[Steve Samuels]

I'll just ask both of you to pleasue put results, as well as code, between CODE delimiters, which is what FAQ 12 asks us to do. The results you show are so faint that I find them difficult to read. Thanks.

[Man Yang]

Hi Steve, sorry about the inconvenience. Please see below for my code and model output. Basically I want to know what does insignificant linear and quadratic effect of time mean in this situation and whether it impacts my postestimation.

 meologit teval_selfesteem c.agec##c.agec i.retain0w || StudentID:, pw(wt_sa1) vce(cluster Cluster)

Fitting fixed-effects model:

Iteration 0:   log likelihood = -1453334.6  
Iteration 1:   log likelihood =   -1451704  
Iteration 2:   log likelihood = -1451703.7  
Iteration 3:   log likelihood = -1451703.7  

Refining starting values:

Grid node 0:   log likelihood = -1446036.9

Fitting full model:

Iteration 0:   log pseudolikelihood = -1446036.9  
Iteration 1:   log pseudolikelihood = -1434064.4  
Iteration 2:   log pseudolikelihood = -1433544.8  
Iteration 3:   log pseudolikelihood = -1433542.2  
Iteration 4:   log pseudolikelihood = -1433542.2  

Mixed-effects ologit regression                 Number of obs     =      2,240
Group variable:       StudentID                 Number of groups  =      1,126

                                                Obs per group:
                                                              min =          1
                                                              avg =        2.0
                                                              max =          3

Integration method: mvaghermite                 Integration pts.  =          7

                                                Wald chi2(3)      =       1.22
Log pseudolikelihood = -1433542.2               Prob > chi2       =     0.7473
                                  (Std. Err. adjusted for 207 clusters in Cluster)
----------------------------------------------------------------------------------
                 |               Robust
teval_selfesteem |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-----------------+----------------------------------------------------------------
            agec |  -.0181226   .0521687    -0.35   0.728    -.1203714    .0841261
                 |
   c.agec#c.agec |   .0048526    .015294     0.32   0.751    -.0251231    .0348283
                 |
      1.retain0w |   .2112243   .2292859     0.92   0.357    -.2381677    .6606163
-----------------+----------------------------------------------------------------
           /cut1 |   -.479159   .1513409    -3.17   0.002    -.7757818   -.1825362
           /cut2 |   2.377513   .2279411    10.43   0.000     1.930757     2.82427
-----------------+----------------------------------------------------------------
StudentID        |
       var(_cons)|   1.073752   .4004319                      .5169723    2.230183
----------------------------------------------------------------------------------

[Weiwen Ng]

... Basically I want to know what does insignificant linear and quadratic effect of time mean in this situation and whether it impacts my postestimation.

 meologit teval_selfesteem c.agec##c.agec i.retain0w || StudentID:, pw(wt_sa1) vce(cluster Cluster)

...
----------------------------------------------------------------------------------
| Robust
teval_selfesteem | Coef. Std. Err. z P>|z| [95% Conf. Interval]
-----------------+----------------------------------------------------------------
agec | -.0181226 .0521687 -0.35 0.728 -.1203714 .0841261
|
c.agec#c.agec | .0048526 .015294 0.32 0.751 -.0251231 .0348283
|
1.retain0w | .2112243 .2292859 0.92 0.357 -.2381677 .6606163
-----------------+----------------------------------------------------------------
/cut1 | -.479159 .1513409 -3.17 0.002 -.7757818 -.1825362
/cut2 | 2.377513 .2279411 10.43 0.000 1.930757 2.82427
-----------------+----------------------------------------------------------------
StudentID |
var(_cons)| 1.073752 .4004319 .5169723 2.230183
----------------------------------------------------------------------------------

Man, from the fixed effects, it looks like the change in probability of response is pretty much constant over time. We don't see your model with just a linear term for time, and we also don't know if the trajectories over time are diverse enough that you might want to fit a random slope for time and see what happens. But as I said, it looks like the mean effect of age is pretty much flat.

margins, at(agec = 20(5)60))
marginsplot, bydimension(_predict)

I am pretty sure that just the plain -marginsplot- command, with no other syntax, will suffice for this example. I am not so familiar with that command's syntax, so anything more advanced and you will want to investigate yourself. But that will enable you to visualize how the probability of a higher response changes over time. If you want to see predictions for just one response category, I would defer to someone more familiar with marginsplot, but I think you can tell margins to just predict one of the outcomes, e.g.

margins, at(agec = 20(5)60)) predict(outcome(1))
marginsplot

In case this turns out to be helpful, I'm adding a link to a primer on margins by our very own Richard Williams.

[Man Yang]

Hi Weiwen, thanks for your response. I understand what you said above that the "the change in probability of response is pretty much constant over time". I used margins and marginsplot to see the predicted probability of outcome1 but the plot actually shows a curve-like trend. Is it because that I include the quadratic term in the model? Please see below

margins retain0w, at(agec = (-5.22(1)4.78)) predict(outcome(1)) atmeans
marginsplot, noci recast(line) ///
    xlabel(-5.22 "7" -4.22 "8" -3.22 "9" -2.22 "10" -1.22 "11" -0.22 "12" 0.78 "13" 1.78 "14" 2.78 "15" 3.78 "16" 4.78 "17") ///
    ytitle("Predicted probability of never having" "low self-esteem by teacher evaluation", size(medsmall)) ///
    xtitle("Student age", size(medsmall)) title("")

Below is the marginsplot

So then I fit in a meologit model with only the linear effect

. meologit teval_selfesteem c.agec i.retain0w || StudentID:, pw(wt_sa1) vce(cluster Cluster)

Fitting fixed-effects model:

Iteration 0:   log likelihood = -1453334.6  
Iteration 1:   log likelihood = -1451740.4  
Iteration 2:   log likelihood = -1451740.1  
Iteration 3:   log likelihood = -1451740.1  

Refining starting values:

Grid node 0:   log likelihood = -1446121.2

Fitting full model:

Iteration 0:   log pseudolikelihood = -1446121.2  
Iteration 1:   log pseudolikelihood = -1434199.2  
Iteration 2:   log pseudolikelihood = -1433665.6  
Iteration 3:   log pseudolikelihood = -1433662.8  
Iteration 4:   log pseudolikelihood = -1433662.8  

Mixed-effects ologit regression                 Number of obs     =      2,240
Group variable:       StudentID                 Number of groups  =      1,126

                                                Obs per group:
                                                              min =          1
                                                              avg =        2.0
                                                              max =          3

Integration method: mvaghermite                 Integration pts.  =          7

                                                Wald chi2(2)      =       1.12
Log pseudolikelihood = -1433662.8               Prob > chi2       =     0.5702
                                  (Std. Err. adjusted for 207 clusters in Cluster)
----------------------------------------------------------------------------------
                 |               Robust
teval_selfesteem |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-----------------+----------------------------------------------------------------
            agec |  -.0191086   .0511663    -0.37   0.709    -.1193927    .0811756
      1.retain0w |   .2098385   .2290332     0.92   0.360    -.2390584    .6587355
-----------------+----------------------------------------------------------------
           /cut1 |  -.5027532   .1331468    -3.78   0.000    -.7637161   -.2417902
           /cut2 |   2.352087   .2015875    11.67   0.000     1.956983    2.747192
-----------------+----------------------------------------------------------------
StudentID        |
       var(_cons)|    1.06942   .3985173                      .5151687    2.219969
----------------------------------------------------------------------------------

The output shows that the linear effect of time is also insignificant. So, which model should I use if I need to examine the trajectories of predicted probability of the outcome (1,2 3)?

[Matt Warkentin]

Edit: I responded to an older message, I did not see that this had carried onto the second page.

[Man Yang]

Hi Mat, I didn't see your response in the previous page. Could you point to me where is it? Thanks.

[Weiwen Ng]

Hi Weiwen, thanks for your response. I understand what you said above that the "the change in probability of response is pretty much constant over time". I used margins and marginsplot to see the predicted probability of outcome1 but the plot actually shows a curve-like trend. Is it because that I include the quadratic term in the model? Please see below

margins retain0w, at(agec = (-5.22(1)4.78)) predict(outcome(1)) atmeans
marginsplot, noci recast(line) ///
xlabel(-5.22 "7" -4.22 "8" -3.22 "9" -2.22 "10" -1.22 "11" -0.22 "12" 0.78 "13" 1.78 "14" 2.78 "15" 3.78 "16" 4.78 "17") ///
ytitle("Predicted probability of never having" "low self-esteem by teacher evaluation", size(medsmall)) ///
xtitle("Student age", size(medsmall)) title("")

Below is the marginsplot
[ATTACH=CONFIG]n1427452[/ATTACH]

So then I fit in a meologit model with only the linear effect

. meologit teval_selfesteem c.agec i.retain0w || StudentID:, pw(wt_sa1) vce(cluster Cluster)

Fitting fixed-effects model:

Iteration 0: log likelihood = -1453334.6
Iteration 1: log likelihood = -1451740.4
Iteration 2: log likelihood = -1451740.1
Iteration 3: log likelihood = -1451740.1

Refining starting values:

Grid node 0: log likelihood = -1446121.2

Fitting full model:

Iteration 0: log pseudolikelihood = -1446121.2
Iteration 1: log pseudolikelihood = -1434199.2
Iteration 2: log pseudolikelihood = -1433665.6
Iteration 3: log pseudolikelihood = -1433662.8
Iteration 4: log pseudolikelihood = -1433662.8

Mixed-effects ologit regression Number of obs = 2,240
Group variable: StudentID Number of groups = 1,126

Obs per group:
min = 1
avg = 2.0
max = 3

Integration method: mvaghermite Integration pts. = 7

Wald chi2(2) = 1.12
Log pseudolikelihood = -1433662.8 Prob > chi2 = 0.5702
(Std. Err. adjusted for 207 clusters in Cluster)
----------------------------------------------------------------------------------
| Robust
teval_selfesteem | Coef. Std. Err. z P>|z| [95% Conf. Interval]
-----------------+----------------------------------------------------------------
agec | -.0191086 .0511663 -0.37 0.709 -.1193927 .0811756
1.retain0w | .2098385 .2290332 0.92 0.360 -.2390584 .6587355
-----------------+----------------------------------------------------------------
/cut1 | -.5027532 .1331468 -3.78 0.000 -.7637161 -.2417902
/cut2 | 2.352087 .2015875 11.67 0.000 1.956983 2.747192
-----------------+----------------------------------------------------------------
StudentID |
var(_cons)| 1.06942 .3985173 .5151687 2.219969
----------------------------------------------------------------------------------

The output shows that the linear effect of time is also insignificant. So, which model should I use if I need to examine the trajectories of predicted probability of the outcome (1,2 3)?

Man, the margins output is showing what your model predicts. You modeled a linear + quadratic term, so yes, the predicted probabilities as a function of age look curved. If you remove the -noci- option, though, I bet you'll see that the CIs mostly overlap. Also note your y-axis: in 7 years, the predicted probability of low self-esteem rises by a bit over 4 percentage points for the average student (the margins plot sets the random effect to 0, I think). A 4-5 percentage point change from 36 to 41% doesn't seem huge - or it may be large given theory, I don't know.

Also worth remembering: the model told you that a one-year change in age and age-squared don't have a significant effect. Over the range of actual student ages, the effect may be more meaningful. I think you should be able to conduct a likelihood ratio test on a model with just a linear term for age versus your current model (rejection means the less restricted model, i.e. your current one, fits better; failure to reject means there's no evidence either fits better).

And, you asked about trajectories. One way to do that is by putting a random slope in the model. If the variance of the random slope is very small, then you have no case for using a random slope model. You can also conduct an LR test versus the random intercept-only model, but I recall that the test statistic doesn't follow a traditional chi-sq distribution.

FYI, you didn't show us the model chi-bar test, which is a likelihood ratio test of your model vs an identical model without a random intercept. But I bet it is significant (because the 95% CI for the intercept variance doesn't include 0), which means that you are justified in fitting a random intercept.

[Man Yang]

Hi Weiwen, thanks so much for your detailed explanation. I tried to run the LR test but -lrtest- won't work when robust vce is specified in the option, which is true to my case. Say let's assume my current model fits better and to summarize what you said above, my current model (linear + quadratic) could be justified and whether the model can be justified has nothing to do with whether the linear and quadratic term is significant or not. Correct? In addition, my marginsplot is correctly displayed the trajectories of the predicted probability of outcome1 according to the model, right?

[Weiwen Ng]

Hi Weiwen, thanks so much for your detailed explanation. I tried to run the LR test but -lrtest- won't work when robust vce is specified in the option, which is true to my case. Say let's assume my current model fits better and to summarize what you said above, my current model (linear + quadratic) could be justified and whether the model can be justified has nothing to do with whether the linear and quadratic term is significant or not. Correct? In addition, my marginsplot is correctly displayed the trajectories of the predicted probability of outcome1 according to the model, right?

Your marginsplot definitely displayed the trajectories of predicted probabilities according to your modelBut remember, that's according to your model. Had you fit a model with only a linear term, your marginsplot would show you a straight line. That could be the wrong model! We don't know for sure.

Also, I notice your syntax said to predict outcome = 1. I haven't personally fit an ordered logit model in a while, but you had /cut1 and /cut2 in the output, so I assume you are coding the outcome as 0, 1, 2. Perhaps it's worth checking, but does that correspond to never having low teacher-evaluated self esteem?

I forgot about the robust vce. I am not sure what test to use to select between models, so I would prefer to defer to others. You might think about using -lincom- to test the predicted probabilities at say, age 14 vs age 7, though.