Testing the Assumptions of Multilevel Models

[Karla Arias]

Hello!
I want to know if Stata has a command to calculate the level 2 residuals. I want to test the assumptions for level 2 of a multilevel model.

Thanks for the help

[Clyde Schechter]

After at least -mixed-, -melogit-, and -mepoisson-, -predict- has an -reffects- option that will do this. I believe this is true of the other -me- commands as well, but I haven't checked or used it personally in those other cases.

For details, type -help mixed postestimation- and click the blue -predict- link.

[Bruce Weaver]

Hello Karla. Clyde has suggested how you can obtain the desired residuals. I'll just add that if you mean that you intend to carry out statistical tests of assumptions, that is generally not a good idea. Take tests of normality as an example.

• H₀ states that the observations (e.g., residuals) are a random sample from a normally distributed population.
• Like other tests, tests of normality have low power when n is low, and higher power as n increases.
• Normality of the errors is most important when n is low, and becomes less and less important as n increases.

Conclusion: Tests of normality fail to detect important departures from normality when n is low, but throw up the red flag far too easily when n is large (and normality of the errors is far less important). Therefore, there is not much point in carrying out such tests. Graphical examinations of the residuals are far more useful, IMO.

HTH.

[Karla Arias]

After at least -mixed-, -melogit-, and -mepoisson-, -predict- has an -reffects- option that will do this. I believe this is true of the other -me- commands as well, but I haven't checked or used it personally in those other cases.

For details, type -help mixed postestimation- and click the blue -predict- link.

Thank you very much, Clyde. I did it!

[Karla Arias]

Hello Karla. Clyde has suggested how you can obtain the desired residuals. I'll just add that if you mean that you intend to carry out statistical tests of assumptions, that is generally not a good idea. Take tests of normality as an example.

• H₀ states that the observations (e.g., residuals) are a random sample from a normally distributed population.
• Like other tests, tests of normality have low power when n is low, and higher power as n increases.
• Normality of the errors is most important when n is low, and becomes less and less important as n increases.

Conclusion: Tests of normality fail to detect important departures from normality when n is low, but throw up the red flag far too easily when n is large (and normality of the errors is far less important). Therefore, there is not much point in carrying out such tests. Graphical examinations of the residuals are far more useful, IMO.

HTH.

Hello Bruce, I understand, I agree. I also prefer graphical examinations.
What would be the graphical analysis for the residuals of level 2? Can you help me please.

[Joseph Coveney]

What would be the graphical analysis for the residuals of level 2?

Run the do-file below and take a look at the resulting Q-Q and P-P plots of the BLUPs. The user manual entries for qnorm and pnorm give you an idea of what to look for in the respective plots.

version 16.1

clear *

set seed `=strreverse("1575692")'

quietly set obs 200
generate int pid = _n
generate double pid_u = rnormal()

quietly expand 10

generate double out = pid_u + rnormal()

mixed out || pid: , nolrtest nolog

*
* Begin here
*
predict double pre, reffects
label variable pre BLUPs

qnorm pre, msize(vsmall) mcolor(black) ///
    ylabel( , angle(horizontal) nogrid) ///
    rlopts(lcolor(black) lpattern(dash))
quietly graph export QQ.png, replace

pnorm pre, msize(vsmall) mcolor(black) ///
    ylabel( , angle(horizontal) nogrid) ///
    rlopts(lcolor(black) lpattern(dash))
quietly graph export PP.png, replace

exit

[Karla Arias]

Run the do-file below and take a look at the resulting Q-Q and P-P plots of the BLUPs. The user manual entries for qnorm and pnorm give you an idea of what to look for in the respective plots.

version 16.1

clear *

set seed `=strreverse("1575692")'

quietly set obs 200
generate int pid = _n
generate double pid_u = rnormal()

quietly expand 10

generate double out = pid_u + rnormal()

mixed out || pid: , nolrtest nolog

*
* Begin here
*
predict double pre, reffects
label variable pre BLUPs

qnorm pre, msize(vsmall) mcolor(black) ///
ylabel( , angle(horizontal) nogrid) ///
rlopts(lcolor(black) lpattern(dash))
quietly graph export QQ.png, replace

pnorm pre, msize(vsmall) mcolor(black) ///
ylabel( , angle(horizontal) nogrid) ///
rlopts(lcolor(black) lpattern(dash))
quietly graph export PP.png, replace

exit

Thank you Joseph!!
I have a question, would it be different if I have a variable as a random effect within the multilevel model?
In such a model, how would you make a residuals vs predicted scatter plot?

[Joseph Coveney]

. . . would it be different if I have a variable as a random effect within the multilevel model?

I'm sorry, I don't understand. Do you mean a random slope? According to the help file for predict after mixed, "By default, BLUPs for all random effects in the model are calculated." So, I believe that it would be exactly the same as I show above, but you will need to specify separate variables for each BLUP prediction, or else specify a variable-name stub. See the help file for more information about how to do that.

. . . how would you make a residuals vs predicted scatter plot?

Here's an example.

version 16.1

clear *

set seed `=strreverse("1575715")'

quietly sysuse auto
summarize rep78, meanonly
quietly replace rep78 = runiformint(r(min), r(max)) if mi(rep78)

xtreg gear_ratio c.trunk, i(rep78) re

*
* Begin here
*
predict double ue, ue // Residuals
predict double xb, xb // Predicted

// Residuals-versus-predicted plot
graph twoway scatter ue xb, mcolor(black) msize(small) ///
    ytitle(Combined Residual) ylabel( , angle(horizontal) format(%03.1f) nogrid) ///
    xtitle(Predicted (Linear Prediction)) ///
        yline(0, lcolor(black) lpattern(dash))

exit

[Karla Arias]

I'm sorry, I don't understand. Do you mean a random slope?]

Yes that's what I mean. I checked it, thanks.


I have a last question, How can I get the predicted values for level 2 in a multilevel model?
Which command can I use?

[Joseph Coveney]

How can I get the predicted values for level 2 in a multilevel model? Which command can I use?

I showed you already. See #6 above. It's the command shown just following the "Begin here" comment in the code block there.

[Karla Arias]

Hello Joseph,

Thank you.
Yes, I get it, my question is how can I get and plot the residuals and predictions for level 2 of a multilevel model. It's because I was reading chapter 2 of Joop Hox's book "Multilevel Analysis (Quantitative methodology series)", he talks about this but I don't know how to do it with Stata.
Let me show you with the following image what I mean:
(The sample size for the example is 2000 students from 100 classes)