model fit statistics using mi estimate

[Traci Wike]

Hi all, when estimating model fit for a multilevel model using multiply imputed data and the command mi estimate, what fit statistics should be/can be used to assess model fit?

[Traci Wike]

anyone?

[Stephen Jenkins]

Please read the Forum FAQ. It provides discussion and recommendations for how to formulate questions effectively in order to maximize the chances of getting a helpful response. Simply re-posting is unlikely to help (and may simply annoy some people)

[Nick Cox]

Traci: Stephen provides very good advice.

I have to guess that your post got no takers because it is both too broad and too vague. You might get more takers if you showed specific results. (I wouldn't be answering, as the question is way outside my experience.)

[Traci Wike]

Thanks for the advice--I apologize for being annoying, messages just get buried quickly in such an active forum. Let me try again.

I am conducting an intervention study using school-based data. My analysis model is a three-level HLM model (time, individuals, schools). I used multiple imputation to handle missingness, which means that my syntax needs to include the mi estimate prefix in order to account for the 10 imputations and get pooled estimates. Because it is impossible to get log-likelihoods for pooled estimates resulting from the mi procedures, I cannot use typical fit statistics like likelihood ratio tests to inform the building/fit of my models. the estat post estimation commands useful for xtmixed also aren't available. I can't find any information about how to monitor fit statistics. I'm wondering if anyone else has encountered this and if they have any information/advice.

Thanks,
Traci

[Rich Goldstein]

thanks for the clarification; below is code from a do file I use to get overall pseudo-r-squared and overall area under the ROC curve with mi data; you should be able to adapt this to get what you want:

local rhs "armg2 armg3 tbsaburn20 tbsaburn21"
noi mi estimate, or saving(miest, replace): logistic hodc `rhs', vce(cluster site)
qui mi query
local M=r(M)
scalar r2=0
scalar cstat=0
qui mi xeq 1/`M': logistic hodc `rhs'; scalar r2=r2+e(r2_p); lroc, nog; scalar cstat=cstat+r(area)
scalar r2=r2/`M'
scalar cstat=cstat/`M'
noi di "Pseudo R=squared over imputed data = " r2
noi di "C statistic over imputed data = " cstat

[Traci Wike]

thanks, Rich

[Rich Goldstein]

you're welcome - and I forgot to mention that Stata tech support helped me with the basic idea

[Jean Kiely]

Rich Goldstein, I am similarly trying to assess model fit and AUC after mi estimate, and, after using your code, I keep getting an error saying "invalid numlist." I assume that is in reference to the r(area), as I did not save area prior. But I am not sure a) if that is the problem, and b) how to remedy it either way. Any thoughts?

Thanks!

[Rich Goldstein]

please post the exact code you used (in a code block - see the FAQ) and maybe someone can give you an answer

[Jean Kiely]

[local model1 "age_yrs mech1 transfer edsbp edgcsmr edpulse mxhdaisX mxextremaisX congenital"
mi estimate, or saving(miest, replace): logit died `model1'
local M = r(M)
scalar cstat = 0
qui mi xeq 1/`M': logistic hodc `model1'; scalar r2=r2+e(r2_p); lroc, nog; scalar cstat=cstat+r(area)
scalar r2=r2/`M'
scalar cstat=cstat/`M'
noi di "Pseudo R=squared over imputed data = " r2
noi di "C statistic over imputed data = " cstat]

[Jean Kiely]

Richard Gloldstein, I literally tried to copy character for character, and still failed. :/

[Rich Goldstein]

first, you did not put your code in a code block as requested - see section 12 of the FAQ

second, there are lines in my code (see #6) that do not appear to be in your code - and your code as shown in #11 will not work without those missing lines

[Annabel Mullin]

Hi, Rich Goldstein I have read your syntax on this matter and thought it very useful.

I tried to use your code (no.6) but without success but I am running an mlogit, would that explain Stata's disquiet? - my code below, I hope in the appropriate manner:

local esmale 'desist2_163034 i.siblings_5 i.readmost_5 i.afterschool_10 i.fatherrole_10 ///
i.familytime_10 i.parentsideas_10 i.birthweight_0 i.socialclass_0 i.smoking2_0 i.breastfeeding_5 ///
i.peducation_5 i.rutter_5 i.malaisemothers_5 truants_10 if a0376==3 & a0248_0==1 & sex2==1, base(0)'

noi mi estimate, or saving(miest, replace): mlogit desist2_163034 `esmale'
qui mi query
local M=r(M)
scalar r2=0
scalar cstat=0
qui mi xeq 1/`M': mlogit desist2_163034 'esmale'; scalar r2=r2+e(r2_p); lroc, nog; scalar cstat=cstat+r(area)
scalar r2=r2/`M'
scalar cstat=cstat/`M'
noi di "Pseudo R=squared over imputed data = " r2
noi di "C statistic over imputed data = " cstat

[Annabel Mullin]

Hi, Rich Goldstein I have read your syntax on this matter and thought it very useful.

I tried to use your code (no.6) but without success but I am running an mlogit, would that explain Stata's disquiet? - my code below, I hope in the appropriate manner:

local esmale 'desist2_163034 i.siblings_5 i.readmost_5 i.afterschool_10 i.fatherrole_10 ///
i.familytime_10 i.parentsideas_10 i.birthweight_0 i.socialclass_0 i.smoking2_0 i.breastfeeding_5 ///
i.peducation_5 i.rutter_5 i.malaisemothers_5 truants_10 if a0376==3 & a0248_0==1 & sex2==1, base(0)'

noi mi estimate, or saving(miest, replace): mlogit desist2_163034 `esmale'
qui mi query
local M=r(M)
scalar r2=0
scalar cstat=0
qui mi xeq 1/`M': mlogit desist2_163034 'esmale'; scalar r2=r2+e(r2_p); lroc, nog; scalar cstat=cstat+r(area)
scalar r2=r2/`M'
scalar cstat=cstat/`M'
noi di "Pseudo R=squared over imputed data = " r2
noi di "C statistic over imputed data = " cstat