ROC curve after a meqrlogit command

[Ida Thagaard]

Hi everybody,

I'm doing a multilevel Mixed effects logistic regression by uing the meqrlogit command. I want to do a ROC curve analysis afterwards but the lroc command is not working. Is there anything I can use instead?
I hope there is someone that can help me,

Regards Ida

[Clyde Schechter]

meqrlogit whatever
predict predicted_probability, mu
roctab actual_outcome predicted_probability

[Ida Thagaard]

Thanks, it works, though the curve looks somewhat strange compared to using the logit command combined with lroc. Do you know why? I'll add the two curves:

merglogit and roctab:


logit and lroc:


Regards Ida

[Ida Thagaard]

Had some problems uploading an image, I'll try again (first the merglogit+roctab then logit+lroc):

[Joseph Coveney]

Thanks, it works, though the curve looks somewhat strange compared to using the logit command combined with lroc. Do you know why?

Why are you using logit on mulitlevel data?

[Ida Thagaard]

I have an binominal outcome. My data is build upon 2600 pregnancies, and some of the women have repeated pregnancies in the dataset. To adjust for that I've moved on from the initial "logit" command to a random effect model (merglogit), with womens Id (mId) as the random effect.

code:
meqrlogit outcome variable, or || mId:, mle

I'm somewhat confused since the random effect is insignificant and the models are performing quite different (see the above curves). Since the effect is insignificant I though I could use the more simple model; logit, but maybe I'm wrong? I've also expected the models to perform simmilar, due to that, and when they are not I'm further confused. Tips on how to move on will be greatly appreciated.

Regards Ida

[Ida Thagaard]

I think I found the problem, Clyde suggested med to code:

predict predicted_probability, mu

Why mu? When I use xb it looks like a normal ROC curve.

[Joseph Coveney]

predict mu, mu includes the individual random effect; predict xb, xb doesn't. Both of what you show above look like "normal" ROC curves in the sense that both are typical of what you can expect for ROC curves.

I'm not sure what facet of your research question drives the need for an ROC curve, but from your description of the dataset, have you considered modeling the outcome as, say, a count variable instead of a multilevel binomial?

[Bruno Moita]

Hi everybody,

I'm struggling to find some references on C statistic (AUC) appropriatenness to evaluate xtmelogit predictive ability.

As far as I get, using roctab,

predict mu, mu

will produce C statistic accounting for random-effects and

predict xb, xb

will produce

C statistic for the fixed-effects only.

Once unobserved level variance is considered in the model, does AUC produce valuable information on predictive ability of the model?
Is AUC for fixed-effects only (xb option) comparable with a traditional logit model?


Thanks in advance for any help on this.

[Clyde Schechter]

Once unobserved level variance is considered in the model, does AUC produce valuable information on predictive ability of the model?

I suppose "valuable" is in the eyes of the beholder. But because the random intercepts are unobservable, predictions that include them cannot be actually applied in the real world. So if by "valuable" you mean "applicable to the real world," the answer is no: the AUC is telling you about the discriminatory power of a hypothetical predictor that can never actually be assessed.

Is AUC for fixed-effects only (xb option) comparable with a traditional logit model?

It depends on what you mean. If you do a multi-level model, and then do outcome predictions based only on the fixed effects, you are creating a prediction rule, and you can quantify its AUC. You can do the same with a one-level logit model. The two models are both legitimate. And if you wanted to say that one of them has a higher AUC than the other, or something like that, you could. In some instances the two outcome prediction approaches might even match most of the time; in other cases not. But basically you can compare these to the same extent and in the same ways you can compare the AUC's from any two different prediction rules.