Model fit issues in multi-level logistic regression

[You Zhang]

Hi I am ran into some issues with model fit in multi-level logit regresstion. I am using melogit to analyze a large cross-national dataset. There are two variables that are consistently significant across models tried, however, using BIC to assess model fit, I find that these two variables do not improve model fit, BIC actually increased slightly. I wonder if I should keep these two variables? Conceptually, I think it can be justified to keep these two variables, but I am not sure why BIC increased...

I would appreciate it if anyone could offer some insights on this. Thanks!

[Clyde Schechter]

Without seeing the actual -melogit- output, it is hard to be specific, but I can make some general remarks about this kind of situation.

The BIC (and the AIC) are basically log-likelihoods penalized for the number of model degrees of freedom. (They differ in the exact nature of the penalty.) So if you add two variables to the model, the log likelihood will go up, but so will the model df, and hence so will the penalty. The situation you are facing is that the penalty for adding two more predictors is greater than the improvement in log likelihood, so, you see a net increase in BIC. This is precisely the reason for using something like BIC in the first place. Statistical significance is a dubious guide to identifying which variables to include in a model. Even if you want to ignore the, in my opinion excellent, advice of the American Statistical Association to abandon the use of the concept altogether,* it really isn't appropriate to use it for this purpose. In large samples, a variable that makes trivial contributions to outcome prediction can be statistically significant. My best guess is that this is the situation you are facing: a very large estimation sample, with two variables that do almost nothing but qualify as "significant." It is precisely such situations that AIC and BIC are designed to detect.

One somewhat pedantic point, but I think it important to make. Neither statistical significance, nor BIC (nor AIC) is a measure of model fit. If you want a measure of goodness of fit, in a one-level model you would use something like the Hosmer-Lemeshow statistic for calibration and an ROC area for discrimination. These approaches encounter some conceptual difficulties with multi-level modeling, i.e. do you want to give the model "credit" for the fit accounted for by the random intercepts? But you can still calculate and plot predicted and observed values in groups (deciles of predicted risk, or other salient subsets) with or without the contribution of the random effects, and examine how well they correspond.

Opinions will vary on what you should do here. "Significance" says include the variables, BIC says not to, and conceptually there is justification for including them. My own approach is to always ignore significance when selecting variables, and to give priority to conceptual/theoretical concerns over AIC/BIC or other statistics, providing the theory is a reasonably well established one or has strong face validity. I would also be persuaded by things like the impact on actual fit statistics of including the variables. And if the sample is relatively small (which, I suspect, is not your situation) another useful approach is to use the models (model with and model without) to generate large samples of simulated ("fake") data and explore those to see which model produces data that are more realistic and contain fewer simulated observations that are simply implausible.

*See https://www.tandfonline.com/doi/full...5.2019.1583913 for the "executive summary" and https://www.tandfonline.com/toc/utas20/73/sup1 for all 43 supporting articles. Or https://www.nature.com/articles/d41586-019-00857-9 for the tl;dr.