Questions on how to interprete AIC and BIC

[crystal feng]

Hi everyone,

I have a question regarding the interpretation of AIC and BIC. Below is the result from my zero inflated Poisson model after fitstat is used.

Measures of Fit for zip of y

Log-Lik Intercept Only: -170698.165 Log-Lik Full Model: -130703.067
D(7524): 261406.133 LR(52): 79990.196
Prob > LR: 0.000
McFadden's R2: 0.234 McFadden's Adj R2: 0.234
ML (Cox-Snell) R2: 1.000 Cragg-Uhler(Nagelkerke) R2: 1.000
AIC: 34.510 AIC*n: 261514.133
BIC: 194194.207 BIC': -79525.680
BIC used by Stata: 261888.516 AIC used by Stata: 261514.133

I understand that the smaller AIC and BIC, the better the model. Compared to the model with other combination of independent variables, this is my smallest AIC and BIC. But is it still too big? Or what we need to care is to just compare AIC/BIC for each set of chosen covariates? Which type of AIC should be reported? And R2?

[Marcos Almeida]

Hi, Crystal,

Please read this text:http://www3.nd.edu/~rwilliam/stats3/L05.pdf

Believe me: it displays and contextualizes all your queries.

Unfortunately, I couldn't identify the author but, judging by the word "rwilliam" in the address, I bet it might well be Richard Williams, a tenured member from this forum.

Hopefull it helps.

Best,

Marcos

[Richard Williams]

First off, based on the format of the output, I am guessing you are using an old version of fitstat. Unless you are using an ancient version of Stata, uninstall fitstat and then do -findit spost13_ado- which has the most current version of fitstat as well as several other excellent programs.

I always use BIC and AIC as ways of comparing alternative models. I don't know of any criteria for saying the lowest values are still too big. If the model is correctly specified, then the BIC and the AIC and the pseudo R^2 are what they are. You'll have to use some other means to assess whether your model is correct, e.g. are there theoretically relevant variables you have excluded?

As for which R^2 to use, McFadden's is the one officially "blessed" by Stata, in that it is the one Stata reports. Fitstat offers several other choices. Paul Allison offers some of his thoughts about the different measures at

http://www.statisticalhorizons.com/r2logistic

[crystal feng]

Hi Marcos and Richard,

Thank you both for valuable thoughts and info! I really appreciate that!

There is another question I want to ask around here, with a hope that someone happens to have some clues. When I tried to use zero-inflated negative binomial model with the same set of variables chosen, the model couldn't converge. My coworker suggested that I could take off some variables, decrease the levels of categorical variables or make the data more balanced in each level for categorical variables. I'm thinking maybe there is not enough zero cases in my data (17%) to run the zero-inflated negative binomial model. Would that also be a potential cause to the convergence issue if the sample size is not big (800 obs) or too many independent variables included (54 vars), so I sacrificed the degree of freedom?

Thanks everyone here to share me your insights! They are all helpful!

Crystal

[Richard Williams]

Crystal, since this is a very different question I would start a new thread on it. My general advice, when a model won't converge, is to simplify it and gradually add more variables. You may then be able to identify variables that are causing you problems.

Also thanks to Marcos for the plug of my handout, which I just now saw. As I gradually revise my handouts I am adding contact info so that people will know where this stuff is coming from.

[Konrad Zdeb]

I'm very happy that this thread appeared. I find fitstat extremely useful. Output-wise it would be amazing if fitstat could be used to nearly summarise different models. My dream syntax would be:

// Data
clear
sysuse census.dta

// Model1
regress death pop poplt5 pop5_17 pop18p pop65p
est sto dummy_one
fitstat

// Model 2
regress death pop poplt5 pop5_17 pop18p pop65p popurban medage marriage divorce
est sto dummy_two

// Fitstat
fitstat, using(dummy_one dummy_two)

[daniel klein]

This is getting away from the topic, but with the save, saving(), diff, and using() options, Konrad's wish seems already fulfilled - theoretically. The example below results in a conformability error, however, indicating some kind of bug, probably.

// Data
clear
sysuse census.dta

// Model1
regress death pop poplt5 pop5_17 pop18p pop65p
fitstat ,save

// Model 2
regress death pop poplt5 pop5_17 pop18p pop65p popurban medage marriage divorce

// Fitstat
fitstat ,diff

Best
Daniel

[crystal feng]

Although it's away from the topic, I'm quite interested to know whether "fitstat, diff" only works for pair comparison.

[Richard Williams]

You can only compare two models at a time, yes.