Iteration not concave problem negative binomial regression

[Guy Swillens]

Dear Statalisters,

I am currently running various xtnbreg regressions on my data for my master thesis. I did not have any problems with most of my regressions (I did several models with various IVs and interactions already). However, now with my last interaction effect I get the not concave message on stata. Oddly, I only get this message when analyzing a particular interaction, even though I do not get this message when I interact with different independent variables.

To illustrate, when I interact the variable (called Repeatedalliance) with IV #1 (called Knowledge similarity), I get the not concave message. However, when I interact the variable (Repeatedalliance) with IV #2 (called Knowledge complementarity), there are no problems whatsoever. Also when I include the Repeatedalliance variable solely as a IV, I get no problems. Thus, only when Repeatedalliance interacts with Knowledge similarity, the not concave problem appears.

Does anybody know how this is possible? and which steps to undertake to solve it?

Any help is much appreciated

Guy Swillens

[Clyde Schechter]

Well, unlike simple linear and one-level logit models, negative binomial regression (and all the more with the panel data version) has a somewhat difficult likelihood function to maximize. If you are getting the "not concave" message along the way, but the model ultimately converges and it does not say "not concave" at the final iteration, then you can just ignore this. If it is still saying "not concave" when it stops, or if it is in a loop and making no progress, then you have to try to help Stata out. There are a few things you can try.

1. You can specify the -difficult- option. This will cause Stata to change its choice of steps during the iteration. Sometimes this does the trick, sometimes not.

2. You can try specifying a -technique()- option. This will cause Stata to use some other technique than the default Newton-Raphson maximization algorithm. The various techniques you can specify are shown at help maximize##algorithm_spec. I have no advice about when which one works best.

3. If you're doing -xtnbreg, re-, then you can estimate the same model with -menbreg-, which has a different algorithm. Also -menbreg- itself can be

4. If you're doing -xtnbreg, fe-, there is no other command that estimates the same model, but you could try -xtpoisson, fe- and then feed the results of that as starting values to --xtnbreg, fe-. Sometimes that's close enough to correct to get Stata out of a ditch. You could also try using the results from -nbreg- as starting results for -xtnbreg, fe- and see if that helps Stata reach a good conclusion.

5. Another approach is to run your -xtnbreg- with the -iterate()- option specified, choosing a number of iterations that will get Stata to stop shortly after you enter the "not concave" region. You will then get a display of interim results. While those results are not valid, they can sometimes show that some other variable is behaving bizarrely. In that case, removing that variable from the model and starting over may solve the problem.

If none of these works, it is possible that the model including that interaction simply is not identified by your data, which is a technical way of saying that the information in your data is equally compatible with several different sets of regression coefficients. This situation is somewhat analogous to collinearity. (Colinearity is a special case of unidentifiability; it is a simple case in that it is easy enough to recognize that Stata automatically detects it and fixes it for you. Unidentifiability, in general, can arise in other circumstances where the data simply are equally compatible with multiple solutions.) If this is the situation you face, you need to simplify your model and remove one or more variables.

[Guy Swillens]

Thank you very much for your reply. The -difficult- option seems to work. Does this mean that I would have to specify the -difficult- option in all other models as well? I reran some of the analyses that worked, but now with the -difficult- option, and the coefficients are slightly different.

[Clyde Schechter]

Yes, -difficult- can sometimes lead to slightly different results. But the differences should be too small to be of any practical importance. I would not bother re-running them just to harmonize the commands. And you could run into another problem: sometimes specifying the -difficult- problem will, itself, lead to convergence problems in a model that did just fine without it! So I only use it when I'm running into the "not concave" problem.

[Guy Swillens]

The -difficult- option does not seem to work for a different mode in which I get the not concave error. I really do not understand this whole problem. When I want to include only my control variables, I get the not concave problem. When I add one independent variable, I do not get the error. Also, sometimes I get the error while if I change the sequence of the variables, I do not get the error. It is quite frustrating..

Concerning the one model that does not work now, I tried the menbreg, and this works. Should I do the menbreg for all the models then? Or would your #4 post be applicable in this situation as well?

[Clyde Schechter]

The model estimated by -menbreg- for two levels is the same model estimated by -xtnbreg-, so you an interchange results from those two models freely. They differ only in the numeric approach to estimating.

The multilevel negative binomial likelihood is just difficult to work with and there is, to my knowledge, no simple characterization of which data sets go smoothly and which cause problems.

[Guy Swillens]

Dear Mr Schechter,

Is there a Stata manual where I can find these options? or a manual on why one has to use these options? I cannot seem to find it.

For my thesis, I have to explain why I had to specify the -menbreg- option instead of -xtnbreg-.

regards

Guy Swillens

[Chris Jiao]

Well, unlike simple linear and one-level logit models, negative binomial regression (and all the more with the panel data version) has a somewhat difficult likelihood function to maximize. If you are getting the "not concave" message along the way, but the model ultimately converges and it does not say "not concave" at the final iteration, then you can just ignore this. If it is still saying "not concave" when it stops, or if it is in a loop and making no progress, then you have to try to help Stata out. There are a few things you can try.

1. You can specify the -difficult- option. This will cause Stata to change its choice of steps during the iteration. Sometimes this does the trick, sometimes not.

2. You can try specifying a -technique()- option. This will cause Stata to use some other technique than the default Newton-Raphson maximization algorithm. The various techniques you can specify are shown at help maximize##algorithm_spec. I have no advice about when which one works best.

3. If you're doing -xtnbreg, re-, then you can estimate the same model with -menbreg-, which has a different algorithm. Also -menbreg- itself can be

4. If you're doing -xtnbreg, fe-, there is no other command that estimates the same model, but you could try -xtpoisson, fe- and then feed the results of that as starting values to --xtnbreg, fe-. Sometimes that's close enough to correct to get Stata out of a ditch. You could also try using the results from -nbreg- as starting results for -xtnbreg, fe- and see if that helps Stata reach a good conclusion.

5. Another approach is to run your -xtnbreg- with the -iterate()- option specified, choosing a number of iterations that will get Stata to stop shortly after you enter the "not concave" region. You will then get a display of interim results. While those results are not valid, they can sometimes show that some other variable is behaving bizarrely. In that case, removing that variable from the model and starting over may solve the problem.

If none of these works, it is possible that the model including that interaction simply is not identified by your data, which is a technical way of saying that the information in your data is equally compatible with several different sets of regression coefficients. This situation is somewhat analogous to collinearity. (Colinearity is a special case of unidentifiability; it is a simple case in that it is easy enough to recognize that Stata automatically detects it and fixes it for you. Unidentifiability, in general, can arise in other circumstances where the data simply are equally compatible with multiple solutions.) If this is the situation you face, you need to simplify your model and remove one or more variables.

Hi Clyde,

I was also using negative binomial regression to estimate the panel data and encountered similar issues.

Thanks for your suggestions but I still have some curiosities.

1. Could you give me a detailed illustration about how to use -difficult- and -technique-, like how to re-write the following code.
e.g., my current -xtnbreg, re- code is

asdoc xtnereg $ylist $xlist $controls, re save(MyFile) cnames(Model 1) dec(4) replace

2. I refer to the STATA manual of -menbreg- and run the code. But the result seems highly like to the result of -xtnbreg, fe-. Do you know how to manually assign random effect for -menereg- by adding additional options, like assigning more weights of random effects in the model?

Thanks in advance.

[Clyde Schechter]

First, it's -xtnbreg-, not -xtnereg-.

To specify the difficult option, you just have to write -difficult- in the option list. There's nothing more to it than that.

For the -technique()- option you have to choose a specific algorithm. You can find detailed information at -help maximize##algorithm_spec-. If the information there is not sufficient to help you, try going to the sections of the PDF documentation that are linked there (in blue).

While you are working on trying to get your model to converge, I suggest removing the -asdoc- prefix and associated options. While it probably isn't hurting anything, there's no reason to add complications to a difficult picture. Once you have the model converging, then re-run it with -asdoc- to get the exportation to Word.

I refer to the STATA manual of -menbreg- and run the code. But the result seems highly like to the result of -xtnbreg, fe-.

I'm not sure what you're trying to say here, but -menbreg- is a random effects model estimator.

Do you know how to manually assign random effect for -menereg- by adding additional options, like assigning more weights of random effects in the model?

Here I truly have no idea what you are asking.

With regard to both of the last two, if you are getting results from -menbreg- that don't make sense to you, why not just post the code you are using and the results Stata is giving you? That might make it possible to help you.

[Abdul Tj]

Hi Clyde,

I am having a similar problem, but in my case, I am using the xtnbreg fe. How can I implement solution 4?

Thanks

[Joseph Coveney]

. . . I am using the xtnbreg fe. How can I implement solution 4?

xtnbreg . . ., i(. . .) fe →
xtnbreg . . ., i(. . .) fe difficult

[Rukman Mecca]

Dear All,
I am currently running a GLM Model binomial family with log function with eform to find risk ratio on my data. As said in the above thread, I did not have problems with most of my regressions. Only in one model, I am getting "the not concave message".
From some other statalist thread, I understood the problem can arise when there are limited observations in certain categories (for example zero observations in one of the categories). However in my case, this issue is coming with two variables which are continuous. Applying technique(bhhh) didnt help either.

glm ferritin_deficiency_category i.(group wealth_quintile) length_baseline bmi_score, family(binomial) link(log) eform

.

Iteration 0: log likelihood = -495.44349 (not concave)
Iteration 1: log likelihood = -484.26282 (not concave)
Iteration 2: log likelihood = -483.94068 (not concave)
Iteration 3: log likelihood = -483.93341 (not concave)
Iteration 4: log likelihood = -483.93323 (not concave)
Iteration 5: log likelihood = -483.93321 (not concave)
Iteration 6: log likelihood = -483.93043 (not concave)
Iteration 7: log likelihood = -483.92874 (not concave)
Iteration 8: log likelihood = -483.92872 (not concave)
Iteration 9: log likelihood = -483.92871 (not concave)
Iteration 10: log likelihood = -483.92818 (not concave)
Iteration 11: log likelihood = -483.92763 (not concave)
Iteration 12: log likelihood = -483.92758 (not concave)
Iteration 13: log likelihood = -483.92756 (not concave)
Iteration 14: log likelihood = -483.92755 (not concave)
Iteration 15: log likelihood = -483.92753 (not concave)
Iteration 16: log likelihood = -483.92751 (not concave)
Iteration 17: log likelihood = -483.9275 (not concave)
Iteration 18: log likelihood = -483.92748 (not concave)
Iteration 19: log likelihood = -483.9269 (not concave)
Iteration 20: log likelihood = -483.92638 (not concave)
Iteration 21: log likelihood = -483.92635 (not concave)
Iteration 22: log likelihood = -483.92633 (not concave)
Iteration 23: log likelihood = -483.92632 (not concave)
Iteration 24: log likelihood = -483.9263 (not concave)
Iteration 25: log likelihood = -483.92628 (not concave)
Iteration 26: log likelihood = -483.92567 (not concave)
Iteration 27: log likelihood = -483.92517 (not concave)
Iteration 28: log likelihood = -483.92515 (not concave)
Iteration 29: log likelihood = -483.92513 (not concave)
Iteration 30: log likelihood = -483.92511 (not concave)
Iteration 31: log likelihood = -483.92479 (not concave)
Iteration 32: log likelihood = -483.92455 (not concave)
Iteration 33: log likelihood = -483.92453 (not concave)
Iteration 34: log likelihood = -483.92449 (not concave)
Iteration 35: log likelihood = -483.92445 (not concave)
Iteration 36: log likelihood = -483.92428 (not concave)
Iteration 37: log likelihood = -483.92416 (not concave)
Iteration 38: log likelihood = -483.92407 (not concave)
Iteration 39: log likelihood = -483.92399 (not concave)
Iteration 40: log likelihood = -483.92374 (not concave)
Iteration 41: log likelihood = -483.92357 (not concave)
Iteration 42: log likelihood = -483.92354 (not concave)
Iteration 43: log likelihood = -483.92298 (not concave)
Iteration 44: log likelihood = -483.92245 (not concave)
Iteration 45: log likelihood = -483.92241 (not concave)
Iteration 46: log likelihood = -483.9224 (not concave)
Iteration 47: log likelihood = -483.92238 (not concave)
Iteration 48: log likelihood = -483.92181 (not concave)
Iteration 49: log likelihood = -483.92129 (not concave)
Iteration 50: log likelihood = -483.92125 (not concave)
Iteration 51: log likelihood = -483.92124 (not concave)
Iteration 52: log likelihood = -483.92122 (not concave)
Iteration 53: log likelihood = -483.9212 (not concave)
Iteration 54: log likelihood = -483.92066 (not concave)
Iteration 55: log likelihood = -483.92012 (not concave)
Iteration 56: log likelihood = -483.92008 (not concave)
Iteration 57: log likelihood = -483.92006 (not concave)
Iteration 58: log likelihood = -483.92004 (not concave)
Iteration 59: log likelihood = -483.92003 (not concave)
Iteration 60: log likelihood = -483.92001 (not concave)
Iteration 61: log likelihood = -483.92 (not concave)
Iteration 62: log likelihood = -483.91998 (not concave)
Iteration 63: log likelihood = -483.91937 (not concave)
Iteration 64: log likelihood = -483.91887 (not concave)
Iteration 65: log likelihood = -483.91885 (not concave)
Iteration 66: log likelihood = -483.91883 (not concave)
Iteration 67: log likelihood = -483.9188 (not concave)
Iteration 68: log likelihood = -483.91855 (not concave)
Iteration 69: log likelihood = -483.91827 (not concave)
Iteration 70: log likelihood = -483.91823 (not concave)
Iteration 71: log likelihood = -483.91821 (not concave)
Iteration 72: log likelihood = -483.91819 (not concave)
Iteration 73: log likelihood = -483.9179 (not concave)
Iteration 74: log likelihood = -483.91764 (not concave)
Iteration 75: log likelihood = -483.91761 (not concave)
Iteration 76: log likelihood = -483.9176 (not concave)
Iteration 77: log likelihood = -483.91758 (not concave)
Iteration 78: log likelihood = -483.91757 (not concave)
Iteration 79: log likelihood = -483.91755 (not concave)
Iteration 80: log likelihood = -483.91753 (not concave)
Iteration 81: log likelihood = -483.9169 (not concave)
Iteration 82: log likelihood = -483.91642 (not concave)
Iteration 83: log likelihood = -483.9164 (not concave)
Iteration 84: log likelihood = -483.91637 (not concave)
Iteration 85: log likelihood = -483.91636 (not concave)
Iteration 86: log likelihood = -483.91578 (not concave)
Iteration 87: log likelihood = -483.91526 (not concave)
Iteration 88: log likelihood = -483.91523 (not concave)
Iteration 89: log likelihood = -483.91521 (not concave)
Iteration 90: log likelihood = -483.91519 (not concave)
Iteration 91: log likelihood = -483.91518 (not concave)
Iteration 92: log likelihood = -483.91454 (not concave)
Iteration 93: log likelihood = -483.91406 (not concave)
Iteration 94: log likelihood = -483.91405 (not concave)
Iteration 95: log likelihood = -483.91402 (not concave)
Iteration 96: log likelihood = -483.91397 (not concave)
Iteration 97: log likelihood = -483.91385 (not concave)
Iteration 98: log likelihood = -483.91369 (not concave)
Iteration 99: log likelihood = -483.91367 (not concave)
Iteration 100: log likelihood = -483.91366 (not concave)
Iteration 101: log likelihood = -483.91302 (not concave)
Iteration 102: log likelihood = -483.91254 (not concave)
Iteration 103: log likelihood = -483.91252 (not concave)
Iteration 104: log likelihood = -483.9125 (not concave)
Iteration 105: log likelihood = -483.91248 (not concave)
Iteration 106: log likelihood = -483.91198 (not concave)
Iteration 107: log likelihood = -483.91142 (not concave)
Iteration 108: log likelihood = -483.91135 (not concave)
Iteration 109: log likelihood = -483.91134 (not concave)
Iteration 110: log likelihood = -483.91132 (not concave)
Iteration 111: log likelihood = -483.9108 (not concave)
Iteration 112: log likelihood = -483.91025 (not concave)
Iteration 113: log likelihood = -483.9102 (not concave)
Iteration 114: log likelihood = -483.91018 (not concave)
Iteration 115: log likelihood = -483.91016 (not concave)
Iteration 116: log likelihood = -483.91015 (not concave)
Iteration 117: log likelihood = -483.91013 (not concave)
Iteration 118: log likelihood = -483.91012 (not concave)
Iteration 119: log likelihood = -483.90957 (not concave)
Iteration 120: log likelihood = -483.90903 (not concave)
Iteration 121: log likelihood = -483.90899 (not concave)
Iteration 122: log likelihood = -483.90897 (not concave)
Iteration 123: log likelihood = -483.90841 (not concave)
Iteration 124: log likelihood = -483.90788 (not concave)
Iteration 125: log likelihood = -483.90785 (not concave)
Iteration 126: log likelihood = -483.90783 (not concave)
Iteration 127: log likelihood = -483.90781 (not concave)>>>>>>>>>>>>>>>>>

Thanks in advance

[Rukman Mecca]

My issue got resolved with option

technique(bfgs)

Thanks for the suggestion

[Shreya Jain]

how can I use this code: technique (bfgs)?

[Joseph Coveney]

how can I use this code: technique (bfgs)?

logit foreign c.mpg →
logit foreign c.mpg , technique(bfgs)