Assumption of proportional odds (brant) in an ordered logistic regression

[Navah Kadish]

Hello everyone,

This is my first time posting on Statalist and I hope I'm presenting all relevant information. I'm using Stata 13.1 for Windows.

I am checking the influence of 2 predictors (both ordinal scale, 5 levels, 3 levels) of an outcome after surgery as dependent variable (ordinal scale with 5 levels) with an ordered logistic regression (N=44).

ologit post_develop_cat pre_develop_cat final_surgery

In order to check the assumption of proportional odds I installed two add-ons (omodel, brant from spost).
I have successfully performed an approximate likelihood-ratio test of proportionality of odds across response categories (omodel):

omodel logit post_develop_cat pre_develop_cat final_surgery

In addition, I wanted to perform the Brant test of parallel regression assumption, as the details option gives coefficients.

ologit post_develop_cat pre_develop_cat final_surgery
brant, detail

While this has worked in a previous analysis, now I receive an error message. There was no further information on error #9993:

not all independent variables can be retained in binary logits
brant test cannot be computed
r(9993);

In order to find out where the problem lies, I repeated the analysis separately for both predictors.
The first one worked:

ologit post_develop_cat pre_develop_cat
brant, detail

The second predictor did not work and produced the error below.

ologit post_develop_cat final_surgery
brant, detail

option rhs_beta() required
r(198);

This error message is related to syntax errors, the quoted option is not specified as one in the help menu, so I'm not sure how to use it.
There seems to be something wrong with my predictor variable "final_surgery" which is coded 0, 1, 2 indicating the extent of surgery.
I would be grateful for suggestions on how to conduct the brant command.

With kind regards,
Navah

[Richard Williams]

Brant does a series of logistic regressions, basically collapsing the DV into a dichotomy. So, if the DV has 4 categories, it is category 1 vs 2, 3, 4, then it is categories 1, 2 vs 3, 4, and finally 1, 2, 3 vs 4. If you get the error "not all independent variables can be retained in binary logits" it probably means that one of your independent vars is a constant in one of those collapses. I don't know how your vars are coded, but if one outcome was "surgery was a spectacular success" and one of the categories of an independent variable was "death is almost certain" I can see where something like that might happen.

If you want the Brant test to work, you may need to drop or recode some of your variables, e.g. combine categories. I suspect you are spreading the data too thin, but if ologit is running ok maybe you don't care.

I don't know where the rhs_beta error is coming from. My guess is there is a problem but the error message is not accurately identifying what it is.

You might also try the -oparallel- command, available from SSC, which offers a couple more tests.

[Richard Williams]

Oh, N = 44 is really small too, which can increase your likelihood of problems.

[Maarten Buis]

Rich and I have done a simulation study to see when the different tests start to fail. This was a project that actually started as a discussion on Statalist. We presented the results on the 2013 German Stata Users' meeting: http://maartenbuis.nl/presentations/gsug13.pdf . In our simulation all tests are already really struggling with sample sizes of 200, so 44 seems way too small to me.

[Richard Williams]

Even aside from the brant-type tests, 44 is an awfully small number for a maximum likelihood analysis. Sample size guidelines (that I stole from others) are on p. 4 of

http://www3.nd.edu/~rwilliam/xsoc73994/L02.pdf

[Navah Kadish]

Dear Richard Williams,
Dear Maarten Buis,

Thank you for your detailed and helpful answers! Please excuse my late reply, I was expecting to but did not receive notifications for answers.

In our area of clinical research 44 presents a large sample and I will not be able to enlarge it, only adapt the statistical analyses. Following your information, I merged some categories and found a predictor that better fit the research question. Now Stata performs the brant test indication no violation of the assumption of proportional odds.
I found different information and am not sure how the small sample affects my type 1 error and power (page 4 in the above linked document L02 mentions a more strict or less strict p-value).
Thank you for your help!

Best, Navah Kadish

[Richard Williams]

Here is what my handout says about sample size:

Both Long and Allison agree that the standard advice is that with small samples you should accept larger p-values as evidence against the null hypothesis. Given that the degree to which ML estimates are normally distributed in small samples is unknown, it is more reasonable to require smaller p-values in small samples.

It sounds, though, that you've done what you can do to minimize the limitations of a small sample. Hopefully you found a strong relationship.

[Navah Kadish]

The paragraph you quote from your handout is the one I was unsure about as it states both accepting larger and also requiring smaller p-values. Luckily, my results showed strong relationships and the assumption of proprotional odds were not violated with p > .3, so that no further adjustement was required.
Thank you so much for your help!

[Richard Williams]

That paragraph basically says that Long and Allison both disagree with the standard advice that you use bigger p values (e.g., .10) in small samples. Instead they think you should go the other way, e.g. use the .01 level.

[Aysha Karim]

How can I perform single score tests of the proportional odds assumption for each covariate in ordered logit model? I have performed the overall score test of equal proportion assumption by 'omodel' command. My code is given below (All the covariates are categorical variable):

omodel logit Dep_disorder pv_exposure vb_exposure Division Region ///
age Work wquintile mstatus education acc_ph acc_ipc adol_exp ///
social_aff adol_aff gen_norm_1 gen_norm_2 gen_norm_3 gen_norm_4