Predicted probabilities, average marginal effects, and confidence intervals

[Stefan Fors]

I´m currently revising a manuscript, and have run into some problems. The aim of the paper is to track differences in health between two groups over time in repeated cross-sectional surveys. As most of the health outcomes in the paper are dichotomous we are mainly using logistic regressions. As a first step in the manuscript we plotted out the development in terms of predicted probabilities. Then we went on to explore it further in terms of Average Marginal Effects. As the predicted probabilities were mainly intended to be descriptive, we did not present any confidence intervals for them. All of this was done using the margins command.

However, now a reviewer have asked that we provide confidence intervals even for the predicted probabilities. This becomes problematic since the predicted probabilities for the different groups often have overlapping confidence intervals (suggesting non-significant differences), while the average marginal effects (from the same models) shows that there are indeed statistically significant differences. Thus, reporting the confidence intervals for the predicted probabilities would lead to an underestimation of the statistically significant differences. Why is this, and is it really reasonable to compare the confidence intervals for the predicted probabilities from different groups in order to establish statistically significant differences?

[Rich Goldstein]

you can have overlapping CI's with statistically significant differences; here are two cites, one aimed at clinicians and the other at statisticians:

Wolfe, R, Hanley, J (2002), "If we're so different, why do we keep overlapping? When 1 plus 1 doesn't make 2", _Canadian Medical Association Journal_, 166: 65-66

Schenker, N and Gentleman, JF (2001), "On Judging the Significance of Differences by Examining the Overlap between Confidence Intervals," _The American Statistician_, 55: 182-186

[Stefan Fors]

Just what I needed. Thank you!

[Richard Williams]

Remember that if two CIs barely overlap, one variable coefficient has to be at the high end of its estimated range and the other has to be at the low end of its estimated range. The odds of both of these things happening are much less than the odds of either thing happening alone. The dydxs are better for showing whether the effects significantly differ. I think you have to be careful when showing CIs because there is a widespread and mistaken believe that overlap means non-significant.

[Stefan Fors]

Thank you Richard. I guess that is the way to go, keep the predicted probabilities for descriptive purposes (without CIs), and do the formal significance testing for the dydxs.