Significant marginal effects but C.I.s for predicted margins overlapping

[Julia Park]

Hi, I have fit a xtlogit model with an interaction term between two variables (x1 & x2). x1 is a dummy variable and x2 is continuous taking values of (0,1,2,3). I found a highly significant marginal effect of x2 using "margins, dydx(x2) at (x1=0)." However, when I get linear predictions for x2=0 and x2=3 respectively while holding x1 constant by running

margins, at(x2==0 x1==0)
margins, at(x2==3 x1==0)
the gap between their outputs is exactly 3 times of the marginal effect previously obtained using "dydx" option, but their C.I.s are big so that they overlap, which probably means that there is no difference in their predicted y between x2=0 and x2=3 (?). (The same thing happens with "predict (pu0)" option.) I expected when the marginal effect is statistically significant, their C.I.s will "not" overlap. So, my question is, can I still say that there is a marginal effect of x2 when x1==0?

Also, when I am using "margins, at(varlist)," do I have to specify values for all variables included in my regression? or can I specify only the ones that I want to fix at specific values, then does Stata automatically set other unspecified variables at mean?

I hope to hear your advice asap. Thank you very much in advance!

[Steve Samuels]

What is true: If the individual CIs do not overlap, then the difference is statistically significant. Though many people believe otherwise, the reverse is not necessarily true: CIs for individual parameters can overlap even if their difference is statistically significant. What is true is that if the difference is statistically significanct, the CI for the difference will exclude zero. I composed this simple example for a colleague who saw a statement like yours in a technical report:

Suppose you have two independent estimated means for data known to be normal

m1 = 70 and m2 = 40 with n= 100 in each and known SD = 100

Thus the SEs for both means are both se = SD/sqrt(n) = 10. The 95% error bound for both mean are is 1.96 se = 19.6

Thus the 95% CIs are:
m1: 70-19.6, 70+19.6 = [50.4 99.6] and m2: 40-19.6, 40+19.6 = [20.4 60.6]

These intervals overlap.

The difference between the estimated means is:

D = 70 -40 =30
The SE for the difference of independent means is sqrt(se1^2 + se2^2) , so the SE of D

se_d = sqrt(200) = 14.14

The 95% bound of error is 1.96 x 14.14 = 27.71

and the 95% CI for the difference is

30-27.71, 30 + 27.71 = [2.29 -57.71]

This CI excludes zero, hence 2-sided p< 0.05. In fact, the Z statistic for the difference is z = 30/14.14 = 2.12, and p = 0.034.

[Julia Park]

Thank you very much for the clarification and detailed example. So, the conclusion is that even though CIs for individual parameters overlap, we can still say there is meaningful conditional effects of the variable as long as the difference is statistically significant. Is this what you mean?

[Maarten Buis]

So, the conclusion is that even though CIs for individual parameters overlap, we can still say there is meaningful conditional effects of the variable as long as the difference is statistically significant. Is this what you mean?

Almost, a statistical test does not test the hypothesis that a effect is meaningful; that is a necessarily subjective assesment that can only be made by humans. It tests the null-hypothesis that the effect is equal to 0 and in your case you can reject that null-hypothesis. If the effect is large enough to be meaningful is just a matter of staring at that coefficient untill you know that it is meaningful (or not).

[Julia Park]

Thank you for the comments!!

[Richard Williams]

I think it is a common mistake to believe that if two confidence intervals overlap the difference between two coefficients is not significant. Remember, when intervals overlap, that means the extreme/least likely possible values of one coefficient are overlapping with the most extreme/least likely possible values of the other. The odds that both coefficients are at the extreme end of the confidence interval are much less than the odds that only one of them will be.

[Charlene Rapsey]

Thank you for a very useful discussion. Given it is a common mistake to believe that if two confidence intervals overlap then the difference between the two conditions is "insignificant", do you think it is best to present margin plots with or without confidence intervals?

[Clyde Schechter]

Given it is a common mistake to believe that if two confidence intervals overlap then the difference between the two conditions is "insignificant", do you think it is best to present margin plots with or without confidence intervals?

As with everything else, a presentation (or publication) should be driven by the goals of the research and the background of the audience.

If one of the goals is to assess the predicted margins, then they should be presented with confidence intervals because every estimate should always be given with an estimate of the associated uncertainty. (The confidence interval represents a bare minimum estimate of the uncertainty of any estimate in that it accounts only for sampling error, but it is better than nothing.) When the goal is to present the margins themselves, it is immaterial whether the differences among the margins are "significant."

If one of the goals is to assess the differences among the predicted margins, then those differences should be resented, along with their confidence intervals. In this situation, the margins themselves are of only peripheral interest and the confidence intervals around them even less interesting. Sometimes the differences need to be placed in the context of the margins themselves but the margins are not otherwise of interest, and in that case one might show the differences and their confidence intervals along with the estimated margins without confidence intervals in a single plot.

It is sometimes the case that understanding both the margins themselves and the differences among them are goals of the research. Then that requires two separate displays. (Yes, of course, you might put them side by side on a single tableau, or even overlay them if the resulting picture would be readable.)

[Jesse Smith]

Thanks all, this discussion is extremely helpful. Is there a way, using marginsplot, to plot the differences among the predicted margins (with CIs), rather than the predicted margins themselves?

[Clyde Schechter]

Use -margins, pwcompare- and then run -marginsplot-.