Calculating the Statistical Significance for Overlapping Confidence Intervals

[Monica Muller]

Hi,

In the following data I want to know whether the b coefficient for low level is significantly different from that for the high level in each month. The only data I have is b itself and its CI. The coefficients and CIs are generated from a monte carlo simulation with n=2000.

Is there a way to calculate whether the difference between the b values in the two levels is statistically significant?

* Example generated by -dataex-. To install: ssc install dataex
clear
input float(ci5 month b ci95) str4 level
 -.03903021  1  -.011272022   .016622718 "high"
 -.04252657  2   -.01650105   .011399727 "high"
 -.05257559  3  -.034638386  -.016130824 "high"
 -.05830632  4   -.03690543  -.018540205 "high"
 -.06023302  5  -.034695968  -.014855707 "high"
 -.05975572  6  -.030589053  -.010161517 "high"
 -.05921155  7  -.026343225  -.006651564 "high"
 -.05927761  8   -.02244832   -.00408389 "high"
 -.05882464  9  -.019107176 -.0025055795 "high"
 -.05799637 10  -.016283397 -.0015027014 "high"
 -.05812236 11  -.013904007 -.0009308802 "high"
 -.05705826 12  -.011891297   -.00049742 "high"
 .004737387  1    .02479452    .04686485 "low" 
 .013566217  2   .036880057    .06039166 "low" 
-.008033013  3   .004375337   .016217021 "low" 
-.017749608  4  -.007688625  .0015991568 "low" 
 -.01862827  5   -.00886435 -.0001028545 "low" 
-.017553983  6  -.007629246 .00055061345 "low" 
 -.01478354  7  -.005569925  .0016163826 "low" 
 -.01321365  8  -.004532443   .001881863 "low" 
 -.01191987  9  -.003787475   .001610453 "low" 
-.011031338 10 -.0033230034  .0011840797 "low" 
-.010346236 11  -.002891431  .0010406288 "low" 
-.009651612 12  -.002524714  .0008282398 "low" 
end

Thank you very much!

[Clyde Schechter]

ttest b, by(level)

That said, the American Statistical Association has recommended that the concept of statistical significance be abandoned. See https://www.tandfonline.com/doi/full...5.2019.1583913. A better practice wold be to report the difference in mean values of b in the two groups along with its confidence interval or standard error (which results you will also get from the above code.)

[Monica Muller]

Thanks Clyde. ttest compares b values for 12 months. I want to see for example whether b_low in month 1 is significantly different from b_high in month 1, etc. What I ended up doing is:

gen se1=(b-ci5)/1.64
reshape wide ci5 b ci95 se1 , i(month) j(level) string
gen tse=1.64*sqrt(se1high^2+ se1low^2)
gen tse=1.64*sqrt(se1high^2+ se1low^2)
gen Difference= bhigh-blow
gen ci5=round(Dif-tse, 0.01)
gen ci95=round(Dif+tse, 0.01)
gen sig=((ci5>0 & ci95>0) | (ci5<0 & ci95<0))

but I am not sure if it's the best approach, specially because it is possible that confidence intervals are not symmetric. I appreciate your feedback.

[Clyde Schechter]

I didn't grasp initially that what you have is paired data. Your approach is correct to the extent that you need to reshape the data, but your calculation for the standard error is not correct.

drop ci*
reshape wide b, i(month) j(level) string

ttest bhigh = blow