Test of significance vs Binomial exact confidence intervals

[Shannon Meadows]

I would appreciate some advice regarding what an appropriate test of significance would be for comparing binomial events, and how this test of significance would compare to the binomial exact confidence intervals.

In Group 1, 30/31 tested positive
In Group 2, 15/21 tested positive

When I get the binomial exact confidence intervals from the following code:
cii 31 30, binomial
cii 21 15, binomial

I get the following confidence intervals:
Group 1: 95%CI 83.3-99.9
Group 2: 95%CI 47.8-88.7

Am I correct to conclude that because the 95% confidence intervals overlap, the prevalence in group 1 is not significantly different from group 2 (p>0.05)?

Secondly, what would be the appropriate statistical test to get the p-value for this comparison?

When I use xi: logit result i.group
I get a p-value of 0.027, but this contrasts with my earlier statement about these groups not being significantly different.

Any suggestions on how to proceed would be appreciated!

[Clyde Schechter]

Am I correct to conclude that because the 95% confidence intervals overlap, the prevalence in group 1 is not significantly different from group 2 (p>0.05)?

Absolutely not!

A significant test for the difference between groups is based on the standard error of the difference. The confidence intervals within each group are based on within-group standard errors only. FInding overlapping confidence intervals when the group-difference is statistically significant is nothing unusual at all. See https://www.cscu.cornell.edu/news/statnews/stnews73.pdf for a simple explanation and illustration.

That said, I would also point out that the statistics underlying exact binomial confidence intervals are rather different from those underlying maximum likelihood estimation of logistic regression models, so discrepancies might be expected even if the principle you assumed were correct.

[Shannon Meadows]

Thank you Clyde for those thoughts. I would like to clarify that the statistical outputs are being appropriately used together- I would report each group prevalence and 95% CI, as identified by the 'cii' coding, and comment on any significant differences between groups from the logit output?

[Clyde Schechter]

Yes, in terms of how you would report the analytic findings, that is what you would do, assuming that one of your research goals is to test for differences between these groups on this outcome.

[Carole J. Wilson]

You can also conduct a two sample normal approximation test.

prtesti 31 30 21 15, count

If I played around with the math, I could probably show the relationship between this and margins after the logit. If you run margins i.group, pwcompare after the logit, you see that the confidence interval and standard error of the difference is identical.

[Phil Besuner]

I prefer the cohort command for risk ratio and difference

csi 30 15 1 6

yields a p value of 0.0086 along with some interesting additional information.

[Clyde Schechter]

A general comment, not directed specifically at anyone who has posted on this thread.

As posts 4, 5, and 6 in this thread demonstrate, there are many approaches to the same data. Yet another would be -tabi 30 15 \ 1 6-, which gives yet a different pvalue, 0.013. And I can think of more, but I think the point has been made. There are pros and cons to each of these particular ways of analyzing this data. Some will appeal more to different audiences who have their own data analysis traditions. Some are easier to explain to uninitiated audiences. Some will satisfy a particular reviewer's (or supervisor's) pet peeve...

While it's nice to have choices, this can lead to problems, as when investigators keep trying different ones until they get the result they were hoping for (e.g. p < 0.05). So, when not just doing exploratory studies, it is best to carefully design studies in advance and commit ahead of time to a specific analysis plan. Sometimes reality gets in the way and the planned analysis turns out to be infeasible with the data at hand, but that should only happen occasionally in well-designed studies.