mixed models and Bonferroni correction

[Tommaso Salvitti]

Hi to everybody,
I wrote the code below (mixed model as an alternative to anova repeated measures)

mixed MaximumFidgety i.Diagnosi_3numbis##i.eta_num, vce(cluster id)|| id:, covariance(unstructured) tolerance(1e-6)

My question is: is it possible to apply the Bonferroni correction only in the post hoc analysis or is it possible to insert it in some part of the model I wrote above? Thanks to everyone

[Joseph Coveney]

. . . is it possible to apply the Bonferroni correction only in the post hoc analysis or is it possible to insert it in some part of the model I wrote above?

I'm not sure what you're trying to get at, but have you looked at the postestimation commands for mixed, especially, for pwcompare here?

In order "to insert it in some part of the model", perhaps you could just dock the p-values that are displayed in the regression output by the Bonferroni adjustment based on the number of pairwise comparisons.

What is your intention with covariance() in your model? Ditto, why do you feel the need to specify a convergence tolerance?

[Tommaso Salvitti]

Thanks Joseph Coveney
I asked this question because when I ran the model I get that, when comparing, both diagnosis 2 and diagnosis 3 with the reference one; I obtain a p-value of 0.030 (contrast=-1.672414) and a p-value of 0.051 respectively. Then when I do the comparisons I get using the command

margins Diagnosi_3numbis, at(eta_num=1) pwcompare(effects) mcompare(bonferroni)

: diagnosi_3numbis 2 vs 1 p-value 0.077 and the same value of the contrast..Is it possible? If I didn't use the corrections I would get the same value. However, Bonferroni corrections are essential, right? thank you in advance

[daniel klein]

Bonferroni's method is straightforward, which is probably why it is so popular. If you have 3 diagnoses, there are 3*(3-1)/2 = 3 possible pairwise comparisons. The Bonferroni adjusted p-value is simply p*3 (or 1, whichever is smaller), where p is the unadjusted p-value. Bonferroni's adjustment tends to be very conservative. By the way, consider accounting for unequal variances across groups (i.e., diagnoses), e.g., using robust standard errors.

[Erik Ruzek]

Bonferroni correction feels like overkill to me in this instance. To Daniel's point about heteroskedastic errors, you can estimate separate residual variances for the diagnosis groups in mixed:

mixed MaximumFidgety i.Diagnosi_3numbis##i.eta_num, || id: , residuals(independent, by(Diagnosi_3numbis))

[Leonardo Guizzetti]

Re #3: yes, the contrast (or more general, any linear combination) and the associated standard errors remain the same. Multiplicity corrections make adjustments to the nominal p-value, or equivalently, the significance threshold.

As Daniel helpfully points out, the Bonferroni adjustment is the most straightforward to implement, by hand or otherwise. Since you mention you are interested in comparing tests 2 vs 1 and 3 vs 1, you will only be comparing 2 of of 3 possible pairwise tests, and so you will only need to adjust by a factor of 2.

[Roman Mostazir]

Thanks Joseph Coveney
However, Bonferroni corrections are essential, right? thank you in advance

It is kind of a gray area of debate. Many actually criticised it for being overly conservative that might increase the chance of Type-II error. Have a look at these relevant papers:

1. Rothman, K. J. (1990). No adjustments are needed for multiple comparisons. Epidemiology, 1(1), 43-46.

2. Ranganathan, P., Pramesh, C. S., & Buyse, M. (2016). Common pitfalls in statistical analysis: The perils of multiple testing. Perspect Clin Res, 7(2), 106-107.

3. O'Keefe, D. J. (2003). Colloquy: Should Familywise Alpha Be Adjusted? Human Communication Research, 29(3), 431-447.

4. Hurlbert, S. H., & Lombardi, C. M. (2012). LOPSIDED REASONING ON LOPSIDED TESTS AND MULTIPLE COMPARISONS. Australian & New Zealand Journal of Statistics, 54(1), 23-42. doi:https://doi.org/10.1111/j.1467-842X.2012.00652.x