Adjusting p-values for testing multiple hypotheses across multiple outcomes/equations

[Ryan Kuo]

Hello!

Does Stata have some functionality with which one can adjust p-values for testing multiple hypotheses across multiple equations and outcome variables?

For context, I have an experimental design wherein I'm testing two treatments (one relating to Language instruction and one to Math instruction) against a common control group (i.e. two treatment arms and one control arm).

Depending on the treatment, I'm looking at different outcome variables (Language test scores for the Language arm, Math test scores for the Math arm), but these variables are nonetheless correlated since high-achieving students are likely to be better at both. My regression specifications are as follows:

foreach subject in Language Math {
xtreg `subject'_endline `subject'_baseline i.treatment, i(stratumID) fe
}

`subject'_endline is my outcome variable, `subject'_baseline is the baseline score, and treatment is a categorical variable denoting treatment status. I also have stratum fixed effects to reflect my experimental design, which drives me to use xtreg instead of reg (too many strata to do i.stratumID).

I want to adjust for the fact that I'm testing multiple hypotheses and using the same control group (similar to Dunnett's test in conception but with differing outcome variables and multiple equations). Does anyone know if/how I could do this in Stata?

Thanks!
Ryan

[Bruce Weaver]

Hello Ryan. I don't have a solution for you, but am offering this comment. Given that you have two treatments and one control, you could use Fisher's LSD as the analogy rather than Dunnett's test. When there are 3 groups, Fisher's LSD preserves the family-wise alpha. If you need sources to support that statement, see the following:

• https://www.ncbi.nlm.nih.gov/pubmed/17128424
• Howell DC. Statistical methods for psychology. Cengage Learning; 2012. (Chapter on MC procedures)

This suggests (to me) that if you use the two-step approach akin to Fisher's LSD, you could apply whatever method you choose for controlling the family-wise alpha to the omnibus test for i.treatment rather than to the contrasts of each treatment level vs. control. This should give you more power than a method analogous to Dunnett's test.

HTH.