Multiple hypothesis Testing

[Latoya Sundack]

Hi,

I am using a diff-in-diff analysis with 5 outcome variables and I was asked to do multiple hypothesis testing. I am not sure how to go about doing this, since I just have one treatment variable. I have seen some examples with many treatment variables and outcomes. Can someone please explain how this can be done?

Best,
Toya

[Carlo Lazzaro]

Toya;
it's impossible to reply positively to queries like this one without an example of your dataset.
in addition: could you confirm that you have 5 ys? Thanks.

[Rich Goldstein]

if you have 5 outcome vars, then I assume you are doing 5 diff-in-diff analyses, correct? if so, the simplest correction is bonferroni - just divide your nominal cutoff by 5 and make that your new cutoff; note that I am here assuming that "multiple hypothesis testing" really means to adjust for multiple tests - if it means something else, please clarify

[Latoya Sundack]

Hi Rich,

Yes, I have 5 different diff-in-diff analyses. I tried running the individual regression for the first outcome variable and then running the bonferroni,

test treat post diff-in-diff female african chinese whites hispanic, mtest(bonferroni)

. I got the results below. Do I need to do the same for all the outcome variables? Also, can you please explain the results below? How do I divide the nominal cutoff in this instance and how do I find the new cutoff? Thanks for your reply.

[Rich Goldstein]

sorry, but I have no idea what you actually did; please read the FAQ and follow its advice

I note that my assumption was that there would be one p-value of particular interest in each DiD and that you would just compare that p-value to a cutoff of .01 (assuming your nominal cutoff was 0.05)

[Latoya Sundack]

Hi Rich,

thank you for your reply.

Ok, so my variable of interest here is dif-in-dif, in the regression using the nominal cutoff of 0.05, the p-value is 0.000.

I have followed this example https://www.statalist.org/forums/forum/general-stata-discussion/general/1499648-adjusting-standard-errors-for-multiple-hypothesis-testing

Assuming that I did the bonferroni correction right, its 0.0015 based on the results above. Therefore, am I to compare these two p values? I am grateful for your feedback. Thanks

[Julian Reif]

I recommend using the Westfall-Young methodology, which is less conservative than Bonferroni. I have developed a user a command for this method in Stata. Download instructions, examples, and methodological details are available on Github:
https://github.com/reifjulian/wyoung

The command also computes the Bonferroni and Sidak multiple testing adjustments for purposes of comparison.

[Latoya Sundack]

Hi Julian,

Thanks for your suggestion.

Toya

[Latoya Sundack]

Hi Julian,

Thanks for your suggestion.

Toya

It worked!! Thanks again.

[Latoya Sundack]

I recommend using the Westfall-Young methodology, which is less conservative than Bonferroni. I have developed a user a command for this method in Stata. Download instructions, examples, and methodological details are available on Github:
https://github.com/reifjulian/wyoung

The command also computes the Bonferroni and Sidak multiple testing adjustments for purposes of comparison.

It worked. Thanks again.