Bonferroni Correction / False Discovery Rate

[Andreas Baltin]

Hi Statalisters,

I have some questions about the bonferroni correction and False Discovery Rates.

What do I do: I have 4 different dependent variables, on which I run 4 seperate regressions. So it looks somewhat like this:

xtreg dependentvariable1 control1 control2 control3 control4, fe vce(cluster state)
xtreg dependentvariable1 control1 control2 control3 control4 control2*control3, fe vce(cluster state)
xtreg dependentvariable1 control1 control2 control3 control4 control1*control2*control3, fe vce(cluster state)
xtreg dependentvariable1 control1 control2 control3 control4 control1*control2*control3*control4, fe vce(cluster state)

xtreg dependentvariable2 control1 control2 control3 control4, fe vce(cluster state)
xtreg dependentvariable2 control1 control2 control3 control4 control2*control3, fe vce(cluster state)
xtreg dependentvariable2 control1 control2 control3 control4 control1*control2*control3, fe vce(cluster state)
xtreg dependentvariable2 control1 control2 control3 control4 control1*control2*control3*control4, fe vce(cluster state)

xtreg dependentvariable2 control1 control2 control3 control4, fe vce(cluster state)
xtreg dependentvariable2 control1 control2 control3 control4 control2*control3, fe vce(cluster state)
xtreg dependentvariable2 control1 control2 control3 control4 control1*control2*control3, fe vce(cluster state)
xtreg dependentvariable2 control1 control2 control3 control4 control1*control2*control3*control4, fe vce(cluster state)

xtreg dependentvariable2 control1 control2 control3 control4, fe vce(cluster state)
xtreg dependentvariable2 control1 control2 control3 control4 control2*control3, fe vce(cluster state)
xtreg dependentvariable2 control1 control2 control3 control4 control1*control2*control3, fe vce(cluster state)
xtreg dependentvariable2 control1 control2 control3 control4 control1*control2*control3*control4, fe vce(cluster state)

I now think that I have to adjust my p-values in order to address the inflated probability of commiting a Type 1 error. I consider 2 approaches:

1. The Bonferroni correction. Here, for every individual regression, I multiply all p-values by 4, as I am carrying out 4 regressions per dependent variable.
That is, for example I run the first regression

xtreg dependentvariable1 control1 control2 control3 control4, fe vce(cluster state)

and then multiply the individual p-values on control 1, control2 and so forth by 4. I do this for all 16 regressions, and then argue that I show statistical significance under Bonferroni-corrected p-values. Is this appraoch correct?

2. I consider employing the False Discovery Rate by Benjamini and Hochberg (1995). However, I am unsure how to exactly do this in my specific case.
I know that you are meant to order your p-values, divide them by the number of tests and then multiply with a specified rate.
However, which p-values do I order? Do I order together all individuals p-values from every set of 4 regression?
As an example, for dependentvariable1 that would be: 20 independent p-values, then order them and then calculate (i/20)*Rate, and then compare with p-values.
Is this the correct way of applying this method in this case?

I am extremely thankful for any help and thank you a lot in advance.

Kind regards,
Andreas

[Andreas Baltin]

Commenting on my own topic:

Regarding my approach 1. I believe rather than multiplying by 4 (the analyses per dependent variable), I have to multiply by number of independent variables FOR EACH of the 16 regressions, right?

For 2., Do i basically order them WITHIN EACH individual regression, i.e. I carry out the procedure 16 times?

[Ivan Manhique]

Hi Andreas,

To which conclusion did you get?

Best,
Ivan

[Julian Reif]

A third option for adjusting p-vals is to control for the familywise error rate. This can be done using the Stata command -wyoung-. Further details and examples available here:
https://github.com/reifjulian/wyoung

[Ivan Manhique]

Dear Professor Reif,

Thank you very much for your response. Though I am trying to go with the false discovery rate (with a step-up method), the examples in

wyoung

are enlightening. I am running a Structural Equation Model with panel data, thus, I have one equation per period. I am open for suggestions..

Thank you,
Ivan Manhique