Post hoc analysis (Bonferroni-holm)

[Daniel Smit]

Dear Statalisters,

My dataset consists out of 500 groups, with each group adding a certain amount of value to their company. I ran a fixed effects regression (xtreg) and performed a F-test to determine whether there are fixed effects present within the groups. ie. whether at least one of the groups is different.

Now, my second step is to determine what group(s) cause the fixed effects and therefore are different. I want to do this with a Bonferroni-holm correction but i am struggling to find out how to do this in Stata. I have found two user-written packages which allow to do this (qqvalue and smileplot) but I fail to understand how to apply this.

Summary on what I did:

- Fixed effects regression

- F-test for fixed effects (significant)

- Next step: identify what groups are different -> Bonferroni-holm ?


This topic is similar but I don't think it's applicable for me: http://www.statalist.org/forums/foru...rom-each-other. Also I read Newson (2010) and Newson (2003) about qqvalue and smileplot but still don't understand how to apply this to a fixed effects regression.

It would be great if anyone could shed some light on how to do this.

Daniel Smit

[Dave Airey]

How do you get the group coefficients' p-values out from xtreg, fe? Maybe in this case you want to run a dummy variables regression, grab the p-values, and submit those to qqvalue?

[Daniel Smit]

Hi Dave,

First of all, thanks allot for the reply
I followed your advice and ran a dummy regression and saved the p-values of that regression (with parmest). Then I used the qqvalue function to calculate the q-values.

This is what I did in code:

reg y x1 x2 age size i.group
parmest, norestore
qqvalue p, method(holm) qvalue(q_group)
list

The strange thing now is: almost all the q-values are 1, except for 2 (out of 500).

Daniel

[Dave Airey]

Well, at least your first problem is solved. Are the p-values you want relative to a reference group or the grand mean? Also, with 500 comparisons, FWERs and strict control of type I error may be conservative if you will follow up with additional data collection on some interesting groups. You might want to consider FDR? What's the end goal?

[Daniel Smit]

I think my explanation with groups was not entirely correct, the 500 ''groups'' are actually just 500 persons. So I need the p-values of the individual investors, which are obtained by doing a regression with a dummy for each person. The end goal is to identify the best performing individuals and maybe see what kind of characteristics these individuals possess. Unfortunately, no other interesting data is available so I have to stick with my current data.

The problem still is that after running qqvalue, the variable created (q_group) is 1 for almost every observation.