Using wyoung to conduct multiple tests on a single model

[Itzhak Rasooly]

Suppose I run a single regression with two independent variables, e.g.

sysuse auto, clear
reg price mpg rep78

Suppose I want to examine the (two) hypotheses that coefficients on mpg and rep78 are zero; and assume that I want to obtain corrected p-values using the wyoung command. Does anyone know how to do this? I have tried a few variants on:

wyoung, cmd("reg price mpg rep78") familyp(mpg rep78) bootstraps(100) seed(20)

However, none of these seem to work. Also, I can't find any examples in the documentation where we run many hypothesis tests but just use one model. Perhaps the issue is that Westfall-Young corrected p-values are not computable in this case?

Many thanks in advance for any suggestions or pointers.

[Itzhak Rasooly]

A small update: I managed to get the command to run by writing out the regression repeatedly, i.e.

wyoung, cmd("reg price mpg rep78" "reg price mpg rep78") familyp(mpg rep78) bootstraps(100) seed(20)

However, the output looks wrong. For example, the Bonferroni and Sidak adjusted p-values are the same as the uncorrected p-values.

[George Ford]

the models are the same and you are testing individual coefficients, not the equality.

sysuse auto, clear

reg price mpg rep78
test mpg
test rep78
test mpg rep78 
test mpg rep78 , mtest(b)
test mpg rep78 , mtest(s)

wyoung, cmd("reg price mpg rep78" "reg price mpg rep78") familyp(mpg-rep78) bootstraps(100) seed(20) familypexp

wyoung, cmd("reg price mpg rep78") familyp(mpg-rep78) bootstraps(100) seed(20) familypexp

[Itzhak Rasooly]

Hi George, thanks for the suggestions. As you note, I want to test two different hypotheses (that the coefficient on mpg is zero, and that the coefficient on rep78 is zero) and am estimating just one model (namely, the regression of price on mpg and rep78). Note that the uncorrected p-values associated with these tests are 0.000 and 0.056: I want the (two) corrected p-values.

Unfortunately, your code does not seem to test both hypotheses (and deliver corrected p-values associated with both hypotheses). When I run the first command, i.e.

wyoung, cmd("reg price mpg rep78" "reg price mpg rep78") familyp(mpg-rep78) bootstraps(100) seed(20) familypexp

the same hypothesis is tested twice. I get one uncorrected p-value (.01341771) and one Westfall-Young corrected p-value (0), although these are printed twice. This uncorrected p-value does not correspond to either of the hypotheses I am testing (as noted, the uncorrected p-values should be 0.000 and 0.056).

When I run your other command, i.e.

wyoung, cmd("reg price mpg rep78") familyp(mpg-rep78) bootstraps(100) seed(20) familypexp

I again get one Westfall-Young corrected p-value (0), which again corresponds to the wrong uncorrected p-value (.0134...).

Is there a way to change the code to do what I want to do?

[George Ford]

maybe

wyoung, cmd("reg price mpg rep78" "reg price mpg rep78" "reg price mpg rep78") familyp(mpg rep78 mpg-rep78) bootstraps(100) seed(20) familypexp

[Julian Reif]

First, please make sure you have the most recent version of -wyoung- installed. (Installation instructions are available on Github.)

You should be able to calculate these p-vals using either of the two syntaxes described in the helpfile:

wyoung price, cmd("reg OUTCOMEVAR mpg rep78") familyp(mpg rep78) bootstraps(100) seed(20)

wyoung, cmd("reg price mpg rep78" "reg price mpg rep78") familyp(mpg rep78) bootstraps(100) seed(20)

[Itzhak Rasooly]

Hi Julian,

Thanks so much for this -- it seems to work!

One slightly strange thing is that the Westfall-Young corrected p-values (0.000 and 0.018) are a lot smaller than the uncorrected p-values (0.000 and 0.056); this comes from running the code with 1000 bootstrap repetitions. But I assume this is just a strange quirk of the data (a strong correlation between the tests?) and not indicative of any issue with the computations.

Many thanks again,

Itzhak

[Julian Reif]

Yes, it is possible for the adjusted p-vals to be smaller than the unadjusted ones. In this particular case, it is probably because of model misspecification. We mention this issue in the technical documentation.

[George Ford]

Julian, how do you add the third test of the two coefficients being equal?

[Julian Reif]

If you want to test a combination of coefficients, you need to specify the -familypexp- option, which will send the expressions specified in -familyp()- to -nlcom- or -lincom-.

sysuse auto, clear

* Unadjusted p-vals
reg price mpg rep78
lincom mpg
lincom rep78
lincom mpg-rep78

* Adjusted and unadjusted p-vals
wyoung, cmd("reg price mpg rep78" "reg price mpg rep78" "reg price mpg rep78") familyp(mpg rep78 "mpg-rep78") bootstraps(100) seed(20) familypexp

[George Ford]

Yikes, I got it right.

[Louise Devos]

Hi there! I have a similar problem, but even with the solutions proposed above, I do not manage to get a working code.

I'm performing a mediation analysis using the gsem command and I want to calculate the corrected p-values for the fact that I have two families of dependent variables (indicated with _SD or _TD in the code below). I can't get my code using the wyoung package to work and hoped anybody could help me out?

Thank you for your time!

My code:

bootstrap, reps(1000): gsem (Can_SD_AmountWork <- Vig_Name Vig_Place_Of_Birth Vig_Residence Vig_Edu_LP_2 Vig_Edu_LP_3 Vig_Edu_LP_4 ///
Vig_Dista_2 Vig_Dista_3 Vig_Male `controls') ///
(Can_SD_Punctuality <- Vig_Name Vig_Place_Of_Birth Vig_Residence Vig_Edu_LP_2 Vig_Edu_LP_3 Vig_Edu_LP_4 ///
Vig_Dista_2 Vig_Dista_3 Vig_Male `controls') ///
(Can_SD_Availability <- Vig_Name Vig_Place_Of_Birth Vig_Residence Vig_Edu_LP_2 Vig_Edu_LP_3 Vig_Edu_LP_4 ///
Vig_Dista_2 Vig_Dista_3 Vig_Male `controls') ///
(Can_SD_QualityInter <- Vig_Name Vig_Place_Of_Birth Vig_Residence Vig_Edu_LP_2 Vig_Edu_LP_3 Vig_Edu_LP_4 ///
Vig_Dista_2 Vig_Dista_3 Vig_Male `controls') ///
(Can_SD_Encounters <- Vig_Name Vig_Place_Of_Birth Vig_Residence Vig_Edu_LP_2 Vig_Edu_LP_3 Vig_Edu_LP_4 ///
Vig_Dista_2 Vig_Dista_3 Vig_Male `controls') ///
(Can_SD_Interaction <- Vig_Name Vig_Place_Of_Birth Vig_Residence Vig_Edu_LP_2 Vig_Edu_LP_3 Vig_Edu_LP_4 ///
Vig_Dista_2 Vig_Dista_3 Vig_Male `controls') ///
(Can_SD_WorkAttitude <- Vig_Name Vig_Place_Of_Birth Vig_Residence Vig_Edu_LP_2 Vig_Edu_LP_3 Vig_Edu_LP_4 ///
Vig_Dista_2 Vig_Dista_3 Vig_Male `controls') ///
(Can_SD_Maturity <- Vig_Name Vig_Place_Of_Birth Vig_Residence Vig_Edu_LP_2 Vig_Edu_LP_3 Vig_Edu_LP_4 ///
Vig_Dista_2 Vig_Dista_3 Vig_Male `controls') ///
(Can_SD_Responsibility <- Vig_Name Vig_Place_Of_Birth Vig_Residence Vig_Edu_LP_2 Vig_Edu_LP_3 Vig_Edu_LP_4 ///
Vig_Dista_2 Vig_Dista_3 Vig_Male `controls') ///
(Can_SD_Authority <- Vig_Name Vig_Place_Of_Birth Vig_Residence Vig_Edu_LP_2 Vig_Edu_LP_3 Vig_Edu_LP_4 ///
Vig_Dista_2 Vig_Dista_3 Vig_Male `controls') ///
(Can_SD_Motivation <- Vig_Name Vig_Place_Of_Birth Vig_Residence Vig_Edu_LP_2 Vig_Edu_LP_3 Vig_Edu_LP_4 ///
Vig_Dista_2 Vig_Dista_3 Vig_Male `controls') ///
(Can_SD_Ambition <- Vig_Name Vig_Place_Of_Birth Vig_Residence Vig_Edu_LP_2 Vig_Edu_LP_3 Vig_Edu_LP_4 ///
Vig_Dista_2 Vig_Dista_3 Vig_Male `controls') ///
(Can_TD_Partic <- Vig_Name Vig_Place_Of_Birth Vig_Residence Vig_Edu_LP_2 Vig_Edu_LP_3 Vig_Edu_LP_4 ///
Vig_Dista_2 Vig_Dista_3 Vig_Male `controls') ///
(Can_TD_Collea <- Vig_Name Vig_Place_Of_Birth Vig_Residence Vig_Edu_LP_2 Vig_Edu_LP_3 Vig_Edu_LP_4 ///
Vig_Dista_2 Vig_Dista_3 Vig_Male `controls') ///
(Can_TD_Client <- Vig_Name Vig_Place_Of_Birth Vig_Residence Vig_Edu_LP_2 Vig_Edu_LP_3 Vig_Edu_LP_4 ///
Vig_Dista_2 Vig_Dista_3 Vig_Male `controls'), ///
nocapslatent ///
vce(cluster ID_Par_Regi)