Oneway Bonferroni

[Aparna Anand]

Hello,

I am trying to compare variables mean and std deviation across three samples. Is oneway bonferroni test is the best way to do it? If yes, when reporting P value in descriptive statistics table, which p value should be included? ( there are p values for each row column mean comparison and another one for F test) Thanks

[Carlo Lazzaro]

Aparna:
welcome to this forum.
The p-values you mentioned have different meanings:
1) the F-test tests the null that -oneway- does not outperform -mean-;
2) each p-value of the head-to-head comparison provides evidence of rejecting the null that the different groups have the same mean (if p-value<0.05) or the other way round (if p-value>0.05).
3) the absence of evidence of differences in variances is a precondition of -oneway- and can be checked via the outcome of the Bartlett's test (although -oneway- is pretty robust to deviations from variance equality).
As an aside, I would consider -regress- instead of -oneway-, as they're both linear model but there's nothing that -oneway- can do that cannot be done far better by -regress-.

[Carlo Lazzaro]

Just elaborating a bit on the last statement of my previous reply, you can see yourself that -oneway- and -regress- (as expected) reach the same results:

use http://www.stata-press.com/data/r15/apple.dta
. oneway weight treatment, bonferroni

                        Analysis of Variance
    Source              SS         df      MS            F     Prob > F
------------------------------------------------------------------------
Between groups      5295.54433      3   1765.18144     21.46     0.0013
 Within groups      493.591667      6   82.2652778
------------------------------------------------------------------------
    Total             5789.136      9   643.237333

Bartlett's test for equal variances:  chi2(3) =   1.3900  Prob>chi2 = 0.708

             Comparison of Average weight in grams by Fertilizer
                                (Bonferroni)
Row Mean-|
Col Mean |          1          2          3
---------+---------------------------------
       2 |   -59.1667
         |      0.001
         |
       3 |     -33.25    25.9167
         |      0.042      0.122
         |
       4 |      -34.4    24.7667      -1.15
         |      0.036      0.146      1.000

. regress weight i.treatment

      Source |       SS           df       MS      Number of obs   =        10
-------------+----------------------------------   F(3, 6)         =     21.46
       Model |  5295.54433         3  1765.18144   Prob > F        =    0.0013
    Residual |  493.591667         6  82.2652778   R-squared       =    0.9147
-------------+----------------------------------   Adj R-squared   =    0.8721
       Total |    5789.136         9  643.237333   Root MSE        =      9.07

------------------------------------------------------------------------------
      weight |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
   treatment |
          2  |  -59.16667   7.405641    -7.99   0.000    -77.28762   -41.04572
          3  |     -33.25   8.279758    -4.02   0.007    -53.50984   -12.99016
          4  |      -34.4   8.279758    -4.15   0.006    -54.65984   -14.14016
             |
       _cons |      111.9   5.236579    21.37   0.000     99.08655    124.7134
------------------------------------------------------------------------------

*Since you have 6 head-to-head comparisons to perform (1 vs 2; 1 vs 3; 1 vs 4; 2 vs 3; 2 vs 4; 3 vs 4) you should multiply by 6 the non-adjusted p-value obtained after -test-*

di 0.000*6
.000
di 0.007*6
.042
di 0.006*6
.036

. test 2.treatment = 3.treatment

 ( 1)  2.treatment - 3.treatment = 0

       F(  1,     6) =    9.80
            Prob > F =    0.0203

. di 0.0203*6
.1218

. test 2.treatment = 4.treatment

 ( 1)  2.treatment - 4.treatment = 0

       F(  1,     6) =    8.95
            Prob > F =    0.0243

. di 0.0243*6
.1458

. test 3.treatment = 4.treatment

 ( 1)  3.treatment - 4.treatment = 0

       F(  1,     6) =    0.02
            Prob > F =    0.9032

*as p-value is capped at 1, there's no scope in multiplying 0.9032 by 6*

.

Moreover, -regress postestimation- offers a suite of test the check possible model misspecifications.

[Daniela Kaiser]

Hi,

I came across this thread and thought those who contributed to it could help me with the folllowing. I'm trying to do a cross-group comparison, where I compare 5 treatment groups (a control group and 4 experimental groups) in terms of a characteristic of the block in which the participants of each group reside (proportion of white residents)

Given that I'm comparing 5 groups, I used a oneway anova with the Bonferroni procedure (syntax and output below).

I'm struggling to understand why the F-test suggests that there is a signficant difference in the mean of the proportion of white residents between the 5 treatment groups, while the p-values of the group-to-group comparison do not suggest that there is a significant difference in any of the pairwise group-to-group comparisons (C and T1, T1 and T2, C and T2, etc.). Given that the F-test suggests that there is a significant difference between groups, I would have expected that to be driven by differences between two pairs of groups or more. Does anyone know how to interpret this or whether I'm misinterpeting the output?

Thank you!

oneway perc_white treatment, bonferroni

                        Analysis of variance
    Source              SS         df      MS            F     Prob > F
------------------------------------------------------------------------
Between groups      1.16692792      4    .29173198      2.52     0.0399
 Within groups       130.64936   1127   .115926672
------------------------------------------------------------------------
    Total           131.816287   1131   .116548442

Bartlett's equal-variances test: chi2(4) =   2.1741    Prob>chi2 = 0.704

                    Comparison of perc_white by treatment
                                (Bonferroni)
Row Mean-|
Col Mean |          C         T1     T2_AT2     T2_AT3
---------+--------------------------------------------
      T1 |   -.060074
         |      0.687
         |
  T2_AT2 |    .004283    .064357
         |      1.000      0.464
         |
  T2_AT3 |   -.039424     .02065   -.043707
         |      1.000      1.000      1.000
         |
   T3_NA |   -.078697   -.018623    -.08298   -.039273
         |      0.174      1.000      0.105      1.000

[Carlo Lazzaro]

Daniela:
the issue here is that the F-test refers to the ANOVA model, not to comparison.
You can experience the very same using -regress- (please note that -regress- outperforms anova in many respects):

. use "C:\Program Files\Stata18\ado\base\a\auto.dta"
(1978 automobile data)

. regress price i.rep78

      Source |       SS           df       MS      Number of obs   =        69
-------------+----------------------------------   F(4, 64)        =      0.24
       Model |  8360542.63         4  2090135.66   Prob > F        =    0.9174
    Residual |   568436416        64     8881819   R-squared       =    0.0145
-------------+----------------------------------   Adj R-squared   =   -0.0471
       Total |   576796959        68  8482308.22   Root MSE        =    2980.2

------------------------------------------------------------------------------
       price | Coefficient  Std. err.      t    P>|t|     [95% conf. interval]
-------------+----------------------------------------------------------------
       rep78 |
          2  |   1403.125   2356.085     0.60   0.554    -3303.696    6109.946
          3  |   1864.733   2176.458     0.86   0.395    -2483.242    6212.708
          4  |       1507   2221.338     0.68   0.500    -2930.633    5944.633
          5  |     1348.5   2290.927     0.59   0.558    -3228.153    5925.153
             |
       _cons |     4564.5   2107.347     2.17   0.034     354.5913    8774.409
------------------------------------------------------------------------------

. oneway price i.rep78, bonferroni
factor-variable and time-series operators not allowed
r(101);

. oneway price rep78, bonferroni

                        Analysis of variance
    Source              SS         df      MS            F     Prob > F
------------------------------------------------------------------------
Between groups      8360542.63      4   2090135.66      0.24     0.9174
 Within groups       568436416     64      8881819
------------------------------------------------------------------------
    Total            576796959     68   8482308.22

Bartlett's equal-variances test: chi2(4) =  11.4252    Prob>chi2 = 0.022

                  Comparison of Price by Repair record 1978
                                (Bonferroni)
Row Mean-|
Col Mean |          1          2          3          4
---------+--------------------------------------------
       2 |    1,403.1
         |      1.000
         |
       3 |    1,864.7    461.608
         |      1.000      1.000
         |
       4 |      1,507    103.875   -357.733
         |      1.000      1.000      1.000
         |
       5 |    1,348.5    -54.625   -516.233     -158.5
         |      1.000      1.000      1.000      1.000

. regress price i.rep78

      Source |       SS           df       MS      Number of obs   =        69
-------------+----------------------------------   F(4, 64)        =      0.24
       Model |  8360542.63         4  2090135.66   Prob > F        =    0.9174
    Residual |   568436416        64     8881819   R-squared       =    0.0145
-------------+----------------------------------   Adj R-squared   =   -0.0471
       Total |   576796959        68  8482308.22   Root MSE        =    2980.2

------------------------------------------------------------------------------
       price | Coefficient  Std. err.      t    P>|t|     [95% conf. interval]
-------------+----------------------------------------------------------------
       rep78 |
          2  |   1403.125   2356.085     0.60   0.554    -3303.696    6109.946
          3  |   1864.733   2176.458     0.86   0.395    -2483.242    6212.708
          4  |       1507   2221.338     0.68   0.500    -2930.633    5944.633
          5  |     1348.5   2290.927     0.59   0.558    -3228.153    5925.153
             |
       _cons |     4564.5   2107.347     2.17   0.034     354.5913    8774.409
------------------------------------------------------------------------------

. mat list e(b)

e(b)[1,6]
           1b.         2.         3.         4.         5.           
        rep78      rep78      rep78      rep78      rep78      _cons
y1          0   1403.125  1864.7333       1507     1348.5     4564.5

. test 2.rep78=3.rep78

 ( 1)  2.rep78 - 3.rep78 = 0

       F(  1,    64) =    0.15
            Prob > F =    0.6984

. oneway price rep78

                        Analysis of variance
    Source              SS         df      MS            F     Prob > F
------------------------------------------------------------------------
Between groups      8360542.63      4   2090135.66      0.24     0.9174
 Within groups       568436416     64      8881819
------------------------------------------------------------------------
    Total            576796959     68   8482308.22

Bartlett's equal-variances test: chi2(4) =  11.4252    Prob>chi2 = 0.022

. mat list e(b)

e(b)[1,6]
           1b.         2.         3.         4.         5.           
        rep78      rep78      rep78      rep78      rep78      _cons
y1          0   1403.125  1864.7333       1507     1348.5     4564.5

.

The difference in the two p-values depend on using Bonferroni correction in -oneway- comparison.
Please also note that, as expected, -oneway- and -regress- give back the very same coefficients.

[Daniela Kaiser]

Hi Carlo,

Thank you so much for your response. Could you expand a bit on the difference between the ANOVA and the comparison? What is the F test in the ANOVA comparing in my case, if not the group means?

I'm not sure if I should interpret my output as indicating that there are significant differences between groups but without being able to pinpoint which group is significantly different to which.

In the example you showed, isn't it the case that both the p values for the regression are non-significant and the p value for the F test is not significant (and the P values for each group to group comparison)?

Regarding -regress- is it still okay to use it despite my variable being a proportion?

Thank you!

[Carlo Lazzaro]

Daniela:
1) the F-test you mentioned refers to the ANOVA model (or to -regress-): it tests the hypothesis that all coefficients excluding the constant are zero. In your case:

di .29173198/.115926672
2.5165217

2) in your case, there's no evidence of difference among the groups. It may be due to the fact that, when these differences were summed up, they make the bw/wth variances ratio statistical significant, but when compared each other, these differences are not enough to reach statistical significance. It is like saying that one man on his pal's shoulders can see what lies to the other side of the wall, even though the difference in height between the two of them is negligible;
3) there's nothing -regress- can do worse than anova, provided that your dependent variable is continuous.

[Daniela Kaiser]

Thank you so much Carlo Lazzaro, apologies for the delayed repsonse!