-pwmc- updated on SSC (Pairwise comparisions of means, unequal variances)

[daniel klein]

Thanks as always to Kit Baum, an update for pwmc is now available from the SSC.

The pwmc command performs pairwise comparisons of means with unequal variances.

To install the latest version from within Stata, type

ssc install pwmc

You can also download the latest version of the command from my GitHub repository.

The update is a complete rewrite of the 2014 version. It fixes a bug on Linux OS, introduces new options, new returned results, and a new reporting routine. I do not expect the changes to break old code; if they do, let me know.

Here is a quick example from the help file:

. sysuse nlsw88
(NLSW, 1988 extract)

. pwmc wage , over(race)

Pairwise comparisons of means with unequal variances
-------------------------------------------------------------------
             |               Robust
        wage |       Diff.  Std. Err.    [95% Conf. Interval]
-------------+-----------------------------------------------------
        race |
     2 vs 1  |  -1.238442   .2566354    -1.841136   -.6357475   (C)
             |                          -1.840673   -.6362105  (GH)
             |                          -1.852083   -.6248002  (T2)
     3 vs 1  |   .4677818   1.032201    -2.100212    3.035776   (C)
             |                          -2.096844    3.032408  (GH)
             |                          -2.165544    3.101108  (T2)
     3 vs 2  |   1.706223    1.04306    -.8858822    4.298329   (C)
             |                          -.8790847    4.291532  (GH)
             |                          -.9474056    4.359853  (T2)
-------------------------------------------------------------------
Key:  C: Dunnett C
     GH: Games and Howell
     T2: Tamhane T2

I hope some of you will find this useful.

[Felix Bittmann]

Thanks Daniel for this interesting command! Can you give some further information on when one should use it? I think margins with pwcompare will do the same but I assume your implementation has some advantages? Thanks for a clarification!

[daniel klein]

Thanks for asking.

tl;dr none of the methods implemented in pwmc is readily implemented in Stata. That is not a very informative answer.

The key difference between pwmc and other commands with the mcompare() option, such as margins with the pwcompare option is that while the former does not assume equal variances the latter typically seem to do, in one way or another. From what I understand, the assumption of equal variances relates to the standard error, the degrees of freedom, and the critical value for constructing the confidence interval. Four out of the seven methods implemented in mcompare() assume equal variances (and two methods assume equal sample sizes). Three methods (bonferroni, sidak, and scheffe) can be used after adjusting standard errors for unequal variances, say with regress. In fact, the (default) standard errors from pwmc are identical to those reported by regress with the vce(hc2) option. However, the methods in mcompare() do not adjust the degrees of freedom and, thus, produce critical values (and confidence intervals) different from pwmc. Yes, you can specify the df() option with mcompare() to supply (externally computed) adjusted degrees of freedom. I believe you can even match Tamhane's (1979) T2 confidence interval with mcompare(sidak) and user-supplied degrees of freedom. However, the degrees of freedom depend on the sample sizes of the compared groups and the df() option only lets you specify one number. Thus, for \(k\) groups with unequal sample sizes, you would have to specify \(\left(k\left(k-1\right)\right)/2\) df() options. I don't think you can match Dunnet's (1980) method or Games and Howell (1976) confidence intervals with mcompare().

You can adjust both standard errors and degrees of freedom with ttest and the the unequal option. When there are only two groups, the updated pwmc produces the same results as ttest with the unequal option. As ttest, the updated pwmc now supports Welch's (1947) degrees of freedom adjustment; it also implements standard errors equal to those reported by regress with the vce(hc3) option. But ttest is limited to comparing two means.



Dunnett, C. W. 1980. Pairwise Multiple Comparisons in the Unequal Variance Case, Journal of the American Statistical Association, 75(372),796--800.

Games, P. A., & Howell, J. F. 1976. Pairwise Multiple Comparison Procedures with Unequal N's and/or Variances: A Monte Carlo study, Journal of Educational Statistics, 1(2), 113--125.

Tamhane, A. C. 1979. A Comparison of Procedures for Multiple Comparisons of Means with Unequal Variances, Journal of the American Statistical Association, 74(366), 471--480.

Welch, B. L. 1947. The generalization of 'student's' problem when several different population variances are involved. Biometrika, 34(1/2), 28--35.

[daniel klein]

OK, so here is a demonstration of my verbal description in #3.

Starting with the (slightly modified) example from the help

. sysuse nlsw88
(NLSW, 1988 extract)

. pwmc wage , over(race) mcompare(t2) vallabel

Pairwise comparisons of means with unequal variances
----------------------------------------------------------------
                |               Robust          Tamhane's T2
           wage |       Diff.  Std. Err.    [95% Conf. Interval]
----------------+-----------------------------------------------
           race |
Black vs White  |  -1.238442   .2566354    -1.852083   -.6248002
Other vs White  |   .4677818   1.032201    -2.165544    3.101108
Other vs Black  |   1.706223    1.04306    -.9474056    4.359853
----------------------------------------------------------------

We can use regress and margins with the pwcompare option (we could also use pwcompare directly) to match the results from pwmc

. quietly regress wage i.race , vce(hc2)

.     // 2 vs 1
. margins race , pwcompare mcompare(sidak) df(`=df_vs[1,1]')

Pairwise comparisons of adjusted predictions             Number of obs = 2,246
Model VCE: Robust HC2

(output omitted)

-----------------------------------------------------------------
                |            Delta-method           Sidak
                |   Contrast   std. err.     [95% conf. interval]
----------------+------------------------------------------------
           race |
Black vs White  |  -1.238442   .2566354     -1.852083   -.6248002

(output omitted)

.     // 3 vs 1
. margins race , pwcompare mcompare(sidak) df(`=df_vs[1,2]')

(output omitted)

Other vs White  |   .4677818   1.032201     -2.165544    3.101108

(output omitted)

.     // 3 vs 2
. margins race , pwcompare mcompare(sidak) df(`=df_vs[1,3]')

(output omitted)

Other vs Black  |   1.706223    1.04306     -.9474056    4.359853

Conversely, we can replicate ttest with pwmc

. ttest wage , by(collgrad) unequal reverse

Two-sample t test with unequal variances
------------------------------------------------------------------------------
   Group |     Obs        Mean    Std. err.   Std. dev.   [95% conf. interval]
---------+--------------------------------------------------------------------
 College |     532    10.52606    .2742596    6.325833    9.987296    11.06483
Not coll |   1,714    6.910561    .1276104    5.283132    6.660273     7.16085
---------+--------------------------------------------------------------------
Combined |   2,246    7.766949    .1214451    5.755523    7.528793    8.005105
---------+--------------------------------------------------------------------
    diff |            3.615502    .3024942                3.021696    4.209308
------------------------------------------------------------------------------
    diff = mean(College) - mean(Not coll)                         t =  11.9523
H0: diff = 0                     Satterthwaite's degrees of freedom =  774.553

    Ha: diff < 0                 Ha: diff != 0                 Ha: diff > 0
 Pr(T < t) = 1.0000         Pr(|T| > |t|) = 0.0000          Pr(T > t) = 0.0000

. pwmc wage , over(collgrad) effects

Two-sample t test with unequal variances
------------------------------------------------------------------------------
             |               Robust                             Unadjusted
        wage |       Diff.  Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
    collgrad |
     1 vs 0  |   3.615502   .3024942    11.95   0.000     3.021696    4.209308
------------------------------------------------------------------------------

[Felix Bittmann]

Thanks for the clarification! I think having a generalization to Welch's approach to more than two groups is a powerful feature. Until now I was not aware of the various limitations that pwcompare has.

[daniel klein]

Thanks as always to Kit Baum, an updated version of pwmc is now available from the SSC.

The update introduces new defaults, a modified syntax, adjusted p-values for Dunnett's C, additional standard errors, and additional degrees of freedom.

More specifically, pwmc has the following changes and improvements:

1. The default adjustment for multiple comparisons is now Games and Howell's (1976) method. Dunnett's C (1980) and Tamhane's T2 (1979) methods remain available as options but must be explicitly requested. This change might break old code; see item 9.

2. The methods for multiple comparisons have new names mcompare(cochran), mcompare(gh), mcompare(tamhane). The new names are also used as row names for some of the returned matrices. This change might break old code; see item 9.

3. Although old syntax continues to work, option hc3 should now be specified as se(hc3). New option se() also supports the default hc2 and new ols standard errors.

4. Although old syntax continues to work, option welch should now be specified as df(welch). New option df() also supports the default satterthwaite, new bm (Bell and McCaffrey), and new residual degrees of freedom.

5. pwmc now computes adjusted p-values for Dunnett's C; see the help file for formulas. I consider this an improvement not a breaking change.

6. Dunnett's C produces different results when combined with se(hc3). I consider this a bug fix.

7. The output table now displays value labels by default. It also includes more specific information on the standard errors and degrees of freedom.

8. There are additional returned results.

9. The pwmc package now includes the wrapper command pwmc_version to undo the changes described in item 1 and item 2 above. The wrapper command is included to facilitate the maintenance of old code. In old do-files (and possibly ado-files), search and replace "pwmc " (note the trailing whitespace) with "pwmc_version 2:" (note the trailing colon) to replicate previous results. To replicate earlier results from the immediate form, pwmci, replace all occurences of "pwmci" with "pwmc_version 2:"; the wrapper automatically switches to the immediate form of the commend. In case it's not obvious: If you're worried about future changes, you may now type pwmc_version 3: ... instead of pwmc (or pwmci).

To install the latest version of pwmc from within Stata, type

ssc install pwmc

Alternatively, the latest version of pwmc may be obtained from GitHub.


Dunnett, C. W. 1980. Pairwise Multiple Comparisons in the Unequal Variance Case. Journal of the American Statistical Association, 75(372), 96--800.

Games, P. A., & Howell, J. F. 1976. Pairwise Multiple Comparison Procedures with Unequal N's and/or Variances: A Monte Carlo study. Journal of Educational Statistics, 1(2), 113--125.

Tamhane, A. C. 1979. A Comparison of Procedures for Multiple Comparisons of Means with Unequal Variances. Journal of the American Statistical Association, 74(366), 471--480.