Friedman post-hoc

[duaa ibrahim]

Hello,

I have 6 participants who repeated a test 5 times. I would like to assess if there was a difference between the repeated tests. I performed Friedman test and it was significant. Now, I would like to perform post hoc but can't find a command for Friedman post-hoc. I read that Nemenyi test might be a good choice but cant find its command. How to proceed?


Truly appreciae your help!

[Marcos Almeida]

Welcome to the Stata Forum / Statalist.

I believe there is a package in R where you can do something similar.

That said, in Stata, theoretically speaking, I gather you may perform parwise Wilcoxon signed rank tests and 'adjust' for familywise error such as Bonferroni's.

To end, the sample size appears to be rather small. It is surprising you found low p-values for the omnibus test.

Indeed, I think you are bound to face type II error, shall you proceed with adjusted post comparisons.

[duaa ibrahim]

Thank you so much Dr. Almeida,
I performed the pairwise Wilcoxon signed rank test and adjusting using Bonferroni's but nothing was significant. However, the p value from Friedman was significant (0.0025).

Any suggestion would be appreciated.. Thanks!

[duaa ibrahim]

I read that Nemenyi test might be the right one but cant find its command though

[Marcos Almeida]

I performed the pairwise Wilcoxon signed rank test and adjusting using Bonferroni's but nothing was significant. However, the p value from Friedman was significant (0.0025).

This striked me as odd. You may wish to see the example below, where I used the SJ - friedman - whose author is Richard Goldstein, an active member of this forum.

. input t1 t2 t3

            t1         t2         t3
  1. 110 150 175
  2. 100 110 130
  3. 120 140 155
  4. 110 130 155
  5. 120 130 145
  6. 130 155 170
  7. end

. signrank t1 = t2

Wilcoxon signed-rank test

        sign |      obs   sum ranks    expected
-------------+---------------------------------
    positive |        0           0        10.5
    negative |        6          21        10.5
        zero |        0           0           0
-------------+---------------------------------
         all |        6          21          21

unadjusted variance       22.75
adjustment for ties       -0.25
adjustment for zeros       0.00
                     ----------
adjusted variance         22.50

Ho: t1 = t2
             z =  -2.214
    Prob > |z| =   0.0269

. signrank t1 = t3

Wilcoxon signed-rank test

        sign |      obs   sum ranks    expected
-------------+---------------------------------
    positive |        0           0        10.5
    negative |        6          21        10.5
        zero |        0           0           0
-------------+---------------------------------
         all |        6          21          21

unadjusted variance       22.75
adjustment for ties        0.00
adjustment for zeros       0.00
                     ----------
adjusted variance         22.75

Ho: t1 = t3
             z =  -2.201
    Prob > |z| =   0.0277

. signrank t2 = t3

Wilcoxon signed-rank test

        sign |      obs   sum ranks    expected
-------------+---------------------------------
    positive |        0           0        10.5
    negative |        6          21        10.5
        zero |        0           0           0
-------------+---------------------------------
         all |        6          21          21

unadjusted variance       22.75
adjustment for ties       -0.63
adjustment for zeros       0.00
                     ----------
adjusted variance         22.13

Ho: t2 = t3
             z =  -2.232
    Prob > |z| =   0.0256

. friedman t?

Friedman =  11.0476
Kendall =    0.7365
P-value =    0.0504

As you see, quite on the contrary of what you found, in spite of Wilcoxon's p-values well below 0.05, we got a barely significant p-value with Friedman's ANOVA.

As a suggestion, you may wish to provide data and command. Perhaps we may identify what has happened.

[Rich Goldstein]

since my name came up, I just want to note that Duaa did email me prior to writing to Statalist; my suggestions were basically the same as Marcos'

[duaa ibrahim]

Thank you so much Dr. Almeida! Truly appreciate your help.
I truly appreciate Dr. Rich Goldstien help, I did reach out for his help with this matter and he was very cooperative in trying to help me reach an answer.
However, since I am relatively new to STATA, I may have done mistakes running the codes, and I am still curious of my results as they stand.
The Wilcoxon sign rank test would be signifiant only if I don't adjust for it.

Data:

Practice 1 Omni	Practice 2 Omni	Practice 3 Omni	Practice 4 Omni	Practice 5 Omni
53.72665995	66.85657838	75.89328604	78.31190716	89.02076946
91.50458223	95.42352433	96.04331537	97.40200777	93.41464764
81.55195409	84.3015418	91.84736411	91.38936236	95.02700639
91.53903198	89.16191657	87.18201087	91.90145989	92.66668696
83.76553697	83.7352378	92.32404343	84.95262814	89.50057555
93.6355134	90.28165779	93.17900586	92.71120876	93.6861227

Codes:

friedman practice1omni practice2omni practice3omni practice4omni practice5omni

Friedman = 18.3714
Kendall = 0.7349
P-value = 0.0025

. signrank practice1omni = practice2omni

Wilcoxon signed-rank test

sign | obs sum ranks expected
-------------+---------------------------------
positive | 3 7 10.5
negative | 3 14 10.5
zero | 0 0 0
-------------+---------------------------------
all | 6 21 21

unadjusted variance 22.75
adjustment for ties 0.00
adjustment for zeros 0.00
----------
adjusted variance 22.75

Ho: practice1omni = practice2omni
z = -0.734
Prob > |z| = 0.4631

.
.
. signrank practice1omni = practice3omni

Wilcoxon signed-rank test

sign | obs sum ranks expected
-------------+---------------------------------
positive | 2 3 10.5
negative | 4 18 10.5
zero | 0 0 0
-------------+---------------------------------
all | 6 21 21

unadjusted variance 22.75
adjustment for ties 0.00
adjustment for zeros 0.00
----------
adjusted variance 22.75

Ho: practice1omni = practice3omni
z = -1.572
Prob > |z| = 0.1159

. signrank practice1omni = practice4omni

Wilcoxon signed-rank test

sign | obs sum ranks expected
-------------+---------------------------------
positive | 1 2 10.5
negative | 5 19 10.5
zero | 0 0 0
-------------+---------------------------------
all | 6 21 21

unadjusted variance 22.75
adjustment for ties 0.00
adjustment for zeros 0.00
----------
adjusted variance 22.75

Ho: practice1omni = practice4omni
z = -1.782
Prob > |z| = 0.0747

. signrank practice1omni = practice5omni

Wilcoxon signed-rank test

sign | obs sum ranks expected
-------------+---------------------------------
positive | 0 0 10.5
negative | 6 21 10.5
zero | 0 0 0
-------------+---------------------------------
all | 6 21 21

unadjusted variance 22.75
adjustment for ties 0.00
adjustment for zeros 0.00
----------
adjusted variance 22.75

Ho: practice1omni = practice5omni
z = -2.201
Prob > |z| = 0.0277

. signrank practice2omni = practice3omni

Wilcoxon signed-rank test

sign | obs sum ranks expected
-------------+---------------------------------
positive | 1 2 10.5
negative | 5 19 10.5
zero | 0 0 0
-------------+---------------------------------
all | 6 21 21

unadjusted variance 22.75
adjustment for ties 0.00
adjustment for zeros 0.00
----------
adjusted variance 22.75

Ho: practice2omni = practice3omni
z = -1.782
Prob > |z| = 0.0747

. signrank practice2omni = practice4omni

Wilcoxon signed-rank test

sign | obs sum ranks expected
-------------+---------------------------------
positive | 0 0 10.5
negative | 6 21 10.5
zero | 0 0 0
-------------+---------------------------------
all | 6 21 21

unadjusted variance 22.75
adjustment for ties 0.00
adjustment for zeros 0.00
----------
adjusted variance 22.75

Ho: practice2omni = practice4omni
z = -2.201
Prob > |z| = 0.0277

. signrank practice2omni = practice5omni

Wilcoxon signed-rank test

sign | obs sum ranks expected
-------------+---------------------------------
positive | 1 1 10.5
negative | 5 20 10.5
zero | 0 0 0
-------------+---------------------------------
all | 6 21 21

unadjusted variance 22.75
adjustment for ties 0.00
adjustment for zeros 0.00
----------
adjusted variance 22.75

Ho: practice2omni = practice5omni
z = -1.992
Prob > |z| = 0.0464

. signrank practice3omni = practice4omni

Wilcoxon signed-rank test

sign | obs sum ranks expected
-------------+---------------------------------
positive | 3 9 10.5
negative | 3 12 10.5
zero | 0 0 0
-------------+---------------------------------
all | 6 21 21

unadjusted variance 22.75
adjustment for ties 0.00
adjustment for zeros 0.00
----------
adjusted variance 22.75

Ho: practice3omni = practice4omni
z = -0.314
Prob > |z| = 0.7532

. signrank practice3omni = practice5omni

Wilcoxon signed-rank test

sign | obs sum ranks expected
-------------+---------------------------------
positive | 2 5 10.5
negative | 4 16 10.5
zero | 0 0 0
-------------+---------------------------------
all | 6 21 21

unadjusted variance 22.75
adjustment for ties 0.00
adjustment for zeros 0.00
----------
adjusted variance 22.75

Ho: practice3omni = practice5omni
z = -1.153
Prob > |z| = 0.2489




I am so thankful and appreciative for all your help.

[Marcos Almeida]

I hope you won't take it amiss, but one of the most cherished ways of showing concern and appreciation of our time, effort and attention, is just following the FAQ advice.

Particularly, if you follow the topic about sharing data, command and output.

The way you presented data and command unveils that you, unfortunately, have not yet taken a look at the FAQ. Please do it. Please act according to the FAQ recommendations in the forthcoming messages. Thank you in advance.

This notwithstanding, and considering I have not enough time (and patience) to type all values of the data set you shared, I decided to type the values as integers. Indeed, but for a few ties, I gather the results won't change much.

This is to say "tout court" that the pairwise comparisons using Nemenyi multiple comparison test won't probably give significant p-values.

By the way, the sample size is tiny, and there is a warning about this in - friedman - helpfiles:

The P-value is an approximation, although it appears to be pretty good as long as there are at least 8 cases and 3 variables.

Actually, there are only 6 cases, you know...

I cannot back you up (theoretically speaking) on this, but you may wish to consider other post-hoc comparisons after Friedman's ANOVA, such as Conover's as well as Quade test.

As a matter of fact, both tests accept Bonferroni's adjustments for p-values (and other methods of correction for familywise error).

Wouldn't ten pairwise comparisons be taken as a fishing expedition, considering the small sample? Well, that is up to you to reflect on.

To end, as far as I'm concerned (maybe I'm wrong), the three above-mentioned post-hoc tests after Friedman's ANOVA are not currently available in Stata.

This is to say that, henceforth, I fear I cannot help you further with this.

I hope to have helped.

[Joseph Coveney]

search Nemenyi

doesn't bring anything up, and so probably isn't available in Stata. You could manually compute it if it's that important to use a nonparametric approach, but have you considered a parametric approach? The residuals from a mixed-model ANOVA don't look too bad—run the do-file below to see whether they're acceptable to you.

I agree with Marcos that you'll likely come up dry with pairwise testing, given what you've got to work with, regardless of which approach you take.

version 15.0

clear *
set more off
input double (omn1 omn2 omn3 omn4 omn5)
53.72665995     66.85657838     75.89328604     78.31190716     89.02076946
91.50458223     95.42352433     96.04331537     97.40200777     93.41464764
81.55195409     84.3015418     91.84736411     91.38936236     95.02700639
91.53903198     89.16191657     87.18201087     91.90145989     92.66668696
83.76553697     83.7352378     92.32404343     84.95262814     89.50057555
93.6355134     90.28165779     93.17900586     92.71120876     93.6861227
end

generate byte pid = _n
quietly reshape long omn, i(pid) j(prc)

anova omn prc pid

predict double omn_res, residuals
pnorm omn_res
qnorm omn_res

contrast prc, pveffects mcompare(bonferroni)

test, showorder
#delimit ;
matrix input Contrasts = (
    -1 1 0 0 0 0 0 0 0 0 0 0 \
    -1 0 1 0 0 0 0 0 0 0 0 0 \
    -1 0 0 1 0 0 0 0 0 0 0 0 \
    -1 0 0 0 1 0 0 0 0 0 0 0 \
    0 -1 1 0 0 0 0 0 0 0 0 0 \
    0 -1 0 1 0 0 0 0 0 0 0 0 \
    0 -1 0 0 1 0 0 0 0 0 0 0 \
    0 0 -1 1 0 0 0 0 0 0 0 0 \
    0 0 -1 0 1 0 0 0 0 0 0 0 \
    0 0 0 -1 1 0 0 0 0 0 0 0);
#delimit cr
test, test(Contrasts) mtest(holm)

exit