Announcement

Collapse
No announcement yet.
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Friedman post-hoc

    Hello,

    I have 6 participants who repeated a test 5 times. I would like to assess if there was a difference between the repeated tests. I performed Friedman test and it was significant. Now, I would like to perform post hoc but can't find a command for Friedman post-hoc. I read that Nemenyi test might be a good choice but cant find its command. How to proceed?


    Truly appreciae your help!

  • #2
    Welcome to the Stata Forum / Statalist.

    I believe there is a package in R where you can do something similar.

    That said, in Stata, theoretically speaking, I gather you may perform parwise Wilcoxon signed rank tests and 'adjust' for familywise error such as Bonferroni's.

    To end, the sample size appears to be rather small. It is surprising you found low p-values for the omnibus test.

    Indeed, I think you are bound to face type II error, shall you proceed with adjusted post comparisons.
    Best regards,

    Marcos

    Comment


    • #3
      Thank you so much Dr. Almeida,
      I performed the pairwise Wilcoxon signed rank test and adjusting using Bonferroni's but nothing was significant. However, the p value from Friedman was significant (0.0025).

      Any suggestion would be appreciated.. Thanks!

      Comment


      • #4
        I read that Nemenyi test might be the right one but cant find its command though

        Comment


        • #5
          I performed the pairwise Wilcoxon signed rank test and adjusting using Bonferroni's but nothing was significant. However, the p value from Friedman was significant (0.0025).
          This striked me as odd. You may wish to see the example below, where I used the SJ - friedman - whose author is Richard Goldstein, an active member of this forum.

          Code:
          . input t1 t2 t3
          
                      t1         t2         t3
            1. 110 150 175
            2. 100 110 130
            3. 120 140 155
            4. 110 130 155
            5. 120 130 145
            6. 130 155 170
            7. end
          
          . signrank t1 = t2
          
          Wilcoxon signed-rank test
          
                  sign |      obs   sum ranks    expected
          -------------+---------------------------------
              positive |        0           0        10.5
              negative |        6          21        10.5
                  zero |        0           0           0
          -------------+---------------------------------
                   all |        6          21          21
          
          unadjusted variance       22.75
          adjustment for ties       -0.25
          adjustment for zeros       0.00
                               ----------
          adjusted variance         22.50
          
          Ho: t1 = t2
                       z =  -2.214
              Prob > |z| =   0.0269
          
          . signrank t1 = t3
          
          Wilcoxon signed-rank test
          
                  sign |      obs   sum ranks    expected
          -------------+---------------------------------
              positive |        0           0        10.5
              negative |        6          21        10.5
                  zero |        0           0           0
          -------------+---------------------------------
                   all |        6          21          21
          
          unadjusted variance       22.75
          adjustment for ties        0.00
          adjustment for zeros       0.00
                               ----------
          adjusted variance         22.75
          
          Ho: t1 = t3
                       z =  -2.201
              Prob > |z| =   0.0277
          
          . signrank t2 = t3
          
          Wilcoxon signed-rank test
          
                  sign |      obs   sum ranks    expected
          -------------+---------------------------------
              positive |        0           0        10.5
              negative |        6          21        10.5
                  zero |        0           0           0
          -------------+---------------------------------
                   all |        6          21          21
          
          unadjusted variance       22.75
          adjustment for ties       -0.63
          adjustment for zeros       0.00
                               ----------
          adjusted variance         22.13
          
          Ho: t2 = t3
                       z =  -2.232
              Prob > |z| =   0.0256
          
          . friedman t?
          
          Friedman =  11.0476
          Kendall =    0.7365
          P-value =    0.0504
          As you see, quite on the contrary of what you found, in spite of Wilcoxon's p-values well below 0.05, we got a barely significant p-value with Friedman's ANOVA.

          As a suggestion, you may wish to provide data and command. Perhaps we may identify what has happened.
          Best regards,

          Marcos

          Comment


          • #6
            since my name came up, I just want to note that Duaa did email me prior to writing to Statalist; my suggestions were basically the same as Marcos'

            Comment


            • #7
              Thank you so much Dr. Almeida! Truly appreciate your help.
              I truly appreciate Dr. Rich Goldstien help, I did reach out for his help with this matter and he was very cooperative in trying to help me reach an answer.
              However, since I am relatively new to STATA, I may have done mistakes running the codes, and I am still curious of my results as they stand.
              The Wilcoxon sign rank test would be signifiant only if I don't adjust for it.

              Data:
              Practice 1 Omni Practice 2 Omni Practice 3 Omni Practice 4 Omni Practice 5 Omni
              53.72665995 66.85657838 75.89328604 78.31190716 89.02076946
              91.50458223 95.42352433 96.04331537 97.40200777 93.41464764
              81.55195409 84.3015418 91.84736411 91.38936236 95.02700639
              91.53903198 89.16191657 87.18201087 91.90145989 92.66668696
              83.76553697 83.7352378 92.32404343 84.95262814 89.50057555
              93.6355134 90.28165779 93.17900586 92.71120876 93.6861227
              Codes:

              friedman practice1omni practice2omni practice3omni practice4omni practice5omni

              Friedman = 18.3714
              Kendall = 0.7349
              P-value = 0.0025

              . signrank practice1omni = practice2omni

              Wilcoxon signed-rank test

              sign | obs sum ranks expected
              -------------+---------------------------------
              positive | 3 7 10.5
              negative | 3 14 10.5
              zero | 0 0 0
              -------------+---------------------------------
              all | 6 21 21

              unadjusted variance 22.75
              adjustment for ties 0.00
              adjustment for zeros 0.00
              ----------
              adjusted variance 22.75

              Ho: practice1omni = practice2omni
              z = -0.734
              Prob > |z| = 0.4631

              .
              .
              . signrank practice1omni = practice3omni

              Wilcoxon signed-rank test

              sign | obs sum ranks expected
              -------------+---------------------------------
              positive | 2 3 10.5
              negative | 4 18 10.5
              zero | 0 0 0
              -------------+---------------------------------
              all | 6 21 21

              unadjusted variance 22.75
              adjustment for ties 0.00
              adjustment for zeros 0.00
              ----------
              adjusted variance 22.75

              Ho: practice1omni = practice3omni
              z = -1.572
              Prob > |z| = 0.1159

              . signrank practice1omni = practice4omni

              Wilcoxon signed-rank test

              sign | obs sum ranks expected
              -------------+---------------------------------
              positive | 1 2 10.5
              negative | 5 19 10.5
              zero | 0 0 0
              -------------+---------------------------------
              all | 6 21 21

              unadjusted variance 22.75
              adjustment for ties 0.00
              adjustment for zeros 0.00
              ----------
              adjusted variance 22.75

              Ho: practice1omni = practice4omni
              z = -1.782
              Prob > |z| = 0.0747

              . signrank practice1omni = practice5omni

              Wilcoxon signed-rank test

              sign | obs sum ranks expected
              -------------+---------------------------------
              positive | 0 0 10.5
              negative | 6 21 10.5
              zero | 0 0 0
              -------------+---------------------------------
              all | 6 21 21

              unadjusted variance 22.75
              adjustment for ties 0.00
              adjustment for zeros 0.00
              ----------
              adjusted variance 22.75

              Ho: practice1omni = practice5omni
              z = -2.201
              Prob > |z| = 0.0277

              . signrank practice2omni = practice3omni

              Wilcoxon signed-rank test

              sign | obs sum ranks expected
              -------------+---------------------------------
              positive | 1 2 10.5
              negative | 5 19 10.5
              zero | 0 0 0
              -------------+---------------------------------
              all | 6 21 21

              unadjusted variance 22.75
              adjustment for ties 0.00
              adjustment for zeros 0.00
              ----------
              adjusted variance 22.75

              Ho: practice2omni = practice3omni
              z = -1.782
              Prob > |z| = 0.0747

              . signrank practice2omni = practice4omni

              Wilcoxon signed-rank test

              sign | obs sum ranks expected
              -------------+---------------------------------
              positive | 0 0 10.5
              negative | 6 21 10.5
              zero | 0 0 0
              -------------+---------------------------------
              all | 6 21 21

              unadjusted variance 22.75
              adjustment for ties 0.00
              adjustment for zeros 0.00
              ----------
              adjusted variance 22.75

              Ho: practice2omni = practice4omni
              z = -2.201
              Prob > |z| = 0.0277

              . signrank practice2omni = practice5omni

              Wilcoxon signed-rank test

              sign | obs sum ranks expected
              -------------+---------------------------------
              positive | 1 1 10.5
              negative | 5 20 10.5
              zero | 0 0 0
              -------------+---------------------------------
              all | 6 21 21

              unadjusted variance 22.75
              adjustment for ties 0.00
              adjustment for zeros 0.00
              ----------
              adjusted variance 22.75

              Ho: practice2omni = practice5omni
              z = -1.992
              Prob > |z| = 0.0464

              . signrank practice3omni = practice4omni

              Wilcoxon signed-rank test

              sign | obs sum ranks expected
              -------------+---------------------------------
              positive | 3 9 10.5
              negative | 3 12 10.5
              zero | 0 0 0
              -------------+---------------------------------
              all | 6 21 21

              unadjusted variance 22.75
              adjustment for ties 0.00
              adjustment for zeros 0.00
              ----------
              adjusted variance 22.75

              Ho: practice3omni = practice4omni
              z = -0.314
              Prob > |z| = 0.7532

              . signrank practice3omni = practice5omni

              Wilcoxon signed-rank test

              sign | obs sum ranks expected
              -------------+---------------------------------
              positive | 2 5 10.5
              negative | 4 16 10.5
              zero | 0 0 0
              -------------+---------------------------------
              all | 6 21 21

              unadjusted variance 22.75
              adjustment for ties 0.00
              adjustment for zeros 0.00
              ----------
              adjusted variance 22.75

              Ho: practice3omni = practice5omni
              z = -1.153
              Prob > |z| = 0.2489




              I am so thankful and appreciative for all your help.

              Comment


              • #8
                I hope you won't take it amiss, but one of the most cherished ways of showing concern and appreciation of our time, effort and attention, is just following the FAQ advice.

                Particularly, if you follow the topic about sharing data, command and output.

                The way you presented data and command unveils that you, unfortunately, have not yet taken a look at the FAQ. Please do it. Please act according to the FAQ recommendations in the forthcoming messages. Thank you in advance.

                This notwithstanding, and considering I have not enough time (and patience) to type all values of the data set you shared, I decided to type the values as integers. Indeed, but for a few ties, I gather the results won't change much.

                This is to say "tout court" that the pairwise comparisons using Nemenyi multiple comparison test won't probably give significant p-values.

                By the way, the sample size is tiny, and there is a warning about this in - friedman - helpfiles:

                The P-value is an approximation, although it appears to be pretty good as long as there are at least 8 cases and 3 variables.
                Actually, there are only 6 cases, you know...

                I cannot back you up (theoretically speaking) on this, but you may wish to consider other post-hoc comparisons after Friedman's ANOVA, such as Conover's as well as Quade test.

                As a matter of fact, both tests accept Bonferroni's adjustments for p-values (and other methods of correction for familywise error).

                Wouldn't ten pairwise comparisons be taken as a fishing expedition, considering the small sample? Well, that is up to you to reflect on.

                To end, as far as I'm concerned (maybe I'm wrong), the three above-mentioned post-hoc tests after Friedman's ANOVA are not currently available in Stata.

                This is to say that, henceforth, I fear I cannot help you further with this.

                I hope to have helped.
                Last edited by Marcos Almeida; 16 Aug 2017, 17:41.
                Best regards,

                Marcos

                Comment


                • #9
                  Code:
                  search Nemenyi
                  doesn't bring anything up, and so probably isn't available in Stata. You could manually compute it if it's that important to use a nonparametric approach, but have you considered a parametric approach? The residuals from a mixed-model ANOVA don't look too bad—run the do-file below to see whether they're acceptable to you.

                  I agree with Marcos that you'll likely come up dry with pairwise testing, given what you've got to work with, regardless of which approach you take.

                  Code:
                  version 15.0
                  
                  clear *
                  set more off
                  input double (omn1 omn2 omn3 omn4 omn5)
                  53.72665995     66.85657838     75.89328604     78.31190716     89.02076946
                  91.50458223     95.42352433     96.04331537     97.40200777     93.41464764
                  81.55195409     84.3015418     91.84736411     91.38936236     95.02700639
                  91.53903198     89.16191657     87.18201087     91.90145989     92.66668696
                  83.76553697     83.7352378     92.32404343     84.95262814     89.50057555
                  93.6355134     90.28165779     93.17900586     92.71120876     93.6861227
                  end
                  
                  generate byte pid = _n
                  quietly reshape long omn, i(pid) j(prc)
                  
                  anova omn prc pid
                  
                  predict double omn_res, residuals
                  pnorm omn_res
                  qnorm omn_res
                  
                  contrast prc, pveffects mcompare(bonferroni)
                  
                  test, showorder
                  #delimit ;
                  matrix input Contrasts = (
                      -1 1 0 0 0 0 0 0 0 0 0 0 \
                      -1 0 1 0 0 0 0 0 0 0 0 0 \
                      -1 0 0 1 0 0 0 0 0 0 0 0 \
                      -1 0 0 0 1 0 0 0 0 0 0 0 \
                      0 -1 1 0 0 0 0 0 0 0 0 0 \
                      0 -1 0 1 0 0 0 0 0 0 0 0 \
                      0 -1 0 0 1 0 0 0 0 0 0 0 \
                      0 0 -1 1 0 0 0 0 0 0 0 0 \
                      0 0 -1 0 1 0 0 0 0 0 0 0 \
                      0 0 0 -1 1 0 0 0 0 0 0 0);
                  #delimit cr
                  test, test(Contrasts) mtest(holm)
                  
                  exit

                  Comment

                  Working...
                  X