Announcement

Collapse
No announcement yet.
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Test-retest reliability?

    Hi Stata forum

    I have repeated measures of the same thing on the same subjects with a variable number (2 to 5) measurements per subject. What is the best way of measuring test-retest reliability in Stata?

    Thankyou

    Karin

  • #2
    If the measures are on a continuous scale, the conventional approach would be the intraclass correlation. -help icc-

    Comment


    • #3
      I had had a look at icc, but I couldn't see how to apply it to my situation. It has the syntax "icc depvar target" but I don't have a dependent variable and a target. I have several measurements per person of equal status. Sorry if I am being stupid.

      Comment


      • #4
        The dependent variable refers to your measurement, and the target refers to the person being measured.

        Comment


        • #5
          Oops I was being stupid. Thankyou Clyde.

          Comment


          • #6
            As I mentioned I have variable numbers of measurements per subject (2-5). When I run icc I get the message that targets have been omitted from computation because of unbalanced data. The results seem to include only the subjects with 5 measurements.

            Comment


            • #7
              That's odd. Here's a different approach:

              Code:
              mixed measurement || subject_id:
              estat icc
              This will not give you any trouble with unbalanced data.

              Comment


              • #8
                Thankyou Clyde--that was just what I needed.

                Comment


                • #9
                  Originally posted by Clyde Schechter View Post
                  That's odd. Here's a different approach:

                  Code:
                  mixed measurement || subject_id:
                  estat icc
                  This will not give you any trouble with unbalanced data.
                  It will likely yield a slightly different result, though. You can replicate icc results with xtreg using the GLS estimator. The ML estimator will yield the same result as the mixed approach. Asymptotically, the two results should be the same, I think.

                  Code:
                  webuse judges , clear
                  icc rating target
                  
                  quietly mixed rating || target :
                  estat icc
                  yields

                  Code:
                  . webuse judges , clear
                  (Ratings of targets by judges)
                  
                  . icc rating target
                  
                  Intraclass correlations
                  One-way random-effects model
                  Absolute agreement
                  
                  Random effects: target           Number of targets =         6
                                                   Number of raters  =         4
                  
                  --------------------------------------------------------------
                                  rating |        ICC       [95% Conf. Interval]
                  -----------------------+--------------------------------------
                              Individual |   .1657418      -.1329323    .7225601
                                 Average |   .4427971      -.8844422    .9124154
                  --------------------------------------------------------------
                  F test that
                    ICC=0.00: F(5.0, 18.0) = 1.79               Prob > F = 0.165
                  
                  Note: ICCs estimate correlations between individual measurements
                        and between average measurements made on the same target.
                  
                  .
                  . quietly mixed rating || target :
                  
                  . estat icc
                  
                  Intraclass correlation
                  
                  ------------------------------------------------------------------------------
                                         Level |        ICC   Std. Err.     [95% Conf. Interval]
                  -----------------------------+------------------------------------------------
                                        target |   .1102339   .1973357      .0023957    .8647096
                  ------------------------------------------------------------------------------
                  Compare with

                  Code:
                  xtset target
                  xtreg rating
                  xtreg rating , mle
                  which yields

                  Code:
                  . xtset target
                         panel variable:  target (balanced)
                  
                  . xtreg rating
                  
                  Random-effects GLS regression                   Number of obs     =         24
                  Group variable: target                          Number of groups  =          6
                  
                  R-sq:                                           Obs per group:
                       within  = 0.0000                                         min =          4
                       between = 0.0000                                         avg =        4.0
                       overall = 0.0000                                         max =          4
                  
                                                                  Wald chi2(0)      =          .
                  corr(u_i, X)   = 0 (assumed)                    Prob > chi2       =          .
                  
                  ------------------------------------------------------------------------------
                        rating |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
                  -------------+----------------------------------------------------------------
                         _cons |   5.291667   .6843996     7.73   0.000     3.950268    6.633065
                  -------------+----------------------------------------------------------------
                       sigma_u |  1.1155467
                       sigma_e |  2.5027762
                           rho |  .16574177   (fraction of variance due to u_i)
                  ------------------------------------------------------------------------------
                  
                  . xtreg rating , mle
                  Iteration 0:   log likelihood = -57.297209
                  Iteration 1:   log likelihood = -57.279795
                  Iteration 2:   log likelihood = -57.279652
                  Iteration 3:   log likelihood = -57.279652
                  
                  Random-effects ML regression                    Number of obs     =         24
                  Group variable: target                          Number of groups  =          6
                  
                  Random effects u_i ~ Gaussian                   Obs per group:
                                                                                min =          4
                                                                                avg =        4.0
                                                                                max =          4
                  
                                                                  Wald chi2(0)      =       0.00
                  Log likelihood  = -57.279652                    Prob > chi2       =          .
                  
                  ------------------------------------------------------------------------------
                        rating |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
                  -------------+----------------------------------------------------------------
                         _cons |   5.291667   .6247685     8.47   0.000     4.067143     6.51619
                  -------------+----------------------------------------------------------------
                      /sigma_u |   .8809323   .8226624                      .1412682    5.493394
                      /sigma_e |   2.502776   .4171294                      1.805324    3.469676
                           rho |   .1102343   .1973356                      .0005208    .7963012
                  ------------------------------------------------------------------------------
                  LR test of sigma_u=0: chibar2(01) = 0.39               Prob >= chibar2 = 0.267
                  Edit: It is interesting that the CIs seem to differ quite substantially.

                  Best
                  Daniel
                  Last edited by daniel klein; 13 Oct 2017, 10:41.

                  Comment


                  • #10
                    Daniel,
                    Code:
                    quietly mixed . . ., reml
                    estat icc

                    Comment


                    • #11
                      Joseph, thanks for the hint. REML indeed matches the point estimate. CIs are still off.

                      Best
                      Daniel

                      Comment


                      • #12
                        Daniel, I'm guessing that the CIs differ because icc is based on the ANOVA model formulation, which allows for negative estimates for variance components above the residual (and thus, in principle, negative ICC estimates), and estat icc is derived from an iterative maximum (restricted) likelihood fit, which constrains all variance component estimates to be nonnegative (maybe even strictly positive). So, the CI for icc is wider (and includes negative values) while the CI for estat icc is conditional on the ICC's being nonnegative and is narrower.

                        Comment


                        • #13
                          Originally posted by Joseph Coveney View Post
                          So, the CI for icc is wider . . . while the CI for estat icc is . . . is narrower.
                          Sorry, I should have said, "So, the CIs for estat ci are shifted rightward relative to those CIs from icc", at least for those of the latter that entertain negative values.

                          Comment


                          • #14
                            Originally posted by Joseph Coveney View Post
                            Daniel, I'm guessing that the CIs differ because icc is based on the ANOVA model formulation, which allows for negative estimates for variance components above the residual (and thus, in principle, negative ICC estimates), and estat icc is derived from an iterative maximum (restricted) likelihood fit, which constrains all variance component estimates to be nonnegative (maybe even strictly positive).
                            Thanks for this further explanation which sounds plausible. I have done a bit of reading in the manuals myself and it seems there are fundamental differences in the way CIs are estimated. In the ANOVA framework (icc) the CIs are based directly on the respective sums of squares and the F-distribution while in the mixed framework a logit transformation and normal approximation is involved. So it seems it is not only the restriction of the estimated variance components but other details that differ between the models as well.

                            Best
                            Daniel

                            Comment


                            • #15
                              Is there a way to incorporate information about the order of waves here? For a given person a score in five successive waves could be (2,3,4,5,6) which would suggest reasonably high test-retest reliability or (2,4,6,3,4) which would not, but I'm not an icc would pick up the difference here. I know how to model auto-regressive residuals in mixed so maybe a formula using that?

                              Best,
                              Chris

                              Comment

                              Working...
                              X