Pool of multiple ICC

[Hayder Alhusseinawi]

Hi
I made an intra-rater reliability test (test-retest)for 8 raters that judged 31 videos 2 times. I got 8 different ICC and CI running a Two-way mixed-effects model. I would like now to report the pool of all ICC and CI. Is it enough to take the median, or do I have to do meta-analysis. in this case, which comment I need to use in stata.
Thank you
Best regards
Hayder

[Leonardo Guizzetti]

It’s not clear how you managed to fit an overidentified two-way random effects model for each rater. To find the overall intrarater ICC you would use an appropriate two way model on the full dataset from all raters.

[Hayder Alhusseinawi]

It’s not clear how you managed to fit an overidentified two-way random effects model for each rater. To find the overall intrarater ICC you would use an appropriate two way model on the full dataset from all raters.

Thank you for your reply, Leonardo. I chose a 2-way mixed effect base on Koo guidline.
Thank you very much, so no need for meta analysis. Do you mean I have to regard all 8 raters as one rater, and then run ICC between 1st and 2nd evaluation?

Koo TK, Li MY. A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. J Chiropr Med. 2016;15(2):155-163. doi:10.1016/j.jcm.2016.02.012

[Joseph Coveney]

Do you mean I have to regard all 8 raters as one rater, and then run ICC between 1st and 2nd evaluation?

You could try fitting a two-way mixed (crossed) model and then computing the raters' ICC from the variance components—maybe something like the following approach. (Begin at the "Begin here" comment; the stuff above is just to create a dataset for illustration.)

.ÿ
.ÿversionÿ17.0

.ÿ
.ÿclearÿ*

.ÿ
.ÿ//ÿseedem
.ÿsetÿseedÿ358594463

.ÿ
.ÿ//ÿRaters
.ÿquietlyÿsetÿobsÿ8

.ÿgenerateÿbyteÿridÿ=ÿ_n

.ÿgenerateÿdoubleÿrid_uÿ=ÿrnormal(0,ÿ2)

.ÿ
.ÿ//ÿSessions
.ÿquietlyÿexpandÿ2

.ÿbysortÿrid:ÿgenerateÿbyteÿtimÿ=ÿ_n

.ÿ
.ÿtempfileÿraters

.ÿquietlyÿsaveÿ`raters'

.ÿ
.ÿ//ÿMovies
.ÿdropÿ_all

.ÿ
.ÿquietlyÿsetÿobsÿ31

.ÿgenerateÿbyteÿmidÿ=ÿ_n

.ÿgenerateÿdoubleÿmid_uÿ=ÿrnormal(0,ÿ1)

.ÿcrossÿusingÿ`raters'

.ÿ
.ÿ//ÿRatings
.ÿgenerateÿdoubleÿscoÿ=ÿ(timÿ-ÿ1)ÿ/ÿ4ÿ+ÿrid_uÿ+ÿmid_uÿ+ÿrnormal(0,ÿsqrt(0.5))

.ÿ
.ÿ*
.ÿ*ÿBeginÿhere
.ÿ*
.ÿmixedÿscoÿi.timÿ||ÿ_all:ÿR.ridÿ||_all:ÿR.mid,ÿremlÿdfmethod(kroger)ÿnolrtestÿnolog

Mixed-effectsÿREMLÿregressionÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿNumberÿofÿobsÿÿÿÿÿ=ÿÿÿÿÿÿÿÿ496
Groupÿvariable:ÿ_allÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿNumberÿofÿgroupsÿÿ=ÿÿÿÿÿÿÿÿÿÿ1
ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿObsÿperÿgroup:
ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿminÿ=ÿÿÿÿÿÿÿÿ496
ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿavgÿ=ÿÿÿÿÿÿ496.0
ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿmaxÿ=ÿÿÿÿÿÿÿÿ496
DFÿmethod:ÿKenward–RogerÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿDF:ÿÿÿÿÿÿÿÿÿÿÿminÿ=ÿÿÿÿÿÿÿ9.49
ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿavgÿ=ÿÿÿÿÿ233.24
ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿmaxÿ=ÿÿÿÿÿ457.00
ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿF(1,ÿÿÿ457.00)ÿÿÿÿ=ÿÿÿÿÿÿÿ7.57
Logÿrestricted-likelihoodÿ=ÿ-617.13024ÿÿÿÿÿÿÿÿÿÿProbÿ>ÿFÿÿÿÿÿÿÿÿÿÿ=ÿÿÿÿÿ0.0062

------------------------------------------------------------------------------
ÿÿÿÿÿÿÿÿÿscoÿ|ÿCoefficientÿÿStd.ÿerr.ÿÿÿÿÿÿtÿÿÿÿP>|t|ÿÿÿÿÿ[95%ÿconf.ÿinterval]
-------------+----------------------------------------------------------------
ÿÿÿÿÿÿÿ2.timÿ|ÿÿÿ.1796117ÿÿÿ.0652999ÿÿÿÿÿ2.75ÿÿÿ0.006ÿÿÿÿÿ.0512863ÿÿÿÿÿ.307937
ÿÿÿÿÿÿÿ_consÿ|ÿÿ-1.500932ÿÿÿÿ.457513ÿÿÿÿ-3.28ÿÿÿ0.009ÿÿÿÿ-2.527814ÿÿÿ-.4740506
------------------------------------------------------------------------------

------------------------------------------------------------------------------
ÿÿRandom-effectsÿparametersÿÿ|ÿÿÿEstimateÿÿÿStd.ÿerr.ÿÿÿÿÿ[95%ÿconf.ÿinterval]
-----------------------------+------------------------------------------------
_all:ÿIdentityÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ|
ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿvar(R.rid)ÿ|ÿÿÿÿ1.42495ÿÿÿ.7662267ÿÿÿÿÿÿ.4966956ÿÿÿÿ4.087985
-----------------------------+------------------------------------------------
_all:ÿIdentityÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ|
ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿvar(R.mid)ÿ|ÿÿÿÿ.901085ÿÿÿ.2412017ÿÿÿÿÿÿ.5332327ÿÿÿÿ1.522701
-----------------------------+------------------------------------------------
ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿvar(Residual)ÿ|ÿÿÿ.5287461ÿÿÿ.0349787ÿÿÿÿÿÿ.4644476ÿÿÿÿ.6019462
------------------------------------------------------------------------------

.ÿ
.ÿtempnameÿVraterÿVmovieÿVresidual

.ÿscalarÿdefineÿ`Vrater'ÿ=ÿexp(_b[lns1_1_1:_cons])^2

.ÿscalarÿdefineÿ`Vmovie'ÿ=ÿexp(_b[lns1_2_1:_cons])^2

.ÿscalarÿdefineÿ`Vresidual'ÿ=ÿexp(_b[lnsig_e:_cons])^2

.ÿ
.ÿdisplayÿinÿsmclÿasÿtextÿ"ICCÿ=ÿ"ÿ%04.2fÿ`Vrater'ÿ/ÿ(`Vrater'ÿ+ÿ`Vmovie'ÿ+ÿ`Vresidual')
ICCÿ=ÿ0.50

.ÿ
.ÿexit

endÿofÿdo-file


.

[Leonardo Guizzetti]

A two-way mixed model would be appropriate if you regard subjects as random representatives of a larger some larger population (i.e., random effect) and only these judges of interest (i.e., a fixed effect). If instead judges should also be considered as a random effect, then you simply switch to a two-way random-effects model. In either case, you can estimate a two-way mixed- (or random) effects models on all data to obtain an estimate of the inter- and intra-rater ICC. The model naturally accommodates replicates. One approach is to structure your data with one observation for each repeated measure on each target, and one variable per judge. This might look like the fake data below, which shows some fake data on targets with repeated measures (identifed by -repl-) and judges scores (yhat1 to yhat4, assuming 4 judges):

     +-----------------------------------------------+
     | target   repl   yhat1   yhat2   yhat3   yhat4 |
     |-----------------------------------------------|
  1. |      1      1       6       3       8       6 |
  2. |      1      2       4       3       6       7 |
  3. |      2      1       5       1       4       7 |
  4. |      2      2       5       0       5       6 |
  5. |      3      1       6       4      10       9 |
..... etc

Then you can use Daniel Klein's -kappaetc- (SSC and Stata Journal 18(4):871--901). This will return the inter- and intra-rater ICCs and (two-sided) confidence intervals.

kappaetc yhat* , icc(mixed) id(target) var

edit: crossed with #4, which shows a nice simulation-based method.

[Hayder Alhusseinawi]

Then you can use Daniel Klein's -kappaetc- (SSC and Stata Journal 18(4):871--901). This will return the inter- and intra-rater ICCs and (two-sided) confidence intervals.

kappaetc yhat* , icc(mixed) id(target) var

edit: crossed with #4, which shows a nice simulation-based method.

Thank you for answering. it's look like i have just to re-arrange my data. I follow your instruction and I get the following results.


kappaetc Rater1 Rater2 Rater3 Rater4 Rater5 Rater6 Rater7 Rater8, icc(mixed) id(target)

Inter/Intrarater reliability Number of subjects = 31
Two-way mixed-effects model Ratings per subject = 16
Replicates per subject = 2
------------------------------------------------------------------------------
| Coef. F df1 df2 P>F [95% Conf. Interval]
---------------+--------------------------------------------------------------
ICC(3,1) | 0.7486 34.66 30.00 210.00 0.000 0.6618 0.8553
ICC_a(3,1) | 0.8892 15.26 45.10 248.00 0.000 0.8359 0.9307
---------------+--------------------------------------------------------------
sigma_s | 1.1200
sigma_sr | 0.4488
sigma_e | 0.4260
------------------------------------------------------------------------------


Best regards.
Hayder

[Hayder Alhusseinawi]

You could try fitting a two-way mixed (crossed) model and then computing the raters' ICC from the variance components—maybe something like the following approach. (Begin at the "Begin here" comment; the stuff above is just to create a dataset for illustration.)

.ÿ
.ÿversionÿ17.0

.ÿ
.ÿclearÿ*

.ÿ
.ÿ//ÿseedem
.ÿsetÿseedÿ358594463

.ÿ
.ÿ//ÿRaters
.ÿquietlyÿsetÿobsÿ8

.ÿgenerateÿbyteÿridÿ=ÿ_n

.ÿgenerateÿdoubleÿrid_uÿ=ÿrnormal(0,ÿ2)

.ÿ
.ÿ//ÿSessions
.ÿquietlyÿexpandÿ2

.ÿbysortÿrid:ÿgenerateÿbyteÿtimÿ=ÿ_n

.ÿ
.ÿtempfileÿraters

.ÿquietlyÿsaveÿ`raters'

.ÿ
.ÿ//ÿMovies
.ÿdropÿ_all

.ÿ
.ÿquietlyÿsetÿobsÿ31

.ÿgenerateÿbyteÿmidÿ=ÿ_n

.ÿgenerateÿdoubleÿmid_uÿ=ÿrnormal(0,ÿ1)

.ÿcrossÿusingÿ`raters'

.ÿ
.ÿ//ÿRatings
.ÿgenerateÿdoubleÿscoÿ=ÿ(timÿ-ÿ1)ÿ/ÿ4ÿ+ÿrid_uÿ+ÿmid_uÿ+ÿrnormal(0,ÿsqrt(0.5))

.ÿ
.ÿ*
.ÿ*ÿBeginÿhere
.ÿ*
.ÿmixedÿscoÿi.timÿ||ÿ_all:ÿR.ridÿ||_all:ÿR.mid,ÿremlÿdfmethod(kroger)ÿnolrtestÿnolog

Mixed-effectsÿREMLÿregressionÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿNumberÿofÿobsÿÿÿÿÿ=ÿÿÿÿÿÿÿÿ496
Groupÿvariable:ÿ_allÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿNumberÿofÿgroupsÿÿ=ÿÿÿÿÿÿÿÿÿÿ1
ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿObsÿperÿgroup:
ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿminÿ=ÿÿÿÿÿÿÿÿ496
ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿavgÿ=ÿÿÿÿÿÿ496.0
ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿmaxÿ=ÿÿÿÿÿÿÿÿ496
DFÿmethod:ÿKenward–RogerÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿDF:ÿÿÿÿÿÿÿÿÿÿÿminÿ=ÿÿÿÿÿÿÿ9.49
ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿavgÿ=ÿÿÿÿÿ233.24
ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿmaxÿ=ÿÿÿÿÿ457.00
ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿF(1,ÿÿÿ457.00)ÿÿÿÿ=ÿÿÿÿÿÿÿ7.57
Logÿrestricted-likelihoodÿ=ÿ-617.13024ÿÿÿÿÿÿÿÿÿÿProbÿ>ÿFÿÿÿÿÿÿÿÿÿÿ=ÿÿÿÿÿ0.0062

------------------------------------------------------------------------------
ÿÿÿÿÿÿÿÿÿscoÿ|ÿCoefficientÿÿStd.ÿerr.ÿÿÿÿÿÿtÿÿÿÿP>|t|ÿÿÿÿÿ[95%ÿconf.ÿinterval]
-------------+----------------------------------------------------------------
ÿÿÿÿÿÿÿ2.timÿ|ÿÿÿ.1796117ÿÿÿ.0652999ÿÿÿÿÿ2.75ÿÿÿ0.006ÿÿÿÿÿ.0512863ÿÿÿÿÿ.307937
ÿÿÿÿÿÿÿ_consÿ|ÿÿ-1.500932ÿÿÿÿ.457513ÿÿÿÿ-3.28ÿÿÿ0.009ÿÿÿÿ-2.527814ÿÿÿ-.4740506
------------------------------------------------------------------------------

------------------------------------------------------------------------------
ÿÿRandom-effectsÿparametersÿÿ|ÿÿÿEstimateÿÿÿStd.ÿerr.ÿÿÿÿÿ[95%ÿconf.ÿinterval]
-----------------------------+------------------------------------------------
_all:ÿIdentityÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ|
ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿvar(R.rid)ÿ|ÿÿÿÿ1.42495ÿÿÿ.7662267ÿÿÿÿÿÿ.4966956ÿÿÿÿ4.087985
-----------------------------+------------------------------------------------
_all:ÿIdentityÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ|
ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿvar(R.mid)ÿ|ÿÿÿÿ.901085ÿÿÿ.2412017ÿÿÿÿÿÿ.5332327ÿÿÿÿ1.522701
-----------------------------+------------------------------------------------
ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿvar(Residual)ÿ|ÿÿÿ.5287461ÿÿÿ.0349787ÿÿÿÿÿÿ.4644476ÿÿÿÿ.6019462
------------------------------------------------------------------------------

.ÿ
.ÿtempnameÿVraterÿVmovieÿVresidual

.ÿscalarÿdefineÿ`Vrater'ÿ=ÿexp(_b[lns1_1_1:_cons])^2

.ÿscalarÿdefineÿ`Vmovie'ÿ=ÿexp(_b[lns1_2_1:_cons])^2

.ÿscalarÿdefineÿ`Vresidual'ÿ=ÿexp(_b[lnsig_e:_cons])^2

.ÿ
.ÿdisplayÿinÿsmclÿasÿtextÿ"ICCÿ=ÿ"ÿ%04.2fÿ`Vrater'ÿ/ÿ(`Vrater'ÿ+ÿ`Vmovie'ÿ+ÿ`Vresidual')
ICCÿ=ÿ0.50

.ÿ
.ÿexit

endÿofÿdo-file


.

Dear Sir,
ُThank you veru much Sir. Actually the method you mention is a little bit advanced for my begginer level as i didn't understand what did you mean with variables mid_u and rid_u. But I did what Mr. Leonardo recommended by changing data ordination.

Best Regards
Hayder