Thanks to Kit Baum there is a package for estimating ICC using mixed crossed-effect REML regression and nlcom.
Textbooks often calculate the ICC using sums of squares on a subject-by-measurement matrix with non-missing cells.
The idea of the ICC is to compare the wanted variation explained by a factor variable on an outcome with the total variation, the total variation being the wanted variation by the factor variable plus the unwanted variation.
Bias occurs sometimes from the measurement repetitions.
The ANOVA-like calculations ignore all measurements by a subject if just one measurement for that subject is missing and may also return ICC estimates below zero.
The latter is theoretically impossible.
To better utilize subjects with missing measurements and avoid obtaining negative ICCs, it is better to use estimates from a mixed, crossed regression.
The command icc2 returns a matrix with the absolute and consistency ICCs, a 95% confidence interval, and a P-value for the ICCs equal to zero.
The user can obtain more precise confidence intervals using the bootstrap.
Examples:
Enjoy
Textbooks often calculate the ICC using sums of squares on a subject-by-measurement matrix with non-missing cells.
The idea of the ICC is to compare the wanted variation explained by a factor variable on an outcome with the total variation, the total variation being the wanted variation by the factor variable plus the unwanted variation.
Bias occurs sometimes from the measurement repetitions.
The ANOVA-like calculations ignore all measurements by a subject if just one measurement for that subject is missing and may also return ICC estimates below zero.
The latter is theoretically impossible.
To better utilize subjects with missing measurements and avoid obtaining negative ICCs, it is better to use estimates from a mixed, crossed regression.
The command icc2 returns a matrix with the absolute and consistency ICCs, a 95% confidence interval, and a P-value for the ICCs equal to zero.
The user can obtain more precise confidence intervals using the bootstrap.
Examples:
Code:
Setup
. webuse judges, clear
Calculate ICCs for one-way random-effects model
. icc2 rating target
| ICC [95% CI] P(ICC=0)
-------------+---------------------------------------
absolute | 0.166 -0.272 0.603 0.458
A mixed crossed regression and nlcom is the basis for the confidence interval whereas the icc uses the F-distribution.
. icc rating target
rating | ICC [95% conf. interval]
-----------------------+--------------------------------------
Individual | .1657418 -.1329323 .7225601
We can use bootstrap options to get a more precise confidence interval.
Setting the option seed alone, the default number of repetitions is 50.
. icc2 rating target, seed(1)
| ICC [95% CI] P(ICC=0)
-------------+---------------------------------------
absolute | 0.166 -0.199 0.531 0.374
Calculate ICCs for two-way random-effects model.
. icc2 rating target judge
| ICC [95% CI] P(ICC=0)
-------------+---------------------------------------
absolute | 0.290 -0.111 0.691 0.157
consistency | 0.715 0.394 1.036 0.000
We can use the bootstrap with 100 repetitions for the confidence interval.
. icc2 rating target judge, seed(1) reps(100)
| ICC [95% CI] P(ICC=0)
-------------+---------------------------------------
absolute | 0.290 0.119 0.461 0.001
consistency | 0.715 0.560 0.870 0.000
The difference between the absolute and consistency ICCs indicates bias from the judges.
Calculate ICCs for two-way mixed-effects model
As argued in 2019 Liljequist, the two-way random-effects model is the same as the two-way mixed-effects model.
. icc2 rating target judge
