Running ICC intra-observer reliability data in STATA

[Pa Hol]

Hi
I am currently trying to figure out how to run ICC intra-observer commands on my data (ICC 3,1).
My excel file currently looks like this:
- First collumn represents participants and each following collumn represents a datapoint derived pre and post, i.e. a1, a2, b1, b2 and so on.
- First row represents particant ID and each following row represents participants scores.

When I try to execute the ICC commands in STATA, i am told to list dependent variable, target variable and rater variable - how can I set the enlisted variables up in a smart way in my excel file?

[wbuchanan]

It'd probably be more useful to show a sample of deidentified data and/or to look at the dataset structure used to illustrate the examples in the help files.

[Pa Hol]

Im an not sure if I understand what you mean?

This is my first experience with reliability statistics in STATA - hence the dificulties.

Here is a small sample of my intra-observer data in excel.

ID	A1	A2	B1
1	0,32	0,28	0,23
2	0,4	0,31	0,34
3	0,39	0,37	0,42
4	0,2	0,21	0,22

As I mentioned, im trying to run ICC two-way mixed effects model (3.1). However, I cant figure out how to enlist "dependent variable", "target variable" and "rater variable".
It would be greatly appreciated if you wbuchanan could help me to enlist the variables correctly. I have read the help-files without much success.

[wbuchanan]

Pa Hol in the help for the icc command if you scroll towards the bottom you should see:

Examples

    Setup
        . webuse judges

    Calculate ICCs for one-way random-effects model
        . icc rating target

    Same as above but test whether ICCs equal 0.5
        . icc rating target, testvalue(.5)

    Calculate ICCs for two-way random-effects model
        . icc rating target judge

    Same as above but estimate consistency of agreement
        . icc rating target judge, consistency

    Calculate ICCs for two-way mixed-effects model
        . icc rating target judge, mixed

    Same as above but estimate absolute agreement
        . icc rating target judge, mixed absolute

If you enter the first command:

webuse judges, clear

You'll load the data used in the examples. Then if you use the -browse- command you can view how the data are structured and compare it to the data in your file.

[Pa Hol]

Thanks wbuchanan, I was finally able to get a result on one of my measurements.

As I initially want to analyse INTRA observer reliability, I have discriminated 1. and 2. ratings from the same observer as 2 independent raters. This is currently the only solution i can come up with, but im not sure if this the correct way of doing an intra-observer calculation.

On the other hand, I have read that ICC is NOT meant for INTRA-observer reliability - if this is the case, then what method should I use?

[wbuchanan]

Could you provide some references as to what you are trying to do? It may help others to identify existing commands/user-written programs that provide the functionality that you are looking for.

[Pa Hol]

I am doing an intra-inter rater reliability study for two observers. First step is to measure the intra-rater reliability for one observer pre and post. For this I have been using ICC twoway mixed effects model. In order to perform the actual command in STATA, first and second observation for the same observer is listed as two independent raters. However, im not sure if this is a correct way to measure the intra-rater reliability. If not, then how should I calculate intra-rater reliability?

[wbuchanan]

Pa Hol I'm getting a bit more confused with your latest response. I'm not sure if the code is published anywhere, but you might try to connect with Andrew Ho and Thomas Kane at Harvard Graduate School of Education regarding their reliability study of educator observations as part of the Measures of Effective Teaching project.

[Nazzarena]

Hi

I also am unclear. The help file and example data clearly point to a long format where rating is the depvar, target is the variable identifying individual measured, and judge is the variable identifying the raters.
In my case, it would look like this, where pid is the record identifier for patients (int 1-118), nag is the category/dimension to be measured (int 1-73) n is the "rater" (2 technical repeats, so 1 and 2) and value is the result of the measurement (float). Note that i encoded nag (and pid for good measure) as they were string, the command takes numeric arguments and would give no observations

pid	nag	value	n
1	1	139	1
1	2	51	1
1	3	62	1
1	4	41	1
1	5	8341	1
1	6	222.5	1
1	7	146	1
1	8	68	1
1	9	128	1
1	10	232	1

thus it should look like

icc value nag n

it says you cannot run the command with more than one measurement. Fair enough, i tried to run by category

bysort nag: icc value pid n

it says

unrecognized command: icc value

helpfile does not specify you cannot combine with by

Thank you for any insight!

[Joseph Coveney]

i tried to run by category

bysort nag: icc value pid n

it says

unrecognized command: icc value

Hmm . . . The following runs for me without a hitch.

set seed 1404860

drawnorm value1 value2, double corr(1 0.5 \ 0.5 1) n(600) clear
generate byte nag = mod(_n, 3) + 1
bysort nag: generate int pid = _n
quietly reshape long value, i(pid nag) j(n)

bysort nag: icc value pid n

exit

[Nicoletta Riva]

Hi,

I would like to calculate the intra-class correlation coefficients for repeated measurements (in my case it is a test-retest of a questionnaire which gives a continuous score), but I cannot find the icc command in STATA. I am using STATA/SE 12 but it says "unrecognized command: icc". I have tried also "ssc install icc" but it says "ssc install: "icc" not found at SSC".

Do you have any suggestions?

Thanks in advance,

Nicoletta

[daniel klein]

As far I remember, icc was added in Stata 12, perhaps Stata 12.1, so you should update your version of Stata. However, the icc command will not allow repeated measures per subject. You may want to download kappaetc from SSC; the latter estimates ICC for repeated measures. Note that the current version of kappaetc has a bug and will give you incorrect results if the data is not sorted by subject identifier. I will send an ad-hoc fix to Kit Baum within the next days.

Best
Daniel

[Joseph Coveney]

Your best avenue would be to upgrade, but if you cannot do that, then you might fit a random effects (or fixed effects) model to your data with xtreg and take a look at the value reported for rho in the output.