ICC reliability with self-reported data

[Mark Wilberforce]

Hi folks

I cannot get my head around ICC when I have self-reported data at two time-points in a test-retest analysis.

I have 100 self-rated observations at T1 and T2. Each person obviously rates themself and no-one else. The "raters" and "targets" are the same.

Can I please check my understanding.

To me, this is akin to saying that we 100 targets of measurement, and each of these is measured by "2 raters" - themselves at two time points (the time between them meaning, in theory, it is an independent rating).

Does this stack up? So in STATA I am reshaping to long, and then running icc depvar id time. Where depvar is the variable of interest, id is the person's unique identifier, and time is the T1 and T2 rating.

Could someone please check my logic?

Thank you so much

Mark Wilberforce

[Carlo Lazzaro]

Mark:
as far as I can get your analysis, I would cast some doubts about any assumption of independence about the two waves of data.
I would feel more comfortable with a two-wave panel data regression (-xt- suite will offer the panel data regression model that fits your -depvar-).

[Mark Wilberforce]

Hi Carlo
Much appreciated. The benefit of Intraclass Correlation Coefficients is that it measures agreement in a common, transportable and widely interpreted format. So I am trying to stay within the ICC "family" of statistical testing. There are widely recognised thresholds etc.

So ICCs allow you to consider fixed and random effects, but in a test-retest reliability the same rater, rates themselves, at two time points.. I'm just struggling to understand the terminology. ICCs were designed with the idea of lots of raters (eg clinicians) measuring the same patients (more inter-rater reliability than test-retest). I'm concerned about choosing the wrong ICC form.

Am very grateful for your reply.

Best wishes, Mark

[Carlo Lazzaro]

Mark:
admittedly, the main concern about your reasoning rests on the fact that each individual rates her/himself.
This is not my reasearch field but, as far as I heard or read, ICC is used to test, for instance, if a panel of independent radiologists consider a given TC scan well done or not by other radiologists.

[Mark Wilberforce]

Yes, ICC was designed with your example in mind. For people's judgement of service quality they receive (my questionnaire) it is necessarily self-report.

[Carlo Lazzaro]

Mark.
thanks for your clarification.
The main issue then is whether the approach you've in mind was already used by other researchers in your scientific field (just in case you had to throw reviewers a bone...).