inter rater reliability

[Tommaso Salvitti]

Hi to everybody.
I have to calculate inter rater reliability.

* Example generated by -dataex-. For more info, type help dataex
clear
input byte(Codice_soggetto Giudici A B C D E F)
 1 1 . . .  .  .  .
 1 2 2 2 4 17 17 40
 2 1 3 1 1 15 16 33
 2 2 2 2 2 16 16 36
 3 1 2 2 3 14 16 35
 3 2 2 2 4 17 18 41
 4 1 2 2 2 13 18 35
 4 2 1 2 4 18 18 42
 5 1 2 2 3 15 17 37
 5 2 2 2 4 17 18 41
 6 1 2 2 2 17 18 39
 6 2 1 2 4 18 18 42
 7 1 3 1 1 15 15 32
 7 2 3 1 2 12 12 27
 8 1 . . .  .  .  .
 8 2 2 2 4 17 16 39
 9 1 2 2 3 16 18 39
 9 2 2 2 4 17 18 41
10 1 2 2 3 15 16 36
10 2 2 2 4 18 18 42
11 1 2 2 2 14 18 36
11 2 3 1 4 15 17 37
12 1 2 2 1 14 16 33
12 2 3 1 2  9 10 22
13 1 1 2 3 18 18 41
13 2 2 2 4 18 18 42
14 1 3 1 1 16 16 34
14 2 3 1 3 17 18 39
15 1 2 2 2 18 18 40
15 2 1 2 4 18 18 42
16 1 2 2 2 16 17 37
16 2 2 2 4 17 18 41
17 1 2 2 1 17 18 38
17 2 3 1 3 18 18 40
18 1 . . .  .  .  .
18 2 2 2 3 14 13 32
19 1 1 2 4 16 16 38
19 2 1 2 4 17 18 41
 . . . . .  .  .  .
end

Out of 19 children (COLUMN A, codice_soggetto) two "judges" (COLUMN B) gave different scores on the variables A, B, C D, E and F (columns C, D, E, F, G and H respectively). How to calculate the inter rater reliability
For three subject (Codice_soggetto 1, Codice_soggetto 8 e codice_soggetto 18) one judge did not express any evaluation

Thanks in advance to everybody

[Joseph Coveney]

I have to calculate inter rater reliability. . . . two "judges" . . . gave different scores on the variables A, B, C D, E and F . . . How to calculate the inter rater reliability

First combine the individual scores for each judge and child into a summary variable. Then use mixed followed by estat icc in order to compute the mixed-model (judges fixed and children random) intraclass correlation coefficient for that summary rating score.

If the dataset were balanced, then you could've used icc . . . , mixed as well.

[Tommaso Salvitti]

Good morning Joseph Coveney and to everybody. I'll try this code. i don't know if it's correct.
I changed the names of the variables A, B,C , D, E, F in item1 and so on

rename *, lower  rename Codice_soggetto sid quietly reshape long item, i(sid) j(tmp) string  rename item sco  gen byte rid=substr(tmp, -1, 1)=="b" gen byte itm=real(substr(tmp,1, strlen(tmp)-1))  mixed sco i.rid ib6.itm || sid:, nolog  estat icc

Is it correct according to you?

[Tommaso Salvitti]

sorry for the way I wrote the syntax
Good morning Joseph Coveney and to everybody. I'll try this code. i don't know if it's correct.
I changed the names of the variables A, B,C , D, E, F in item1 and so on

rename *, lower

rename soggetti sid
quietly reshape long item, i(sid) j(tmp) string

rename item sco
 gen byte rid=substr(tmp, -1, 1)=="b"
gen byte itm=real(substr(tmp,1, strlen(tmp)-1))

mixed sco i.rid ib6.itm || sid:, nolog

estat icc

Is it correct according to you?

[Tommaso Salvitti]

If I did it correctly, I get this output

mixed sco i.rid ib6.itm || sid:, nolog  Mixed-effects ML regression                         Number of obs    =     210 Group variable: sid                                 Number of groups =      19                                                     Obs per group:                                                                  min =       6                                                                  avg =    11.1                                                                  max =      12                                                     Wald chi2(6)     = 9583.63 Log likelihood = -445.17174                         Prob > chi2      =  0.0000  ------------------------------------------------------------------------------          sco | Coefficient  Std. err.      z    P>|z|     [95% conf. interval] -------------+----------------------------------------------------------------        1.rid |   .6289804    .268051     2.35   0.019       .10361    1.154351              |          itm |           1  |      -35.4   .4499856   -78.67   0.000    -36.28196   -34.51804           2  |  -35.68571   .4499856   -79.30   0.000    -36.56767   -34.80376           3  |  -34.57143   .4499856   -76.83   0.000    -35.45338   -33.68947           4  |  -21.42857   .4499856   -47.62   0.000    -22.31053   -20.54662           5  |  -20.65714   .4499856   -45.91   0.000     -21.5391   -19.77519              |        _cons |   37.09132   .4311816    86.02   0.000     36.24622    37.93642 ------------------------------------------------------------------------------  ------------------------------------------------------------------------------   Random-effects parameters  |   Estimate   Std. err.     [95% conf. interval] -----------------------------+------------------------------------------------ sid: Identity                |                   var(_cons) |   1.154822   .4828033      .5089185    2.620487 -----------------------------+------------------------------------------------                var(Residual) |   3.543524    .362476      2.899772     4.33019 ------------------------------------------------------------------------------ LR test vs. linear model: chibar2(01) = 30.72         Prob >= chibar2 = 0.0000  .  . estat icc  Residual intraclass correlation  ------------------------------------------------------------------------------                        Level |        ICC   Std. err.     [95% conf. interval] -----------------------------+------------------------------------------------                          sid |   .2457933   .0810449      .1215277    .4343049 ------------------------------------------------------------------------------

so My icc is 0.24 or this value is the residual intraclass correlation?
Sorry if the question is stupid but I've recently approached this analysis thanks to everybody in advance

[Tommaso Salvitti]

If I did it correctly, I get this output

mixed sco i.rid ib6.itm || sid:, nolog

Mixed-effects ML regression                         Number of obs    =     210
Group variable: sid                                 Number of groups =      19
                                                    Obs per group:
                                                                 min =       6
                                                                 avg =    11.1
                                                                 max =      12
                                                    Wald chi2(6)     = 9583.63
Log likelihood = -445.17174                         Prob > chi2      =  0.0000

------------------------------------------------------------------------------
         sco | Coefficient  Std. err.      z    P>|z|     [95% conf. interval]
-------------+----------------------------------------------------------------
       1.rid |   .6289804    .268051     2.35   0.019       .10361    1.154351
             |
         itm |
          1  |      -35.4   .4499856   -78.67   0.000    -36.28196   -34.51804
          2  |  -35.68571   .4499856   -79.30   0.000    -36.56767   -34.80376
          3  |  -34.57143   .4499856   -76.83   0.000    -35.45338   -33.68947
          4  |  -21.42857   .4499856   -47.62   0.000    -22.31053   -20.54662
          5  |  -20.65714   .4499856   -45.91   0.000     -21.5391   -19.77519
             |
       _cons |   37.09132   .4311816    86.02   0.000     36.24622    37.93642
------------------------------------------------------------------------------

------------------------------------------------------------------------------
  Random-effects parameters  |   Estimate   Std. err.     [95% conf. interval]
-----------------------------+------------------------------------------------
sid: Identity                |
                  var(_cons) |   1.154822   .4828033      .5089185    2.620487
-----------------------------+------------------------------------------------
               var(Residual) |   3.543524    .362476      2.899772     4.33019
------------------------------------------------------------------------------
LR test vs. linear model: chibar2(01) = 30.72         Prob >= chibar2 = 0.0000

. 
. estat icc

Residual intraclass correlation

------------------------------------------------------------------------------
                       Level |        ICC   Std. err.     [95% conf. interval]
-----------------------------+------------------------------------------------
                         sid |   .2457933   .0810449      .1215277    .4343049
------------------------------------------------------------------------------

[Tommaso Salvitti]

so My icc is 0.24 or this value is the residual intraclass correlation?
Sorry if the question is stupid but I've recently approached this analysis thanks to everybody in advance

[Tommaso Salvitti]

Joseph Coveney
Did I understand your suggestions correctly? thnaks a lot

[Joseph Coveney]

Did I understand your suggestions correctly?

No.

I don't know what these scores represent, but by inspection, their distributions are disparate, for example, B takes on only two values; A only three values and C only four. Only with the last three scores (D-F) could you make some kind of case that they're continuous, and even then their sample variances range from 3½ to over 20.

Moreover, I don't know whether this is just happenstance peculiar to this sample or built into the rating system, but scores A and B are collinear; likewise, score F and some subset of the other scores are collinear.

If you "have to calculate inter rater reliability", then my suggestion is to go back to whoever gave you this task for guidance on how to combine these scores into a single summary score that will be less discrete. I wouldn't worry too much about whether the summary score would satisfy psychometric unidimensionality criteria (you have only four linearly independent scores, anyway). For the task at hand, I would consider more important that it be amenable to fitting by a linear mixed model.

[Tommaso Salvitti]

Sorry Joseph Coveney

* Example generated by -dataex-. To install: ssc install dataex
clear
input byte(Soggetti ITEM1A ITEM1B ITEM2A ITEM2B ITEM3A ITEM3B ITEM4A ITEM4B ITEM5A ITEM5B)
 1 . 2 . 4  . 17  . 17  . 40
 2 1 2 1 2 15 16 16 16 33 36
 3 2 2 3 4 14 17 16 18 36 41
 4 2 2 2 4 13 18 18 18 35 42
 5 2 2 3 4 15 17 17 18 37 41
 6 2 2 2 4 17 18 18 18 39 42
 7 1 1 1 2 15 12 15 12 32 27
 8 . 2 . 4  . 17  . 16  . 39
 9 2 2 3 4 16 17 18 18 39 41
10 2 2 3 4 15 18 16 18 36 42
11 2 1 2 4 14 15 18 17 36 37
12 2 1 1 2 14  9 16 10 33 22
13 2 2 3 4 18 18 18 18 41 42
14 1 1 1 3 18 17 16 18 34 39
15 2 2 2 4 18 18 18 18 40 42
16 2 2 2 4 16 17 17 18 37 41
17 2 1 1 3 17 18 18 18 38 40
18 . 2 . 3  . 14  . 13  . 32
19 2 2 4 4 16 17 16 18 38 41
end

this is the correct database.
I'll try to explain the variables. the variable item5 is the sum of the variables item1, item2, item3 and item4. The item1 scores represent hand movements and can take values of 1 and 2; item2 scores represent head movements and can take values of 1,2,3 and 4; item 3 scores represent upper limb movements and can take values from 0 to 18; item 4 scores represent the movements of the lower limbs and can take values from 0 to 18. Item 5 scores generally represent how the child moves and can have a maximum value of 42.
From these premises how is the inter rater reliability?

[Tommaso Salvitti]

Is anybody can help me?

[Stacey Green]

I am just learning about ICR as well.

1. There are a number of ways to calculate the inter coder reliability
2. Hruschka et al. argue for the use of "Cohen's Kappa." Perhaps you can search for how to calculate that value.
3. The kappa range runs from -1 to 1. Therefore the authors propose these criteria,

0.811.00 = almost perfect;
0.61–0.80 = substantial;
0.41–0.60 = moderate;
0.210.40 = fair; 0.00–0.20 = slight;
and < 0.00 = poor.

These values might help interpret your Stata output.

This paper has been a tremendous help to me: Hruschka, Daniel J., et al. "Reliability in coding open-ended data: Lessons learned from HIV behavioral research." Field methods 16.3 (2004): 307-331.
Direct link to paper: https://journals.sagepub.com/doi/pdf...fDM8BktyeRLtay

Good luck!

[Tommaso Salvitti]

thanks a lot.
But i refer #10 Joseph Coveney
Is the problem clearer now? thanks a lot

[Tommaso Salvitti]

Dear everyone, I ask for advice and an opinion on how I am proceeding since I cannot consult anyone. Yesterday Joseph Coveney suggested me to combine these scores into a single summary score. So i consider only the variable item5 (the variable item5 is the sum of the variables item1, item2, item3 and item4; see#10) and i've done in this way:

 * Example generated by -dataex-. For more info, type help dataex clear input byte(Soggetti item1a item1b)  1  . 40  2 33 36  3 36 41  4 35 42  5 37 41  6 39 42  7 32 27  8  . 39  9 39 41 10 36 42 11 36 37 12 33 22 13 41 42 14 34 39 15 40 42 16 37 41 17 38 40 18  . 32 19 38 41 end

rename *, lower rename Soggetti sid quietly reshape long item, i(sid) j(tmp) string rename item sco gen byte rid=substr(tmp, -1, 1)=="b" gen byte itm=real(substr(tmp,1, strlen(tmp)-1)) mixed sco i.rid ib1.itm || sid:, nolog estat icc

rename *, lower  rename Soggetti sid quietly reshape long item, i(sid) j(tmp) string  rename item sco  gen byte rid=substr(tmp, -1, 1)=="b" gen byte itm=real(substr(tmp,1, strlen(tmp)-1))  mixed sco i.rid ib1.itm || sid:, nolog  estat icc

am I doing well? Thanks to everybody in advance

[Tommaso Salvitti]

Dear everyone, I ask for advice and an opinion on how I am proceeding since I cannot consult anyone. Yesterday Joseph Coveney suggested me to combine these scores into a single summary score. So i consider only the variable item5 (the variable item5 is the sum of the variables item1, item2, item3 and item4; see#10) and i've done in this way:

* Example generated by -dataex-. For more info, type help dataex
clear
input byte(Soggetti ITEM5A ITEM5B)
 1  . 40
 2 33 36
 3 36 41
 4 35 42
 5 37 41
 6 39 42
 7 32 27
 8  . 39
 9 39 41
10 36 42
11 36 37
12 33 22
13 41 42
14 34 39
15 40 42
16 37 41
17 38 40
18  . 32
19 38 41
end

rename *, lower

rename Soggetti sid
quietly reshape long item, i(sid) j(tmp) string

rename item sco
 gen byte rid=substr(tmp, -1, 1)=="b"
gen byte itm=real(substr(tmp,1, strlen(tmp)-1))

mixed sco i.rid ib1.itm || sid:, nolog

estat icc

am I doing well? Thanks to everybody in advance