[Joseph Coveney]

Sorry . . . this is the correct database. . . . variable item5 is the sum of the variables . . . From these premises how is the inter rater reliability?

Okay, so the convention is that the sum of the first four scores can be taken for the summary score that I twice requested (in #2 and #9) above? If that's the convention, then that's what you would use in estimating the intraclass correlation coefficient for the scale.

A couple of comments:

1. You still have anomalies in your dataset: check the sumscores (item5) for child IDs 3 and 14 for judge ID 1.

2. Although the simple sumscore,seems to be the convention and I suppose reflects intention by the experts to weight the individual scores unequally, if you wanted to weight the individual scores equally, then you could for example rescale each score from zero to one before summing them. Something like the following.

version 18.0

clear *

quietly input byte(Soggetti ITEM1A ITEM1B ITEM2A ITEM2B ITEM3A ITEM3B ITEM4A ///
    ITEM4B ITEM5A ITEM5B)
 1 . 2 . 4  . 17  . 17  . 40
 2 1 2 1 2 15 16 16 16 33 36
 3 2 2 3 4 14 17 16 18 36 41
 4 2 2 2 4 13 18 18 18 35 42
 5 2 2 3 4 15 17 17 18 37 41
 6 2 2 2 4 17 18 18 18 39 42
 7 1 1 1 2 15 12 15 12 32 27
 8 . 2 . 4  . 17  . 16  . 39
 9 2 2 3 4 16 17 18 18 39 41
10 2 2 3 4 15 18 16 18 36 42
11 2 1 2 4 14 15 18 17 36 37
12 2 1 1 2 14  9 16 10 33 22
13 2 2 3 4 18 18 18 18 41 42
14 1 1 1 3 18 17 16 18 34 39
15 2 2 2 4 18 18 18 18 40 42
16 2 2 2 4 16 17 17 18 37 41
17 2 1 1 3 17 18 18 18 38 40
18 . 2 . 3  . 14  . 13  . 32
19 2 2 4 4 16 17 16 18 38 41
end

rename (ITEM?A ITEM?B) (sco?1 sco?2)
rename Soggetti cid

forvalues i = 1/5 {
    local varlist `varlist' sco`i'
}
quietly reshape long `varlist', i(cid) j(rid)

*
* Begin here
*
recast double sco?
forvalues i = 1/4 {
    summarize sco`i', meanonly
    quietly replace sco`i' = (sco`i' - r(min)) / (r(max) - r(min))
}
egen double sum = rowtotal(sco1-sco4)
mvdecode sum, mv(0)

mixed sum i.rid || cid: , reml dfmethod(kroger) nolog
estat icc

exit

[Tommaso Salvitti]

it's very clear Joseph Coveney Thank you so much!!!
i have to ask you another question.
Could it be a validated solution to evaluate the reliability of the evaluator for each variable separately? That's what they told me to do. is it possible? and in this case the instruction is valid in #15?

[Tommaso Salvitti]

Dear Joseph Coveney my output in stata, following your code is this:

estat icc

Residual intraclass correlation

------------------------------------------------------------------------------
Level | ICC Std. err. [95% conf. interval]
-----------------------------+------------------------------------------------
sid | .3458195 .2159164 .0752929 .7743709
------------------------------------------------------------------------------

so THE INTER RATER RELIABILITY BETWEEN THE TWO JUDGE IS 34,5% EVEN IF IN THE OUTPUT IS WRITTEN "LEVEL sid"

Sorry for the question but I'm approaching these things for the first time

[Tommaso Salvitti]

is anybody can help me??@Joseph Coveney

[Joseph Coveney]

my output in stata, following your code is this: . . .

I don't know what you did to get that value (0.35), but following the code that I show above in #16, I get an ICC coefficient of 0.62 ± 0. 15; I suppose that the difference can be chalked up to the Spearman-Brown prophecy.

so THE INTER RATER RELIABILITY BETWEEN THE TWO JUDGE IS 34,5% EVEN IF IN THE OUTPUT IS WRITTEN "LEVEL sid"

The child is the higher level in the hierarchical regression model; the two raters' scores are what contribute the residual variance. So child ID (cid) is properly shown under Level in the output table.

[Tommaso Salvitti]

it's very clear Joseph Coveney Thank you so much!!!
i have to ask you another question.
Could it be a validated solution to evaluate the reliability of the evaluator for each variable separately (ITEM1, ITEM2, ITEM3, ITEM4)? That's what they told me to do. is it possible? and in this case the instruction is valid in #15?

rename *, lower
rename Soggetti sid
quietly reshape long item, i(sid) j(tmp) string


rename item sco
gen byte rid=substr(tmp,1, strlen(tmp)-1))
mixed sco i.rid || sid:, reml dfmethod(kroger) nolog
estat icc

Can i repeat this instructions for each variable?

What do you think Joseph Coveney?

thanks also in advance to everybody

[Tommaso Salvitti]

Joseph Coveney
I honestly don't understand your answer. To the question, is the intraclass correlation coefficient between the two judges 0.62 ?(following the code #16). i don't understand "The child is the higher level in the hierarchical regression model; the two raters' scores are what contribute the residual variance. So child ID (cid) is properly shown under Level in the output table".

[Tommaso Salvitti]

thanks in advance to Joseph Coveney and to everybody