kappaetc won't calculate confidence intervals

[Julia Dabravolskaj]

Hi,
I have a dataset with 23 raters (ie vars ID1 - ID23) who rated 7 statements (var 'statement') on the scale from 1 (strongly disagree) to 5 (strongly agree). The same 7 statements were assessed at two other time points (ie Round 2 and Round 3 of the Delphi survey).

First, I ran this command (see below) to calculate Gwet's AC1 for all 7 statements separately in Round 1, 2, and 3. As expected, the degree of agreement is increasing in every following round AND the output contains 95% CI:
kappaetc ID1 - ID23

However, I'd also like to look at Gwet's AC for each of the statements separately, but while I do get Gwet's AC1 there are dots/empty spaces instead of CIs in these outputs:
kappaetc ID1 - ID23 if statement==1

I'm pretty sure it has to do with the fact that when trying to calculate Gwet's AC1 for each statement I have only 1 'subject' that is being rated by 23 raters, but I see papers that are designed in the same way and they do provide CIs for each item/statement. Any insight? Any help would be very much appreciated.
Kindly,
Julia

[daniel klein]

kappaetc is either from SSC or SJ (18-4).

I have discussed conceptual and technical concerns regarding agreement coefficients for a single subject. In your situation, you have (many) more raters than rating categories, meaning that the lower bound for agreement coefficients is always positive (see this post). I express a more fundamentally theoretical concern in this post. Finally, I summarize the problem with agreement coefficients that are based on the observed rating frequencies here. Gwet's AC is such a coefficient.

Technically speaking, you should specify the categories() option with kappaetc because Gwet's AC uses the number of rating categories and it is unlikely that you observe all rating categories for single subjects. The categories() option at least ensures the correct mathematical answer in terms of the agreement coefficient.

I assume you could mathematically get a standard error and confidence intervals from the so-called analytic approach as a function of the observed agreement and expected agreement. I also believe that those standard errors and CI generally do not have a straightforward interpretation and are meaningless in the case of a single subject. I guess that Gwet has similar feelings towards the analytic approach to standard errors as he did not even try to derive one for his AC.

Given more than two raters, you can get standard errors and confidence intervals conditional on the subject. Specify the option se(conditional subject).

Overall, my advice still is: do not do that; the results are probably misleading.

[Julia Dabravolskaj]

daniel klein, this is incredibly helpful! Thank you so much for this info. I can get 95% CIs when I add se(conditional subject) to the command. However, I completely agree that the CI estimates might be quite misleading (which I will make sure to relay to my team).

Another quick question... Kappaetc does not run if the ratings do not vary, but is there a way to force Stata to 'skip' the statements for which there is no variation and produce the output for all subsequent statements? That's what I'm getting:

-> Round1 = 1, number = 39
ratings do not vary
r(459);

end of do-file

Thanks!
Julia

[daniel klein]

The by prefix has a rc0 option, see

help by

capture might also be relevant, see

help capture