Assuming ICC

[Anika Zaman]

Hello everyone, This is my first post here. I need icc for power calculation. I have outcome variables test scores and student's effort to study .Can you hint me any source to assume icc ? I have been told to look at similar study so I was checking data of a similar study. I am trying to find out the intraclass correlation between testscores and then student's effort separately, not between them. I have the test score, student ids and they are clustered by village id.I also have 2 treatment groups and 1 control. I have tried the stata command but I can't understand who is my rater/target? Not sure whether to pick one way or two way.It is a Randomised Controlled Trial model.for power calculation I should be using : power twomeans 0, cluster k1(75) k2(75) m1(15) m2(15) power(.8 .9) rho() I need to assume icc/rho.Thanks.

[Clyde Schechter]

Yes, this is a dilemma that is often faced when designing a trial with multi-level data. I can tell you from long and painful experience with this that, despite doing your best, you likely will end up with a useless power analysis. Going to the literature to find studies using similar variables is, frankly, a fool's errand. First, very few studies actually publish the ICC they achieved. Second, even if you find one that did, unless they are studying the same outcome and explanatory variables you are and model it the same way, the ICC they found may have little or nothing to do with the ICC you will attain in your study. Finally, my experience is that even if you can find a published study that is an exact match, ICC's generally seem to vary wildly in replications of the same study, even when based on large samples. Consequently, I would never take seriously any power/sample-size/effect-size analysis based on a single assumed value of ICC. I think you need to assume a range of values that seems plausible (in light of whatever past studies you can find plus any other information you might know about the constructs involved) and present multiple corresponding power/sample-size/effect-size analyses. So you can end up saying things like "our expected case scenario will provide us with a minimum detectable effect size of xxx, based on an ICC of yyy. Under unfavorable circumstances, with ICC = zzz, the minimum detectable effect size will be www; and under an optimistic scenario with ICC = vvv, the minimum detectable effect size decreases to uuu."

Probably for your expected case scenario you can do something with the data you currently have.* For example:

mixed test_score i.treatment_group || village_id:
estat icc

and use that result as your base case. Then allow a generous margin around that for optimistic and pessimistic scenarios. Similarly for the effort measurement.

*Note: I'm assuming here that the data you are working with is some kind of preliminary data, not the actual complete study data. If the data you have is the complete data from your study, it is generally understood that it is not appropriate to do a post-hoc power analysis based on that data.

[Anika Zaman]

Thanks a lot for your kind help. I have noted your suggestions.