Power calculation for a multi-arm experiment

[danishussalam]

Hello everyone,

I'm facing difficulty finding resources to estimate the Minimum Detectable Effect (MDE) in STATA for an online experiment I'm conducting. I know others have asked somewhat similar questions before but I still haven't been able to find a concrete answer.

My total sample size is 3000, evenly divided among 6 arms. Here's the experiment setup:

1. Out of the 3000, we initially randomize participants into two groups of 1500 each. Group 1 plays a revealed preference game before answering some questions, while Group 2 plays the game after answering the questions. This order of the game serves as a treatment in itself.
2. Within each group (Group 1 and Group 2), we randomly select three groups of 500 (T1, T2, C1). Here, we vary the way we ask questions to gauge stated preferences.

In total, I'll have 6 groups, each with 500 participants. The primary outcome is the difference between revealed and stated preferences (in %). Based on past studies, the mean for similar outcomes is 60% with a standard deviation of 15%. My interest lies in both: a) comparing treatment groups (T1 and T2) with C1, and b) comparing treatments within the same initial experimental category (played game before or after).

Could someone please guide me on the best approach to calculate the MDE with a power of 0.8 and a significance level of 0.05? I'm unsure whether I should treat this as a 6-arm experiment or a multi-layered design. Any assistance on this would be greatly appreciated.

[Luca Calianno]

Hi,
I will suggest some useful links that helped me a lot.

• https://blog.stata.com/2019/01/10/ca...-1-the-basics/ (READ all blogs, should be 4)
• https://nickch-k.github.io/Econometr...ons_Stata.html

I believe the best approach is to to create a dataset with the characteristics you are after, do your statistical analyses on that dataset and store estimates. Repeat the process changing the value of the treatment effect. (If you want, repeat the process many times to generate some measure of uncertainty.)