Interpreting Interaction Effect over Time

[Robert Niewoehner]

Hello everyone, I hope you all are having a wonderful week (and weekend)!

I have a quick question regarding the interpretation of an interaction effect, and I want to make sure I'm interpreting the effect correctly. See the following output of an 'xtreg' Fixed-Effect regression.

I'm ultimately trying to determine the treatment effect. I have 2 years of data (2017 and 2018) and two different treatments in addition to a control group.
- The "Factor_Arm" variable is set to a 1 (Control), 2 (Treatment 1), or 3 (Treatment 2)

I'm getting turned around a bit. My primary variables of interest are the coefficients on the interaction terms at the bottom, but I'm wondering if this is actually the treatment effect.

I'm getting turned around trying to interpret this. I know that with the interaction, what I'm seeing is in Factor_Arm#Year is how the coefficient on i.Year will change in 2018 for each of the treated groups.

So in that instance, what I have below is indeed the treatment effect.
But then what is the p-value comparing? Is this saying (as I think) that even after we control for how the control group changed in 2018, the change in Treated Group 1 is statistically different from 0 (p = 0.05), ie, the "Treatment Effect" is statistically different from zero?

Any/all help greatly appreciated. Thank you!

[Clyde Schechter]

I would like to be able to explain your results in terms of the specific numbers in your output, but the color scheme renders on my screen in such a way that I cannot read any of the numbers: the yellow framework is easily read, but the numbers are just indistinguishable from the black background. So I'll try to do it in the abstract.

The coefficient for 2018.Year is the estimated difference between ShotPerPx in 2018 and ShotPerPx in 2017 when Factor_Arm = 1 (i.e., in the control group.) The corresponding differences for Factor_Arm = 2 or 3 are not directly shown in the output (though they could be calculated).

The coefficient for 2.Factor_Arm#2018.Year is the amount that you have to add to the coefficient of 2018.Year in order to come up with the estimated difference between ShotPerPx in 2018 and ShotPerPx in 2017 when Factor_Arm = 2 (i.e. in the 1st treatment group). That is why these analyses are called "difference in differences" analyses. The interaction coefficients are the difference between groups of the between-time period differences.

Now the reasoning goes like this: since Factor_Arm = 1 is the control group, we imagine that had the treatment group(s) not been treated, their change from 2017 to 2018 would have been the same as what the control group experienced. That is the expected value of ShotPerPx would have changed by the same amount (plus some noise) as the change observed between 2017 and 2018 in the control group. So if we take the actual change in ShotPerPx in the treatment group(s) and subtract the change observed in the control group, that amount is, arguably, attributable to the treatment. (Evidently there is the necessary assumption that it isn't attributable to something other than the treatment that coincidentally happened at the same time!) So the coefficient of 2.Fasctor_Arm#2018.Year is interpreted as the effect of the treatment when Factor_Arm = 2 (i.e. in the 1st treatment group).

The situation for the second treatment group (Factor_Arm = 3) is completely analogous to that of the first treatment group.

[Robert Niewoehner]

Clyde: you are absolutely right - I had second thoughts about changing my color back to a default before uploading that. Thank you for the gentle tip. I cannot edit my figure, but I will re-upload here for future reference.




Couple of comments:
1. The reason Factor_Arm = 2 or 3 are not shown is this is a FE regression, and these values are time invariant. Thus, they are differenced-out in a FE regression, but could be estimated via a Pooled OLS or Random Effects regression (I assume this is how you imply "they could be calculated").
2. Thank you for the clear, concise explanation of the treatment effect. I thought I was on the right track, but sometimes when you stare at these things for 2 hours you start to see things that are not there.

Quick Follow-Up:
If I estimated this regression via a Pooled OLS (or a Random Effects model), the output would include an estimate for Factor_Arm = 2 and Factor_Arm = 3. See attached image.

Q: These values would represent the different intercepts for each of the treatment groups in 2017. Is that correct?

Thus, the interaction would continue to capture the "difference in the differences" in 2018, but we would also be able to see the difference between the groups in the first year. Please correct me if I'm wrong.

Thank you for the help!

[Clyde Schechter]

If I estimated this regression via a Pooled OLS (or a Random Effects model), the output would include an estimate for Factor_Arm = 2 and Factor_Arm = 3. See attached image.

Q: These values would represent the different intercepts for each of the treatment groups in 2017. Is that correct?

Yes, sort of. The "sort of" is because you have to consider whether the use of an OLS or RE estimator is appropriate here. An OLS estimator would completely eliminate any representation of effects at the ClinicID level. Unless those effects are negligible (you don't show the sigma_u and rho outputs from the random effects model, so I can't tell if they are or not) this would be an important mis-specification of the model, with effects on both the coefficient estimates and the standard errors. The random effects model would be less egregious. But it is important to remember that the random effects estimator only produces unbiased estimates if the error terms are truly independent of the predictors. In particular, the RE estimator is vulnerable to confounding by omitted variables. With a randomized trial this is, of course, basically not a problem, but in observational data it can be a serious problem.

So, if the RE estimator is appropriate to your situation, your interpretation of those values is correct. But you need to think about the appropriateness of the RE estimator here. It's not automatically wrong, but it's not automatically right either.

[Robert Niewoehner]

Thank you for the robust clarification. In this particular case, I do actually have a randomized trial, improving my confidence in using the RE estimator. I will probably still make some effort to demonstrate the randomization was truly successful, but I'm at least starting from the right spot.

The corresponding differences for Factor_Arm = 2 or 3 are not directly shown in the output (though they could be calculated).

Did you have an alternate recommendation for how to calculate these values?

[Clyde Schechter]

[quote]

Did you have an alternate recommendation for how to calculate these values?

[quote]

lincom 2018.year + 2.Factor_Arm#2018.year
lincom 2018.year + 3.Factor_Arm#2018.year

[Robert Niewoehner]

Ah, misunderstood. Yes I see that. Thank you, Clyde! Really appreciate the time and thoughtful feedback.