Interpretation of Subsample analysis for Difference in Differences Analysis

[Pranshu Tripathi]

Hello All
I am working on a legal policy that has been enacted in 2014. For this purpose, I created the treated and control groups. Now, the issue is that while finding the impact of the policy on the groups individually, the results show that this policy has a significant impact on both groups. However, when I do a DiD analysis, the results are insignificant, showing that the difference between the treatment effects on both groups is insignificant.

How should I interpret it?

Can we say that the individual effect is there, but the parallel trend continues even after the treatment, and the treatment effect is similar in both groups?

Is it right to do so?

[Clyde Schechter]

Can we say that the individual effect is there, but the parallel trend continues even after the treatment, and the treatment effect is similar in both groups?

Well, because you observed similar differences in both groups, you can't really call it a "treatment" effect. Something happened in 2014, but it affected both groups to the same extent. So it represents some kind of secular trend, not a treatment effect.

Now, it is a widespread, but entirely incorrect, practice to interpret the lack of statistical significance of an effect estimate as meaning "no effect." It does not mean that. It means that the data are compatible with no effect. But if you look at the confidence interval around your estimate, that will give you a whole range of possible effect sizes that the data are compatible with. And some of them might be large enough to be of practical importance, should they turn out to be the correct values. In that case it would be best to characterize the study as inconclusive with regard to the treatment effect.

Or, if you have very sharp data and a large enough sample size, you may have a confidence interval that is pretty narrow, so that even if the largest compatible effect size in the confidence interval were the correct value, it would be too small to be of any practical importance anyway. In that case it would be fair to say that the study excludes any meaningful effect.

[Pranshu Tripathi]

Thanks a lot, Clyde.

Regards