Differences between DD and DDD results

[Kanika Phogat]

Hello everyone,

I'm analyzing the impact of a conditional cash transfer program launched in 2013 in India on adolescent girls' secondary school enrollment and learning outcomes (math and reading tests). The analysis uses repeated cross-sectional data from 2008 to 2022, with a few missing years, comparing one treated state to three neighboring control states.

I’m conducting two types of analyses: a simple Difference-in-Differences (DD) and a Difference-in-Difference-in-Differences (DDD) approach. I’ve observed some differences in the results that I need help reconciling.

DD Analysis:

• Treated Group: 13-16-year-old girls in the treated state
• Control Group: 13-16-year-old girls in control states
• Pre-Period: Before 2013
• Post-Period: After 2013
• The parallel trends assumption holds for this specification.

Results:

• Enrollment: Positive and higher coefficients, but non-significant.
• Learning Outcomes (Math and Reading): Negative coefficients, both non-significant.

DDD Analysis:

• Additional Control Layer: 13-16-year-old boys across treated and control states
• Pre-Period: Before 2013
• Post-Period: After 2013
• The parallel trends assumption holds for this specification as well.

Results:

• Enrollment: Smaller coefficients, but significant.
• Learning Outcomes (Math and Reading): Positive coefficients, with only the math coefficient being significant.

Placebo Test:


I also ran DD and DDD regressions using 13-16-year-old boys as the placebo-treated group.

Results:

• DD: Positive for enrollment, negative for learning outcomes, but no significant results.
• DDD: Small, positive, and significant results for all outcomes when boys are the treated group.

The key difference I'm trying to reconcile is that the DD specification shows a negative but insignificant effect on learning outcomes for 13-16-year-old treated girls, while the DDD specification shows a very small but positive and significant effect for the same group.

Questions:

1. How should I reconcile the differences observed between the DD and DDD results?
2. Could the differences be attributed to differential trends for these outcomes across boys in treated vs. control states, or does this suggest that boys may not be a suitable control group to begin with?
3. Given that, theoretically, the intervention shouldn't affect boys, how should I interpret the significant results from the DDD analysis when boys are used as the placebo-treated group?
4. Should I identify which set of results (DD vs. DDD) provides the preferred estimates, or is it more appropriate to present both sets and discuss potential reasons for the differences?

Your insights and suggestions on how to approach and interpret these differences in results would be greatly appreciated. Please let me know if I’ve made any mistakes in explaining my problem or if you need additional information or code.

Thank you in advance!

Best,
Kanika

[Kanika Phogat]

Further information about the DD and DDD estimation equations:

The DD estimation equation is as follows:

𝑦_𝑖𝑠𝑡 = 𝛽₀ + 𝛽₁. 𝑝𝑜𝑠𝑡_𝑡 + 𝛽₂. WB_𝑠 + 𝛽₃. (𝑝𝑜𝑠𝑡_𝑡 × WB_𝑠) + 𝜗𝑋_𝑖𝑠𝑡 + 𝜕_s + 𝛾_𝑡 + (𝜕_𝑠 × 𝑡) + 𝜖_𝑖𝑠𝑡

The outcome variable 𝑦_𝑖𝑠𝑡 represents either enrollment (binary: 1 if enrolled in any educational institution, 0 otherwise) or learning outcomes (scored 0 to 4, where 4 is the highest). The key variable WB_𝑠 indicates if the child is from the treated state, and 𝑝𝑜𝑠𝑡_𝑡 marks post-program years (after 2013). The coefficient β3 estimates the intent-to-treat effect on 13-16-year-old girls in the treated state, controlling for child, household, and village characteristics, along with year (𝛾_𝑡) and state (𝜕_s) fixed effects and state-specific linear trends (𝜕_𝑠 × 𝑡). The regressions are weighted for state-level representativeness.

The DDD estimation equation is as follows:

𝑦_𝑖𝑠𝑡 = 𝛽₀ + 𝛽_1. WB_𝑠 + 𝛽₂. 𝑝𝑜𝑠𝑡_𝑡 + 𝛽₃. Female_i + 𝛽₄ (𝑝𝑜𝑠𝑡_𝑡 × Female_i) + 𝛽₅ (𝑝𝑜𝑠𝑡_𝑡 × WB_𝑠) + 𝛽₆ (WB_𝑠 × Female_i) + 𝛽₇ (WB_𝑠 × 𝑝𝑜𝑠𝑡_𝑡 × Female_i) +𝜗𝑋_𝑖𝑠𝑡 + 𝜕_d + 𝛾_𝑡 + (𝜕_d × 𝑡) + (θ_i × 𝑡) + 𝜖_𝑖𝑠𝑡

In the DDD model, (θ_i × 𝑡) represents the gender-specific linear trend, added alongside state-specific trends to account for differences in enrollment and learning trends between males and females over time. This controls for potential confounding factors, such as other simultaneous policy interventions that may differentially impact these outcomes for girls vs boys. The key coefficient of interest, 𝛽₇ estimates the intent-to-treat effect.

Please let me know if any further information is required!

Thanks in advance!

[Andreas Backhaus]

Dear Kanika,

In your DDD regression, is beta_4 statistically different from beta_7?

[Kanika Phogat]

Hello Andreas,

I ran the 'test' command to check the null hypothesis that beta_4 equals beta_7. The p-value from the test is 0.2861, indicating that these coefficients are not statistically different from each other.

[Andreas Backhaus]

Hi Kanika,

My intuition is the following:

The Triple Diff is the differences of two Difference-in-Differences - in your case, (1) the difference between girls in the treated and non-treated states, and (2) the difference between girls and boys in the treated states.
In your DD analysis, you have computed (1) using girls in the non-treated states as the control group. Note you could also compute (2) by using boys in the treated states as the control group. I would expect (2) to be significant given that the DDD is significant and we already know that (1) is insignificant.
In this sense, there is no inconsistency in your results, but they are telling you that the treatment seems to make a difference only between girls and boys, not between girls despite some of the girls not being treated. One potential explanation could be spillovers from the treatment to girls across state boundaries, which would require you to investigate this possibility with institutional knowledge etc. A different, more problematic explanation could be that the treatment actually does nothing and there is a downward trend for boys that your regression does not capture, which again would prompt further investigation.

[Kanika Phogat]

Dear Andreas,

Thank you so much for your insightful response!

I’ve actually added a DD specification where boys from the treated state are used as the control group, and as you suggested, I do get significant results. However, I decided not to include these in the main results because the parallel trends assumption didn’t hold for this specification.

Regarding the possible explanations for my findings, you might be right about the control states potentially not being true controls. While there may not be direct spillovers from the treated state, it’s possible that similar policies in the control states are affecting the outcomes. Almost every state in India has some form of policy aimed at improving education and health outcomes of girls, even if they’re not as widely documented online. So, it’s possible that these states might have simultaneous policies impacting the same outcomes, even if they aren’t directly related to the CCT. Do you think this may be a reason for the results I am seeing? However, please note that I also conducted a robustness check by using the Synthetic Control Method where I tried to restrict the donor pool to states that do not have or have ever had a policy like the I am looking at and found that the results were in a similar direction to those from DD Specification 1 (comparing treated girls vs. control girls).

As for the second, more problematic explanation, I did check for trends while testing the parallel trends assumption. Interestingly, I didn’t observe any clear downward trends for boys. Enrollment was generally increasing, and learning outcomes were mostly stagnated with some minor fluctuations. Do you think this observation is enough to rule out the second issue you mentioned?

Looking forward to your thoughts!

Thank you again.

Best,
Kanika

[Andreas Backhaus]

Dear Kanika,

If the parallel trends assumption is violated in the DD using boys in the treated states as controls, note that this violation may but does not have to carry over to the DDD. See this very informative paper in The Econometrics Journal: https://academic.oup.com/ectj/article/25/3/531/6545797
May I ask what kind of violation of the parallel trends assumption you have detected? Meaning who trended the wrong way before treatment?
As to the less technical questions regarding whether readers may find your explanations sufficient - there I clearly know too little about your audience and the specifics of your setting. But showing in your writing that you do have this knowledge will certainly help your case.

Best,
Andreas

[Kanika Phogat]

Dear Andreas,

Thank you for your follow-up.

Visually, when plotting the group means over time, it’s difficult to definitively say whether the parallel trends assumption holds for DD Specification 2 (comparing girls in the treated state to boys in the treated state). However, the trends do sometimes converge or diverge slightly in other specifications as well. Interestingly, when I plotted the means for enrollment of treated girls vs. control girls (DD Specification 1), there appears to be a parallel trends violation as the trends intersect once during the pre-period, but nothing like this happens in DD Specification 2.

Nevertheless, I conducted regression analysis with individual year dummies and performed a joint F test on the interaction coefficients. The results were insignificant for all outcomes in DD Specification 1 and the DDD Specification 3, which supports the assumption of parallel trends in these cases. However, for DD Specification 2 (comparing girls in the treated state to boys in the treated state), the parallel trends assumption did not hold for learning outcomes, which is why I decided not to include this specification in the main analysis.

Looking forward to your thoughts!

Best,
Kanika