Robustness Checks for Triple DiD model

[Taiba Chau]

Hi, I was wondering if I have a dataset for individuals in many countries. I am trying to analyse the implication of the policy in those countries. I am running a triple DiD to estimate the implication of the policy on the mental health of individuals after the policy was introduced for the selected countries.

For this I know I don't need to rely on the parallel trend assumption as I only have two time periods. However is there any way of doing a robustness check? Or is there a need to do a robustness check?
Thank you

[Jared Greathouse]

With only two time periods, it'll be hard to apply design based or sampling based robustness checks. However, it is typically possible to apply model/estimator based robustness checks. Such as, estimating a linear probability model after doing a logit or probit analysis, or similar changes to the functional form that may change the results.

Note that yes, you do have to rely on parallel trends holding. There's just no way to test for it in this setting, unless you're going to combine your DDD with matching.

[Taiba Chau]

I understand with only two time periods the parallel trends assumption is not possible to test. Im just confused in terms of robustness checks what to do?

[Taiba Chau]

I have read some papers and have seen that robustness checks could also include running alternative models. Such as OLS or FE or in my case also a regular DiD

[Clyde Schechter]

I think Jared Greathouse has nicely outlined the kind of robustness checks that are feasible in #2. I would, however, in this setting, be more concerned about other threats to validity. Countries that adopt policies together often share other common features--they might be in similar stages of economic development, or be geographically near each other, or have some important cultural similarities. And all of those things could influence the trajectory of mental health among people in those countries. If I were reviewing this as a paper, I would want to see some "placebo" tests. In particular, I would want to see if replacing the treatment variable by a variable indicating geographic region instead of the actual policy implementation makes the effect go away, or nearly so. If the effect is due to the policy, it should. Similarly I would want to see whether the effect disappears when some good indicator of economic development is used in place of the policy variable. Another related issue would be the age distributions in the countries (which is related to both economic development and culture). So you might look at the median age in each country and then group the countries into a high vs low median age groups and see what happens when that replaces the policy variable. Cultural similarities are harder to explicitly test, but indicators for the countries' dominant religion are sometimes a reasonable proxy for that, I would think especially so with outcomes like mental health.

[Taiba Chau]

I also came across a placebo test but since I only have two groups and one measure of well-being. What could an alternative measure for this placebo test?

[Clyde Schechter]

Strictly speaking, a placebo test would be done by replacing the policy variable by a variable that selected two groups at random. But I think even more important than that would be the kind of tests I mentioned in #5.

[Jared Greathouse]

One possible way of getting around the violations of SUTVA (the kind that Clyde Schechter speaks of in 5 where geography/similar policies can influence the outcomes in neighboring areas) is to see if your findings are robust to changes of the geography of your comparison group. I'll give an example from a current paper where I use synthetic controls. I write

We have m_n ≥ 1 . . . 13 treated units and 65 never-treated donor units. I use two sets of donor units: Puerto Rico and all counties in California, New Mexico, Arizona, Nevada, Florida and Texas. I use mainland counties as donors because there could be spillovers, a violation of SUTVA.

In this case, I'm studying Puerto Rico and a series of mass vaccination sites they used in March of 2021-May of the same year. One possible cause for concern could be that people from neighboring counties/municipios may flock to these sites they day they occur, thus affecting the COVID-19 numbers in that county, too. So, if I select a subset of highly Hispanic areas on the mainland United States, there's no argument to be made that the effects of one policy is spilling over into another area. If your effect size holds or at least is similar when you do this, you've got more confidence that your results are plausible. The fact that certain areas share much in common, geography, culture and so on, that's a good thing. The time that it comes to be a problem is when a policy occurs in areas that're in close proximity to each other, since the possibility for spillovers exist unless these can be ruled out.

Recall that the point of having parallel trends is the idea that were it not for the policy, the trends of your treated units would continue in the same direction as the untreated units; the goal is to simulate a randomized experiment where the control group us like the treatment group in every single conceivable way EXCEPT for the fact that the intervention happened in one area versus not.


Having said all of this, I should note that there's room for deviation here depending on the context. If you're comparing countries, perhaps the spillover effects won't matter as much. The United States legalized marijuana (recreationally anyways) in 2012/2014 in Colorado. If we were comparing crime rates in the United States, Mexico and Canada, it wouldn't make as much sense to be concerned about spillovers and other things that might confound your intervention and outcome. I know I've sort of said a lot here, but these are the kinds of contextual requirements you'll need to take into consideration when doing a real paper (i.e., not for class where your instructor likely won't expect this level of detail).

[Taiba Chau]

That was very insightful and appreciate both your thoughts and knowledge. Thank you