DiD and parallel trend assumption

[Taiba Chau]

Hi, I currently have a DiD method for testing the implication of a policy on the mental wellbeing of natives in Germany and the EU. The ran my regression and thought to draw some graphs to see trends for pre-treatment. So when doing this I found that the average mental well-being of people in the EU and Germany differ. So they do not follow the same trend. I am wondering how this now would affect my results and whether I have to adjust my methodology or interest groups to make a valid analysis of the policy on these groups of people. Thank you!

[Clyde Schechter]

First, since Germany is part of the EU, it is not valid to compare with statistical tests anything about Germany with anything about the EU. You can compare things about Germany with the corresponding things of the rest of the EU other than Germany.

So if you really analyzed Germany vs EU, there is nothing further to say: your analyses are invalid and there is no point trying to interpret them.

If you properly analyzed Germany vs the rest of the EU, then the relevant issue is not whether the average mental well being in the two differ: the relevant is whether their trends over time are parallel. These are two different things and either can be true or false independently of the other. So you need to be clear on what exactly you have looked at. Showing your results here for more concrete advice would be helpful.

If you have, in fact, properly looked at the parallelism of the trends in mental health in the two groups (not the average levels of mental health) and found them to be materially different, then you simply cannot use DID methodology: the core assumption justifying its use is violated. Some other approach would be needed.

[Jared Greathouse]

My advice here is, IF you have data for multiple nations across multiple years, is to use synthetic controls.

Synthetic Controls is the cooler, more rigorous older brother of DD that does what DD does in a better, more transparent way

[Taiba Chau]

Thank you for your responses. It is the comparison of the policy imposed in Germany and how that affects the wellbeing of people in Germany VS other countries in the EU. I have looked at the trends of wellbeing over time for the two groups. For which I find that it is not the same. I used another form of wellbeing which is happiness and that tends to follow a similar trend for both groups. I understand I cannot use DiD if the trends in mental health do not follow the same trend. What would be the alternative method then?

[Clyde Schechter]

Well, the first thing that came to my mind as an alternative is the one you have already tried: look for an alternate outcome measure that does follow parallel trends. You might be able to modify your current wellbeing measure. If it is a multi-item scale from a survey, it may be that an abbreviated form of the scale would work. You might find that by checking for parallel trends on the individual items in the scale and eliminating items that show a badly non-paralllel trend. Putting the rest together as a subscale might work out. Of course, you would need to check the reliability of the subscale before using it as an outcome.

Propensity scores may be another useful approach if you have enough other variables that are suitable for the purpose.

Added: Might there be some subpopulation of the EU that could serve as a control group and would exhibit parallel trends with Germany on the wellbeing outcome?

[Jared Greathouse]

The alternative method in my mind would be synthetic controls. You likely have a medium amount of dimensions, a reasonable donor pool. My question for you, is how many years of pre-intervention data do you have?


EDIT: I don't have anything against Clyde's suggestion, but following recent work by Athey and co, they make the argument that matching is more appropriate when you've got a relatively large number of donor units and not many pre intervention periods, and that synthetic controls are more appropriate when you've got a small to moderately sized donor pool, and a long pre-intervention time series.

Of course, you could use Athey's proposed nuclear norm matrix completion estimator, but this isn't for the faint of heart since it's a mathematically challenging command simply to understand.

So if this were my problem, SCM or even interrupted time series (maybe both!) is what I'd do.

[George Ford]

See Mastering Metrics, p. 196.

[Clyde Schechter]

Re #6.

but following recent work by Athey and co, they make the argument that matching is more appropriate when you've got a relatively large number of donor units and not many pre intervention periods,

That is certainly correct. I just want to point out that I never said matching. Matching is only one way in which propensity scores can be used, and many feel that it is never the best way to use them. There is simply adjustment for propensity score, and there is stratification by propensity score ranges. To be honest, I'm not very fond of propensity score approaches and seldom use them myself. It just struck me that this might be a situation where it would be useful.

That said, synthetic control is another good methodological suggestion here. It is, in a sense, a more sophisticated version of my suggestion to seek out a subpopulation to use as a control group.

[Jared Greathouse]

Yeah you're right, I'm so used to PS in the matching context, but like you implied, King and others say that you shouldn't use them for that. Presumably, other matching methods exist too.

In fact, a related version that could be employed is IPW analysis.

[George Ford]

One way that is commonly used is to plot the pre-treatment graphs for each EU member and see which are closest (visual inspection). Synthetic Counterfactual is a mechanical way to do the same, sort of, and gets the researcher out of it (so to speak). I've never had much luck with SC, but plenty of people do. There's a bit of a learning curve since the technique is somewhat new and under advancement.

In Mastering Metrics, AP suggest adding a time trend unique to each unit. I'd start there as it takes about two minutes to implement. But there are other options.