Unparallel pre-intervention trends in synthetic control

[Eran Itskovich]

Hi all,

I'm trying to run a synthetic control model using the -synth_runner- package and have a general question.

My data contains 201 localities over 9 years - 8 pre-treatment and 1 post-treatment. In the model, I include several socio-demographic variables (e.g., total population, mean age, male ratio, mean income, etc.), as well as the outcome variable, which is crime. The problem is that whatever I'm trying, the trends of the treated and synthetic units are far from being parallel. I also tried using monthly data instead of yearly data, but it didn't make things much better.

Is there something particular that can cause this? maybe my dependent variable changes too much from year to year?

Thanks!

[Tom Scott]

Its hard to find comparable units in an observational study. It is likely there just isn't a weighted combination of localities that match your treated unit's trend. Your fit may go up if you shorten the pre-intervention period but then you risk having biased results. Is there any way you can increase your donor pool by obtain data from more localities? Especially localities that seem similar to your treated unit(s)?

[Dimitriy V. Masterov]

Are your outcomes and matching variables in levels (count of burglaries) or in rates (burglaries per 100K residents)? If it's the former, try the latter and remove pop from the matching covariates.

You also don't need trends to be parallel: synth makes weaker assumptions than diff-in-diff. You do need the synthetic cohort to lie on top of the treated during the pre-period.

[Jeff Wooldridge]

Eran: How many treated and control units do you have out of 201? It appears you have only one treated period, but at least you have several pre-treatment periods.

If you have one treated unit, can you show, graphically -- as suggested by Dimitriy -- how the SC and treated units compare? And you might initially just try matching on pre-treatment outcomes to see how that works.

[Eran Itskovich]

Thanks everyone for your answers!

1. Unfortunately, there's no way to increase my donor pool. This is the data I received.

2. I have 7 treated units and 194 controls. And yes, I have one treated period.

3. I tried using crime rates as suggested by Dimitry, and here's what I got:



Doesn't seem like I get much out of it, is it?

Dimitry, can you please elaborate on why the trends don't have to be parallel, and what it means that the synthetic cohort lies on top of the treated during the pre-treatment period?
It's just that in every article I read, the pre-treatment lines are almost identical.

Thanks again for your kind help!

Eran

[Dimitriy V. Masterov]

You want the treated unit(s) and the SC cohort for it/them to look almost identical in the pre-period. Take a look at

Abadie, Alberto. 2021. "Using Synthetic Controls: Feasibility, Data Requirements, and Methodological Aspects." Journal of Economic Literature, 59 (2): 391-425. https://www.aeaweb.org/articles?id=10.1257/jel.20191450

for the parallel assumption discussion and some examples of what a good fit looks like.

Your graph does not look great. Could you make the same plot for the seven treated units separately? Sometimes, that can let you see if you can model some units better than others and find where the lack of fit originates.

You may also want to add your synth_runner code here in case something is going awry there.

With annual data, it might be hard to make this work since you don't have enough points to estimate the weights well. I would try monthly or quarterly if monthly is too noisy. The other trick to using monthly/quarterly data is that crime tends to be seasonal, higher in summer and lower in winter. But if you have almost every country in the world, you will be mixing seasonalities. You may want to try restricting the potential donor pool for each treated one to countries in the same hemisphere.

[Jeff Wooldridge]

I agree the trends don't have to parallel, but I also agree that's not a great fit before the intervention. In fact, that drop in what I assume is the treated group (solid line) just before the intervention is concerning. I've been teaching an alternative approach -- based on some work with a student of mine -- that seems to work well in some cases. It involves removing unit-specific trends and then projecting into the treated period. When applied to the California smoking data in Abadie, Diamond, and Hainmueller (2010), and using all donor states, it effectively mimics synthetic control (while providing a small-sample confidence interval errors). Somewhat remarkably, it does notably better than SC when you have a "poor" set of controls -- such as Alabama, Arkansas, Louisiana, and Mississippi in the graphs below. I see I should've reversed the colors in the second graph as the blue is the treated unit (CA).

[Jeff Wooldridge]

Eran: I'd be happy to try my method using your data and report back to see if it does better matching the pre-treatment outcomes. And I'm certainly not claiming that it will. But I've found some situations where it does, including the above.