Robustness Tests for DiD in 2SLS Fixed Effects Models

[Kristoffer Sundstroem]

Hello everyone,

I have recently gotten some valuable feedback on one of my papers which suggested to run some robustness checks on the DiD interaction term estimates in my regressions.

The panel dataset contains approximately 800 observations spread across 40 years and 20 firms.

Starting off I have the following code:

 xtivreg2 COD ProducedPulp i.Year (AllowedCOD AllowedCOD_EC = Lag1PP TSL Lag1PP_TSL Lag1PP_O2D), fe endog(AllowedCOD AllowedCOD_MB)

The COD is the dependent variable for emissions. One control variable is added together with 2 independent variables, Allowed COD and Allowed COD interacted with the treatment, that are believed to suffer from endogeneity and is therefore dealt with through the use of instrumental variables. The model also employ Fixed Effects for both firm and year.

The code runs smoothly and presents results that are in line with the hypothesis presented in the paper.

The results look like the following see Model 2 marked in red:

	Model 1 FEM	Model 2 2SLS FE
COD Limit Ton/year	0.538***	0.570***
	(0.0241)	(0.0585)
COD Limit x EC Dummy	-0.350***	-0.528***
	(0.0306)	(0.0605)
Produced Pulp Ton/year	0.00461*	0.00573**
	(0.00199)	(0.00203)

The suggested robustness checks that I got from the reviewers were to try changing the implementation of the treatment (a change in regulatory regime in this specific case) so that it occurs a couple of years earlier and a couple of years later in order to investigate the impact on the estimates for the interaction term shown above.

As I change the year of implementation of the treatment I get the following results.

When I move the implementation to an earlier date (4 years before the actual year) the coefficients changes to approximately:
COD Limit = 0.65
COD Limit x EC = -0.68

As I move the implementation to an later date (4 years after the actual year) the coefficients changes to approximately:
COD Limit = 0.53
COD Limit x EC = -0.50

My intuition just suggest that I can, after these robustness checks, conclude that the impact of the treatment does not seem to have a specific effect on the effectiveness of the COD limits. But I am worried that I have missed something crucial as these numbers are quite similar to the original regression. Is it something to worry about or can I just conclude that the treatment do not have any real impact on the COD limits effectiveness as it seems to capture something else occurring over time?

Thanks in advance.

Kind regards,
Kristoffer

[George Ford]

I think you need to delete all treated periods when you move the implementation date back in time. If there's an effect, then it will make the coefficient non-zero even if you drag is back in time (it's a weighted average of 0 and an effect).

Or, add a dummy for the 4 years prior (or 5, or 3) to the full model (should be 0 in the treated period). It should not be significant.

[Kristoffer Sundstroem]

I think you need to delete all treated periods when you move the implementation date back in time. If there's an effect, then it will make the coefficient non-zero even if you drag is back in time (it's a weighted average of 0 and an effect).

Or, add a dummy for the 4 years prior (or 5, or 3) to the full model (should be 0 in the treated period). It should not be significant.

Thank you George,

This is clearly the mistake that was made. When the earlier treated period was removed the following results were obtained:

Implementation to an earlier date (10 years before the actual year and during a 5 year period) the coefficients changes to approximately:
COD Limit = 0.08
COD Limit x EC = 1.57^***

As I move the implementation to a later date (5 years after the actual year and during a 5 year period) the coefficients changes to approximately:
COD Limit = 0.49^***
COD Limit x EC = -1^***

This time I included the significance levels for the coefficients as they changed after this new specification.

This is in line with the hypothesis that the untreated period should have a positive coefficient for the interaction whilst the period afterwards should have a negative coefficient.

Kind regards,
Kristoffer