DID interpretation

[Ashley Lee]

Hi, I am working on my master's thesis and am having trouble interpreting the result.

Here is the command I used:
xtreg light i.t##i.treat i.year lnconsgds lngvtrev lngvtexp lnfixasset lnindentp lnpop lnhh lnavwage lngdp lnarea, fe vce(cl id)

The dependent variable is night light intensity, and the results show the average light intensity in treatment group increased by 3.15 after the opening of the railway line.
The light ranges from 0-63, and I am having trouble finding out how much 3.15 means by comparing it with the counterfactual trends.
t=1 if post period, t=0 if pre period; treat=1 if the township is located less than 10km from the station and treat=0 otherwise.



. xtreg light i.treat##i.t i.year lnconsgds lngvtrev lngvtexp lnfixasset lnindent
> p lnpop lnhh lnavwage lngdp lnarea, fe vce(cl id)
note: 1.treat omitted because of collinearity
note: 2013.year omitted because of collinearity

Fixed-effects (within) regression Number of obs = 7,444
Group variable: id Number of groups = 1,113

R-sq: Obs per group:
within = 0.2637 min = 1
between = 0.3537 avg = 6.7
overall = 0.1687 max = 8

F(18,1112) = 52.76
corr(u_i, Xb) = -0.0201 Prob > F = 0.0000

(Std. Err. adjusted for 1,113 clusters in id)
------------------------------------------------------------------------------
| Robust
light | Coef. Std. Err. t P>|t| [99% Conf. Interval]
-------------+----------------------------------------------------------------
1.treat | 0 (omitted)
1.t | -1.729807 .7768026 -2.23 0.026 -3.734158 .2745437
|
treat#t |
1 1 | 3.15423 .8405881 3.75 0.000 .9852956 5.323164
|
year |
2007 | .1854319 .1151499 1.61 0.108 -.1116845 .4825483
2008 | -.8745761 .236328 -3.70 0.000 -1.484363 -.264789
2009 | -2.520318 .3622885 -6.96 0.000 -3.455116 -1.585521
2010 | .7224478 .3533836 2.04 0.041 -.189373 1.634269
2011 | -.1411444 .207068 -0.68 0.496 -.6754332 .3931443
2012 | -.8863937 .1356414 -6.53 0.000 -1.236383 -.5364039
2013 | 0 (omitted)
|
lnconsgds | 2.069105 .4375113 4.73 0.000 .9402133 3.197997
lngvtrev | 1.054407 .3054798 3.45 0.001 .2661902 1.842624
lngvtexp | .6141107 .1877861 3.27 0.001 .1295741 1.098647
lnfixasset | -.6692963 .1378422 -4.86 0.000 -1.024965 -.3136279
lnindentp | -.2801087 .2391813 -1.17 0.242 -.8972581 .3370406
lnpop | 1.461806 .6951386 2.10 0.036 -.3318307 3.255443
lnhh | .6500884 .6054552 1.07 0.283 -.9121421 2.212319
lnavwage | .1599649 .2070536 0.77 0.440 -.3742869 .6942166
lngdp | .7224559 .251562 2.87 0.004 .0733613 1.371551
lnarea | -4.082284 1.487769 -2.74 0.006 -7.92111 -.2434567
_cons | -15.06321 13.04124 -1.16 0.248 -48.71296 18.58654
-------------+----------------------------------------------------------------
sigma_u | 8.7995316
sigma_e | 1.8401743
rho | .95810041 (fraction of variance due to u_i)
------------------------------------------------------------------------------

.
I am attaching the parallel as a reference. Thank you!

[Clyde Schechter]

The light ranges from 0-63, and I am having trouble finding out how much 3.15 means by comparing it with the counterfactual trends.

Kudos to you for caring about whether you have a meaningful effect, and not just shouting "p < 0.05" and doing a victory lap!!!!

In the end, this is not really a statistical issue: it depends on what this 0-63 light intensity measurement scale is about. And in the end, you need to consult with somebody who is really familiar with that measurement, how it works, and what it means. You might ask whether a difference in light intensity of 3.15 is large enough that people living there would notice the difference? Is it large enough to interfere with sleeping at night? Does it interfere with nearby atronomical observatories (if there are any)? Those are the kind of substantive questions you might consider.

But there are some statistical clues in the output that might also influence your thinking. First, let's not focus too much on the point estimate of 3.15. You have an interval estimate from 0.99 to 5.3. Since the question in your mind is apparently whether you should take the findings as indicating that something meaningful is going on, let's be conservative and imagine that the effect is only 0.99. We can compare that 0.99 to the estimated effects of other things in your model. We can notice that it is very similar to the effect of a unit change in lnvtrev (whatever that is). In fact, it is larger in magnitude than all of your other covariates with the exception of lnconsgds, lnpop, and lnarea. Assuming you were thoughtful about the choice of which covariates were worth entering in the model, I will interpret your decision to include all those other covariates as evidence that their effects are meaningful in some real world sense--if not, why would you have included them? So if those were meaningful, it seems reasonable to conclude that this treatment effect (Near HSR station) is as well.

The above statistical argument is clearly not bullet-proof. It assumes that you selected your covariates with their effect sizes in mind--but sometimes we include covariates just for comparability with other research, and sometimes we expect them to have meaningful effects but, for whatever reason, in the present study, perhaps they do not. Also, I don't know what lnvtrev is, and maybe (especially since I suppose it is the log-transform of some variable vtrev) a unit change in it is much larger than is every observed in reality, so the comparison with a unit change in a dichotomous treatment variable could be misleading.

Ultimately, you need input from somebody who understands that light intensity measure. But, there is at least a prima facie argument that the effect you have estimated her is large enough to matter in the real world.

[Ashley Lee]

Thank you so much Clyde for your insightful comment!

The way I want to interpret is by using the trends in the control group as counterfactual trends. So, for example, it would be something like: "The light intensity of townships near the stations would have been about (let's say) 35 in the absence of the railway. Hence, the estimated increase in light intensity of 3.15 implies that there was about (let's say) 9% increase in the night light intensity."

How can I calculate the average counterfactual value of treatment group? If I do, I guess I can fill up the sentence.

Thanks in advance.

[Clyde Schechter]

It is difficult to coax the counterfactual out of -xtreg, fe- results because the fixed effects themselves play a role but are not really identified without assumptions. If you run a simpler model along the lines of:

regress light i.treat##i.t i.year lnconsgds lngvtrev lngvtexp lnfixasset lnindent
> p lnpop lnhh lnavwage lngdp lnarea
margins year, at(t = 0 treat = 1)

the -margins- output corresponding to treat = 1 will give you an estimate of the counterfactual.

Now for some values of year, there may be no observations for that year with t = 0 and treat = 1 and I suspect -margins- will say those are not estimable. If that occurs, add the -noestimcheck- option to the -margins- command and I believe Stata will comply with a counterfactual.

[Ashley Lee]

Thank you so much Clyde. I really appreciate it. Have a great day!