Diff-in-diff with two different treatment years

[Dirk van Beers]

First of all, thank you to all the participants in this forum for everything I have learned just by browsing here during my studies.

Now I have encountered a problem that I have not found an answer to by searching existing posts on this forum.

About my study:
I have two groups of companies that I have panel data for, namely their annual accounting numbers from 2006 to 2017. One group receives a treatment in either 2010 or 2011, and the other group is a control group which does not receive treatment at all. This control group was obtained through propensity score matching based on accounting data from the year -before- treatment was given.

What I want to find is the effect of this treatment on the (log) labor costs of these companies. It is not certain that the effect occurs immediately, so I want to follow the effects year-by-year after the treatment is given.

About my model:
My variable treat is 1 for the treated companies, 0 for the control companies
year denotes which accounting year the numbers are from
post is a variable generated to create a common baseline year, where treatment occurs at post = 10, the first year after treatment is post = 11 (look at the last digit). This is true regardless of whether you received treatment in 2010 or 2011.
loglaboris the outcome variable I am trying to estimate the effect of treatment on.

My question is this:
Should I include both the post and year variables, or just one of them in my regression? In other words, how should I control for year fixed effects? Should I even control for them at all? I have three intuitive options:

With both:
(1) reg loglabor i.year i.treat##i.post

Without year:
(2) reg loglabor i.treat##i.post

Without post:
(3) reg loglabor i.year i.treat i.treat#i.post

Option 1 and 2 yields almost the exact same treat#post result. I have attached the results at the bottom of my post.

In Option 1, I have trouble understanding how to interpret what the post variable should mean. It means something like "change in y for a control variable from x years after its treated neighbor received treatment".

In Option 2, it makes no intuitive sense for me to completely leave out the accounting year effects. Is it in that case completely adjusted for by the post variable instead?

Option 3 yields a different result from the other two, and that makes me unsure, although I find it the most intuitively appealing.

Bonus question: I see a lot of people using "xtreg , fe cluster (id)" for regressions similar to mine. I have tried to read up on clustering and fixed effects without quite understanding whether it applies to my case or not, without being able to conclusively tell. Any input on this would also be greatly appreciated.

I hope I have made this post understandable. Please let me know if I can clarify anything.

Output for anyone interested:

Output Option 1:

. reg loglabor i.year i.treat##i.post

      Source |       SS           df       MS      Number of obs   =    23.985
-------------+----------------------------------   F(25, 3959)     =      0,72
       Model |    41,47029        25   1,6588116   Prob > F        =    0,8422
    Residual |   9124,8716     3.959  2,30484254   R-squared       =    0,0045
-------------+----------------------------------   Adj R-squared   =   -0,0018
       Total |  9166,34189     3.984  2,30078863   Root MSE        =    1,5182

------------------------------------------------------------------------------
    loglabor |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
        year |
       2010  |   ,0404684   ,1540413     0,26   0,793    -,2615393    ,3424761
       2011  |   ,2050142   ,2178473     0,94   0,347    -,2220891    ,6321176
       2012  |   ,4441197   ,2668073     1,66   0,096     -,078973    ,9672124
       2013  |   ,6752693   ,3080826     2,19   0,028      ,071254    1,279285
       2014  |   ,8767399   ,3444468     2,55   0,011     ,2014301     1,55205
       2015  |   1,068865   ,3773225     2,83   0,005        ,3291     1,80863
       2016  |   1,220168   ,4091586     2,98   0,003     ,4179866    2,022349
       2017  |   1,396722   ,4397593     3,18   0,002     ,5345464    2,258898
             |
     1.treat |    ,065928   ,1406558     0,47   0,639    -,2098366    ,3416927
             |
        post |
         10  |  -,1470907   ,1833596    -0,80   0,422    -,5065788    ,2123974
         11  |    -,30442   ,2394775    -1,27   0,204    -,7739308    ,1650907
         12  |  -,4960301   ,2847423    -1,74   0,082    -1,054285    ,0622252
         13  |  -,7010035   ,3237389    -2,17   0,030    -1,335714   -,0662928
         14  |  -,8598441   ,3585186    -2,40   0,017    -1,562743   -,1569456
         15  |  -1,044088   ,3903514    -2,67   0,008    -1,809397   -,2787797
         16  |  -1,183748   ,4222374    -2,80   0,005    -2,011572   -,3559254
         17  |  -1,251803   ,4553166    -2,75   0,006     -2,14448   -,3591257
             |
  treat#post |
       1 10  |  -,0898922   ,1989174    -0,45   0,651    -,4798822    ,3000979
       1 11  |  -,0854031   ,1989174    -0,43   0,668    -,4753931     ,304587
       1 12  |  -,0202425   ,1989174    -0,10   0,919    -,4102326    ,3697476
       1 13  |  -,0179358   ,1989174    -0,09   0,928    -,4079259    ,3720543
       1 14  |  -,0394882   ,1989174    -0,20   0,843    -,4294783    ,3505018
       1 15  |  -,0257552   ,2000188    -0,13   0,898    -,4179047    ,3663944
       1 16  |  -,0589348   ,2023696    -0,29   0,771    -,4556932    ,3378237
       1 17  |  -,0996602   ,2255088    -0,44   0,659    -,5417845    ,3424642
             |
       _cons |   10,87102   ,1094211    99,35   0,000     10,65649    11,08554
------------------------------------------------------------------------------

Output Option 2:

. reg loglabor i.treat##i.post

      Source |       SS           df       MS      Number of obs   =    23.985
-------------+----------------------------------   F(17, 3967)     =      0,40
       Model |  15,5372062        17  ,913953304   Prob > F        =    0,9866
    Residual |  9150,80469     3.967  2,30673171   R-squared       =    0,0017
-------------+----------------------------------   Adj R-squared   =   -0,0026
       Total |  9166,34189     3.984  2,30078863   Root MSE        =    1,5188

------------------------------------------------------------------------------
    loglabor |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
     1.treat |    ,065928   ,1407134     0,47   0,639    -,2099494    ,3418055
             |
        post |
         10  |  -,0698784   ,1407134    -0,50   0,619    -,3457558    ,2059991
         11  |  -,0405819   ,1407134    -0,29   0,773    -,3164594    ,2352955
         12  |   ,0045574   ,1407134     0,03   0,974      -,27132    ,2804349
         13  |   ,0219446   ,1407134     0,16   0,876    -,2539329     ,297822
         14  |   ,0618069   ,1407134     0,44   0,661    -,2140705    ,3376844
         15  |   ,0543002   ,1417988     0,38   0,702    -,2237052    ,3323057
         16  |   ,0738036   ,1443347     0,51   0,609    -,2091736    ,3567808
         17  |   ,1329355   ,1609941     0,83   0,409    -,1827034    ,4485745
             |
  treat#post |
       1 10  |  -,0898922   ,1989989    -0,45   0,651    -,4800418    ,3002575
       1 11  |  -,0854031   ,1989989    -0,43   0,668    -,4755527    ,3047466
       1 12  |  -,0202425   ,1989989    -0,10   0,919    -,4103921    ,3699071
       1 13  |  -,0179358   ,1989989    -0,09   0,928    -,4080854    ,3722138
       1 14  |  -,0394882   ,1989989    -0,20   0,843    -,4296378    ,3506614
       1 15  |  -,0238457   ,2000908    -0,12   0,905    -,4161361    ,3684447
       1 16  |  -,0572985    ,202447    -0,28   0,777    -,4542084    ,3396115
       1 17  |  -,0996602   ,2256012    -0,44   0,659    -,5419654    ,3426451
             |
       _cons |     10,883   ,0994994   109,38   0,000     10,68793    11,07808
------------------------------------------------------------------------------

Output Option 3:

. reg loglabor i.year i.treat i.treat#i.post

      Source |       SS           df       MS      Number of obs   =    23.985
-------------+----------------------------------   F(25, 3959)     =      0,72
       Model |    41,47029        25   1,6588116   Prob > F        =    0,8422
    Residual |   9124,8716     3.959  2,30484254   R-squared       =    0,0045
-------------+----------------------------------   Adj R-squared   =   -0,0018
       Total |  9166,34189     3.984  2,30078863   Root MSE        =    1,5182

------------------------------------------------------------------------------
    loglabor |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
        year |
       2010  |   ,0404684   ,1540413     0,26   0,793    -,2615393    ,3424761
       2011  |   ,2050142   ,2178473     0,94   0,347    -,2220891    ,6321176
       2012  |   ,4441197   ,2668073     1,66   0,096     -,078973    ,9672124
       2013  |   ,6752693   ,3080826     2,19   0,028      ,071254    1,279285
       2014  |   ,8767399   ,3444468     2,55   0,011     ,2014301     1,55205
       2015  |   1,068865   ,3773225     2,83   0,005        ,3291     1,80863
       2016  |   1,220168   ,4091586     2,98   0,003     ,4179866    2,022349
       2017  |   1,396722   ,4397593     3,18   0,002     ,5345464    2,258898
             |
     1.treat |    ,065928   ,1406558     0,47   0,639    -,2098366    ,3416927
             |
  treat#post |
       0 10  |  -,1470907   ,1833596    -0,80   0,422    -,5065788    ,2123974
       0 11  |    -,30442   ,2394775    -1,27   0,204    -,7739308    ,1650907
       0 12  |  -,4960301   ,2847423    -1,74   0,082    -1,054285    ,0622252
       0 13  |  -,7010035   ,3237389    -2,17   0,030    -1,335714   -,0662928
       0 14  |  -,8598441   ,3585186    -2,40   0,017    -1,562743   -,1569456
       0 15  |  -1,044088   ,3903514    -2,67   0,008    -1,809397   -,2787797
       0 16  |  -1,183748   ,4222374    -2,80   0,005    -2,011572   -,3559254
       0 17  |  -1,251803   ,4553166    -2,75   0,006     -2,14448   -,3591257
       1 10  |  -,2369829   ,1833596    -1,29   0,196     -,596471    ,1225052
       1 11  |  -,3898231   ,2394775    -1,63   0,104    -,8593339    ,0796877
       1 12  |  -,5162726   ,2847423    -1,81   0,070    -1,074528    ,0419827
       1 13  |  -,7189393   ,3237389    -2,22   0,026     -1,35365   -,0842286
       1 14  |  -,8993323   ,3585186    -2,51   0,012    -1,602231   -,1964338
       1 15  |  -1,069843   ,3903514    -2,74   0,006    -1,835152   -,3045349
       1 16  |  -1,242683   ,4215941    -2,95   0,003    -2,069245   -,4161213
       1 17  |  -1,351463   ,4542815    -2,97   0,003     -2,24211   -,4608152
             |
       _cons |   10,87102   ,1094211    99,35   0,000     10,65649    11,08554
------------------------------------------------------------------------------

[Dirk van Beers]

Does anyone have some input? Can i clarify the question in any way to make it easier to understand?

[Carlo Lazzaro]

Dirk:
post as long as this one are often skipped because the potentially interested listers:
- do not have the time to delve into such many issues;
- spot some issues that can be easily solved just taking a look at Stata .pdf manual and/or any decent panelm data econometric textbooks;
- believe it's error-prone to consider three different regression models and give a clear-cut advice, especially if your research field is not hers/his.

That said:
- I do not understand if you want to go diff in diff (see http://www.princeton.edu/~otorres/DID101.pdf) or -xtreg- (see Stata .pdf manual and related reference);
- under -xttreg- enttry, Stata .pdf manual explains why clustering standard errors is the way to go when your data shows heteroskedasticity and/or autocorrelation (please also note that, unlike -regress-, -robust- and -cluster- options do the very same job under -xtreg-);
- as far as your intearction is concerned, I would consider interacting -year- with -treat-.

[Dirk van Beers]

Mr. Lazzaro,

Thank you so much for getting back to me. Your advice on how to write a better post next time and whether to use clustering or not has been duly noted.

About the interaction:
I might be misunderstanding you, but wouldn't having a year#treat interaction variable defeat my purpose of creating a common baseline? "Treat#year2012" for example, would be the first year after treatment for the group receiving in 2011 and the second year for the group receiving in 2010. I'm trying to estimate the treatment effect x years after treatment has been given, not the treatment effect in calendar year x.

About model choice:
Those slides by Torres are the exact ones I have been using to create my diff-in-diff model. And a diff-in-diff model is certainly what I'm trying to do. What I am essentially wondering is whether to do the diff-in-diff with -reg- (like Torres) or with -xtreg- (because I have panel data).

The case for using -reg- is that Torres is using it. However, what confuses me is that a lot of posts on this forum seem to indicate that using reg for panel data is a mistake. For example this post from Clyde Schechter in a thread which you also participate in, Mr. Lazzaro:

As for -xtreg- vs -reg-, you have panel data: the observations are therefore presumably correlated within firms. So -reg- would be incorrect as it treats all observations as independent. If the output from -xtreg- says that rho is near 0 and that there is no difference between -xtreg- and OLS, then if you want to go back to -reg- for simplicity, you can.

Source: https://www.statalist.org/forums/for...95#post1321095

My rho with xtreg is nowhere near 0. Should I then disregard Torres' use of -reg-, and consider it as a model choice which cannot be applied in my case?

[Carlo Lazzaro]

Dirk:
- thanks for providing further details;
- as far as I can get your query, now, interaction between -treat- and -post- seems the way to go;
- you can also plug in -year- among predictors-. You can then estimate post-regression whether they're jointly significant via -testparm(i.year)-;
- if the F-test appraring a sa footnote of the -xtteg,fe- outcome table fails to reach statistical significance, go -regress-. Provided that I rarely use diff-in-diff- I would consider the user-written programme -diff- (type -net sj 16-1 st0424- from within Stata to spot and install it).

As an aside, please call me Carlo, as all on (and many more off) the list do. Thanks.

[Dirk van Beers]

Carlo, thank you so much!