Diff-in-diff with xtreg

[Luca As]

Hi everyone,

I have some questions about the difference-in-differences (DID) model with fixed effects.

I am dealing with a monthly panel dataset – from 2005 to 2016 – for workers and I want to study the effect of a policy introduced on January 2015.

I run the following fixed effect model, where I decided also to put the fixed effects for years and months.

Post is a dummy equal to one if year>=2015 and treat is the dummy equal to one for workers who were eligible for the treatment. To make to question simple to understand, Y is the outcome variable and it is equal to one if the worker finds a job and zero otherwise.

xtset id_worker year_m
xtreg Y i.treat##i.post i.year i.month, fe vce(robust)

The outcome is the following:

. xtreg Y i.treat##i.post i.year i.month, fe vce(robust)
note: 2016.year omitted because of collinearity

Fixed-effects (within) regression               Number of obs     = 14,449,651
Group variable: id_worker                       Number of groups  =    170,670

R-sq:                                           Obs per group:
     within  = 0.0244                                         min =          6
     between = 0.0015                                         avg =       84.7
     overall = 0.0191                                         max =        156

                                                F(25,170669)      =    1094.01
corr(u_i, Xb)  = -0.3293                        Prob > F          =     0.0000

                    (Std. Err. adjusted for 170,670 clusters in id_lavoratore)
------------------------------------------------------------------------------
             |               Robust
           Y |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
     1.treat |   .0109051   .0000801   136.08   0.000      .010748    .0110622
      1.post |   .0017448   .0004856     3.59   0.000     .0007931    .0026965
             |
  treat#post |
        1 1  |    .003317   .0001612    20.57   0.000      .003001     .003633
             |
        year |
       2005  |   .0022443   .0004834     4.64   0.000     .0012969    .0031917
       2006  |   .0020643   .0004861     4.25   0.000     .0011117     .003017
       2007  |    .001732   .0004855     3.57   0.000     .0007803    .0026836
       2008  |   .0020534   .0004858     4.23   0.000     .0011013    .0030055
       2009  |   .0016452   .0004855     3.39   0.001     .0006936    .0025967
       2010  |   .0013525   .0004855     2.79   0.005     .0004009    .0023042
       2011  |   .0013109   .0004856     2.70   0.007     .0003592    .0022625
       2012  |   .0013895   .0004855     2.86   0.004     .0004379    .0023411
       2013  |   .0014791   .0004856     3.05   0.002     .0005272    .0024309
       2014  |   .0010583   .0004855     2.18   0.029     .0001067    .0020098
       2015  |  -.0012254   .0000607   -20.20   0.000    -.0013444   -.0011065
       2016  |          0  (omitted)
             |
       month |
          2  |  -.0174946   .0001178  -148.47   0.000    -.0177256   -.0172637
          3  |  -.0174899   .0001178  -148.44   0.000    -.0177208   -.0172589
          4  |  -.0175043   .0001179  -148.43   0.000    -.0177355   -.0172732
          5  |  -.0175121    .000118  -148.40   0.000    -.0177434   -.0172808
          6  |  -.0175214   .0001181  -148.38   0.000    -.0177529     -.01729
          7  |  -.0179749   .0001211  -148.39   0.000    -.0182123   -.0177375
          8  |  -.0179795   .0001212  -148.38   0.000     -.018217    -.017742
          9  |  -.0180004   .0001213  -148.39   0.000    -.0182382   -.0177627
         10  |  -.0180103   .0001214  -148.38   0.000    -.0182482   -.0177724
         11  |  -.0180065   .0001214  -148.35   0.000    -.0182444   -.0177686
         12  |  -.0180912   .0001219  -148.46   0.000    -.0183301   -.0178524
             |
       _cons |   .0135575   .0004917    27.57   0.000     .0125937    .0145212
-------------+----------------------------------------------------------------
     sigma_u |  .00646775
     sigma_e |   .0373691
         rho |  .02908457   (fraction of variance due to u_i)
------------------------------------------------------------------------------

I have the following three questions:
1) Is it correct to also use the months and years fixed effects? I performed the -testparm- test and it seems ok; but this cause a collinearity problem and it is quite different from the standard diff-in-diff strategy.

2) My outcome variable is a dummy equal to zero or one. Is it ok to perform a linear model? I read that for a diff-in-diff model it should be ok – I am also using vce(robust) for the SE. In this context, I tried to perform a -xtlogit- model but it takes too much time – more than 10 hours and it does not show any results.

3) How should I interpret the coefficients? I tried to use the -margins- command but I only get "(not estimable)". Instead, when I run the xtreg without the months and years fixed effects, the command works well. What should I do?

. margins treat#post

Predictive margins                              Number of obs     = 14,449,651
Model VCE    : Robust

Expression   : Linear prediction, predict()

---------------------------------------------------------------------------------------
                      |            Delta-method
                      |     Margin   Std. Err.      z    P>|z|     [95% Conf. Interval]
----------------------+----------------------------------------------------------------
           treat#post |
                 0 0  |          .  (not estimable)
                 0 1  |          .  (not estimable)
                 1 0  |          .  (not estimable)
                 1 1  |          .  (not estimable)
---------------------------------------------------------------------------------------

Thanks a lot!

[Clyde Schechter]

1) This really has nothing to do with DID. You just have to ask yourself what you are hoping to accomplish by including month. Including month adds to the model a "seasonal" shock (where season = 1 month, and the same shock recurs each year in that month) to the outcome variable. If there is reason to believe that the real world outcome exhibits such variation, then it is perfectly fine to add it to the model, whether it is a DID analysis or just any regression. In that regard, it's really no different from any other covariate you might reasonably add to the model.

2) If the predicted outcomes are staying away from 0 and 1, then a linear probability model (which is what you are getting from -xtreg-) is perfectly OK. If you are getting predicted values near 0 or 1, or even negative and greater than 1, then, you may not like this model. You don't provide enough information about what went wrong with your -xtlogit- model, but that is the usual next step. Were you making no progress towards convergence, and getting "not concave" or "backed up" messages all the time? If so, after how many iterations did Stata get stuck? You could rerun the -xtlogit-, adding the -iterate(#)- option, setting # to enough iterations to just get you into the place where Stata gets stuck. Then -xtlogit- will stop and show you interim results. These results are not correct and cannot be used as results, but they may show you why -xtlogit- is having difficulty converging. The clue will be some variable whose coefficient is some outlandishly large number, or has some outlandishly large standard error. Removing that (those) variable(s) from the model will usually lead to convergence. Such variables are often problematic in their own right: predictors that are nearly perfect, or dichotomous predictors that are nearly always 1 or nearly always 0, etc.

3) When you run a fixed effects model with both treat and worker_id effects, and both post and year effects, you have introduced some colinearities. Stata identifies the model by omitting some of the involved variables. Notice that you have been warned that 2016.year was omitted due to colinearity. The problem is that the predictive margins you are trying to estimate with your -margins- command depend on which variable gets omitted to identify the model. In other words, you are trying to calculate statistics that are not well defined: they are artifacts of the particular choice of omitted variables. Here's an example where you can see what happens if you override Stata's concerns about estimability:

webuse nlswork, clear

keep if inrange(idcode, 1, 10)
gen byte treat = (idcode > 5)
gen byte post = (year >= 80)

regress ln_wage i.treat##i.post i.idcode,  vce(cluster idcode)
margins treat#post, noestimcheck

regress ln_wage i.treat##i.post ib5.idcode, vce(cluster idcode)
margins treat#post, noestimcheck

Notice that the -margins- results are different for the two regressions; yet these two models are not substantively different--they are just two different ways of breaking the colinearity between treat and idcode. You have the same problem. When you use -xtreg, fe- Stata has one particular strategy for dealing with the colinearity. But the -margins- results would be different if another one were used. This is why these margins are not estimable: they are just artifacts of the particular choices made to break colinearity. In fact, you have this problem twice: once for worker_id and treat, and again for post and year. Nevertheless, note that your key outcome, the DID estimate of the treatment effect, comes out the same in both models--it is the coefficient of the interaction term in the regression output. And that is what really counts here.

That said, I have concerns about the data. Your output does not look right. If your treat variable is truly set to 0 for those who were not eligible for the policy and 1 for those who were, it should be colinear with the worker_id fixed effects and Stata should omit it. The fact that it was not admitted is proof that you have some worker_id's who are inconsistently coded 0 in some observations and 1 in others. Similarly, there is something wrong with your post variable. It should be colinear with the year variables. Consequently, in addition to omitting the lowest year in your data, 2005, Stata should be omitting either the post variable itself or an additional year. Yet your output shows that only one year, 2016 has been omitted: you should have lost either the post variable or another year. So this is proof that your post variable is not consistently coded the same in all observations for some year, or, alternatively, your data actually begin in 2004 (so that Stata has omitted both 2004 and 2016, which would be fine). You need to fix these problems before you go any further.

[Luca As]

Thank you very much for your answer!

I begin from the end. (i) The eligibility criteria for the treatment varies over time – i.e. workers may be eligible in some months but not in others, since the criterion also depends on their past working history, which varies over time. This is the reason why the variable is not collinear with worker_id. (ii) Yes, you are right: there was an issue with the year variable. Now the dataset correctly begins in 2005 and Stata dropped 2005 and 2016. This is the new output:

 xtreg Y i.treat##i.post i.year i.month, fe vce(robust)
note: 2016.year omitted because of collinearity

Fixed-effects (within) regression               Number of obs     = 14,445,756
Group variable: id_worker                    Number of groups  =    170,671

R-sq:                                           Obs per group:
     within  = 0.0244                                         min =          6
     between = 0.0015                                         avg =       84.6
     overall = 0.0190                                         max =        138

                                                F(24,170670)      =    1138.10
corr(u_i, Xb)  = -0.3294                        Prob > F          =     0.0000

                             (Std. Err. adjusted for 170,671 clusters in id_lavoratore)
---------------------------------------------------------------------------------------
                      |               Robust
                    Y |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
----------------------+----------------------------------------------------------------
              1.treat |   .0108976   .0000801   136.03   0.000     .0107406    .0110546
               1.post |   -.000505   .0000381   -13.27   0.000    -.0005796   -.0004305
                      |
           treat#post |
                 1 1  |   .0033141   .0001611    20.57   0.000     .0029983      .00363
                      |
                 year |
                2006  |  -.0001907     .00004    -4.77   0.000    -.0002691   -.0001123
                2007  |  -.0005198   .0000372   -13.97   0.000    -.0005928   -.0004469
                2008  |  -.0002006   .0000407    -4.93   0.000    -.0002804   -.0001208
                2009  |  -.0006042   .0000375   -16.10   0.000    -.0006777   -.0005307
                2010  |  -.0008945   .0000371   -24.10   0.000    -.0009672   -.0008217
                2011  |   -.000939   .0000379   -24.75   0.000    -.0010134   -.0008646
                2012  |  -.0008637   .0000392   -22.03   0.000    -.0009405   -.0007868
                2013  |  -.0007731   .0000405   -19.11   0.000    -.0008524   -.0006939
                2014  |  -.0011924   .0000384   -31.07   0.000    -.0012676   -.0011172
                2015  |  -.0012255   .0000607   -20.20   0.000    -.0013444   -.0011066
                2016  |          0  (omitted)
                      |
                month |
                   2  |   -.017479   .0001178  -148.39   0.000    -.0177099   -.0172482
                   3  |  -.0174743   .0001178  -148.36   0.000    -.0177052   -.0172434
                   4  |  -.0174888   .0001179  -148.34   0.000    -.0177199   -.0172577
                   5  |  -.0174965    .000118  -148.31   0.000    -.0177278   -.0172653
                   6  |  -.0175059    .000118  -148.29   0.000    -.0177373   -.0172746
                   7  |  -.0179589   .0001211  -148.31   0.000    -.0181963   -.0177216
                   8  |   -.017964   .0001211  -148.29   0.000    -.0182014   -.0177265
                   9  |  -.0179849   .0001213  -148.30   0.000    -.0182226   -.0177472
                  10  |  -.0179947   .0001213  -148.29   0.000    -.0182325   -.0177569
                  11  |   -.017991   .0001213  -148.26   0.000    -.0182288   -.0177531
                  12  |  -.0180757   .0001218  -148.38   0.000    -.0183144   -.0178369
                      |
                _cons |   .0157947   .0001007   156.81   0.000     .0155973    .0159921
----------------------+----------------------------------------------------------------
              sigma_u |  .00646177
              sigma_e |  .03734938
                  rho |  .02906214   (fraction of variance due to u_i)
---------------------------------------------------------------------------------------

1) Ok, perfect. I understand.

2) As you suggested, I come back to -xtlogit-. After some iterations, the output is the following one. I also left the code run for more than 100 iterations and results are all the same: log likelihood = -17976.42 (not concave).

. xtlogit Y i.treat##i.post i.year i.month, fe iterate(10)
note: 2016.anno omitted because of collinearity
note: multiple positive outcomes within groups encountered.
note: 150,917 groups (12,339,809 obs) dropped because of all positive or
      all negative outcomes.

Iteration 0:   log likelihood = -64422.007  
Iteration 1:   log likelihood = -18841.716  
Iteration 2:   log likelihood = -18024.155  
Iteration 3:   log likelihood = -17980.667  
Iteration 4:   log likelihood = -17976.605  
Iteration 5:   log likelihood = -17976.432  
Iteration 6:   log likelihood =  -17976.42  
Iteration 7:   log likelihood =  -17976.42  
Iteration 8:   log likelihood =  -17976.42  (not concave)
Iteration 9:   log likelihood =  -17976.42  (not concave)
Iteration 10:  log likelihood =  -17976.42  (not concave)
convergence not achieved

Conditional fixed-effects logistic regression   Number of obs     =  2,105,947
Group variable: id_workers .                    Number of groups  =     19,754

                                                Obs per group:
                                                              min =         18
                                                              avg =      106.6
                                                              max =        138

                                                LR chi2(22)       =  152394.65
Log likelihood  =  -17976.42                    Prob > chi2       =     0.0000

---------------------------------------------------------------------------------------
                    Y |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
----------------------+----------------------------------------------------------------
              1.treat |   11.38261   1.000831    11.37   0.000     9.421017     13.3442
               1.post |  -12.93925   .0463526  -279.15   0.000     -13.0301   -12.84841
                      |
           treat#post |
                 1 1  |   15.30532          .        .       .            .           .
                      |
                 year |
                2006  |  -.8287109   .0656597   -12.62   0.000    -.9574015   -.7000203
                2007  |  -.1558317   .0558165    -2.79   0.005    -.2652301   -.0464334
                2008  |   .4988535   .0501728     9.94   0.000     .4005166    .5971903
                2009  |   .6166546   .0491428    12.55   0.000     .5203365    .7129727
                2010  |   .6924009   .0480835    14.40   0.000      .598159    .7866428
                2011  |   .7659856   .0468074    16.36   0.000     .6742448    .8577263
                2012  |   1.035521   .0446783    23.18   0.000      .947953    1.123089
                2013  |    1.30401   .0418449    31.16   0.000     1.221995    1.386024
                2014  |   1.160855          .        .       .            .           .
                2015  |  -.9101875   .0427507   -21.29   0.000    -.9939774   -.8263977
                2016  |          0  (omitted)
                      |
                month |
                   2  |  -27.71943   10846.76    -0.00   0.998    -21286.97    21231.53
                   3  |  -27.72012   10785.01    -0.00   0.998    -21165.95    21110.51
                   4  |  -27.71645   10639.92    -0.00   0.998    -20881.59    20826.15
                   5  |  -27.71452   10492.52    -0.00   0.998    -20592.67    20537.24
                   6  |  -27.70921   10319.25    -0.00   0.998    -20253.06    20197.64
                   7  |  -27.51706    6516.38    -0.00   0.997    -12799.39    12744.35
                   8  |  -27.52333   6481.301    -0.00   0.997    -12730.64    12675.59
                   9  |  -27.52944    6414.36    -0.00   0.997    -12599.44    12544.39
                  10  |  -27.53725   6353.014    -0.00   0.997    -12479.22    12424.14
                  11  |  -27.54103   6318.917    -0.00   0.997    -12412.39    12357.31
                  12  |   -27.5213   6148.496    -0.00   0.996    -12078.35    12023.31
---------------------------------------------------------------------------------------

3) Ok, I understand. I also tried to use the noestimcheck command as you did and this is the output. Should I use this coefficient as the result? I read that this is not always a good approach. Moreover, in many economics paper, the DID coefficient from the standard xtreg is read as the percentage effect on the outcome – in my cases: an increase in the probability of finding a job about 3.3%. Is this completely wrong?

Thanks again!

[Clyde Schechter]

Well, those month coefficients and standard errors in the -xtlogit- output are clearly pathological. I would take a look at a cross tabulation of month with your outcome variable -tab Y month if e(sample)- to see if the problem is some months with too few observations, or maybe in some months nearly all observations have the same outcome, or something like that. In any case, it is clear that something about the month variable and its relationship to Y is interfering here. It may or may not be fixable, and you may need to just omit month if you want to use a logistic model. Or you might consider monthly adjustment important enough to warrant sticking with a linear probability model instead of logistic.

I also tried to use the noestimcheck command as you did and this is the output. Should I use this coefficient as the result? I read that this is not always a good approach. Moreover, in many economics paper, the DID coefficient from the standard xtreg is read as the percentage effect on the outcome – in my cases: an increase in the probability of finding a job about 3.3%. Is this completely wrong?

Well, what I showed with the -noestimcheck- option is that the results you can force Stata to produce are just an artifact of the way in which colinearity is broken, so the results are not usable. So you definitely should not use the results produced by -margins, noestimcheck- here. The interaction coefficient from -xtreg- is the DID estimator of the effect. It implies that during the transition from the pre- to post-policy period, the change in Y was 0.0033 greater in the treatment group than in the control group. A shorter way of saying this is that the estimated causal effect of the policy was to increase the probability of a positive outcome by 0.33 percentage points. (Not 0.33 percent!!).

[Luca As]

Thank you very much for your answer; it has been very precious.

I checked the dataset and, indeed, for some months the observations for the outcome were always zero. This was due to an error in the construction of the panel dataset and I corrected it.

Now the -xtlogit- command works and results are the following. Are they reasonable? I also run the -xtreg- regression and coefficients of interest are very different from each other. Is this ok?

Thanks again!

-xtlogit-

. xtlogit Y i.treat##i.post i.year i.month, fe
note: 2016.year omitted because of collinearity
note: multiple positive outcomes within groups encountered.
note: 132,783 groups (11,042,781 obs) dropped because of all positive or
      all negative outcomes.

Iteration 0:   log likelihood = -36675.213  
Iteration 1:   log likelihood = -36006.181  
Iteration 2:   log likelihood = -35980.464  
Iteration 3:   log likelihood = -35980.394  
Iteration 4:   log likelihood = -35980.394  

Conditional fixed-effects logistic regression   Number of obs     =  2,409,337
Group variable: id_worker                       Number of groups  =     23,140

                                                Obs per group:
                                                              min =          2
                                                              avg =      104.1
                                                              max =        138

                                                LR chi2(24)       =  155677.12
Log likelihood  = -35980.394                    Prob > chi2       =     0.0000

---------------------------------------------------------------------------------------
                    Y |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
----------------------+----------------------------------------------------------------
              1.treat |   4.722902    .041916   112.68   0.000     4.640748    4.805056
               1.post |   1.421016   .1699885     8.36   0.000     1.087844    1.754187
                      |
           treat#post |
                 1 1  |   .9190997   .0792322    11.60   0.000     .7638074    1.074392
                      |
                 year |
                2006  |   .7021247   .1503696     4.67   0.000     .4074057    .9968438
                2007  |    .955885   .1528308     6.25   0.000     .6563421    1.255428
                2008  |   1.349586   .1528857     8.83   0.000     1.049935    1.649236
                2009  |   1.182648   .1537537     7.69   0.000     .8812959    1.483999
                2010  |   1.182182   .1539311     7.68   0.000     .8804827    1.483882
                2011  |   1.161383   .1540853     7.54   0.000      .859381    1.463384
                2012  |   1.297923   .1541221     8.42   0.000     .9958495    1.599997
                2013  |   1.384475   .1543875     8.97   0.000     1.081881    1.687069
                2014  |   1.245497   .1550285     8.03   0.000     .9416464    1.549347
                2015  |   -.569596   .0378503   -15.05   0.000    -.6437813   -.4954107
                2016  |          0  (omitted)
                      |
                month |
                   2  |   -4.19179   .0728082   -57.57   0.000    -4.334491   -4.049088
                   3  |   -4.40417   .0783918   -56.18   0.000    -4.557815   -4.250525
                   4  |  -4.639207   .0839954   -55.23   0.000    -4.803835   -4.474579
                   5  |   -4.67984   .0837219   -55.90   0.000    -4.843932   -4.515748
                   6  |  -4.711035    .083957   -56.11   0.000    -4.875588   -4.546482
                   7  |   -4.85544   .0794886   -61.08   0.000    -5.011235   -4.699645
                   8  |  -4.955462   .0826207   -59.98   0.000    -5.117396   -4.793529
                   9  |  -4.633235   .0694872   -66.68   0.000    -4.769428   -4.497043
                  10  |  -4.431261   .0624429   -70.96   0.000    -4.553647   -4.308875
                  11  |  -4.735451   .0718098   -65.94   0.000    -4.876196   -4.594707
                  12  |  -4.725697   .0709184   -66.64   0.000    -4.864694   -4.586699
---------------------------------------------------------------------------------------

Odds ratio

. xtlogit, or

Conditional fixed-effects logistic regression   Number of obs     =  2,409,337
Group variable: id_worker                       Number of groups  =     23,140

                                                Obs per group:
                                                              min =          2
                                                              avg =      104.1
                                                              max =        138

                                                LR chi2(24)       =  155677.12
Log likelihood  = -35980.394                    Prob > chi2       =     0.0000

---------------------------------------------------------------------------------------
                    Y | Odds Ratio   Std. Err.      z    P>|z|     [95% Conf. Interval]
----------------------+----------------------------------------------------------------
              1.treat |   112.4942   4.715304   112.68   0.000     103.6219    122.1263
               1.post |   4.141325   .7039775     8.36   0.000      2.96787    5.778748
                      |
           treat#post |
                 1 1  |   2.507032   .1986378    11.60   0.000     2.146433    2.928212
                      |
                 year |
                2006  |   2.018036   .3034513     4.67   0.000     1.502914    2.709716
                2007  |   2.600971   .3975085     6.25   0.000     1.927728    3.509339
                2008  |   3.855828   .5895009     8.83   0.000     2.857466    5.203004
                2009  |   3.263002   .5016985     7.69   0.000     2.414026    4.410549
                2010  |   3.261484   .5020439     7.68   0.000     2.412064    4.410031
                2011  |   3.194347    .492202     7.54   0.000     2.361698    4.320557
                2012  |   3.661685   .5643467     8.42   0.000     2.707023    4.953019
                2013  |    3.99273   .6164275     8.97   0.000     2.950225     5.40362
                2014  |    3.47466   .5386713     8.03   0.000       2.5642    4.708394
                2015  |    .565754    .021414   -15.05   0.000     .5253024    .6093206
                2016  |          1  (omitted)
                      |
                month |
                   2  |   .0151192   .0011008   -57.57   0.000     .0131085    .0174383
                   3  |   .0122262   .0009584   -56.18   0.000     .0104849    .0142567
                   4  |   .0096654   .0008118   -55.23   0.000     .0081982     .011395
                   5  |   .0092805    .000777   -55.90   0.000      .007876    .0109354
                   6  |   .0089955   .0007552   -56.11   0.000     .0076306    .0106044
                   7  |   .0077859   .0006189   -61.08   0.000     .0066627    .0090985
                   8  |   .0070448    .000582   -59.98   0.000     .0059916    .0082832
                   9  |   .0097233   .0006756   -66.68   0.000     .0084852    .0111419
                  10  |   .0118995    .000743   -70.96   0.000     .0105287    .0134487
                  11  |   .0087785   .0006304   -65.94   0.000      .007626    .0101052
                  12  |   .0088645   .0006287   -66.64   0.000     .0077142    .0101864
---------------------------------------------------------------------------------------

-xtreg-

. xtreg Y i.treat##i.post i.year i.month, fe vce(robust)
note: 2016.year omitted because of collinearity

Fixed-effects (within) regression               Number of obs     = 13,452,118
Group variable: id_worker                       Number of groups  =    155,923

R-sq:                                           Obs per group:
     within  = 0.0282                                         min =          1
     between = 0.0032                                         avg =       86.3
     overall = 0.0214                                         max =        138

                                                F(24,155922)      =    1151.67
corr(u_i, Xb)  = -0.3536                        Prob > F          =     0.0000

                             (Std. Err. adjusted for 155,923 clusters in id_worker)
---------------------------------------------------------------------------------------
                      |               Robust
                    Y |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
----------------------+----------------------------------------------------------------
              1.treat |   .0143493   .0001015   141.36   0.000     .0141504    .0145483
               1.post |  -.0008561   .0000499   -17.17   0.000    -.0009538   -.0007583
                      |
           treat#post |
                 1 1  |   .0051838   .0002095    24.75   0.000     .0047733    .0055944
                      |
                 year |
                2006  |  -.0004047   .0000498    -8.12   0.000    -.0005024    -.000307
                2007  |  -.0006825   .0000474   -14.40   0.000    -.0007754   -.0005896
                2008  |  -.0003382   .0000512    -6.60   0.000    -.0004385   -.0002378
                2009  |  -.0009904   .0000473   -20.93   0.000    -.0010831   -.0008976
                2010  |  -.0012933   .0000473   -27.35   0.000     -.001386   -.0012007
                2011  |  -.0014034   .0000481   -29.15   0.000    -.0014977    -.001309
                2012  |  -.0012901   .0000493   -26.14   0.000    -.0013868   -.0011934
                2013  |  -.0013164   .0000499   -26.38   0.000    -.0014142   -.0012185
                2014  |  -.0017393   .0000481   -36.15   0.000    -.0018337    -.001645
                2015  |  -.0013099    .000072   -18.19   0.000    -.0014511   -.0011687
                2016  |          0  (omitted)
                      |
                month |
                   2  |  -.0209951   .0001388  -151.29   0.000     -.021267   -.0207231
                   3  |  -.0210382   .0001389  -151.50   0.000    -.0213104   -.0207661
                   4  |  -.0210946    .000139  -151.72   0.000    -.0213671   -.0208221
                   5  |  -.0211071   .0001392  -151.63   0.000    -.0213799   -.0208343
                   6  |  -.0211097   .0001393  -151.50   0.000    -.0213828   -.0208366
                   7  |  -.0213578   .0001407  -151.75   0.000    -.0216336   -.0210819
                   8  |  -.0213892   .0001409  -151.79   0.000    -.0216654   -.0211131
                   9  |  -.0213918   .0001415  -151.17   0.000    -.0216692   -.0211145
                  10  |  -.0213656   .0001419  -150.61   0.000    -.0216437   -.0210876
                  11  |  -.0214394   .0001419  -151.08   0.000    -.0217175   -.0211613
                  12  |  -.0214728   .0001422  -151.00   0.000    -.0217516   -.0211941
                      |
                _cons |   .0191357   .0001214   157.57   0.000     .0188977    .0193737
----------------------+----------------------------------------------------------------
              sigma_u |  .01003397
              sigma_e |  .04272026
                  rho |  .05228251   (fraction of variance due to u_i)
---------------------------------------------------------------------------------------

[Clyde Schechter]

Well, I'm glad you found the data problems and got -xtlogit- to run.

You should not expect linear regression and logistic regression coefficients to resemble each other at all. They are not only different models, but they are modeling the outcome on different metrics.

That said, there are some things that disturb me about both models.

The -xtreg- results have a within-R² of only 0.0282. That is, your model explains hardly any of the within-worker variation in Y. Because of the gargantuan sample size, your effects are "statistically significant," but I think statistical significance becomes utterly meaningless in samples this large. Just the fact that a linear model is almost certainly not the real data generating process is likely to create statistically significant findings when the sample size is this large. You'd really have to work hard at finding data that didn't give you statistically significant results! Similarly, the coefficients themselves are really tiny. In the untreated group, the transition from pre- to post is associated with a reduction in probability of Y of only 0.00086, and in the treated group it is an increase of only 0.0043277; both are tiny fractions of a percentage point changes. Can these possibly be meaningful?

As for the -xtlogit- results, the problem is the opposite. The odds ratios you are getting seem implausibly large. At least in my world (epidemiology) it is exceedingly rare to find any real-world effect of a dichotomous variable that has an odds ratio > 4 (or < 1/4). Your odds ratio for treat is > 100 and would be utterly without precedent, or even anything close to it, in my area. Generally if we see extremely large odds ratios like this it is either due to there being only a handful of observations having treat = 1 and those happen, by chance, to be observations with Y = 1 whereas most observations have Y = 0. Alternatively, this can be seen if the overall probability of Y is already very close to 1: at that point, a very tiny increment in probability will correspond to a huge odds ratio. For example if the probability of Y is .9990 in the untreated group and .9999 in the treated group, then despite the minuscule difference in probability, the odds ratio is just a tad over 10. At the other end of your output, the month adjustments have extremely small odds ratios. These may, in effect, be "canceling out" the very high odds ratios associated with treat and post, so it's very hard to see.

There is something else that is disturbing here. When you go from -xtreg, fe- to -xtlogit- fe- you are losing the vast majority of the sample (both in terms of total observations and in terms of distinct worker_id's.) Now, these are due to omissions of any worker_id for whom the value of Y is constant across his/her observations. See where it says

note: 132,783 groups (11,042,781 obs) dropped because of all positive or
      all negative outcomes.

This finding also suggests that there is very little variation in your outcome Y and you are straining to estimate very tiny effects on something that is nearly a universal constant.

For the moment, put aside the fancy models. What does -table treat post, c(mean Y)- show?

[Luca As]

Thanks!

Well, what you are saying is perfectly correct. I need to be more precise on my output here: my Y is equal to one when a worker changes his contractual type and zero otherwise. I am doing the same also with the probability of finding a job – zero when unemployed and one when employed – and results are very similar.

As one can imagine, both the events – changing a contractual type or finding a job – are quite rare. So, yes, most values of Y are equal to zero:

tab Y
 Y          |      Freq.     Percent        Cum.
------------+-----------------------------------
          0 | 13,426,773       99.81       99.81
          1 |     25,345        0.19      100.00
------------+-----------------------------------
      Total | 13,452,118      100.00

Looking at the previous -xtlogit- model, also in may opinion the odd ratio linked to the treated group is extremely large – I will think about the way I defined the treated group again. On the other hand, the odd ratio linked to the DID dummy seems reasonable: "the odds of changing contract for the treated group is 2.5 times that of the control group" – I hope this is the correct interpretation.

As for the table:

table treat post, c(mean Y)

------------------------------
treat     |          post      
          |        0         1
----------+-------------------
        0 | .0001116  .0001412
        1 | .0074528  .0109297
------------------------------

[Clyde Schechter]

So, yes the -table- output does seem compatible with both the -xtreg- and -xtlogit- outputs. What you have is analysis of a very rare event. There are small variations from pre- to post. There are also small absolute differences between untreated and treated, but as ratios the same numbers give a very large result. So it seems as if both models are capturing what we see in the simple cross-tab analysis.

[Steffen Mauch]

Your output does not look right. If your treat variable is truly set to 0 for those who were not eligible for the policy and 1 for those who were, it should be colinear with the worker_id fixed effects and Stata should omit it.

Clyde Schechter Does this mean that if the treatment criterion is not varying over time - as it is the case here (see #3) - but is assigned only once and constant thereafter, it is never possible to get an estimate on the -treat- variable in a fixed effects panel DD design?

[Clyde Schechter]

Does this mean that if the treatment criterion is not varying over time - as it is the case here (see #3) - but is assigned only once and constant thereafter, it is never possible to get an estimate on the -treat- variable in a fixed effects panel DD design?

Yes, that is correct. In the fixed-effects model you can never get an effect estimate for the treat variable. But that's of no importance at all. For example, if you were to use a random effects model instead of fixed, effects, you could get an estimate of the treat variable effect--but you would ignore it because it is more or less irrelevant anyway.

What you are interested in when doing a DD analysis is not the effect of the treat variable but the effect of the treatment itself. And that is given by the coefficient of the treatment-time interaction term, not the treat variable. One of the things that makes interaction models difficult to learn to use is that the variables no longer mean what their names suggest.

[Steffen Mauch]

Ok, thank you very much for the explanation.

[Steffen Mauch]

Clyde Schechter Dear Mr. Schechter,

I'm sorry to bother you once again but re-reading this post another uncertainty came up (btw you seem to be the person answering all the post which are relevant for me, so thank you for patiently answering all the my questions).

2) If the predicted outcomes are staying away from 0 and 1, then a linear probability model (which is what you are getting from -xtreg-) is perfectly OK. If you are getting predicted values near 0 or 1, or even negative and greater than 1, then, you may not like this model.

How can I interpret

predicted values near 0 or 1

? To make this more plastic, in my DD Analysis I get the following output:

. reghdfe anyMMactivity_consistent i.pre_post##i.treatment_2, noabsorb vce(cluster tc_name)
(MWFE estimator converged in 1 iterations)

HDFE Linear regression                            Number of obs   =      4,152
Absorbing 1 HDFE group                            F(   3,    143) =     348.74
Statistics robust to heteroskedasticity           Prob > F        =     0.0000
                                                  R-squared       =     0.2034
                                                  Adj R-squared   =     0.2028
                                                  Within R-sq.    =     0.2034
Number of clusters (tc_name) =        144         Root MSE        =     0.4144

                                      (Std. Err. adjusted for 144 clusters in tc_name)
--------------------------------------------------------------------------------------
                     |               Robust
anyMMactivity_cons~t |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
---------------------+----------------------------------------------------------------
          1.pre_post |   .3872571   .0246981    15.68   0.000     .3384365    .4360777
       1.treatment_2 |   .0345634   .0215728     1.60   0.111    -.0080794    .0772061
                     |
pre_post#treatment_2 |
                1 1  |   .0368784   .0272733     1.35   0.178    -.0170326    .0907893
                     |
               _cons |   .4657895   .0220047    21.17   0.000      .422293     .509286
--------------------------------------------------------------------------------------

If I add all these estimates to get the predicted value, i.e. outcome = 1*_cons + 1*pre_post + 1*treatment_2 + 1*pre_post#treatment_2 = .9244884, the resulting value is arguably close to 1. Am I right in my assumption that this result would still not imply to use a xtlogit model?

Should this conclusion change if I add time and individual fixed effects and I'm only getting a result for the DD estimate, i.e. in the situation depicted below?

. reghdfe anyMMactivity_consistent i.pre_post##i.treatment_2, absorb(i.id i.wave) vce(cluster tc_name) //Include tc fixed effects?
(dropped 202 singleton observations)
note: 1bn.pre_post is probably collinear with the fixed effects (all partialled-out values are close to zero; tol = 1.0e-09)
note: 1bn.treatment_2 is probably collinear with the fixed effects (all partialled-out values are close to zero; tol = 1.0e-09)
(MWFE estimator converged in 2 iterations)
note: 1.pre_post omitted because of collinearity
note: 1.treatment_2 omitted because of collinearity

HDFE Linear regression                            Number of obs   =      3,950
Absorbing 2 HDFE groups                           F(   1,    143) =       2.79
Statistics robust to heteroskedasticity           Prob > F        =     0.0973
                                                  R-squared       =     0.6431
                                                  Adj R-squared   =     0.2856
                                                  Within R-sq.    =     0.0034
Number of clusters (tc_name) =        144         Root MSE        =     0.3865

                                      (Std. Err. adjusted for 144 clusters in tc_name)
--------------------------------------------------------------------------------------
                     |               Robust
anyMMactivity_cons~t |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
---------------------+----------------------------------------------------------------
          1.pre_post |          0  (omitted)
       1.treatment_2 |          0  (omitted)
                     |
pre_post#treatment_2 |
                1 1  |   .0710889    .042591     1.67   0.097    -.0131004    .1552782
                     |
               _cons |   .6767765   .0153866    43.98   0.000     .6463619    .7071911
--------------------------------------------------------------------------------------

Absorbed degrees of freedom:
-----------------------------------------------------+
 Absorbed FE | Categories  - Redundant  = Num. Coefs |
-------------+---------------------------------------|
          id |      1975           0        1975     |
        wave |         2           1           1     |
-----------------------------------------------------+

[Clyde Schechter]

I'm partial to linear probability models, so I would be comfortable with one when the predicted probabilities are as you describe them.

I would not say it implies "not" to use an -xtlogit- model. -xtlogit- models are allowable and sometimes describe the data better. I think you could work with either model this way. You might try doing both and looking at how well each fits the data.

[Jared Greathouse]

I'm currently a Ph.D student who's taking my second methods course, and I'm almost not kidding when I say the first half of the semester was too demonstrate that when we're above .2 and less than .8 on the S curve, LPM and logit are PRETTY MUCH the same.

Not totally- the results and coefficients are different, with different interpretations and implications, but if I was doing this, I'd estimate both and then graph their predicted probabilities or otherwise report my effect sizes in an graphical manner to show the impact.

[Steffen Mauch]

Thank you very much for your answer, Mr. Schechter!

I hope it is not too embarassing that I have to ask how such a model comparison would be conducted. I posted the models in question below. For the xtreg model I would consult the adjusted R^2 in order to assess the fit of the model. However, what is to be done with the xtlogit model? Is there something like a Pseudo R^2 (as in the cross-sectional case) and how can these two different models be compared?

. reghdfe anyMMactivity_consistent i.pre_post##i.treatment_4, absorb(i.id i.wave) vce(cluster tc_name)
(dropped 202 singleton observations)
note: 1bn.pre_post is probably collinear with the fixed effects (all partialled-out values are close to zero; tol = 1.0e-09)
note: 1bn.treatment_4 is probably collinear with the fixed effects (all partialled-out values are close to zero; tol = 1.0e-09)
(MWFE estimator converged in 2 iterations)
note: 1.pre_post omitted because of collinearity
note: 1.treatment_4 omitted because of collinearity

HDFE Linear regression                            Number of obs   =      3,950
Absorbing 2 HDFE groups                           F(   1,    143) =       5.14
Statistics robust to heteroskedasticity           Prob > F        =     0.0249
                                                  R-squared       =     0.6437
                                                  Adj R-squared   =     0.2868
                                                  Within R-sq.    =     0.0051
Number of clusters (tc_name) =        144         Root MSE        =     0.3862

                                      (Std. Err. adjusted for 144 clusters in tc_name)
--------------------------------------------------------------------------------------
                     |               Robust
anyMMactivity_cons~t |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
---------------------+----------------------------------------------------------------
          1.pre_post |          0  (omitted)
       1.treatment_4 |          0  (omitted)
                     |
pre_post#treatment_4 |
                1 1  |   .0782339   .0345095     2.27   0.025     .0100193    .1464485
                     |
               _cons |   .6828487   .0086604    78.85   0.000     .6657299    .6999676
--------------------------------------------------------------------------------------

Absorbed degrees of freedom:
-----------------------------------------------------+
 Absorbed FE | Categories  - Redundant  = Num. Coefs |
-------------+---------------------------------------|
          id |      1975           0        1975     |
        wave |         2           1           1     |
-----------------------------------------------------+


. xtlogit anyMMactivity_consistent i.pre_post##i.treatment_2 i.wave, fe
note: 2.wave omitted because of collinearity
note: multiple positive outcomes within groups encountered.
note: 1,263 groups (2,324 obs) dropped because of all positive or
      all negative outcomes.
note: 1.treatment_2 omitted because of no within-group variance.

Iteration 0:   log likelihood = -310.95627  
Iteration 1:   log likelihood = -219.43408  
Iteration 2:   log likelihood = -210.98249  
Iteration 3:   log likelihood = -210.90257  
Iteration 4:   log likelihood = -210.90249  
Iteration 5:   log likelihood = -210.90249  

Conditional fixed-effects logistic regression   Number of obs     =      1,828
Group variable: id                              Number of groups  =        914

                                                Obs per group:
                                                              min =          2
                                                              avg =        2.0
                                                              max =          2

                                                LR chi2(2)        =     845.27
Log likelihood  = -210.90249                    Prob > chi2       =     0.0000

------------------------------------------------------------------------------------------
anyMMactivity_consistent |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------------------+----------------------------------------------------------------
              1.pre_post |   2.115809   .2037187    10.39   0.000     1.716528    2.515091
           1.treatment_2 |          0  (omitted)
                         |
    pre_post#treatment_2 |
                    1 1  |    .899093   .2744844     3.28   0.001     .3611135    1.437072
                         |
                  2.wave |          0  (omitted)
------------------------------------------------------------------------------------------