Using a Difference-in-Difference approach with a logit model

[Huda Azfar]

I am trying to run a simple DID model with a binary dependent variable, i.e. a logit model. Is this an approach that is possible? Simply running a regression as follows:

logit treat time did

[Clyde Schechter]

Well, what you write could never be a correct analysis because you have treat as the outcome variable. I assume you meant -logit outcome treat time did-. Actually, assuming you have a classical DID set up, where the treatment begins at the same time in all treated entities, you can simplify the coding by using factor variable notation: - logit outcome i.treat##i.time-.

But it may not be appropriate to use -logit-. This depends on your data. If you have longitudinal (panel) data, where the same people (or firms, or states, or whatever your units of analysis are) are followed over time, then you need a panel-data analysis.

xtset unit_of_analysis time
xtlogit outcome i.treat##i.time, fe

If, however, what you have a a series of cross sections, some of which precede the time when treatment began for the treated units, and others follow, but they do not involve the same units, then just -logit outcome i.treat##i.time- is fine.

Now, that's in terms of the bare bones model. You may want to include some covariates that are relevant. And you may want to use cluster robust standard errors if you are using longitudinal (panel) data.

[Huda Azfar]

Sorry, I did mean -logit outcome treat time did-....

I set up the panel as follows,

xtset personid int_year, yearly

(where person is the individual's unique identifier, and int_year shows the interview year, such that this command would set up the panel to show the personid over time (observation of the independent variables for eat year individual was interviewed)).

Not much variation within the individuals characteristics is anticipated, is it then better to do just a DID without fixed effects?
And would this simply be the code for that?:

logit dep i.treat##i.time

(where "dep" is the outcome binary variable)

[Clyde Schechter]

If you have panel data you must use a panel estimator, at least initially. So, no, it is not OK to just do it with -logit-. Start with -xtlogit, fe-. Then look at the last part of the output. If sigma_u is close to zero, and rho is close to 0, then that says that the fixed effects are not actually accounting for much variation and you can relax and go back to using -logit-. But you can't reach this conclusion without first seeing that -xtlogit, fe- output.

[Huda Azfar]

Thanks a bunch Clyde! I understand now, on a conceptual level
The output I receive though doesn't include sigma_u or rho - have I inputted the code incorrectly perhaps?

 xtset personid int_year, yearly 
       panel variable:  personid (unbalanced)
        time variable:  int_year, 2004 to 2011, but with gaps
                delta:  1 year

xtlogit dep i.treat##i.time, fe

And this is what it churns out

note: multiple positive outcomes within groups encountered.
note: 1,638 groups (2,950 obs) dropped because of all positive or
      all negative outcomes.

Iteration 0:   log likelihood = -104.66559  
Iteration 1:   log likelihood = -104.52719  
Iteration 2:   log likelihood = -104.52643  
Iteration 3:   log likelihood = -104.52643  

Conditional fixed-effects logistic regression   Number of obs     =        478
Group variable: personid                        Number of groups  =        202

                                                Obs per group:
                                                              min =          2
                                                              avg =        2.4
                                                              max =          3

                                                LR chi2(3)        =     130.99
Log likelihood  = -104.52643                    Prob > chi2       =     0.0000

------------------------------------------------------------------------------
         dep |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
     1.treat |   .3432175   .8406951     0.41   0.683    -1.304515     1.99095
      1.time |   .3427126   .5699821     0.60   0.548    -.7744317    1.459857
             |
  treat#time |
        1 1  |   2.035563   .8818109     2.31   0.021     .3072454     3.76388
------------------------------------------------------------------------------

[Clyde Schechter]

You're right. Sorry. -xtlogit, fe- conditions out the fixed effects, so it doesn't estimate their variance. There is really no escaping the need to use -xtlogit- over -logit-. With -xtlogit, re- you get a sigma_u and rho (which is what I was thinking of). So you may want to look into random effects regression and then see if the sigma_u and rho are small enough to make -logit- an acceptable alternative.

[Huda Azfar]

Right, so I did perform an RE and the sigma_u and rho and not close to zero, indicating the need to stick to an RE framework..

However, I conducted a Hausman test (it should be valid for an xtlogit stop, right?), with a P-value < 0.05 it appears I have to indeed go with the Fixed Effects though? Or is it an incorrect test to have performed?

hausman fe re 

                 ---- Coefficients ----
             |      (b)          (B)            (b-B)     sqrt(diag(V_b-V_B))
             |       fe           re         Difference          S.E.
-------------+----------------------------------------------------------------
     1.treat |    .3432175     .4256876       -.0824701        .6297125
      1.time |    .3427126    -.5866406        .9293532        .4353551
  treat#time |
        1 1  |    2.035563     2.302691       -.2671278        .6485319
------------------------------------------------------------------------------
                         b = consistent under Ho and Ha; obtained from xtlogit
          B = inconsistent under Ha, efficient under Ho; obtained from xtlogit

    Test:  Ho:  difference in coefficients not systematic

                  chi2(3) = (b-B)'[(V_b-V_B)^(-1)](b-B)
                          =        9.39
                Prob>chi2 =      0.0245

[Clyde Schechter]

Right, so I did perform an RE and the sigma_u and rho and not close to zero, indicating the need to stick to an RE framework..

So, yes that would rule out using just -logit-. By itself, it says nothing about choosing between FE and RE.

Your Hausman test is one approach to that. And if you belong to the school of thought that uses Hausman tests to choose between FE and RE estimators, your results do indicate that you should use FE. Personally, I do not usually rely on statistical tests to select models in this way, but looking at the Hausman output, I, too, would recommend using FE over RE here. I say that because the differences between the FE and RE estimates of both the 1.time and 1.treat#1.time coefficients are very different. For 1.time the difference is actually larger in magnitude than either estimate, and the signs are different. For the interaction term (which is the main focus of your analysis) it is larger in magnitude than 10% of the FE estimate itself--a large enough difference to matter for practical purposes in most settings. So, yes, go with FE.

[Huda Azfar]

Right! I see. Thanks a million for the confirmation!

And 3 final confirmations:

1. Can these also be used after an -xtlogit, fe- ?

margins time#treat

margins treat, dydx(time)

2. If time and treat are insignificant and time*treat is significant, clear case of policy having a impact right? Coefficient significance for Time and Treat by themselves is not necessary for practical reasons, they exist just by construction in such a DID framework

3. There is no sigma_u or rho in a FE when using panel data, that would exist for a cross-section analysis rather.

[Clyde Schechter]

1. You cannot use -margins time#treat- after -xtlogit, fe-. If you try, it will tell you that the parameters are not estimable, and indeed they are not. As for -margins treat, dydx(time)-, Stata will also say this is not-estimable, but in this case Stata is wrong and if you try it as -margins treat, dydx(time) noestimcheck- you will get it.

(You might wonder what would happen if you tried the -noestimcheck- trick with -margins time#treat-. The answer is that you will get results, but the results are misleading garbage.)

2. The coefficient of the treat#time interaction is the DID estimator of the effect. The output also gives you its standard error, 95% confidence intervals, and a statistical test. The American Statistical Association recommend abandoning the use of the concept of "statistical significance." See https://www.tandfonline.com/doi/full...5.2019.1583913. Better to focus on the magnitude of the effect estimate, how precise the estimate is (as reflected in the standard error or confidence intervals) and whether the effect is of practical importance both as estimated and over the range of the confidence interval.

In any case, even if we were still going to work with the concept of statistical significance, the significance of the coefficients of time and treat, by themselves, are not relevant, as you note.

3. Correct that there is no sigma_u or rho in -xtlogit, fe- output, and, in fact, those parameters are not estimable in the FE logit model. In a cross sectional analysis, there would also be no sigma_u or rho in the output, but that output would not come from -xtlogit, fe- in the first place, and the sigma_u and rho parameters would not even exist. In a random effects analysis of panel data, you do get estimates of sigma_u and rho.

[Huda Azfar]

Wonderful, I believe you have answered all my questions on this one, it was a great help. Thank you!!

[Steffen Mauch]

Clyde Schechter Dear Mr. Schechter,

I have a follow-up question on what you write in #10:

1. You cannot use -margins time#treat- after -xtlogit, fe-. If you try, it will tell you that the parameters are not estimable, and indeed they are not. As for -margins treat, dydx(time)-, Stata will also say this is not-estimable, but in this case Stata is wrong and if you try it as -margins treat, dydx(time) noestimcheck- you will get it.

(You might wonder what would happen if you tried the -noestimcheck- trick with -margins time#treat-. The answer is that you will get results, but the results are misleading garbage.)

To my understanding that implies that it is impossible to obtain an estimate on the size of the DD estimator in a -xtlogit- setup. Rather, one can only say something about the direction and the significance of the effect. Did I understand you correctly?

[Clyde Schechter]

Yes, you understood me correctly.

[Steffen Mauch]

Thank your for your answer. Now I understand why you are partial towards a linear probability model in a DD design with binary outcome. A DD analysis without a final result on the size of the DD estimator, appears pretty useless to me...