Margins after reghdfe?

[Charlotte Webber]

Hello Statalisters,.

I am running a triple differences linear probability regression, where I am evaluating the effects of a state-level insurance policy change (i.policy -- ie, the treated group), that could have been implemented anytime after 2010 (i.post -- i.e., TWFE not DID/DDD), that we are theorizing had differential effects on two different age-groups of people (i.age). Our outcome is binary (i.screened -- i.e.,screened for cancer). We have a large national data set.

My objective is to get not only the "DDD" coefficient, but also the predicted screening rates for the each of the two age groups both pre- and post- the policy change in treated vs control states, however I am both confused as to the proper margins command to be using and also confused about margins output I am getting.

The regression I am using is:

reghdfe screen i.age##i.post#i.policy ib2.RACE i.EDUC, absorb(state year) vce(cluster state) pool(1)
(MWFE estimator converged in 4 iterations)
note: 1.age#1.post omitted because of collinearity
note: 1.post#0b.policy omitted because of collinearity
note: 1.post#1.policy omitted because of collinearity

HDFE Linear regression                            Number of obs   =  9,130,961
Absorbing 2 HDFE groups                           F(  11,     36) =     214.87
Statistics robust to heteroskedasticity           Prob > F        =     0.0000
                                                  R-squared       =     0.0329
                                                  Adj R-squared   =     0.0329
                                                  Within R-sq.    =     0.0111
Number of clusters (state_num) =         37       Root MSE        =     0.4595

                                            (Std. Err. adjusted for 37 clusters in state_num)
---------------------------------------------------------------------------------------------
                            |               Robust
                     screen |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
----------------------------+----------------------------------------------------------------
                 age#policy |
                 younger#1  |  -.0208939   .0221751    -0.94   0.352     -.065867    .0240792
                   older#0  |   .0778104   .0140845     5.52   0.000     .0492457    .1063752
                   older#1  |          0  (omitted)
                            |
                post#policy |
                       0 1  |   .0101914     .00729     1.40   0.171    -.0045934    .0249762
                       1 0  |          0  (omitted)
                       1 1  |          0  (omitted)
                            |
            age#post#policy |
                 older#1#0  |  -.0032618   .0067761    -0.48   0.633    -.0170043    .0104808
                 older#1#1  |   .0129391    .007872     1.64   0.109     -.003026    .0289041
                            |
                       RACE |
                  Hispanic  |    .018485   .0070756     2.61   0.013     .0041351     .032835
                 Black, NH  |   .0871647    .007065    12.34   0.000     .0728363    .1014932
                 Asian, NH  |   .0334286   .0105761     3.16   0.003     .0119793    .0548778
                            |
                  EDUCATION |
                        HS  |  -.0457087   .0028512   -16.03   0.000    -.0514911   -.0399263
                  Some Col  |  -.0610266   .0045289   -13.47   0.000    -.0702116   -.0518415
                 Bach/Grad  |  -.0554348   .0064723    -8.56   0.000    -.0685612   -.0423083
                            |
                      _cons |   .2794276   .0089781    31.12   0.000     .2612192     .297636
---------------------------------------------------------------------------------------------

Absorbed degrees of freedom:
-----------------------------------------------------+
 Absorbed FE | Categories  - Redundant  = Num. Coefs |
-------------+---------------------------------------|
   state_num |        37          37           0    *|
        year |         9           1           8     |
-----------------------------------------------------+
* = FE nested within cluster; treated as redundant for DoF computation

When I run margins, I get "not estimable". I am assuming this is because my policy variable is correlated with the state FEs, and that post is correlated with the year FEs, so I thought it would still be kosher to run the margins with -noestimcheck option, however the following happens:

margins i.age##i.post##policy, atmeans noestimcheck

Adjusted predictions                            Number of obs     =  9,130,961
Model VCE    : Robust

Expression   : Linear prediction, predict()
at           : 0.age          =     .681696 (mean)
               1.age          =     .318304 (mean)
               0.policy       =    .4287486 (mean)
               1.policy       =    .5712514 (mean)
               0.post         =    .4573222 (mean)
               1.post         =    .5426778 (mean)
               1.RACE         =     .393269 (mean)
               2.RACE         =    .4001108 (mean)
               3.RACE         =    .1699526 (mean)
               4.RACE         =    .0366676 (mean)
               2.EDUCATION    =    .3711704 (mean)
               3.EDUCATION    =    .3085791 (mean)
               4.EDUCATION    =    .1976261 (mean)
               5.EDUCATION    =    .0968208 (mean)

---------------------------------------------------------------------------------------------
                            |            Delta-method
                            |     Margin   Std. Err.      z    P>|z|     [95% Conf. Interval]
----------------------------+----------------------------------------------------------------
                        age |
                   younger  |   .3067197   .0035853    85.55   0.000     .2996927    .3137467
                     older  |   .3552687   .0073577    48.29   0.000     .3408478    .3696897
                            |
                       post |
                         0  |   .3242972   .0025805   125.67   0.000     .3192396    .3293549
                         1  |    .320383   .0023248   137.81   0.000     .3158265    .3249395
                            |
                   age#post |
                 younger#0  |   .3098791   .0040138    77.20   0.000     .3020122     .317746
                 younger#1  |   .3040572   .0042475    71.59   0.000     .2957324    .3123821
                   older#0  |   .3551759   .0100741    35.26   0.000      .335431    .3749208
                   older#1  |    .355347   .0063635    55.84   0.000     .3428747    .3678193
                            |
                     policy |
                         0  |   .3401969    .008783    38.73   0.000     .3229825    .3574113
                         1  |   .3086454   .0067858    45.48   0.000     .2953455    .3219453
                            |
                 age#policy |
                 younger#0  |   .3159929    .009111    34.68   0.000     .2981357    .3338502
                 younger#1  |   .2997598   .0126543    23.69   0.000     .2749578    .3245617
                   older#0  |   .3920333   .0121487    32.27   0.000     .3682222    .4158443
                   older#1  |   .3276754   .0072428    45.24   0.000     .3134799    .3418709
                            |
                post#policy |
                       0 0  |   .3407603   .0089327    38.15   0.000     .3232524    .3582681
                       0 1  |    .311941    .007286    42.81   0.000     .2976608    .3262213
                       1 0  |   .3397221   .0087769    38.71   0.000     .3225197    .3569244
                       1 1  |   .3058682   .0085404    35.81   0.000     .2891293     .322607
                            |
            age#post#policy |
               younger#0#0  |   .3159929    .009111    34.68   0.000     .2981357    .3338502
               younger#0#1  |   .3052904   .0128555    23.75   0.000     .2800941    .3304868
               younger#1#0  |   .3159929    .009111    34.68   0.000     .2981357    .3338502
               younger#1#1  |    .295099   .0134205    21.99   0.000     .2687954    .3214026
                 older#0#0  |   .3938034   .0144269    27.30   0.000     .3655271    .4220796
                 older#0#1  |   .3261843   .0121676    26.81   0.000     .3023363    .3500323
                 older#1#0  |   .3905416   .0108428    36.02   0.000       .36929    .4117931
                 older#1#1  |    .328932   .0065028    50.58   0.000     .3161868    .3416772
---------------------------------------------------------------------------------------------

I have a few questions/confusions:
(1) Is it ok to be using -noestimcheck in this case?

(2) If it is (or regardless), why am I getting identical predictions for younger#0 (ie, younger age group, in non-treated states) as for the triple interaction predictions of younger#0#0 (ie, younger, pre-policy, in non-treated states) and younger#1#0 (ie, younger, post-policy, non-treated states)? I guess I was expecting that even if my regession coefficients were not statistically significant (which the DDD coefficient is not), these numbers still shouldn't be exactly the same.

(3) Am I using the corrent margins command? It confuses me that Stata puts up all the "at" means of all of the variables, when (I think) I am asking it to evaluate the predicted probabilities at specific values of variables (or at least my 3 interaction variables).

I have read a lot on this listserv and other places, but still couldn't answer these questions -- any insights greatly appreciated!

[Clyde Schechter]

1) No, it isn't OK to use -noestimcheck- here. If you don't believe me, do an experiment: using the ib?. factor-variable notation, change the base (omitted) categories of your key variables, and you will see that the -margins- estimates change too. That means that those margins have no referent in the real world: they are artifacts of the way choose to parameterize the model. They are estimates of nothing.

It is often permissible to override the estimability issue with -noestimcheck- when you are estimating marginal effects, not the predictive margins you estimated. In fact, based on your statement of the problem, even if you could get those predictive margins, they aren't what you need. What you need are the marginal effects.

2) I don't know. Probably if I invested enough time in it I could figure it out, but since those are estimates of nothing meaningful anyway, it's not worth the effort of discovering why two useless statistics turn out to be equal.

3) When you use -atmeans-, Stata routinely prints out the mean values for all the variables in the model. But your concern that Stata is using the means of policy, age, and post is not true: those means are not being used. The values of policy, age, and post are being used directly. That said, this is still not the correct -margins- command for the reason noted in my response to 1).

Here's what I would do. While I normally counsel people not to create home-brew interaction terms and just rely on factor variable notation to do it for them, the one situation where that is bad advice is when we are doing a generalized difference-in-differences model (i.e. the policy is implemented at different times in different states. Here it is more convenient to make a single variable representing policy#post: call it something like policy_in_effect. Doing this simplifies the modeling and the interpretation (because in this kind of model the effects of policy and post, separately, have a complicated meaning that doesn't relate to anything concrete in the real world, and they are pretty much never of any interest).

reghdfe screen i.age##i.policy_in_effect ib2.RACE i.EDUC, absorb(state year) vce(cluster state) pool(1)
margins age, dydx(policy_in_effect) atmeans noestimcheck

will give you the triple difference estimate of the causal effect of the policy on screen, separately in each age group. If you are interested in whether the policy effects differ between the two age groups, you get that from the coefficient of older#1.policy_in_effect in the -reghdfe- output, not from the -margins- output.

[Charlotte Webber]

Hi Clyde, thank you, this is incredibly helpful. I am playing around and re-running the regressions as you suggested and will post back with any results asap.

In the meantime, I wonder if you could briefly explain *why* it's not ok to use -noestimcheck? I believe you : ), I just don't understand why the predictive margins are garbage while the marginal effects are valid, given that (I thought) the marginal effects are just differences in predictive margins/probabilities. Thanks again.

[Clyde Schechter]

I just don't understand why the predictive margins are garbage while the marginal effects are valid, given that (I thought) the marginal effects are just differences in predictive margins/probabilities.

Well, actually you think correctly, and that is precisely why it's OK for marginal effects but not predictive margins.

The underlying problem that triggers the (not estimable) output is usually colinearity among variables. If you think about the typical situation of a multi-level category variable represented by indicators ("dummy variables") using factor variable notation, those indicators are necessarily colinear with the constant term. So, to identify the model, either the constant term or one (reference) category of the variable must be omitted from the model. Omitting one reference category is the way it is usually done. And what do those coefficients represent? They represent the differences between their categories and the omitted category. Even though the full set of indicators cannot produce a valid model of the levels, they do produce a valid model of differences. If you want to find the difference between two non-omitted categories, you can use -lincom- to calculate that difference and get a valid answer, because differences between coefficients of colinear variables are identifiable and are the same regardless of which level of the category variable is chosen as the omitted reference category. This is the general principle for any group of colinear variables: coefficients of individual variables are artifacts of the choice of omitted category, but differences of coefficients are invariant.

When you calculate marginal effects you are looking at differences across categories--so these are well defined, and independent of the choice of reference category, even if the levels at the categories aren't. The problem is that -margins- investigates the colinearity, and when it finds it, it throws up its hands and says (not estimable) rather than figuring out whether the particular things being asked for might be invariant under change of reference category. (I don't blame it for doing that--it's a hard problem.)

[Charlotte Webber]

Ok, I've had a little time to play and unfortunately, I'm more confused, though it's no longer about the margins. I'm going to back up and show you some prior regessions:

Our hypothesis was that, post-policy, the older age-group would be doing better compared to the younger age-group in states that adopted the insurance policy (but not in the states that didn't adopt). Initially, to check our hypothesis, we ran DIDs (age##post), stratified by adopters/non-adoptors. For this, we used just post-2010 (ie, a true DID, not generalized DID, since *most* states that adopted did so in 2010). We got the following results:

(1) Regression just for adopters/treated states

reghdfe screen i.age##i.post `control_vars_allStates' if policy==1, absorb(state_num year) vce(cluster state_num)
(MWFE estimator converged in 4 iterations)
note: 1bn.post is probably collinear with the fixed effects (all partialled-out values are close to zero; tol = 1.0e-09)

HDFE Linear regression                            Number of obs   =  4,356,230
Absorbing 2 HDFE groups                           F(  13,     21) =    1648.05
Statistics robust to heteroskedasticity           Prob > F        =     0.0000
                                                  R-squared       =     0.0587
                                                  Adj R-squared   =     0.0587
                                                  Within R-sq.    =     0.0475
Number of clusters (state_num) =         22       Root MSE        =     0.3920

                                   (Std. Err. adjusted for 22 clusters in state_num)
------------------------------------------------------------------------------------
                   |               Robust
            screen |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------------+----------------------------------------------------------------
               age |
            older  |   .0343195   .0177587     1.93   0.067    -.0026118    .0712508
            1.post |          0  (omitted)
                   |
          age#post |
          older#1  |   .0127438   .0062995     2.02   0.056    -.0003567    .0258443
                   |
              RACE |
        White, NH  |  -.0128497   .0052539    -2.45   0.023    -.0237757   -.0019238
        Black, NH  |   .0843454   .0049176    17.15   0.000     .0741188     .094572
        Asian, NH  |   .0142412   .0124185     1.15   0.264    -.0115846     .040067
                   |
                   |
         EDUCATION |
               HS  |  -.0837713    .018153    -4.61   0.000    -.1215227     -.04602
     SOME COLLEGE  |   -.130703   .0231313    -5.65   0.000    -.1788072   -.0825989
BACHELOR/GRADUATE  |  -.1887754    .026167    -7.21   0.000    -.2431927   -.1343582
                   |
             _cons |   .3435236   .0147295    23.32   0.000     .3128919    .3741553
------------------------------------------------------------------------------------

(2) Regression just for non-adopters/control states:

reghdfe screen i.age##i.post `control_vars_allStates' if policy==0, absorb(state_num year) vce(cluster state_num)
(MWFE estimator converged in 4 iterations)
note: 1bn.post is probably collinear with the fixed effects (all partialled-out values are close to zero; tol = 1.0e-09)

HDFE Linear regression                            Number of obs   =    106,818
Absorbing 2 HDFE groups                           F(  8,     14) =     4545.79
Statistics robust to heteroskedasticity           Prob > F        =     0.0000
                                                  R-squared       =     0.0709
                                                  Adj R-squared   =     0.0709
                                                  Within R-sq.    =     0.0592
Number of clusters (state_num) =         15       Root MSE        =     0.4230

                                   (Std. Err. adjusted for 15 clusters in state_num)
------------------------------------------------------------------------------------
                   |               Robust
            screen |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------------+----------------------------------------------------------------
               age |
            older  |   .0644621     .01235     5.22   0.000     .0379739    .0909503
            1.post |          0  (omitted)
                   |
          age#post |
          older#1  |   .0104903   .0066171     1.59   0.135     -.003702    .0246826
                   |
              RACE |
        White, NH  |  -.0085867   .0095678    -0.90   0.385    -.0291077    .0119343
        Black, NH  |   .0914607   .0075749    12.07   0.000      .075214    .1077073
        Asian, NH  |   .0119648   .0111102     1.08   0.300    -.0118642    .0357939
                   |
         EDUCATION |
               HS  |  -.1019737   .0125699    -8.11   0.000    -.1289334   -.0750139
     SOME COLLEGE  |  -.1597237   .0127239   -12.55   0.000    -.1870138   -.1324336
BACHELOR/GRADUATE  |  -.2295448   .0121472   -18.90   0.000    -.2555978   -.2034917
                   |
             _cons |    .399562   .0060886    65.62   0.000     .3865034    .4126207
------------------------------------------------------------------------------------

So far, these results matched our priors, since the coefficient for the older age group post-poilcy was bigger in the treated states (.0127438) than control states (.0104903).

So then, (3), we ran the DDD:

  
reghdfe screen i.age##i.post##i.policy `control_vars_allStates', absorb(state_num year) vce(cluster state_num)
(MWFE estimator converged in 4 iterations)
note: 1bn.post is probably collinear with the fixed effects (all partialled-out values are close to zero; tol = 1.0e-09)
note: 1bn.policy is probably collinear with the fixed effects (all partialled-out values are close to zero; tol = 1.0e-09)

HDFE Linear regression                            Number of obs   =  4,463,048
Absorbing 2 HDFE groups                           F(  14,     36) =     194.80
Statistics robust to heteroskedasticity           Prob > F        =     0.0000
                                                  R-squared       =     0.0677
                                                  Adj R-squared   =     0.0677
                                                  Within R-sq.    =     0.0524
Number of clusters (state_num) =         37       Root MSE        =     0.4052

                                            (Std. Err. adjusted for 37 clusters in state_num)
---------------------------------------------------------------------------------------------
                            |               Robust
                     screen |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
----------------------------+----------------------------------------------------------------
                        age |
                     older  |   .0670305   .0135282     4.95   0.000      .039594    .0944671
                     1.post |          0  (omitted)
                            |
                   age#post |
                   older#1  |   .0092792   .0067143     1.38   0.175    -.0043381    .0228964
                            |
                   1.policy |          0  (omitted)
                            |
                 age#policy |
                   older#1  |  -.0346395   .0258725    -1.34   0.189    -.0871114    .0178324
                            |
                post#policy |
                       1 1  |  -.0062925   .0061999    -1.01   0.317    -.0188664    .0062814
                            |
            age#post#policy |
                 older#1#1  |   .0041347   .0096289     0.43   0.670    -.0153935    .0236629
                            |
                       RACE |
                  Hispanic  |   .0107493   .0053622     2.00   0.053    -.0001257    .0216243
                 Black, NH  |   .0987545   .0063902    15.45   0.000     .0857947    .1117144
                 Asian, NH  |   .0268186   .0075218     3.57   0.001     .0115638    .0420734
                            |
                  EDUCATION |
                        HS  |  -.0912755   .0123685    -7.38   0.000    -.1163599    -.066191
              SOME COLLEGE  |  -.1428968   .0156572    -9.13   0.000    -.1746511   -.1111425
         BACHELOR/GRADUATE  |  -.2054314   .0181552   -11.32   0.000    -.2422519   -.1686109
                            |
                      _cons |   .3582151    .011577    30.94   0.000     .3347359    .3816943
---------------------------------------------------------------------------------------------

Here, the coefficient of interest (age#post#policy) is not significant, but still has the right sign (.0041347) --> older ages doing better post-policy in adopting states.

Finally, (4), taking your suggestion I created the home-grown interaction of post & treated (and NB: I continued to use the post-2010 for consistency, not the generalized post-variable that can vary by state), but counter to my intuition, the coefficient of interest switched signs:

. reghdfe screen i.age##i.policy_in_effect_1 `control_vars_allStates', absorb(state_num year) vce(cluster state_num)
(MWFE estimator converged in 4 iterations)

HDFE Linear regression                            Number of obs   = 14,463,048
Absorbing 2 HDFE groups                           F(  12,     36) =     339.16
Statistics robust to heteroskedasticity           Prob > F        =     0.0000
                                                  R-squared       =     0.0675
                                                  Adj R-squared   =     0.0675
                                                  Within R-sq.    =     0.0522
Number of clusters (state_num) =         37       Root MSE        =     0.4052

                                            (Std. Err. adjusted for 37 clusters in state_num)
---------------------------------------------------------------------------------------------
                            |               Robust
                     screen |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
----------------------------+----------------------------------------------------------------
                        age |
                     older  |   .0556058   .0121758     4.57   0.000     .0309123    .0802994
       1.policy_in_effect_1 |  -.0027725   .0092757    -0.30   0.767    -.0215845    .0160394
                            |
     age#policy_in_effect_1 |
                   older#1  |  -.0087877   .0088589    -0.99   0.328    -.0267544     .009179
                            |
                       RACE |
                  Hispanic  |   .0116799   .0052269     2.23   0.032     .0010792    .0222806
                 Black, NH  |   .0986197   .0062354    15.82   0.000     .0859737    .1112657
                 Asian, NH  |   .0254365   .0074194     3.43   0.002     .0103892    .0404838
                            |
                  EDUCATION |
                        HS  |  -.0911458   .0123827    -7.36   0.000    -.1162591   -.0660326
              SOME COLLEGE  |  -.1427811    .015681    -9.11   0.000    -.1745837   -.1109785
         BACHELOR/GRADUATE  |  -.2050967   .0183299   -11.19   0.000    -.2422713    -.167922
                            |
                      _cons |   .3566576   .0122077    29.22   0.000     .3318992     .381416
---------------------------------------------------------------------------------------------

If I'm reading this correctly, it says that, post-policy, older ages do *worse* compared to younger in states that have adopted...Is there something I'm missing in why these results differ so greatly from my mechanical DDD (reg 3), not to mention the built-up DID results (regs 1 & 2)?

Thanks in advance.

[Clyde Schechter]

Here, the coefficient of interest (age#post#policy)

Except that is not the coefficient of interest in that model. In fact, there is no single coefficient of interest in that model. That's the additional complication you get with models having 3-way interactions. The coefficient of age#post#policy only tells you the difference between the older and younger groups in any single fixed combination of post and policy, it does not contrast the effect of the policy in the two groups. To get that requires doing some adding and subtracting of some coefficients in that model. Notice that that triple interaction model has 2 more df than the model using policy_in_effect in a two-way interaction with age.

Next, you are taking this change in sign much, much too seriously. In your first two models, the age#post coefficients have confidence intervals that overlap extensively. In fact, the CIs are almost completely overlapping each other. So you should not be surprised if when you formally estimate the difference between those, your results are largely indeterminate. In particular, notice that the coefficient of age#policy_in_effect has a CI that runs from about -.03 to about +0.01, so the sign of the effect is indeterminate. While more of that CI is in negative than positive territory, the data are not giving you a conclusion about the direction of this effect. Perhaps when you go to the "real" analysis where you stop using 2010 a a proxy date for policy adoption things will be more clear cut. After all, using that proxy date is, in effect, adding noise to the analysis.

[Charlotte Webber]

Thank you for your continued insights - I better understand the challenges of interpreting the triple interactions coeff and agree with you about putting too much stock in the sign change.

I'm still, however, perplexed about my results & interpretation therein. First, let me say that I switched from a generalized DD to a traditional DD, by dropping states who adopted the policy change after 2010. (I know this may sound like data-mining, but there are legitimate data reasons for doing so that are not worth discussing here, plus there were very few states who adopted post-2010.) So now I only have the treatment group of states (adopted in 2010) and control group (never adopted). That said, I again ran 2 DDs, stratified by adopters/non-adopters, then ran the DDD. Here is what I get:

(1) DD reg for non-adopters (+ marginal effects) -- My interpretation: MEs shows older group has higher screening rates than younger group both pre and post-2010, but pre-post difference *not* significant (in other words: policy likely had no differential effect btw age groups in control states)

. reghdfe screen i.age##i.post `control_vars_allstates' if policy==0 , absorb(state_num year) vce(cluster state_num)
(MWFE estimator converged in 4 iterations)
note: 1bn.post is probably collinear with the fixed effects (all partialled-out values are close to zero; tol = 1.0e-09)

HDFE Linear regression                            Number of obs   = 10,106,818
Absorbing 2 HDFE groups                           F(   8,     14) =    4545.79
Statistics robust to heteroskedasticity           Prob > F        =     0.0000
                                                  R-squared       =     0.0709
                                                  Adj R-squared   =     0.0709
                                                  Within R-sq.    =     0.0592
Number of clusters (state_num) =         15       Root MSE        =     0.4230

                                  (Std. Err. adjusted for 15 clusters in state_num)
-----------------------------------------------------------------------------------
                  |               Robust
           screen |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
------------------+----------------------------------------------------------------
              age |
           older  |   .0644621     .01235     5.22   0.000     .0379739    .0909503
           1.post |          0  (omitted)
                  |
              age#|
             post |
         older#1  |   .0104903   .0066171     1.59   0.135     -.003702    .0246826
                  |
             RACE |
        Hispanic  |   .0085867   .0095678     0.90   0.385    -.0119343    .0291077
       Black, NH  |   .1000473   .0113135     8.84   0.000     .0757824    .1243123
       Asian, NH  |   .0205515   .0065605     3.13   0.007     .0064806    .0346224
                  |
        EDUCATION |
              HS  |  -.1019737   .0125699    -8.11   0.000    -.1289334   -.0750139
    SOME COLLEGE  |  -.1597237   .0127239   -12.55   0.000    -.1870138   -.1324336
BACHELOR/GRADU~E  |  -.2295448   .0121472   -18.90   0.000    -.2555978   -.2034917
                  |
            _cons |   .3909754   .0131814    29.66   0.000      .362704    .4192467
-----------------------------------------------------------------------------------
 
. margins age, dydx(post) atmeans noestimcheck

Conditional marginal effects                    Number of obs     = 10,106,818
Model VCE    : Robust

-------------------------------------------------------------------------------
              |            Delta-method
              |      dy/dx   Std. Err.      z    P>|z|     [95% Conf. Interval]
--------------+----------------------------------------------------------------
     1.post   |
          age |
     younger  |          0  (omitted)
       older  |   .0104903   .0066171     1.59   0.113     -.002479    .0234596
-------------------------------------------------------------------------------


. margins post, dydx(age) atmeans noestimcheck

Conditional marginal effects                    Number of obs     = 10,106,818
Model VCE    : Robust

------------------------------------------------------------------------------
             |            Delta-method
             |      dy/dx   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
     1.age   |
        post |
          0  |   .0644621     .01235     5.22   0.000     .0402565    .0886677
          1  |   .0749524   .0079614     9.41   0.000     .0593484    .0905564
------------------------------------------------------------------------------
Note: dy/dx for factor levels is the discrete change from the base level.

(2) DD reg for adopters (+ marginal effects) -- My interpretation: MEs shows older group has higher screening rates than younger group both pre and post-2010, and post (vs. pre) difference larger but only *marginally* significant (in other words: policy likely increased screening rates in older vs younger groups in treatment states)

 reghdfe screen i.age##i.post `control_vars_allstates' if policy==1 , absorb(state_num year) vce(cluster state_num)
(MWFE estimator converged in 4 iterations)
note: 1bn.post is probably collinear with the fixed effects (all partialled-out values are close to zero; tol = 1.0e-09)

HDFE Linear regression                            Number of obs   = 12,102,473
Absorbing 2 HDFE groups                           F(   8,     17) =    2823.73
Statistics robust to heteroskedasticity           Prob > F        =     0.0000
                                                  R-squared       =     0.0543
                                                  Adj R-squared   =     0.0543
                                                  Within R-sq.    =     0.0434
Number of clusters (state_num) =         18       Root MSE        =     0.3876

                                  (Std. Err. adjusted for 18 clusters in state_num)
-----------------------------------------------------------------------------------
                  |               Robust
           screen |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
------------------+----------------------------------------------------------------
              age |
           older  |   .0340021   .0180044     1.89   0.076    -.0039838     .071988
           1.post |          0  (omitted)
                  |
              age#|
             post |
         older#1  |   .0108415   .0060194     1.80   0.089    -.0018584    .0235414
                  |
             RACE |
        Hispanic  |   .0117636    .005919     1.99   0.063    -.0007245    .0242516
       Black, NH  |   .0955278   .0099261     9.62   0.000     .0745856    .1164699
       Asian, NH  |   .0231415   .0073517     3.15   0.006     .0076308    .0386522
                  |
        EDUCATION |
              HS  |  -.0706255   .0153914    -4.59   0.000    -.1030986   -.0381525
    SOME COLLEGE  |  -.1152681   .0206704    -5.58   0.000    -.1588787   -.0716574
BACHELOR/GRADU~E  |  -.1701706   .0227621    -7.48   0.000    -.2181944   -.1221468
                  |
            _cons |   .3131806   .0118273    26.48   0.000     .2882271    .3381341
-----------------------------------------------------------------------------------

margins age, dydx(post) atmeans noestimcheck

Conditional marginal effects                    Number of obs     = 12,102,473
Model VCE    : Robust

-------------------------------------------------------------------------------
              |            Delta-method
              |      dy/dx   Std. Err.      z    P>|z|     [95% Conf. Interval]
--------------+----------------------------------------------------------------
     1.post   |
          age |
     younger  |          0  (omitted)
       older  |   .0108415   .0060194     1.80   0.072    -.0009564    .0226394
-------------------------------------------------------------------------------
Note: dy/dx for factor levels is the discrete change from the base level.

. margins post, dydx(age) atmeans noestimcheck

Conditional marginal effects                    Number of obs     = 12,102,473
Model VCE    : Robust

------------------------------------------------------------------------------
             |            Delta-method
             |      dy/dx   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
     1.age   |
        post |
          0  |   .0340021   .0180044     1.89   0.059    -.0012859      .06929
          1  |   .0448436   .0132257     3.39   0.001     .0189216    .0707655
------------------------------------------------------------------------------
Note: dy/dx for factor levels is the discrete change from the base level.

(3) DDD reg for adopters vs non-adopters (+ marginal effects) -- My interpretation: Policy marginally lowered screen rates in older groups post-2010, not in younger groups, but difference between two not statistically significant?

reghdfe screen i.age##i.policy_in_effect_1 `control_vars_allstates', absorb(state_num year) vce(cluster state_num)
(MWFE estimator converged in 4 iterations)

HDFE Linear regression                            Number of obs   = 22,209,291
Absorbing 2 HDFE groups                           F(   9,     32) =     382.28
Statistics robust to heteroskedasticity           Prob > F        =     0.0000
                                                  R-squared       =     0.0668
                                                  Adj R-squared   =     0.0668
                                                  Within R-sq.    =     0.0504
Number of clusters (state_num) =         33       Root MSE        =     0.4043

                                  (Std. Err. adjusted for 33 clusters in state_num)
-----------------------------------------------------------------------------------
                  |               Robust
           screen |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
------------------+----------------------------------------------------------------
              age |
           older  |   .0569065   .0123951     4.59   0.000     .0316584    .0821545
1.policy_in_eff~1 |   .0002755   .0100359     0.03   0.978    -.0201669    .0207179
                  |
              age#|
policy_in_effec~1 |
         older#1  |  -.0144023   .0092727    -1.55   0.130      -.03329    .0044855
                  |
             RACE |
        Hispanic  |   .0107843   .0056422     1.91   0.065    -.0007086    .0222771
       Black, NH  |   .0980496   .0069817    14.04   0.000     .0838283    .1122709
       Asian, NH  |   .0235578    .007257     3.25   0.003     .0087757    .0383399
                  |
        EDUCATION |
              HS  |  -.0847184    .011647    -7.27   0.000    -.1084426   -.0609942
    SOME COLLEGE  |   -.135675   .0153167    -8.86   0.000     -.166874   -.1044759
BACHELOR/GRADU~E  |  -.1965977    .017987   -10.93   0.000    -.2332361   -.1599593
                  |
            _cons |   .3488202    .011987    29.10   0.000     .3244034     .373237
-----------------------------------------------------------------------------------

. margins age dydx(policy_in_effect_1) atmeans noestimcheck

Conditional marginal effects                    Number of obs     = 22,209,291
Model VCE    : Robust

-----------------------------------------------------------------------------------
                  |            Delta-method
                  |      dy/dx   Std. Err.      z    P>|z|     [95% Conf. Interval]
------------------+----------------------------------------------------------------
1.policy_in_eff~1 |
              age |
         younger  |   .0002755   .0100359     0.03   0.978    -.0193944    .0199454
           older  |  -.0141267   .0077947    -1.81   0.070     -.029404    .0011505
-----------------------------------------------------------------------------------
Note: dy/dx for factor levels is the discrete change from the base level.

My question: how do I reconcile the stratified DD reg results (screening rates seem to have increased for older vs younger groups in treatment states, post-policy adoption?) with the DDD reg results (screening rates seem to fall for older groups in states where policy was in effect?)? Am I misunderstanding/misinterpreting something? Incorrectly using the homegrown policy#post interaction now that it's a traditional DD/DDD? Thanks again

[Clyde Schechter]

The discrepancy between the two approaches (separate DD regressions vs DDD) arises from the covariates in the model. The DDD model you are using is not equivalent to the pair of DD models because in the DDD model, the effects of the covariates (race, education) are constrained to be the same for the adopters and non-adopters, whereas they are estimated independently in the pair of DD models.

If you were to rerun your DDD model, enhancing it by including interactions of policy with race and education, you would replicate the results of the paired DD models. (Well, you would replicate coefficients and marginal effect estimates, the standard errors, z-stats, p-values, and confidence intervals would differ.)

The question is which is the correct approach? Looking at the pair of DD models, the effects of the education variable, in particular, seem to be fairly large and substantially different in the policy and non-policy groups. So I think the discrepancy is telling you that a simple DDD that does not also interact policy with education is a mis-specified model, and I would not put much stock in those results. The race effects seem a bit more similar across policy and non-policy groups than those of education--and the race effects are also smaller. So you might be able to get away with not having a policy#race interaction in the DDD model. But I think you need to at least include policy#education to get a reasonably well specified DDD model.

[Charlotte Webber]

Hi Clyde, I'm back (after many months), still plugging away to understand fully. I want to re-visit you advice from post#2 where you wrote: "While I normally counsel people not to create home-brew interaction terms and just rely on factor variable notation to do it for them, the one situation where that is bad advice is when we are doing a generalized difference-in-differences model (i.e. the policy is implemented at different times in different states. Here it is more convenient to make a single variable representing policy#post: call it something like policy_in_effect. Doing this simplifies the modeling and the interpretation (because in this kind of model the effects of policy and post, separately, have a complicated meaning that doesn't relate to anything concrete in the real world, and they are pretty much never of any interest)."

My question is: the home-brew leaves out the separate interaction effects of policy and post with age - why is this ok? Thanks

[Clyde Schechter]

the home-brew leaves out the separate interaction effects of policy and post with age - why is this ok?

No, it doesn't. It leaves out the separate "main" effects of policy and post, but it includes the interaction. That's the whole point.

The reason it's OK not to include separate variables for the "main" effects of policy and post is that you capture that information with the panel and year effects (expressed in the -absorb()- option of -reghdfe- ). Those must be part of the model in order for the interaction term to represent the treatment effect.

[Charlotte Webber]

Thanks. Yes, I understood leaving out the main effects of policy and post (as they are absorbed), but I was thinking about policy#age and post#age. In my original DDD, I had i.policy##i.post##i.age, which included not just the main effects, but also the two-way interactions policy#post, policy#age, post#age. When I create policy_in_effect=policy*post, and run i.policy_in_effect##i.age, the two-way interactions between post#age and policy#age don't get included. I am curious if/why it is ok to leave out those two-way interactions?

[Clyde Schechter]

I am not certain, but I suggest you try running the regression again, explicitly including post#age and policy#age. I think you will find that both of them get omitted due to colinearity. My expectation is that policy#age will be colinear with the combination of age and the state-fixed effect, and post#age with the combination of age and year.