[Clyde Schechter]

I don't know how to say better what I said in #14. In this kind of model there is no such thing as "the" DID estimator of the effect. There is a different effect for each value of S, given by the formula b2 + b3*S. But there is no one value that can be called "the" DID estimator of the effect.

b2 can be interpreted as the effect of the intervention for those situations (if there are any) where S = 0, but not for any other circumstance. b3 is interpretable as the rate at which the marginal effect of the intervention increases per unit increase in S.

If you are desperate for a single number, you could, following your regression, assuming it was run with proper factor-variable notation, run -margins, dydx(activetreatment)-. The number you get from that could be described as the average effect of treatment (averaged over the distribution of S in the data). But again, it would be wrong to call that "the effect." It is the average value of the effect over all observed values of S, and depending on the circumstances, it may or may not be useful.

[Lom Newton]

Thank you Clyde once again. I run the -margins, dydx(activetreatment)- and a got a value. Now may I please run by if it is fit for purpose?:

Let's say my outcome variable is hours_worked. I wanted to see the percentage change in hours_worked (in terms of averages) after the policy was implemented relative to pre-intervention period for the treated group. My initial plan was to use the coefficient of activetreatment (b₂) and divide it by the average value of the pre-intervention hours_worked for the treated group. For example if b₂ = -40 and pre-intervention mean of hours_worked (treated group) = 120, then percentage change = -30/120 = -25%. Which I tend to interpret as a 25% drop in average hours_worked among the treated group following the introduction of the policy. But in my model since there is the interaction between activetreatment and S, and as you have explained, I obviously cannot just use b₂ this way. So can the -margins, dydx(activetreatment)- value be taken as the percentage change I seek to compute? Or could using that value in place of the b₂ in my percentage change formula above give me the desired output?

I am grateful for your attention and time.

[Clyde Schechter]

The marginal effect you get from -margins, dydx()- is a rate of absolute change in the absolute outcome, not a relative change. And calculating a relative change by using the mean outcome as a denominator will give you a misleading result. If you want a relative change in outcome per unit change in S then you need the semi-elasticity given by -margins, eydx(activetreatment)-. That will be the average proportional change in outcome per unit change in S. If you want it in percents, then multiply by 100.

[Lom Newton]

I got the following error by running -margins, eydx(activetreatment)-

. margins, eydx(activetreatment)
inconsistent estimation sample levels 0 and 1 of factor activetreatment
r(459);

Moreover, I feel I may not have communicated my earlier request properly. Or better still I am not sure I get the concept of the "relative change" in outcome variable per unit change in S you mentioned. In other words, what I intend to compute is that, after treatment, what is percentage change in the mean outcome for the treated group based on their pre-treatment mean value. If this measure is what -margins, eydx(activetreatment)- captures, then it's fine. The only issue left will be to address the error I got : inconsistent estimation sample levels 0 and 1 of factor activetreatment
r(459);

Thanks.

[Clyde Schechter]

A relative change means the change in proportion to the starting value. So if I start from a value of 5 and I go to 6 that is a relative change of (6-5)/5 = 20% or 0.2. (By contrast, as an absolute change it is 6-5 = 1).

I have never seen this "inconsistent estimation samples level 0 and 1..." before and I do not know what it means. My guess is that perhaps the expected values at activetreatment = 0 and 1 are of opposite signs. It is meaningless to speak of a relative change if a variable can equal 0 or can take on both positive and negative values. You have not shown any of your actual outputs, so I can't say anything more specific than that. If you want more specific advice, show the complete commands and output of the regression, of -margins activetreatment-, -margins, dydx(activetreatment)- and the complete output (if there was anything besides the error message) of -margins, eydx(activetreatment)-.

I will re-emphasize, though, that taking the average marginal effect and dividing it by the mean value of the outcome is not a meaningful answer. That is the quotient of two means, whereas you need the mean of the quotient, which is a very different thing.

[Lom Newton]

I have attached the outputs as requested. They are two files: Full regression output with the code and the margins output with the codes. You would observe that the -margins, eydx(activetreatment)- only produced an error; no real output.

[Clyde Schechter]

Many of us, especially me, do not download attachments from strangers. The best way to post code and output is described in Forum FAQ #12--copy/paste it into the Forum editor surrounded by code delimiters. Read the Forum FAQ for instructions on how to use code delimiters if you are not familiar with them.

[Lom Newton]

Duly noted. I have here now the codes and the outputs. The first is the full regression and the subsequent ones are the -margins-.

. reg y activetreatment##c.s treat c_tot f_ant met, absorb(dcomm) cluster(dcomm)
note: met omitted because of collinearity

Linear regression, absorbing indicators         Number of obs     =     20,155
                                                F(5, 116)         =          .
                                                Prob > F          =          .
                                                R-squared         =     0.3386
                                                Adj R-squared     =     0.3346
                                                Root MSE          =     87.039

                                       (Std. Err. adjusted for 117 clusters in dcomm)
-------------------------------------------------------------------------------------
                    |               Robust
                  y |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
--------------------+----------------------------------------------------------------
  1.activetreatment |  -13.80783   3.704243    -3.73   0.000    -21.14455   -6.471109
                  s |  -.0049947   .0010826    -4.61   0.000    -.0071389   -.0028504
                    |
activetreatment#c.s |
                 1  |   .0191151   .0012229    15.63   0.000     .0166929    .0215372
                    |
              treat |   8.685059   5.809645     1.49   0.138    -2.821674    20.19179
              c_tot |  -.0003844   .0001192    -3.23   0.002    -.0006205   -.0001483
              f_ant |   24.33651   3.480985     6.99   0.000     17.44198    31.23104
                met |          0  (omitted)
              _cons |   79.07902    3.44525    22.95   0.000     72.25527    85.90277
-------------------------------------------------------------------------------------

. margins activetreatment 

Predictive margins                              Number of obs     =     20,155
Model VCE    : Robust

Expression   : Linear prediction, predict()

---------------------------------------------------------------------------------
                |            Delta-method
                |     Margin   Std. Err.      t    P>|t|     [95% Conf. Interval]
----------------+----------------------------------------------------------------
activetreatment |
             0  |   114.2238   .9019402   126.64   0.000     112.4374    116.0102
             1  |    105.723   2.783682    37.98   0.000     100.2096    111.2365
---------------------------------------------------------------------------------

. margins, dydx(activetreatment)

Average marginal effects                        Number of obs     =     20,155
Model VCE    : Robust

Expression   : Linear prediction, predict()
dy/dx w.r.t. : 1.activetreatment

-----------------------------------------------------------------------------------
                  |            Delta-method
                  |      dy/dx   Std. Err.      t    P>|t|     [95% Conf. Interval]
------------------+----------------------------------------------------------------
1.activetreatment |  -8.500799   3.683099    -2.31   0.023    -15.79564   -1.205957
-----------------------------------------------------------------------------------
Note: dy/dx for factor levels is the discrete change from the base level.

. margins, eydx(activetreatment)
inconsistent estimation sample levels 0 and 1 of factor activetreatment
r(459);

end of do-file

r(459);

.

Thank you.

[Clyde Schechter]

Thank you. I'm sorry, but I don't know what to tell you. I was able to locate the part of the -margins- code that throws that error message, but I do not sufficiently understand what that code is doing to help you out here.

I'll just point out that I have said from the beginning that I think the attempt to reduce an interaction model's findings to a single number is misguided (though I did not think it would prove difficult to do!) And even had this proven easy, I still think that the best way to show interaction model results is along the lines I suggested in #14. Any attempt to reduce this to a single number is a) contrary to the very meaning of an interaction model, b) provides at best a limited glimpse of a complicated picture, and c) is usually misleading.

[Lom Newton]

Thank you Clyde for your time and assistance. I am grateful.

[Guest]

See https://www.ipr.northwestern.edu/wor.../Day%204.2.pdf for a lucid explanation of generalized difference-in-differences modeling, which applies to your situation.

You need a variable, call it treat, which is 1 in the group that receives the treatment (and is 1 in those observations at all times, including before treatment started) and 0 in all observations for the untreated group, another which we can call active_treatment which is 1 in the treatment group after treatment begins, but is 0 in the treatment group before treatment begins and is 0 in all observations in the control group. Then you do a fixed effects regresion that looks more or less like this:

xtset region
xtreg outcome i.treat i.activetreatment i.year, fe // OR RE AS THE CASE MAY BE

The coefficient of activetreatment is the DID estimator of the effect of treatment.

Hi Clyde,

Do you have any examples of papers which use this approach, or explain the theory behind this approach? I've used the approach successfully (so thanks for your help!), I'm just struggling to understand the theory behind it.

[Clyde Schechter]

https://www.annualreviews.org/doi/pd...-040617-013507

[Lucas Lourenco]

See https://www.ipr.northwestern.edu/wor.../Day%204.2.pdf for a lucid explanation of generalized difference-in-differences modeling, which applies to your situation.

Hi Clyde, is there a way to access this file? I've seen you recommending it in other posts and I'd like to read it, but I got "page not found". Thank you.

[Clyde Schechter]

It seems to have been taken down, and I don't know of any other place to find that presentation.

Fortunately, there is a very good article about DID estimation, including generalized DID, at https://www.annualreviews.org/doi/pd...-040617-013507. From the comments I have seen by people here who have looked at both, most seem to think this one is actually better.

[Dominik Mueller]

Hi, I'm referring to #9 and the answer #10.

There is data for the treatment and control groups, and the matched control groups are assigned the same treatment date as the treatment groups. However, the treatment dates are staggered at different points of time. What I wonder about is , even though I can design a pre/post design by it and then using the suggested regression - would I still have the problem with staggered adoption which is describes in many new papers such as Godman Bacon in this paper https://andrewcbaker.netlify.app/201...s-methodology/ ? I mean, do I have the problem that the "interaction term" i.type##.i.pre_post would also include the 2x2 differences to the earlier adopter? Or do it just take into account the control groups?