Comparing ATT coefficients between two separate Difference-in-difference model estimates.

[Kasia Jasiniewska]

Hi!

I need help with comparing two ATT coefficients between two separate Difference-in-difference models. I run the following regressions:

reg score i.countryDiD##i.post_treatment escs gender lang_home [aw=w_fstuwt] if (cnt=="POL" | cnt == "CZE") & escs_low1==1, cluster(schoolid)

reg score i.countryDiD##i.post_treatment escs gender lang_home [aw=w_fstuwt] if (cnt=="POL" | cnt == "CZE") & escs_high1==1, cluster(schoolid)

The first one conditions on being part of the lowest 10% of the socioeconomic spectrum, while the second one conditions on being in the top 10%.

I wish to test for the equality of 'i.countryDiD##i.post_treatment' coefficients (which represent the average treatment effect for each group). As I am using clustering, which is necessary in the context of my data set, I struggle to find a command that would allow me to compare the coefficients in STATA, and not manually.

I get the comment:

're-estimate without the cluster() option, and
specify the cluster() option with suest'

'option eststo not allowed'.

I would be grateful for your suggestions.

[Carlo Lazzaro]

Kasia:
-suest-, as you've already found out yourself, is the way to go.
However, the only way to proceed is to go default standard errors in -regress- and then rely on -suest- cluster-robust standard errors, as in the following plain vanilla toy-example:

. sysuse auto.dta
(1978 automobile data)

. regress price i.foreign##i.rep78
note: 1.foreign#1b.rep78 identifies no observations in the sample.
note: 1.foreign#2.rep78 identifies no observations in the sample.
note: 1.foreign#5.rep78 omitted because of collinearity.

      Source |       SS           df       MS      Number of obs   =        69
-------------+----------------------------------   F(7, 61)        =      0.39
       Model |    24684607         7  3526372.43   Prob > F        =    0.9049
    Residual |   552112352        61  9051022.16   R-squared       =    0.0428
-------------+----------------------------------   Adj R-squared   =   -0.0670
       Total |   576796959        68  8482308.22   Root MSE        =    3008.5

-------------------------------------------------------------------------------
        price | Coefficient  Std. err.      t    P>|t|     [95% conf. interval]
--------------+----------------------------------------------------------------
      foreign |
     Foreign  |   2088.167   2351.846     0.89   0.378     -2614.64    6790.974
              |
        rep78 |
           2  |   1403.125   2378.422     0.59   0.557    -3352.823    6159.073
           3  |   2042.574   2204.707     0.93   0.358    -2366.011    6451.159
           4  |   1317.056   2351.846     0.56   0.578    -3385.751    6019.863
           5  |       -360   3008.492    -0.12   0.905    -6375.851    5655.851
              |
foreign#rep78 |
   Foreign#1  |          0  (empty)
   Foreign#2  |          0  (empty)
   Foreign#3  |  -3866.574   2980.505    -1.30   0.199    -9826.462    2093.314
   Foreign#4  |  -1708.278   2746.365    -0.62   0.536    -7199.973    3783.418
   Foreign#5  |          0  (omitted)
              |
        _cons |     4564.5   2127.325     2.15   0.036      310.651    8818.349
-------------------------------------------------------------------------------

. estimates store A

. regress price i.foreign##c.trunk

      Source |       SS           df       MS      Number of obs   =        74
-------------+----------------------------------   F(3, 70)        =      3.45
       Model |  81822937.2         3  27274312.4   Prob > F        =    0.0211
    Residual |   553242459        70   7903463.7   R-squared       =    0.1288
-------------+----------------------------------   Adj R-squared   =    0.0915
       Total |   635065396        73  8699525.97   Root MSE        =    2811.3

---------------------------------------------------------------------------------
          price | Coefficient  Std. err.      t    P>|t|     [95% conf. interval]
----------------+----------------------------------------------------------------
        foreign |
       Foreign  |   1114.855    2657.69     0.42   0.676    -4185.741    6415.451
          trunk |   261.6024   91.41565     2.86   0.006     79.27966    443.9252
                |
foreign#c.trunk |
       Foreign  |   6.257639   211.4828     0.03   0.976    -415.5316    428.0468
                |
          _cons |   2213.787    1403.61     1.58   0.119    -585.6244    5013.199
---------------------------------------------------------------------------------

. estimates store B

. suest A B

Simultaneous results for A, B                               Number of obs = 74

---------------------------------------------------------------------------------
                |               Robust
                | Coefficient  std. err.      z    P>|z|     [95% conf. interval]
----------------+----------------------------------------------------------------
A_mean          |
        foreign |
       Foreign  |   2088.167   889.0552     2.35   0.019     345.6505    3830.683
                |
          rep78 |
             2  |   1403.125   1220.527     1.15   0.250    -989.0639    3795.314
             3  |   2042.574   744.2031     2.74   0.006     583.9628    3501.185
             4  |   1317.056   568.2894     2.32   0.020     203.2288    2430.882
             5  |       -360   306.3387    -1.18   0.240    -960.4128    240.4128
                |
  foreign#rep78 |
     Foreign#1  |          0  (empty)
     Foreign#2  |          0  (empty)
     Foreign#3  |  -3866.574   1283.502    -3.01   0.003    -6382.192   -1350.956
     Foreign#4  |  -1708.278   1184.953    -1.44   0.149    -4030.743    614.1875
     Foreign#5  |          0  (omitted)
                |
          _cons |     4564.5   263.0594    17.35   0.000     4048.913    5080.087
----------------+----------------------------------------------------------------
A_lnvar         |
          _cons |   16.01839   .2087498    76.73   0.000     15.60925    16.42753
----------------+----------------------------------------------------------------
B_mean          |
        foreign |
       Foreign  |   1114.855   1707.457     0.65   0.514    -2231.699    4461.408
          trunk |   261.6024    64.3181     4.07   0.000     135.5413    387.6636
                |
foreign#c.trunk |
       Foreign  |   6.257639   162.7648     0.04   0.969    -312.7556    325.2708
                |
          _cons |   2213.787   888.1394     2.49   0.013      473.066    3954.508
----------------+----------------------------------------------------------------
B_lnvar         |
          _cons |   15.88281   .2420743    65.61   0.000     15.40835    16.35727
---------------------------------------------------------------------------------

. mat list e(b)

e(b)[1,26]
         A_mean:      A_mean:      A_mean:      A_mean:      A_mean:      A_mean:      A_mean:      A_mean:      A_mean:      A_mean:      A_mean:      A_mean:      A_mean:
             0b.           1.          1b.           2.           3.           4.           5.  0b.foreign#  0b.foreign#  0b.foreign#  0b.foreign#  0b.foreign#  1o.foreign#
        foreign      foreign        rep78        rep78        rep78        rep78        rep78     1b.rep78     2o.rep78     3o.rep78     4o.rep78     5o.rep78     1b.rep78
y1            0    2088.1667            0     1403.125    2042.5741    1317.0556         -360            0            0            0            0            0            0

         A_mean:      A_mean:      A_mean:      A_mean:      A_mean:     A_lnvar:      B_mean:      B_mean:      B_mean:      B_mean:      B_mean:      B_mean:     B_lnvar:
     1o.foreign#   1.foreign#   1.foreign#  1o.foreign#                                    0b.           1.               0b.foreign#   1.foreign#                          
       2o.rep78      3.rep78      4.rep78     5o.rep78        _cons        _cons      foreign      foreign        trunk     co.trunk      c.trunk        _cons        _cons
y1            0   -3866.5741   -1708.2778            0       4564.5    16.018388            0    1114.8547    261.60243            0    6.2576392    2213.7872    15.882812

. test 1.foreign#3.rep78 1.foreign#c.trunk

 ( 1)  [A_mean]1.foreign#3.rep78 = 0
 ( 2)  [B_mean]1.foreign#c.trunk = 0

           chi2(  2) =    9.19
         Prob > chi2 =    0.0101

As an aside, you may want to consider -didregress- and -xtdidregress- for DID-related regressions.

[Kasia Jasiniewska]

Dear Carlo,
Thanks. This worked!
Best regards, Kasia

[Kasia Jasiniewska]

I have a follow-up question. After performing the tests for my coefficients I came into conclusion that STATA might incorrectly interpreting my variables as categorical and thus using the Chi-squared instead of a t-test. My explanatory variable is not categorical, on the contrary, it's students' performance (mean score), thus it's continuous, and the coefficients which I wish to compare are the average treatment effects from the two identically specified difference-in-difference models, but conditioned on being in two different socioeconomic spectrums. When I employ the t-test by hand it gives me different results from the Chi-sq test performed in STATA (these are significantly different - t-test implies that the difference in means is significantly different from each other, while Chi-squared finds no distinguishable effect between the two groups, which totally changes the interpretation of the results).

T-test: The calculated t-statistic for the difference in average treatment effects (ATT) between the bottom 10% SES and the top 10% SES is approximately -113.07. -> highly significant. What I wish to do is to simply calculate the same thing, but in STATA, so as I not do it manually.

I below attach my code and the table with results I wish to compare (ATT of M3a and M4a)

.

//Comparing coefficients: Test ATT 10/90 static DD (Table 5)//

reg score i.countryDiD##i.post_treatment escs gender lang_home [aw=w_fstuwt] if (cnt=="POL" | cnt == "CZE") & escs_low1==1
estimates store A

gen post_treatmentHIGH10 = post_treatment
reg score i.countryDiD##i.post_treatmentHIGH10 escs gender lang_home [aw=w_fstuwt] if (cnt=="POL" | cnt == "CZE") & escs_high1==1
estimates store B

suest A B, cluster(schoolid)
mat list e(b)
ttest 1.countryDiD#1.post_treatment = 1.countryDiD#1.post_treatmentHIGH10

Please note, I am generating a new variable as these have the same names and otherwise STATA would not allow me to compare those.

- Is my code correct?
- Why would a t-test yield different results?
- Why would STATA choose to perform a Chi-sq in this case?
- Which of the two tests is appropriate in my case?

I would be really grateful for your help.

Best,
Kasia

[Kasia Jasiniewska]

I have a follow-up question. After performing the tests for my coefficients I came into conclusion that STATA might incorrectly interpreting my variables as categorical and thus using the Chi-squared instead of a t-test. My explanatory variable is not categorical, on the contrary, it's students' performance (mean score), thus it's continuous, and the coefficients which I wish to compare are the average treatment effects from the two identically specified difference-in-difference models, but conditioned on being in two different socioeconomic spectrums. When I employ the t-test by hand it gives me different results from the Chi-sq test performed in STATA (these are significantly different - t-test implies that the difference in means is significantly different from each other, while Chi-squared finds no distinguishable effect between the two groups, which totally changes the interpretation of the results).

//Comparing coefficients: Test ATT 10/90 static DD (Table 5)//

reg score i.countryDiD##i.post_treatment escs gender lang_home [aw=w_fstuwt] if (cnt=="POL" | cnt == "CZE") & escs_low1==1
estimates store A

gen post_treatmentHIGH10 = post_treatment
reg score i.countryDiD##i.post_treatmentHIGH10 escs gender lang_home [aw=w_fstuwt] if (cnt=="POL" | cnt == "CZE") & escs_high1==1
estimates store B

suest A B, cluster(schoolid)
mat list e(b)
ttest 1.countryDiD#1.post_treatment = 1.countryDiD#1.post_treatmentHIGH10

// After careful evaluation I think the first method is still valid in the context of my work as the 'test' that STATA is performing is the Wald test described here: https://www.stata.com/manuals13/rtestnl.pdf#rtestnl. It is based on the ratio of the estimated coefficient to its standard error, squared, which, under the null hypothesis, indeed follows a chi-square distribution with degrees of freedom equal to the number of restrictions being tested. Based on the STATA article provided, I think it's a legitimate way to compare coefficients between two distinct models. Therefore, I believe this resolves my questions, although I would be grateful if anyone familiar with this method could confirm my findings. Thanks!