How to test whether difference in differences is statistically significant without regression?

[Anna Jongma]

Dear all,

I have two groups and have tested whether the mean of my variable of interest is significantly different in each of two periods (using the ttest command; i.e. ttest "var" if year==1, by(treated) and ttest "var" if year==2, by(treated). However, I would also like to test whether the difference between those differences themselves is statistically significant (i.e. a get a raw diff-in-diff estimate); is it possible to do this without a regression?

Many thanks!

[Carlo Lazzaro]

Anna:
I fail to get why linear regression intimidates you.
That said, you may want to try something along the following lines:

. use "http://www.stata-press.com/data/r15/union.dta"
(NLS Women 14-24 in 1968)

. tab not_smsa

   1 if not |
       SMSA |      Freq.     Percent        Cum.
------------+-----------------------------------
          0 |     18,767       71.63       71.63
          1 |      7,433       28.37      100.00
------------+-----------------------------------
      Total |     26,200      100.00


. ttest age if year==70,by(not_smsa) unequal

Two-sample t test with unequal variances
------------------------------------------------------------------------------
   Group |     Obs        Mean    Std. Err.   Std. Dev.   [95% Conf. Interval]
---------+--------------------------------------------------------------------
       0 |   1,240    22.12661    .0716767    2.523997    21.98599    22.26723
       1 |     418    22.04306    .1214546    2.483145    21.80432     22.2818
---------+--------------------------------------------------------------------
combined |   1,658    22.10555    .0617231    2.513275    21.98449    22.22661
---------+--------------------------------------------------------------------
    diff |            .0835507    .1410275               -.1933183    .3604197
------------------------------------------------------------------------------
    diff = mean(0) - mean(1)                                      t =   0.5924
Ho: diff = 0                     Satterthwaite's degrees of freedom =  728.314

    Ha: diff < 0                 Ha: diff != 0                 Ha: diff > 0
 Pr(T < t) = 0.7231         Pr(|T| > |t|) = 0.5537          Pr(T > t) = 0.2769

. ttest age if year==71,by(not_smsa) unequal

Two-sample t test with unequal variances
------------------------------------------------------------------------------
   Group |     Obs        Mean    Std. Err.   Std. Dev.   [95% Conf. Interval]
---------+--------------------------------------------------------------------
       0 |   1,365    22.66374    .0722338    2.668741    22.52203    22.80544
       1 |     428    22.50234    .1391961    2.879711    22.22874    22.77593
---------+--------------------------------------------------------------------
combined |   1,793    22.62521    .0642511    2.720637    22.49919    22.75122
---------+--------------------------------------------------------------------
    diff |            .1613998    .1568224               -.1465205    .4693201
------------------------------------------------------------------------------
    diff = mean(0) - mean(1)                                      t =   1.0292
Ho: diff = 0                     Satterthwaite's degrees of freedom =  672.672

    Ha: diff < 0                 Ha: diff != 0                 Ha: diff > 0
 Pr(T < t) = 0.8481         Pr(|T| > |t|) = 0.3038          Pr(T > t) = 0.1519

. g age_diff=age if year==70
(24,542 missing values generated)

. replace age_diff=age if year==71
(1,793 real changes made)


. bysort idcode: gen flag=age_diff[_n+1]-age_diff[_n] if age_diff[_n+1]!=.
(25,037 missing values generated)

. ttest flag, by(not_smsa) unequal

Two-sample t test with unequal variances
------------------------------------------------------------------------------
   Group |     Obs        Mean    Std. Err.   Std. Dev.   [95% Conf. Interval]
---------+--------------------------------------------------------------------
       0 |     886    1.002257    .0052964    .1576504    .9918624    1.012652
       1 |     277     1.01444    .0071809    .1195135    1.000304    1.028577
---------+--------------------------------------------------------------------
combined |   1,163    1.005159    .0043836    .1494944    .9965583     1.01376
---------+--------------------------------------------------------------------
    diff |           -.0121831    .0089228               -.0297067    .0053405
------------------------------------------------------------------------------
    diff = mean(0) - mean(1)                                      t =  -1.3654
Ho: diff = 0                     Satterthwaite's degrees of freedom =  602.375

    Ha: diff < 0                 Ha: diff != 0                 Ha: diff > 0
 Pr(T < t) = 0.0863         Pr(|T| > |t|) = 0.1726          Pr(T > t) = 0.9137

.

[Marcos Almeida]

I'm not sure whether I understood your question, but maybe you meant the use of a time-series operator, such as "D." to create the differences beforehand.

[Dick Campbell]

I'm not sure I understand this either, but as you pose the question it appears that you are ianterested in a simple group by time interaction. But I can't tell from what you posted if you have panel data or not. In any case, you can easily do this in some form of regression (depending on whether you have panel data or repeated cross sectons) and if you use something else, e.g. ANOVA, the resulting linear model is regression-based.