Announcement

Collapse
No announcement yet.
X
Collapse
  • Page of 1
  • Filter
  • Time
  • Show
Clear All
new posts
  • #1

    Compare ATT results from propensity score matching and coarsened exact matching method

    Hi, all.

    I am trying to use propensity score matching(PSM) and coarsened exact matching methods to obtain ATT( average treatment effect on treated) of quantile expenditure.
    my PSM codes are as below:
    1.Q1: psmatch2 sampleb `expcova' if exp_all4q==1, outcome(exp_all) n(1) ate ties logit common
    2.Q2: psmatch2 sampleb `expcova' if exp_all4q==2, outcome(exp_all) n(1) ate ties logit common
    3.Q3: psmatch2 sampleb `expcova' if exp_all4q==3, outcome(exp_all) n(1) ate ties logit common
    4.Q4: psmatch2 sampleb `expcova' if exp_all4q==4, outcome(exp_all) n(1) ate ties logit common

    and the ATTs are 552.28, 874.18, 1946.76, 7887.9 respectively. They are all statistically significant, while most covariates are not.

    My question is, I am not sure if I can just directly compare above ATTs. if I can't then, are there any good suggestions?
    Cause I have tried dummy method as this link suggests: https://stats.idre.ucla.edu/stata/fa...r-more-groups/
    and I also tried command suest, but it seems they are all not working.

    Thank you in advance.
    Tags: None
  • #2
    Below is an approximate yet easy way of comparing ATTs from different subsamples. Essentially, ATTs from -psmatch2- can largely be replicated by weighted least squares estimates and one can further compare the estimates from different linear regressions based on usual methods.

    Code:
    . webuse cattaneo2, clear
(Excerpt from Cattaneo (2010) Journal of Econometrics 155: 138-154)

. psmatch2 mbsmoke mage medu if mmarried == 1, outcome(bweight) n(1) ties logit common

Logistic regression                             Number of obs     =      3,248
                                                LR chi2(2)        =      89.54
                                                Prob > chi2       =     0.0000
Log likelihood = -1184.8054                     Pseudo R2         =     0.0364

------------------------------------------------------------------------------
     mbsmoke |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
        mage |  -.0317064   .0114785    -2.76   0.006    -.0542038    -.009209
        medu |   -.176328   .0221354    -7.97   0.000    -.2197126   -.1329435
       _cons |    1.20868   .3694059     3.27   0.001     .4846576    1.932702
------------------------------------------------------------------------------
--------------------------------------------------------------------------------------
> --
        Variable     Sample |    Treated     Controls   Difference         S.E.   T-st
> at
----------------------------+---------------------------------------------------------
> --
         bweight  Unmatched |  3195.7555   3463.84255  -268.087049   28.6664817    -9.
> 35
                        ATT |  3195.7555   3447.11171  -251.356207   33.7286788    -7.
> 45
----------------------------+---------------------------------------------------------
> --
Note: S.E. does not take into account that the propensity score is estimated.

           | psmatch2:
 psmatch2: |   Common
 Treatment |  support
assignment | On suppor |     Total
-----------+-----------+----------
 Untreated |     2,839 |     2,839 
   Treated |       409 |       409 
-----------+-----------+----------
     Total |     3,248 |     3,248 

. gen weight = _weight if mmarried == 1
(1,962 missing values generated)

. psmatch2 mbsmoke mage medu if mmarried == 0, outcome(bweight) n(1) ties logit common

Logistic regression                             Number of obs     =      1,394
                                                LR chi2(2)        =      28.37
                                                Prob > chi2       =     0.0000
Log likelihood =  -866.2637                     Pseudo R2         =     0.0161

------------------------------------------------------------------------------
     mbsmoke |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
        mage |   .0497414   .0106289     4.68   0.000     .0289092    .0705737
        medu |  -.0878898   .0249567    -3.52   0.000     -.136804   -.0389755
       _cons |  -.8695609   .3294632    -2.64   0.008    -1.515297   -.2238248
------------------------------------------------------------------------------
--------------------------------------------------------------------------------------
> --
        Variable     Sample |    Treated     Controls   Difference         S.E.   T-st
> at
----------------------------+---------------------------------------------------------
> --
         bweight  Unmatched | 3085.43736   3258.92545   -173.48809   34.7690292    -4.
> 99
                        ATT | 3085.43736   3274.18222  -188.744859   40.7322773    -4.
> 63
----------------------------+---------------------------------------------------------
> --
Note: S.E. does not take into account that the propensity score is estimated.

           | psmatch2:
 psmatch2: |   Common
 Treatment |  support
assignment | On suppor |     Total
-----------+-----------+----------
 Untreated |       939 |       939 
   Treated |       455 |       455 
-----------+-----------+----------
     Total |     1,394 |     1,394 

. replace weight = _weight if mmarried == 0
(1,277 real changes made)

. 
. reg bweight mbsmoke [aw=weight] if mmarried == 1, r
(sum of wgt is 818.0000046305358)

Linear regression                               Number of obs     =      2,680
                                                F(1, 2678)        =      48.76
                                                Prob > F          =     0.0000
                                                R-squared         =     0.0481
                                                Root MSE          =     559.11

------------------------------------------------------------------------------
             |               Robust
     bweight |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
     mbsmoke |  -251.3562   35.99586    -6.98   0.000    -321.9387   -180.7737
       _cons |   3447.112   22.45341   153.52   0.000     3403.084    3491.139
------------------------------------------------------------------------------

. reg bweight mbsmoke [aw=weight] if mmarried == 0, r
(sum of wgt is 910.0000005736947)

Linear regression                               Number of obs     =      1,277
                                                F(1, 1275)        =      20.16
                                                Prob > F          =     0.0000
                                                R-squared         =     0.0250
                                                Root MSE          =     589.67

------------------------------------------------------------------------------
             |               Robust
     bweight |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
     mbsmoke |  -188.7449   42.03683    -4.49   0.000    -271.2138   -106.2759
       _cons |   3274.182   33.26249    98.43   0.000     3208.927    3339.437
------------------------------------------------------------------------------

. 
. reg bweight c.mbsmoke##c.mmarried [aw=weight], r
(sum of wgt is 1,728.00000520423)

Linear regression                               Number of obs     =      3,957
                                                F(3, 3953)        =      40.24
                                                Prob > F          =     0.0000
                                                R-squared         =     0.0494
                                                Root MSE          =     575.35

------------------------------------------------------------------------------
             |               Robust
     bweight |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
     mbsmoke |  -188.7449   42.02514    -4.49   0.000    -271.1379   -106.3519
    mmarried |   172.9295   40.12564     4.31   0.000     94.26058    251.5984
             |
   c.mbsmoke#|
  c.mmarried |  -62.61135   55.33677    -1.13   0.258    -171.1026    45.87994
             |
       _cons |   3274.182   33.25324    98.46   0.000     3208.987    3339.377
------------------------------------------------------------------------------
    • 1 like

    Comment

    • #3
      Hi，Fei Wang,

      Thank you so much for your explanation. it's really helpful.
      I am just wondering if I can conduct Chow test over ATT results among the groups?

      and I have one more question.
      when I conduct CEM method as below:

      imb `vrqol_paidtime', treatment(sampleb)
      cem `vrqol_paidtime', treatment(sampleb) k2k
      reg activity_paid sampleb `vrqol_paidtime' [iweight=cem_weights]

      I found I got different results even every time I run the same codes. is it because of the "weight"?


      Thanks again.
      Last edited by Jane Quan; 07 Dec 2021, 04:14.

      Comment

      • #4
        Originally posted by Jane Quan View Post
        Hi，Fei Wang,

        Thank you so much for your explanation. it's really helpful.
        I am just wondering if I can conduct Chow test over ATT results among the groups?

        and I have one more question.
        when I conduct CEM method as below:

        imb `vrqol_paidtime', treatment(sampleb)
        cem `vrqol_paidtime', treatment(sampleb) k2k
        reg activity_paid sampleb `vrqol_paidtime' [iweight=cem_weights]

        I found I got different results even every time I run the same codes. is it because of the "weight"?


        Thanks again.
        I think I found the problem. it is the "k2k". Thank you anyway. your codes are really helpful



        Comment

        • #5
          Originally posted by Jane Quan View Post
          Hi，Fei Wang,
          I am just wondering if I can conduct Chow test over ATT results among the groups?
          I think the last regression in #2 does Chow test.

          Comment

          • #6
            Originally posted by Fei Wang View Post
            Below is an approximate yet easy way of comparing ATTs from different subsamples. Essentially, ATTs from -psmatch2- can largely be replicated by weighted least squares estimates and one can further compare the estimates from different linear regressions based on usual methods.

            Code:
            . webuse cattaneo2, clear
(Excerpt from Cattaneo (2010) Journal of Econometrics 155: 138-154)

. psmatch2 mbsmoke mage medu if mmarried == 1, outcome(bweight) n(1) ties logit common

Logistic regression Number of obs = 3,248
LR chi2(2) = 89.54
Prob > chi2 = 0.0000
Log likelihood = -1184.8054 Pseudo R2 = 0.0364

------------------------------------------------------------------------------
mbsmoke | Coef. Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
mage | -.0317064 .0114785 -2.76 0.006 -.0542038 -.009209
medu | -.176328 .0221354 -7.97 0.000 -.2197126 -.1329435
_cons | 1.20868 .3694059 3.27 0.001 .4846576 1.932702
------------------------------------------------------------------------------
--------------------------------------------------------------------------------------
> --
Variable Sample | Treated Controls Difference S.E. T-st
> at
----------------------------+---------------------------------------------------------
> --
bweight Unmatched | 3195.7555 3463.84255 -268.087049 28.6664817 -9.
> 35
ATT | 3195.7555 3447.11171 -251.356207 33.7286788 -7.
> 45
----------------------------+---------------------------------------------------------
> --
Note: S.E. does not take into account that the propensity score is estimated.

| psmatch2:
psmatch2: | Common
Treatment | support
assignment | On suppor | Total
-----------+-----------+----------
Untreated | 2,839 | 2,839
Treated | 409 | 409
-----------+-----------+----------
Total | 3,248 | 3,248

. gen weight = _weight if mmarried == 1
(1,962 missing values generated)

. psmatch2 mbsmoke mage medu if mmarried == 0, outcome(bweight) n(1) ties logit common

Logistic regression Number of obs = 1,394
LR chi2(2) = 28.37
Prob > chi2 = 0.0000
Log likelihood = -866.2637 Pseudo R2 = 0.0161

------------------------------------------------------------------------------
mbsmoke | Coef. Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
mage | .0497414 .0106289 4.68 0.000 .0289092 .0705737
medu | -.0878898 .0249567 -3.52 0.000 -.136804 -.0389755
_cons | -.8695609 .3294632 -2.64 0.008 -1.515297 -.2238248
------------------------------------------------------------------------------
--------------------------------------------------------------------------------------
> --
Variable Sample | Treated Controls Difference S.E. T-st
> at
----------------------------+---------------------------------------------------------
> --
bweight Unmatched | 3085.43736 3258.92545 -173.48809 34.7690292 -4.
> 99
ATT | 3085.43736 3274.18222 -188.744859 40.7322773 -4.
> 63
----------------------------+---------------------------------------------------------
> --
Note: S.E. does not take into account that the propensity score is estimated.

| psmatch2:
psmatch2: | Common
Treatment | support
assignment | On suppor | Total
-----------+-----------+----------
Untreated | 939 | 939
Treated | 455 | 455
-----------+-----------+----------
Total | 1,394 | 1,394

. replace weight = _weight if mmarried == 0
(1,277 real changes made)

.
. reg bweight mbsmoke [aw=weight] if mmarried == 1, r
(sum of wgt is 818.0000046305358)

Linear regression Number of obs = 2,680
F(1, 2678) = 48.76
Prob > F = 0.0000
R-squared = 0.0481
Root MSE = 559.11

------------------------------------------------------------------------------
| Robust
bweight | Coef. Std. Err. t P>|t| [95% Conf. Interval]
-------------+----------------------------------------------------------------
mbsmoke | -251.3562 35.99586 -6.98 0.000 -321.9387 -180.7737
_cons | 3447.112 22.45341 153.52 0.000 3403.084 3491.139
------------------------------------------------------------------------------

. reg bweight mbsmoke [aw=weight] if mmarried == 0, r
(sum of wgt is 910.0000005736947)

Linear regression Number of obs = 1,277
F(1, 1275) = 20.16
Prob > F = 0.0000
R-squared = 0.0250
Root MSE = 589.67

------------------------------------------------------------------------------
| Robust
bweight | Coef. Std. Err. t P>|t| [95% Conf. Interval]
-------------+----------------------------------------------------------------
mbsmoke | -188.7449 42.03683 -4.49 0.000 -271.2138 -106.2759
_cons | 3274.182 33.26249 98.43 0.000 3208.927 3339.437
------------------------------------------------------------------------------

.
. reg bweight c.mbsmoke##c.mmarried [aw=weight], r
(sum of wgt is 1,728.00000520423)

Linear regression Number of obs = 3,957
F(3, 3953) = 40.24
Prob > F = 0.0000
R-squared = 0.0494
Root MSE = 575.35

------------------------------------------------------------------------------
| Robust
bweight | Coef. Std. Err. t P>|t| [95% Conf. Interval]
-------------+----------------------------------------------------------------
mbsmoke | -188.7449 42.02514 -4.49 0.000 -271.1379 -106.3519
mmarried | 172.9295 40.12564 4.31 0.000 94.26058 251.5984
|
c.mbsmoke#|
c.mmarried |  -62.61135  55.33677 -1.13 0.258 -171.1026 45.87994
|
_cons | 3274.182 33.25324 98.46 0.000 3208.987 3339.377
------------------------------------------------------------------------------
            Hi,

            Is it possible to display or calculate treatment effect on treated or untreated for each observation? then I can use command "sum" to obtain ATT or ATU. thanks.
            Last edited by Jane Quan; 05 Feb 2022, 07:50.

            Comment

            • #7
              Originally posted by Jane Quan View Post
              Is it possible to display or calculate treatment effect on treated or untreated for each observation? then I can use command "sum" to obtain ATT or ATU. thanks.
              Unfortunately not, because in general it is impossible to tell how a treatment affected an individual case. (If it were possible, researchers wouldn't need samples, average treatment effects, and measures of variance and statistical significance.)

              If perhaps you mean that you want to sum up the ATT or ATU of all four groups with varying values of exp_all4q, then please see the section titled "Matching within strata" in the help file of psmatch2 (Leuven and Sianesi, available from SSC).
              David Radwin
              Senior Researcher, California Competes
              californiacompetes.org
              Pronouns: He/Him

              Comment

              • #8
                Originally posted by David Radwin View Post

                Unfortunately not, because in general it is impossible to tell how a treatment affected an individual case. (If it were possible, researchers wouldn't need samples, average treatment effects, and measures of variance and statistical significance.)

                If perhaps you mean that you want to sum up the ATT or ATU of all four groups with varying values of exp_all4q, then please see the section titled "Matching within strata" in the help file of psmatch2 (Leuven and Sianesi, available from SSC).
                Hi David，

                Thank you so much for your explanation.
                It helps a lot.

                Regards.

                Comment

                Previous Next
                Working...
                X