Propensity score matching methodology

[Kalvis Altens]

Dear All,

Your help with propensity score matching method in STATA will be greatly appreciated. I have access to STATA 12. All questions refer to commands employed in STATA 12.

The focus of my research study is: Do EU structural fund support in Latvia stimulate firms to invest additional funds in their research and development (R&D) activities. I want to see if EU innovation support programms either complement or crowd-out (substitute) private R&D investment.

I want to use propensity score matching method to estimate the average causal effect of participating in the programm using a cross-sectional microdataset based on the Latvian edition of Eurostat’s community innovation survey for 2010-2012. The sample consists of 121 firms in the treatment group and 307 in the control group. Treatment variable is dummy for having/not having received support from the program and outcome variable is the amount of R&D expenditure a company reported in 2012.

The questions.

1. I have some 20 covariates in probit. Most of them have p-values close to 100%. Some have around 30%. Two of them have values below 5%. How is the propensity score calculated in this case? Does it take into account only variables whose significance is below 5% or all of them? If no coeffcient has a p-value below 5% how is then the pscore is calculated and does the pscore matching makes statistical sense?
2. Also, I estimate pscores in both T and C group and try to find overlap between them in both groups. Then it follows, that some individuals in C group have higher probability of receiving treatment than those who were actually treated. How is that possible?
3. Generally it is confusing that you need to calculate propensity score for treated individuals since they are treated and would have 100% probability of receiving the treatment. Where is the logic here?
4. How do we account for the fact that we don’t know when the subsidy was received in the three year period the data covers however the R&D expenditures are reported for just the most recent year?
5. Which matching type (kernel, nearest neighb with/ nneighb without replacement, caliper, etc) would you recommend for our case with sample size of 121 firms in the treatment group and 307 in the control group and most of coefficients being insignificant at 5% in probit?
6. In STATA 12 what commands would you recommend to obtain the significance level of the Average treatment effect?
7. How to evaluate what is satisfactory overlap of propensity scores, since if the overlap is small the method is unappropriate?
8. The dataset also has sampling weights. How to account for those in probit, t-tests and matching? How do they change the analysis and do we need to bother with them at all?
9. How should the variables be modified/transformed and which variables should be included as control variables to make more coefficients significant in probit and improve the overall matching quality?
10. What is common support and is it important to us?

Thank you very much in advance!

[Melissa Garrido]

Hi Kalvis,
If you're using Stata 12, you'll likely want to use the user-written commands: -psmatch2- and -pscore-.
I'd suggest you read the help files for both of these commands as well as some tutorials on propensity score matching (some suggestions are listed below) and then return to Statalist with any specific questions that remain.

The basic logic of propensity scores is that you are accounting for as many observed covariates as possible that you think are associated with both treatment receipt and outcome. The goal of propensity score matching is to create treated and comparison groups that are as similar as possible on observed confounders.
The p-value of a covariate in a probit regression has no bearing on whether the covariate should be included in a probit regression.

Here are some references to get you started:

Garrido MM, Kelley AS, Paris J, Roza K, Meier DE, Morrison RS, Aldridge MD. Methods for constructing and assessing propensity scores. Health Services Research. 2014; 49(5): 1701-1720. (Includes step by step instructions for assessing covariate balance and common support; appendix includes sample code)

Garrido MM. Propensity scores: A practical method for assessing treatment effects in pain and symptom management research. Journal of Pain & Symptom Management. 2014; 48(4): 711-718. (Discusses covariate selection for propensity scores)

Stuart EA. Matching Methods for Causal Inference: A Review and Look Forward.Statistical Science. 2010; 25 (1): 1-21. (Discusses different options for matching)

Hope this helps!
Melissa

[David Radwin]

I agree with Melissa Garrido and suggest these additional resources in no particular order:

Sekhon, J.S. (2009). Opiates for the matches: Matching methods for causal inference. Annual Review of Political Science, 12: 487-508.

Morgan, S.L., and Winship, C. (2014). Counterfactuals and causal inference (2nd ed.). New York: Cambridge University Press.

Stuart, E.A., and Rubin, D.B. (2008). Best practices in quasi-experimental designs: Matching Methods for Causal Inference. In J.W. Osborne (Ed.), Best practices in quantitative methods (pp. 155-176). Thousand Oaks, CA: Sage.

Imai, K., King, G., and Stuart, E.A. (2008). Misunderstandings between experimentalists and observationalists about causal inference. Journal of the Royal Statistical Society: Series A, 171(2): 481-502.

Guo, S., & Fraser, M. W. (2014). Propensity score analysis: Statistical methods and applications (2nd ed). Thousand Oaks, CA: Sage.