Propensity scoring for multilevel, longitudinal data

[salvear]

I am using data from a school district with the intent to measure the impact of Programs A and B on student achievement growth between kindergarten and grade 4. However, I need to address the issue of selection bias, as more economically advantaged students tend to enroll in Program B. Also, not all campuses offer both Program A and Program B.

I'm interested in pursuing a form of propensity scoring, but I'm not sure how to use teffects for multilevel data. My data are longitudinal, and the outcome is achievement growth across five years. Previously, I used a cross-classified multilevel growth model because some students switch schools over time and this interrupts the nesting of students within campuses. I have a theoretical framing for how to approach propensity scoring based on Thoemmes and West (2011), which is multilevel modeling with broad inference space, conditioning across clusters (estimation of propensity scores are based on fixed effects in the model). But, I am not sure how to program this in Stata. Any assistance is greatly appreciated!

[Sebastian Geiger]

I think -teffects psmatch- is not able to explicitly account for multilevel structures. The best thing you can do is (probably) to use dummy variables for the school districts in the propensity score estimation.

You may set up loops estimating the propensity score for each school district separately in order to make sure that only students from the same district are matched. However, this involves some coding as well as some methodological issues.

Another option would be to use an alternative matching method like coarsened exact matching, where you can choose exact matches for the indicator variable of the school districts.

Edit: I was a little fast with my response. I though you were referring to propensity score matching (!). Of course, the propensity score can also be used as a weight within other models (see for example: the different command options of teffects). The propensity score may also be used with models which are designed to deal with multi-level data. However, I'm not an expert in this field. I will read your cited paper later, and maybe I come up with some additional ideas.

[salvear]

Thank you for your response. I have started reading up on teffects inverse-probability-weighted regression adjustment (IPWRA). Again, there's nothing that I've read to explicitly account for the multilevel aspect of my data (nesting within student over time, between students, across campuses). The data are from just one school district, but I can certainly apply your comments to the school level.

[Sebastian Geiger]

I still didn't have time to read the paper you cited. Therefore, I don't know exactly what the authors are doing, but with inverse-probability-weighted regression adjustment you can, at least, use the level variables as dummies (e.g. i.campus) in the treatment and the outcome level. In doing so, you allow for fixed effects at the different levels. Theoretically, this would also be possible with individual-fixed effects. However, I guess the required number of dummy variables would be a little to high.

Again, I would not consider myself an expert in this field, but you may consider obtaining the propensity score from -teffects- or alternative commands like -psmatch2- (psmatch2 provide a more convenient way of storing the propensity score), and then you can use fixed or random effects models (e.g. xtreg, fe or xtreg, re) with the estimated propensity score as a weight. This is essentially also what -teffects ipwra- does but without the possibilities that xtreg provides with respect to multilevel data. With respect to the estimation of the correct standard error, it may be an issue that the propensity score is estimated, but from may practical experience I would conclude that it does not matter very much as long as your sample is quite large.

Another problem you probably will encounter is that it may be inappropriate to assume that the students can choose between the programs in every year of your sample. If the students choose the program at the beginning, my initial idea is t to run the propensity score model only for the first year and spread the estimated values to all observations of the same individual. Subsequently, you can run the (multilevel) regression.

[Ben Lazrus]

Just wondering if there been progress on this? e.g. using teffects with mixed models