Balance check in Comparative Interrupted Times Series

[Zhao Kai]

Hi there,

I am conducting a research that evaluate the differential policy effects on students based on their exemption status. Some students became exempt from placement tests because of the policy while non-exempt students still need to take placement tests and may be assigned into remediation classes based on their performance.

The outcome variable is a binary indicator of whether a student passed a college-level course within the first year. Post represents post-policy cohorts. We have six cohorts of students, with Cohort 2011-2013 entering college before the policy and Cohort 2014-2016 entering college after the policy. We used a Comparative Interrupted Time Series design to examine how exempt and non-exempt students may benefit differently from the policy. I include a number of control variables (Student Backgrounds in the equation), including gender, race, age, and free lunch status. Below is the equation I modelled:

Logit (y_ijt)= β₀+ β₁(Post) + β₂(Exempt)_ijt+ β₃(Post*Exempt)_ijt+ β₄(S)_ijt + ξ_j+ λ_t


Now one reviewer asked me to prove that the composition changes for exempt and non-exempt students are similar after the reform. She suggested that I replace the outcome variable in the equation with control variables (e.g., gender, race….) to see if the coefficients for the interaction term are significant. Insignificant coefficients would provide evidence for similar composition changes for exempt and non-exempt students. I ran the analyses and found that in many cases, the coefficients were significant. The changes in the distribution of age and race were different after the reform for exempt and non-exempt students. Any additional analyses I can ran to address the reviewer's concern?

[Jared Greathouse]

I love a good theoretical stats question. In the world I work in (I guess the broader econometrics world), we'd call this a structural break or, maybe in this case, perfect collinearity of common factors in the pre-intervention period.

That is, assume we have a data generating process of unit-level common factors which affect a set of factor loadings over time. Together, these produce the outcomes we see for interventions under control. One main assumption, particularly with time series data, is that there's no perfect collinearity of common factors. Or, put a bit differently, there's no major change in the post-policy period that you didn't measure before the intervention.

In a paper I'm doing right now, I'm assessing the causal impact of mass vaccination events on COVID-19 cases in Puerto Rico. I originally wanted to study from March 2021 to August 2021, but then I remembered that Delta became a thing on May 20th, 2021- thus, a more contagious variant that didn't exist in most of the pre-intervention periods now exists in some post-intervention periods. Its higher contagiousness produces a structural break in the data generating process of the time series.

So what did I do? I limited my sample of treated units to those before May 20th, 2021. And, it looks like you may need to do the same thing as the case may be. If the composition of these cohorts changes in the post-intervention period, you either must control for this somehow or limit your sample to more stable cohorts- otherwise, we'll end up over or underestimating the effect of interest.

[Zhao Kai]

Thanks Jared. I am also thinking of running some subgroup (by gender, race, and age) analyses to see if the results still hold.