Difference-in-differences with attrition

[Nicolas Orgeira]

Hi,

I would like to run a Difference-in-differences regression and I'd appreciate some advice regarding methodology.

The dataset I am working with has a high level of attrition. When running a DID, I get two different coefficients depending on whether or not I only keep respondents who took part in both phases of the study. I was wondering which one was methodologically correct. Should I run the DID regression but only keep the respondents who took part in both periods? Or keep the attriters in the DID regression?

Out of precaution, I tested whether there was attrition bias and added Inverse-probability weights for each respondent with complete data.

Thank you

[Clyde Schechter]

This is a problem with no truly satisfactory solution.

Keeping only those who remained in the study for both phases is a biased sample, as you demonstrated by getting different results when you omitted those who dropped out. As for testing for attrition bias, the only tests you can do are based on things observed prior to the attrition. But the real concern from attrition is whether those entities would have differed after the point of attrition had you been able to observe them. It is the unknowable missing data that is the real concern, and there are no tests you can do that will tell you anything about that.

The first thing you should do at this point is try to gain an understanding of why the attrition occurred. To some extent comparing characteristics of completers and non-completers may give you some clues. But that is not enough. You need to think, in real world terms, about the mechanisms and reasons driving attrition, and what those mechanisms and reasons imply about the data that you couldn't observe. These considerations may enable you to construct a qualitative model of the attrition process.

If you can find convincing reason to believe that the attrition was due to an entirely random mechanism (i.e. completely unrelated to the missing data) then you can just analyze the data you have and not worry about it. This is seldom the case, however.

A slightly more likely possibility is that although the forces that drive attrition are related to the values you would have observed for the missing data, you may be able to remove or strongly attenuate that relationship by conditioning on other data that you have observed. This situation is referred to as missing at random (MAR), and in that case you may be able to apply multiple imputation to salvage the situation.

If this condition does not hold, then you have data that are missing not at random (MNAR), and in the absence of a good quantitative model of the missing data (which is seldom possible) there is pretty much no hope of getting a trustworthy analysis. What you can do in that situation is a sensitivity analysis trying different scenarios of assumptions about the missing data, and, with luck, you might find that at least some broad qualitative or semi-quantitative conclusions will hold under all, or nearly all, reasonable scenarios about the missing data.

[Nicolas Orgeira]

If you can find convincing reason to believe that the attrition was due to an entirely random mechanism (i.e. completely unrelated to the missing data) then you can just analyze the data you have and not worry about it. This is seldom the case, however.

Clyde, thank you very much for your insight and for your very informative response.

To test for attrition bias, I performed a probit test and a BGLW test. Despite some signs of attrition bias, given that the pseudo-R2 is relatively low (baseline variables and indicators explain about 7% of panel attrition), I believe that attrition is random. The country went through an epidemic between the two data collection, killing a significant share of the population and leading to many displacements.

Given this, would you recommend to carry the analysis without the attriters? I feel that keeping the attriters would provide an estimator similar to an Intent-to-Treat estimator (at the policy level) whereas removing attriters would be closer to an Average-Treatement Effect. I was wondering how studies experiecing high levels of attrition present the results - with or without attriters.

My sincerest thanks for any thoughts that you would be willing to share.

[Clyde Schechter]

Well, under the circumstances, the notion that attrition might be completely random seems plausible, assuming that your outcome variable is not health related!

Your characterization of the analysis that retains all the data as being something like an intention to treat estimator seems right. For most purposes, that is the effect that people are interested in and my inclination would be to present that one. In a situation where there is a large amount of missing data, as here, it is usually good to present some sensitivity analyses that handle missing data differently as well. Even though you are persuaded that the epidemic accounts for the attrition and is exogenous, some readers might disagree, and it would be good to try to convince them that whatever findings you have are, at least qualitatively, reasonably robust to different assumptions about the missing data.

I don't see much value in an analysis that excludes the people with incomplete data. At best, assuming you are correct that the missingness is exogenously generated and is ignorable, you are just throwing away information and reducing the precision of your estimates. At worst, you are introducing a potentially very large bias into the analysis. Remember, too, that you cannot make any generalizations outside the sample if you exclude the people with missing data because outside your existing study you cannot identify who is part of the population if you do that: it is not possible to predict who will and who will not die or be displaced by a future epidemic!

[Nicolas Orgeira]

Thank you very much Clyde, this is very helpful. I will follow your sound recommendations and keep the attriters in the analysis.