Significant T-test means difference

[Daniel Perez Parra]

Good morning everyone!! I'm Daniel, a master student in development economics that is currently struggling a bit with his master thesis (The effect of law on female education). This is my first post, don't hesitate to let me know whether I'm doing something wrong


Well, I have seen that many papers that use the pre-treatment diff in means using the command ttest var, by (group) unequal to start testing the exogeneity of an intervention (to test if results would be biased due to a selection issue). In my case p-values are significant (see the attached photo).

Does it mean that my results will be intrinsically biased? Is it okay to start en empirical research with these results?

Note: I'm comparing girls living in regions that have passed the law with regions that have not approved the law yet, capitalizing the staggered implementation of the law across Ethiopia. More control will be added to the table but so far I'm deeply worried of the t-test diff of my outcome variables.


Thanks in advance,


Daniel.

[Jared Greathouse]

The t-test doesn't really measure exogeneity, it measures balance between groups (in the language of propensity scores anyways).

Ideally, all your t stats would be insignificant, in fact, 0. What you've done here doesn't appear to be wrong.

[Daniel Perez Parra]

The t-test doesn't really measure exogeneity, it measures balance between groups (in the language of propensity scores anyways).

Ideally, all your t stats would be insignificant, in fact, 0. What you've done here doesn't appear to be wrong.

Thanks a lot for your response!!!

Well, actually what I did using the aforementioned ttest comand is to check whether the pre-treatment means between regions are significantly different. Being the null hypothesis H0: diff=0 and the p-value 0.03; we reject the null hypothesis and conclude that regions are significantly different before the treatment (right?). Having said that, I wonder what does exactly means in terms of 1) Selection Bias, endogenous timing of the reform? and 2) Pararell trends assumption. Kindly check an example of the P-value (1) for Educational attaintment, comparing early with late-adopters. Thanks

[Jared Greathouse]

Yes you're pretty much correct. With selection bias, it's hard to say.

But your post implies that your goal is to do difference-in-differences analysis. Since the policy took hold at different times, what i would do if I were you, if I would use csdid by FernandoRios, didmultiple_gt by Chaisemartin and d'Haultfoeuille.

There's a massive literature on staggered adoption, and everyone (including me) has their own opinions on how to go about addressing it. A really good paper you should read, is this one. They discuss validation of PTA and other matters you'd be interested in.

[Øyvind Snilsberg]

also, if you are doing (staggered) difference-in-differences, differences in the levels of outcomes need not be a threat to validity.