[sanjay nawaz]

Okay I now doing the trip diff algebra

Y=A + B*RAJASTHAN + C*YOUNG + D*POST + E*RAJASTHAN*YOUNG + F*RAJASTHAN*POST + G*YOUNG*POST + H*RAJASTHAN*YOUNG*POST

DDD estimator is: [YoungRajasthanPOST-YoungRajasthanPRE]-[YoungGujaratPOST-YoungGujaratPRE]-[OldRajasthanPOST-OldRajasthanPRE]

Which is equal to [(A+B+C+D+E+F+G+H) - (A+B+C+E)]-[(A+C+D+G)-(A+C)]-[(A+B+D+F)-(A+B)] which equals:

A+B+C+D+E+F+G+H-A-B-C-E-A-C-D-G+A+C-A-B-D-F+A+B=(A-A-A+A-A+A)+(B-B-B+B)+(C-C-C+C)+(D-D-D)+(E-E)+(F-F)+(G-G)+H=H-D which does not equal the DDD coefficient of H.

Where am I going wrong?

[Clyde Schechter]

You're not going wrong. Your algebra is correct. But your interpretation of the DDD is incorrect. The DDD is not represented by H. H can be interpreted in several ways, but all of them are differences in effects between some groups. Perhaps the most salient interpretation of H in your context is that it is the difference between the effect of passage into the post period on the young (presumably you want to attribute this to the law) in Rajasthan and the difference between the effect of passage into the post period on the young in Gujarat. But that is not the DDD.

Here's how I would think about this. If you were to look just at the difference in outcome among the young in Rajasthan and the young in Gujarat, taking nothing else into account, you would like to call this the effect of the intervention. But you can't because this difference is "contaminated" with other effects: the mere passage of time may be associated with a change in the outcome that would have taken place whether there was a new law or not. Hence the DD estimator which subtracts out the observed change in outcome among the young in Gujarat. But we remain concerned that we have not yet eliminated all of the "contamination." So it is possible that there are other things going on in Rajasthan that are distinctive from those going on in Gujarat but also have nothing to do with the law. Since the law applied only to the young, we could estimate any distinctive Rajasthan effects other than the law by looking at the outcome difference among the young in Rajasthan (in theory--I question whether this is reasonable in your context). So we subtract that out too. Your DDD is structured in exactly this way. The first [] is the difference between pre and post among the Young of Rajasthan. The second [] is the corresponding change in Gujarat, and the third [] is the corresponding change among Rajasthan's old. So the DDD works exactly in this way: it starts with the total observed change and then subtracts out two different estimates of contaminating effects.

The reason I am skeptical of the validity of the DDD estimator here is that I don't think that the third component is applicable in your study. Although I would defer to the opinion of an expert in the field (this is way out of my field), as an educated lay person I find it difficult to believe that the change in savings behavior among the old in Rajasthan (or anywhere else) is a reasonable estimate of the changes that would have been seen among the young there in the absence of the law. From what I have observed in life, age is perhaps the single strongest determinant of savings behavior. If the outcome of your study were something that doesn't vary much with age, then I would accept the DDD estimator here.

[sanjay nawaz]

Ok I think I have cleared up my confusion. Basically, the DDD is:

[(YoungRajasthanPOST-YoungRajasthanPRE)-(YoungGujaratPOST-YoungGujaratPRE)]-[(OldRajasthanPOST-OldRajasthanPRE)-(OLDGujaratPOST-OLDGujaratPRE)]

There should be 8 terms and not 6 as the DDD is simply differencing 2 DD estimates. So in my model I have:

DDD=DD_1 - DD_2

My plan is now:

1. To run DD_1 on the YOUNG=1 sample
2. To run DD_2 on the YOUNG=0 sample

Hopefully 1 shows significant results, and 2 shows insignificant results. Then I combine the two into the DDD estimate. Is this a correct way of proceeding?

Are there any more DD to conduct before I do the DDD? For example; ALLGujarat vs ALLRajasthan, or YoungGujarat vs OLD gujarat? I believe neither of these are necessary and my original plan should be working?

Apologies again for length

[sanjay nawaz]

Okay sorry clyde I posted this before I read your post.
I have read your post, and I think I understand it. My question is that:

a) What is the DDD estimate in my equation if it is not H? I have basically coded that into STATA and have been interpreting H in the results as the DDD coefficient. Is my equation wrong? I believe this is where we need to start using the margins command in STATA?

b) Reading lecture notes online have led me to believe the correct specification is given by my previous post (#33). Basically in that equation, H would be my DDD estimate. Correct? If so, what is the disadvantage/advantage of using DDD in that way?

Again what is your opinion on my sub-sample approach?

sending you million thanks Clyde

[Clyde Schechter]

Sanjay, your calculation

DDD estimator is: [YoungRajasthanPOST-YoungRajasthanPRE]-[YoungGujaratPOST-YoungGujaratPRE]-[OldRajasthanPOST-OldRajasthanPRE]

Which is equal to [(A+B+C+D+E+F+G+H) - (A+B+C+E)]-[(A+C+D+G)-(A+C)]-[(A+B+D+F)-(A+B)] which equals:

A+B+C+D+E+F+G+H-A-B-C-E-A-C-D-G+A+C-A-B-D-F+A+B=(A-A-A+A-A+A)+(B-B-B+B)+(C-C-C+C)+(D-D-D)+(E-E)+(F-F)+(G-G)+H=H-D

is correct. The DDD estimator is not H, it is H-D.

[sanjay nawaz]

Okay thanks clyde,

And your opinion on my specification in post #33?
Particularly the running of different DD on the two samples and then combining it for final DDD?
I ask because all lecture notes in economics are claiming DDD estimate is one in post #33,
This is notes from jeff wooldridge on page 6

https://www.cemmap.ac.uk/resource/id/imbens_wooldridge/slides_11.pdf

and some others

http://econweb.rutgers.edu/killings/Econ_520/difference_models.pdf

[Clyde Schechter]

Sanjay, I'm can't advise you. We have a bunch of different estimators floating around here. The 8 term estimator is another one that eliminates yet another source of "contamination." The choice among them is not a statistical question, it's a substantive scientific question.

I know it can feel comforting to use approaches that are cited in the basic textbooks. But all statistical formulas, no matter how solid their mathematical pedigree, run into limitations when you apply them in the real world. I will again emphasize that I am not an economist, but I have lived almost 70 years now and known a lot of people of all ages, and I know something about their savings habits. To my mind, there is no credibility to the notion that old people can serve as controls for young people when the outcome is savings. So my perspective has been all along, and remains, that the simple DD estimator calculated only from the data on the young is the most credible estimator in your situation. I do not see the data from the old as being helpful here. So I would calculate only what you call DD_1 in #33 and stop there. But perhaps things are different in India, and in any case, perhaps there is literature showing that my personal observations about savings as a function of age are simply not true beyond the limited sphere of informal observations I have made. I think that to resolve this you need advice from people with real expertise about savings habits. Your problem is not mathematical and not statistical: you have all of that straight at this point. Your question is substantive.

There is one statistical point lurking in #33 that I will make. The difference between statistically significant and not statistically significant is not, itself, statistically significant. If you are going to use DD_1 - DD_2 as your estimate of the effect of the law, it is entirely irrelevant whether DD_1 or DD_2 are, separately statistically significant or not. Just focus on DD_1 - DD_2 and its confidence interval.

[sanjay nawaz]

Yes clyde I understand your point about savings behaviour, it's just that I have a number of other outcome variables that I will also implement. But your main point is this I think:

In any DD framework, treatment and control groups have to be similar. So for example If I can restrict the `OLD' in my study to be those only maybe 5 years older than the YOUNG in the study, it could be used as a satisfactory (conceptually at least) control group. Further, you are arguing that if I do a DDD design, then there is no point in conducting any other DD because they will tell me nothing about my DDD, as DD will be a different model.

I will implement a DD now, and maybe go for DDD and only use OLD=YOUNG+5 or something similar if I have the times.

[Clyde Schechter]

Yes, we are in agreement now.

[sanjay nawaz]

Hi

I am a bit confused about the interpretation of the DD

If suppose I find that post LAW rajasthanis increased savings by 10,000

does this mean that over the entire post-law period they increased savings by 10,000 over gujuratis. And if suppose my post period is 10 years, does that means on avergae they are increasing it about 1000 a year ?

[Clyde Schechter]

As I understand your data, you have periodic (annual?) observations on the two cities, and that the dependent variable, SAVINGSUSD, is some aggregate measure (mean, total, median--I don't think you ever said) for (the young in ) that city in that year. I also do not recall seeing in this thread any explanation of exactly what the SAVINGSUSD variable represents--in particular, whether it is measuring accumulated savings in hand (stock) or new savings accumulated in the current year (flow). Subject to those areas of un-clarity, your model of this looks something like:

xtreg SAVINGSUSD i.Rajasthan##i.pre_post other variables, fe
margins Rajasthan#pre_post
margins Rajasthan, dydx(pre_post)

So if the output of the -margins, dydx()- command shows the marginal effect for Rajasthan to be +US$10,000, then that would mean that, adjusted for the other variables in your model, the (mean, total, median, whatever it was) savings, averaged across all years, increased by $10,000. One could not, from that, conclude that they are increasing about $1000 a year. If the outcome variable represents stock of savings, that is one possibility, but the findings would be equally consistent with a one-time jump of $10,000 that was sustained, or no increase at all in the first 5 years and a $20,000 increase in the final five years, etc.

[sanjay nawaz]

I think "averaged across all years is crucial here". So Rajasthanis over my entire period increased their savings. Yes it is annual data, so basically rajasthanis over the entire period increased their savings, and not per year. I know this is a basic question, but I have been getting different advice.
Also, what is the advantage of pooled vs panel data here? Basically I had a huge pool of people and then deleted all observations for which I did not have at least one observation either side. This of course reduced the total number of observations, so sometimes I think maybe I should have just used the pooled cross section data.

[Clyde Schechter]

I know this is a basic question, but I have been getting different advice.

My advice is conditional on the assumptions about your data set out in your post. In particular, whether it is per year or not depends on whether the variable SAVINGSUSD measures, in each observation, total accumulated savings to that point (stock) or incremental savings over that year (flow). I do not recall that you have ever stated anything about that one way or the other. If the variable is stock of savings, then, yes, it is definitely an increase over the entire period. But if the variable represents incremental savings over that year, then the interpretation would be per year.

This of course reduced the total number of observations, so sometimes I think maybe I should have just used the pooled cross section data.

In my view, it is not really as if you can simply choose between using pooled OLS and panel estimators with observational panel data. With observational panel data, you should use panel estimators: pooled OLS is a mis-specification and will usually introduce serious omitted-variable bias. There is one situation where you can properly use OLS with panel data (though it hardly ever happens in real life): if there really is no variation at the panel level. You can tell if you have that situation easily enough because the final lines of output from -xtreg- show the values of sigma_u, rho, and an F-test that all sigma_u are zero. Only when rho is very close to zero and the F-test does not reject that null hypothesis does the OLS estimator become valid. But if rho is appreciably greater than 0, or if that F-test rejects the null hypothesis, then the OLS approach is an inappropriate model, in my opinion. The choice you can make when dealing with panel data is not OLS vs FE but RE vs FE.

[sanjay nawaz]

Thanks for that Clyde

I have the following question. There is another interesting situation I am aware of, although in different states. Basically one state has implemented a program, whilst another state has not. This might make a DD design useful. The entire point of the program is that people who are selected for it, learn how to increase savings. In order to be part of the program however, people need to signal ability to save. Basically program runners look at ability to save and decide whether or not they want the person in the program. The issue is that I don't have data on which person was part of the program, but am interested in signalling incentives this program generated. Obvious problem is that If I see an increase in savings it might be because of signalling or maybe because of the program. Is the following an approximate solution to this:

Split the post-program period in two, and delete all observations in the second half. This reduces the liklihood that a person in my dataset has joined the program and so maybe the training effects might be less? Then run a simple DD. At the moment I have done this and I am getting similar results to when I use the entire period, and when I only use the first half of the post program period.

Any comments are welcome.

[John lenon]

A basic question that I am trying to figure out but I am still not sure. So I thought someone here can help me with these questions.

1) areg y treatment i.year##i.state, vce(state)

2) areg y treatment c.year##i.state, vce(state)


Let's suppose I am studying the impact of school policy implemented at the state level in different time periods (staggered setup). What will this model do in terms of this question?
Is there a difference between the two models? How should I interpret this interaction term, what is the model exactly doing? Some results are totally different when I use c or i in the year. Also, am I including the year fixed effect in the 2nd model, because the value is for just one term "year". So I am confused as the results are also different when I use these different models.

Also when I use c.year##i.state my results are much better. What can be the reason I can look for?

Thank you