Bootstrapping difference

[Charlotte Rogers]

Hello,

I want to test whether the differential of two coefficients is significantly larger than the differential of other two coefficients. If I want to bootstrap this, do I need to bootstrap in the estimation of each coefficient and then perform the test or not bootstrap in the estimation of the coefficients and bootstrap in the test?

Thanks

[Carlo Lazzaro]

Charlotte:
I would say to bootstrap the test.
However, please consider the above as a tempative reply, due to the very limited details about your research goal.

[Charlotte Rogers]

To be more specific.

I run the same two regressions on different samples:
reg y x z [pw=w_f] , vce(cl id) /*sample 1*/
reg y x z [pw=w_f] , vce(cl id) /*sample 2*/

Then I calculate the differential of the x coefficient in the first regression and the x coefficient in the second regression

Then I run two other regressions on two other samples. And once again I calculate differential of the x coefficient in the latter two regressions.

Finally I want to test whether the first differential is significantly larger than the second differential. How can I bootstrap this test?

[Mike Lacy]

You would need to write your own program to feed to bootstrap:

// This presumes you have a variable in your data that marks each "sample" you want to analyze.
cap prog drop myregdiff
prog myregdiff, rclass
//
   reg y x z [pw=w_f] if sample == 1, vce(cl id) /*sample 1*/
   local b = _b[x]
   reg y x z [pw=w_f]  if sample == 2, vce(cl id) /*sample 2*/
   local diff12 = `b' - _b[x]
//
   reg y x z [pw=w_f] if sample == 3, vce(cl id) /*sample 3*/
   local b = _b[x]
   reg y x z [pw=w_f] if sample == 4, vce(cl id) /*sample 4*/
   local diff34 = `b' - _b[x]
//
   return scalar diff12_34 = `diff12' - `diff34'
end
//
bootstrap diff = r(diff12_34), ....... options ..... : myregdiff

This will give you a bootstrap standard error and confidence interval on "diff." I would not regard this as a test, since the standard error is not estimated under conditions that enforce a null hypothesis.

Regards, Mike

[Charlotte Rogers]

Thank you Mike!! When running "
bootstrap diff = r(diff12_34), ....... options ..... : myregdiff I recieve the following error: insufficient observations to compute bootstrap standard errors no results will be saved Do you have an idea why this could be?

[Carlo Lazzaro]

Charlotte:
this may be due to the fact that there's no variance in some of the items included in -bootstrap-.

[Charlotte Rogers]

Carlo: Thanks for your reply. I am following Mike's code. The bootstrap is performed to obtain a standard error and a confidence interval on a scalar. Could you be more specific with what you mean with "no variance in some of the items included"?

[Carlo Lazzaro]

Charlotte:
I meant that -bootstrap- procedure finds hard times in calculatng SE possibly due to too many reopeated observations in your simulation.
There are some interesting threads about the isue you're facing. The one I like the most is the following one: http://www.stata.com/statalist/archi.../msg01156.html.
You can access all of them by googling with the following string: -insufficient observations to compute bootstrap standard errors stata-.

[Mike Lacy]

I don't have any brilliant ideas here. I'd look at the size of the subgroups. If one of them is small, and if the variance of one or another variable is low in that group in the full sample, I can imagine a bootstrap sample happening to not have much variance.
You might try putting in a -tabstat- request (e.g., -tabstat x1 x2 ..., by(sample) statistics(mean sd)- ) at the top of the program, and then run -bootstrap- with the noisily option. That will create a lot of messy output, but might show you what's going on.

Regards, Mike

[Charlotte Rogers]

Mike, my problem could be that I have a panel. I copy my code in case it clarifies my issue:


xtset id

cap prog drop myregdiff
prog myregdiff, rclass

reg y x $correl i.sub [pw=w_f] if y2==1, vce(cl sch)
local b = _b[x]
reg y x $correl i.sub [pw=w_f] if y2==0, vce(cl sch)
local diff12 = `b' - _b[x]
//
xtreg y x i.sub if y2==1, fe vce(cl sch)
local b = _b[x]
xtreg y x i.sub if y2==0,fe vce(cl sch)
local diff34 = `b' - _b[x]
//
return scalar diff12_34 = `diff12' - `diff34'
end
//
bootstrap diff = r(diff12_34): myregdiff

The error I receive is:
insufficient observations to compute bootstrap standard errors
no results will be saved
r(2000);

[Mike Lacy]

I haven't used -bootstrap- with an -xt- estimation command. While I know that it can be done (see http://www.stata.com/support/faqs/st...th-panel-data/), I don't know if it can be done in a way that also fits with your -reg- command that does *not* treat the data as a panel. I don't know enough to be helpful with that problem, sorry.

Regards, Mike

[Charlotte Rogers]

Mike: I suspect that my problem is having more than one sample. When I run the same code only for one of the samples, then it works perfectly fine. Do you know how I can fix this problem?

[Charlotte Rogers]

So basically the question is whether STATA allows bootstrap from more than one sample

[Mike Lacy]

Per my earlier comment, no, I don't have any detailed knowledge about bootstrapping with an -xt- command.

I would also say that I don't understand your observation that "question is whether [Stata] allows bootstrap from more than one sample," since the code I provided you earlier manifestly does that. I think the conceptual issue here is that -xt- corresponds to one kind of sampling model, and non-xt estimation commands to another, so I don't see how both sampling models could be accommodated within one bootstrapping experiment. Again, bootstrapping with -xt- commands is outside my experience and likely outside my expertise; I am only pointing to what I see as a possible problem, and I don't have a solution.

Regards, Mike

[Laurence Lester]

I have used very similar code to that shown by Charlotte at #10 and at times get the same error as at #10. In my experience it is always a small-sample problem; if I increase the sample (artificially or otherwise) Stata completes the bootstrap.
Regards
Laurence