p-values and confidence intervals from bootstrapped t-test

[Matthias Greiff]

Dear colleagues,

I want to test whether the mean of a distribution is significantly smaller than zero. Since the data is not normally distributed, I don't want to use a standard t-test. Rather, I would like to do a bootstrapped version of the t-test. I do this using the command

bs "ttest d_certainty_uncertainty=0" "r(mu_1) r(p)"

where d_certainty_uncertainty is the name of the variable. As result, I get confidence intervals and a p-value. Here, my first question is how the p-value is computed. What puzzles me is the following:
- the p-value is always the same, no matter how many replications are used, confidence intervals are different
- the p-value is exactly the same as the p-value from the standard t-test

Also, is it possible to plot or tabulate the data from the bootstrapped confidence intervals? My idea is to run 1000 replications, which gives me 1000 mean values (i.e., the means of the 1000 bootstrapped samples). Then, I could get the p-value by just counting how many out of these 1000 values are larger than zero. Any ideas on how this can be done?

Below is my output: First, there is a standard t-test, then the bootstrapped t-test with 2 replications and the bootstrapped t-test with 1000 replications.

Many thanks for your help in advance.

Best

Matthias

STATA OUTPUT


. ttest d_certainty_uncertainty==0

One-sample t test
------------------------------------------------------------------------------
Variable | Obs Mean Std. Err. Std. Dev. [95% Conf. Interval]
---------+--------------------------------------------------------------------
d_cert~y | 96 -3.026078 1.361934 13.34417 -5.729858 -.3222981
------------------------------------------------------------------------------
mean = mean(d_certainty_uncertainty) t = -2.2219
Ho: mean = 0 degrees of freedom = 95

Ha: mean < 0 Ha: mean != 0 Ha: mean > 0
Pr(T < t) = 0.0143 Pr(|T| > |t|) = 0.0287 Pr(T > t) = 0.9857

. bs "ttest d_certainty_uncertainty=0" "r(mu_1) r(p)", reps(2) nodots nowarn

command: ttest d_certainty_uncertainty=0
statistics: _bs_1 = r(mu_1)
_bs_2 = r(p)

Bootstrap statistics Number of obs = 96
Replications = 2

------------------------------------------------------------------------------
Variable | Reps Observed Bias Std. Err. [95% Conf. Interval]
-------------+----------------------------------------------------------------
_bs_1 | 2 -3.026078 -1.429353 .7483916 -12.5353 6.483139 (N)
| -4.984624 -3.926239 (P)
| . . (BC)
_bs_2 | 2 .028664 -.025153 .0022792 -.0002963 .0576244 (N)
| .0018994 .0051227 (P)
| . . (BC)
------------------------------------------------------------------------------
Note: N = normal
P = percentile
BC = bias-corrected

. bs "ttest d_certainty_uncertainty=0" "r(mu_1) r(p)", reps(1000) nodots nowarn

command: ttest d_certainty_uncertainty=0
statistics: _bs_1 = r(mu_1)
_bs_2 = r(p)

Bootstrap statistics Number of obs = 96
Replications = 1000

------------------------------------------------------------------------------
Variable | Reps Observed Bias Std. Err. [95% Conf. Interval]
-------------+----------------------------------------------------------------
_bs_1 | 1000 -3.026078 .0299752 1.405738 -5.784616 -.2675405 (N)
| -6.025728 -.4036439 (P)
| -6.152806 -.4625959 (BC)
_bs_2 | 1000 .028664 .0812439 .1836033 -.3316283 .3889564 (N)
| .0001762 .7139603 (P)
| .0001446 .677225 (BC)
------------------------------------------------------------------------------
Note: N = normal
P = percentile
BC = bias-corrected

[Marta Garcia-Granero]

Hi Matthias:

I think that you need a resampling test, not a bootstrap CI

Regards,
Marta

[Maarten Buis]

What you do is not a bootstrap test. You can see how to get p-values by looking at example three (achieved significance level) of the manual entry on bootstrap.

[Matthias Greiff]

Marta, Maarten, thanks for your answers.

Shouldn't it be possible to compute the p-value form the confidence interval, using the method I described?

Maarten, with regard to example three, it is not clear to me how the code should be modified. My original data is paired data from two treatments. In the example above I computed the difference and tested whether the (mean of) difference is significantly smaller than zero. I can use my original data from both treatments and modify the code from example 3. However, I'm not sure if my modifications are correct. It would be really nice if you could take a look at it. Thanks a lot!



STATA CODE / Example 3 from bootstrap, modified to paired t-test

keep svo_certainty svo_uncertainty
ttest svo_certainty == svo_uncertainty
scalar tobs = r(t)
summarize svo_certainty, meanonly
scalar mean_certainty = r(mean)
summarize svo_uncertainty, meanonly
scalar mean_uncertainty = r(mean)
scalar omean = 0.5 * mean_uncertainty + 0.5 * mean_certainty
*summarize svo_certainty, meanonly
replace svo_certainty = svo_certainty - mean_certainty + scalar(omean)
*summarize svo_uncertainty, meanonly
replace svo_uncertainty = svo_uncertainty - mean_uncertainty + scalar(omean)
set seed 1
bootstrap t=r(t), rep(1000) saving(bsauto3) nodots: ttest svo_certainty = svo_uncertainty
use bsauto3, clear
generate indicator = abs(t)>=abs(scalar(tobs))
summarize indicator, meanonly
display "ASLboot = " r(mean)

[Matthias Greiff]

And the achieved significance level is 0.031.

[Clyde Schechter]

This isn't directly responsive to the original question, but I would also note that the syntax of the bootstrap command there is quite out of date. See -help bootstrap- for the modern syntax, and many options that weren't available in the past (though they may not be useful in this particular situation).

[Steve Samuels]

You cannot obtain a correct p-value from a bootstrap confidence interval. The reason is that a p-value is a probability computed under the null hypothesis of no difference. The bootstrap estimates that difference, under the "alternative", if you will. See page 10 of the Stata 13 manual entry for the bootstrap:

[Carlo Lazzaro]

Matthias:
you may be interested in taking a look at the following article: http://www.ncbi.nlm.nih.gov/pubmed/10180748