Bootstrap number of replications

[Sergiy Radyakin]

Dear Statalisters,

I am looking for references similar to

Efron, B., and Tibshirani, R. "Bootstrap Methods for Standard Errors, Confidence Intervals, and Other Measures of Statistical Accuracy", Statistical Science, Volume 1, Issue 1 (Feb., 1986), 54-75.

which describe number of required bootstrap replications for various statistics. Especially valuable would be ones that analyze it with regards to modern non-parametric methods like matching, as well as other where no analytical derivation of the standard errors and other statistics is possible.

Similarly of interest would be to hear which rules members of Statalist apply in practice to determine the number of required bootstrap iterations.

Finally, if there appears to be an estimator for which, say, 1000 replications are not practically sufficient, please mention it.

Thank you in advance for your suggestions. Sergiy

[Carlo Lazzaro]

Sergiy:
- my favourite application of bootstrap method is for bootstrap ttest and confidence intervals (CIs). I follow the rule of 1,000 or (more often) 10,000 replications. As far as my resarch field is concerned (economic evaluation of health care programmes), the guidance on (at least) 1,000 replications for CIs is reported in: Heyse JF et al. Statistical consideration in analysing health care resource utilization and cost data. In Drummond MF, McGuire A. Economic evaluation in health care. Merging theory with practice. Oxford: Oxford University Press, 2001: 230-232. The same source recommends bias-corrected and accelerated bootstrap CIs.

Kind regards,
Carlo

[Maarten Buis]

If you do a bootstrap test, i.e. use the bootstrap to compute p-values rather than confidence intervals, a 1,000 replications is often not enough. Say you find a p-value of 5%, then that is based on only 50 replications in which the statistic in the bootstrap sample was more extreme than the statistic in the actual sample. As a rule of thumb I use 20,000 replications for a bootstrap test, so that a p-value of 5% corresponds to 1,000 replications in which the the statistic in the bootstrap sample was more extreme than the statistic in the actual sample. You can also compute the Monte Carlo confidence interval to quantify how much uncertainty there is left due to the randomness in the bootstrap procedure. A 95%MCCI is just based on the 2.5th and 97.5th quanitles of the binomial distribution with parameters the number of replications and the number of rejections.

Untill now I just assumed that I computed the p-value as the proportion of samples in which the statistic in the bootstrap sample was more extreme than the statistic in the actual sample. You can be a bit more fancy than that, and for programs like asl_norm and oparallel (both available from SSC) I have done so, but if the number of replications is large enough it does not matter. It certainly does not matter for the specific question by Sergiy.

[Sergiy Radyakin]

Thank you very much for your recommendations, Carlo and Maarten. They are very helpful. Sergiy.

[Tom Jefferson]

There was an article by Andrews and Buchinsky 14 or 15 years ago in Econometrica that described a method to determine the "optimal" number of bootstraps to use. Brian Poi wrote a command called bssize that implements it. Type in

findit bssize

to find it. I used the command a few times, but since most of the estimators I use are relatively fast, I just arbitrarily use 5,000 bootstraps most of the time. The more the merrier.

Tom

[gllammuser]

Deal All,
I am trying to bootstrap using model-2 of fmm in STATA (the model in which probabilities are set to be function of variables). Every time I try, I get this message:

"provide starting values or estimate 2 component Normal model with constant component probabilities an error occurred when bootstrap executed fmm"

I have already tried running model-1 first (constant probabilities) and then run bootstraping on model-2 based on that, it doesn't seem to work. The bootstrap works fine for model-1 but seems to have problems running model-2. I would really appreciate it if you can help me with that.

Thank you very much for your time.
Aidin

[Carlo Lazzaro]

gllammuser:
-as per FAQ, please note the preference on this forum for full real names;
-as per FAQ, please do not start out a new thread under an old topic.

Kind regards,
Carlo