Bootstrap Confidence interval

[Evelyn Hilly]

Hi Stata users

I have a set of patient-level data evaluating the cost-effectiveness (CEA) of new treatment, A (n=400 patients) vs standard treatment, B (n=2000 patients).

The non-parametric technique of re-sampling from the patient-level dataset was used to obtain 95%CI.

The bootstrap result shows the following:

Step 1: . bootstrap r(cost_change) r(effect_change) r(icer) , reps(5000) nodots


Step 2: . estat bootstrap

Bootstrap results Number of obs = 1,983
Replications = 5000

command: icer
_bs_1: r(cost_change)
_bs_2: r(effect_change)
_bs_3: r(icer)


Observed Bootstrap
Coef. Bias Std. Err. [95% Conf. Interval]

_bs_1 1170.0921 -1.049031 891.3307 -570.8432 2897.692 (BC)
_bs_2 .06131481 .0009527 .04546892 -.0429747 .13592 (BC)
_bs_3 38103.532 -26603.11 1374521.7 -37622.84 3676503 (BC)

(BC) bias-corrected confidence interval


I reported the base-case ICER result obtained from the RCT and for the 95%CI, I've used (95%CI -37622.84 to 3676503) obtained from the bootstrap.

I was quizzed about the wide 95%CI. Isnt this the confidence interval we report for bootstrap estimates? or not?

Please advise. thanks

[Joseph Coveney]

Isnt this the confidence interval we report for bootstrap estimates? or not?

The bias-corrected bootstrap confidence intervals is one type that I've seen reported. Does it make sense to you that the bias for the icer coefficient is such a large proportion of its value? That you have 2400 patients, but the bootstrap command reports fewer than 2000?

[Evelyn Hilly]

hi Joseph - thanks for your kind reply. No, its a typo error on my part. I was trying to round it off. I don't think this has affected the estimates.


Should i report the wide 95%CI in my paper? or is this CI that should be cited/reported at all?

thanks and please advise anyone?!

[Carlo Lazzaro]

Evelyn:
when non parametric bootstrap is used to perform stochastic sensitivity analysis alongside clinical trials, there's no explicit guidelines about which 95% CI to report in the paper, as all the available post-bootstrap 95% CIs have their pros and cons (https://www.ncbi.nlm.nih.gov/pubmed/11113956).
If I have to report the 95% CI for incremental cost and effectiveness, I usually prefer the percentile bootstrap (https://www.ncbi.nlm.nih.gov/pubmed/11910068); that approach was never criticized by referees, but might show some drawbacks, too (https://www.ncbi.nlm.nih.gov/pubmed/11113956).

However, it is noteworthy that bootstrap 95% CI for the incremental cost-effectiveness ratio (ICER) is pretty meaningless unless you also report in your paper the joint density of the incremental cost and effectiveness bootstrap replications on the cost-effectiveness plane (https://www.ncbi.nlm.nih.gov/pubmed/2115096).

If you cannot rule out that the health care programme under investigation may be strongly dominated (ie, is more costly and less effective than the selected comparator), the bootstrap replications can include couples of incremental cost and effectiveness<0 that fall on the North West sector of the cost-effectiveness plane, but can be mistakenly interpreted as a sign of strong dominance (as the North West and the South East sectors of the cost-effectiveness plane both have minus sign, but for quite opposite reasons).

That’s why it is usually better to switch from ICER to the Net Monetary Benefit (https://www.ncbi.nlm.nih.gov/pubmed/9566468).

Eventually, I personally prefer cost-effectiveness acceptability curve (CEAC) and frontier (CEAF) to show the uncertainty surrounding the base case estimate of the ICER (https://www.ncbi.nlm.nih.gov/pubmed/11747057).

Recently, I‘ve had the chance to convey most of this stuff in a Markov-model supported cost-utility analysis (in that case, the uncertainty surrounding the base case estimate was obviously addressed via a Monte Carlo simulation instead of an [unfeasible] non-parametric bootstrap, but cost-effectiveness plane, CEAC and CEAF were reported as well); with a bit of shameless self-promotion, I refer to it, if interested (https://www.ncbi.nlm.nih.gov/pubmed/29641931).