Calculate p value for bias-corrected confidence intervals after bootstrap

[Hannes Gerstel]

Dear Statalist members,

I struggle with calculating p-values for bias-corrected confidence intervals after I run a mediation analysis with bootstrapping. Unfortunately, Stata 17 only shows z and p-values for normal based confidences intervals and not for bias-corrected CIs. The issue should be applicable to other statistical models using bootstrap (e.g., regressions, see this thread for basically the same issue).

Mediation with 60 bootstrap replications showing an insignificant link between Mediator 1 and Mediator 2 based on normal confidence intervals (M1 -> M2, relevant line is colered in red)

. sem (X -> Y, ) (X -> M2, ) (M2 -> M1, ) (M1 -> Y, ), vce(bootstrap, reps(60) seed(2) dots(1)) level(90) nocapslatent
(running sem on estimation sample)

Bootstrap replications (60)

Structural equation model                                  Number of obs = 114
Estimation method: ml                                      Replications  =  60

Log likelihood = -937.89911

------------------------------------------------------------------------------
             |   Observed   Bootstrap                         Normal-based
             | coefficient  std. err.      z    P>|z|     [90% conf. interval]
-------------+----------------------------------------------------------------
Structural   |
  -----------+----------------------------------------------------------------
  M1         |
          M2 |   2.080746   1.334327     1.56   0.119    -.1140266    4.275519
       _cons |   60.87975   4.511078    13.50   0.000     53.45969    68.29981

------------------------------------------------------------------------------

Bias-corrected confidence intervals for the above analysis, showing that the link between M1 -> M2 should be significant at the 10% level (colered in orange)

. estat bootstrap, bc

Structural equation model                       Number of obs     =        114
                                                Replications      =         60

------------------------------------------------------------------------------
             |    Observed               Bootstrap
             | coefficient       Bias    std. err.  [90% conf. interval]
-------------+----------------------------------------------------------------
M1           |
          M2 |   2.0807463   .1086629   1.3343272    .0238429    4.09717  (BC)
       _cons |   60.879751    .078569   4.5110779    53.49619   68.20156  (BC)

------------------------------------------------------------------------------
Key: BC: Bias-corrected

60 replications are simply chosen for illustration purposes, since the normal based CI p-value is above 10% in this case, whereas the bias-corrected p-value should be < 10%.

I tried the recommendation in this older thread by taking the observed coefficient minus the bias to calculate a z value and then taking 2*normal(-abs(‘z’)) for the p-value. However, this yields a p-value of 0.139 when it actually should be < 0.10 [using di 2*normal(-abs((2.0807463 - .1086629) / 1.3343272))] since the bias-corrected confidence interval range does not contain zero at the 90% confidence level.

Bruteforcing the confidence level shows that the p-value should be between a significance level of 90.4 and 90.5%.

I assume that the solution goes into the direction of this paper, which currently is above the scope of my knowledge.

For brevity, I didn't attach the dataex code. If it helps, I can attach it.

Many thanks for your valuable help,
Hannes

[Carlo Lazzaro]

Hannes:
you might find the following article interesting, too: Desgagné A, Castilloux AM, Angers JF, LeLorier J. The use of the bootstrap statistical method for the pharmacoeconomic cost analysis of skewed data. Pharmacoeconomics. 1998 May;13(5 Pt 1):487-97. doi: 10.2165/00019053-199813050-00002. PMID: 10180748.


.

[Maarten Buis]

Bootstrap p-values are very different animals than bootstrap confidence intervals. I don't think that the paper you linked to is appropriate for your problem, based on only a quick glance at it. You can see how to compute bootstrap p-values in the Stata manual entry of bootstrap, that is, the pdf not the helpfile. It is in the example under the heading "Achieved significance level".

[Hannes Gerstel]

Carlo Lazzaro
Many thanks for your suggestion. If my understanding isn't completely wrong, I believe that following the steps in section three of your suggested paper yield normal-based p-values instead of bias-corrected p-values.


Maarten Buis
Also thank you very much for your helpful hint towards the stata manual. Unfortunately, this example exceeds my knowledge on how to apply this method in a mediation case and whether this actually can be used to calculate bias-corrected p-values. I have the feeling that this also calculates normal based p-values, similar to just using "bs p=r(p), reps(1000): ttest mpg, by(foreign) unequal".

The linked paper from Asparouhov/Muthen might actually describe the exact case I'm interested in (just for another software) "If in addition, a P-value is needed to support the result of the bias-corrected confidence interval, the manual method described below for computing such a P-value can be utilized."

[Maarten Buis]

The example on achieved significance levels in the manual does not use normal based p-values, that is the whole point of an achieved significance level. The quick scan I made of the paper you linked suggest to me that they are using normal based p-values (with some bells and whistles to account for the estimate of the bias).

Anyhow if the manual is beyond your capabilities, then the paper you linked to is way beyond your capabilities. So I guess the only alternative open to you is be pragmatic and look for alternatives. The confidence interval would be the obvious choice, as you already have those. In principle, you could also solve this problem by improving your capabilities, but that is a long term strategy.

[Carlo Lazzaro]

Hannes:
you're correct. The quoted article was meant to intoduce non-parametric bootstrap -ttest- in health economics and was well received from its publication up to the end of the first decade of the new millennium. In more recent years, decision models and, in part, health econometrics are increasing their relevance in this research field.
That said, another article that might be interesting to read (with a focus on 95% CIs, though) is: https://pubmed.ncbi.nlm.nih.gov/11113956/.
I do share Maarten's point about sticking with (bootstrap) 95% CIs that in my experience are more informative than (boostrap) p-values.

[Hannes Gerstel]

@Carlo
Thanks for providing another helpful paper.

@Maarten
Thanks for taking the time to provide another helpful answer. If I can find the time, I will try to apply your stated t-test example to my mediation analysis.
For now I will go with your pragmatic solution of just using the given bias-corrected CIs (even though my field (unfortunately) relies on p-values).