Calculating 95% CI of difference in AUC

[Natalie Malek]

After running 2 different logistic regressions with the predict command and then running roccomp- how do we calculate the 95% confidence interval for the difference in AUC? STATA only spits out the p-value and X2.

[Felix Bittmann]

Bootstrapping is an option here. If you want to do so, make sure you use roctab as only this command provides the area as a return value.

[Leonardo Guizzetti]

Can you say more about your models? Do they have the same outcome variable? Are the predictor variables of one model a subset of the other?

[Natalie Malek]

They are two logistic models- one nested within the other.

[Leonardo Guizzetti]

Thank you, Natalie. There are two ways yon can go about this:

1) if you only care whether the AUC values are different for each model, then it is sufficient to use a likelihood ratio test to compare the nested model, and the point estimate of this difference is the difference in AUC estimates. The justiifcation is provided in Pepe et al (2013).

2) if a confidence interval is in addition is desired (as your post suggests) then Zou and Yue's methodology is appropriate for correlated AUC statistics.

Pepe, M. S., Kerr, K. F., Longton, G., & Wang, Z. (2013). Testing for improvement in prediction model performance. Statistics in Medicine, 32(9), 1467–1482. https://doi.org/10.1002/sim.5727

Zou, G. Y., & Yue, L. (2013). Using confidence intervals to compare several correlated areas under the receiver operating characteristic curves. Statistics in Medicine, 32(29), 5077–5090. https://doi.org/10.1002/sim.5889

[Joseph Coveney]

. . . running roccomp- how do we calculate the 95% confidence interval for the difference in AUC? STATA only spits out the p-value and X2.

You can use that chi-square test statistic to compute the confidence interval. Unfortunately, you also need the individual ROC AUCs, too, and although -roccomp- displays them both, it doesn't return them. (Note to StataCorp: this seems like an oversight.) So, you'll have to get them individually from -roctab-. Once you have them, you can compute the confidence bounds. The individual steps are illustrated below using the autos dataset, and the result is cross-checked against what bootstrapping the difference gives.

.ÿ
.ÿversionÿ17.0

.ÿ
.ÿclearÿ*

.ÿ
.ÿ//ÿseedem
.ÿsetÿseedÿ307720499

.ÿ
.ÿquietlyÿsysuseÿauto

.ÿ
.ÿ*
.ÿ*ÿBeginÿhere
.ÿ*
.ÿ//ÿGettingÿindividualÿROCÿAUCsÿ&ÿtheirÿdifference
.ÿtempnameÿmpgÿpriceÿdifference

.ÿ
.ÿquietlyÿroctabÿforeignÿmpg

.ÿscalarÿdefineÿ`mpg'ÿ=ÿr(area)

.ÿquietlyÿroctabÿforeignÿprice

.ÿscalarÿdefineÿ`price'ÿ=ÿr(area)

.ÿ
.ÿscalarÿdefineÿ`difference'ÿ=ÿ`mpg'ÿ-ÿ`price'

.ÿ
.ÿ//ÿGettingÿtestÿstatisticsÿofÿdifference
.ÿtempnameÿchi2ÿdfÿv

.ÿ
.ÿroccompÿforeignÿmpgÿprice

ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿROCÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿAsymptoticÿnormal
ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿObsÿÿÿÿÿÿÿareaÿÿÿÿÿStd.ÿerr.ÿÿÿÿÿÿ[95%ÿconf.ÿinterval]
-------------------------------------------------------------------------
mpgÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ74ÿÿÿÿÿ0.7286ÿÿÿÿÿÿÿ0.0675ÿÿÿÿÿÿÿÿ0.59622ÿÿÿÿÿ0.86095
priceÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ74ÿÿÿÿÿ0.5769ÿÿÿÿÿÿÿ0.0747ÿÿÿÿÿÿÿÿ0.43053ÿÿÿÿÿ0.72331
-------------------------------------------------------------------------
H0:ÿarea(mpg)ÿ=ÿarea(price)
ÿÿÿÿchi2(1)ÿ=ÿÿÿÿÿ1.44ÿÿÿÿÿÿÿProb>chi2ÿ=ÿÿÿ0.2302

.ÿ
.ÿscalarÿdefineÿ`chi2'ÿ=ÿr(chi2)

.ÿscalarÿdefineÿ`df'ÿ=ÿr(df)

.ÿscalarÿdefineÿ`v'ÿ=ÿ(`mpg'ÿ-ÿ`price')^2ÿ/ÿ`chi2'

.ÿ
.ÿ//ÿassertÿ(`difference')^2ÿ/ÿ`v'ÿ==ÿ`chi2'
.ÿdisplayÿinÿsmclÿasÿresultÿ"SEÿ=ÿ"ÿsqrt(`v')
SEÿ=ÿ.12639943

.ÿ
.ÿ//ÿComputingÿ95%ÿCIÿupperÿandÿlowerÿbounds
.ÿdisplayÿinÿsmclÿasÿresultÿ"95%ÿCIÿLBÿ=ÿ"ÿabs(`difference')ÿ-ÿsqrt(invchi2(`df',ÿ0.95)ÿ*ÿ`v')
95%ÿCIÿLBÿ=ÿ-.09607748

.ÿdisplayÿinÿsmclÿasÿresultÿ"95%ÿCIÿUBÿ=ÿ"ÿabs(`difference')ÿ+ÿsqrt(invchi2(`df',ÿ0.95)ÿ*ÿ`v')
95%ÿCIÿUBÿ=ÿ.39939916

.ÿ
.ÿ//ÿBootstrappingÿ(conventional,ÿnotÿwild)
.ÿprogramÿdefineÿbootem,ÿrclass
ÿÿ1.ÿÿÿÿÿÿÿÿÿversionÿ17.0
ÿÿ2.ÿÿÿÿÿÿÿÿÿsyntaxÿ[if]
ÿÿ3.ÿ
.ÿÿÿÿÿÿÿÿÿroctabÿforeignÿmpgÿ`if'
ÿÿ4.ÿÿÿÿÿÿÿÿÿtempnameÿmpgÿn
ÿÿ5.ÿÿÿÿÿÿÿÿÿscalarÿdefineÿ`mpg'ÿ=ÿr(area)
ÿÿ6.ÿÿÿÿÿÿÿÿÿscalarÿdefineÿ`n'ÿ=ÿr(N)
ÿÿ7.ÿ
.ÿÿÿÿÿÿÿÿÿroctabÿforeignÿpriceÿ`if'
ÿÿ8.ÿÿÿÿÿÿÿÿÿreturnÿscalarÿdeltaÿ=ÿ`mpg'ÿ-ÿr(area)
ÿÿ9.ÿÿÿÿÿÿÿÿÿreturnÿscalarÿNÿ=ÿ`n'
ÿ10.ÿend

.ÿ
.ÿquietlyÿbootstrapÿdeltaÿ=ÿr(delta),ÿreps(1000)ÿbcaÿnowarnÿnodots:ÿbootem

.ÿestatÿbootstrap,ÿall

BootstrapÿresultsÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿNumberÿofÿobsÿÿÿÿÿ=ÿÿÿÿÿÿÿÿÿ74
ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿReplicationsÿÿÿÿÿÿ=ÿÿÿÿÿÿÿ1000

ÿÿÿÿÿÿCommand:ÿbootem
ÿÿÿÿÿÿÿÿdelta:ÿr(delta)

------------------------------------------------------------------------------
ÿÿÿÿÿÿÿÿÿÿÿÿÿ|ÿÿÿÿObservedÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿBootstrap
ÿÿÿÿÿÿÿÿÿÿÿÿÿ|ÿcoefficientÿÿÿÿÿÿÿBiasÿÿÿÿstd.ÿerr.ÿÿ[95%ÿconf.ÿinterval]
-------------+----------------------------------------------------------------
ÿÿÿÿÿÿÿdeltaÿ|ÿÿÿ.15166084ÿÿÿÿ.003649ÿÿÿ.12594597ÿÿÿ-.0951887ÿÿÿ.3985104ÿÿÿ(N)
ÿÿÿÿÿÿÿÿÿÿÿÿÿ|ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ-.1090707ÿÿÿ.4010282ÿÿÿ(P)
ÿÿÿÿÿÿÿÿÿÿÿÿÿ|ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ-.1124402ÿÿÿ.3921568ÿÿ(BC)
ÿÿÿÿÿÿÿÿÿÿÿÿÿ|ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ-.1136853ÿÿÿ.3921568ÿ(BCa)
------------------------------------------------------------------------------
Key:ÿÿÿN:ÿNormal
ÿÿÿÿÿÿÿP:ÿPercentile
ÿÿÿÿÿÿBC:ÿBias-corrected
ÿÿÿÿÿBCa:ÿBias-correctedÿandÿaccelerated

.ÿ
.ÿexit

endÿofÿdo-file


.

The simple computation from the chi-square test statistic returned by -roccomp- (SE = 0.126; 95% CI = [-0.096, 0.399]) is confirmed in the Normal-based interval from bootstrapping (SE = 0.126, 95% CI = [-0.095, 0.399]).

In your case, both ROC AUCs are pretty poor (< 0.7) and so their difference might not be worth worrying too much about; nevertheless, you might want to check on the coverage of the CI computed in this way by using simulation under conditions (sample size, proportion of positive cases, difference in predicted cases) similar to those obtained with your particular dataset.