95%CI after roctab

[thiago ardenghi]

Dear all
Is it possible to compute the confidence interval (CI) of the sensitivity and specificity of each Cutpoint after running the roctab command?
I am using the following command:
roctab disease rating, detail graph summary
But ir only give-me the 95%CI for the AUC.
Thank you

[Joseph Coveney]

Maybe you could bootstrap it.

version 15.0

clear *
set seed `=strreverse("1415835")'

program define cutci
    version 15.0
    syntax [if], [Cutoff(real 0.5)]

    logit foreign gear `if'
    estat classification, cutoff(`cutoff')
end

sysuse auto
logit foreign gear, nolog
predict double pr, pr
quietly levelsof pr, local(prs)
foreach pr of local prs {
    bootstrap sen = r(P_p1) spe = r(P_n0), reps(40): cutci , c(`pr')
}

exit

Caveat emptor.

What are you going to do with these?

[Bruce Weaver]

Hello Thiago. This is not completely automated, but depending on exactly what you want, it might serve your purpose.

clear
webuse hanley
roctab disease rating, detail graph summary

generate byte nodisease = !disease
generate byte tpos = .
generate byte tneg = .

forvalues i = 1(1)6 {
quietly replace tpos = rating >= `i'
quietly replace tneg = !tpos
quietly ci proportion tpos if disease, wilson   // Wilson-score CI for sensitivity
display "Cut-point:  " `i'
display "  Sens, 95% CI:  " r(proportion) ", " r(lb) " to " r(ub)
quietly ci proportion tneg if nodisease, wilson // Wilson-score CI for specificity
display "  Spec, 95% CI:  " r(proportion) ", " r(lb) " to " r(ub)
}

Selected bits of the output:

. roctab disease rating, detail graph summary

Detailed report of sensitivity and specificity
------------------------------------------------------------------------------
                                           Correctly
Cutpoint      Sensitivity   Specificity   Classified          LR+          LR-
------------------------------------------------------------------------------
( >= 1 )          100.00%         0.00%       46.79%       1.0000     
( >= 2 )           94.12%        56.90%       74.31%       2.1835       0.1034
( >= 3 )           90.20%        67.24%       77.98%       2.7534       0.1458
( >= 4 )           86.27%        77.59%       81.65%       3.8492       0.1769
( >= 5 )           64.71%        96.55%       81.65%      18.7647       0.3655
( >  5 )            0.00%       100.00%       53.21%                    1.0000
------------------------------------------------------------------------------


                      ROC                    -Asymptotic Normal--
           Obs       Area     Std. Err.      [95% Conf. Interval]
     ------------------------------------------------------------
           109     0.8932       0.0307        0.83295     0.95339


Cut-point:  1
  Sens, 95% CI:  1, .92995338 to 1
  Spec, 95% CI:  0, 0 to .06211786
Cut-point:  2
  Sens, 95% CI:  .94117647, .84075374 to .97979336
  Spec, 95% CI:  .56896552, .44118105 to .68818201
Cut-point:  3
  Sens, 95% CI:  .90196078, .79021765 to .95739193
  Spec, 95% CI:  .67241379, .54424055 to .77916709
Cut-point:  4
  Sens, 95% CI:  .8627451, .74278308 to .93188898
  Spec, 95% CI:  .77586207, .65338882 to .86406339
Cut-point:  5
  Sens, 95% CI:  .64705882, .50986058 to .76365512
  Spec, 95% CI:  .96551724, .88270855 to .99049207
Cut-point:  6
  Sens, 95% CI:  0, 0 to .07004662
  Spec, 95% CI:  1, .93788214 to 1

HTH.

[Madu Abuchi]

Maybe you could bootstrap it.

version 15.0

clear *
set seed `=strreverse("1415835")'

program define cutci
version 15.0
syntax [if], [Cutoff(real 0.5)]

logit foreign gear `if'
estat classification, cutoff(`cutoff')
end

sysuse auto
logit foreign gear, nolog
predict double pr, pr
quietly levelsof pr, local(prs)
foreach pr of local prs {
bootstrap sen = r(P_p1) spe = r(P_n0), reps(40): cutci , c(`pr')
}

exit

Caveat emptor.

What are you going to do with these?

Joseph Coveney Do you have any suggestions on how to bootstrap cutoff points? Assuming the model adjusts for another factor and you want to bootstrap the cutpoint of gear.

I found that Phil Claytons -cutpt- ado does bootstrap but doesn't allow for adjusting another variable

[Joseph Coveney]

I'm not sure what you mean. Do you mean bootstrapping what are called optimum cutoffs?

[Madu Abuchi]

Yes bootstrapping the optimum cut-off point i.e the cut-off point that maximizes sensitivity and specificity (Youden's index)

[Joseph Coveney]

There have been numerous threads on the list over the years about so-called optimum cutoff points along the receiver operating characteristic curve—for example, this one—that you might want to seek out and peruse before you get too deeply involved in this exercise.

You can use cutpt on predictions from a logistic regression model that has made adjustments for a covariate.

[Leonardo Guizzetti]

To add my opinion, you may want to rethink Youden's J as an index of "optimal". What you are doing will maximize the sum of sensitivity and specificity, which means, you may end up with one of them being very high and the other very low, which may be suboptimal for your purposes. This is often used when the costs of false negatives and false positives are the same, but this assumption is hardly ever justifiable in medical research, if it is ever examined at all. An alternative is to use Liu's cutpoint (also estimated by -cutpt-), which maximizes over the product of the sensitivity and specificity, ensuring that both parameters are at least not too small.

[Madu Abuchi]

Thanks, Joseph and Leonard for your inputs

Joseph using the cutpt on predictions from a logistic regression model gives the prediction cut-off point and not the cut-off point of the actual variable of interest. That has been my concern

[Joseph Coveney]

cutpt on predictions from a logistic regression model gives the prediction cut-off point and not the cut-off point of the actual variable of interest.

Once you adjust for another variable, you no longer have an ROC curve of the actual variable of interest.