Can I use the pmcalplot command after cross-validation (cvauroc) to assess calibration in model prediction?

[Virginia Zarama]

I am conducting internal validation of clinical score in an entire dataset (7000 patients) by k-fold cross-validation using the cvauroc command. Can I use the pmcalplot or calibrationbelt command after the cross-validation to assess model calibration ?

[Rich Goldstein]

I have never used -cvauroc- but it looks like you can use the option "fit" to get the predicted probabilities and then use that variable in your call to -pmcalplot- or -calibrationbelt-

[Virginia Zarama]

Thank you!. I used the _fit to get the predicted values and ran the pmcalplot. It worked fine! I am not sure if I can do the calibration in the same sample that I did k-fold cross-validation. I wouldn't need to split the sample to do cv in one sample and calibration in other sample, right?

[Rich Goldstein]

sorry, but for validation I usually use -bootstrap- not cross-validation; maybe someone else can help you

[Virginia Zarama]

I understand. Thank you very much for your help. Last two questions,
- After you validate with bootstrap, do you use the same complete original sample to test calibration? or you split the sample : one for bootstrap and other to calibrate?
- Have you worked with the bootstrap validation command: bsvalidation?

[Rich Goldstein]

after I validate with bootstrap I use the same original sample for calibration

I have never used -bsvalidation-

[Virginia Zarama]

Thank you!!