Calibration plot

[Ankit Bhardwaj]

I need to create a calibration chart for my risk model and generate a predicted probabilities column based on my new risk score. Can you please guide me through the process?

[Rich Goldstein]

there are a number of ways to do this, but it sounds like a ready-made solution might be best for you; use -search- to find, download and install -pmcalplot- which is a user-written package; then read the help file

help pmcalplot

then, if you have specific questions, try again

[Leonardo Guizzetti]

There is also the calibration belt (-search calibrationbelt- on SSC and also StataJournal, if I recall correctly).

[Ankit Bhardwaj]

Could you please provide the video link? I couldn't find it in the video tutorial section. Thank you!

[Leonardo Guizzetti]

Sorry I’m not familiar with any video tutorials for these commands.

[Rich Goldstein]

and neither am I

[Ankit Bhardwaj]

Ok, no problem. I need to create predictive probabilities based on my new risk score. Any suggestions please.

[Leonardo Guizzetti]

Those commands should provide examples of syntax in their respective help files. You don't give any details of your risk score or how its modelled, though I'd wager it's a logistic regression model. In any case, you need a way to get estimates of the predicted probability of the outcome and pair those with the actual outcome to do any kind of calibration in this area.

I'm not going to suggest any specific code because it's not always a straightforward task. It's my belief that if you are seriously developing or validating a clinical prediction model, then you should have understanding and awareness of how to do it rigorously, or seek out professional expertise.

I would start with reading the papers referenced in those user-contributed packages. You may also like this open-access article (Steyerberg & Vergouwe 2014) to give you an overview of the broad steps needed. Steyerberg also has a nice textbook-level treatment on the development of clinical prediction models now in it's second edition.

Ewout W. Steyerberg, Yvonne Vergouwe, Towards better clinical prediction models: seven steps for development and an ABCD for validation, European Heart Journal, Volume 35, Issue 29, 1 August 2014, Pages 1925–1931, https://doi.org/10.1093/eurheartj/ehu207

[Ankit Bhardwaj]

Thank you, Prof. Leonardo Guizzetti, for the help. I'm sorry for not explaining this model in detail. I derived a risk model based on logistic regression analysis with the help of a statistician. I am a clinical epidemiologist. Then, I validated this model very well in the validation cohort with good AUROC. Now, I want to prepare a calibration plot as asked by my clinical physician. I typed the help pmcalplot command in the STA TA. it asked me to install the package first but I failed to do so. It is giving me the below message again and again:

. pmcalplot RiskSCOREVALIDATION
Package 'running' is required for this version on pmcalplot
Installation will begin now ...
checking running consistency and verifying not already installed...
could not copy http://fmwww.bc.edu/repec/bocode/r/running.hlp
(no action taken)
--Break--
r(1);

[Leonardo Guizzetti]

That's all very well and good. Sometimes these issues happen where the program has dependencies that need to be installed. They're usually documented in the help file (often near the top or bottom). In this case, there's a link at the very bottom of -help pmcalplot- that says to see -running-, which itself is not a great name to use to search.

Use -ssc describe running- to find the package for -running-, and then install it from the user interface that pops up or else -ssc install running- to do that directly.

[Ankit Bhardwaj]

Thank You

[Ankit Bhardwaj]

Dear Statisticians,
Kindly help me with the calibration plot. I am not able to get it after running the pmcalplot command; instead, I am getting the below error message.

pmcalplot p_dev VNTSept if VNTSept ==1, ci
Binary option selected: Calibration plot for logistic prediction model displaying...
r(2000);

[Nick Cox]

r(2000) as return code means no observations [to use to do what you ask].

It is hard to say more without knowing about your data (no data example here) or knowing where the command fails. On the latter, you could

set trace on

before running the command and show the trace before it fails.

You may need to

set tracedepth

with a particular depth, either bumping it up if the trace reveals too little or bumping it down if the trace reveals too much.

r(2000) can be triggered by various problems, such as

* missing values

* working with a subset that is empty in your data

* supplying a string variable where a numeric variable is needed

[Ankit Bhardwaj]

Thank You. I have run the mdesc command, which shows there are no missing values in the dataset.

[Nick Cox]

That may not be the issue. Commands often create temporary variables and the issue may be that there are missing values in those.

I can't say more but that is the problem. Without a way to reproduce your problem or precise information about where the program fails even the program developers may be hard put to say more.

If you're unable or unwilling to add more information, your best bet may be to contact the developers directly. They haven't responded here and may not be active on Statalist.