Calculating predicted and expected outcomes after risk adjustment using hierarchical logistic regression models

[Tim Anderson]

Dear Statalist,

I am interested in using Stata to examine hospital variation in risk-adjusted readmission rates, accounting for difference in the patients seen by each hospital, as is done by Medicare.

This approach, as described by Medicare uses hierarchical logistic regression model to simultaneously models data at the patient and hospital levels to account for the variance in patient outcomes within and between hospitals...link and longer description below

https://www.cms.gov/Medicare/Medicar...n-Program.html
https://www.qualitynet.org/dcs/Conte...etTier3&c=Page

Paraphrased

The hospital-level 30-day all-cause risk-standardized readmission rate (RSRR) is estimated using a hierarchical logistic regression model. In brief, the approach simultaneously models data at the patient and hospital levels to account for the variance in patient outcomes within and between hospitals. At the patient level, it models the log-odds of hospital readmission within 30 days of discharge using age, selected clinical covariates, and a hospital-specific effect. At the hospital level, the approach models the hospital-specific effects as arising from a normal distribution. The hospital effect represents the underlying risk of a readmission at the hospital, after accounting for patient risk. The hospital-specific effects are given a distribution to account for the clustering (non-independence) of patients within the same hospital. If there were no differences among hospitals, then after adjusting for patient risk, the hospital effects should be identical across all hospitals.

The RSRR is calculated as the ratio of the number of “predicted” readmissions to the number of “expected” readmissions at a given hospital, multiplied by the national observed readmission rate. For each hospital, the numerator of the ratio is the number of readmissions within 30 days predicted based on the hospital’s performance with its observed case mix, and the denominator is the number of readmissions expected based on the nation’s performance with that hospital’s case mix. This approach is analogous to a ratio of “observed” to “expected” used in other types of statistical analyses.

The “predicted” number of readmissions (the numerator) is calculated by using the coefficients estimated by regressing the clinical risk factors (covariates) and the hospital-specific effect on the risk of readmission. The estimated hospital-specific effect is added to the sum of the estimated regression coefficients multiplied by the patient characteristics. The results are log transformed and summed over all patients attributed to a hospital to calculate a predicted value. The “expected” number of readmissions (the denominator) is obtained in the same manner, except that a common effect using all hospitals in our sample is added in place of the hospital-specific effect. The results are log transformed and summed over all patients attributed to a hospital to calculate an expected value.

Medicare has published methods using SAS and the PROC GLIMMIX command, for which I believe the closest Stata alternative would be melogit. However, I am not sure how to structure this, as if I include the hospital variable as a clustering term, I do not know how to generate predicted probabilities for that hospital term using post-estimation command such as predict and margins.

As a simplified example - I hope to run the following model on a sample of 500,000 patients from 150 hospitals including the following variables
readmitted = the binary outcome variable
age = continuous covariate
gender = binary covariate
hospital_id = hospital identifier

I believe to obtain the "predicted number of readmissions" for each hospital I would run the following model:

melogit readmitted age i.gender i.hospital_id  || hospital_id: 
margins hospital_id

Though I'm unsure if this is correct to include the hospital term as both a fixed and random effect. If I include only the random effect, I cannot generate predicted probabilities

Secondly, to obtain the "expected number of readmissions" as described above, I am not sure how to include "a common effect using all hospitals in our sample, in place of the hospital-specific effect"

Appreciate your thoughts.

Sincerely,
Tim Anderson

[Erik Ruzek]

Tim,

You can include the hospital dummy variables in your model (I typically wouldn't do that but it is permitted), but to get similar estimates as the Medicare study, I think you want the empirical Bayes predictions of the random hospital intercepts. See

help melogit postestimation

and look under predict. You want the

reffects

for the empirical Bayes predictions and

reses

for the standard errors of the predictions.

With regards to the last point, about a common hospital effect, I believe they are referring to the random intercept, and not to anything else.

[Tim Anderson]

Hi Erik,

Thanks for this suggestion. I suppose I'm still unsure of the steps to achieve predicted vs expected readmissions once I run the reffects post-estimation...

Once I have the reffects term, how to add it "to the sum of the estimated regression coefficients multiplied by the patient characteristics"?

The “predicted” number of readmissions (the numerator) is calculated by using the coefficients estimated by regressing the clinical risk factors (covariates) and the hospital-specific effect on the risk of readmission. The estimated hospital-specific effect is added to the sum of the estimated regression coefficients multiplied by the patient characteristics. The results are log transformed and summed over all patients attributed to a hospital to calculate a predicted value.

Do I need to develop a loop adding the _cons +reffect+fixed effects for all patients in each hospital, seperately?

For expected readmissions, I think this is more straightforward

The “expected” number of readmissions (the denominator) is obtained in the same manner, except that a common effect using all hospitals in our sample is added in place of the hospital-specific effect. The results are log transformed and summed over all patients attributed to a hospital to calculate an expected value.

Could I just run the post estimation predict, mu command which should incorporate both fixed and random effects and then summing the predicted probability for each patient at each hospital....for example

melogit readmitted age i.gender || hospital_id:  
predict adjusted_readmission_risk, mu
bysort hospital_id: egen expected_readmissions = mean(adjusted_readmission_risk)

Tim

[Erik Ruzek]

Hi Tim,

I will admit that the language they use is quite confusing. It is hard to tell exactly, but I would think that what you want for both of these is the following:

 melogit readmitted age i.gender || hospital_id:    

**Expected

predict adjusted_readmission_risk, xb // note the change from mu to xb, which is the linear prediction w/ the random effect set to 0  

bysort hospital_id: egen expected_readmissions = total(adjusted_readmission_risk) // note change to total vs. mean - they say "summed"  

**Predicted

predict BLUP_readmission_hosp, reffect

gen hosp_specfic_readmission = adjusted_readmission_risk + BLUP_readmission_hosp

bysort hospital_id: egen predicted_readmissions = total(hosp_specific_readmission)

I'm not totally sure about this, though. Again, their wording is confusing. Others might have different thoughts.

[Erik Ruzek]

OK, I looked at the documentation a bit more and I think you were on to the right track but also a little off. See below:

**Expected 

predict adjusted_readmission_risk, mu conditional(fixedonly) // just having mu on it's own will include the random effects whereas this sets them to 0  

bysort hospital_id: egen expected_readmissions = total(adjusted_readmission_risk) // note change to total vs. mean - they say "summed"  

**Predicted  

predict murand_readmission_risk, mu  

bysort hospital_id: egen predicted_readmissions = total(eta_readmission_risk) 

[Giovanni Ferreira]

Hi Tim, I wonder if this could help you https://journals.sagepub.com/doi/abs...urnalCode=stja, but hopefully you've already found out what the answer to your question was!

[Christopher Dall]

Hi,

Wanted to revive this thread a little bit, if possible. Tim, were you ever able to figure this out? I have tried using the above code from Eric, and it doesn't quite give me what is expected. Wondering if anyone has found a definitive resource on it.