Predicted Risk of Mortality Models: Surgeons Need to Understand Limitations of the University HealthSystem Consortium Models

Benjamin D. Kozower, Gorav Ailawadi, David R. Jones, Robert D. Pates, Christine L. Lau, Irving L. Kron, George J. Stukenborg

Research output: Contribution to journalArticle

30 Citations (Scopus)

Abstract

Background: The University HealthSystem Consortium (UHC) mortality risk adjustment models are increasingly being used as benchmarks for quality assessment. But these administrative database models may include postoperative complications in their adjustments for preoperative risk. The purpose of this study was to compare the performance of the UHC with the Society of Thoracic Surgeons (STS) risk-adjusted mortality models for adult cardiac surgery and evaluate the contribution of postoperative complications on model performance. Study Design: We identified adult cardiac surgery patients with mortality risk estimates in both the UHC and Society of Thoracic Surgeons databases. We compared the predictive performance and calibration of estimates from both models. We then reestimated both models using only patients without any postoperative complications to determine the relative contribution of adjustments for postoperative events on model performance. Results: In the study population of 2,171 patients, the UHC model explained more variability (27% versus 13%, p < 0.001) and achieved better discrimination (C statistic = 0.88 versus 0.81, p < 0.001). But when applied in the population of patients without complications, the UHC model performance declined severely. The C statistic decreased from 0.88 to 0.49, a level of discrimination equivalent to random chance. The discrimination of the Society of Thoracic Surgeons model was unchanged (C statistic of 0.79 versus 0.81). Conclusions: Although the UHC model demonstrated better performance in the total study population, this difference in performance reflects adjustments for conditions that are postoperative complications. The current UHC models should not be used for quality benchmarks.

Original languageEnglish (US)
Pages (from-to)551-556
Number of pages6
JournalJournal of the American College of Surgeons
Volume209
Issue number5
DOIs
StatePublished - Nov 1 2009
Externally publishedYes

Fingerprint

Risk Adjustment
Benchmarking
Mortality
Thoracic Surgery
Databases
Population
Calibration
Surgeons

ASJC Scopus subject areas

  • Surgery

Cite this

Predicted Risk of Mortality Models : Surgeons Need to Understand Limitations of the University HealthSystem Consortium Models. / Kozower, Benjamin D.; Ailawadi, Gorav; Jones, David R.; Pates, Robert D.; Lau, Christine L.; Kron, Irving L.; Stukenborg, George J.

In: Journal of the American College of Surgeons, Vol. 209, No. 5, 01.11.2009, p. 551-556.

Research output: Contribution to journalArticle

Kozower, Benjamin D. ; Ailawadi, Gorav ; Jones, David R. ; Pates, Robert D. ; Lau, Christine L. ; Kron, Irving L. ; Stukenborg, George J. / Predicted Risk of Mortality Models : Surgeons Need to Understand Limitations of the University HealthSystem Consortium Models. In: Journal of the American College of Surgeons. 2009 ; Vol. 209, No. 5. pp. 551-556.
@article{049cb5098f8c4ef98a13dd963aed8880,
title = "Predicted Risk of Mortality Models: Surgeons Need to Understand Limitations of the University HealthSystem Consortium Models",
abstract = "Background: The University HealthSystem Consortium (UHC) mortality risk adjustment models are increasingly being used as benchmarks for quality assessment. But these administrative database models may include postoperative complications in their adjustments for preoperative risk. The purpose of this study was to compare the performance of the UHC with the Society of Thoracic Surgeons (STS) risk-adjusted mortality models for adult cardiac surgery and evaluate the contribution of postoperative complications on model performance. Study Design: We identified adult cardiac surgery patients with mortality risk estimates in both the UHC and Society of Thoracic Surgeons databases. We compared the predictive performance and calibration of estimates from both models. We then reestimated both models using only patients without any postoperative complications to determine the relative contribution of adjustments for postoperative events on model performance. Results: In the study population of 2,171 patients, the UHC model explained more variability (27{\%} versus 13{\%}, p < 0.001) and achieved better discrimination (C statistic = 0.88 versus 0.81, p < 0.001). But when applied in the population of patients without complications, the UHC model performance declined severely. The C statistic decreased from 0.88 to 0.49, a level of discrimination equivalent to random chance. The discrimination of the Society of Thoracic Surgeons model was unchanged (C statistic of 0.79 versus 0.81). Conclusions: Although the UHC model demonstrated better performance in the total study population, this difference in performance reflects adjustments for conditions that are postoperative complications. The current UHC models should not be used for quality benchmarks.",
author = "Kozower, {Benjamin D.} and Gorav Ailawadi and Jones, {David R.} and Pates, {Robert D.} and Lau, {Christine L.} and Kron, {Irving L.} and Stukenborg, {George J.}",
year = "2009",
month = "11",
day = "1",
doi = "10.1016/j.jamcollsurg.2009.08.008",
language = "English (US)",
volume = "209",
pages = "551--556",
journal = "Journal of the American College of Surgeons",
issn = "1072-7515",
publisher = "Elsevier Inc.",
number = "5",

}

TY - JOUR

T1 - Predicted Risk of Mortality Models

T2 - Surgeons Need to Understand Limitations of the University HealthSystem Consortium Models

AU - Kozower, Benjamin D.

AU - Ailawadi, Gorav

AU - Jones, David R.

AU - Pates, Robert D.

AU - Lau, Christine L.

AU - Kron, Irving L.

AU - Stukenborg, George J.

PY - 2009/11/1

Y1 - 2009/11/1

N2 - Background: The University HealthSystem Consortium (UHC) mortality risk adjustment models are increasingly being used as benchmarks for quality assessment. But these administrative database models may include postoperative complications in their adjustments for preoperative risk. The purpose of this study was to compare the performance of the UHC with the Society of Thoracic Surgeons (STS) risk-adjusted mortality models for adult cardiac surgery and evaluate the contribution of postoperative complications on model performance. Study Design: We identified adult cardiac surgery patients with mortality risk estimates in both the UHC and Society of Thoracic Surgeons databases. We compared the predictive performance and calibration of estimates from both models. We then reestimated both models using only patients without any postoperative complications to determine the relative contribution of adjustments for postoperative events on model performance. Results: In the study population of 2,171 patients, the UHC model explained more variability (27% versus 13%, p < 0.001) and achieved better discrimination (C statistic = 0.88 versus 0.81, p < 0.001). But when applied in the population of patients without complications, the UHC model performance declined severely. The C statistic decreased from 0.88 to 0.49, a level of discrimination equivalent to random chance. The discrimination of the Society of Thoracic Surgeons model was unchanged (C statistic of 0.79 versus 0.81). Conclusions: Although the UHC model demonstrated better performance in the total study population, this difference in performance reflects adjustments for conditions that are postoperative complications. The current UHC models should not be used for quality benchmarks.

AB - Background: The University HealthSystem Consortium (UHC) mortality risk adjustment models are increasingly being used as benchmarks for quality assessment. But these administrative database models may include postoperative complications in their adjustments for preoperative risk. The purpose of this study was to compare the performance of the UHC with the Society of Thoracic Surgeons (STS) risk-adjusted mortality models for adult cardiac surgery and evaluate the contribution of postoperative complications on model performance. Study Design: We identified adult cardiac surgery patients with mortality risk estimates in both the UHC and Society of Thoracic Surgeons databases. We compared the predictive performance and calibration of estimates from both models. We then reestimated both models using only patients without any postoperative complications to determine the relative contribution of adjustments for postoperative events on model performance. Results: In the study population of 2,171 patients, the UHC model explained more variability (27% versus 13%, p < 0.001) and achieved better discrimination (C statistic = 0.88 versus 0.81, p < 0.001). But when applied in the population of patients without complications, the UHC model performance declined severely. The C statistic decreased from 0.88 to 0.49, a level of discrimination equivalent to random chance. The discrimination of the Society of Thoracic Surgeons model was unchanged (C statistic of 0.79 versus 0.81). Conclusions: Although the UHC model demonstrated better performance in the total study population, this difference in performance reflects adjustments for conditions that are postoperative complications. The current UHC models should not be used for quality benchmarks.

UR - http://www.scopus.com/inward/record.url?scp=70350130994&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=70350130994&partnerID=8YFLogxK

U2 - 10.1016/j.jamcollsurg.2009.08.008

DO - 10.1016/j.jamcollsurg.2009.08.008

M3 - Article

C2 - 19854393

AN - SCOPUS:70350130994

VL - 209

SP - 551

EP - 556

JO - Journal of the American College of Surgeons

JF - Journal of the American College of Surgeons

SN - 1072-7515

IS - 5

ER -