ABSTRACT - UCSF’s 30-day unplanned readmission rates of 9-11% over the past several years is above the target of being in the top decile of peer Academic Medical Centers. There are several readmission reduction programs at UCSF, but enrollment is driven by patient insurance or primary care provider rather than risk of readmission. Our proposal is to use a Machine Learning (ML) model to predict risk of unplanned readmission for patients after discharge from UCSF, and to use this model to enroll high-risk patients into targeted discharge support programs. Our goal is to reduce 30-day unplanned readmissions by 15%. Currently, all Accountable Care Organization patients discharged from UCSF are enrolled in a discharge support program through the Office of Population Health (OPH) that provides an average of one phone call per week in the first month after discharge. If this proposal is selected, we would identify a subset of ACO patients at high-risk for readmission and leverage existing OPH infrastructure to design and provide higher-touch post-discharge support to these patients. Adjusting our ML model to match the resource constraints for this high-touch program, we plan to set our model at a sensitivity of 70%, and a positive predictive value of 17%. Over the course of one year, we estimate we can prevent 63 unplanned readmissions in this small pilot ACO population. This equates to savings of 403 bed days, resulting in nearly $1.2 Million/year of additional contribution margin from backfill. If successful, we aim to expand use of this model beyond ACO patients to the 28,000 annual discharges at UCSF, which would multiply the above benefits five-fold.
PROJECT LEAD(S):
Xinran Liu, MD, MS, FAMIA. Assistant Professor, Division of Hospital Medicine. Director of Clinical Informatics at SMMC. Associate Program Director, Clinical Informatics Fellowship. Director, GME Clinical Informatics and Data Science Pathway.
Timothy Judson, MD, MPH. Assistant Professor, Division of Hospital Medicine. Medical Director of Care Delivery Transformation, Office of Population Health. Medical co-Director, Adult Care Transitions, UCSF. Physician Informaticist, Center for Digital Health Innovation.
EXECUTIVE SPONSOR(S):
Gina Intinarelli, RN, MS, PhD. Vice President and Chief Population Health Officer at UCSF Health. Associate Chief Nursing Officer, Institute for Nursing Excellence. Associate Dean of Clinical Affairs, UCSF School of Nursing
Adrienne Green, MD, SFHM, FACP. Chief Medical Officer, Adult Services. VP Regulatory and Medical Affairs. Professor of Clinical Medicine, Division of Hospital Medicine.
TEAM
Anoop Muniyappa, MD, MS. Assistant Professor, Division of Hospital Medicine. Associate Director of Population Health Analytics. Physician Informaticist, DOM Data Core.
Stephanie Rogers, MS, MPH, MD. Associate Professor of Medicine, Division of Geriatrics. Associate Chief of Clinical Geriatrics Programs, UCSF. Co-Medical Director, Adult Transitions, UCSF. Medical Director, UCSF's Age-Friendly Health System.
Michael Helle, NRP/CCP, FP-C, MHA, MBA. Director Clinical Programs, Office of Population Health.
- Ahmed Alaa, PhD, Assistant Professor, UC Berkeley/UCSF Computation Precision Health Program.
PROBLEM
UCSF’s 30-day unplanned readmission rate has been between 9% and 11% for several years, and we have been unable to achieve our readmissions target – to be in the top decile of peer Academic Medical Centers. Readmissions are not only harmful to patients and their families, but are also financially detrimental to the health system, resulting in millions of dollars of penalties, including through the federal Hospital Readmission Reduction Program (HRRP). Approximately 8 of 100 beds at UCSF are occupied by readmitted patients, worsening capacity and ED overcrowding issues, and preventing UCSF from accepting transfers of tertiary/quaternary patients. The readmissions rate is also an important national quality metric that impacts the rankings and reputation of UCSF Health.
There are currently several programs in place to support patients after discharge to prevent readmissions. However, enrollment into these programs is determined by factors like the patient’s health insurance and primary care provider, rather than by their risk of readmission.
TARGET
We propose to apply a Machine Learning (ML) model to risk stratify individual patients after discharge based on their unplanned readmission risk, and then use that risk stratification to drive enrollment in targeted post-discharge support interventions. By doing so, our goal is to decrease the 30-day unplanned readmission rate in these patients by 15%. Similar interventions in the literature have resulted in 10-25% reductions [1,2].
GAPS
UCSF has not previously used readmission prediction algorithms or scores to drive enrollment in discharge support programs for three primary reasons. First, published readmission prediction scores (such as the LACE score and HOSPITAL score) have not performed well in the highly complex population of patients served by UCSF. Second, until recently, UCSF did not have HIPAC, which is necessary infrastructure to operationalize modern ML models. Third, many discharge support programs are open to only subsets of patients, such as those in Accountable Care Organizations (ACOs), without regard to readmission risk.
INTERVENTION
The proposal team has already developed a Machine Learning model that can effectively predict the risk of unplanned readmissions for individual patients after discharge at UCSF. The model is designed to run daily, on all adult patients discharged the previous day. The model will output a “yes” or a “no” for whether or not that specific patient is high risk to have an unplanned 30-day readmission event (based on set sensitivity and positive predictive value [PPV] thresholds), as well as the risk score itself. In addition, the model will display the top reasons why the model made that individual patient prediction (see figure 1, attached).
An initial model was built using deidentified data from the Information Commons [3], after which a prototype model was created using identified data from APeX’s Clarity database specifically tuned to the 30-day unplanned readmission definition used at UCSF (Vizient). The model incorporates data including demographics, social determinants of health, utilization, procedures, labs, vitals, and EHR orders.
The prototype model built with Clarity data currently demonstrates an AUC of 0.749, and AUPRC of 0.250 on the validation set. The model’s performance on the hold out test set (data from 2022, not used for any model training or tuning) is very similar with an AUC of 0.738 and AUPRC of 0.245. The consistency in performance across the validation and test set suggests that the model will be able to generalize well to new data and is not overfit to the validation set. The top performing features in the model can be seen in figure 2 (attached). The precision-recall (PR) curve from the test set can be seen in figure 3 (attached). These results are consistent with the performance of machine-learning models at other US health systems [4-7]. All of the features in the model were designed so that they would be available the day after discharge. We have confirmed that this is working as intended.
Metrics such as AUC and AUPRC should ideally be translated to sensitivity (recall) and positive predictive value (precision) to better evaluate potential clinical utility [8]. To do this, we need to determine what threshold to set the model to. The PR curve in figure 3 demonstrates all possible values for precision and recall based on possible thresholds. One important factor to account for when selecting the threshold to use is resource capacity. The downstream clinical workflow that we propose to support patients at high risk for readmission (detailed later in proposal) is limited in how many patients it can manage per year - estimated under 3000 patients. We have chosen a threshold that results in a recall of 70% and precision of ~17%. This achieves the goal of enrolling less than 3000 patients into our intervention program, and maintains a reasonably high recall and precision.
Specifically, in a population of 100,000 ACO patients, we estimate there would be 6,000 discharges per year, of which there would be ~600 readmissions. With a set recall of 70% and precision of 17% for predicting a patient to be high risk, approximately 2470 of those patient discharges will be classified as high risk for readmissions. We estimate that we can prevent 63 unplanned 30-day readmissions annually within this small ACO population. Please see the below confusion matrix and outcome metrics for more details.
Case positives | Case negatives | |
Test positives | 420 | 2050 |
Test negatives | 180 | 3350 |
Anticipated sensitivity: 70%
Anticipated cases that test positive: 2470 (41% of all patients)
Anticipated PPV: 17%
Anticipated FPR: 38%
Anticipated NPV: 95%
Anticipated specificity: 62%
In testing the model above, we have gained several insights that can help us not only improve the model further, but also refine the downstream clinical workflows that result from the model’s predictions.
In the time interval since Caring Wisely round 1, we have added a large number of additional features into the model. This includes: national and state ADI indexes, outpatient utilization prior to hospitalization, # of days on antibiotics, # days of on pressors, if the patient required ICU level care, Charlson Comorbidity Index (CCI), PMH groupers based on CCI, and whether or not the patient had PCP appointments set up at the time of discharge. Despite the addition of these features, the model's performance only improved slightly. We suspect that adding additional structured features will have similarly limited impact. This is an insight from Dr. Ahmed Alaa, who is a faculty member in the joint UCSF/Berkeley Computational Precision Health Program. Dr. Alaa will be joining our team as a ML subject expert. Based on his experience, he suggests that we add other modalities of data such as clinical notes, as there might be important information captured in case management, occupational and physical therapy, physician, or social work notes that are not available as structured data elements. This will be our main area of focus in terms of next steps to refine the model further.
In reviewing the results of the model, we noticed that there seemed to be multiple discrete cohorts of patients. For example, patients with the highest predicted risk scores tend to have very high prior utilization, be middle-aged, and medically complex. The group of patients with the 2nd highest predicted risk scores tend to be among the oldest, with the highest medically complexity, but have lower prior utilization compared to the group with the highest predicted risk scores. Based on this insight, we believe it makes sense to study these cohorts further to determine whether they may benefit from different types of post-discharge support interventions.
If this program is selected, we will start by using the model on a subset of ACO patients (~100,000) who are already eligible for discharge support programs through the Office of Population Health (OPH). In the current state, all of these ACO patients are enrolled in a program that offers an average of one call per week for the first month after discharge. In the future state, patients who are high risk for readmission would be enrolled in more high-touch programs, involving intensive care support from a multidisciplinary team. These protocols would be developed by OPH leadership in collaboration with the Caring Wisely team, and the Medical Directors for Adult Care Transitions. We are excited by the opportunity to partner with the Caring Wisely team and leverage their expertise in program design, lean-thinking, process improvement, and implementation science to develop and implement these novel workflows. Some specific areas where Caring Wisely support and funding will help accelerate our initiative forward include:
Explore additional features to add to the model that have not been tried yet
Review cohorts of patients identified by the model that have different levels of risk, and design possible interventions and workflows that are best suited to address the needs of the different cohorts.
Collaborate to develop a strategy for how we enroll patients into post-discharge support programs in a way that is both equitable for patients and maximizes benefit to the health system.
Collaborate to create an infrastructure that will allow us to monitor, study, and improve this care model.
On the financial side, the funding support from Caring Wisely will be critical to the success of this initiative. The funding will be partly used to help build the reports in APeX that are necessary to enable these new workflows, as well as offer salary support for team members so that they have time to design and implement these highly complex and multidisciplinary workflows.
Our ultimate goal, should the initial pilot be successful, is to expand the use of this model to all patients discharged from a UCSF hospital, matching higher risk patients with higher touch discharge support programs. However, that would necessitate additional FTE in population health, so is not included in our cost/benefit calculations.
There are two key technical barriers to implementation. One is being able to pull all of the relevant data from Clarity, on a daily basis, to a secure environment where it can undergo the necessary processing steps, and then be run through the model to generate outputs. Fortunately, UCSF has HIPAC, which is built for this exact purpose, and we hope to leverage HIPAC and its standards and policies to implement the model. All of the code to pull and process data from Clarity has already been written, which will significantly decrease the time needed to accomplish this integration. The DSI team, which manages HIPAC, has expressed their support for this work, especially if it is selected as a Caring Wisely awardee. A second barrier is to take the outputs from the model, and then send them back to APeX where they can be used. If our proposal is selected, we plan to use the Caring Wisely funds to build the necessary interfaces and reports with the help of the APeX interface team.
The main adverse outcome that the proposal team wants to prevent against is the situation of having the model propagate and/or magnify disparities and inequities among vulnerable patient populations. To prevent this risk, the proposal team plans to test the model’s performance across different racial/ethnicity groups and insurance classes. If there are disparities in model performance across these groups, we would either try to modify the model at the global level to see if we can correct these disparities, or customize the model to have different thresholds for different patient populations so that the model’s performance is uniform across these populations.
PROPOSED EHR MODIFICATIONS
The goals of implementing this machine learning model are (1) to identify patients at high risk for readmission and (2) use that risk prediction as one of the components to drive enrollment into discharge support programs, so as to better allocate discharge support resources toward the goal of decreasing unnecessary readmissions. In order to achieve these goals, we will need to create an APeX report of patients identified as high risk for readmission. This will be used by OPH and Case Management to track and provide additional support to these patients after discharge.
COST
Costs to the health system: All costs are covered within the budget below.
Savings/benefits to the health system
Immediate financial benefit:
Increase contribution margin from backfill: By preventing moderate acuity readmissions in this small ACO pilot population, we anticipate saving 403 bed days, which we expect to lead to additional contribution margin of nearly $1.2 Million/year.
Number of ACO patient discharges in pilot | 6,000 |
Number of readmissions expected | 600 |
Number of readmissions captured in ML model | 420 |
Readmissions prevented | 63 (15%) |
Average bed days per readmission | 6.4 |
Bed days saved | 403.2 |
Backfill revenue per bed day* | $ 2,937 |
Net impact | $ 1,184,198 |
Other immediate benefits:
Reduce ED overcrowding: Although this is difficult to predict based on available literature, we would anticipate that this proposal would result in a 5% absolute reduction in ED revisits, or 300 prevented ED visits per year, decreasing ED overcrowding.
Improve quality of care: UCSF publicly reports its readmissions metrics, and they are a commonly cited quality metric. An improved readmission rate could help further improve the already excellent reputation of UCSF for providing world-class care.
Long term benefits
Note: Calculations for this section are based on the long-term assumption that after demonstrating net financial benefit of this model, we would be able to apply it to all discharged patients, rather than just those in ACOs. Based on the literature, we predict that that by better determining which patients to target with discharge support programs, we can improve 30-day unplanned readmission rates by 15%, from 9.7% during the last fiscal year to 8.25%. This would have several benefits:
Decreased Hospital Readmission Reduction Program penalties: UCSF has been penalized under the CMS HRRP each of the last 8 years. The median penalty in the past 3 years has been approximately $1 Million. If we can selectively enroll high-risk patients into discharge support programs, particularly for conditions covered in HRRP, we anticipate UCSF would outperform the expected readmission rate in the 6 conditions that are part of this program, eliminating that penalty.
Increased backfill revenue for general population: If we can demonstrate net financial benefit using this model, we can advocate for additional resources for population health teams to be able to enroll high risk non-ACO patients into discharge support programs, which would multiply the benefit of this program five-fold.
SUSTAINABILITY
Incorporating readmission risk score will become part of standard work for patient enrollment into key discharge support programs within the OPH, and leaders in this space (GI, MH, TJ, SR) will champion this practice going forward, and hope to expand this practice further to all UCSF patients.
From a technical side, once the ML model is deployed in HIPAC, and the interface to deliver its results into APeX is built, there should be minimal technical upkeep needs to keep this pipeline intact. However, the model’s performance will need to be observed regularly (e.g. every 6-12 months), as ML model performance can degrade over time as trends in diseases, workflows, and patient characteristics change. The model will likely need to be updated at some regular interval, although this would be a good opportunity to improve the model as well. These updates will need significantly less resources than the initial build, and the proposal team is committed to doing so.
BUDGET
APeX Interface team - $15,000 to build interfaces in APeX to display model results in APeX
Salary support for technical work (algorithm optimization, integration): $20,000
Salary support for operational work (oversee report building, protocol creation in OPH): $15,000
Works Cited
[1] Romero-Brufau S, Wyatt KD, Boyum P, Mickelson M, Moore M, Cognetta-Rieke C. Implementation of Artificial Intelligence-Based Clinical Decision Support to Reduce Hospital Readmissions at a Regional Hospital. Appl Clin Inform. 2020 Aug;11(4):570-577
[2] Wu CX, Suresh E, Phng FWL, Tai KP, Pakdeethai J, D'Souza JLA, Tan WS, Phan P, Lew KSM, Tan GY, Chua GSW, Hwang CH. Effect of a Real-Time Risk Score on 30-day Readmission Reduction in Singapore. Appl Clin Inform. 2021 Mar;12(2):372-382.
[3] Luo AL, Ravi A, Arvisais-Anhalt S, Muniyappa AN, Liu X, Wang S. Development and Internal Validation of an Interpretable Machine Learning Model to Predict Readmissions in a United States Healthcare System. Informatics. 2023; 10(2):33.
[4] Huang Y, Talwar A, Chatterjee S, Aparasu RR. Application of machine learning in predicting hospital readmissions: a scoping review of the literature. BMC Med Res Methodol. 2021 May 6;21(1):96.
[5] Eckert C, Nieves-Robbins N, Spieker E, et al. Development and Prospective Validation of a Machine Learning-Based Risk of Readmission Model in a Large Military Hospital. Appl Clin Inform. 2019;10(2):316-325.
[6] Ko M, Chen E, Agrawal A, et al. Improving hospital readmission prediction using individualized utility analysis. J Biomed Inform. 2021;119:103826.
[7] Schiltz NK, Dolansky MA, Warner DF, Stange KC, Gravenstein S, Koroukian SM. Impact of instrumental activities of daily living limitations on hospital readmission: an observational study using machine learning. J Gen Intern Med. 2020;35(10):2865-2872.
[8] Liu X, Anstey J, Li R, Sarabu C, Sono R, Butte AJ. Rethinking PICO in the Machine Learning Era: ML-PICO. Appl Clin Inform. 2021 Mar;12(2):407-416.
Comments
A big thank you to the Caring
A big thank you to the Caring Wisely team for advancing our proposal to round 2. We have reviewed your feedback and edited our proposal to address your questions. In addition, we have highlighted our responses below for ease of viewing. We look forword to your next round of review, and are happy to answer any additional questions.
As discussed in prior emails, due to the time limitations for edits, and the nature of our outcome of interest (future 30 day unplanned readmissions), we cannot fully prospectively validate our model's performance by March 31st. However, what we can do is report results from the hold out test set (data from 2022, not used for any model training or tuning), in addition to our results from the validation set (data from 2021, used for model tuning). Consistency in performance across the validation and test sets will help us feel more confident that the model will generalize well, and not simply overfitting to the validation set. Our model demonstrates an AUC of 0.749, and AUPRC of 0.250 on the validation set. The model’s performance on the hold out test set is very similar with an AUC of 0.738 and AUPRC of 0.245. We feel like this offers strong support that the model will perform well to new data from UCSF as well. In addition, we have highlighted some requested outcome metrics starting from paragraph 5 in the intervention section.
In testing the model, we have gained several insights that can help us not only improve the model further, but also refine the downstream clinical workflows that result from the model’s predictions.
1) In the time interval since Caring Wisely round 1, we have added a large number of additional features into the model. This includes: national and state ADI indexes, outpatient utilization prior to hospitalization, # of days on antibiotics, # days of on pressors, if the patient required ICU level care, Charlson Comorbidity Index (CCI), PMH groupers based on CCI, and whether or not the patient had PCP appointments set up at the time of discharge. Despite the addition of these features, the model's performance only improved slightly. We suspect that adding additional structured features will have similarly limited impact. This is an insight from Dr. Ahmed Alaa, who is a faculty member in the joint UCSF/Berkeley Computational Precision Health Program. Dr. Alaa will be joining our team as a ML subject expert. Based on his experience, he suggests that we add other modalities of data such as clinical notes, as there might be important information captured in case management, occupational and physical therapy, physician, or social work notes that are not available as structured data elements. This will be our main area of focus in terms of next steps to refine the model further.
2) In reviewing the results of the model, we noticed that there seemed to be multiple discrete cohorts of patients. For example, patients with the highest predicted risk scores tend to have very high prior utilization, be middle-aged, and medically complex. The group of patients with the 2nd highest predicted risk scores tend to be among the oldest, with the highest medically complexity, but have lower prior utilization compared to the group with the highest predicted risk scores. Based on this insight, we believe it makes sense to study these cohorts further to determine whether they may benefit from different types of post-discharge support interventions.
We are excited by the opportunity to partner with the Caring Wisely team and leverage their expertise in program design, lean-thinking, process improvement, and implementation science to develop and implement these novel workflows. Some specific areas where Caring Wisely support and funding will help accelerate our initiative forward include:
1) Explore additional features to add to the model that have not been tried yet
2) Review cohorts of patients identified by the model that have different levels of risk, and design possible interventions and workflows that are best suited to address the needs of the different cohorts.
3) Collaborate to develop a strategy for how we enroll patients into post-discharge support programs in a way that is both equitable for patients and maximizes benefit to the health system.
4) Collaborate to create an infrastructure that will allow us to monitor, study, and improve this care model.
5) On the financial side, the funding support from Caring Wisely will be critical to the success of this initiative. The funding will be partly used to help build the reports in APeX that are necessary to enable these new workflows, as well as offer salary support for team members so that they have time to design and implement these highly complex and multidisciplinary workflows.
All of the features in the model were designed so that they would be available the day after discharge. We have confirmed that this is working as intended.
This is quite an amazing
This is quite an amazing proposal with a tremendous amount of pre-work and testing - kudos! I'm very excited to see how new informatics approaches can be applied to real-world problems like this one. This would be an outstanding demonstration of how a learning health system should work.
Margaret Fang, Chief of UCSF Hospital Medicine
Thanks for your support
Thanks for your support Margaret!
This superb innovative work
This superb innovative work focused on applying machine learning to a very important area of study will undoubtedly be hugely beneficial to further our understanding on readmissions and allow for clinical informatics to be applied to answer many more valuable clinical questions to support the post discharge process.
Thanks for your support
Thanks for your support Lauren!
Great work--can you please
Great work--can you please clarify if there is somebody on the team who will be dedicated to project management activities throughout the year--the budgeted support for project FTE seem to be more involved in technical/operational aspects of the work.
Thanks Brian! Yes, to clarify
Thanks Brian! Yes, to clarify, I (Xinran Liu) will be dedicated to project management activities throughout the year in additional to technical work. This was assumed in my mind, but forgot to formally document until now.
Hope that helps!
Leo
Thank you for your excellent
Thank you for your excellent submission. Feedback below from the Selection Committee.