The UCSF Health Problem

Despite the availability of clinical practice guidelines from the American Association for the Study of Liver Diseases (AASLD) and the American Gastroenterological Association (AGA), adherence to recommended quality measures for patients with cirrhosis remains suboptimal.  This gap leads to a high burden of readmissions and inpatient mortality bore by patients with cirrhosis – avoidable healthcare costs.  Cirrhosis is a dynamic condition that affects multiple organ systems outside of the liver, e.g. brain (encephalopathy), hematology (cytopenias), renal (hepatorenal syndrome), cardiology (cirrhosis related cardiomyopathy), and infectious diseases (spontaneous bacterial peritonitis); this complexity is reflected in the electronic health record, which typically includes relevant clinical data across multiple parts of the patient record.  Additionally, busy clinicians may find it challenging to quickly reference and integrate multiple clinical guidelines.  There is an urgent need for a streamlined solution that ensures guideline-concordant care tailored to each patient’s specific clinical profile and decompensations.

How Might AI Help?

We have previously constructed a liver disease specific Large Language Model (LLM) called “LiVersa” based on integration of AASLD clinical practice guidelines via Retrieval Augmented Generation (RAG-LLM).  LiVersa is the first clinical assistant in the UCSF Versa platform and is currently available for select clinicians in hepatology and liver transplantation (PMID 38451962, PMID 38935858).  We have also constructed “CirrhosisRx,” which is a rule-based (non-AI) clinical decision support system for guideline adherent inpatient cirrhosis care, built on the EngageRx platform using SMART-on-FHIR.  CirrhosisRx is currently deployed in a pragmatic randomized clinical trial (NCT05967273, PMID 38407255) at UCSF Medical Center.  In this proposal, we plan to link the two technologies, specifically integrating LiVersa into CirrhosisRx via Fast Healthcare Interoperability Resources (FHIR) application progressing interface (API) calls within the APeX EPIC EHR system, to provide a patient-personalized dynamic information retrieval system to provide guideline-based clinical recommendations to end-users.

The integrated LiVersa-CirrhosisRx system would provide the following advantages:

• Both Rule-Based and Generative Hybrid Recommendations:  Integration of LiVersa RAG-LLM into CirrhosisRx will allow providers to query and access up-to-date AASLD clinical guidelines without navigating outside of the EHR.
• Personalized Medicine: The proposed application would automatically incorporate relevant patient-specific data, enabling real-time, personalized recommendations specific to the patient and clinical scenario (“Right Patient, Right Time”).
• Streamlined Workflow: Embedding the LLM in a user-friendly CDS interface eliminates the need to toggle between multiple resources, reducing cognitive load and time constraints on clinicians.

How Would an End-User Find and Use It?

The combined LiVersa-CirrhosisRx application will be deployed within the existing “CirrhosisRx” tab that is enabled for select EPIC user contexts under the ongoing pragmatic randomized clinical trial.  See screenshot of CirrhosisRx:

In the LiVersa-CirrhosisRx integration, we will use the FHIR data connected with LiVersa to generate a summary of the patient with displayed data.  A customized free-text box will be built into the CirrhosisRx application that allows the end user to ask open-ended questions regarding the management of the patient.  The embedded LiVersa RAG-LLM will return answers that reference the latest AASLD guidelines while reflecting the patient’s unique clinical situation.  Future expansions could include automated pending of orders and order sets consistent with LiVera’s recommendations.

Example of AI Output

What Are the Risks of AI Errors?

There several potential risks arise when introducing an LLM-based solution into clinical workflows: 1. Hallucinations or Misinformation, 2. Clinician bias due to overreliance on AI, 3. Biases in recommendations, 4. Clinical scenarios un-accounted for by the RAG-LLM model.  To mitigate these risks, we propose a robust pilot testing phase embedded within the existing CirrhosisRx trial.  We will record both clinical outcomes, defined as adherence to AASLD/AGA practice guidelines for inpatient cirrhosis care, and implementation outcomes through structured usability assessments and semi-structured interviews.  We have been developing LiVersa for a use case of hepatology e-consultations and have a 12-question survey developed to evaluate its effectiveness versus human-written consultation recommendations in 54 previous e-consult cases.  The preliminary data from our analyses have demonstrated that the e-consultation drafts produced by LiVersa were helpful 71% of the time with a 4% potential harm rate.  Given that all clinical actions taken resulting from the LiVersa-CirrhosisRx application would have to be confirmed/finalized by a human clinician, there is an integral “human-in-the-loop” mechanism for this proposal.

How Will We Measure Success?

We will track both clinical outcomes and implementation outcomes to determine the effectiveness of our LLM-enhanced CDS tool:

• Clinical Outcomes:

• Guideline adherence - Measure adherence to five AASLD/AGA guideline recommendations, comparing EHR-based metrics before and after the tool’s introduction.
• Rates of Hospital Readmission: Evaluate changes in readmissions in patients with cirrhosis taking place with 90 days.
• Mortality and Morbidity Trends: Evaluate changes in inpatient mortality rates for patients with cirrhosis.

• Implementation Outcomes:

• Time Savings: Track time spent in chart review and decision-making tasks.
• Frequency of CDS Usage: Evaluate how often providers use LiVersa-CirrhosisRx interface in the context of how many potential encounters where it could be used.
• Clinician Surveys: Gather feedback on trust in AI recommendations, ease of use, and perceived impact on practice.

• Equity and Bias Analysis

• Demographic Subgroup Performance: Examine whether the AI recommendations remain consistent across different patient populations with varied demographic backgrounds.

Describe Your Qualifications and Commitment

• This project is led by Dr. Jin Ge, MD, MBA.  Dr. Ge is a transplant hepatologist and a data science and AI researcher within the Division of Gastroenterology and DoC-IT.  He serves as the Director of Clinical AI for the Division of Gastroenterology.  He has experiences in developing, testing, and deploying digital technologies for liver disease care and is the principal investigator for the CirrhosisRx pragmatic randomized controlled trial and LiVersa liver-disease specific LLM.  He has worked closely with both APeX-Enabled Research and the AI Tiger Team.  If selected, he will commit at least 10% effort for 1 year towards this project to ensure its success.
• Dr. Valy Fontil is a former faculty member at UCSF and currently serves as the Director of Research at Family Health Centers at NYU Langone Health.  He is a primary care physician, health services researcher, and digital health entrepreneur who specializes in innovations for high-risk, low-income populations.  He is the innovator of the EngageRx platform, on which CirrhosisRx is based, and has expertise in building digital health interventions in the EHR.  He will serve as an advisor to this effort.