1. The UCSF Health Problem
Advanced care planning (ACP) is a process that allows patients to name a surrogate decision-maker and to discuss and document their preferences for medical care.^1,2 Centers for Medicare and Medicaid Services (CMS) and other national organizations recommend ACP, and it is an important quality metric often tied to reimbursement.² When ACP information in the electronic health record (EHR) cannot be found during a medical crisis, patients often receive care that is not aligned with their goals, which is widely recognized as an important patient safety issue.^3,4 

To improve ACP documentation, with our team’s guidance, UCSF adopted “clinically meaningful” ACP as a quality metric, which includes both documented discussions and legal forms (e.g., advance directives). UCSF has also created and adopted an “.ACP” SmartPhrase and ACP note title that, along with ACP documents, are “pulled into” a central location in the EHR for easy clinician retrieval.

However, threats to patient safety and adherence to patients’ goals of care abound, particularly in inpatient medicine and surgical settings.^5,6 At UCSF, both inpatient and outpatient clinicians and ACP leaders have spent many hours, year after year, attempting to educate clinicians about using the .ACP SmartPhrase and the ACP note titles to pull this information into the central ACP location in the EHR. With frequent trainee and inpatient clinician turnover, despite ongoing education, clinician use of these documentation innovations is low (e.g., < 5% in surgery). As our team has shown, important ACP information is often buried in clinical notes and ignored during medical crises.^3,7 While semi-automated natural language processing (NLP) can be used to manually search for a list of known ACP terms in the EHR, this manual abstraction is time-consuming, resource-intensive, and not practical during a medical crisis. 

Given national guidelines for ACP, quality metrics tied to reimbursements, clinician burden of documentation, and the need to honor patients’ medical wishes, it is imperative to be able to efficiently gather ACP-related information into a central usable location. Artificial Intelligence (AI) and large language models (LLM) have tremendous promise to identify all documented goals of care conversations in the medical record and ensure this critical information is available to all clinicians in a central ACP location in the EHR. We propose that an AI model would run autonomously in the background without the need for additional clinician or staff support. 

2. How Might AI Help?
Generative AI and large language models (LLMs) can transform the identification of goals of care documentation in the EHR by identifying unstructured clinical data. Our AI solution will analyze EHR data, including all inpatient and outpatient notes, to identify documented ACP discussions by any provider type, including clinicians, social workers, nurses, nurse practitioners, chaplains, and healthcare navigators. The AI tool will function as an assistive technology to find ACP information that clinicians can use at the bedside to help patients and families make medical decisions. The AI and LLM solution will increase the number of listed documented goals of care conversations in the central ACP activity in APeX, decrease the time required per chart to complete surveillance that would otherwise require NLP and manual chart review, and decrease clinician burden in finding and documenting ACP information. Importantly, automating this process has the potential to improve patient safety by ensuring that patients’ stated wishes for medical care are honored and respected. This initiative will improve care for patients across clinical disciplines and across inpatient and outpatient care and will be scalable in multiple other care settings.

3. How Would an End-User Find and Use It?
The AI tool will be designed to run automatically in the background on a daily basis. When documented goals of care conversations are identified in the EHR, they will be added to the established central ACP location in the EHR. The goals of care information will then be made available to clinicians without the need for NLP search queries or manual chart review. In addition, as the model improves over time, it will obviate the need for ongoing clinician education to use the .ACP SmartPhrase and ACP note titles when documenting goals of care.

4. Example of AI Output

Using UCSF Versa with the prompt: “Pull wording from notes related to advance care planning verbatim.”

Discussed with XXXX regarding XXXX’s code status and XXXX would not like to pursue heroic measures should he have a cardiac arrest, ie. Prefers XXXX to be DNR/DNI. XXXX worries that given his already poor baseline quality of life, an event like that may make this worse for him, and it may not be in his best interest to pursue full life measures. XXXX is yet to fill a POLST form but will do so soon.

Surrogate decision maker: XXXX (Conservator)

Life sustaining treatment preferences (i.e. Code): DNR/DNI

5. What Are the Risks of AI Errors?
AI-based identification may introduce risks such as false negatives, false positives, and AI hallucinations. False negatives could lead to a lack of identification of ACP documentation in medical notes, impacting patient safety and quality metric compliance. However, any improvement over the current state, in which most of ACP information in notes is being missed, has the potential to improve patient safety. False positives may result in non-ACP information being included in the EHR central activity (e.g., “noise”). However, because ACP information is so crucial in a medical crisis, it is generally agreed that false positives are preferable to false negatives as this documentation can be quickly screened by clinicians if it is readily accessible in a central location. Through prompt engineering, we will ask Versa to generate notes verbatim to lessen the risk of hallucinations, but we will continue to check for and refine the model.

To mitigate these risks, the model will undergo rigorous validation to ensure reliability and clinical utility: (a) retrospective validation – the AI models will be developed and then validated against historical goals of care notes with .ACP SmartPhrases as the gold standard; (b) among 1000 patients over 6 months we will conduct a prospective validation through manual chart review of clinical notes; (c) qualitative patient queries (n=20) about the accuracy of ACP documentation (i.e., 10 patients on the medical and 10 on the surgical inpatient services representing the average self-reported race/ethnicity background of UCSF inpatients; 10 White (50%), 4 Asian (20%), 3 Black (15%), and 3 Hispanic/Latinx (15%) patients); and (d) clinician feedback. We will continually monitor and manually validate the model to correct for bias and refine it to ensure accuracy. 

6. How Will We Measure Success?
We will follow a cohort of hospitalized patients, 65 years of age and older, admitted to the inpatient medical and surgical services over 6 months. Success will be measured by:

• The increase in documented goals of care discussions in the central ACP EHR activity 6 months after applying the AI/LLM tool compared to a demographically matched comparison cohort 6 months prior to applying the tool
• Using chart review:
  • The positive predictive value and negative predictive value of AI-generated ACP documentation  
  • The time saved using the AI/LLM tool compared to using NLP ACP term queries and manual chart review
  • Qualitative analysis of potential bias of ACP documentation across demographic and clinical subgroups
  • A survey of 20 patients (i.e., 10 medicine and 10 surgical inpatients) to query whether goals of care documentation in the EHR is aligned with their preferences
  • A survey of 20 inpatient clinicians (10 medicine and 10 surgical clinicians) about their satisfaction (5-point Likert scale) with documenting, finding, and using ACP information in the inpatient setting with open-ended questions on how to improve 

7. Describe Your Qualifications and Commitment
This project is co-led by Rebecca Sudore, MD (Division of Geriatrics) and Elizabeth Wick, MD (Colorectal Surgery), and includes Logan Pierce, MD (Hospital Medicine). This interdisciplinary team comprises ACP experts, surgery and hospital medicine clinicians, and an informaticist with a proven successful track record of improving ACP workflows at UCSF. Currently, this team holds a large, multi-center NIH Pragmatic Trial Collaboratory project to test patient-facing, automated ACP EMR interventions. The current proposal builds on their successful collaboration and the glaring need they have identified to improve ACP documentation, decrease clinician burden, and improve patient safety. Please see the letter of support provided by our clinical and operational partners, Drs. Michelle Mourad and Molly Cantor.

Rebecca Sudore, MD, is a geriatrician, palliative medicine physician, implementation scientist, Professor of Medicine at UCSF, and co-director of the Vulnerable Aging Research Core of the NIH-funded Pepper Center. Her research focuses on aging, health literacy, and developing and testing tools to facilitate ACP, particularly for historically marginalized older adults. Dr. Sudore has created ACP interventions and clinical workflows, including automated EHR-based patient interventions, that have been tested in randomized trials.^8-10 In close collaboration with Population Health and Primary Care Strategies, these automated workflows have been adopted by UCSF Health. She will provide leadership in ACP documentation and measurement and ensure the project aligns with the priorities of UCSF Population Health, the inpatient ACP team (please see the letter of support), and national guidelines for ACP.

Elizabeth Wick, MD, is a Professor of Surgery and Vice Chair for Quality and Safety in the Department of Surgery at UCSF. She is an expert in surgical quality improvement and has led multiple national initiatives focused on improving surgical outcomes. Dr. Wick has extensive experience with ACP in surgery and has been instrumental in developing strategies to increase ACP documentation at UCSF, along with Dr. Sudore. She will provide leadership in integrating AI solutions into surgical inpatient workflows. Dr. Wick will work closely with Dr. Rochelle Dicker, section chief acute care surgery, UCSF Health, operational leader, to support this project.

Logan Pierce, MD, is board-certified in both clinical informatics and internal medicine. He is the Managing Director of UCSF Data Core, a team of physician data scientists dedicated to utilizing EHR data to improve healthcare outcomes. He has experience using large language models to extract data from clinical text. Dr. Pierce will actively contribute throughout the development lifecycle, ensuring alignment with UCSF Health priorities and participating in regular progress reviews with Health AI and AER teams.

References:

1.         Sudore RL, Lum HD, You JJ, et al. Defining Advance Care Planning for Adults: A Consensus Definition From a Multidisciplinary Delphi Panel. J Pain Symptom Manage. May 2017;53(5):821-832 e1. doi:10.1016/j.jpainsymman.2016.12.331

2.         Hickman SE, Lum HD, Walling AM, Savoy A, Sudore RL. The care planning umbrella: The evolution of advance care planning. J Am Geriatr Soc. Feb 25 2023;doi:10.1111/jgs.18287

3.         Walker E, McMahan R, Barnes D, Katen M, Lamas D, Sudore R. Advance Care Planning Documentation Practices and Accessibility in the Electronic Health Record: Implications for Patient Safety. J Pain Symptom Manage. Feb 2018;55(2):256-264. doi:10.1016/j.jpainsymman.2017.09.018

4.         Allison TA, Sudore RL. Disregard of patients' preferences is a medical error: comment on "Failure to engage hospitalized elderly patients and their families in advance care planning". JAMA Intern Med. May 13 2013;173(9):787. doi:10.1001/jamainternmed.2013.203

5.         Colley A, Lin J, Pierce L, et al. Experiences with targeting inpatient advance care planning for emergency general surgery patients: A resident-led quality improvement project. Surgery. Oct 2023;174(4):844-850. doi:10.1016/j.surg.2023.04.031

6.         Colley A, Lin JA, Pierce L, Finlayson E, Sudore RL, Wick E. Missed Opportunities and Health Disparities for Advance Care Planning Before Elective Surgery in Older Adults. JAMA Surg. Oct 1 2022;157(10):e223687. doi:10.1001/jamasurg.2022.3687

7.         McMahan RD, Tellez I, Sudore RL. Deconstructing the Complexities of Advance Care Planning Outcomes: What Do We Know and Where Do We Go? A Scoping Review. J Am Geriatr Soc. Jan 2021;69(1):234-244. doi:10.1111/jgs.16801

8.         Sudore RL, Schillinger D, Katen MT, et al. Engaging Diverse English- and Spanish-Speaking Older Adults in Advance Care Planning: The PREPARE Randomized Clinical Trial. JAMA Intern Med. Dec 1 2018;178(12):1616-1625. doi:10.1001/jamainternmed.2018.4657

9.         Sudore RL, Walling AM, Gibbs L, Rahimi M, Wenger NS, Team UCHCPS. Implementation Challenges for a Multisite Advance Care Planning Pragmatic Trial: Lessons Learned. J Pain Symptom Manage. Aug 2023;66(2):e265-e273. doi:10.1016/j.jpainsymman.2023.04.022

10.       Walling AM, Sudore RL, Bell D, et al. Population-Based Pragmatic Trial of Advance Care Planning in Primary Care in the University of California Health System. J Palliat Med. Sep 2019;22(S1):72-81. doi:10.1089/jpm.2019.0142