1. The UCSF Health problem: Imaging surveillance represents a cornerstone of brain tumor management and includes surveillance of incidental lesions that may require future treatment or post-treatment follow-up to ensure disease control. Intracranial lesions are relatively common on magnetic resonance imaging and are incidentally seen in 0.7-1.6% of the general population ^1,2. These typically include “benign” tumors that are slow growing and may require years of follow-up. Timely detection of tumor growth on imaging surveillance represents a clinical opportunity for intervention, potentially with less invasive methods such as radiosurgery. Similarly, adherence to follow-up for patients with previously treated non-malignant brain tumors is critical as these individuals often have decades of expected life with ongoing need for imaging surveillance to detect recurrence. Follow-up non-compliance remains a hurdle to care with a paucity of prior studies evaluating rates of follow-up loss and the impact of missed opportunities for intervention. Studies that have previously evaluated this are sparse ^3–6 with some studies demonstrating non-compliance rates of more than 20% ^3,4. Thus, there is a need for novel healthcare interventions that can improve compliance with imaging surveillance for patients with brain tumors and minimize follow-up loss over long periods of time.

This proposal aims to create a closed-loop adaptive AI system in Epic that incorporates a large language model (LLM) to 1) identify provider-requested imaging modality and follow-up time period and 2) predict the probability of follow-up loss to generate tailored reminders for follow-up scheduling. This system could also adapt and modify reminder frequency longitudinally based on patient compliance. End-users include neurosurgical and neuro-oncology providers who evaluate brain tumor patients as well as patients who will receive reminders about imaging and clinic follow-up. Closed loop systems without the use of LLMs have been used in other clinical contexts to encourage follow-up ^7,8. However, this would be a novel approach to this clinical challenge.

2. How might AI help? LLMs provide the opportunity to improve imaging and clinical follow-up through several avenues. An LLM can assess the “assessment and plan” section in clinical documentation to identify provider-recommended follow-up time-period and imaging modality. More importantly, the model may assess clinical data (e.g. past medical history, age, functional status, etc), demographic data (e.g. distance traveled, insurance status, race/ethnicity, etc), and social history (family support, employment status, etc) at the time of the encounter to aid in prediction of risk of follow-up loss. “Non-compliance” risk assessment by the model could provide the ability for an adaptive and tailored closed-loop reminder system with risk assessment and reminder schedules updated at each subsequent visit.

Currently, follow-up compliance relies on clinical staff booking an appointment, ordering follow-up imaging, and reaching out to patients. Staff often set Epic self-reminders that will notify them to initiate the next appointment. However, this still relies on personnel following up on these reminders with potential oversight due to human errors and staff turnover. Additionally, there are difficulties tracking this long-term especially when follow-up may extend out to more than 10 years from diagnosis or treatment. This proposed LLM-based closed-loop system could produce automated reminders (MyChart, email, text, Epic Letters) in a tailored, data-driven fashion to serve as an automated aid to clinical staff and ensure imaging follow-up. The model could also be expanded to continue to assess risk of non-compliance and adapt reminders longitudinally along a patient’s imaging surveillance course.

3. How would an end-user find and use it? The AI support program could be activated through Apex at the time of a clinic visit by any neurosurgical or neuro-oncology providers. At this timepoint, the model would be able to 1) identify the imaging follow-up timepoint and modality placed in the note by the provider, 2) provide a LLM-based risk assessment of follow-up non-compliance which would be reported to providers and be fed forward into a reminder system framework, and 3) initiate a tailored set of reminders to patients at pre-specified time points based on risk profile and tumor type. Additionally, the model could initiate reminders to providers to ensure that imaging orders are placed in preparation for clinic appointments. As the visit approaches, the model could detect when appointments are created and if patients attend those appointments. If compliance is not met, the model could trigger additional patient reminders and incorporate this for future improvements to its predictive performance. Reminders will be mediated through MyChart digital, text message, and automated telephone outreach, depending on patient-selected contact preference. There is an opportunity to tailor the outreach modality as well.

4. Embed a picture of what the AI tool might look like.  We anticipate the AI tool can initially be an “opt in” patient care option for practitioners. There could be a “Yes/No” option in the “Wrap up” section of Apex during an outpatient encounter. Once this is selected, the model would be activated and report a “risk profile” group for that patient, display the provider’s recommended imaging/clinic follow-up pulled from the note for practitioner review, and display a tailored reminder schedule for the patient with the ability for minor customizations by the practitioner to reminder templates. As the next follow-up date is nearing, the reminders would be sent and the model would adapt based on whether the patient schedules follow-up or not. If no appointment is scheduled within the specified follow-up time period, then the model will initiate additional reminders with feedback to the LLM for risk prediction refinement. If compliance is met with scheduling of a visit, the model can re-initiate new reports on provider-specific imaging follow-up for that visit, an updated risk profile, and new reminder schedules. See supporting Figure 1 and 2 for images of mockup AI tool and workflow.

5. What are the risks of AI errors?  AI errors in the context of this platform could lead to misinterpretation of planned imaging follow-up secondarily leading to incorrectly timed follow-up reminders. This could lead to patient confusion and potential need for clarification/error correction from clinical staff. The model may incorrectly identify “at risk” patients for follow-up non-compliance which may change practitioner interactions or management strategy considerations. However, at this time, clinical management strategies will not be changed based on the perceived risk of loss of follow-up. There are several methods to mitigate these errors. Standard practice in neurosurgery clinics is for patient navigators to set reminders on scheduling imaging follow-up. This could be continued on initial implementation of the platform to serve as a “backstop” to ensure that patients are appropriately scheduled. Providers would also be able to check the AI-provided imaging modality and time-period for follow-up as this information would be displayed in the Epic function. Additionally, reminders will be a template outreach with contact information provided so that patients may call back with any questions or requests for clarification.

6. How will we measure success?  Success with this proposal can be measured in several ways, which relate to model development, completion of a closed-loop adaptive framework, and implementation in the clinical setting with the possibility of a prospective, randomized, interventional study. Here are the specific aims/goals of the proposal:

Aim 1: Develop an LLM to reliably extract practitioner-recommended follow-up time-period and the corresponding follow-up imaging modality from clinical and identify patients at risk of non-compliance and provide this data as an output. The target population will include those who undergo upfront imaging surveillance or post-treatment imaging surveillance for brain tumors within the neurosurgery practice. The initial model can be developed based on de-identified clinical documentation in a retrospective fashion with further model performance assessment and refinement based on prospective data collection feedback from Aim 3. We will quantify the accuracy of LLM identification of recommended follow-up time and modality with a target of over 98% accuracy. The performance of the model for risk assessment should include an AUC of greater than 0.7.

Aim 2: Develop a closed-loop framework in Epic to incorporate LLM-identified imaging follow-up timepoint and risk profile to implement tailored patient reminders with adaptive capabilities in the setting of non-compliance. Patient reminders will be mediated through MyChart digital letters, email, text messages, and automated telephone outreach. Reminders to practitioners for imaging orders can also be triggered within this framework. We will quantify the accuracy of execution of reminders based on risk assessment by the model as a marker of success.

Aim 3: Conduct a prospective, randomized, interventional study with a comparison of follow-up adherence between the closed-loop LLM adaptive reminder system (intervention arm) and standard clinical practice reminders (control arm) for patients undergoing upfront imaging surveillance or post-treatment imaging surveillance for brain tumors within the neurosurgery practice. The study endpoint will be to examine the rate of follow-up within 1 year of a specified follow-up time point between the closed loop LLM-based system reminder intervention arm and the standard clinical practice reminder system. Additionally, prospective validation of the model’s ability to identify patients at risk of follow-up non-compliance will be assessed with refinement of the model to improve predictive capabilities.

7. Describe your qualifications and commitment:I am an Assistant Professor and clinical faculty within the UCSF Department of Neurological Surgery with a surgical practice focused on brain tumors and skull base lesions. Many of the patients I manage have tumors that are considered “benign” or slow growing and require long time periods for follow-up, often over 5-10 years. Many of these patients will either require future treatment with an initial period of imaging surveillance to assess for tumor growth or will need long periods of follow-up after upfront treatment. There are limited consensus recommendations for duration of imaging follow-up for non-malignant tumors either for patients who undergo imaging surveillance as an imaging strategy or in the postoperative period. In general, for many of these patients, imaging every 1-3 years is required. I have been working directly with Dr. Madhumita Sushil, who is an Assistant Professor in the Division of Clinical Informatics and Digital Transformation (DoC-IT) - Department of Medicine, Department of Neurological Surgery, and the Bakar Computational Health Sciences Institute (BCHSI). Dr. Sushil has additional expertise in the development of LLMs and will additionally provide guidance on model training, development, and implementation. I have support from my department to participate in regular work-in-progression sessions and collaborate with the Health AI and AER teams to develop and implement this proposal. The framework from this proposal could be more broadly implemented in other disease contexts across the institution (outside of brain tumors), improve the utilization of UCSF-based imaging centers, provide for improved patient quality of care, and streamline outpatient workflow.