1. The UCSF Health Problem  

     UCSF Neurology receives approximately 125 outpatient referrals daily, with average patient wait times for some subspecialties ranging 6 - 9 months. Wait times for patients to see a neurologist at UCSF are significantly higher compared to the national average, where a recent study found that the average wait time for an appointment is 34 days for Medicare participants (1). While there are many factors to explain this discrepancy, including geographic location, specific neurological conditions managed and healthcare system capacity, there is significant room for improvement in the current referral processing system.    

     To schedule a patient in the appropriate subspeciality clinic, the current pathway for external referrals relies on non-clinicians to manually review unstructured and often incomplete referral packets and then use the APeX scheduling decision trees to select the single most appropriate input for chief complaint to ultimately schedule patients with the appropriate subspecialty clinic.  Accommodating rapidly changing clinical/paraclinical criteria for acceptance of referrals in this rules-based system requires labor-intensive collaboration between clinicians and specialized programmer support. 

     These factors contribute to the scheduling of lower priority referrals, improper utilization of subspecialty clinic slots, extended wait times for appropriate referrals, and increased no show/cancellation rates, as patients opt to seek care elsewhere.  Delays in diagnosis and management can lead to poor health outcomes, increased acute care utilization and increased healthcare expenditures.  Furthermore, mis-triage negatively impacts patient satisfaction, increases healthcare system burden, leads to staff and clinician burnout and fosters moral injury. 

2. How Might AI Help?  

The administrative (and clinical) staff require the proper tools to efficiently identify and interpret complex medical information and improve the precision and accuracy referral routing.  Large Language Models (LLMs) are increasingly being used for clinical decision support and administrative automation, leading to better workflow efficiency and improved access to specialized care. An AI-driven process can enhance efficiency in neurology referrals by:  

Since many referrals do not have a clear clinical indication or may cite multiple clinical indications, an AI tool could also be employed to gather information outside referral letters to better prioritize and categorize referrals (2). AI-based solutions have helped to streamline referral processing and have shown promise in decreasing time to treatment for oncology patients and improving overall patient experience (3).   

We are proposing a staggered two-step process to integrate ATriA: 

3. How Would an End-User Find and Use It?  

Versa API Enhancement  

Versa is now available to all staff who take the prerequisite AI training and request access. The individual would select the triaging assistant for that department (API, department-specific database), upload referral PDFs, and then provide queries related to that individual staff members role. 

APeX Build forATriA 

The ATriA tool would integrate directly into the existing APeX system, processing referral documents through an automated pipeline:  

There will be an actionable button labeled “Process referral” near the top of the Appointment Request interface in APeX.  There will be another actionable button labeled “Generate indication” under the “Indications” section of the Appointment Request interface in APeX. 

4. AI Tool Visualization 

Versa Enhancement Request 

ATriA Apex Build Request 

5. What Are the Risks of AI Errors?  

Limitations to the implementation of LLMs include concerns regarding accuracy, reliability and potential bias (4). While AI can streamline referral processing, potential risks include, misinterpretation of clinical data, AI hallucinations, bias in model training leading to potential underperformance in underrepresented populations or rare neurological disorders, and breaches in regulatory compliance particularly with HIPAA compliance and data security. To mitigate risks, a human-in-the-loop approach will be used, ensuring patient coordinators review AI-generated summaries before finalizing referrals. Additionally, an initial validation phase will be conducted to assess AI accuracy and regulatory compliance before full implementation.  

6. How Will We Measure Success?  

Initial validation and workflow optimization will focus on a cohort of internal referrals.  Subsequent efforts will focus on external referrals, which are often more fragmented in terms of referral information.  

Key Metrics: collected in APeX:  

Key Metrics: Ideal  

This AI-driven approach represents a hybrid model between rule-based and machine-learning methodologies, offering a scalable and sustainable solution to the challenges of neurology referral processing.   We anticipate that with optimization these methodologies can be utilized in non-neurological specialities.

7. Describe Your Qualifications and Commitment  

     The project will be spearheaded by UCSF General Neurology Division Technology and Division Chiefs (Pierre Martin, Maggie Waung) in collaboration with the UCSF Clinical Administrative Director (Mark Datuin). We request salary support for the academic co-leads, Pierre Martin and Maggie Waung.   

     Pierre Martin is an outpatient general neurologist with SmartUser certification for Epic and has an academic focus on educational technology.  He helps to develop SmartTools and other resources for colleagues to improve clinical efficiency.  He recently secured Innovations Funding for Education to develop an immersive mobile application for medical trainees to learn clinical neuroanatomy via an interactive 3D model.  

     As the General Neurology Division Chief, Maggie Waung has been intimately involved in clinical operations over the past 3 years. She assisted with development of the clinical decision tree and has worked closely with the Ambulatory Clinical Informatics lead, Katie Grouse on optimizing patient triage for General Neurology over the past year. She also works closely with all stakeholders that might benefit from this project including clinical providers (MDs and APPs), LVNs, RNs, patient coordinators, clinic managers, other Neurology Division Chiefs, and the Neurology Vice Chair for Clinical Affairs, John Engstrom.       

     We will plan on weekly meetings and incorporate time for work-in-progress sessions with the Health AI and AER teams. We plan to provide monthly updates to the General Neurology Division and Clinical Division Chief meetings to solicit feedback as needed.