Section 1. The UCSF Health Problem. Despite major advancements in antiretroviral therapy (ART), challenges remain for people with HIV (PWH) who are sub-optimally engaged in care due to treatment misalignment, stigma, unstable housing, mental health needs, and other barriers that compromise adherence.^1,2 Clinical decisions often follow standardized protocols, overlooking patient preferences and psychosocial factors—typically buried in unstructured electronic healthcare record (EHR) notes. This gap impedes the delivery of personalized HIV care and contributes to persistent disparities in treatment engagement and viral suppression, particularly among underserved populations.³ Previous approaches to improving care engagement at UCSF have leveraged structured data to flag patients at risk for non-adherence, but such models offer little insight into why disengagement is occurring or how to respond effectively.⁴ We propose a novel clinical decision support (CDS) tool that combines natural language processing (NLP) of unstructured clinical notes with real-time visualization of key structured data to support individualized treatment planning for PWH. The tool will extract and display patient-stated treatment preferences and barriers, overlayed with ART history, resistance, lab trends, adherence, and engagement—within a collapsible interface. End-users will be HIV and primary care providers at UCSF and affiliated sites, who need timely, actionable insights to tailor care for patients who need it most. By highlighting this information at the point of care, the tool aims to transform hidden insights into proactive, equitable, and patient-centered HIV care.

Section 2. How might AI help? Artificial intelligence and NLP can highlight patient-centered insights buried in unstructured EHR notes.⁴ While structured EHR data can flag PWH at risk for non-adherence, it rarely reveals why engagement is faltering or how best to intervene in a patient-specific, actionable way.⁵ Predictive models may offer promise in identifying insights to retention to care,⁶ and our approach will use NLP to extract tailored treatment preferences, psychosocial barriers, and care facilitators from both provider documentation and electronic patient intake and feedback forms (e.g., treatment delivery preferences, concerns about stigma or side effects, housing needs, food support, or transportation assistance). These extracted insights will be integrated into a provider-facing CDS tool that also synthesizes structured data elements critical to HIV care. By combining intelligent data extraction with thoughtful interface design, this tool will empower providers to deliver personalized and equity-informed care to PWH--particularly those sub-optimally engaged.

Section 3. How would an end-user find and use it? The AI-enabled CDS tool will be embedded within the provider-facing EHR interface and designed for ease of discovery and use during routine workflows. It will be most valuable during pre-visit planning, in-visit decision-making, and multidisciplinary case reviews—particularly when caring for PWH who are sub-optimally engaged or have a detectable viral load. The tool will appear as a collapsible sidebar or within a dedicated clinical tab, requiring no change to existing documentation practices. When opened, the interface will display key structured and unstructured patient insights. NLP-derived summaries of treatment and care delivery preferences, barriers (e.g., stigma, confidentiality concerns, mental health, unstable housing), and facilitators (e.g., mobile clinic access, social work support) will appear alongside structured data visualizations such as an ART medication history timeline (with individual drugs and resistance overlays), a hoverable tooltip summarizing the current regimen and resistance concordance, HIV viral load and CD4 trends, a refill-based adherence timeline, appointment adherence percentage, and a timestamped record of counseling on comprehensive support services. High-yield features such as a traffic light–style engagement risk score and a follow-up alert for patients without upcoming appointments provide rapid, actionable context. Each AI-derived item will include a confidence score and a link to its original source (e.g., provider note, intake form, lab result). Providers will be able to validate, dismiss, or amend insights with a single click; trigger referrals; or initiate preference-aligned ART discussions directly from the interface. Hover tools and minimal clicks keep the CDS intuitive—not disruptive. Designed around principles of efficient interpretability and low cognitive burden, the tool transforms traditionally siloed insights into a unified, actionable view that enables providers to deliver more personalized, responsive, and equitable HIV care.

Section 4. Embed a picture of what the AI tool might look like. See Supplement Figure 1. CDS Tool Example

Section 5. What are the risks of AI errors?  This project involves several potential risks related to AI-driven summarization and data integration, including false negatives (missed insights), false positives (misinterpreted or overstated insights), hallucinations (unsupported or fabricated content), and inconsistent summary performance across populations.⁷ The AI may fail to detect or highlight key patient preferences—such as a desire for long-acting ART or a transportation barrier—limiting personalization. It may also misread clinical context (e.g., flag outdated issues or produce summaries not grounded in documentation). There is also a risk that the model underperforms for certain populations, particularly those with sparse or non-standard documentation styles, which could reinforce existing health disparities.⁸ To mitigate these risks, we will take a multi-layered approach. First, the model will be designed as an extraction (not generative) tool, constrained to surface directly documented concepts with confidence scoring and source-linked outputs for transparency. Second, we will develop and test the model against a gold-standard annotated dataset of clinical notes, allowing us to benchmark performance using recall, precision, and false discovery metrics. Manual audits and provider/patient validation and feedback during the pilot phase will enable real-time refinement. Third, performance will be evaluated across demographic and clinical subgroups (e.g., race/ethnicity, gender identity, language) to identify and address equity gaps. Finally, the interface will be designed to support human review and easy correction of outputs, keeping the clinician-patient partnership in control at all times. This layered strategy ensures the tool remains safe, transparent, and equitable.

Section 6. How will we measure success? We will assess success using an evaluation approach informed by the RE-AIM framework, focusing on adoption (provider usage), effectiveness (impact on clinical decisions and patient outcomes), and maintenance (potential for integration into long-term clinical workflows).⁹ We will compare results pre- and post-implementation, track variation across provider groups and patient subpopulations, and incorporate both quantitative and qualitative metrics. If the tool demonstrates high clinical utility and provider trust, we will pursue further scaling within UCSF Health. If adoption or impact is limited, we will revise or discontinue implementation. Data measurements already collected in APeX include: 1) CDS tool interaction logs: Frequency of tool access, hover expansions, link clicks, and edits to summaries; 2) Documentation changes: Increase in capture of preferences, barriers, and facilitators post-implementation; 3) ART regimen adjustments: Number of preference-aligned regimen changes made after tool use; 4) Referral patterns: Uptake of case management, behavioral health, and social services; 5) Appointment engagement: Change in visit attendance and no-show rates among flagged patients; 6) Clinical outcomes: Viral suppression among sub-optimally engaged patients over time; 7) Provider usage patterns: Tool usage by role (e.g., HIV PCPs, ID specialists, case managers). Additional ideal measurements to evaluation success include: 1) Provider satisfaction and utility (via surveys/interviews); 2) Time saved: Reduction in pre-visit preparation or chart review time; 3) Accuracy and trustworthiness: Validated through clinician review and audit of AI outputs; 4) Equity analysis: Tool performance and outcome disparities by race, ethnicity, language, or insurance status; 5) Patient-reported experience: Patient perspectives on care personalization, engagement, and satisfaction; 6) Annotation benchmarking: Recall, precision, and F1 scores against a gold-standard annotated dataset

Section 7. Describe your qualifications and commitment: This project will be co-led by Jose Gutierrez, PhD, FNP-BC, AAHIVS, Malcolm John, MD, and Michelle Cohen, FNP-BC, MPH, AAHIVS— an interdisciplinary group of UCSF clinician-researchers with complementary expertise in HIV care, health equity, informatics, and implementation science. Dr. Gutierrez is an Assistant Professor in the UCSF Department of Family Health Care Nursing. His research focuses on understanding patient preferences to develop AI-enabled decision support tools for personalized HIV treatment. He is also co-founder of the UCSF School of Nursing Artificial Intelligence Interest Group and serves on the UCSF CRIO Advisory Council. Dr. Malcolm John is a Professor of Medicine and Director of the UCSF Health Black Health Initiative. A long-standing leader in HIV care and health equity, Dr. John brings deep expertise in operational innovation and culturally responsive care for marginalized communities to shape system-level strategies to reduce disparities and improve care engagement across UCSF Health. Michelle Cohen is a family nurse practitioner and HIV specialist at both the UCSF 360 Positive Care Center and UCSF Women’s HIV Program. In addition to her HIV clinical knowledge, Michelle brings prior experience in global HIV prevention advocacy with AVAC and frontline HIV clinical research, including the landmark iPrEx PrEP study in Peru and HIV cure studies based at ZSFG. Together, they bring expertise in HIV care, informatics, and system-level implementation. All team members are committed to active participation in regular work-in-progress meetings and ongoing collaboration with the UCSF Health AI and AER teams throughout the project year to ensure rigorous, real-world integration of the CDS tool.

References

1.             Marcus R, Tie Y, Dasgupta S, et al. Characteristics of Adults With Diagnosed HIV Who Experienced Housing Instability: Findings From the Centers for Disease Control and Prevention Medical Monitoring Project, United States, 2018. J Assoc Nurses AIDS Care 2022; 33(3): 283-94.

2.             Rooks-Peck CR, Adegbite AH, Wichser ME, et al. Mental health and retention in HIV care: A systematic review and meta-analysis. Health Psychol 2018; 37(6): 574-85.

3.             Shade SB, Marseille E, Kirby V, et al. Health information technology interventions and engagement in HIV care and achievement of viral suppression in publicly funded settings in the US: A cost-effectiveness analysis. PLOS Medicine 2021; 18(4): e1003389.

4.             Patra BG, Sharma MM, Vekaria V, et al. Extracting social determinants of health from electronic health records using natural language processing: a systematic review. J Am Med Inform Assoc 2021; 28(12): 2716-27.

5.             Ridgway JP, Uvin A, Schmitt J, et al. Natural Language Processing of Clinical Notes to Identify Mental Illness and Substance Use Among People Living with HIV: Retrospective Cohort Study. JMIR Med Inform 2021; 9(3): e23456.

6.             Oliwa T, Furner B, Schmitt J, Schneider J, Ridgway JP. Development of a predictive model for retention in HIV care using natural language processing of clinical notes. J Am Med Inform Assoc 2021; 28(1): 104-12.

7.             Ji Z, Lee N, Frieske R, et al. Survey of Hallucination in Natural Language Generation; 2022.

8.             Obermeyer Z, Powers B, Vogeli C, Mullainathan S. Dissecting racial bias in an algorithm used to manage the health of populations. Science 2019; 366(6464): 447-53.

9.             Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health 1999; 89(9): 1322-7.