The UCSF Health problem
Patients seek preventive health recommendations from their primary care providers (PCP). Health topics include but are not limited to topics like nutrition, musculoskeletal health, injury prevention, and aging well. These broad topics extend beyond the standard 20-minute annual visit and current APeX-derived healthcare maintenance banner topics.
PCPs are ill-equipped to provide tailored recommendations at scale. For instance, one survey showed that 73% of physicians believe nutrition guidance should be part of patient visits, but only 15% feel fully prepared to provide it. Moreover, insurance coverage rules for nutrition services vary by plans. Some plans cover registered dieticians and nutritionists, while others, like Medicare Part B cover for limited qualifying conditions of diabetes, chronic kidney disease, and transplant. Similarly, the confidence levels among PCPs in providing exercise and injury prevention recommendations are considered moderate, ranging between 50% and 67%.
The knowledge gaps extend to trainees. A 2018 survey reported only 29% of US medical schools met the recommended minimum of 25 hours of nutrition education. Fast forward to 2023, only 7.8% of US medical schools now meeting the 25-hour minimum. Similarly, while 83% of US medical schools require some form of musculoskeletal education, explicit didactics on injury prevention is limited.
PCPs and clinicians in training could utilize ProUCare AI tool to generate tailored, shareable preventive medicine guidance which adapts throughout the arc of our patients’ lives.
How might AI help?
My vision is for the ProUCare AI tool to generate tailored recommendations based on the patients’ EHR inputs complemented by evidence-based nutrition science, physical medicine, and occupational medicine foundations.
The recommendations would (1) replace the current BMI-generated AVS nutrition handouts, (2) empower patients in recognizing and reducing occupation-related health impacts, (3) improve PCP confidence in creating preventive health advice at scale, (4) promote patient-clinician practice shifts to wellness from chronic disease management, and (5) improve the medical education foundation across these intersections.
How would an end-user find and use it?
PCPs and trainees would access the ProUCare AI tool during specific settings e.g. the annual visit (pull) and scenarios e.g. patient-led preventive health inquires (push). The tool can be embedded on the screen left side closer to Care Gaps and SDoH icons. Alternatively, the tool can be added in the order section of the Annual Exam SmartSet. (Fig 1, see attachment).
Piloting primary care (faculty, APP, trainee) users would enter their nuanced question in the ProUCare question field. ProUCare would compile data sourced from the patient's record (structured and free text), EHR inputs from questionnaries, social determinants of health data (SDoH), insurance plans coupled with UC resources and vetted references (society guidelines, publications) by the project's panel of experts from Nutrition, Osher, and Physical therapy. (Fig 2, see attachment).
ProUCare's outputs could populate the After visit summary (AVS) (Fig 3, see attachment). Providers could copy/paste relevant subsections into the assessment/plan section within the “Problem List”, and as part of portal message replies for the scenario based inquiries. The three locations are visible to the patient via MyChart between visits and to their care teams across UCSF.
What are the risks of AI errors? What are the different types of risks relevant to the proposed solution and how might we measure whether they are occurring and mitigate them?
Risk 1. Inaccurate recommendations due to incomplete data, and inconsistent data entry practices.
Measurement and mitigation strategies:
- Conduct an inventory of annual exam EHR-based questionnaires, medical, and social history fields.
- Perform a consensus review with nutritionists, physical and occupational medicine to define pertinent components from the inventory which require revision.
- Optimize for structured field versus free-text entry.
- Educate clinicians and care teams on EHR annual exam workflow changes.
- Perform bimonthly to monthly checks for data completeness and AI tool output accuracy.
Risk 2. Misdirection. Recommendations could lead to unnecessary interventions.
Measurement and mitigation strategies:
- Educate and remind PCPs and trainee users of the AI tool’s role as an extender.
- Create a feedback system which enables provider and patient feedback for reporting and analyzing errors in preventive health recommendations. Use the generated feedback to refine input parameters.
- Perform an analysis of the order and referral pattern changes and outcomes based on the AI tool’s recommendations.
Risk 3. Bias. AI-generated recommendations may not account for cultural, regional, or socioeconomic dietary patterns. Similar dataset underrepresentation of occupations and functional status could introduce biases and hallucinations.
Measurement and mitigation strategies:
- Use diverse datasets that include varied demographics, occupations, and dietary patterns
- Employ techniques like Retrieval-Augmented Generation (RAG) to ensure outputs are grounded in patient-specific contexts.
- Use the same feedback system outlined above to incorporate clinical and patient feedback loops to refine recommendations based on usability and relevance.
How will we measure success? What can we measure that will tell us whether intended end-users are using the tool, changing what they do, and improving health for patients or otherwise solving the problem you described above? What evidence would you need to convince UCSF Health leadership to continue supporting the AI in APeX? What results would make you consider abandoning the implementation? Please include 2 subsections:
- ProUCare usage patterns. Number of times opened, timing (during clinic, after clinic hours), setting (annual, Medicare wellness, Mychart message, recommendations based on test results), and time spent accessing the tool.
ProUCare education assessment. Users and patients rate the output utility (thumbs up/down, Likert scale 0-5), number of times reference links are opened per category and by audience (clinician type - faculty, APP, trainee vs patient).
- Practice changes. Measure the amount and rate of change in the completeness of information collected on SDoH questionnaires. Track the number of wellness, preventive-focused ApeX tickets submitted as a result of ProUCare's inclusion e.g. social history structured fields for occupation categories, commute (expressed in hours), screen time (expressed in hours, reasons), cultural influences, budget ranges, stores related to food procurement.
Visit metrics. Number of completed annual and Medicare wellness visits, referral numbers, wait times.
- Population health metrics. Preventive test completion rates (short term during the pilot year), percentages with well-controlled chronic conditions or precursor conditions which normalized e.g. prediabetes to normal.
Patient satisfaction scores: Add a wellness question on practice surverys to gauge ProUCare's utlity to the patients.
- User surveys. During the development phase and post deployment which
- Assess changes in the confidence levels, external tools, resource knowledge and time spent creating tailored preventive medicine recommedations.
- Assess AI usability, design, and alert fatigue.
What evidence would you need to convince UCSF Health leadership to continue supporting the AI in APeX
- Population health
- Preventive health.
- Increased annual exam and Medicare wellness completion rates.
- Improved preventive health test and referral completion rates.
- Indentification and provision of early detection guidance for nuanced and under-represented patient populations currently missed by the current care gaps baner.
- Chronic disease management.
- Reduce the long-term incidence and severity of metabolic and musculoskeletal diseases like diabetes, hypertension, obesity and MASH.
- Reduce total healthcare spend on medications, labs, imaging.
- Education
- Improve providers and trainees' evidence-based foundations in nutrition, complementary alternative medicines, and physical health.
- Encourage patients to seek advice from vetted, compiled, evidence-based sources.
- Community engagement 1. Publicize current and future UCSF resources e.g. condition-focused nutrition group classes, webinars.
- Community engagement 2. Strengthen clinical team and patient links to community-based organizations who may provide low-cost and assistance programs. e.g. food banks, sliding scale chiropractors.
- Clinical team satisfaction
- Reduce administrative tasks required for referrals, appointment tracking.
- Reduce providers' non-billable time spent in chart review and researching information from different sources to devise a crafted recommendations.
- Patient satisfaction. CGHAPs, Press Ganey.
- Service access improvements. Reduce long wait times to impacted referral services like neurology and physical therapy.
- Care coordination 1. Increase patients' awareness of insurance and employer contracted services based on ProUCare's recommendations.
- Care coordination 2. Improve patients' confidence in using ProUCare's advice as a bridge until the specialist appointment.
Dr. Chan Tack is a dual boarded internal medicine and obesity medicine specialist. She is trained in human-centered design and public health management. She successfully led Primary Care Service preventive health colorectal cancer screening initiatives. The program's foundations extended across primary care sites and featured by the California Healthcare Safety Net Institute.
Dr. Chan Tack is an experienced Epic certified leader in digital health innovations across video visits and remote patient monitoring from pilot to system-wide launches. Her work as an informed physician leader enabled UCSF' 3,500+ clinicians, 4MM+ ambulatory video visits.
She is supported by colleagues in the Nutrition Counseling, Osher and Physical Therapy departments. Dr. Chan Tack is committed to collaboration with the Health AI and AER teams for development and implementation of the AI algorithm.
- https://www.pcrm.org/news/news-releases/poll-most-doctors-want-discuss-nutrition-patients-feel-unprepared
- Lee AK, Muhamad RB, Tan VPS. Physically active primary care physicians consult more on physical activity and exercise for patients: A public teaching-hospital study. Sports Med Health Sci. 2023 Nov 20;6(1):82-88. doi: 10.1016/j.smhs.2023.11.002. PMID: 38463668; PMCID: PMC10918360.
- Papastratis, I., Konstantinidis, D., Daras, P. et al. AI nutrition recommendation using a deep generative model and ChatGPT. Sci Rep 14, 14620 (2024). https://doi.org/10.1038/s41598-024-65438-x
- https://blogs.und.edu/cnpd/2024/09/diet-related-diseases-are-the-no-1-ca...
- Special thanks to clinic managers and practitioners in UCSF Nutrition Counseling, Osher Center, and Physical Therapy departments.
Comments
I could not find your Figure
I could not find your Figure 1...you can attach as a PDF if needed.
Have you done any testing
Have you done any testing with patients to see what kind of summaries for preventive care they might want to see? Could you envision this being an interactive tool for patients (e.g. via a chatbot)?