Chief Health AI Officer  |  Executive VP of Physician Services  |  Chief Medical Information Officer
Chief Nursing Informatics Officer  |  Chief Innovation Officer  |  AVC Clinical Research  |  CTSI
Chief Research Informatics Officer  |  BCHSI  |  Division of Clinical Informatics & Digital Transformation
Chief Information Officer  |  Chief Data Officer

1. The UCSF Health problem.

Tacrolimus is a major immunosuppressive drug in solid organ transplantation.¹ Due to its narrow therapeutic index, however, tacrolimus administration requires strict monitoring and adjustment to achieve optimal therapeutic dosages. Excessive dosages may increase the risk of nephrotoxicity while insufficient dosages can lead to acute rejection.² Consequently, transplant patients require lifelong monitoring of tacrolimus trough concentrations.

1. The UCSF Health Problem
Orthodontic treatment requires regular radiographic imaging to monitor tooth movement, ensure proper bracket positioning and evaluate the risk of root resorption [1]. However, due to long term orthodontic treatment (average of 24 months), it is not uncommon for orthodontists to occasionally overlook the timing of radiographs, potentially leading to delayed treatment adjustments and prolonged overall treatment duration. Additionally, orthodontic patients should receive dental checkups and cleanings every six months, or more frequently, if necessary, to maintain oral health and prevent issues such as interproximal caries. Unfortunately, orthodontists may overlook these routine reminders, increasing the risk of caries development during treatment [2,3].
A solution is needed to automatically track patient radiographic and dental cleaning records, issuing timely reminders to ensure adherence to these critical steps in the orthodontic workflow. Previous attempts at addressing similar issues include manual tracking and reminders from administrative staff, but these approaches are prone to human error. The intended end-users of this solution include orthodontists, dental assistants, and administrative staff who manage patient records and appointments.
2. How Might AI Help?
AI can assist in monitoring and analyzing patient records by identifying patterns and issuing timely reminders. The AI system would utilize patient electronic health records, including past radiographs and dental cleaning history, to track due dates for necessary imaging and oral hygiene appointments.
The AI model would produce automated reminders when a patient is due for a radiograph or dental cleaning, ensuring timely intervention. It would help solve the problem by reducing the likelihood of missed imaging or hygiene appointments, thereby improving treatment efficiency and preventing oral health complications.

3. How Would an End-User Find and Use It?
The AI support would be integrated into the existing orthodontic patient management system (Apex), functioning within the standard workflow. When a patient’s record is accessed, the system would check whether an X-ray or cleaning is due based on predefined intervals. If due, a pop-up notification would alert the orthodontist or staff, prompting immediate action.
End-users would see these notifications as part of the patient’s digital chart. The recommendations would be explained through a brief summary indicating the last recorded radiograph or cleaning date and the recommended next steps. Users can then schedule the necessary appointments directly from the interface. Additionally, the system may allow customization of reminder frequency based on patient-specific needs. This AI-driven automation would enhance patient care by ensuring timely interventions and reducing reliance on manual tracking.
      

5.What are the risks of AI errors? 

• False Positive Errors: The system may incorrectly remind a patient as needing an X-ray or dental cleaning when they are not actually due. This could lead to unnecessary imaging or redundant appointments, increasing patient costs and exposure to radiation.
• Mitigation: Upon receiving a reminder, the orthodontist needs to determine whether an X-ray should be taken or if the patient should be referred for a dental cleaning. Additionally, if the patient’s general dentist does not use the APeX system and the cleaning was performed at another clinic but is not recorded in the system, the orthodontist can select the “refuse with a comment” option and manually enter the cleaning date.
• False Negative Errors: The AI may fail to flag a patient who is due for an X-ray or dental cleaning, leading to missed appointments and potential treatment delays or oral health complications.
• Mitigation: Regular audits of AI predictions against actual clinical schedules and patient histories can help identify and correct systematic under-reporting.
• Hallucination Errors: Generative AI components, if used, may provide incorrect or misleading recommendations based on incomplete or misinterpreted data.
• Mitigation: Restrict AI-generated outputs to structured data analysis rather than free-form text generation, and ensure recommendations are based on verified clinical guidelines.
• To measure and mitigate these risks, continuous monitoring of AI performance metrics—such as accuracy, recall, and precision—will be necessary. Additionally, user feedback from orthodontists and dental staff should be collected to refine AI predictions and minimize errors over time.

6.How will we measure success? 
To determine whether the AI system is being effectively used and is achieving its intended goals, we will measure outcomes in two categories: data already being collected in APeX and ideal supplementary measurements.

  a. A list of measurements using data that is already being collected in APeX

• Reminder Utilization Rate: Track how often the AI-generated reminders are triggered and viewed within the system.
• Follow-up Compliance: Measure the percentage of patients who complete dental cleanings and radiographs within recommended intervals.
• Caries Incidence: Compare caries incidence rates before and after orthodontic treatment among patients with and without AI-supported reminders (if general dentists also use APeX).
• Treatment Plan Audits: Evaluate completed treatment plans for documentation of oral hygiene checkups and imaging completion.
• Time-to-Treatment Adjustment: Assess whether timely radiographic imaging correlates with more prompt bracket repositioning and treatment plan modifications.

  b. A list of other measurements you might ideally have to evaluate success of the AI

• User Feedback and Satisfaction: Surveys and interviews with orthodontists and staff about AI usability, helpfulness, and alert fatigue.
• Clinical Outcome Improvement: Reduction in the number of delayed treatments or undiagnosed caries due to missed cleanings or imaging.
• Reduction in Manual Tracking: Quantify decrease in staff workload related to manual scheduling or tracking of appointments.
• Patient Satisfaction Scores: Assess patient perception of care quality, particularly regarding timely follow-ups and preventive care.
• False Alert Rate: Monitor and categorize false positives and negatives to continuously refine AI accuracy.
• To convince UCSF Health leadership to continue supporting the AI, we would need to demonstrate improved patient care outcomes (e.g., reduced caries, faster treatment completion), higher adherence to recommended dental care timelines, and increased provider satisfaction. If the AI results in low adoption, excessive false alerts, or no measurable improvement in clinical outcomes, it may indicate a need to reevaluate or discontinue implementation.

The UCSF Problem:

Lung nodules are a common finding on CT and are often benign but rarely may represent malignancy. To monitor for potential malignancy, follow-up imaging is often recommended. If there is concern for malignancy, adequate follow-up allows earlier diagnosis and treatment, while lack of follow-up may lead to progression and worse prognosis.