Section 1. The UCSF Health problem. 

• Vaginal bleeding with a positive pregnancy test is a common reason for emergency department visits.¹ The most critical concern in this context is ectopic pregnancy (EP), which occurs in 1–2% of pregnancies and, if undiagnosed, can result in serious morbidity and loss of fertility.^2,3 Diagnosis is challenging because the presenting symptoms are non-specific and often overlap with other early pregnancy complications.⁴ Evidence-based clinical guidelines recommend early use of transvaginal ultrasound (TVS) and serial β-hCG testing to identify intrauterine pregnancy (IUP) or EP.¹ However, when the pregnancy location cannot be determined on initial presentation—a situation termed Pregnancy of Unknown Location (PUL)—further testing and follow-up are required to reach a final diagnosis.⁵

Numerous diagnostic algorithms have been proposed to improve classification and outcomes for women with PUL.⁶ While a 2006 consensus statement laid the foundation for PUL management, inconsistencies remain due to varying definitions and classifications of at-risk populations and outcome categories.¹ More recently the M6 prediction model was developed and modified to include BhCG, ultrasound, and clinical characteristics with and without progesterone measures. This model has been published and externally validated, serving as a starting point for this proposal. (REF) A renewed effort is needed to standardize and streamline approaches and improve real-world implementation (including diagnosis and management) of evidence-based PUL care.⁴

At UCSF/ZSFG, a robust clinical guideline for the management of Pregnancy of Unknown Location (PUL) was updated in 2024 and is currently used by residents and attendings to guide risk assessment, triage, and follow-up decisions (ZSFG PUL Guidelines, 2024). Despite this evidence base, implementation remains manual, variable, and burdensome, as exploratory conversations with residents and attendings uncovered, right now:

• Gyn interns rotate every 5 weeks and spend 1–3 hours daily managing a “floater list” of 15–30+ PUL patients with supervision by a daily rotating gyn attending physician.
• Follow-up plans are documented in free text in APeX and vary based on attending style, leading to inefficiencies and inconsistencies.
• There is no embedded clinical decision support, no structured tool for follow-up tracking, and no patient-facing communication platform. Interns spend hours calling floater list patients daily
• Management depends on interpretation of lab results ultrasound and evolving history, making room for error and delayed diagnosis and intervention for this high-stakes condition.

Section 2. How might AI help? 

Recent developments in artificial intelligence (AI) and machine learning (ML) have shown strong potential to improve PUL management, particularly in outcome prediction, risk stratification, ultrasound interpretation, and individualized care. Key advances include:

• Risk-prediction models such as M6 and modified M6 with clinical characteristics, which use serial β-hCG with and without progesterone values to estimate likelihood of ectopic pregnancy, have demonstrated high diagnostic accuracy (AUCs up to 0.89).^7–10 More complex models—such as neural networks and support vector machines trained on clinical, lab, and imaging data—have shown comparable or better performance.¹¹
• These models now power decision-support algorithms, including the validated two-step protocol (progesterone + M6), which reduces unnecessary follow-up while maintaining high sensitivity for ectopic detection.^6,12
• AI in ultrasound image interpretation has been piloted using deep learning to identify subtle features of ectopic pregnancies that may be missed by less experienced clinicians.^13,14
• AI is also enabling personalized management strategies, including expectant management for stable ectopic cases.^6,10,12

Despite this promising evidence base, AI tools for PUL remain largely absent from routine clinical workflows and are rarely integrated with patient-facing communication systems. UCSF has the opportunity to lead in this space for this complex high-stakes condition often requiring weeks of individual follow-up, by developing an embedded AI tool built on its rich historical EHR data, leveraging the existing M6 model, and up to date care algorithm— enhancing efficiency, accuracy, and equity in PUL care.

We propose TRACE (Triage, Risk Assessment, and Communication Engine) a 3-in-1 AI tool embedded in APeX and MyChart to support both clinicians and patients across the PUL pathway:

1. Prediction Model: We will train a machine learning model—building on the published and validated M6 prediction model— working closely with UCSF data scientists for model validation. We will use data from the UCSF EMR following the procedures described in the publications referenced.  We will identify cases using the ICD 10 diagnosis code for PUL and extract data on model inputs including BhCG, ultrasound results, and clinical characteristics (risk factors, symptoms, sociodemographics) for our population to assess the ability of the model to predict ectopic pregnancy, viable intrauterine pregnancy, miscarriage, or persistent PUL. Initially, we expect to replicate the model using only the initial visit and the first follow-up, however, going forward we hope to improve the predictive model utilizing information from all the follow-up visits so that it can inform decisions (such as when to return for the next visit/lab test/US) at each visit as described in step 2 below.
2. Decision Support + Triage Algorithm: We will translate the ZSFG PUL Guidelines (2024 update) into a structured AI-driven triage algorithm, which will be updated with the prediction model results. The tool will generate next-step clinical recommendations based on predicted outcome and patient-specific factors. Example outputs: “Repeat hCG in 48 hours”, “Schedule transvaginal ultrasound 7 days”, “Consider methotrexate”, “Continue expectant management with safety counseling”
3. Patient-Facing Communication Assistant: A chatbot within MyChart will deliver timely, consistent, and language-accessible communication to patients, reducing follow-up delays and clinical workload:

• Confirm follow-up appointments and lab tests
• Notify patients of test results and recommended next steps
• Deliver safety information and education (e.g., signs of ectopic pregnancy)
• Provide reminders and check-ins

This end-to-end system will augment the current time intensive manual “floater list” process, creating a centralized, streamlined, consistent, and patient-centered workflow for PUL management at UCSF.

Section 3. How would an end-user find and use it?

The TRACE AI tool would be most useful when the gyn team is called to consult on a patient with a positive pregnancy test and early pregnancy symptoms (e.g., bleeding, pain), either in the emergency department or outpatient settings (a frequent occurrence). When the consulting gyn clinician initiates hCG testing or flags a case as suspected Pregnancy of Unknown Location (PUL), the AI tool would automatically populate a PUL Management Panel in the APeX EHR.

As shown in Figure 1 (left panel), the tool would provide:

• A predicted outcome classification (ectopic, intrauterine pregnancy, miscarriage, or persistent PUL)
• A risk level (e.g., 43% ectopic), generated by the model
• A visual timeline of serial hCG and US results
• Management recommendations based on embedded clinical guidelines (e.g., "Repeat hCG in 48 hrs," "Order transvaginal ultrasound")
• Action buttons to pend orders or insert a templated clinical note

The patient-facing chatbot interface (see Figure 2, right panel) would operate in parallel via MyChart. Patients would receive:

• Timely updates about lab results and next steps
• Appointment scheduling assistance
• Safety warnings (e.g., symptoms of ectopic rupture)
• Follow-up reminders and options for clarification

Both interfaces are designed to reduce cognitive burden on providers, support standardized care, and ensure that patients-particularly those at high risk-receive clear, timely, and actionable information.

Section 4. Embed a picture of what the AI tool might look like. 

Section 5. What are the risks of AI errors? 

• False negatives, such as classifying a high-risk ectopic pregnancy as low-risk, could result in delayed or missed diagnosis and potentially serious patient harm.
• False positives, such as overclassifying a low-risk case, could lead to unnecessary labs, visits, or patient anxiety.
• Equity risks are possible if the model underperforms for subpopulations (e.g., patients with limited English proficiency, Medicaid insurance, or historically marginalized groups) due to biased training data.

These risks will be mitigated through the following approaches:

• Begin with validated models (e.g., M6) and retrain using UCSF historical data
• Evaluate and report model performance stratified by race/ethnicity, language, and insurance status
• Keep AI recommendations advisory, allowing clinician discretion
• Include transparent outputs and user-facing explanation of AI logic (e.g., “based on hormone trends and symptoms, this is considered high risk”)
• Build an audit dashboard to track tool usage, override frequency, and outcome correlation

Section 6. How will we measure success? 

We will draw on our expertise as implementation researchers to evaluate the effectiveness of this AI tool and will seek additional funding to do so. Prior to introduction we will survey residents and conduct retrospective chart review examine measures described below before and after introduction. We will estimate time demands for both clinicians and patients prior to and after introduction. We will assess whether end-users (clinicians and patients) are using the system as intended, whether it leads to improved decision-making and workflow efficiency, and whether it improves timely and accurate diagnosis of PUL. We will also examine whether the tool supports equity, reduces unnecessary follow-ups, and minimizes risk of delayed ectopic diagnosis.

We will monitor adoption, usage patterns, and clinical outcomes using a combination of APeX-derived metrics and ideal supplemental measures at ZSFG. These metrics will help determine whether to continue expanding the tool within UCSF or to modify or abandon implementation.

A. Measurements using data that is already being collected in APeX:

• Time from first positive pregnancy test to definitive PUL diagnosis
• Number of hCG draws, pelvic ultrasounds, and follow-up visits per PUL patient
• Proportion of ectopic pregnancies diagnosed before rupture
• Proportion of PUL patients receiving timely follow-up
• Frequency of AI tool usage by eligible clinicians (e.g., OB/Gyn residents, Gyn attendings, ED providers)
• Frequency with which AI-generated recommendations are followed vs. overridden
• Number and type of orders pended via the AI interface
• MyChart chatbot message delivery and open rates

B. Additional measurements ideally needed to evaluate success of the AI:

• Clinician-reported satisfaction, time savings, trust, and perceived utility (via surveys or focus groups)
• Patient understanding of follow-up instructions (e.g., short MyChart-based surveys)
• Time spent by interns managing the floater list before and after implementation
• Disparities in model performance and outcomes stratified by insurance status, language, and race/ethnicity
• Rate of errors or safety concerns identified through audit (e.g., missed follow-up for high-risk cases)
• Longitudinal reduction in unnecessary follow-up testing for low-risk cases

Success would be indicated by high tool adoption, improved timeliness and consistency of PUL management, reduced follow-up burden for clinicians and patients, cost-effectiveness, and equitable model performance across subgroups. Low adoption, significant workflow disruption, or evidence of biased or unsafe predictions would be grounds for modification or discontinuation. 

Qualifications and commitment.

This project will be led by Dilys Walker, MD, FACOG.  Dr Walker is a practicing obstetrician gynecologist at ZSFG and a member of the leadership team at the Bixby Center for Global Reproductive Health. Her expertise is in implementation research across the life course. Dr. Walker has decades of experience managing PUL in various settings. She was awarded a Gates Grand Challenges and J and J grant to build and test a Virtual Mentor chatbot for postpartum hemorrhage. Her team has attended 3 works-in-progress sessions with the AER team.

References

1.         Barnhart KT. Ectopic Pregnancy. N Engl J Med. 2009;361(4):379-387. doi:10.1056/NEJMcp0810384

2.         ACOG Practice Bulletin No. 193: Tubal Ectopic Pregnancy - PubMed. Accessed April 1, 2025. https://pubmed.ncbi.nlm.nih.gov/29470343/

3.         Marion LL, Meeks GR. Ectopic pregnancy: History, incidence, epidemiology, and risk factors. Clin Obstet Gynecol. 2012;55(2):376-386. doi:10.1097/GRF.0b013e3182516d7b

4.         Kirk E, Papageorghiou AT, Condous G, Tan L, Bora S, Bourne T. The diagnostic effectiveness of an initial transvaginal scan in detecting ectopic pregnancy. Hum Reprod Oxf Engl. 2007;22(11):2824-2828. doi:10.1093/humrep/dem283

5.         Condous G, Kirk E, Lu C, et al. Diagnostic accuracy of varying discriminatory zones for the prediction of ectopic pregnancy in women with a pregnancy of unknown location. Ultrasound Obstet Gynecol Off J Int Soc Ultrasound Obstet Gynecol. 2005;26(7):770-775. doi:10.1002/uog.2636

6.         Bobdiwala S, Christodoulou E, Farren J, et al. Triaging women with pregnancy of unknown location using two-step protocol including M6 model: clinical implementation study. Ultrasound Obstet Gynecol Off J Int Soc Ultrasound Obstet Gynecol. 2020;55(1):105-114. doi:10.1002/uog.20420

7.         Maheut C, Panjo H, Capmas P. Diagnostic accuracy validation study of the M6 model without initial serum progesterone (M6NP) in triage of pregnancy of unknown location. Eur J Obstet Gynecol Reprod Biol. 2024;296:360-365. doi:10.1016/j.ejogrb.2024.03.010

8.         Kyriacou C, Ledger A, Bobdiwala S, et al. Updating M6 pregnancy of unknown location risk-prediction model including evaluation of clinical factors. Ultrasound Obstet Gynecol Off J Int Soc Ultrasound Obstet Gynecol. 2024;63(3):408-418. doi:10.1002/uog.27515

9.         Christodoulou E, Bobdiwala S, Kyriacou C, et al. External validation of models to predict the outcome of pregnancies of unknown location: a multicentre cohort study. Bjog. 2021;128(3):552-562. doi:10.1111/1471-0528.16497

10.       Hou L, Liang X, Zeng L, Wang Q, Chen Z. Conventional and modern markers of pregnancy of unknown location: Update and narrative review. Int J Gynaecol Obstet Off Organ Int Fed Gynaecol Obstet. 2024;167(3):957-967. doi:10.1002/ijgo.15807

11.       Rueangket P, Rittiluechai K, Prayote A. Predictive analytical model for ectopic pregnancy diagnosis: Statistics vs. machine learning. Front Med. 2022;9:976829. doi:10.3389/fmed.2022.976829

12.       Jurman L, Brisker K, Ruach Hasdai R, et al. Enhancing decision-making in tubal ectopic pregnancy using a machine learning approach to expectant management: a clinical article. BMC Pregnancy Childbirth. 2024;24:825. doi:10.1186/s12884-024-07035-4

13.       Training and testing performance of ectopic pregnancy prediction model... ResearchGate. Accessed April 1, 2025. https://www.researchgate.net/figure/Training-and-testing-performance-of-...

14.       An automated ectopic pregnancy prediction system using ultrasound images with the aid of a deep learning technique. ResearchGate. Published online December 16, 2024. doi:10.1007/s00500-024-10333-w

Summary of Open Improvement Edits

The 17 comments provided on our proposal raised valuable points that have been used to strengthen the proposal.  The majority of the comments came from clinical faculty and residents who manage the problem of PUL on a daily basis and believe this tool could be transformational on multiple levels.  We have tried to strengthen section 2 on How might AI help? in response to these helpful comments.

Summary of comments:

Comments supportive of the TRACE tool:

• Quality of care- The tool will improve the quality of patient care for this high-stakes condition. Though guidelines exist and have been validated in various populations, they are not always followed at ZSFG (or around the country for that matter) as one reader stated “I have found as a resident that these guidelines are not always followed, often due to providers simply not having enough time to refer to them every time, and instead going with their own personal preference when it comes to PUL management, and that leads to a lot of variation in practice and sometimes near misses. because providers simply don’t have the time to carefully map the specifics through the guideline (see attached) and instead rely on
• Administrative Burden- residents spend up to 3 hours daily following the floater list of PUL patients.
• Costs- The majority of patients with PUL are covered by Medicaid.  By streamlining follow up, costs will be saved while maintaining or improving quality.
• Patient-centered- Many patients are required to be followed from days to weeks with serial blood draws and ultrasounds to determine the location and viability of their pregnancy. This is disruptive and anxiety provoking with the potential for errors and missed ectopic pregnancies.  By creating a patient facing platform, communication can be automated and consolidated. Additionally, one reader commented on the value of being able to create language accessible messaging to our multi-lingual population served.  Often requires 10-15 min per call to secure interpreter and make certain message is understood

 In summary

On reader wrote, “AI can handle complex algorithmic decision-making that’s difficult for (tired) human brains to do quickly and it can complete important administrative tasks better than physicians (tracking, patient communication and reminders) making it a win for patients and physicians alike. If successful, TRACE would be a highly sought-after tool across the United States.” 

Comments to strengthen the proposal:

• One reader commented, how much of this could you do without AI? 

The fact is, all of it is currently done without AI which is suboptimal, time intensive, and risky.  We have added clarification to the description of the problem in section 1 to better justify the benefits of the TRACE tool. One of the readers, who is an ObGyn fellow, said it best- “PUL is a challenging clinical diagnosis because it carries significant uncertainty, is managed differently by different clinicians despite robust clinical guidelines on best practices, and is highly time and resource intensive for the trainees at our institution who do the patient-facing work of helping patients navigate the process of determining pregnancy location. This proposal has the potential to revolutionize care for patients facing an uncertain prognosis which can be confusing, scary and very burdensome, and also to revolutionize clinicians' work of managing PULs” 

• In response to the comment- It sounds as if developing or refining the initial prediction model is key -- driving the two other pieces. Can you expand a bit on how you might work with the validated tools you've mentioned? Has any preliminary work been done with UCSF data?

This is correct, validating and potentially modifying the M6 prediction model with UCSF data would be the first step. We have added additional clarification and references in the description of risk prediction models and the pathway we describe for development. Specifically, we will use the modified M6 model from the UK/Belgium and validate its performance with UCSF data.  The M6 model has been published and validated externally. We would need to work closely with UCSF data scientists for model validation using data points available from the UCSF EMR following the procedures described in the publications listed below. We will identify cases using the ICD 10 diagnosis code for PUL and extract data on BhCG, ultrasound results, and clinical characteristics (risk factors, symptoms) for our population. Initially, we expect to replicate the model using the initial visit and the first follow-up, however, going forward we hope to improve the predictive model utilizing information from all the follow-up visits so that it can inform decisions (such as when to return for the next visit/lab test/US) at each visit.

 Kyriacou C, Ledger A, Bobdiwala S, Ayim F, Kirk E, Abughazza O, Guha S, Vathanan V, Gould D, Timmerman D, Van Calster B, Bourne T; Collaborators. Updating M6 pregnancy of unknown location risk-prediction model including evaluation of clinical factors. Ultrasound Obstet Gynecol. 2024 Mar;63(3):408-418. doi: 10.1002/uog.27515. PMID: 37842861.

 Maheut C, Panjo H, Capmas P. Diagnostic accuracy validation study of the M6 model without initial serum progesterone (M6NP) in triage of pregnancy of unknown location. Eur J Obstet Gynecol Reprod Biol. 2024 May;296:360-365. doi: 10.1016/j.ejogrb.2024.03.010. Epub 2024 Mar 8. PMID: 38552504.