Chief Health AI Officer  |  Executive VP of Physician Services  |  Chief Medical Information Officer
Chief Nursing Informatics Officer  |  Chief Innovation Officer  |  AVC Clinical Research  |  CTSI
Chief Research Informatics Officer  |  BCHSI  |  Division of Clinical Informatics & Digital Transformation
Chief Information Officer  |  Chief Data Officer

Overview - AI Pilots 2025  

A Learning Healthcare System, as envisioned by the Institute of Medicine and the NIH Collaboratory, aims to bring new knowledge quickly into the healthcare delivery system, including advances in data science, machine learning, and artificial intelligence.  

This is the second call for UCSF’s Artificial Intelligence/Machine Learning Demonstration Projects - “AI Pilots” - aimed at advancing UCSF as a Learning Healthcare System. This AI Pilots call is sponsored by UCSF and UCSF Health Leadership, including the Chief Health AI Officer, Executive Vice President of Physician Services, Chief Medical Information Officer, Chief Nursing Informatics Officer, Chief Innovation Officer, Associate Vice Chancellor of Clinical Research, Clinical and Translational Science Institute (CTSI), Chief Research Informatics Officer, Bakar Computational Health Sciences Institute (BCHSI), Division of Clinical Informatics & Digital Transformation, Chief Information Officer, and Chief Data Officer.

Funded project teams will receive investigator salary support for 1 year (up to 10% effort for project lead(s) starting July 1, 2025), with an option to extend for a second year, depending upon project progress.   

Goals of the AI Pilot 

1. Identify an exciting and realistic opportunity for AI or machine learning tools to drive significant improvement in the health system, with a focus on solutions that may scale to drive value in multiple care settings.  These may be clinical decision support tools or tools used to improve care delivery in other capacities. 
2. Work with the UCSF Health AI team to develop and validate an AI tool using data extracted from the UCSF EHR (or elsewhere, as appropriate) 
3. Design how the AI tool will integrate into clinical workflows and clinical decision-making, using evidence-based approaches and in collaboration with clinical champions, and work with technical teams to build an interface that allows use of the tool in the EHR, if appropriate 
4. Plan an intervention study to evaluate effectiveness of the AI tool – including its implementation and impact on outcomes 
5. Work with the appropriate governance committees (e.g. UCSF Health AI Oversight Committee) to ensure safe and effective use 

To accomplish these goals, funded project teams will participate in twice-monthly work-in-progress seminars. The technical work of developing the AI tool, designing the integration approach, and implementing it in the health system for the prospective validation will be completed by the UCSF Health AI Team, in consultation with the project lead, who will provide subject-matter expertise and deep knowledge of the clinical problem and the relevant workflows.

Therefore, technical AI expertise is not required to be competitive in the application process.

The project lead will be responsible for working with the APeX-Enabled Research Program team and the Health AI Team to design and build the user-interface with engagement from proposed users and DoC-IT researchers with experience in health AI integration. As needed during the development phase of the AI algorithm and the APeX interface, the project lead will facilitate discovery interviews with potential users of the tool.

The project lead will also plan an intervention study to evaluate the effectiveness of the implementation and impact of the AI Decision Support Tool with support from the CTSI Informatics Program.  By the end of the first year of the project:

• the prospective validation study should be launched
• the APeX interface built and pilot-tested
• the intervention study planned with a written protocol 

Upon successful prospective validation of the tool and obtaining relevant approvals, the project lead will assist with developing and delivering clinician training materials as needed for the pilot in collaboration with clinical stakeholders. 

Salary support will be provided for 1 year.  For projects progressing as expected and meeting milestones, there may be an option to extend salary support for an additional year, with the goal of launching an implementation study and developing evidence of real-world utility.  

Office Hours in Support of Proposal Creation

To aid in the development and improvement of proposals, "office hours" will be offered for open discussions around possible projects and submissions. Attendees can brainstorm ideas for submissions or work on refining ideas and addressing proposal feedback. Office hours will include a data scientist and a data engineer from the Health AI team to help scope and refine project goals. They will also include a member of the APeX Enabled Research (AER) team to discuss the feasibility of projects. People who have identified problems in the Health System are encouraged to join an office hour during the Open Submission period to discuss how AI tools may be used to alleviate problems for the Health System, so they can craft competitive proposals. People are also encouraged to join an office hour during the Open Improvement period to discuss modifications that can be done to address feedback on and strengthen their proposals. 

Review Criteria: 

Proposals will be evaluated based on the following criteria: 

1. Project addresses a high-priority area with opportunity for improvement at UCSF Health and large potential impact from use of AI.
   
2. Project lead has expertise (clinical or operational) in the problem of interest that will help in designing workflows that leverage the AI tool. Research experience is a plus.
3. Availability of rich source(s) of accessible data that might support development of a useful AI algorithm. 
4. A clear evaluation framework, with a set of health outcome and process metrics identified to measure real-world uptake, meaningful impact(s), safety, and equity/fairness.  
5. Engagement and support from critical operational and clinical champions. 

Open Proposals Format 

A.  Paste your 3-page proposal into the submission form in the Project Description section.  The full proposal, including all sections described above, may be no more than 3 pages, with 1-inch margins and Arial 11-point font.   Additional pages of References may also be included, if desired

Please strictly follow this format using the following section headings: 

1. The UCSF Health problem.  What problem are you trying to solve?  Why is it important?  What has been tried previously (predictive modeling or otherwise)? Who are the intended end-users? 
2. How might AI help?  Artificial intelligence can help summarize and analyze data or generate content.  What data would your AI use?  What would it produce?   How might AI help solve the problem? 
3. How would an end-user find and use it? To be useful, AI support must be actionable, easily discovered, and user-friendly.  Describe when the AI support might be most useful (at what point in the usual health system workflow), what end-users might see, how the predictions/recommendations are explained to the end-user, what end-users are supposed to do based on the AI output, and how they might interact with it (if at all).   
4. Embed a picture of what the AI tool might look like.  Sketch what an end-user might see (hand draw, take a picture, upload as jpeg, and embed in the file).  While we won’t be doing any major software development, for this purpose you can assume that any sort of illustration of results of an AI algorithm might be possible to produce (don’t worry about APeX EHR limitations), and that users could interact with the tool via buttons, checkboxes, or any other interfaces (e.g., to enter data, pend an order, etc). Help us understand how the tool will fit into the intended users’ workflow and how it might add value. This may be embedded in the middle of the document or at the end.  Refer to it in Section 3 where you describe what end-users might see. 
5. What are the risks of AI errors?  Different types of errors (such as a false positive or false negative prediction, or a “hallucination” from a generative AI model) have different consequences. What are the different types of risks relevant to the proposed solution and how might we measure whether they are occurring and mitigate them? 
6. How will we measure success?  What can we measure that will tell us whether intended end-users are using the tool, changing what they do, and improving health for patients or otherwise solving the problem you described above?  What evidence would you need to convince UCSF Health leadership to continue supporting the AI in APeX?  What results would make you consider abandoning the implementation? Please include 2 subsections: 
   • A list of measurements using data that is already being collected in APeX 
   • A list of other measurements you might ideally have to evaluate success of the AI 
7. Describe your qualifications and commitment: You must be a UCSF faculty member with a clinical or operational role in the health system that is relevant to your clinical problem or co-lead with a UCSF faculty member.  As project lead, describe your qualifications and your commitment to dedicate effort to this project during the coming year, including participating in regular work-in-progress sessions and collaborating with the Health AI and AER teams for development and implementation of the AI algorithm.  

The full proposal, including all sections described above, may be no more than 3 pages, with 1-inch margins and Arial 11-point font.  Additional pages of References may also be included, if desired.

B. Upload a short Letter of Support from a clinical or operational leader who can attest to the importance of the problem you are trying to address, and their support for trying an AI approach to the problem. 

C. NOTE: there is a section on the submission form to upload a budget. This is not a requirement for submission. 

Timeline