IT Innovation Contest

A team-based contest for creative IT solutions

A mobile image-processing based application for the identification of pills in the Emergency Department

Proposal Status: 

Description of Project: It is estimated that there are more than 30,000 deaths related to accidental poisonings each year. While there are databases of prescription drugs, there have been no robust efforts to automate the identification of medications in clinical practice. We seek to implement and test a mobile application that uses image processing to identify medications.



Deliverables: We will evaluate a mobile application and corresponding platform for the identification of medications in the SFGH ED.


  1. Mobile Application With
    1. Camera functionality to send images to a remote server
    2. A database of common FDA approved medications
    3. The ability to create local drug lists that can be used by individual patients to set reminders and find more information about their medications
  2. A Stand with lighting to optimize image acquisition for precise identification
  3. A Server that receives images and returns a ranked list of pills most likely to match based on shape, color, and character imprint (


Impact on UCSF’s mission and/or community: We have estimated that there are approximately 75 events of unidentifiable medications or medications that are brought to the SFGH ED each month. Such image processing can be used in the hospital, at local pharmacies, and by patients themselves to identify and look up information on medications. We believe there is significant opportunity for commercial integration and acquisition.


List of team members and their roles: 
David Ouyang, MS3 – Project Manager (200 hours) 
Aron Yu – Electrical Engineering PhD student – Image Processing Developer (150 hours) 
Dennis Qian – Mobile Developer (50 hours) 
Janet Chu, MS2 – Designer and IRB Approval Facilitator (50 hours) 
Rahul Deo, MD, PhD  (Assistant Professor of Cardiology) – Statistical Analysis Expert (20hr)
Derrick Lung, MD, MPH (Assistant Professor of Emergency Medicine) – ER UI Expert (20hr)


This project would certainly fill a need for the ED and can be used by the public at large as well. Nice concept!

Very intriguing project. Naively I assumed that this would have already been done given that the simple shape, color, pattern recognition needs for the app have been solved long time ago. I really liked the YouTube video submission. Appears that you have a working prototype. A quick Google search led to some interesting findings. It appears that most apps out there are simply asking for two of three pieces of information, shape, color or imprint on the pill. Here's a head-to-head comparison of the top ones: I didn't easily find an image based app. However I found a 2010 RFP for developing an image context based pill identification system by the FOB: Some of the vendors on the RFP list may have solved this problem? There is also this NIH project: Which seems to be relevant to what you are proposing. Questions: 1. why limit to only shapes, color? how about attempting an OCR on the pill for the imprint? 2. how about using image content based matching against a database of pills like the one from nih? 3. Do you handle partial or broken pills as well as multiple pills at the same time or is it only for one pill at a time? Assuming this has not yet been done, I hope you guys continue development as it would be an app that someone like might just buy from you, as they have not developed it yet.

Apologies for the miscommunication. We are actively working on the OCR now and have been making good progress, that is why we think it would be reasonable to have a working product by October 1st. The demo video is from Feb, which was earlier in the the development and we have more to show.

In our search, we have not found any functional systems. The research projects have very controlled environments and are very dependent on lighting, these are things we are actively working on and feel like have a handle on. As you have shown, there is no image based system for pill identification on the market. A careful search from any of the solicitors of the FBO do not turn up any viable systems.

The system is currently focused on one pill at a time now, but there is no reason why it cannot scale to more pills - simply a matter of resources and priorities. We currently do not handle broken or partial pills.

For this project, we specifically want to see our app used in the EDs which can validate our approach and be useful for emergency room physicians.

here's a content based system reference that includes imprint recognition and attempts to go beyond prescription drugs with about 80% accuracy on a 15K+ drug sample. The techniques described there as well as database sources might be applicable to your project:

That is a great article Vivek, thanks! We have already been using the databases for FDA approved pills, although we have not tried illicit drugs. That is very interesting and something we should try in the future. We have been using similar approaches, and it will be good to read this thoroughly and see if we missed anything or if they have other insights.

Commenting is closed.