2013 CTSI Annual Pilot Awards to Improve the Conduct of Research

To facilitate the development, conduct or analysis of clinical & translational research

Application Management System

Proposal Status: 

Application Management System

Rationale: There are several programs on campus that manage the dissemination, submission, review, and selection of applications for student positions, grant awardees, or other candidate selectees. The Research Allocation Program (RAP) is one such program where this process is perhaps the most complex due to the number of intramural funding mechanisms, the number of review committees, and the nature of the selection process. Currently, RAP applications are submitted online but the review and decision processes are done manually, thus limiting continued efficiency improvement and lowering the ease of program reporting. To remedy this situation for RAP and other related program processes on campus, we would like to develop a single electronic solution that will accommodate the processes for dissemination, submission, review, selection, and reporting. The solution may leverage parts of existing software such as NUCATS Assist (Northwestern University) or CTSI-ART, or be built completely customized with Drupal interfaces.

Plan: The overall goal is to develop a centralized application management system that would coordinate opportunity dissemination, application submission, application review, application selection, and follow-up processes for multiple programs across multiple agencies. Configurable application components and flexible workflows would accommodate changing business requirements and support process improvements. Such a process solution would result in significant longer terms savings for all collaborating programs.

Criteria and metrics for success: An automated application management system can decrease the amount of time and resources administrative staff spends on application processes and follow-ups, as well as on compiling and summarizing application information for program reporting. Designated administrators would have direct access into the system to customize forms and run reports. Communication with reviewers and applicants can improve through automated deadline reminders and standardized decision notices. Broader and deeper analysis of application and review processes would be possible through reporting measures that have not been available before. The system will be a central repository for applications and progress reports and will offer greater ability to measure programs and funding opportunity impact.

Advantages for all users

Applicants can find funding information, apply via a website that is open to the public, and can upload a single proposal PDF. The application can be saved until submission allowing to be built in stages. Potentially applicants could also use the system to enter and submit progress reports online.

Reviewers can find all materials in one place (proposals, review forms, previous reviews). They can receive automated notifications and reminders to complete tasks before deadlines. They can write their reviews in stages and assign initial scores. During the review meeting, they can score proposals “live” as they are discussed.

Administrators can replace uploaded PDF files, delete applications, and submit on applicant’s behalf. They will take advantage of automated messages/notifications/reminders to communicate with applicants and awardees. They will benefit from a system that tracks many variables, including applicant attributes (e.g., name, department, gender, award status, etc.), and reviewers’ attendance and productivity. Summarized financial award information will be available in the system. Administrators can use the system to manage progress reports starting with automated requests for project/progress report updates sent through the system at intervals designated by each funding agency.

Potential Collaborators Development team - Depending upon whether a system is developed from scratch or leverages an existing one, the system development responsibility will lay with Edwin Martin and ISU or our colleagues at CTSI Virtual Home or the SF Coordinating Center or potentially an outside contractor. Requirement owner - RDO [Emy Volpe (RAP), Gail Fisher (LSP)], Suya Colorado-Caldwell (EVC&P Committees), and many other campus programs that utilize application review and selection processes.

Budget: This project can be developed modularly. With the $50,000, we could plan the development of all modules and develop several of the modules. Our intention would be that matching funds be identified that would enable to the full development of the modular plan. Leveraging another existing system may enable the full system build for $50,000.


Hi Emy,


As you know, I'm the product manager for CTSI's Application Review and Tracking (ART) system, a combination of custom sfotware and an off-the-shelf form builder that does some of what you reference in your proposal. 

Our expereince with building ART was that it cost significantly more than $50K to get where we are today. And that didn't include some of the major features in your proposal, like flexible work flows. Nor did it include the off-the-shelf form builder ART uses. In developing your proposal, did you get input from software developers to arrive at the estimate?

Also taking from our learning with ART, would it be possible to enhance RAP, as an alternative to an entirely new system? We initially built the application piece, then added online review and tracking. It seems like there are good things about RAP. Maybe adding or enhancing some components would be a more practical way to meet the needs of your programs.

Lastly, I wonder if one application management solution truly can satisfy all programs. Some of the programs that use ART do so because of the "single proposal PDF" (which means the same application form for all programs, right?) requirement in RAP. Some of our programs require flexible forms with branched logic based on the choices about sub-programs the appliant chooses.

I'm happy to explain or provide more detail to any of my questions as needed. Thanks for putting in your idea!




Hi there -


Brian, as I read this, it looks to me like the proposal is indeed an extension or upgrade to the current Application, Review & Tracking (ART) system that we are currently managing, rather than a start from scratch kind of thing.   Emy -- is that correct?


We've definitely had some discussion about this and as some may recall, when we initially scoped the work for ART, we deliberately excluded RAP mainly because of the complexity of the review process.


If it's the right time to re-visit this and there is time, resources and funding, we'd love to discuss. I really appreciate the vote of confidence that the CTSI Virtual Home team has the expertise to develop such a system, but need to think a bit as to whether we have the "resources" at this time ;)  

Look forward to others' thoughts and to discussions on feasibility and resourcing.

Oh yes, finally, other potential collaborators in addition to the Virtual Home team at CTSI, would be folks from the SF Coordinating Center. These guys were brains and brawn behind the ART system :)




Hi Leslie and Brian,

Thanks for your feedback. I will address both your comments at once.

We envision this more like an expansion or enhancement of ART.  We are aware the cost will likely be more than $50k but were thinking that we might be able to design a phased or modular approach. For example, we currently have an electronic submission system (built by a contractor) in JAVA with an associate SQL database. We could maintain this for a time and have it "dump" into something new - a phase I Review Committees management system that woud respect the move of our website to a Drupal structure.

We can imagine that the plannning will be part of this grant upfront and the planning could include how to operationalize a phased approch.

With respect to the universal applicability of a RAP management solution, while it's true that a RAP mgt. system would not directly apply to a fellowship application process, a system built to deal with the complexity of RAP could be more easily modified and used by other programs.

Thanks much,


Emy Volpe


You present a compelling rationale for developing a solution to support the RAP process.

As you know the Catalyst Program uses the RAP mechanism to solicit and accept applications, but performs all reviews and tracking independently. We have been working with Brian Turner over the last year to implement the management and tracking of submitted projects through ART. In its current form, ART is certainly more efficient than our former 'manual' approach, but there are some critical challenges as well. Our assessment regarding modifying ART as opposed to identifying a new solution altogether is ongoing, and we'd be happy to share our experiences with you.



Thanks Ruben,


Good to know you are also working with the ART team to find ways to improve it so that could be used by other programs.

It's good to have a common agenda! We'll be in touch.



Something like this would be great for the Limited Submission Program (LSP), as well!  The LSP oversees distinct review committees for every limited submission opportunity (LSO) and with close to 100 LSOs annually, reducing the manual management of particular aspects of application submission and review would be great.  

After talking to Gretchen, there might be other options out there to investigate - whether for ideas on features or software itself.  Late last year, there was a presentation via the CTSA Consortium from Northwestern on their system called NUCATS Assist.  I wasn't able to attend this presentation, I think some other folks at UCSF did though.  In any event, the info on this system along with a link to a video recording of the presentation itself is online here:  https://ctsacentral.org/tools/tool_shop#NUCATS 


Perhaps it will be of help/use.

Thanks Leslie!


I will check that out. We are scheduled to receive a NUCATS demo by Dr. Warren Kibbe at the end of the month and Nina Jameson and Ed Martin (from ITS) will be assisting us in determining if that system is a good fit for us.



This is a timely proposal that strengthens the foundation of the research enterprise and mirrors the UCSF Committees Service and Management Portal (CSMP) proposal submitted for the 2012 IT Innovation Contest showing that it could be adapted beyond the research sphere and be of service to the University enterprise-wide.  For reference the CSPM proposal description follows:      

Project Description: This proposal outlines a project for developing, deploying, and supporting a UCSF Committee Service and Management Portal built on the UCSF Salesforce platform and leveraging Chatter capabilities.

Faculty and staff provide support to UCSF’s academic mission by serving on a number of research, clinical, and administrative committees, and their active participation is critical to achieving the institution’s goal of advancing health worldwide.  Currently, there is no single, quick way to determine the number, composition, context, collaborative relationships, and charge of the many committees currently convened at UCSF or what actual impact they have on the success of the enterprise. For example, if one wants to identify existing resources or forums to help move along a specific goal or facilitate a specific decision, there is no existing repository that helps identify that.  UCSF needs a tool that provides quick and easy access to enterprise-wide committee information to senior leadership, committee chairs and members and staff.  The following are key components that such a tool must address:

  • Have committees completed their charge and if so, they should be disbanded?
  • If committees are currently active are they still relevant?
  • Do standing committees make sense or is there efficiency and effectiveness in convening short-lived, task/decision-oriented task forces and/or work groups?  
  • How can a committee’s or committee member’s completion of service and/or work product be appropriately and efficiently acknowledged?
    • How can committee staff work be coordinated more centrally and efficiently?
    • How can ongoing, centralized collaboration be promoted and facilitated?

 The UCSF Salesforce platform offers key features to facilitate committee management, such as standardized contact records for all UCSF staff and faculty (fed by EDS), single sign on through MyAccess, Chatter group and activity streams, support for file versioning, and the ability to leverage evolving Salesforce functionality (less need for costly customization).  Building a committee management tool on the UCSF Salesforce platform will allow users to access all relevant data related to committee management in order to efficiently support the organization, maintenance, and decision making processes for Committee Chairs and Managers (staff support) as well as key Executives needing insight about committee efficacy and outcomes at UCSF. 


This project will deliver a web based application using the Salesforce platform that will:

  1. Track relevance of committee charge, purpose, and oversight
  2. View, organize, update, and track tasks and data related to committee management (e.g. agendas, notes, attendance, completed action items, etc.)  
    1. Identify who is currently serving on what committee(s) and in what capacity
    2. Determine how many committees individual members are serving on and from what departments members are being pulled
      1. Track terms of service and manage committee member rotation
      2. Record and report on “credit” for committee service
      3. Manage and archive committee communication
      4. Produce committee analytics
      5. Communicate data with Advance and the Academic Senate’s Service Portal

 Impact and Rationale: 

A centralized repository and service/management portal will have universal impact and:

  1. Minimize administrative burden on all members of our community
  2. Provide up to date information and documentation Allow committee staff to centrally facilitate and manage administrative functions simultaneously for multiple committees
  3. Assist leadership in evaluating existing committee model as well as individual committees and make data informed decisions regarding the need for further investment and/or action
  4. Classify and convene appropriate groups depending on input needed, decision to be reached, or work to be accomplished
    1. Eliminate duplicate committees
    2. Reduce in-person meetings
    3. Create a foundation for collaboration across departments

 Measures of Success:

 The following will measure the success of the project:

  1. Stakeholder support and use of the tool
  2. Enterprise-wide acceptance and use of the tool (estimated at 3000+ users)
  3. Reduction in the use of listservs associated with committees
  4. Ability to report on membership and activities across committees 

Terrific comments, thank you Suya.  Not to mention that RAP only deals with 10 standing review committees whose membership is updated annually.  It would be tremendous to have a tool that does all of that!

Emy Volpe

Emy and Suya have laid out a compelling case for a committee/review management system that can be used across a wide variety of functions, be it campus-wide committee management, evaluating students, the RAP program, scientific review committees in departments for human subjects or animal research, and any other group that needs to manage an evaluation process. 

Commenting is closed.