2013 IT Innovation Contest

A team-based contest for creative IT solutions

E*Value Efficiency Effort for Faculty Advancement Packets

Proposal Status: 
  1. Description of project: Develop a new process to save thousands of hours of work each year by automating the process of pulling required faculty evaluation data and delivering to a secure HR drive. Currently, each faculty member or their representative must manually extract this data from E*value.This will allow HR generalists to efficiently upload the required evaluations into Advance to support faculty appointments and advancement.  This is a pilot for the School of Medicine departments, with the future goal to invite other Schools to experience a reduction in their administrative burden as well. 
  2. Deliverables:
    1. Identified and vetted E*Value standardized report that provides the required evaluation data on an annual basis for eligible faculty.  The vetting would include the Committee on Academic Personnel as well as the Vice and Associate Deans.
      1. Using a unique identifier (ie employee id)
      2. Pdf files would be used, 1 per faculty member for all teaching in the School of Medicine that evaluates its program through E*Value. This includes required UME and GME teaching.
    2. Secure location on an HR drive
    3. Communication to affected campus community, disclosure to faculty members
  3. Impact on UCSF's mission and/or community:
    1. At least 3,000 less hours of administrative work per year for faculty, departments, HR, SOM
  • 2012-13 had 2,973 appointment and advancement packets created across all departments at UCSF. In 2011-12, there were 3,054.
  • It takes approximately one hour compile and pull a faculty member’s evaluations for the year.
  • For the approximately 40 Academic HR Generalists, they will have the ability to obtain the information directly from the secure file as required without phone calls, emails, and follow up.
  • UCSF can do a better job meeting the requirements for faculty appointments and advancement.  Evidence of teaching is required, and evaluations form E*Value represent the majority of that evidence.
  • This project will ensure accurate, timely, and complete evaluation data within the advancement packets of all faculty so that the Committee on Academic Personnel (CAP) can trust that all information is included and is consistent.
  • There will be a reduction in the number of individuals handling evaluation documents, which protects the data.
  • List of team members, their roles, estimated time devoted
  • Team Member

    Role

    Estimated Time Devoted

    Luke Hones, HRIS

    Technology Advisor, provide HR IT resources

    5%

    Cynthia Lynch-Leathers, Assistant Vice Provost

    Advance System, Executive Sponsor

    In kind support

    Kevin Souza, Associate Dean Medical Education

    Executive Sponsor

    In kind support

    Bonnie Hellevig, Director, Educational Data

    Technology Advisor, E*Value Programmer

    70%

    MarkLovett, Evaluations Research Analyst

    E*Value Programmer

    10%

    Arianne Teherani, Medical Education Evaluations Unit

    Executive Sponsor

    In kind support

    Allison Chen, Medical Education Evaluations Unit

    Evaluations Specialist

    5%

    Amy Day, Director of GME

    Executive Sponsor

    In kind support

    Susan Sall, HR

    Project Coordinator

    5%

     

    Comments

    I wholeheartedly support this project.  It would lead to HR staff time savings as described above.  In addition, it will also help with the efficiency of processing the merit and promotion actions by ensuring that the teaching evaluations are automatically included with the packet avoiding last minute discoveries that this item is missing.  This project will save HR staff time and improve the quality of the advacement packets!

    Terrific Idea!  This will create a more streamlined process for both Faculty going throughout the advancement process and Staff who administer the process. Information which exists in one system should be leveraged for as many purposes as possible to eliminate redudancy across campus.

    From the persepctive of a faculty member who teaches across programs, units, departments and learner levels, this would be SO helpful! As embarrasing as it is, I will go on record here to note that currently, no matter how much time I devote to trying to do this, I find it impossible to actually collect and sythesize all available information on my teaching evalutions - let alone advocate for myself with the most thorough/complete data. In the end, lots of time spent without complete results. So most definitely a project that would save time and effort AND improve efficiency and quality!

    I am in favor - Elizabeth Garrity

    I manage a program that relies on E*Value scores as part of our application process. It would be a tremendous advance for faculty to have access to a single PDF with a year's worth of evaluative scores, and it would have a positive impact beyond the critical needs outlined in the proposal.

    Hello- I wanted to clarify that this project is related to how HR would get access to the E*Value data, and wouldn't include any changes for how the faculty would access their data.  The time savings will be that the department or individual faculty doesn't have to pull and compile the data to send to HR for the advancement packet. 

    Would it be possible for HR to provide faculty with the pdf summary upon request, or to access it from ADVANCE? It seems that the universal issue is the difficulty of pulling E*Value data; if this project streamlined this, perhaps phase II (operational, not technological) would be to maximize the use of the existing summary.

    Hi Cynthia, The current project plans to deliver the faculty evaluations to HR. Connecting directly to ADVANCE is a logical next step. I know there is a lot of interest in that, and it's something we will likely investigate in the future. 

    This is a great project to save so much time!

     

    I support this innovative proposal because when I worked in HR, we spent a good deal of time trying to gather evals for the faculty for posting in ADVANCE. It was a very cumbersome process.

    Now, as a coordinator, I help to obtain the eval info for HR and agree that it can be done is a much better, more streamlined process.

    The time and effort we save would certainly be put to good use completing other important work.

    Thank you for your very worthwhile, sensible and necessary proposal.

    Joannie O'Leary

    Something that saves 3000 work hours that span faculty and staff alike should be developed.  That's 3000 hours that could be used in pursuit of more productive endeavors.

    I support this project. It would be best if both housestaff evaluations and medical student data could be combined in the report as well as ranking of faculty with in divisions and department.

    I am absolutely in support of the project. As someone who teaches (and is responsible for physiology teaching and, therefore, collection and distribution of the evaluations for all the faculty in my Department) in all professional schools this would be of great help. 

    I think this is a great time-savings initiative and enthusiatically support it. We do have some questions though (perhaps all of these are to be addressed in the vetting process?)... 

    Since faculty members typically teach in several courses, is the plan to have a separate evaluation for each course?  If not, then does this mean the data would be somewhat compromised, since differences between ratings in, for instance, didactic and clinical courses, would be averaged?  

    If the evaluations are to be done course by course, how will the information be collected?  Will reference be made to the CV in Advance?  

    We're suggesting that the optimal system would be one where the teaching activity section in the CV in Advance identifies specific courses for which there are E*Value data.  Perhaps the owner of the CV clicks on information available on pull down menus to identify courses that have evaluations for them.  Some courses in which they teach may not be in E*Value; this would help the faculty member identify those courses in which they are teaching but for which there will be no evaluations; corrective action then may be taken, by having the faculty member communicate with the course director. 

    Would the system allow for reviewers to see COURSE evaluations for courses where the faculty member under review is director?

    These questions may be premature but perhaps a little more detail about the project would assist in its approval.

     

     

    Shirley brings up insightful questions that demonstrate the tremendous downstream potential for this project. Clearly there is much support for streamlining this cumbersome process. Designing a system that would save 3,000 hours of staff and faculty time is a great place to start.

    Great questions, Shirley. Here's the Q & A's together:

     

    Since faculty members typically teach in several courses, is the plan to have a separate evaluation for each course?  If not, then does this mean the data would be somewhat compromised, since differences between ratings in, for instance, didactic and clinical courses, would be averaged?   The evaluations would be pulled by faculty member employee id number and there would be separate evaluations for each course.  This project would not consolidate ratings or do any calculations.

    If the evaluations are to be done course by course, how will the information be collected?  Will reference be made to the CV in Advance?  See answer above.

    We're suggesting that the optimal system would be one where the teaching activity section in the CV in Advance identifies specific courses for which there are E*Value data.  Perhaps the owner of the CV clicks on information available on pull down menus to identify courses that have evaluations for them.  Some courses in which they teach may not be in E*Value; this would help the faculty member identify those courses in which they are teaching but for which there will be no evaluations; corrective action then may be taken, by having the faculty member communicate with the course director.  This is a great idea for future improvement, but the limited time frame and resources mean we’re only be able to address getting the data available to HR in an automated way.  This could be considered for a future enhancement/extension.

    Would the system allow for reviewers to see COURSE evaluations for courses where the faculty member under review is director?  I may need more clarification on this question, but we will pull all evaluations that are appropriate for someone's advancement packet based on the current procedures/processes.

    Thank you Susan for the response.  Our last comment is based on the fact that faculty who direct courses should list that activity as a contribution to their teaching.  The quality of their contribution could be assessed by examining the evaluations of the course as a whole, independent of the faculty member's own lectures, etc.  There is evidently no plan to include such course evaluations in the packet, if I understand correctly the process.  

    Excellent idea. This proposal would save a lot of faculty and staff time, and also standardize the way that data is pulled and presented. Currently, when faculty pull data, there can be a lot of time spent, and still there are questions about whether the data is current and complete.

    No question that this will be helpful: it should improve accuracy in reporting scores and save time and headaches.  It is another question whether E*Value scores demonstrate true teaching ability vs other surrogate markers - but maybe this project will also help clarify that.

    I think this is a terrific idea.  Often faculty are asked to provide the average scores for the course or the department as a point of comparison.  It would be great if there was a way to easily integrate that information into the report as well, or perhaps have available separate reports for the courses and departments that the faculty could pull in addition to their own.

    Hi Carrie, Your suggestion about incorporating the comparison data is a good one. It is something we can consider for a second phase or expansion. 

    I had reached out to others I have worked with, and the following people also provided their support in a separate email:  Keith Yamamoto, Ron Arenson, David Julius, Allan Basbaum, Stephen McLeod, and Jacque Duncan.

    I am in favor of this important project!

    As someone who works with evaluations and reports, I am in complete support of this.  The process for generating faculty evaluation reports for promotion packets or portfolio updates out of E*Value is not a simple or intuiative, this would free up time for everyone involved from faculty, staff, department academic affairs and especially HR.  Although we have a user guide for faculty in how to do this, faculty constantly contact me about pulling their evaluations since it's not simple or they cannot figure out when or which course they have taught in our complex curriculum.  Currently if the faculty is unclear on the timeframes that they teach in, the report generated could be not accurate.  If this project becomes reality, there would be no confusion for the faculty and their evaluations and the data would be ensured to be accurate, which would be beneficial for everyone.  This has huge time saving potential for all involved in the promotion process for faculty.

    Absolutely, Sylvia.  As an administrator who also assists faculty in obtaining this data I am in favor of a streamlined and automated system.  It will save a tremendous amount of time and greatly reduce stress and anxiety for everyone involved. 

    I want to express my enthusiasm and support for this project. One of my responsibilities is to support department staff with E*Value use, which includes obtaining faculty evaluation data in E*Value for advancement purposes. Faculty teach in multiple settings and across different departments, and it would be great to be able to obtain faculty performance data in an efficient and accurate manner. Administrative staff and faculty often have to jump through multiple hoops in order to gather all the necessary reports, which are not always centrally available or easily accessible. Having a centralized repository of faculty evaluation data would save a tremendous amount of time for both staff and faculty. It would be helpful to have the evaluation data contextualized by different settings and comparison data.

    Everything about this sounds good. As a clerkship coordinator I periodically receive urgent requests for faculty evaluations to be used in their promotion packet. Since I only have access to those courses I am involved with, the faculty member or HR are often sent elsewhere to get the remaining evaluations from other courses the faculty member teaches in. Centralizing this data and removing the urgent requests for multiple people sounds like a good thing.

    I would love to see this happen! It's a natural bridge between systems that will save faculty and administrative staff time, and potentially improve the completeness of data used in the academic review process.