CTSI Annual Pilot Awards to Improve the Conduct of Research

An Open Proposal Opportunity

Encouraging research collaboration and integrity by optimizing post-publication peer review for researcher participation

Submitted by Robert Judson on February 27, 2012 - 3:21pm Last revised by Anonymous on May 2, 2014 - 12:08pm.
Type: 
Proposal
Topics: 
Community Policies & Practices
Proposal Status: 
Final Proposal

Rationale

There is an integrity crisis in the life sciences. Since 1990, retraction rates for peer-reviewed medical research have increased by a factor of 4 (Cohol (2007) EMBO 792-3). This crisis highlights how traditional mechanisms of peer review are straining under the ever expanding volume of research being produced.
Post-publication peer review offers a promising solution: researchers could aggregate their opinions online to create an alternative metric of integrity. A number of systems have been built to allow this kind of peer review, but none have succeeded in engaging the research community.

Plan

Our team has spent close to a year conducting research in labs on the way that papers are read and discussed, and has created an alternative system which generates significantly more traction. The system is based around figure-by-figure summaries of a paper, soliciting comments on individual figures rather than on papers as a whole.
Initial tests with this system have found extremely promising improvements in both reading time and discussion rates over competing platforms. We are seeking funding to refine the system for launch across and beyond UCSF, and to cover the costs of user engagement.

Criteria and metrics for success

As a web-based application, we will have access to a plethora of data on user behavior. We are seeking to optimize around the following metrics:
 -In order to establish that we becoming a useful part of scientists’ workflows, we will seek to maximize the
number of papers viewed on our system per user per week.
 -In order to establish whether we are serving as a platform for meaningful discussion, we will seek to maximize the number of comments per paper view, and the amount of response (replies and upvotes) per comment.

Approximate cost and very brief justification ($10k-max $100k)

We estimate that launching the project will require a budget of $250k in 2012. The majority of this sum goes to salaries for web development and community management staff. We are seeking $100k to supplement our other sources of funding.

Collaborators

Core team:
Robert Judson- Completing PhD in UCSF BMS program. Focus on complex genetic networks & stem cell biology
David Jay, MBA- Consultant focusing in online community development, web developer.   

Core Advisers:
India Hook-Barnard, PhD- Program Officer, National Academy of Sciences
Keith Yomamato, PhD- Vice Chancellor for Research, UCSF

Comments

Maninder Kahlon - March 2, 2012 - 10:42am
Quite interesting. A couple of things to consider - a. If integrity is the issue, then replication based on openning up access to the data underlying figures is key. We're doing work at CTSI(with a few partners) to see if we can catalyze the open data/open science piece, so would be useful to understand the other 'movements' your idea ties to, such as this one. b. For this kind of a proposal, you'll want to share with us what you've done thus far and any data that shows you have traction already (e.g. a platform that UCSF grad students are already using, or that is being used by X# of journal clubs). Feel free to add attachments, or reach out offline. c. You should explain how what you're doing is different from PLoS approach to enabling online discussion and annotation around articles. How successful do you think that has been (or any other such effort by Elsevier, Science, etc), and what are the implications for your project?

David Jay - March 16, 2012 - 2:05pm
Excellent feedback! To respond to your questions: We’ve getting as active as we can in the open access/open science movements (it’s why we’re doing this). We’ve organized a meetup group of open science stratups in the Bay area, and are working with staff from PLoS to plan an Open Access Hackathon in June. We’d love to get further connected to open science advocates at UCSF. Our alpha is currently being used by one lab, though we’ll be expanding to more (and beyond UCSF) in the next few months. Our system is different than PLoS (we tie comments to figures rather than to entire papers or lines of text, and there are a few other differences), and is getting considerably higher traction.

Dennis Hsieh - March 3, 2012 - 11:36am
Reposting here in case you don't get notified I responded: I love your idea. I think it provides a public forum for discussion on articles and figures that is missing. Similar to news articles and other forums, having the ability to discuss outside of journal club/conferences but across institutions and across the field I think is a great way to critically evaluate papers. Additionally, I think having these comments available for those reading the paper would be helpful who may not have as much experience (ie students/junior investigators) or those who do not have the benefits of discussing the paper in a conference/at journal club. I think one challenge is figuring out how to organize the comment threads so that the comments themselves can be easy to browse through and can serve as a resource as well. On a side note it would be great to figure out how to get free access to the articles if you are not at a university without having to pay 30-40 for the article - I think the limited access also makes the work less accessible and less valuable. Would love to chat more if you have time the next few weeks. I could see us working together to figure out how to gather and evaluate the evidence and then use for a variety of purposes. -Dennis

David Jay - March 16, 2012 - 2:06pm
Thanks for your feedback! We're hoping that our system will be able to generate open access summaries of papers (and help convince journals to make their papers OA, but that's a longer term goal.)

Commenting is closed.