1.    Rationale Modern basic science rewards large papers in highly cited journals.  However, it is difficult for translational and clinical researchers to assess the quality of a basic science paper.  A proxy is the number of citations, but that is a very inaccurate measure; some papers that cannot be replicated have hundreds of citations.  Given the lack of safeguards ensuring publication quality, the intense competition to produce high profile publications incentivizes publication bias (i.e. tendencies for journals to publish experiments confirming its original hypothesis or "positive" results).  An obvious way to affirm or dispute the quality of a paper is through replication.  Unfortunately, at present replication studies are not valued by the basic science community, and they typically go unpublished.  A large amount of translational and clinical research likely yields negative results because it was based on invalid basic science premises.  Our solution is MicroPub, a platform for soliciting and indexing “micro publications” that replicate data from published basic science articles.

2.    Plan 

We will create a website that publishes and indexes short replication studies.  Each “MicroPub” will contain an abstract, one figure, detailed methodology, and a short discussion.  Editorial moderation, rather than peer-review, will be used to evaluate the soundness of research. The short format will hasten the publication process.  Validation or refutation by independent investigators will serve as a high-quality measure of a finding’s reproducibility, and by extension, its validity.  This will be a critical resource for determining whether a basic science finding merits investigation at the translational/clinical level.  All papers will be Open Access. 

Initially, we will solicit MicroPubs from UCSF researchers.  We will utilize social media and traditional media outlets to popularize MicroPub.  Ultimately, we envision MicroPub being linked to the original article listings on PubMed, as well as related follow-up translational/clinical papers.   

2013-03-12 Addendum to Proposal:

We are discussing our goals with the UCSF Library, the Open Science Framework and the Reproducibility Initiative, and we have expanded our proposal as follows:

Many Small-Scale, Technical Basic Biological Experiments Can Only be Replicated by Academic Labs

Although core facilities and outside vendors are able to perform numerous specialized techniques at the same level or in some cases better than most academic labs, many high impact publications use novel methodologies that are technically challenging to replicate. These “artisanal” laboratories rely on apprentices (in the form of graduate students and postdocs), who learn from experts within the laboratory and then develop their own line of research. Thus, a paper can often only be replicated by members of the same academic lab or a competing academic lab.  There is no current outlet for publication of these replication studies, so they are very rarely published.  The scientific community misses out on this valuable information. MicroPub fills this niche.

MicroPub Re-aligns Cultural Incentives and Promotes Transparency

There are numerous reasons for the widespread lack of reproducibility in basic biomedical sciences.  But one reason is preeminent—you are not allowed to be wrong. This is institutional (NIH funding) and cultural. Given the current state of NIH funding, allowing the publication of a replication study that failed – admitting you are wrong – is to risk jeopardizing one's career.  And if many replication studies are never revealed to the wider scientific community, why do them? 

MicroPub aims to shift this culture.  It provides a venue that recognizes that irreproducibility often does not reflect scientific fraud or sloppiness, but may be due to other reasons.  Instead, irreproducibility is often due to subtle differences in experimental conditions or analyses, or publication bias.  Currently there is no published venue for discussion of these issues.  Although open access publications that utilize post-publication peer-review offer an avenue for online discussions, we feel that commenting per se does not carry the same strength as first-hand experimental data.  MicroPub provides a way to quickly publish first-hand data, and integrates it with open access and post-publication discussions.

Moreover, MicroPub provides a venue for scientists other than the initial authors to publish replication studies.  Right now, the peer-review barrier for doing so, particularly when a replication study contradicts the original study, is enormously high.  MicroPub is specifically devoted to studies of this type, so the barriers will be surmountable and the studies can be published and disseminated.

Of course, replication studies by initial authors will also be welcome.  It is often members of the same lab who replicate (or cannot replicate) each other’s results.  Again there are enormous disincentives (job security, NIH funding, reputation) against publishing such studies. MicroPub becomes a way for labs to publicly acknowledge and explore reasons for irreproducibility, and ultimately, to establish a reputation for honing long-term, course-correcting, scientifically valid results.  It is our hope that as cultural barriers against admitting error shift, so too will the institutional (NIH funding) barriers. 

2013-03-14 Addendum to Proposal:

MicroPub Keeps Track of Methodology Requests and Serves as a Detailed Methods Repository

One of the major barriers to replicating biological experiments is the unavailability of clear step-by-step methodologies and help with reagents and equipment. Due to journal space limitations, authors are unable to explain exactly how they did their experiments, and rely on citing previous publications that describe similar methods. However, methods are usually modified and these changes are often the key to getting the published results. As there is currently no accountability, requesting for protocols and reagents can be a slow, tedious process. MicroPub aims to solve this issue by providing a custom contact and tracking service that emails the first and the corresponding authors, explaining that a replicator wishes to replicate a particular figure in their publication. MicroPub will then announce this request on the website and keeps track of how long it has been since the original authors have been notified, and whether they have responded. These protocols are published onto MicroPub directly and become available to not only the replicator, but also to the wider community of scientists who may also wish repeat the experiment. To provide incentive, both the original authors and the replicator are credited for the methodology section. There are also cultural incentives, as collaborations are often predicated upon shared protocols and reagents. This act of sharing is one of the many benefits of attending scientific conferences, and MicroPub can provide a platform for these kinds of productive social interactions online.

3.    Criteria and metrics for success

MicroPub fits squarely within the mission of CTSI.  MicroPub “nurtures communication, encourages collaboration, fosters innovation, and catalyzes the successful conduct of research”.  Most importantly, MicroPub has potential to revolutionize translational and clinical research because it will ensure that researchers only pursue investigations on basic science findings that have been independently validated many times.  The success of the initiative will be measured by the total number of MicroPubs and by the number of MicroPubs used as justification for translational/clinical follow-up research.

4.    Approximate cost and very brief justification ($50K max)

The primary costs will be the development of a standardized publication format and of the website. Approximate initial cost: $35K

5.    Collaborators

We are a team of two, S.Y. Christin Chong, Ph.D., postdoctoral fellow, and Jonathan Russell, M.D.-Ph.D. student.  Both of us have extensive experience in scientific publication, science writing, and social media.  We will hire a developer experienced in creating web-based publishing platforms and social web services.