Today, I'll briefly be speaking about what preprints mean for critical care science and what we can expect from them in the future. I currently have funding from the NHLBI as well as I serve as editor-in-chief of Annals of the American Thoracic Society. Here I've provided an overview in a schematic format describing scientific publishing in its current form. As we all know, manuscripts are typically submitted to a journal, subsequently undergo administrative processing followed by peer review. Manuscripts that are selected as high priority are then revised, potentially being sent back to the journal and then either rejected by the editors of the journal or ultimately accepted for publication. Upon acceptance, there is a publication lag between when a manuscript is ultimately accepted and when it's finally published, either in print or potentially even online, before which the public can consume that information. This process, on average, ultimately takes a median of about 100 days for the majority of scientific journals. So on average, by the time a manuscript has been submitted, it won't see acceptance or publication for at least three months. There is general consensus that this three-month lag between a time of paper submission to a journal and its ultimate acceptance is far too slow to create meaningful knowledge in areas where timely information is necessary. This was particularly relevant during the 2003 SARS epidemic when approximately only 8% of papers were ultimately published during the outbreak that were relevant for care of those patients. There was an additional 7% of papers that were accepted, but this left the vast majority of scientific information left unaccessible to the practicing individuals who needed information about these patients. In this figure, you can see that this publication lag spanned all research domains from search for causative agents all the way to treatments as well as risk factors. SARS epidemic is shown here by the gray distribution. In response to the lack of readily available information around the SARS epidemic, as well as growing discontent within the scientific community around publication lag, the scientific community advanced a new mode for publishing papers early, and thus the preprint was born. The preprint sought to bypass the lengthy peer review and revision process that typically occupied many weeks to months in the journal's cycle of paper evaluation to immediately put research in the hands of the public to allow them to consume and potentially benefit care for patients. Preprints by definition are considered publicly accessible scholarly content that has not yet been certified by a journal's peer review process. There are a number of preexisting preprint servers whose sole purpose is to catalog and maintain scientific information that's not yet been peer reviewed. These initially began in 1991 with ARXiv, pronounced archive, which cataloged scientific discoveries in the physical sciences. In 2013, BioRxiv was developed, cataloging scientific information from the biological sciences, and ultimately in 2019, MedRxiv was launched, which represents information surrounding the clinical sciences. Since the creation of BioRxiv in 2013, there has been a steady and increasing use of preprints as a way to disseminate scientific information. Shown in these figures are the number of papers published versus in preprint, both during the Zika epidemic as well as the Ebola epidemic in 2014 and 2016. You can see that approximately only 10% of papers during both of these outbreaks were published in preprints relative to the total number of papers published in the scientific literature. In contrast, at the beginning of the COVID pandemic in 2020, nearly 20 to 25% of all science was represented in preprint form relative to the only 10% represented during the most recent Zika outbreak in 2016. While there is general acceptance that preprints as an important way to rapidly disseminate new knowledge, they have also brought several other additional benefits to the scientific community, but also have brought several ongoing challenges. I've broken these benefits as well as challenges into several areas that preprints have impacted, including authors, editors, funders, and readers. There are a number of benefits to preprints from the perspective of an author. First and perhaps most importantly is the ability to disseminate new knowledge rapidly in the scientific community that is delayed when publishing a manuscript in a traditional journal. This allows for earlier community engagement where scientists can potentially interact with other clinicians or other scientists to discuss the findings at an earlier stage. Inclusion of manuscripts that are on a preprint server in one's CV may also allow for authors to more timely address issues related to promotion. Some authors use preprints as a way to protect the provenance of intellectual content, specifically staking a claim in a scientific finding in an area that might be hotly contested or is otherwise competitive. Authors often benefit because there is no paywall to preprint publication, allowing a broad reach of the science for free. This has become increasingly important to authors who are concerned about equity at the global level. And also, preprints allow potentially a manuscript or the scientific findings in a manuscript to gain greater visibility and recognition. This was nicely demonstrated in a paper published in JAMA in 2018 where authors examined over 8,000 preprints published on BioRxiv, matched them to up to five published articles on PubMed that didn't have a preprint, matched for the same journal or publication, as well as timeframe. They then compared altimetric attention scores between articles who were published in the scientific literature that did have a preprint versus those that didn't have a preprint, but were also published in the medical literature. Altimetric attention scores, as shown here, is a crude measure of engagement with the scientific community as well as the public around a specific manuscript. Authors in the scientific literature with a preprint had significantly higher altimetric attention scores than those published in the medical literature who did not have a preprint, suggesting that preprints significantly increase the engagement with the scientific and public community around a scientific finding. In contrast to the significant benefits of preprints to authors, there are relatively few potential harms. The first and most often cited harm has to do with an interpretation of the Engelfinger Rule, which states that most medical journals will not consider for publication a manuscript that's been previously published elsewhere. However, the vast majority of journals consider preprints an exception to this rule. In fact, over 86% of the top 100 journals by impact factor will consider for publication manuscripts that have been uploaded to a preprint server. An additional potential harm has to do with the added work necessary to engage with the preprint community. However, a majority of authors that do upload to preprints find this engagement to be beneficial. There are a number of potential benefits to preprints from the perspective of an editor. Because preprints typically receive feedback from a broader and more diverse group of readers than afforded by the typical peer-review process, editors may view submissions with a preprint as more thoroughly peer-reviewed and thus a superior product. Moreover, as discussed in prior slides, journal publications that had a preprint received greater attention, perhaps driving greater traffic and citations, two key metrics that journals rely upon to measure success. Many journals have also streamlined preprints into their workflow, specifically allowing the transfer of a manuscript from a preprint server directly to a journal submission process. This may reduce administrative burden within a journal when processing manuscripts. However, there are also some potential harms. Editors view themselves and their journals as credible sources of information and likewise take some responsibility to combat misinformation. Misleading non-peer-reviewed science in preprint form that garners significant attention may demand that editors spend real estate in their journals to combat such misinformation. In addition, on occasion, authors will continue to cite preprints instead of the final peer-reviewed journal publication, which diverts traffic away from the journal, potentially in a harmful way. Authors or other editors may also feel responsible for reviewing a preprint's history and feedback when commenting on a manuscript's acceptability for publication in the journal. Ultimately, this creates more work for what is already a volunteer effort for the vast majority of peer reviewers and editors. From the perspective of funders, preprints may be beneficial in that they allow writers and reviewers access to the most up-to-date science. They may also allow junior investigators to better demonstrate productivity in an area that they're interested in being funded, or they may allow more seasoned investigators to demonstrate productivity in an area that is rapidly evolving or relatively novel. However, most funding agencies have yet to determine the right role for preprint publications or how they should be weighted in the peer-review process when assessing a proposal's strengths. From the perspective of the consumer, preprints have the most opportunity to benefit the community when there is an immediate need for knowledge, such as what we've experienced with the COVID-19 Real-time dissemination of emerging epidemiology, prognosis, cases, case theories about novel diseases potentially can be life-saving in some circumstances. However, there are some potential harms that come as well. Correct interpretation of the findings of preprints may require attention to the comments on the preprint that emerge from the scientific community. In some cases, poorly conducted science may raise safety concerns about suboptimally vetted treatments. In other words, shoddy science has the potential to be disseminated via this means. Ultimately, the central argument surrounding the benefits versus harms of preprints comes down to timeliness versus credibility. The word preprint neither implies shoddy science nor absolves investigators of the need for critical review, yet it provides opportunities for both. So let's take a look at what the literature says about this competing tension. In the modern era, preprints still dramatically reduce the time to data access by at least two to three months. This was nicely demonstrated by Fraser and colleagues published in PLOS Biology recently. In their analysis, they looked at thousands of preprint manuscripts that were ultimately published in the medical literature by most of the major publishing houses across the world. On the left panel, you'll see the time it took for preprint servers to take the uploaded manuscript and ultimately release it to the public. You can see for both COVID preprints and non-COVID preprints, this averaged about one to two days of screening time before a manuscript was ultimately released for public consumption. In contrast, on the right panel, you'll see the time between when the paper was uploaded to the preprint server and when it was ultimately published by the medical journals. And you can see that there was significant delays across all both COVID and non-COVID manuscripts. And while COVID-19 preprints were ultimately published in the medical literature much faster than non-COVID-19 preprints, there was considerable delays across all groups. With the red bars, on average, anywhere between 50 to 100 days post-preprint posting before it was published in the medical literature. These data demonstrate that while publishers responded to the need for rapid COVID-19 information by publishing COVID-19 papers faster than non-COVID papers, there was still considerable delays in data dissemination that ultimately were caused by peer review of at least 50 to 100 days. But what about preprint credibility? Is the information that's being released earlier quality information? Some of the most important discoveries of the COVID pandemic were published first in preprint form. The manuscript first describing the COVID-19 outbreak in China was uploaded to MedArchive on February 9, 2020. Ultimately, the New England Journal peer reviewed and accepted this manuscript and posted it online on their website in March 2020, and finally published the manuscript in its paper form in April 2020. On the one hand, this demonstrates how journals can rise to the occasion and rapidly publish incredibly important findings. On the other hand, that one to two month lag may have had incredible consequences for public health. Similarly, the initial recovery trial, which demonstrated a mortality benefit to individuals with COVID-19 treated with dexamethasone, was posted in preprint form in June of 2020, and ultimately accepted by the New England Journal and posted online one month later. The recovery investigators also published their study examining early tocilizumab and COVID-19 in preprint form in February 2021, which was ultimately accepted and published by The Lancet in May of 2021. Advocates for preprints may highlight these last three examples as evidence as to their great credibility and importance and potential to save lives that might be lost had such findings not been disseminated via preprint form. However, during the pandemic, there are examples of how preprints may have spread misinformation to the detriment of the scientific community and ultimately to patients. Many of you are aware of the initial trial suggesting that hydroxychloroquine in combination with azithromycin was beneficial as a treatment for patients with COVID-19. This initially was posted to MedArchive in March 2020 and published in July of 2020. The study was widely and rapidly criticized. Among other concerns, it did not meet a priori sample size requirements. It incorporated a trial arm that was not pre-specified. And finally, the study was rapidly contradicted by numerous other studies. Despite these issues, the preprint was widely picked up by the media and ultimately called a game-changing treatment by the sitting president of the United States. At the time of this talk, the published yet refuted study was cited almost 3,000 times. And while it's not clear to what extent the preprint contributed to the widespread hysteria surrounding this potential treatment, it certainly allowed greater opportunity for dissemination of misinformation surrounding these therapies. So if we're convinced that preprints have the capability of providing credible information in a more timely way than peer review, the question then is, what is the value of the peer review process? And does it add to the quality of the manuscripts being considered? In a fascinating analysis, Carniero and colleagues compared manuscript quality between preprints and manuscripts published in the medical literature. They did this two ways. First, they compared the quality of a random sample of preprints uploaded to BioRxiv to manuscripts published in the medical literature indexed by PubMed, shown in the panel on the left. Second, they conducted a pre-post analysis examining the quality of manuscripts uploaded to a preprint server that were ultimately published in the medical literature by comparing quality at the preprint stage versus the publication stage, shown in the panel on the right. To measure quality, they calculated the percent adherence of each manuscript to its respective manuscript reporting standard. For example, a clinical trial manuscript would be compared to the consort reporting standard to assess adherence. In both analyses, the quality of the published manuscripts was higher than the preprint manuscripts. There was a 4.2 percent greater adherence to reporting standards that resulted from the peer review process. So while peer review clearly improved the quality of manuscripts, it's not yet clear whether such improvement outweighs any delays in data dissemination that are required for peer review to occur. There is some evidence that the preprint community can be self-policing, and bad preprints can be removed via community engagement. One particular article, as an example, was posted in January of 2020, which suggested that the COVID-19 virus was actually man-made. The scientific community ultimately posted 135 comments critiquing the work, and the authors of the manuscript eventually withdrew the preprint from the preprint server. We also know that peer review is not a panacea for bad science, as evidenced by a couple of trials posted here that were published in high-impact journals after peer review, and ultimately withdrawn once investigation identified significant scientific integrity issues related to the conduct of these particular studies. With a more nuanced understanding of preprints, instead of labeling them as universally good or bad, we can focus our efforts on strategies to try and maximize the benefits as well as minimize the harms. A number of existing strategies are currently in place to minimize the harms of preprints. This includes preprint servers, automatic screening of pseudoscience and conspiracy theories prior to posting preprints, an active engagement with the community to solicit input and feedback about specific manuscripts, disclaimers to label preprints as not peer-reviewed or preliminary. There needs to be an important link for preprints to point to articles that are ultimately published in the medical literature post-peer review as the most credible source of information surrounding that specific topic. And finally, there needs to be some acceptance of consumer responsibility that there are pros and cons to preprints in their native form. There are also some emerging and novel ways in which we can reduce the harms of preprints. These include providing a mechanism through which we can recognize individuals who actively engage as commenters or peer reviewers of preprints. Unfortunately we have no standard by which that academic work receives credit. We also should develop as a community standards for how to incorporate preprints into our discourse. Publishers and funders need to figure out ways in which we can incorporate preprints into our discussion while acknowledging some of the pros and cons of using them in those discussions. And finally, there is some discussion around whether or not preprints should have an expiration date if they ultimately aren't published in the medical literature. This has some pushback because it may create some publication bias in that we're potentially suppressing data that journals ultimately are not interested enough in to be published, but nonetheless is an emerging way in which people are thinking about how to minimize harms of preprints. So what does the future of preprints in critical care look like? Well, it's clear that they're increasingly used and here to stay. And overall, they're largely beneficial, allowing access to new knowledge earlier and providing the community with a greater exposure to primary data. Consumer standards, however, are necessary to really minimize the harms which are potentially great when preprints are used to disseminate misinformation. However, medical journals are still in good shape as the default of peer review is still clearly additive and confers powerful credibility that the preprints have yet to really accomplish. Thank you so much for your attention. I'd be more than happy to discuss any topics presented in my talk offline. Feel free to email me at your leisure if you have questions or issues you'd like to discuss further.

preprints

critical care science

scientific publishing

timely information

peer review