5 Innovations in Peer Review

Introduction

Masthead of The Woman Inventor publication, 1891
“The ‘Woman inventor’ greets the members and visitors of the Patent Centennial Convention and presents this number, the first ever published and devoted to the cause of woman inventors of this country.”

When web technologies became ubiquitous, they disrupted the scholarly dissemination ecosystem and provided opportunities to grow journal and publication audiences and expanded the manner in which research was shared. These technologies also allowed researchers to more quickly share ancillary and supplemental materials like data sets, instruments, lab notebooks, videos, audio files, and other behind-the-scenes research documentation.

Similarly, these technologies provided opportunities to innovate peer-review practices, taking them from the realm of print publication to online platforms. In this section we will explore some of the recent innovations in peer review and consider their merits and the potential ethical challenges they introduce.

By the end of this module you should be able to:

  • Recognize basic online systems used for peer review
  • Consider the ethical implications that technology has on peer review
  • Discover recent peer-review innovations
  • Evaluate innovations for their potential to implement opportunities and diminish challenges presented by traditional peer-review systems

Activities and Exercises

Icon of a play buttonWatch, Icon of an open bookRead, and Reflect: Journal Systems (5:1)

Watch this video about the peer-review process from PLOS. It depicts a straightforward review submission in their journal management system called Editorial Manager.

Read the Open Journal Systems (OJS) Review Process page, the Springer How to Target and Invite Reviewers page, and the reviewer invitation template letter from Radiology.

Read What Are Innovations in Peer Review and Editorial Assessment For?  By Halffman and Horbach

Now that you have seen a bit how technical systems work, what do you think? Set a timer for 5 minutes to write down your thoughts or record them aloud. Having trouble? Here are some guiding questions:

  • What does an online system do for peer review?
  • How can online systems be personalized or depersonalized?
  • From this introductory glance at systems, what is your gut reaction?

Icon of a play buttonWatch, Icon of an open bookRead, and Reflect: Innovations (5:2)

In this section you’ll be asked to read and reflect on some innovations in peer review. Listed first are the readings and content on each topic, followed by reflection questions. There are a lot of innovations to explore. They are all worthwhile. However, if you are short on time, complete either the Artificial Intelligence section or the Open Peer Review section.

Artificial Intelligence

Reflection Questions:

  • Generally, what is your reaction to AI innovations in peer review based on what you read and watched?
  • What advantages and disadvantages do you see in using AI for peer-review tasks?
  • Who is represented in AI and who is not?
  • Would the use of AI make more sense in some disciplines than others?
  • Do you think using AI in peer review is ethical? Why or why not?

Open Peer Review

Reflection Questions:

  • Generally, what is your reaction to open peer review based on what you read?
  • What advantages and disadvantages do you see in using open or transparent peer review?
  • Who is represented in open review and who is not?
  • Would the use of open review make more sense in some disciplines than others?
  • Do you think open peer review is ethical? Why or why not?

Other Innovations

Icon of a pencilDo: Find Other Innovations

Search the web to see if you can identify 1-3 additional peer-review innovations. Do some reading, and reflect using the following questions:

  • Generally, what is your reaction to the innovation based on what you have read?
  • What advantages and disadvantages do you see in the innovation?
  • Who is represented and who is not?
  • Would the use of this innovation make more sense in some disciplines than others?
  • Do you think this innovation is ethical? Why or why not?

Struggling to identify an innovation? Here is one suggestion:

  • Peer Community In (PCI) – “a free recommendation process of scientific preprints based on peer reviews”

Icon of a pencilDo: Vision Board (5:3)

Make a vision board* (digital or physical) of the future of peer review. Your vision board should be thoughtful and take into consideration everything you have learned and reflected upon in this course so far. Does your vision include a technology that has not yet been invented? Great! Does your vision eliminate all technologies in the process? Also great! Consider the innovations that you have learned about in this module, and decide whether you want to further develop them with your vision or completely eliminate them. Whatever your vision, consider these guiding questions:

  • What problems can technology solve in peer-review processes?
  • What aspects of peer review require a human approach vs. what can be automated in a human-centered way?
  • How does this vision board reflect my values?

After you have completed your vision board, write a short narrative (one or two paragraphs) explaining your vision. Be sure to address your reasoning for your vision, making apparent how you have evaluated the options you present within it. Share your vision board on social media using the course hashtag.

*What is a vision board? A vision board is a collection of images (or sounds or other objects) or other inspirations that represent your vision. Think of it as a mural or collage of things that represent what you see for the future. Here’s a short blog post about vision boards.

References

Burley, R. (2021, November 1). How AI is accelerating research publishing. Research Information. https://www.researchinformation.info/analysis-opinion/how-ai-accelerating-research-publishing

CACM. (2017, January 31). Computational support for academic peer review. Vimeo. https://vimeo.com/201905108

Domingo, M., & Harris, S. (2020). Transparent peer review—A practical solution to implement open peer review at scale: A case study. Science Editor, 43(3), 72–76. https://doi.org/10.36591/SE-D-4303-72

eLife. (n.d.). Peer review. ELife. Retrieved February 22, 2022, from https://elifesciences.org/about/peer-review

EU Science & Innovation. (n.d.). Open peer review explained [Video]. YouTube. Retrieved February 22, 2022, from https://www.youtube.com/watch?v=aEMLEZ-FKIc

Griffith, C. (2019, June 23). 4 reasons why you should create a vision board even if you don’t believe in the Law of Attraction. PropelHer. Retrieved February 22, 2022, from https://www.propelher.co.uk/reasons-create-a-vision-board-even-if-dont-believe-law-of-attraction

Halffman, W., & Horbach, S. P. J. M. (2020). What are innovations in peer review and editorial assessment for? Genome Biology, 21(1), 87. https://doi.org/10.1186/s13059-020-02004-4

Hecht, B., Wilcox, L., Bigham, J. P., Schöning, J., Hoque, E., Ernst, J., Bisk, Y., De Russis, L., Yarosh, L., Anjum, B., Contractor, D., & Wu, C. (2021). It’s time to do something: Mitigating the negative impacts of computing through a change to the peer review process. ArXiv, 2112.09544 [Cs]. http://arxiv.org/abs/2112.09544

PLOS. (2021, August 31). How to submit a peer review to PLOS ONE [Video]. Vimeo. https://vimeo.com/595595314

Radiological Society of North America. (n.d.). Potential reviewer letter. Radiology. Retrieved February 22, 2022, from https://pubs.rsna.org/doi/abs/10.1148/radiology?publicationCode=radiology

RAx. (n.d.). Research paper review assistant tool: One-stop solution for reviewers to conduct high-quality review [Video]. YouTube. Retrieved February 22, 2022, from https://www.youtube.com/watch?v=qEhqJOptqBw

Reviewing Process. (n.d.). Open Journal Systems. Retrieved February 22, 2022, from https://openjournalsystems.com/ojs-3-user-guide/reviewing-process/

Price, S., & Flach, P. (2017, March). Computational support for academic peer review. Communications of the ACM, 60(3), 70–79. https://cacm.acm.org/magazines/2017/3/213825-computational-support-for-academic-peer-review/fulltext

Springer Nature. (n.d.). How to target and invite reviewers. Retrieved February 22, 2022, from https://www.springer.com/gp/authors-editors/editors/how-to-target-and-invite-reviewers/32892

License

Icon for the Creative Commons Attribution-ShareAlike 4.0 International License

Peer Review: A Critical Primer and Practical Course Copyright © 2022 by Emily Ford is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License, except where otherwise noted.

Share This Book