Status: This module is currently in development
Rationale
Concurrent with broader developments in Open Science and increased transparency in research, Open Peer Review is a complex, and rapidly evolving topic. Alongside this, more diverse criteria of research evaluation beyond traditional methods are emerging, and with these come a range of practical, ethical, and social factors to consider. This module will provide insight into current developments in Open Peer Review and research evaluation.
Learning outcomes
- The researcher will be able to describe the history of peer review in the context of scholarly publishing, the criticisms levied against traditional peer review, and the ongoing developments with Open Peer Review.
- The researcher will be able to use a range of post-publication review, commenting, and annotation services.
- The researcher will be able to describe the issues associated with the use of traditional metrics in research evaluation, and the role that peer evaluation and next-generation metrics (or altmetrics) play in this.
- The researcher will be able to use a range of services to build and demonstrate their personal research impact profile, both quantitatively and qualitatively.
- The researcher will become familiar with the relevant criteria for research evaluation to them, and be able to have a critical discussion about them with their colleagues and those who drafted them.
Resources
Tools
- Peer review report template, Authorea
- Eigenfactor project
- Publons Academy
- Metrics Toolkit
- Open Badges
Research Articles and Reports
- Why the impact factor of journals should not be used for evaluating research, Seglen, 1997
- Effect of open peer review on quality of reviews and on reviewers’ recommendations: a randomised trial, van Rooyen et al., 1999
- A Reliability-Generalization Study of Journal Peer Reviews: A Multilevel Meta-Analysis of Inter-Rater Reliability and Its Determinants, Bornmann et al., 2010
- Effect on peer review of telling reviewers that their signed reviews might be posted on the web: randomised controlled trial, van Rooyen et al., 2010
- Open peer review: A randomised controlled trial, Walsh et al., 2010
- Deep impact: unintended consequences of journal rank, Brembs et al., 2013
- Excellence by Nonsense: The Competition for Publications in Modern Science, Binswanger, 2014
- Attention! A study of open access vs non-open access articles, Adie, 2014
- Publishing: Credit where credit is due, Allen et al., 2014
- The Metric Tide report, Wilsdon et al., 2015
- Grand challenges in altmetrics: heterogeneity, data quality and dependencies, Haustein, 2016
- Badges to Acknowledge Open Practices: A Simple, Low-Cost, Effective Method for Increasing Transparency, Kidwell et al., 2016
- A framework to monitor open science trends in the EU, Smith et al., 2016
- Peer Review Survey 2015: Key Findings, Mark Ware Consulting, 2016
- Point of View: How open science helps researchers succeed, McKiernan et al., 2016
- Peer Review Quality and Transparency of the Peer-Review Process in Open Access and Subscription Journals, Wicherts, 2016
- Next-generation metrics: Responsible metrics and evaluation for open science, European Commission, 2017
- Evaluation of Research Careers fully acknowledging Open Science Practices: Rewards, incentives and/or recognition for researchers practicing Open Science, European Commission, 2017
- Research: Gender bias in scholarly peer review, Helmer et al., 2017
- “Excellence R Us”: university research and the fetishisation of excellence, Moore et al., 2017
- Metrics for openness, Nichols and Twidale, 2017
- What is open peer review? A systematic review, Ross-Hellauer, 2017
- Survey on open peer review: Attitudes and experience amongst editors, authors and reviewers, Ross-Hellauer et al., 2017
- A multi-disciplinary perspective on emergent and future innovations in peer review, Tennant et al., 2017
- Reviewer bias in single- versus double-blind peer review, Tomkins et al., 2017
- Prestigious science journals struggle to reach even average reliability, Brembs, 2018
- Making research evaluation more transparent: Aligning research philosophy, institutional values, and reporting, Dougherty et al., 2018
- Research excellence indicators: time to reimagine the ‘making of’?, Ferretti et al., 2018
- The Journal Impact Factor: A brief history, critique, and discussion of adverse effects, Lariviere and Sugimoto, 2018
Key posts
- Six essential reads on peer review, ASAPbio
- Peer reviews are open for registering at Crossref, Jennifer Lin
- Why we don’t sign our peer reviews, Jeremy Yoder
- The Fractured Logic of Blinded Peer Review in Journals, Hilda Bastian
- The peer review process: challenges and progress, Irene Hames
- Responsible metrics: Where it’s at?, Lizzie Gadd
- Goodhart’s Law and why measurement is hard, David Manheim
- Academe’s prestige problem: We’re all complicit in perpetuating a rigged system, Maximillian Alvarez
- Let’s move beyond the rhetoric: it’s time to change how we judge research ,Stephen Curry
- Blockchain offers a true route to a scholarly commons, Lambert Heller
Other
- Metrics and Research Assessment, ScienceOpen collection
- The Open Access Citation Advantage, ScienceOpen collection
- Citation Behaviour and Practice, ScienceOpen collection
- Scholarly Publication Practices and Impact Factor Calculation and Manipulation, ScienceOpen collection
- Peer Review in the Age of Open Science, Tony Ross-Hellauer, 2017
- The San Francisco Declaration on Research Assessment (DORA)
- The Leiden Manifesto
- The Humane Metrics Initiative
- Open Research Badges
- OpenUpHub must reads
- NISO Alternative Assessment Metrics (Altmetrics) Initiative
- Snowball Metrics, standardized research metrics