Status: This module is currently in development
Rationale
Reproducible research is at the heart of science. There has been an increased need and willingness to open and share research from the data collection right through to the interpretations of results. This has come with its own set of challenges, which include designing workflows that can be adopted by collaborators in a way that does not compromise the integrity of their contribution. This module will introduce the necessary tools required for transparent reporting which is reproducible and readable.
Learning outcomes
- Researchers will be able to describe the key factors that affect the reproducibility of research, including workflow design, data management, and reporting.
- The researcher will be able to use a range of resources to create and implement a workflow for reproducible research, including using lab notebooks and tools for sharing code and data.
Resources
Tools
- Center for Open Science (COS)
- Open Science Framework (OSF)
- Reproducibility Project: Cancer Biology
- Reproducibility Project: Psychological Science
- Registered Reports
- Existing reproducible research workshops/practical resources:
- Reproducible Research, Workshop, CC-BY, April Clyburne-Sherin & Courtney Soderberg
- Initial steps towards reproducible research. Karl Broman
- The Open Science and Reproducible Research course, CC-BY, Christie Bahlai
- Reproducibility Workshop, Best practices and easy steps to save time for yourself and other researchers, Code Ocean
- Reproducibility in Science: A guide to enhancing reproducibility in scientific results and writing,ROpenSci
- Reproducible Research using Jupyter Notebooks workshop, The Carpentries
- R markdown workshop, (Liberate Science)
- rrtools: Tools for Writing Reproducible Research in R, Ben Marwick
- ReproZip, an Open source tool for full computational reproducibility
- Software Carpentry and Data Carpentry lessons
- Jupyter notebooks and JupyterLab, R Markdown, Stencila
- Virtual Machines:
- Docker
- Vagrant
- BinderHub
- nteract.io
- Binder Documentation for creating custom computing environments that can be shared and used by multiple remote users
- Statcheck, GRIM
- Scienceroot, the first blockchain-based scientific ecosystem
- Online repositories for open hardware:
- PLOS open source toolkit channel
- Open Neuroscience
- Open Plant Science
- Appropedia
- DocuBricks
- Hackaday.io
- Bio-protocol, a peer reviewed protocol journal
- BMJ Open Science, a new journal that aims to improve the transparency, integrity and reproducibility of biomedical research
- Evernote
- Labguru
- sciNote
- AsPredicted
- The Sci-Gaia Open Science Platform
- Improving your statistical inferences, Daniel Lakens
- Open Stats Lab, Kevin McIntyre
- R for Data Science
- R tutorial: Introduction to cleaning data with R, DataCamp
- Nextflow, open source tool than enables reproducible and portable computational workflows across cloud and clusters
Research Articles and Reports
- Reproducibility, Virtual Appliances, and Cloud Computing, Howe,2012
- The Ironic Effect of Significant Results on the Credibility of Multiple-Study Articles, Schimmack, 2012
- Power failure: why small sample size undermines the reliability of neuroscience, Button et al., 2013
- Git can facilitate greater reproducibility and increased transparency in science. Ram, 2013
- Ten simple rules for reproducible computational research, Sandve et al., 2013
- Investigating Variation in Replicability: A “Many Labs” Replication Project, Klein et al., 2014
- An introduction to Docker for reproducible research, Boettiger, 2015
- Opinion: Reproducible research can still be wrong: Adopting a prevention approach, Leek and Peng, 2015
- Replicability vs. reproducibility – or is it the other way around?, Liberman, 2015
- The GRIM test: A simple technique detects numerous anomalies in the reporting of results in psychology, Brown and Heathers, 2016
- What does research reproducibility mean?, Goodman et al., 2016
- Tools and techniques for computational reproducibility, Piccolo and Frampton, 2016
- Transparency, Reproducibility, and the Credibility of Economics Research, Christensen and Miguel, 2017
- A trust approach for sharing research reagents, Edwards et al., 2017
- Estimating the Reproducibility of Psychological Science, Nosek et al., 2017
- Digital Open Science – Teaching digital tools for reproducible and transparent research, Toelch and Ostwald, 2017
- Terminologies for reproducible research, Barba, 2018
- An introduction to statistical and data sciences via R, Ismay and Kim, 2018
- The practice of reproducible research: case studies and lessons from the data-intensive sciences, Kitzes et al., 2018
- bookdown: Authoring Books and Technical Documents with R Markdown, Xie, 2018
- Our path to better science in less time using open data science tools, Lowndes et al. 2017
- Haves and Have nots must find a better way: The case for Open Scientific Hardware, Chagas, 2018
- Computational Reproducibility via Containers in Social Psychology, Green and Clyburne-Sherin, 2018
Key Posts
- Data hygiene and data provenance:
- A Data Cleaner’s Cookbook
- Storify by Dawn Bazely
- Failure is moving science forward, Christie Aschwanden
- 5 keys to building open hardware, Joshua Pearce
- How to make replication the norm, Gertler et al., 2018
- Reproducibility PI Manifesto, Lorena Barba
- How to run a lab for reproducible research, Lorena Barba
- Essential skills for reproducible research computing, Barba et al., 2017
Other
- Institutions, projects, and companies using or providing open hardware/materials:
- CERN’s Open Hardware Repository and Open Hardware License
- UFRGS Centro de Tecnologia Academica, CTA
- Michigan Tech Open Sustainability Technology research group
- Open Plant
- Trend in Africa
- Open Lab Tools Cambridge University
- PhotosynQ
- PublicLab
- BackyardBrains
- OpenPCR
- OpenROV
- Prometheus Science
- senseBox
- Addgene
- Definition of Open Reproducible Research, FOSTER
- Global Open Science Hardware Roadmap, GOSH
- Open and Reproducible Science syllabus, Campbell, 2018
- EQUATOR network, Enhancing the QUAlity and Transparency Of health Research)
- Knitr, Elegant, flexible, and fast dynamic report generation with R, Yihui Xie
- Using Sweave and knitr, RStudio Support