Systematic Reviews: Methods & Resources

Many organizations have created guidelines to standardize reporting of analytical research. See some of the main ones below. The NIH offers a useful chart of Research Reporting Guidelines, and you can find over 500 on the EQUATOR network.

When comparing therapies/interventions:

• PRISMA (Guideline on how to perform and write-up a systematic review and/or meta-analysis of the outcomes reported in multiple clinical trials of therapeutic interventions. PRISMA replaces the previous QUORUM statement guidelines): Liberati, A,, Altman, D,, Moher, D, et al. (2009). The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. Plos Medicine, 6(7):e1000100. doi:10.1371/journal.pmed.1000100 
• AHRQ (2014)'s Methods Guide for Effectiveness and Comparative Effectiveness Reviews

When comparing diagnostic methods:

• Checklist for Artificial Intelligence in Medical Imaging (CLAIM). CLAIM is modeled after the STARD guideline and has been extended to address applications of AI in medical imaging that include classification, image reconstruction, text analysis, and workflow optimization. The elements described here should be viewed as a “best practice” to guide authors in presenting their research. Reported in Mongan, J., Moy, L., & Kahn, C. E., Jr (2020). Checklist for Artificial Intelligence in Medical Imaging (CLAIM): A Guide for Authors and Reviewers. Radiology. Artificial intelligence, 2(2), e200029. https://doi.org/10.1148/ryai.2020200029
• STAndards for the Reporting of Diagnostic accuracy studies (STARD) Statement. (Reporting guidelines for writing up a study comparing the accuracy of competing diagnostic methods) http://www.stard-statement.org/

When evaluating clinical practice guidelines:

• AGREE Research Trust (ART) (2013). Appraisal of Guidelines for Research & Evaluation (AGREE-II). (A 23-item instrument for assessing the quality of Clinical Practice Guidelines. Used internationally for evaluating or deciding which guidelines could be recommended for use in practice or to inform health policy decisions.)
• National Guideline Clearinghouse Extent of Adherence to Trustworthy Standards (NEATS) Instrument (2019). (A 15-item instrument using scales of 1-5 to evaluate a guideline's adherence to the Institute of Medicine's standard for trustworthy guidelines. It has good external validity among guideline developers and good interrater reliability across trained reviewers.)

When you need to re-analyze individual participant data:

• If you wish to collect, check, and re-analyze individual participant data (IPD) from clinical trials addressing a particular research question, you should follow the PRISMA-IPD guidelines as reported in Stewart, L.A., Clarke, M., Rovers, M., et al. (2015). Preferred Reporting Items for a Systematic Review and Meta-analysis of Individual Participant Data: The PRISMA-IPD Statement. JAMA, 313(16):1657-1665. doi:10.1001/jama.2015.3656.

When comparing studies involving animals, livestock, or food:

• Sargeant JM, et al. (2010). The REFLECT Statement: Reporting Guidelines for Randomized Controlled Trials in Livestock and Food Safety: Explanation and Elaboration. Zoonoses Public Health. 57(2):105-36. Epub 2010/01/15. doi: JVB1312 [pii] 10.1111/j.1863-2378.2009.01312.x. PubMed PMID: 20070652.
• NC3Rs ARRIVE Guidelines (Animal Research: Reporting of In Vivo Experiments)

When reporting the results of a Randomized Controlled Trial:

• Consolidated Standards of Reporting Trials (CONSORT) Statement. (2010 reporting guideline for writing up a Randomized Controlled Clinical Trial). http://www.consort-statement.org. Since updated in 2022, see Butcher, M. A., et al. (2022). Guidelines for Reporting Outcomes in Trial Reports: The CONSORT-Outcomes 2022 Extension. JAMA : the Journal of the American Medical Association, 328(22), 2252–2264. https://doi.org/10.1001/jama.2022.21022
• Kilkenny, C., Browne, W. J., Cuthill, I. C., Emerson, M., & Altman, D. G. (2010). Improving bioscience research reporting: The ARRIVE guidelines for reporting animal research. PLoS Biology, 8(6), e1000412–e1000412. https://doi.org/10.1371/journal.pbio.1000412 (A 20-item checklist, following the CONSORT approach, listing the information that published articles reporting research using animals should include, such as the number and specific characteristics of animals used; details of housing and husbandry; and the experimental, statistical, and analytical methods used to reduce bias.)

The following is a list of useful resources and articles.

Tools:

• Right Review: helps you choose the proper review methodology for your project
• The Systematic Review Toolbox: an online catalogue of tools that support various tasks within the systematic review and wider evidence synthesis process. Tools include the 'Polyglot Search Translator'. 

General Guidance:

• Institute of Medicine. (2011). Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: National Academies 
• International Committee of Medical Journal Editors (2022). Recommendations for the Conduct, Reporting, Editing, and Publication of Scholarly work in Medical Journals. Guidance on how to prepare a manuscript for submission to a Medical journal.
• Cochrane Handbook of Systematic Reviews of Interventions 
• Joanna Briggs Institute (JBI) Manual of Evidence Synthesis: provides guidance on how to analyze both quantitative and qualitative research.
• Cochrane Qualitative & Implementation Methods Group. (2019). Training resources. Retrieved from https://methods.cochrane.org/qi/training-resources. (How to do a meta-synthesis, or qualitative evidence synthesis). 
• Center for Reviews and Dissemination (University of York, England) (2009). Systematic Reviews: CRD's guidance for undertaking systematic reviews in health care. (British systematic review guidelines). 
• Sutton, A., Clowes, M., Preston, L., & Booth, A. (2019). Meeting the review family: exploring review types and associated information retrieval requirements. Health information and libraries journal, 36(3), 202–222. doi:10.1111/hir.12276  (Describes different types of literature reviews).
• Riegelman R.  Studying a Study and Testing a Test: Reading Evidence-based Health Research, 6th Edition.  Lippincott Williams & Wilkins, 2012. (Textbook of quantitative statistical methods used in health sciences research).
• Guyatt, G., Rennie, D., Meade, M., & Cook, D. (2015). Users' guides to the medical literature (3rd ed.). New York: McGraw-Hill Education Medical.  (This is a foundational textbook on evidence-based medicine and of particular use to the reviewer who wants to learn about the different types of published research article e.g. "what is a case report?" and to understand what types of study design best answer what types of clinical question).

Guidance for searching:

• Hunter, K. E., et al. (2022). Searching clinical trials registers: guide for systematic reviewers. BMJ (Clinical research ed.), 377, e068791. https://doi.org/10.1136/bmj-2021-068791
• Canadian Agency for Drugs and Technologies in Health (CADTH) (2019). Grey Matters: a practical tool for searching health-related grey literature. Retrieved from https://www.cadth.ca/resources/finding-evidence/grey-matters.
• Glanville, J., Duffy, S., Mccool, R., & Varley, D. (2014). Searching ClinicalTrials.gov and the International Clinical Trials Registry Platform to inform systematic reviews: what are the optimal search approaches? Journal of the Medical Library Association : JMLA, 102(3), 177–183. https://doi.org/10.3163/1536-5050.102.3.007
• McGowan J, Sampson M, Salzwedel DM, Cogo E, Foerster V, Lefebvre C. PRESS Peer Review of Electronic Search Strategies: 2015 Guideline Statement. J Clin Epidemiol. 2016;75:40–46. doi:10.1016/j.jclinepi.2016.01.021. PRESS is a guideline with a checklist for librarians to critically appraise the search strategy for a systematic review literature search.

Quality of Evidence Grading:

• Grading of Recommendations Assessment, Development and Evaluation (GRADE). (An international group of academics/clinicians working to promote a common approach to grading the quality of evidence and strength of recommendations.) 
• Phillips, B., Ball, C., Sackett, D., et al. (2009). Oxford Centre for Evidence Based Medicine: Levels of Evidence. Retrieved 3/20/17 from https://www.cebm.net/wp-content/uploads/2014/06/CEBM-Levels-of-Evidence-2.1.pdf. (Another commonly used criteria for grading the quality of evidence and strength of recommendations, developed in part by EBM guru David Sackett.) 

• Campbell, M. (2020). Synthesis without meta-analysis (SWiM) in systematic reviews: reporting guideline. BMJ, 368. doi: https://doi.org/10.1136/bmj.l6890 (guideline on how to analyse evidence for a narrative review, to provide a recommendation based on heterogenous study types).
• Community Preventive Services Task Force (2021). The Methods Manual for Community Guide Systematic Reviews. (Public Health Prevention systematic review guidelines)
• Enhancing the QUAlity and Transparency Of health Research (EQUATOR) network. (Tracking and listing over 550 reporting guidelines for various different study types including Observational studies, Qualitative research, Quality improvement studies, and Economic evaluations). http://www.equator-network.org/resource-centre/library-of-health-research-reporting/
• Cochrane Qualitative & Implementation Methods Group. (2019). Training resources. Retrieved from https://methods.cochrane.org/qi/training-resources. (Training materials for how to do a meta-synthesis, or qualitative evidence synthesis). 
• Cornell University Library (2019). Planning worksheet for structured literature reviews. Retrieved 4/8/22 from https://osf.io/tnfm7/ (offers a framework for a narrative literature review).
• Green, B. N., Johnson, C. D., & Adams, A. (2006). Writing narrative literature reviews for peer-reviewed journals: secrets of the trade. Journal of Chiropractic Medicine, 5(3): 101-117. DOI: 10.1016/S0899-3467(07)60142-6.  This is a very good article about what to take into consideration when writing any type of narrative review.

When reviewing observational studies/qualitative research:

• STrengthening the Reporting of OBservational studies in Epidemiology (STROBE) statement. (Reporting guidelines for various types of health sciences observational studies). http://www.strobe-statement.org 
• Meta-analysis of Observational Studies in Epidemiology (MOOSE) http://jama.jamanetwork.com/article.aspx?articleid=192614
• RATS Qualitative research systematic review guidelines. https://www.equator-network.org/reporting-guidelines/qualitative-research-review-guidelines-rats/

GW researchers may want to consider using Refworks to manage citations, and GW Box to store the full text PDF's of review articles. You can also use online survey forms such as Qualtrics, RedCAP, or Survey Monkey, to design and create your own coded fillable forms, and export the data to Excel or one of the qualitative analytical software tools listed above.