Systematic Reviews: Study selection and appraisal

1. First level screening: title and abstract review

At the initial screening stage, read just the title and abstract of the studies, and make a decision to include or exclude the study from your review. Record the decision and reasons for inclusion/exclusion.

2. Second level screening: full text review

Read and critically appraise the full text of each study you selected at the first pass screening stage to determine whether you wish to include them in your discussion and analysis. Evaluate each study based on the following criteria:

Does this study address a clearly focused question?

Did the study use valid methods to address this question?

Are the valid results of this study important?

Are these valid, important results applicable to my patient or population?

Does this study meet my inclusion/exclusion criteria?

If the answer to any of these questions is “no”, you may wish to read no further and exclude the study, or you may decide to include the study to inform your discussion but not include the results in your analysis. 

Make a decision on whether or not to include the study in your review, and write your decision and reasons for inclusion/exclusion at this second level/full text review stage on the study screening form. You will summarize the reasons for exclusion on the PRISMA flow diagram.

3. Reporting your screening decisions

In the final report in the methods section the PRISMA checklist Item 9 study selection will be reported as:

• How studies were screened e.g. by reading title & abstract, and how they were critically appraised e.g. by applying a standardized appraisal form appropriate for that study type.
• What sort of studies were excluded e.g. letters, conference abstracts, etc.
• Who reviewed/appraised the studies
• What the process was for resolving disagreements e.g. reporting the level of inter-rater agreement, how often arbitration about selection was required, & what efforts were made to resolve disagreement e.g. were original authors contacted

The PRISMA Flow Diagram maps out the screening process by showing the number of records identified, included and excluded, and the reasons for exclusions. Covidence keeps track of your screening decisions and generates a PRISMA flow diagram for you. Alternatively there is a PRISMA flow diagram generator at https://www.prisma-statement.org/prisma-2020-flow-diagram

PRISMA checklist Item 16 states that study selection should be reported as follows:

• "Describe the results of the search and selection process, from the number of records identified in the search to the number of studies included in the review, ideally using a flow diagram"
• "Cite studies that might appear to meet the inclusion criteria, but which were excluded, and explain why they were excluded"

Courtesy of Page, M. J., McKenzie, J. E., Bossuyt, P. M., et al. (2021). The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ (Clinical research ed.), 372, n71. https://doi.org/10.1136/bmj.n71

The research team should agree on the inclusion and exclusion criteria for studies you wish to review and put together a study screening form. To help identify your inclusion/exclusion criteria, revisit the PICOS of interest you came up with for your search strategy. The screening form may look similar to Table 3 of Brown et al (2013). You may write down your decision to include or exclude an article on an Excel spreadsheet like this one.

Courtesy of the University of Melbourne: https://unimelb.libguides.com/c.php?g=492361&p=3368110

To help with the screening process

There are three web-based software applications that can help with screening and tracking your selection decisions:

1. Covidence (free for GW affiliates, see tab). Provides a decision dashboard and annotation tool, and the ability to screen candidate citations you locate in your literature search. You can invite team members to collaborate on your Covidence review.
2. Abstrackr (free, Beta, open-source). Abstrackr comprises two components; a web-based annotation tool that allows participants in a review to collaboratively screen citations for relevance, and machine learning technologies that semi-automate the screening process. The machine learning technology permits reviewers to screen roughly half of the set of citations imported for a given review, and then let the software automatically screen the remaining citations.  
3. DistillerSR (requires subscription). Enables you to create forms for making screening decisions, and extract data.

Critical Appraisal tools:

Study Quality Assessment Tools developed in 2013 by the National Heart Lung & Blood Institute: https://www.nhlbi.nih.gov/health-topics/study-quality-assessment-tools  Choose an appraisal tool that matches the type of study you are reviewing from one of the following 6 study types: Controlled Intervention Studies, Systematic Reviews and Meta-Analyses, Observational Cohort and Cross-Sectional Studies, Case-Control Studies, Before-After (Pre-Post) Studies With No Control Group, & Case Series Studies.

Worksheets from the Oxford University Center for Evidence Based Medicine - choose a worksheet that matches the type of study:
Systematic Review article Critical Appraisal Sheet
Diagnosis study Critical Appraisal Sheet
Prognosis study Critical Appraisal Sheet
Therapy / Randomized Controlled Trial Critical Appraisal Sheet

Alternatively the CASP: Critical Appraisal Skills Checklists are eight critical appraisal tools designed to be used when reading and evaluating the quality of Systematic Reviews, Randomized Controlled Trials, Cohort Studies, Case Control Studies, Economic Evaluations, Diagnostic Studies, Qualitative studies and Clinical Prediction Rule.

Another alternative set of Critical Appraisal checklists are from the Joanna Briggs Institute (JBI). JBI require you use their critical appraisal checklists if you are conducing a JBI systematic review following the methods described in the JBI Manual for Evidence Synthesis.