Systematic Reviews: Study selection and appraisal

Inclusion/Exclusion criteria

See http://unimelb.libguides.com/c.php?g=492361&p=3368110

First level screening - title and abstract review

At the initial screening stage read just the title and abstract of the candidate studies and make a decision to include or exclude the study from your review.  

For small reviews of a few studies (e.g. <100)

The research team should agree on the inclusion and exclusion criteria for studies you wish to review and put together a study screening form.  To help identify your inclusion/exclusion criteria, revisit the PICOS of interest you came up with for your search strategy and gain agreement/approval from your colleagues or supervisor. The screening form may look similar to Table 3 of Brown et al (2013). You may write down your decision to include or exclude an article on an Excel spreadsheet like this one, or if you have a small number of records you may choose to print out one copy for each record, although printing will be impractical for larger numbers of records.  Screen each potentially useful article identified in the literature search as follows:

1. Read the title and abstract (where available) and apply the inclusion/exclusion criteria from the screening form.
2. Make a decision on whether or not to include the study in the review.
3. Record the decision and reasons for inclusion/exclusion on the study screening form or spreadsheet. You will summarise the reasons for exclusion on the PRISMA flow diagram - see Study Selection PRISMA item # 17 below.

For large reviews of many studies (e.g. >100) - in case you need to partially automate the screening process

There are three web-based software applications that can help with screening and tracking your selection decisions:

1. Covidence (GW in 2019 bought a subscription so you can use this tool now). Provides a decision dashboard and annotation tool, and the ability to screen candidate citations you locate in your literature search. Covidence is used by Cochrane review teams as their first level screening tool, the resulting study characteristics and decision data can be exported to RevMan (free for academic use) or Excel. 
2. Abstrackr (free, Beta, open-source). Abstrackr comprises two components; a web-based annotation tool that allows participants in a review to collaboratively screen citations for relevance, and machine learning technologies that semi-automate the screening process. The web-based annotation tool allows project leads to import the citations that are to be screened for a review from either RefMan or Pubmed. Participants can then join the project and begin screening; the tool maintains a digital paper trail of all screening decisions.  The machine learning technology permits reviewers to screen roughly half of the set of citations imported for a given review, and then let the software automatically exclude a (hopefully large) portion of the remaining citations; the reviewers will then only need to screen the articles classified as relevant by the software.  A recent article evaluating the use of Abstrackr in the systematic review process is Rathbone, J., Hoffmann, T., & Glasziou, P. (2015). Faster title and abstract screening? Evaluating Abstrackr, a semi-automated online screening program for systematic reviewers. Systematic Reviews, 480. doi:10.1186/s13643-015-0067-6
3. DistillerSR (requires subscription). Enables you to create forms for making screening decisions, and extract data.

Second level screening - full text review

Having excluded candidate studies that did not meet your inclusion/exclusion criteria you should have a smaller number of potentially relevant studies. GW affiliates at GW and Children's National Health System can use Box to store and share the full text PDF's of copyrighted journal articles https://it.gwu.edu/backup-storage-document-management. Read and critically appraise the full text of each study you selected at the first pass screening stage to determine whether you wish to include them in your discussion and analysis. Specifically each study must be evaluated based on the following criteria:

Does this study address a clearly focused question?
Did the study use valid methods to address this question?
Are the valid results of this study important?
Are these valid, important results applicable to my patient or population?

If the answer to any of these questions is “no”, you may wish to read no further and exclude the study, or you may decide to include the study to inform your discussion but not include the results in your analysis. 

To help with this process you may wish to download and apply one of the following Critical Appraisal tools:

Study Quality Assessment Tools developed in 2013 by the National Heart Lung & Blood Institute: https://www.nhlbi.nih.gov/health-topics/study-quality-assessment-tools  Choose an appraisal tool that matches the type of study you are reviewing from one of the following 6 study types: Controlled Intervention Studies, Systematic Reviews and Meta-Analyses, Observational Cohort and Cross-Sectional Studies, Case-Control Studies, Before-After (Pre-Post) Studies With No Control Group, & Case Series Studies.

Worksheets from the Oxford University Center for Evidence Based Medicine - choose a worksheet that matches the type of study:
Systematic Review article Critical Appraisal Sheet
Diagnosis study Critical Appraisal Sheet
Prognosis study Critical Appraisal Sheet
Therapy / Randomized Controlled Trial Critical Appraisal Sheet

Alternatively the CASP: Critical Appraisal Skills Checklists are eight critical appraisal tools designed to be used when reading and evaluating the quality of Systematic Reviews, Randomised Controlled Trials, Cohort Studies, Case Control Studies, Economic Evaluations, Diagnostic Studies, Qualitative studies and Clinical Prediction Rule.

Another alternative set of Critical Appraisal checklists are from the Joanna Briggs Institute.

Make a decision on whether or not to include the study in your review, and write your decision and reasons for inclusion/exclusion at this second level/full text review stage on the study screening form. You will summarise the reasons for exclusion on the PRISMA flow diagram - see Study Selection PRISMA item # 17 below.

Reporting your screening decisions

In the final report in the methods section the PRISMA checklist Item 9 study selection will be reported as:

• How studies were screened e.g. by reading title & abstract, and how they were critically appraised e.g. by applying a standardised appraisal form appropriate for that study type - see above.
• What sort of studies were excluded e.g. letters, conference abstracts, etc.
• Who reviewed/appraised the studies
• What the process was for resolving disagreements e.g. reporting the level of inter-rater agreement, how often arbitration about selection was required, & what efforts were made to resolve disagreement e.g. were original authors contacted

Researchers must keep the screening forms to create a summary descriptive flow diagram of study selection.

In the final report in the results section the PRISMA checklist Item 17 study selection should be reported as follows:

• Record the number of studies screened, assessed for eligibility and included in the review, with reasons for exclusions both in the text and in form of a PRISMA flow diagram of study selection e.g. similar to Fig 2 of Liberati et al. (2009). Covidence keeps track of your screening decisions and generates a PRISMA flow diagram for you, GW affiliates can register for a Covidence account here. Alternatively there is a PRISMA flow diagram generator at http://www.prisma-statement.org/PRISMAStatement/FlowDiagram