Systematic Reviews: Study selection and appraisal

1. First level screening: title and abstract review

At the initial screening stage, read just the title and abstract of the studies, and make a decision to include or exclude the study from your review. Record the decision and reasons for inclusion/exclusion.

2. Second level screening: full text review

Read and critically appraise the full text of each study you selected at the first pass screening stage to determine whether you wish to include them in your discussion and analysis. Evaluate each study based on the following criteria:

Does this study address a clearly focused question?

Did the study use valid methods to address this question?

Are the valid results of this study important?

Are these valid, important results applicable to my patient or population?

Does this study meet my inclusion/exclusion criteria?

If the answer to any of these questions is “no”, you may wish to read no further and exclude the study, or you may decide to include the study to inform your discussion but not include the results in your analysis. 

Make a decision on whether or not to include the study in your review, and write your decision and reasons for inclusion/exclusion at this second level/full text review stage on the study screening form. You will summarize the reasons for exclusion on the PRISMA flow diagram.

3. Reporting your screening decisions

In the final report in the methods section the PRISMA checklist Item 9 study selection will be reported as:

• How studies were screened e.g. by reading title & abstract, and how they were critically appraised e.g. by applying a standardized appraisal form appropriate for that study type.
• What sort of studies were excluded e.g. letters, conference abstracts, etc.
• Who reviewed/appraised the studies
• What the process was for resolving disagreements e.g. reporting the level of inter-rater agreement, how often arbitration about selection was required, & what efforts were made to resolve disagreement e.g. were original authors contacted

The PRISMA Flow Diagram maps out the screening process by showing the number of records identified, included and excluded, and the reasons for exclusions. Covidence can keep track of your screening decisions and generate a PRISMA flow diagram for you. 

PRISMA checklist Item 16 states that study selection should be reported as follows:

• "Describe the results of the search and selection process, from the number of records identified in the search to the number of studies included in the review, ideally using a flow diagram"
• "Cite studies that might appear to meet the inclusion criteria, but which were excluded, and explain why they were excluded"

Courtesy of Page, M. J., McKenzie, J. E., Bossuyt, P. M., et al. (2021). The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ (Clinical research ed.), 372, n71. https://doi.org/10.1136/bmj.n71

The research team should agree on the inclusion and exclusion criteria for studies you wish to review and put together a study screening form. To help identify your inclusion/exclusion criteria, revisit the PICOS of interest you came up with for your search strategy. 

Above image courtesy of the University of Melbourne: https://unimelb.libguides.com/c.php?g=492361&p=3368110

To help with the screening process:

Critical appraisal tools: