Skip to Main Content

Systematic Reviews

Learn about conducting systematic reviews

Selecting & Appraising Studies

As you are conducting your search, results are usually imported to a citation management tool or a spreadsheet to manage and document your results. After de-duplicating results, the remaining articles are selected for inclusion though a two-step process. To avoid bias, two investigators independently vote and review each article. 

  1. The first step is selection based on title and abstract review. Often the large majority of search findings is rejected at this step.
  2. The second step looks at the full text of  the remaining articles to create a final set of articles to be included in review.

Selecting and appraising studies is a time-consuming aspect of conducting a systematic review. You may choose to utilize a software or tool to facilitate this process. See the Tools & Software tab. 

Chapter 4 Section 6 of the Cochrane Handbook for Systematic Reviews of Interventions provides detailed guidance for study selection. 

Study Selection: Step by Step

Below is a typical process for selecting studies for inclusion, according to  the Cochrane Handbook. Your selection process should be documented in your review protocol.

  1. Merge search results from different sources using reference management software, and remove duplicate records of the same report (i.e. records reporting the same journal title, volume and pages).
  2. Examine titles and abstracts to remove obviously irrelevant reports (authors should generally be over-inclusive at this stage).
  3. Retrieve the full text of the potentially relevant reports.
  4. Link together multiple reports of the same study (see Section 4.6.2).
  5. Examine full-text reports for compliance of studies with eligibility criteria.
  6. Correspond with investigators, where appropriate, to clarify study eligibility (it may be appropriate to request further information, such as missing methods information or results, at the same time). If studies remain incomplete/unobtainable they should be tagged/recorded as incomplete, and should be listed in the table of ‘Studies awaiting assessment’ in the review.
  7. Make final decisions on study inclusion and proceed to data collection.
  8. Tag or record any ongoing trials which have not yet been reported so that they can be added to the ongoing studies table.

Note that studies should not be omitted from a review solely on the basis of measured outcome data not being reported (see MECIR Box 4.6.b and Chapter 13).

From: Cochrane Handbook for Systematic Reviews of Interventions

Critical Appraisal Tools

From the Center for Evidence-Based Medicine (CEBM) from University of Oxford:

Critical appraisal is the systematic evaluation of clinical research papers in order to establish:

  1. Does this study address a clearly focused question?
  2. Did the study use valid methods to address this question?
  3. Are the valid results of this study important?
  4. Are these valid, important results applicable to my patient or population?

If the answer to any of these questions is “no”, you can save yourself the trouble of reading the rest of it.

CEBM Critical Appraisal Worksheets:

Additional Critical Appraisal Tools:

Defining Eligibility and Reducing Bias

Cochrane Handbook for Systematic Reviews of Interventions 

Campbell Collaboration: Meta-Evidence Blog

Catalog of Bias (CEBM & Oxford University)

Office of Health Assessment and Translation (OHAT) Division of the National Toxicology Program

Reporting Your Selection Process & Results

The PRISMA checklist provides reporting guidelines for documenting the study selection process.

PRISMA checklist, item 9:

  • State the process for selecting studies (i.e., screening, eligibility, included in systematic review, and, if applicable, included in the meta-analysis). 

PRISMA checklist item 12:

  • Describe methods used for assessing risk of bias of individual studies (including specification of whether this was done at the study or outcome level), and how this information is to be used in any data synthesis. 

PRISMA checklist item 17:

  • Give numbers of studies screened, assessed for eligibility, and included in the review, with reasons for exclusions at each stage, ideally with a flow diagram.