Systematic and Literature Reviews

Steps in the Systematic Review Process

1. Identify your research question.  Formulate a clear, well-defined research question of appropriate scope. Define your terminology. Find existing reviews on your topic to inform the development of your research question, identify gaps, and confirm that you are not duplicating the efforts of previous reviews. Consider using a framework like PICO to define you question scope.
2. Define inclusion and exclusion criteria. Clearly state the criteria you will use to determine whether or not a study will be included in your search.  Consider study populations, study design, intervention types, comparison groups, measured outcomes.
3. Search for studies. Run your searches in the databases that you've identified as relevant to your topic. Work with a librarian to help you design comprehensive search strategies across a variety of databases. Approach the gray literature methodically and purposefully. Collect ALL of the retrieved records from each search into a reference manager, such as Endnote, and de-duplicate the library prior to screening.
4. Select studies for inclusion based on pre-defined criteria. Start with a title/abstract screening to remove studies that are clearly not related to your topic. Use your inclusion/exclusion criteria to screen the full-text of studies. It is highly recommended that two independent reviewer screen all studies, resolving areas of disagreement by consensus.
5. Extract data from included studies. Use a spreadsheet, or systematic review software, to extract all relevant data from each included study. It is recommended that you pilot your data extraction tool, to determine if other fields should be included or existing fields clarified.
6. Evaluate the risk of bias of included studies. Use a Risk of Bias tool (such as the Cochrane RoB Tool) to assess the potential biases of studies in regards to study design and other factors. You can adapt existing tools to best meet the needs of your review, depending on the types of studies included.
7. Present results and assess the quality of evidence. Clearly present your findings, including detailed methodology (such as search strategies used, selection criteria, etc.) such that your review can be easily updated in the future with new research findings. Perform a meta-analysis if the studies allow. Provide recommendations for practice and policy-making if sufficient, high quality evidence exists, or future directions for research to fill existing gaps in knowledge or to strengthen the body of evidence.

IOM Standards for Systematic Reviews: Standard 3.1: Conduct a comprehensive systematic search for evidence

The goal of systematic review searches is to identify all relevant studies on a topic. Therefore, systematic review searches are typically quite extensive. It is necessary, however, to strike a balance between striving for comprehensiveness and maintaining relevance when developing a search strategy. Increasing the comprehensiveness (or sensitivity) of a search will reduce its precision and will retrieve more non-relevant articles.

The goal of a systematic review search is to maximize recall and precision while keeping results manageable. Recall (sensitivity) is defined as the number of relevant reports identified divided by the total number of relevant reports in existence. Precision (specificity) is defined as the number of relevant reports identified divided by the total number of reports identified.  

Issues to consider when creating a systematic review search:   

• All concepts are included in the strategy
• ​All appropriate subject headings are used
• Appropriate use of explosion
• Appropriate use of subheadings and floating subheadings
• Use of natural language (text words) in addition to controlled vocabulary terms
• Use of appropriate synonyms, acronyms, etc.
• Truncation and spelling variation as appropriate
• Appropriate use of limits such as language, years, etc.
• Field searching, publication type, author, etc.
• Boolean operators used appropriately
• Line errors: when searches are combined using line numbers, be sure the numbers refer to the searches intended
• Check indexing of relevant articles
• Search strategy adapted as needed for multiple databases

TIP:  Look for systematic reviews that have already been published.

         This serves 2 purposes:

1. Helps ensure that the work has not already been done.
2. Provides examples of search strategies for your topic.