Evidence Synthesis Methodologies in the Health Sciences

A typical systematic search includes searches of:

• Major databases in your discipline.
• Important databases from related disciplines.
• A large interdisciplinary database

At minimum, you should include three databases. In the health sciences, the most common combination disciplinary databases are MEDLINE (searched via PubMed or Ovid MEDLINE) and Embase. For an interdisciplinary databases, searchers generally choose Web of Science or Scopus. The tabs in this box provide links to these and other recommended databases.

Databases have different interfaces, syntaxes, and controlled vocabulary. Your core search will need to be translated, or adapted, to the other databases you choose to search.

Grey literature is frequently defined as:

"Information produced on all levels of government, academia, business and industry in electronic and print formats not controlled by commercial publishing i.e. where publishing is not the primary activity of the producing body." (ICGL Luxembourg definition, 1997, expanded in New York, 2004)

Why include grey literature in evidence syntheses?

• Thoroughness: Evidence synthesis aims to include all information relevant to a research question. Not all information will appear in traditionally published literature sources. For example, research shared at conferences is often never formally published.
• Publication bias: Published literature tends to focus on successful and more established research. Things like clinical trials with no effect or negative results, emerging areas of research, and supplementary data from published materials may be located in grey literature rather that more traditional sources. Bias may also lead to a lack of underrepresented, marginalized, or non-Western voices in published literature.
• Methodological guidelines emphasize it: For example, systematic reviews performed using Cochrane or Campbell Collaboration methods are strongly encouraged or expected to include searches of grey literature. Peer reviewers or journal editors may also be looking for its inclusion.

What kinds of materials count as grey literature?

Common grey literature sources include, but are not limited to: trial registrations and trial data, government documents, regulatory information, guidelines, technical reports, white papers, annual reports, preprints, patents, unpublished studies, theses and dissertations, conference materials, data sets and statistics, NGO reports, and fact sheets. Web content such as blog and social media posts, slide decks, videos, and organizational or government websites would also fall under this umbrella.

How do I find grey literature?

Locating grey literature can be challenging. One highly effective method is to consult experts in the subject matter, including team members. Also consider what groups or organizations are likely to have studied or written about your topic.

The tabs in this box provide resources for finding grey literature in a number of different categories. Neither the resources nor the categories are exhaustive.

There's so much information out there. How do I decide what to search and when to stop?

There's no way to be sure you've found everything, and balancing sensitivity and specificity in grey literature searching is less clear than in searching databases. Consider what types of grey literature may be relevant, and what sources you consider reliable.

General recommendations include:

• Decide on your search plan in advance, including types of grey literature, possible sources, and search terms.
• Search or browse multiple sources (i.e., don't just do a Google search).
• Adapt your search strategies to the functionality of the sources, which may be quite limited.
• Exploratory searching may be useful, but final searches that may or may not yield additional sources for screening should be fully documented.

The SuRe (Summarized Research in Information Retrieval for HTA) chapter on locating grey literature provides excellent guidance for planning your strategy.

How do I report grey literature searches?

Be sure to document:

• Where you searched, including resource name and, if necessary, URL.
• The search strategy and date searched.
• The number of results.

If using the PRISMA reporting standards, grey literature numbers belong in the right-hand column under "Identification of studies via other methods."

Citation searching is a recommended practice in evidence synthesis. The recently published TARCiS Statement (Terminology, Application, and Reporting of Citation Searching) provides excellent guidance on this process.

Important definitions for citation searching (described in Recommendation 1):

• Citation searching is an umbrella term that entails various methods of citation based literature retrieval
• Backward citation searching is retrieving and screening cited references
• Forward citation searching is retrieving and screening citing references

The TARCiS Statement includes a number of recommendations. A summary of some essential points:

• Typical references to subject to citation searching include seed articles, included studies after full text screening, and highly relevant review articles.
• Backward citation searching can be used to test a search strategy's sensitivity.
• Backward and forward citation searching on all included studies is a valuable supplementary search technique; if the resulting set is too large to screen, it might be best to select a sample set of included studies for this process.
• Citation searching methods must be reported in full in the methods section; the authors recommend both PRISMA-S (see the Document Your Searches page of this guide) as well as the TARCiS Statement checklist.