UTC RAVE Alert

Steps in an Evidence Synthesis

Steps in an Evidence Synthesis

Click each step to read a description of each evidence synthesis step in more detail, as well as provide resources and guidance.

0. Develop a Protocol

What is a Protocol?

An evidence synthesis protocol states your rationale, hypothesis, and planned methodology. Much like a blueprint for a house, a protocol outlines the planned framework for the evidence synthesis. Members of the team use the protocol as a guide to conduct the research. It is recommended that you register your protocol prior to conducting your review. This will improve transparency and reproducibility, reduce bias, and will also ensure that other research teams do not duplicate your efforts. A protocol template and checklist are included on this page, as well as a checklist for structured literature reviews that serves as a similar document to an evidence synthesis protocol.

  • Evidence Synthesis Protocol Template
  • Planning Worksheet for Structured Literature Reviews
  • PRISMA Checklist
  • PRISMA Extension for Scoping Reviews
  • PRISMA Extension for Network Meta-Analysis
1. Draft your Research Question

Developing a Research Question

Developing your research question is one of the most important steps in the evidence synthesis process. At this stage in the process, you and your team have identified a knowledge gap in your field and are aiming to answer a specific question:

  • If X is prescribed, then Y will happen to patients?

OR assess an intervention:

  • How does X affect Y?

OR synthesize the existing evidence:

  • What is the nature of X?

Whatever your aim, formulating a clear, well-defined research question of appropriate scope is key to a successful evidence synthesis. The research question will be the foundation of your synthesis and from it your research team will identify 2-5 possible search concepts. These search concepts will later be used in step 5 to build your search strategy.

Research Question Frameworks

Formulating a research question takes time and your team may go through different versions until settling on the right research question. To help formulate your research question, some research question frameworks are listed below (there are dozen of different types of these frameworks--for a comprehensive overview, see this guide from the University of Maryland: https://guides.library.cornell.edu/evidence-synthesis/databases)

Think of these frameworks as you would for a house or building. A framework is there to provide support and to be a scaffold for the rest of the structure. In the same way, a research question framework can also help structure your evidence synthesis question. Probably the most common framework is PICO:

PICO for Quantitative Studies

  • P - Population/Problem
  • I - Intervention/Exposure
  • C - Comparison
  • O - Outcome

Example: Is gabapentin (intervention), compared to placebo (comparison), effective in decreasing pain symptoms (outcome) in middle aged male amputees suffering phantom limb pain (population)?

While PICO is a helpful framework for clinical research questions, it may not be the best choice for other types of research questions, especially outside the health sciences. Here are a few others (for a comprehensive, but concise, overview of the almost 40 different types of research question frameworks, see this review from the British Medical Journal: Rapid review of existing question formulation frameworks)

PICO for Qualitative Studies

  • P - Population/Problem
  • I - Phenomenon of Interest
  • Co - Context

Example: What are the experiences (phenomenon of interest) of caregivers providing home based care to patients with Alzheimer's disease (population) in Australia (context)?

SPICE

  • S - Setting
  • P - Perspective (for whom)
  • I - Intervention/Exposure
  • C - Comparison
  • E - Evaluation

Example: What are the benefits (evaluation) of a doula (intervention) for low income mothers (perspective) in the developed world (setting) compared to no support (comparison)?

SPIDER

  • S - Sample
  • PI - Phenomenon of Interest
  • D - Design
  • E - Evaluation
  • R - Study Type

Example: What are the experiences (evaluation) of women (sample) undergoing IVF treatment (phenomenon of interest) as assessed?

Design: questionnaire or survey or interview

Study Type: qualitative or mixed method

Inclusion and Exclusion Criteria

Inclusion and exclusion criteria are developed after a research question is finalized but before a search is carried out. They determine the limits for the evidence synthesis and are typically reported in the methods section of the publication. For unfamiliar or unclear concepts, a definition may be necessary to adequately describe the criterion for readers.

2. Select Grey Literature Sources

What is Grey Literature?

Grey (or gray) literature is literature produced by individuals or organizations outside of commercial and/or academic publishers. This can include information such as government reports, conference proceedings, graduate dissertations, unpublished clinical trials, and much more. The sources you select will be informed by your research question and field of study, but should likely include, at a minimum, theses and dissertations.

Why Search the Grey Literature?

The intent of an evidence synthesis is to synthesize all available evidence that is applicable to your research question. There is a strong bias in scientific publishing toward publishing studies that show some sort of significant effect. Meanwhile, many studies and trials that show no effect end up going unpublished. But knowing that an intervention had no effect is just as important as knowing that it did have an effect when it comes to making decisions for practice and policy-making. While not peer-reviewed, grey literature represents a valuable body of information that is critical to consider when synthesizing and evaluating all available evidence.

How Do I Search the Grey Literature?

Finding grey literature and searching it systematically is challenging. But there are a few approaches that you can take to add some structure to your search of this type of information:

  • Refer to grey literature sources used for related evidence syntheses. Refer to both published evidence syntheses and registered protocols.
  • Ask experts in the field for relevant grey literature sources. If you are an expert, include important grey literature sources, and ask colleagues for their recommendations.
  • Search databases that specialize in grey literature: See the "Grey Literature Sources" box at the bottom of this page for more information.
  • Search for theses and dissertations: There are a number of databases dedicated to theses and dissertations, which you can search using your search terms. See the "Grey Literature Sources" box at the bottom of this page for links to these resources.
  • Search clinical trials: There may be clinical trials being conducted that are relevant to your research question, but that haven't been published yet or never were published. See the "Grey Literature Sources" box at the bottom of this page for links to these resources.
  • Identify government agencies and international and non-governmental organizations that might publish technical papers and reports on your topic. Search their websites or any online libraries that they may provide. See the "Grey Literature Sources" box at the bottom of this page for links to some examples.
  • Search conference proceedings and newsletters: Identify professional organizations that have and/or conferences at which researchers might be presenting work related to your topic. Search those conference proceedings or newsletters on the organization's website or by contacting organizational boards for access to past proceedings that may not be online. See the "Grey Literature Sources" box at the bottom of this page for some examples.
  • Contact known researchers in the field to determine if there are any ongoing or unpublished studies that s/he may be aware of.
  • Search professional and trade magazines. Professional magazines contain literature that is written by professionals in the field for other professionals in the field, but that may not be about research. Trade magazines contain advertisements and news very specific to a topic or industry.

How Do I Manage the Grey Literature Search?

  • Identify and record the sources you will search. The sources you search will be informed by your research question and where you expect to find information related to your question.
  • Document where you are searching and your search strategies, including document resource name, URL, search terms, and date searched.
  • Collect citation information as you go.
  • Adhere to your established inclusion and exclusion criteria when selecting sources.

See below for guidance documents specific to grey literature searching.

Grey Literature Resources

Grey Literature Databases:

Theses and Dissertations:

Clinical Trials:

Public Policy:

Conference Proceedings:

This list of conference proceedings is not comprehensive, but are ideas of where to search. We recommend that you search proceedings for the three most relevant conferences for your evidence synthesis.

Preprint Repositories:

Other Resources:

3. Write a Search Strategy

Writing a Search Strategy

It is recommended that you work with a librarian to help you design comprehensive search strategies across a variety of databases. Writing a successful search strategy takes an intimate knowledge of bibliographic databases.

Using Boolean logic is an important component of writing a search strategy:

  • "AND" narrows the search, e.g. children AND exercise
  • "OR" broadens the search, e.g. (children OR adolescents) AND (exercise OR diet)
  • "NOT" excludes terms, e.g. exercise NOT diet
  • "*" at the root of a word finds all forms of that word, e.g. (child* OR adolescen*) AND (exercise* OR diet*)
  • parentheses ensure all terms will be searched together as a set quotations around a phrase searches that exact phrase, e.g. (child* OR adolescen* OR "young adult*")

Evidence Synthesis Search Strategy Examples

Agriculture Example:

Nutrition Example:

Sources and more information:

Other Methodology Search Filters:
4. Register a Protocol

Where to Register Protocols

Open Science Framework

An open source, multidisciplinary web application that connects and supports the research workflow. Researchers use the OSF to collaborate, document, archive, share, and register research projects, materials, and data. OSF can be used to pre-register a systematic review protocol and to share documents such as a Zotero library, search strategies, and data extraction forms.

Unlike other registries, evidence synthesis author teams do not submit their protocols for review by an editorial board before they are accepted and pre-registered on OSF. Instead, create your own pre-registration (instructions here).

Example of a pre-registered scoping review protocol on OSF (click "registrations" to view registration info)

5. Translate Search Strategies

Translating Search Strategies Background

Evidence synthesis methods require authors to search multiple databases, and not all databases accept the same search "syntax." Each individual database requires use of specialized search syntax, and therefore evidence synthesis search strategies must be 'translated' between databases.

For example, a search for vitamin D[tiab] in PubMed will show you all citations with the phrase "vitamin D" in the title, abstract, or keywords, but a search for vitamin D[tiab] in Web of Science will not work at all.

Below is a template that you can use to document your search strategy translations and results, as well as search syntax translation tools and examples. Contact a librarian for assistance with search syntax translation.

 

Search Syntax Translation Resources

Grey Literature and Regional Database Search Strategies

Search strategies for grey literature and regional databases often have to be distilled significantly from the main search strategy. This is due to the fact that many grey literature and regional databases cannot handle complex search strategies and special syntax. In addition, searching grey literature and regional databases with all the terms from the main search strategy sometimes returns far too many results to screen. The recommended method of simplifying a search strategy is to combine a few of the most important terms from each key concept of your research question. See an example below.

Research Question: What is the effectiveness of Vitamin B12 supplements in reducing morbidity in pregnant women with HIV infection?

Key Concept 1 distilled terms: B12, B 12, cobalamin

Key Concept 2 distilled terms: pregnancy/pregnant, gestate/gestation/gestational

Key Concept 3 distilled terms: ​HIV, human immunodeficiency virus

Distilled Search Strategy: (B12 OR "B 12" OR cobalamin) AND (pregnan* OR gestat*) AND (HIV OR "human immunodeficiency virus")

6. Citation Management

Citation Management

A citation management program will save you a lot of time when doing your evidence synthesis. Programs like Endnote, Zotero or Mendeley will store and organize the citations collected during your screening, deduplicate the results and automatically format in-text citations and bibliographies in your manuscript.

You may download Zotero and Mendeley for free, while Endnote is freely available to the UTC Campus and can be downloaded here (https://www.utc.edu/library/help/endnote).

Citation Management Tips

Once you've finished translating your search strategy to the syntax of each database you're searching, you will then:

1. Run the search in each database.

2. Export the results as a BibTeX, MEDLINE, RIS, or XML file and save those files.

3. Import those files into a citation management program.

Deduplication

You will likely retrieve multiple versions of the same study as you search many databases and will need to deduplicate your results before article screening. After you've performed your searches and imported the results into your citation management software:

  • In Endnote, from the menu select References -> "Find Duplicates". Select which duplicate record to keep by selecting "Keep This Record".
  • In Zotero, click on the "Duplicate Items" collection in your library. You can resolve duplicates by merging the files.
  • In Mendeley, select your folder of interest. Go to your Tools menu and select "Check for Duplicates". Select the details that you would like to keep from each of the documents. Click merge to create one entry containing the complete document details.
  • Covidence also automatically deduplicates your results. Click here for more information about Covidence deduplication.
7. Article Screening

Article Screening

The purpose of article screening to remove studies that are clearly not related to your topic. Use your inclusion/exclusion criteria to first screen the title and abstracts of your studies and determine whether they are relevant to your research question. Once titles and abstracts have been screened, the full text must be retrieved and screened to definitely decide whether the study fits the eligibility criteria of your synthesis.

It is highly recommended that two independent reviewers screen all studies, resolving areas of disagreement by consensus or by a third party who is an expert in the field. Listed below are tools that can be used for article screening.

Tools for Managing Evidence Syntheses

Covidence:

Covidence is an online systematic review management tool that allows for independent title/abstract screening, full text screening, data extraction and risk of bias assessment. Although it is designed for use with systematic reviews, its utility translates fairly well to some other evidence synthesis methods, like scoping reviews.

Covidence is available for purchase for individual research projects at https://www.covidence.org/

Rayyan:

Rayyan is a free online tool that can be used for independent screening and coding of studies in an evidence synthesis. Rayyan will pre-populate inclusion and exclusion criteria, but you can customize these criteria. It also uses tagging and filtering to code and organize references. Title and abstract screening can be conducted in one project, while full text screening can be conducted in a second project.

Access Rayyan here.

Excel:

Excel is the most basic tool for the management of article screening. Lists of references can be exported from citation managers into Excel format for screening. A more advanced approach to using Excel for this purpose is the PIECES approach, designed by a librarian at Texas A&M. The PIECES workbook is downloadable at this guide.

Because it is easy to accidentally overwrite cells in Excel, it is recommended that you keep a clean copy of your exported articles and work only on a copy.

8. Risk of Bias Assessment

Risk of Bias Assessment

Risk of bias assessment (sometimes called "quality assessment" or "critical appraisal") helps to establish transparency of evidence synthesis results and findings. A risk of bias assessment is often performed for each included study in your review. Evidence syntheses strive to eliminate bias in their findings. Individual studies that are included in a synthesis may include biases in their results or conclusions, for example design flaws that raise questions about validity of findings or an overestimate of intervention effect. Risk of bias assessment generally is not required with evidence synthesis methods outside of systematic reviews. However, this may depend on the evidence synthesis method that you are utilizing.

Risk of Bias Assessment in a Discipline Outside of Human Medicine

This Cochrane Training presentation helps to navigate the steps of Risk of Bias Assessment, but some things might not apply to other disciplines. For disciplines outside of human medicine, the Critical Appraisal Skills Programme provides checklists that can be applied to a diverse array of study types.

Risk of Bias Assessment in Scoping Reviews

Scoping reviews don't typically include a risk of bias assessment

“A key difference between scoping reviews and systematic reviews is that the former are generally conducted to provide an overview of the existing evidence regardless of methodological quality or risk of bias (4, 5). Therefore, the included sources of evidence are typically not critically appraised for scoping reviews.”

From Tricco et al., PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation. See the full article for more methodology guidelines specific to scoping reviews.

Representing your Risk of Bias Assessment

Risk of bias assessments can be represented in table format in an evidence synthesis, showing each included study and how strong it is across several quality criteria for that particular study type for example:

From Liu, Zhenlei; Tao, Xixi; Chen, Yuexin; Fan, Zhongjie; Li, Yongjun (2015): Summary of risk of bias for RCTs (RevMan 5.3).. PLOS ONE. Figure. https://doi.org/10.1371/journal.pone.0121388.g002

If a high proportion of studies are assessed with a high risk of bias, caution should be used when interpreting results for your evidence synthesis. More information can be found in Chapter 8 of the Cochrane Handbook.

 
9. Data Extraction

Data Extraction

Whether you plan to perform a meta-analysis or not, you will need to establish a regimented approach to extracting data. Researchers often use a form or table to capture the data they will then summarize or analyze. The amount and types of data you collect, as well as the number of collaborators who will be extracting it, will dictate which extraction tools are best for your project. Programs like Excel or Google Spreadsheets may be the best option for smaller or more straightforward projects, while systematic review software platforms can provide more robust support for larger or more complicated data.

It is recommended that you pilot your data extraction tool, especially if you will code your data, to determine if fields should be added or clarified, or if the review team needs guidance in collecting and coding data.

Data Extraction Tools

Excel

Excel is the most basic tool for the management of the screening and data extraction stages of the systematic review process. Customized workbooks and spreadsheets can be designed for the review process. A more advanced approach to using Excel for this purpose is the PIECES approach, designed by a librarian at Texas A&M. The PIECES workbook is downloadable at this guide.

Covidence

Covidence is a software platform built specifically for managing each step of a systematic review project, including data extraction. Read more about how Covidence can help you customize extraction tables and export your extracted data.

RevMan

RevMan is free software used to manage Cochrane reviews. For more information on RevMan, including an explanation of how it may be used to extract and analyze data, watch Introduction to RevMan - a guided tour.

DistillerSR

DistillerSR is a systematic review management software program, similar to Covidence. It guides reviewers in creating project-specific forms, extracting, and analyzing data.

Sumari

JBI Sumari (the Joanna Briggs Institute System for the United Management, Assessment and Review of Information) is a systematic review software platform geared toward fields such as health, social sciences, and humanities. Among the other steps of a review project, it facilitates data extraction and data synthesis. View their short introductions to data extraction and analysis for more information.

The Systematic Review Toolbox

The SR Toolbox is a community-driven, searchable, web-based catalogue of tools that support the systematic review process across multiple domains. Use the advanced search option to restrict to tools specific to data extraction.

Additional Information

These resources offer additional information and examples of data extraction forms:​

Brown, S. A., Upchurch, S. L., & Acton, G. J. (2003). A framework for developing a coding scheme for meta-analysis. Western Journal of Nursing Research, 25(2), 205–222. https://doi.org/10.1177/0193945902250038

Elamin, M. B., Flynn, D. N., Bassler, D., Briel, M., Alonso-Coello, P., Karanicolas, P. J., … Montori, V. M. (2009). Choice of data extraction tools for systematic reviews depends on resources and review complexity. Journal of Clinical Epidemiology, 62(5), 506–510. https://doi.org/10.1016/j.jclinepi.2008.10.016

Higgins, J.P.T., & Deeks, J.J. (Eds.) (2011). Chapter 7: Selecting studies and collecting data. In J.P.T.Higgins, & S. Green (Eds.), Cochrane handbook for systematic reviews of interventions Version 5.1.0 (updated March 2011). The Cochrane Collaboration. Available from www.handbook.cochrane.org.

Research guide from the George Washington University Himmelfarb Health Sciences Library: https://guides.himmelfarb.gwu.edu/c.php?g=27797&p=170447

10. Synthesize, Map, or Describe the Results

Synthesize, Map, or Describe the Results

In the data synthesis section, you need to present the main findings of your evidence synthesis. As an evidence synthesis summarizes existing research, there are a number of ways in which you can synthesize results from your included studies.

If the studies you have included in your evidence synthesis are sufficiently similar, or in other words homogenous, you can synthesize the data from these studies using a process called “meta-analysis”. As the name suggests, a meta-analysis uses a statistical approach to bring together results from multiple studies. There are many advantages to undertaking a meta-analysis.

If the studies you have included in your evidence synthesis are not similar (e.g. you have included different research designs due to diversity in the evidence base), then a meta-analysis is not possible. In this instance, you can synthesize the data from these studies using a process called “narrative or descriptive synthesis”.

A word of caution here – while the process underpinning meta-analysis is well established and standardized, the process underpinning narrative or descriptive synthesis is subjective and there is no one standard process for undertaking this.

In recent times, evidence syntheses of qualitative research is gaining popularity. Data synthesis in these studies may be termed as “meta-synthesis”. As with narrative or descriptive synthesis, there are a myriad of approaches to meta-synthesis.

(the above content courtesy of University of South Australia Library)

Regardless of whether an evidence synthesis presents qualitative or quantitative information, reporting out using the PRISMA flow diagram is recommended. The PRISMA website and its many adaptations can be very helpful in understanding components of systematic reviews, meta-analyses and related evidence synthesis methods

Librarians can help write the methods section of your review for publication, to ensure clarity and transparency of the search process. However, we encourage evidence synthesis teams to engage statisticians to carry out their data syntheses.

Library

Library