Reviews of published scientific literature are a valuable resource that can underline best practices in medicine and clarify clinical controversies. Among the various types of reviews, the systematic review of the literature is ranked as the most rigorous since it is a high-level summary of existing evidence focused on answering a precise question. Systematic reviews employ a pre-defined protocol to identify relevant and trustworthy literature. Such reviews can accomplish several critical goals that are not easily achievable with typical empirical studies by allowing identification and discussion of best evidence, contradictory findings, and gaps in the literature. The Association of University Radiologists Radiology Research Alliance Systematic Review Task Force convened to explore the methodology and practical considerations involved in performing a systematic review. This article provides a detailed and practical guide for performing a systematic review and discusses its applications in radiology.
Introduction
Review of the existing scientific literature is a valuable resource for generating best practices in medicine and clarifying clinical controversies with greater study power than an individual study. Several terms are commonly used to describe these types of post hoc review articles, including systematic review, literature review, overview, scoping review, meta-analysis, practice guidelines, and many others. Although there are some similarities between these types of reviews, there are substantial differences in the scope, goals, and methodology of each. For example, “literature review” is a generic term commonly used for reviews that qualitatively discuss a topic, often in an informal and unstructured manner that is subject to author bias . In contrast, “meta-analysis” is a method of statistically combining findings from multiple quantitative studies to increase statistical power and validity .
A “systematic review” is a high-level summary of existing evidence focused on answering a precise question . To be considered truly “systematic,” a review must ask a specific research question and apply an explicit and thorough methodology to comprehensively review all the available information . The use of a pre-defined protocol to identify relevant and trustworthy literature makes the systematic review a research study in itself. This methodology reduces author bias and allows identification and discussion of best evidence, contradictory findings, and gaps in the literature. Systematic reviews commonly underpin clinical practice guidelines, which aim to incorporate this scientific evidence with non-empirical information, such as risks and costs. These reviews may also result in the proposal of new theories and development of future research directions . For these reasons, systematic reviews are often placed highly on the hierarchy of evidence. If sufficient high-quality data are obtained during a systematic review, they may be pooled using a meta-analysis.
In this article, we will explore the methodology and practical considerations involved in performing a systematic review. Following a brief overview of systematic reviews, the first section will address the preparatory steps involved in creating a systematic review. The second section will describe how to conduct the literature search using a pre-defined search protocol. The third section will address the data analysis and quality assurance steps that take place after the literature search. The article will then conclude with a discussion of the role of systematic reviews in radiology. For convenience, a glossary of commonly used methodology terms related to systematic reviews has been included in Appendix 1 .
OVERVIEW OF SYSTEMATIC REVIEW
The most widely used methodology guidelines for systematic reviews are published by the Cochrane Collaboration. The Cochrane Collaboration is an independent non-profit organization with over 37,000 contributors from more than 130 countries that organizes medical research findings by developing systematic reviews of healthcare interventions and diagnostic tests. These “Cochrane Reviews” are published in the Cochrane Library and periodically updated with the latest research findings. The Cochrane Collaboration also publishes the Cochrane Handbook , which serves not only as a guide for authors writing a Cochrane Review but also as a widely accepted guide for writing any systematic review. The Cochrane Collaboration divides the general methods for conducting systematic reviews into the following steps: define the question, develop inclusion criteria, search for studies, select studies and collect data, assess risk of bias in included studies, analyze data, and report results . Combining this framework with a similar framework from Gough, et al , we outline the steps to performing a systematic review as follows (see Figure 1 ).
SECTION 1: PREPARATORY STEPS
Developing the Research Question
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Using the PICOS model, the research question commonly takes the form of, “What is the effect of (intervention/exposure) on (population) relative to (control) as measured by (outcomes), according to the data reported in all (study design) studies?”
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Evaluating the Quality of the Research Question
Get Radiology Tree app to read full this article<
Establishing Inclusion Criteria
Get Radiology Tree app to read full this article<
Developing a Study Protocol
Get Radiology Tree app to read full this article<
Registering the Review
Get Radiology Tree app to read full this article<
SECTION 2: LITERATURE SEARCH
Selecting Databases
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Conducting the Study Search
Get Radiology Tree app to read full this article<
(“multiple sclerosis” OR “Multiple Sclerosis” OR “MS” OR “Demyelinating disease” OR “demyelination” OR “demyelinating disorder” OR “optic neuritis with demyelination”)
However, such search strings are often not adequate for diagnostic test accuracy studies, since the methods may not be reported in the title or abstract and indexing terms may not be assigned, which will result in the missing of relevant studies and addition of irrelevant studies. One study found that none of the methodological filters have the combined sensitivity and precision for identifying primary diagnostic accuracy studies . Hence, it is recommended to use more than just the index condition, diagnostic test, and methodological filter when searching for studies related to diagnostic test accuracy .
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Evaluating the Quality and Comprehensiveness of the Search
Get Radiology Tree app to read full this article<
Table 1
Summarized version of PRESS 2015 Guideline Evidence-Based Checklist
Translation of research question Evaluate the quality of the search strategy: does it match the research question, assesses the width of the search terms and looks for the number of records retrieved Boolean and proximity operators Assess for the use and appropriateness of Boolean operators such as “AND”, “OR” as well as the proximity operators such as “adjacent”, “near”, “within”; within the search string Subject headings (database specific) Evaluating relevance and appropriate use of subject headings and subheadings Text word searching (free text) Evaluates appropriateness of specific words in terms of spelling variations, synonyms and antonyms, acronyms and abbreviations; review search results to ensure irrelevant studies are not included Spelling, syntax, and line numbers Evaluate for spelling and syntactical errors Limits and filters Assess the width and appropriateness of the filters or limitations applied or need to be applied
Get Radiology Tree app to read full this article<
Filtering Studies
Get Radiology Tree app to read full this article<
SECTION 3: ANALYSIS
Extracting Data
Get Radiology Tree app to read full this article<
Assessing for Bias in Included Studies
Get Radiology Tree app to read full this article<
Table 2
Summarized version of QUADAS- 2 system for evaluating quality of diagnostic accuracy studies
DOMAIN PATIENT SELECTION INDEX TEST REFERENCE STANDARD FLOW AND TIMING Description • Describe patient-selection methods
• Describe included patients (presentation, previous testing, intended use of index test, setting) Describe the index test, its execution and interpretation Describe the reference standard, its execution and interpretation • Describe patients who did not receive the index test and/or reference standard or those who were excluded from 2 × 2 table
• Describe the time interval and any interventions between index test(s) and reference standard Signaling questions (yes/no/unclear) • Was the patient selection consecutive or random?
• Was case-control design avoided?
• Were inappropriate exclusions avoided? \* • Were the index tests interpreted without knowledge of results of the reference standard?
• Was the threshold pre-specified (if used)? • Is the reference standard likely to correctly classify the target condition?
• Were the reference standard results interpreted without knowledge of index test results? • Was there an appropriate interval between index test(s) and reference standard?
• Did all patients get the reference standard?
• Did all patients get the same reference standard?
• Were all patients included in the analysis? Risk of bias: (High/low/unclear) Could patient-selection have introduced bias? Could the execution or interpretation of the index test have introduced bias? Could the execution or interpretation of the reference test have introduced bias? Could the patient flow have introduced bias? Concerns about applicability
(High/low/unclear) Are there concerns that the included patients do not match the review question? Are there concerns that the index test, its execution, or interpretation differ from the review question? Are there concerns that the target condition as defined by the reference standard does not match the review question?
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Analyzing Data
Get Radiology Tree app to read full this article<
Synthesizing and Interpreting Results
Get Radiology Tree app to read full this article<
Reporting Results
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
DISCUSSION
Applications in radiology
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Limitations
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Appendix 1-
Glossary of systematic review vocabulary used in manuscript
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
References
1. Uman LS: Systematic reviews and meta-analyses. J Can Acad Child Adolesc Psychiatry 2011; 20: pp. 57-59.
2. Pati D, Lorusso LN. How to Write a Systematic Review of the Literature. HERD . 2017:1937586717747384.
3. Higgins JPTGreen SCochrane Handbook for Systematic Reviews of Interventions.2011.The Cochrane Collaborationpp. xxi, 649.
4. Gough D, Oliver S, Thomas J: An introduction to systematic reviews.2012.Thousand Oaks, Calif.: SAGELondonpp. xiii, 288.
5. Richardson WS, Wilson MC, Nishikawa J, Hayward RS: The well-built clinical question: a key to evidence-based decisions. ACP J Club 1995; 123: pp. A12-A13.
6. Liberati A, Altman DG, Tetzlaff J, et. al.: The PRISMA Statement for Reporting Systematic Reviews and Meta-Analyses of Studies That Evaluate Health Care Interventions: Explanation and Elaboration. Plos Medicine 2009; 6:
7. McInnes DF, Bossuyt PMM. Systematic Reviews: Information for Radiologists . 2015 10-23-2015 1-19-2018]; Available from: http://pubs.rsna.org/page/radiology/pia/systematic-reviews .
8. Hulley SB, Cummings SR, Browner WS, Grady DG, Newman TB: Designing clinical research: an epidemiologic approach.3rd ed.2007.Williams & WilkinsBaltimorepp. xi, 247.
9. Group CDTAW. Handbook for Diagnostic Test Accuracy (DTA) Reviews . 1-19-2018]; Available from: http://methods.cochrane.org/sdt/handbook-dta-reviews .
10. Research NIfH. PROSPERO International Prospective Register of Systematic Reviews . 1-19-2018; Available from: https://www.crd.york.ac.uk/PROSPERO/ .
11. Reviews S. Systematic Reviews . 1-19-2018]; Available from: www.systematicreviewsjournal.com .
12. Grey Literature Checklist . 4-17-2018]; Available from: http://guides.mclibrary.duke.edu/greyliterature .
13. Beynon R, Leeflang MM, McDonald S, et. al.: Search strategies to identify diagnostic accuracy studies in MEDLINE and EMBASE. Cochrane Database Syst Rev 2013;
14. McGowan J, Sampson M, Salzwedel DM, et. al.: PRESS Peer Review of Electronic Search Strategies: 2015 Guideline Statement. J Clin Epidemiol 2016; 75: pp. 40-46.
15. Young T, Hopewell S: Methods for obtaining unpublished data. Cochrane Database Syst Rev 2011;
16. Jonnalagadda SR, Goyal P, Huffman MD: Automating data extraction in systematic reviews: a systematic review. Syst Rev 2015; 4: pp. 78.
17. Bristol Uo. QUADAS-2 . 1-19-2018; Available from: http://www.bristol.ac.uk/population-health-sciences/projects/quadas/quadas-2/ .
18. Collaboration TC. Assessing Risk of Bias in Included Studies . Available from: http://methods.cochrane.org/bias/assessing-risk-bias-included-studies .
19. Higgins JP, Altman DG, Gotzsche PC, et. al.: The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ 2011; 343: pp. d5928.
20. Stang A: Critical evaluation of the Newcastle-Ottawa scale for the assessment of the quality of nonrandomized studies in meta-analyses. Eur J Epidemiol 2010; 25: pp. 603-605.
21. Stroup DF, Berlin JA, Morton SC, et. al.: Meta-analysis of observational studies in epidemiology: a proposal for reporting. Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group. JAMA 2000; 283: pp. 2008-2012.
22. Kiewiet JJ, Leeuwenburgh MM, Bipat S, et. al.: A systematic review and meta-analysis of diagnostic performance of imaging in acute cholecystitis. Radiology 2012; 264: pp. 708-720.
23. (PRISMA) PRIfSRaM-A. PRISMA Statement . 1-19-2018; Available from: http://www.prisma-statement.org/ .
24. Moher D, Liberati A, Tetzlaff J, Altman DG, Group P: Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med 2009; 6:
25. Moher D, Shamseer L, Clarke M, et. al.: Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev 2015; 4: pp. 1.
26. Shamseer L, Moher D, Clarke M, et. al.: Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation. BMJ 2015; 350: pp. g7647.
27. Tunis AS, McInnes MD, Hanna R, Esmail K: Association of study quality with completeness of reporting: have completeness of reporting and quality of systematic reviews and meta-analyses in major radiology journals changed since publication of the PRISMA statement?. Radiology 2013; 269: pp. 413-426.
28. van der Pol CB, McInnes MD, Petrcich W, Tunis AS, Hanna R: Is quality and completeness of reporting of systematic reviews and meta-analyses published in high impact radiology journals associated with citation rates?. PLoS One 2015; 10:
29. McInnes MD, Bossuyt PM: Pitfalls of Systematic Reviews and Meta-Analyses in Imaging Research. Radiology 2015; 277: pp. 13-21.
30. Bossuyt PM: Informative reporting of systematic reviews in radiology. Radiology 2013; 269: pp. 313-314.
31. Tang A, Bashir MR, Corwin MT, et. al.: Evidence Supporting LI-RADS Major Features for CT- and MR Imaging-based Diagnosis of Hepatocellular Carcinoma: A Systematic Review. Radiology 2018; 286: pp. 29-48.
32. Kim HY, Park JH, Lee YJ et al. Systematic Review and Meta-Analysis of CT Features for Differentiating Complicated and Uncomplicated Appendicitis. Radiology . 2017:171260.
33. Xu XL, Liu XD, Liang M, and Luo BM. Radiofrequency Ablation versus Hepatic Resection for Small Hepatocellular Carcinoma: Systematic Review of Randomized Controlled Trials with Meta-Analysis and Trial Sequential Analysis. Radiology . 2017:162756.
34. Weng HH, Noll KR, Johnson JM et al. Accuracy of Presurgical Functional MR Imaging for Language Mapping of Brain Tumors: A Systematic Review and Meta-Analysis. Radiology . 2017:162971.
35. Symanski JS, Subhas N, Babb J, Nicholson J, Gyftopoulos S: Diagnosis of Superior Labrum Anterior-to-Posterior Tears by Using MR Imaging and MR Arthrography: A Systematic Review and Meta-Analysis. Radiology 2017; 285: pp. 101-113.
36. Behzadi AH, Zhao Y, Farooq Z, and Prince MR. Immediate Allergic Reactions to Gadolinium-based Contrast Agents: A Systematic Review and Meta-Analysis. Radiology . 2017:162740.
37. Bennani-Baiti B, Baltzer PA: MR Imaging for Diagnosis of Malignancy in Mammographic Microcalcifications: A Systematic Review and Meta-Analysis. Radiology 2017; 283: pp. 692-701.
38. Westerlaan HE, van Dijk JM, Jansen-van der Weide MC, et. al.: Intracranial aneurysms in patients with subarachnoid hemorrhage: CT angiography as a primary examination tool for diagnosis–systematic review and meta-analysis. Radiology 2011; 258: pp. 134-145.
39. van der Paardt MP, Zagers MB, Beets-Tan RG, Stoker J, Bipat S: Patients who undergo preoperative chemoradiotherapy for locally advanced rectal cancer restaged by using diagnostic MR imaging: a systematic review and meta-analysis. Radiology 2013; 269: pp. 101-112.
40. Pickhardt PJ, Hassan C, Halligan S, Marmo R: Colorectal cancer: CT colonography and colonoscopy for detection–systematic review and meta-analysis. Radiology 2011; 259: pp. 393-405.
41. Siddiqui A, Dreyer KJ, Gupta S: Meaningful use a call to arms. Acad Radiol 2012; 19: pp. 221-228.
42. Gupta S, Klein K, Singh AH, Thrall JH: Analysis of Low Appropriateness Score Exam Trends in Decision Support-based Radiology Order Entry System. J Am Coll Radiol 2017; 14: pp. 615-621.
43. Giardino A, Gupta S, Olson E, et. al.: Role of Imaging in the Era of Precision Medicine. Acad Radiol 2017; 24: pp. 639-649.
44. Toor SS, Jaberi A, Macdonald DB, et. al.: Complication rates and effectiveness of uterine artery embolization in the treatment of symptomatic leiomyomas: a systematic review and meta-analysis. AJR Am J Roentgenol 2012; 199: pp. 1153-1163.
45. McInnes MD, Kielar AZ, Macdonald DB: Percutaneous image-guided biopsy of the spleen: systematic review and meta-analysis of the complication rate and diagnostic accuracy. Radiology 2011; 260: pp. 699-708.
46. Whiting PF, Rutjes AW, Westwood ME, et. al.: QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med 2011; 155: pp. 529-536.