Home How to Perform a Systematic Review and Meta-analysis of Diagnostic Imaging Studies
Post
Cancel

How to Perform a Systematic Review and Meta-analysis of Diagnostic Imaging Studies

Posted Apr 30, 2018 Updated Apr 10, 2023
By Paul Cronin MD MS , Aine Marie Kelly MD MS MA , Duaa Altaee BA , Bradley Foerster MD PhD , Myria Petrou MD MS , Ben A. Dwamena MD
22 min read

A systematic review is a comprehensive search, critical evaluation, and synthesis of all the relevant studies on a specific (clinical) topic that can be applied to the evaluation of diagnostic and screening imaging studies. It can be a qualitative or a quantitative (meta-analysis) review of available literature. A meta-analysis uses statistical methods to combine and summarize the results of several studies. In this review, a 12-step approach to performing a systematic review (and meta-analysis) is outlined under the four domains: (1) Problem Formulation and Data Acquisition, (2) Quality Appraisal of Eligible Studies, (3) Statistical Analysis of Quantitative Data, and (4) Clinical Interpretation of the Evidence. This review is specifically geared toward the performance of a systematic review and meta-analysis of diagnostic test accuracy (imaging) studies.

Introduction

Systematic reviews and meta-analyses have become popular in medicine and are very commonly applied to treatment trials. However, they are still less common for diagnostic imaging studies. Systematic reviews and meta-analyses aim to provide summaries of the average result. In the case of imaging tests, this is diagnostic performance such as sensitivity or specificity, and the uncertainty of this average. In radiology, the smaller patient size and limited methodological quality of the primary studies can limit the quality of the review and meta-analysis. However, systematic reviews and meta-analyses may be the best assessment of the published literature available at any point in time, especially in the absence of large, definitive trials. They may provide important information to guide patient care and direct future clinical research. Performing and interpreting systematic reviews in radiology can be challenging given the paucity of available clinical studies. However, if investigators adhere to proper methodology, systematic reviews may provide useful information from a comprehensive study of the literature with limited bias.

In this review, a 12-step framework for performing systematic reviews (and meta-analyses) is outlined under the four domains: (1) Problem Formulation and Data Acquisition, (2) Quality Appraisal of Eligible Studies, (3) Statistical Analysis of Quantitative Data, and (4) Clinical Interpretation of the Evidence ( Table 1 ). We will subsequently use “systematic review” and “meta-analysis” to represent the whole process of evidence synthesis. The steps in “problem formulation and data acquisition” are “define the question and objective of the review,” “establish criteria for including studies in the review,” and “conduct a literature search to retrieve the relevant literature.” The steps in “quality appraisal of eligible studies” are “extract data on variables of interest,” “assess study quality and applicability to the clinical problem at hand,” and “summarize the evidence qualitatively and, if appropriate, quantitatively (meta-analysis).” The steps in “statistical analysis of quantitative data” are “estimate summary diagnostic test performance metrics and display the data,” “assess heterogeneity,” “investigate data for publication bias,” “assess the robustness of estimates of diagnostic accuracy using sensitivity analyses,” and “explore and explain heterogeneity in test accuracy using subgroup analysis (if applicable).” The steps in “clinical interpretation of the evidence” are “graphically display how the evidence alters the posttest probability using a Fagan plot (Bayes nomogram), likelihood ratio scatter graph, or probability-modifying plot.” This review is tailored for radiologists who are new to the process of performing a systematic review and meta-analysis. However, we hope that those with experience with systematic review and meta-analysis will also find new information in this article.

TABLE 1

An Outline of the Main Steps in Doing a Meta-analysis of Diagnostic Test Accuracy

1. Problem formulation and data acquisition Step 1. Define the question and objective of the review Step 2. Establish criteria for including studies in the review Step 3. Conduct a literature search to retrieve the relevant literature2. Quality appraisal of eligible studies Step 4. Extract data on variables of interest Step 5. Assess study quality and applicability to the clinical problem at hand Step 6. Summarizing the evidence qualitatively and if appropriate, quantitatively (meta-analysis)3. Statistical analysis of quantitative data Step 7. Estimate diagnostic accuracy and display the data Step 8. Assess heterogeneity Step 9. Assess for publication bias Step 10. Assess the robustness of estimates of diagnostic accuracy using sensitivity analyses (if applicable) Step 11. Explore and explain heterogeneity in test accuracy using subgroup analysis (if applicable)4. Clinical interpretation of the evidence Step 12. Graphically display how the evidence alters the posttest probability

Problem Formulation and Data Acquisition

Step 1. Define the Question and Objective of the Review

A good review question addresses a clinical problem for which there is uncertainty. Therefore, the first step is to identify the relevant clinical problem. This includes specifying the patient, the index test(s) and reference test being studied, and the outcome measurements (diagnostic test accuracy) . In evidence-based practice, these components can be abbreviated to PICO (Patient, Intervention, Comparator, and Outcome) or in the Cochrane guidelines for diagnostic accuracy tests as PICTS (Patient, Index test, Comparator test, Target disorder and Study design) . Patients can refer to patients presenting signs and symptoms of the disease (diagnostic studies), patients with the disease (prognostic studies), or population at risk of the disease (screening studies). The index test is the test to be evaluated. A meta-analysis may consider and compare several index tests. The comparator test is standard practice or the reference standard or the “gold standard” that the index tests are compared to. It is the test or procedure used to classify patients as having the target condition or disease or not. The target disorder is the disease that one is trying to diagnose. Examples of PICO questions or statements are shown in Table A1 . These include “In patients with symptomatic carotid stenosis, how does computed tomographic angiography (CTA) compare with magnetic resonance angiography (MRA) for the detection and quantification of carotid stenosis?” or “In patients with known or suspected coronary artery disease, how does CT coronary angiography compare with invasive catheter coronary angiography for identifying one (or more) potentially or probably hemodynamically significant (≥50% coronary artery luminal diameter) stenosis in terms of sensitivity, specificity and diagnostic accuracy?” or “In patients with a solitary pulmonary nodule, how well does dynamic contrast material–enhanced CT, dynamic contrast material–enhanced MR imaging, FDG PET, and 99m Tc-depreotide SPECT compare for the diagnosis of malignancy (diagnostic accuracy)?” or “In patients with known or suspected rotator cuff tears, how does ultrasound compare to MRI for diagnosis?” or “Is low-dose CT colonography equivalent to optical colonoscopy in identifying clinically meaningful colonic polyps?” It should be remembered that evidence synthesis can be derailed by not asking a focused question. It is also important to have a focused research question as this is used to direct the search.

Table A1

Examples of PICOS (Patient, Intervention, Comparator, Outcome, and Study Design) or in the Cochrane Guidelines for Diagnostic Accuracy Tests as PICTS (Patient, Index Test, Comparator test, Target Disorder and Study Design) Statements

(PICOS)—Patient, Population, Problem Intervention Comparator Outcome Study design (PICTS)—Patient, Population, Problem Index test Comparator test Target disorder Study design Symptomatic carotid stenosis Computed tomographic angiography (CTA) Magnetic resonance angiography (MRA) Sensitivity, specificity, and diagnostic accuracy

Detection and quantification of carotid stenosis Known or suspected coronary artery disease CT coronary angiography Invasive catheter coronary angiography Sensitivity, specificity, and diagnostic accuracy

Identifying one (or more) potentially or probably hemodynamically significant (≥50% coronary artery luminal diameter) stenosis A solitary pulmonary nodule Dynamic contrast material–enhanced CT

Dynamic contrast material–enhanced MRI

FDG PET

99m Tc-depreotide SPECT Histology Sensitivity, specificity, and diagnostic accuracy

Diagnosis of malignancy Known or suspected rotator cuff tears Ultrasound MRI Sensitivity, specificity, and diagnostic accuracy Low-dose CT colonography (CTC) Optical colonoscopy (OC) Sensitivity, specificity, and diagnostic accuracy

Clinically meaningful colonic polyps

CT, computed tomography; MRI, magnetic resonance imaging.

Step 2. Establish Criteria for Including Studies in the Review

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Step 3. Conduct a Literature Search to Retrieve the Relevant Literature

Get Radiology Tree app to read full this article<

Table A2

Examples of Search Sources

Computerized bibliographic databases (examples) PubMed— www.ncbi.nlm.nih.gov/pubmed/ MEDLINE— www.medline.com EMBASE https://embase.elsevier.com/ Health Technology Assessment (HTA)— www.york.ac.uk/inst/crd/crddatabases.htm#HTA Database of Abstracts of Reviews of Effects (DARE)— www.york.ac.uk/inst/crd/crddatabases.htm#DARE Turning Research into Practice (TRIP)— www.tripdatabase.com/Aboutus/Publications/index.html?catid=11 TRIP for guidelines see www.tripdatabase.com/Aboutus/Publications/index.html?catid=4 Aggressive Research Intelligence Facility (ARIF) www.arif.bham.ac.uk/ Cochrane Central Register of Controlled Trials (CENTRAL)— http://www.cochranelibrary.com/about/central-landing-page.html Search Medica— www.searchmedica.com Google Scholar— www.scholar.google.com Google search engine— www.google.com Yahoo search engine— www.search.yahoo.com Science Citation Index— scientific.thomson.com/products/sci/ Web of Science— scientific.thomson.com/products/wos/ Web of Knowledge— isiwebofknowledge.com/ Scopus— info.scopus.com/overview/what/ Gale Directory of Online Portable and Internet Databases— http://library.dialog.com/bluesheets/pdf/bl0230.pdf Continental and regional and national databasesSubject-specific databasesFull-text journals available electronically (examples) Public Library of Science (PLoS)— www.plos.org/journals/ PubMed Central— www.pubmedcentral.nih.gov/ BiomedCentral— www.biomedcentral.com Free Medical Journals— freemedicaljournals.com/ HighWire Press— highwire.stanford.edu/lists/freeart.dtl Journal reference listsAncestor and descendent search Always examine the references of articles which have been decided to be included in meta-analysis to see if they contain any relevant studies of which the researcher is unaware.Conference abstracts or proceedings (examples) Biological Abstracts/RRM (Reports, Reviews, Meetings)— scientific.thomsonreuters.com/products/barrm/ BMC Meeting Abstracts (free)— www.biomedcentral.com Conference Papers Index— www.csa.com/factsheets/cpi-set-c.php Programs from professional \meetingsResearch registersDissertations and theses databases (examples) ProQuest Dissertations & Theses Database: indexes more than 2 million doctoral dissertations and masters’ theses and includes US dissertations since 1861 and British dissertations since 1988— www.proquest.co.uk/products_pq/descriptions/pqdt.shtml Letters to active researchersPersonal contact and peer consultationGray literature databasesOther reviews, (evidence-based) guidelines and sources of studies (examples) National Guideline Clearinghouse (US)— www.guideline.gov/ Canadian Medical Association—Infobase: Clinical Practice Guidelines— www.cma.ca/index.cfm/ci_id/54316/1a_id/1.htm National Library of Guidelines (UK)— www.library.nhs.uk NICE Clinical Guidelines (UK)— www.nice.org.uk/aboutnice/whatwedo/aboutclinicalguidelines/about_clinical_guidelines.jsp Australian National Health and Medical Research Council: Clinical Practice Guidelines— www.nhmrc.gov.au/publications/subjects/clinical.htm New Zealand Guidelines Group— www.nzgg.org.nz Citation alertsHandsearchingWeb searchingUnpublished and ongoing studies

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Figure 1, Example of a flowchart of a literature search.

Get Radiology Tree app to read full this article<

Quality Appraisal of Eligible Studies

Step 4. Extract Data on Variables of Interest

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Step 5. Assess Study Quality and Applicability to the Clinical Problem at Hand

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Figure 2, (a) Methodological quality, study validity, and risk of bias summary for each study showing authors' judgments about each domain for each included study using the QUADAS tool. (b) Study quality scores. Graph illustrates study quality based on QUADAS criteria, expressed as a percentage of studies meeting each criterion. (c) Risk of bias and applicability summary for each study showing authors' judgments about each domain for each included study using the QUADAS 2 tool. (d) Study quality scores. Graph illustrates study quality based on QUADAS 2 criteria, expressed as a percentage of studies meeting each criterion.

Get Radiology Tree app to read full this article<

Step 6. Summarize the Evidence Qualitatively and, if Appropriate, Quantitatively

Get Radiology Tree app to read full this article<

Statistical Analysis of Quantitative Data

Step 7. Estimate Diagnostic Accuracy

Meta-analysis of Diagnostic Test Accuracy Differ From Meta-analysis of Interventions

Get Radiology Tree app to read full this article<

Analyzing the Data

Get Radiology Tree app to read full this article<

Table A3

The Commonly Used Summary Statistics for Test Accuracy Including a 2 × 2 Contingency Table with Sensitivity and Specificity Positive- and Negative-predictive Values and Accuracy Calculated

Disease True False Test outcome_Positive_ True positive False positive → Positive predictive value = TP/(TP + FP)Negative False negative True negative → Negative predictive value = TN/(TN + FN) ↓

Sensitivity =

True positive rate =

True positive fraction =

Detection rate = TP/(TP + FN) ↓

Specificity =

True-negative rate =

True-negative fraction = TN/(FP + TN) → Accuracy = (TP +TN)/(TP + FP + FN + TN)

→ Prevalence = (TP + FN)/(TP + FP + FN + TN)

FN, false-negative; FP, false positive; TN, true negative; TP, true positive.

Sensitivity = TP/(TP + FN).

Specificity = TN/(TN + FP).

Positive-predictive value = TP/(TP + FP).

Negative-predictive value = TN/(TN + FN).

Accuracy = TP + TN/(TP + FP + FN + TN).

Get Radiology Tree app to read full this article<

Model Fitting and Statistical Methods for Pooling Data

Moses-Littenberg SROC curves

Get Radiology Tree app to read full this article<

Hierarchical and Bivariate Models

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Display the Data

Forest Plot

Get Radiology Tree app to read full this article<

Figure 3, Forest plot showing study-specific and mean sensitivity and specificity. Each black square is a study-specific sensitivity and specificity. The size of the black square reflects the weight of the study in the meta-analysis, and the horizontal line reflects the 95% confidence interval (CI). The vertical broken line represents the pooled sensitivity or specificity and the boundaries of the hollow diamond displayed at the bottom represent the 95% CI of the pooled results.

Get Radiology Tree app to read full this article<

Positivity Thresholds

Get Radiology Tree app to read full this article<

Summary ROC Plots

Get Radiology Tree app to read full this article<

Linked ROC Plots

Get Radiology Tree app to read full this article<

Figure 4, SROC curve with confidence and prediction regions around mean operating sensitivity and specificity point. Area under the summary ROC curve. Each circle represents an individual study result. The diamond in the center represents intersection of the summary sensitivity and specificity, the inner dashed line represents the 95% confidence interval of the summary sensitivity and specificity, and the outer dotted line represents the 95% predicted interval. SENS, sensitivity; SPEC, specificity; SROC, summary receiver operating characteristic; AUC,  area under the curve.

Get Radiology Tree app to read full this article<

Test Results Are Available Only as a Dichotomy

Get Radiology Tree app to read full this article<

Test Results Are Available in More Than Two Categories

Get Radiology Tree app to read full this article<

Step 8. Assess Heterogeneity

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Heterogeneity Due to Threshold Effect

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Heterogeneity Due to Non-threshold Effect

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Meta-regression

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Step 9. Assess Publication Bias

Get Radiology Tree app to read full this article<

Figure 5, Assessing publication bias. Funnel plot with superimposed regression line. Formal testing for publication bias may be conducted by a regression of diagnostic log odds ratio against 1/square root of the effective sample size (1/root (ESS)), weighting by effective sample size, with P < .10 for the slope coefficient indicating significant asymmetry. The statistically nonsignificant P value (.89) for the slope coefficient suggests symmetry in the data and a low likelihood of publication bias. However, the test is known to have low power (41) .

Get Radiology Tree app to read full this article<

Step 10. Assess the Robustness of Estimates of Diagnostic Accuracy Using Sensitivity Analyses

Get Radiology Tree app to read full this article<

Figure 6, Forest plot of results of multiple univariate meta-regression and subgroup analyses. This figure shows the results of a meta-regression of the dataset. Covariates, such as study design (prospective cohort [prodesign]), study size ≥ 30 patients (ssize 30), and partial verification bias (fulverif) may be introduced into a regression with any test performance measure as the dependent variable. In diagnostic studies, heterogeneity in sensitivity and specificity can result from many causes related to definitions of the test and reference standards, operating characteristics of the test, methods of data collection, and patient characteristics. Meta-regression is the use of regression methods to incorporate the effect of covariates on summary measures of performance and can be used to explore between-study heterogeneity. As with any meta-regression, however, the sample size will correspond to the number of studies in the analysis with small number of studies, limiting the power of regression to detect significant effects.

Get Radiology Tree app to read full this article<

Step 11. Explore and Explain Heterogeneity in Test Accuracy Using Subgroup Analysis

Get Radiology Tree app to read full this article<

Comparing Index Tests

Get Radiology Tree app to read full this article<

Analysis With Small Numbers of Studies

Get Radiology Tree app to read full this article<

Clinical Interpretation of the Evidence

Step 12. Graphically Display How the Evidence Alters the Posttest Probability

Fagan Plot (Bayes Nomogram)

Get Radiology Tree app to read full this article<

Pretest probability=Prevalence of target condition PTP=LR×Pretest probability/[(1−Pretest probability)×(1−LR)] Pretest probability

=

Prevalence of target condition PTP

=

LR

×

Pretest probability

/

[

(

1

−

Pretest probability

)

×

(

1

−

LR

)

]

This concept is depicted visually with Fagan nomograms . When Bayes theorem is expressed in terms of log-odds, the posterior log-odds are linear functions of the prior log-odds and the log-likelihood ratios. A Fagan plot, as shown in Figure 7 , consists of a vertical axis on the left with the prior log-odds, an axis in the middle representing the log-likelihood ratio, and a vertical axis on the right representing the posterior log-odds. Lines are then drawn from the prior probability on the left through the likelihood ratios in the center and extended to the posterior probabilities on the right ( Fig 7 ).

Figure 7, Likelihood ratio or Fagan nomogram for different pretest probability of disease: 25%, 50% and 75% for two tests. Posttest probability is derived by drawing a straight line from the pretest probability vertical axis to the appropriate likelihood ratio and continuing the straight line to the vertical posttest probability axis. Where this line intersects the vertical posttest probability axis is the posttest probability. When Bayes theorem is expressed in terms of log-odds, the posterior log-odds are linear functions of the earlier log-odds and the log-likelihood ratios. A Fagan plot consists of a vertical axis on the left with the earlier log-odds, an axis in the middle representing the log-likelihood ratio and a vertical axis on the right representing the posterior log-odds (43) .

Get Radiology Tree app to read full this article<

Likelihood Ratio Scatter Graph

Get Radiology Tree app to read full this article<

Figure 8, Likelihood ratio scatter graph shows summary point of likelihood ratios obtained as functions of mean sensitivity and specificity in the right upper quadrant, suggesting that the test is useful for confirmation of presence of disease (when positive) and not for its exclusion (when negative). Informativeness may also be represented graphically by a likelihood ratio scatter graph or matrix. It defines quadrants of informativeness based on established evidence-based thresholds: Left upper quadrant, likelihood ratio positive > 10, likelihood ratio negative < 0.1, confirmation and exclusion, suggesting that the test is useful for confirmation of presence of disease (when positive) and for its exclusion (when negative). Right upper quadrant, likelihood ratio positive > 10, likelihood ratio negative > 0.1, confirmation only, suggesting that the test is useful for confirmation of presence of disease (when positive) and not for its exclusion (when negative). Left lower quadrant, likelihood ratio positive < 10, likelihood ratio negative < 0.1, exclusion only, suggesting that the test is not useful for confirmation of presence of disease (when positive) but is for its exclusion (when negative). Right lower quadrant, likelihood ratio positive < 10, likelihood ratio negative > 0.1, no exclusion or confirmation, suggesting that the test is neither useful for confirmation of presence of disease (when positive) nor for its exclusion (when negative) (44) .

Get Radiology Tree app to read full this article<

Predictive Values and Probability-modifying Plot

Get Radiology Tree app to read full this article<

Figure 9, Probability-modifying plot with posttest probabilities for hypothetical populations with different prevalence of disease according to Bayes theorem. Dashed line indicates a positive test result, dashed line with dots indicate a negative result. Posttest probability for a positive result is derived by drawing a vertical line up to the dashed curved lin e and then across to the y-axis. Posttest probability for a negative result is derived by drawing a vertical line up to the dashed line with dots curved line and then across to the y-axis. The probability-modifying plot is a graphical sensitivity analysis of predictive value across a prevalence continuum defining low- to high-risk populations. It depicts separate curves for positive and negative tests. The user draws a vertical line from the selected pretest probability to the appropriate likelihood ratio line and then reads the posttest probability off the vertical scale.

Get Radiology Tree app to read full this article<

Conclusion

Get Radiology Tree app to read full this article<

Appendix

Software for Diagnostic Accuracy Meta-analysis

Get Radiology Tree app to read full this article<

midas

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

RevMan

Get Radiology Tree app to read full this article<

dr-ROC

Get Radiology Tree app to read full this article<

metandi

Get Radiology Tree app to read full this article<

Metadas

Get Radiology Tree app to read full this article<

mada

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

HSROC

Get Radiology Tree app to read full this article<

Meta-DiSc

Get Radiology Tree app to read full this article<

Sensitivity and Specificity

Get Radiology Tree app to read full this article<

Predictive Values

Get Radiology Tree app to read full this article<

Accuracy

Get Radiology Tree app to read full this article<

Likelihood Ratios

Get Radiology Tree app to read full this article<

Diagnostic Odds Ratios

Get Radiology Tree app to read full this article<

References

  • 1. Berman N.G., Parker R.A.: Meta-analysis: neither quick nor easy. BMC Med Res Methodol 2002; 2: pp. 10.

  • 2. Whiting P., Rutjes A.W., Reitsma J.B., et. al.: Sources of variation and bias in studies of diagnostic accuracy: a systematic review. Ann Intern Med 2004; 140: pp. 189-202.

  • 3. Whiting P., Rutjes A.W., Reitsma J.B., et. al.: The development of QUADAS: a tool for the quality assessment of studies of diagnostic accuracy included in systematic reviews. BMC Med Res Methodol 2003; 3: pp. 25.

  • 4. Whiting P., Rutjes A.W., Dinnes J., et. al.: Development and validation of methods for assessing the quality of diagnostic accuracy studies. Health Technol Assess 2004; 8: pp. iii. 1–234

  • 5. Leeflang M.M., Deeks J.J., Gatsonis C., et. al.: Systematic reviews of diagnostic test accuracy. Ann Intern Med 2008; 149: pp. 889-897.

  • 6. Simes R.J.: Publication bias: the case for an international registry of clinical trials. J Clin Oncol 1986; 4: pp. 1529-1541.

  • 7. Lijmer J.G., Mol B.W., Heisterkamp S., et. al.: Empirical evidence of design-related bias in studies of diagnostic tests. JAMA 1999; 282: pp. 1061-1066.

  • 8. McGrath T.A., McInnes M.D.F., Langer F.W., et. al.: Treatment of multiple test readers in diagnostic accuracy systematic reviews-meta-analyses of imaging studies. Eur J Radiol 2017; 93: pp. 59-64.

  • 9. Buscemi N., Hartling L., Vandermeer B., et. al.: Single data extraction generated more errors than double data extraction in systematic reviews. J Clin Epidemiol 2006; 59: pp. 697-703.

  • 10. Jones A.P., Remmington T., Williamson P.R., et. al.: High prevalence but low impact of data extraction and reporting errors were found in Cochrane systematic reviews. J Clin Epidemiol 2005; 58: pp. 741-742.

  • 11. Gotzsche P.C., Hrobjartsson A., Maric K., et. al.: Data extraction errors in meta-analyses that use standardized mean differences. JAMA 2007; 298: pp. 430-437.

  • 12. Cook D.J., Sackett D.L., Spitzer W.O.: Methodologic guidelines for systematic reviews of randomized control trials in health care from the Potsdam Consultation on meta-analysis. J Clin Epidemiol 1995; 48: pp. 167-171.

  • 13. Whiting P.F., Rutjes A.W., Westwood M.E., et. al.: QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med 2011; 155: pp. 529-536.

  • 14. Moses L.E., Shapiro D., Littenberg B.: Combining independent studies of a diagnostic test into a summary ROC curve: data-analytic approaches and some additional considerations. Stat Med 1993; 12: pp. 1293-1316.

  • 15. Midgette A.S., Stukel T.A., Littenberg B.: A meta-analytic method for summarizing diagnostic test performances: receiver-operating-characteristic-summary point estimates. Med Decis Making 1993; 13: pp. 253-257.

  • 16. van Houwelingen H.C., Arends L.R., Stijnen T.: Advanced methods in meta-analysis: multivariate approach and meta-regression. Stat Med 2002; 21: pp. 589-624.

  • 17. Macaskill P.: Empirical Bayes estimates generated in a hierarchical summary ROC analysis agreed closely with those of a full Bayesian analysis. J Clin Epidemiol 2004; 57: pp. 925-932.

  • 18. Reitsma J.B., Glas A.S., Rutjes A.W., et. al.: Bivariate analysis of sensitivity and specificity produces informative summary measures in diagnostic reviews. J Clin Epidemiol 2005; 58: pp. 982-990.

  • 19. Chu H., Cole S.R.: Bivariate meta-analysis of sensitivity and specificity with sparse data: a generalized linear mixed model approach. J Clin Epidemiol 2006; 59: pp. 1331-1332. author reply 1332–1333

  • 20. Rutter C.M., Gatsonis C.A.: A hierarchical regression approach to meta-analysis of diagnostic test accuracy evaluations. Stat Med 2001; 20: pp. 2865-2884.

  • 21. Arends L.R., Hamza T.H., van Houwelingen J.C., et. al.: Bivariate random effects meta-analysis of ROC curves. Med Decis Making 2008; 28: pp. 621-638.

  • 22. Harbord R.M., Deeks J.J., Egger M., et. al.: A unification of models for meta-analysis of diagnostic accuracy studies. Biostatistics 2007; 8: pp. 239-251.

  • 23. McGrath T.A., McInnes M.D., Korevaar D.A., et. al.: Meta-analyses of diagnostic accuracy in imaging journals: analysis of pooling techniques and their effect on summary estimates of diagnostic accuracy. Radiology 2016; 281: pp. 78-85.

  • 24. Mulrow C.D.: Rationale for systematic reviews. BMJ 1994; 309: pp. 597-599.

  • 25. Lau J., Ioannidis J.P., Terrin N., et. al.: The case of the misleading funnel plot. BMJ 2006; 333: pp. 597-600.

  • 26. Lewis S., Clarke M.: Forest plots: trying to see the wood and the trees. BMJ 2001; 322: pp. 1479-1480.

  • 27. Hanley J.A.: Receiver operating characteristic (ROC) methodology: the state of the art. Crit Rev Diagn Imaging 1989; 29: pp. 307-335.

  • 28. Ioannidis J.P., Patsopoulos N.A., Evangelou E.: Uncertainty in heterogeneity estimates in meta-analyses. BMJ 2007; 335: pp. 914-916.

  • 29. Higgins J.P., Thompson S.G., Deeks J.J., et. al.: Measuring inconsistency in meta-analyses. BMJ 2003; 327: pp. 557-560.

  • 30. Higgins J.P., Thompson S.G.: Quantifying heterogeneity in a meta-analysis. Stat Med 2002; 21: pp. 1539-1558.

  • 31. Zhou Y., Dendukuri N.: Statistics for quantifying heterogeneity in univariate and bivariate meta-analyses of binary data: the case of meta-analyses of diagnostic accuracy. Stat Med 2014; 33: pp. 2701-2717.

  • 32. McInnes M.D., Hibbert R.M., Inacio J.R., et. al.: Focal nodular hyperplasia and hepatocellular adenoma: accuracy of gadoxetic acid–enhanced MR imaging—a systematic review. Radiology 2015; 277: pp. 413-423.

  • 33. Dinnes J., Deeks J., Kirby J., et. al.: A methodological review of how heterogeneity has been examined in systematic reviews of diagnostic test accuracy. Health Technol Assess 2005; 9: pp. 1-113. iii

  • 34. Lee J., Kim K.W., Choi S.H., et. al.: Systematic review and meta-analysis of studies evaluating diagnostic test accuracy: a practical review for clinical researchers—part II. Statistical methods of meta-analysis. Korean J Radiol 2015; 16: pp. 1188-1196.

  • 35. Lijmer J.G., Bossuyt P.M., Heisterkamp S.H.: Exploring sources of heterogeneity in systematic reviews of diagnostic tests. Stat Med 2002; 21: pp. 1525-1537.

  • 36. Egger M., Davey Smith G., Schneider M., et. al.: Bias in meta-analysis detected by a simple, graphical test. BMJ 1997; 315: pp. 629-634.

  • 37. Sterne J.A., Egger M.: Funnel plots for detecting bias in meta-analysis: guidelines on choice of axis. J Clin Epidemiol 2001; 54: pp. 1046-1055.

  • 38. Egger M., Smith G.D.: Misleading meta-analysis. BMJ 1995; 311: pp. 753-754.

  • 39. Sterne J.A., Egger M., Smith G.D.: Systematic reviews in health care: investigating and dealing with publication and other biases in meta-analysis. BMJ 2001; 323: pp. 101-105.

  • 40. Terrin N., Schmid C.H., Lau J.: In an empirical evaluation of the funnel plot, researchers could not visually identify publication bias. J Clin Epidemiol 2005; 58: pp. 894-901.

  • 41. Deeks J.J., Macaskill P., Irwig L.: The performance of tests of publication bias and other sample size effects in systematic reviews of diagnostic test accuracy was assessed. J Clin Epidemiol 2005; 58: pp. 882-893.

  • 42. Peters J.L., Sutton A.J., Jones D.R., et. al.: Comparison of two methods to detect publication bias in meta-analysis. JAMA 2006; 295: pp. 676-680.

  • 43. Fagan T.J.: Letter: nomogram for Bayes theorem. N Engl J Med 1975; 293: pp. 257.

  • 44. Stengel D., Bauwens K., Sehouli J., et. al.: A likelihood ratio approach to meta-analysis of diagnostic studies. J Med Screen 2003; 10: pp. 47-51.

  • 45. Li J., Fine J.P., Safdar N.: Prevalence-dependent diagnostic accuracy measures. Stat Med 2007; 26: pp. 3258-3273.

  • 46. Dwamena B.: MIDAS: Meta-analytical integration of diagnostic accuracy studies in Stata, West Coast Stata Users’ Group meetings.2007.University of Michigan MIDAS Web site; Published August 15, 2007; Available at http://sitemaker.umich.edu/metadiagnosis/midas_home

  • 47. Dwamena B.: MIDAS: Meta-analytical integration of diagnostic accuracy studies in Stata, North American Stata Users’ Group meetings.2007.University of Michigan MIDAS Web site; Published August 15, 2007; Available at http://sitemaker.umich.edu/metadiagnosis/midas_home

  • 48. Dwamena B.A.: MIDAS: Stata module for meta-analytical integration of diagnostic test accuracy studies.2008.Boston College Department of EconomicsBoston, MA Available at http://ideas.repec.org/c/boc/bocode/s456880.html

  • 49. Van Houwelingen H.C., Zwinderman K.H., Stijnen T.: A bivariate approach to meta-analysis. Stat Med 1993; 12: pp. 2273-2284.

  • 50. Riley R.D., Abrams K.R., Lambert P.C., et. al.: An evaluation of bivariate random-effects meta-analysis for the joint synthesis of two correlated outcomes. Stat Med 2007; 26: pp. 78-97.

  • 51. Riley R.D., Abrams K.R., Sutton A.J., et. al.: Bivariate random-effects meta-analysis and the estimation of between-study correlation. BMC Med Res Methodol 2007; 7: pp. 3.

  • 52. Rabe-Hesketh S.: GLLAMM manual. University of California-Berkeley, Division of Biostatistics, Working Paper Series Paper No. 1602004.

  • 53. Rabe-Hesketh S., Skrondal A., Pickles A.: Reliable estimation of generalized linear mixed models using adaptive quadrature. Stata J 2002; 2: pp. 1-21.

  • 54. Littenberg B., Moses L.E.: Estimating diagnostic accuracy from multiple conflicting reports: a new meta-analytic method. Med Decis Making 1993; 13: pp. 313-321.

  • 55. Harbord R., Whitting P., Sterne J.: metandi: Stata module for statistically rigorous meta-analysis of diagnostic accuracy studies.Methods for evaluating medical tests.2008.Department of Public Health, Epidemiology and Biostatistics, University of BirminghamBirmingham, UK: 1st Symposium; July 24–25, 23

  • 56. Zamora J., Abraira V., Muriel A., et. al.: Meta-DiSc: a software for meta-analysis of test accuracy data. BMC Med Res Methodol 2006; 6: pp. 31.

Academic Radiology, Volume 25, Issue 5
Journals General Radiology
This post is licensed under CC BY 4.0 by the author.
Recently Updated
Contents