Radiology practice involves extracting diagnostically useful information from medical images. Judgment is a necessary component of this process, as it is in all other medical specialties. Although based on clinical expertise, a radiologist’s judgment remains a subjective skill.
The subjective component of radiology practice can make it difficult to measure the level of diagnostic performance. Simple diagnostic performance measures such as sensitivity and specificity are a start, but they can mask the underlying variability of radiology readers. For example, radiologists are constantly faced with the tradeoff between sensitivity and specificity caused by differing thresholds for calling an exam abnormal or indicative of a particular disease. We know that in radiology practice, some radiologists tend to be “overcallers” (higher sensitivity but lower specificity, on average), and some tend to be “undercallers” (lower sensitivity but higher specificity, on average) . Who has the “right” sensitivity and specificity?
Receiver operating characteristic (ROC) analysis adjusts for possible differences in reader thresholds so that “overcallers” and “undercallers” can be considered on the same scale. If we compare two different imaging techniques, we want to know if there is a difference in sensitivity and/or specificity that is not due to differences in reader thresholds. ROC analysis allows us to tease out these real underlying differences.
This issue of Academic Radiology is the first of two issues honoring the memory of Dr. Charles E. Metz, a pioneer in the application of ROC analysis to the evaluation of radiology exams . It is fitting that Dr. Metz is honored in this journal because Academic Radiology has published some of the seminal articles in ROC analysis . Among Academic Radiology articles published in the past 5 years, 2 of the 10 most cited articles have been about ROC analysis, and they happened to be coauthored by Dr. Metz .
This issue features 10 articles sampling the latest original work and reviews on ROC analysis. Many of the authors are protégés and/or former collaborators of Dr. Metz. These associations were not requirements for inclusion in this issue and simply reflect the tremendous influence that Dr. Metz had on the work in this field.
The first article describes an exercise that I developed to teach basic principles of ROC analysis . I have given didactic lectures about ROC analysis but have not been completely satisfied with their apparent effectiveness or connection with the audience. In other words, I saw some students dosing off, and I thought I was boring. When asked to teach a class in a new course, I decided to address the boredom issue by adding an interactive element. The ROC laboratory exercise was the result.
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
References
1. Miller G.M.: A radiologist with a ruler. Am J Neuroradiol 2003; 24: pp. 556.
2. Nishikawa R.M.: Charles E. Metz, PhD. Acad Radiol 2012; 19: pp. 1537-1538.
3. Dorfman D.D., Berbaum K.S., Metz C.E., et. al.: Proper receiver operating characteristic analysis: the bigamma model. Acad Rad 1997; 4: pp. 138-149.
4. Roe C.A., Metz C.E.: Dorfman-Berbaum-Metz method for statistical analysis of multireader, multimodality receiver operating characteristic data: validation with computer simulation. Acad Rad 1997; 4: pp. 298-303.
5. Wagner R.F., Metz C.E., Campbell G.: Assessment of medical imaging systems and computer aids: a tutorial review. Acad Rad 2007; 14: pp. 723-748.
6. Pesce L.L., Metz C.E.: Reliable and computationally efficient maximum-likelihood estimation of “proper” binormal ROC curves. Acad Rad 2007; 14: pp. 814-829.
7. Eng J.: Teaching receiver operating characteristic analysis: an interactive laboratory exercise. Acad Radiol 2012; 19: pp. 1452-1456.
8. Alemayehu D., Zou K.H.: Applications of ROC analysis in medical research: recent developments and future directions. Acad Radiol 2012; 19: pp. 1457-1464.
9. Parast L., Cai B., Bedayat A., et. al.: Statistical methods for predicting mortality in patients diagnosed with acute pulmonary embolism. Acad Radiol 2012; 19: pp. 1465-1473.
10. Chakraborty D.P., Yoon H.J., Mello-Thoms C.: Application of threshold-bias independent analysis to eye-tracking and FROC data. Acad Radiol 2012; 19: pp. 1474-1483.
11. McClish D.K.: Evaluation of the accuracy of medical tests in a region around the optimal point. Acad Radiol 2012; 19: pp. 1484-1490.
12. Hillis S.L., Metz C.E.: An analytic expression for the binormal partial area under the ROC curve. Acad Radiol 2012; 19: pp. 1491-1498.
13. Dorfman D.D., Berbaum K.S., Metz C.E.: Receiver operating characteristic analysis: generalization to the population of readers and patients with the jackknife method. Invest Radiol 1992; 27: pp. 723-731.
14. Hillis S.L., Berbaum K.S., Metz C.E.: Recent developments in the Dorfman-Berbaum-Metz procedure for multireader ROC study analysis. Acad Radiol 2008; 15: pp. 647-661.
15. Skaron A., Li K., Zhou X.H.: Statistical methods for MRMC ROC studies. Acad Radiol 2012; 19: pp. 1499-1507.
16. Obuchowski N.A., Gallas B.D., Hillis S.L.: Multi-reader ROC studies with split-plot designs: a comparison of statistical methods. Acad Radiol 2012; 19: pp. 1508-1517.
17. Hillis S.L.: Simulation of unequal-variance binormal multireader ROC decision data: an extension of the Roe and Metz simulation model. Acad Radiol 2012; 19: pp. 1518-1528.
18. Li C., Glüer C.C., Eastell R., et. al.: Tree-structured subgroup analysis of receiver operating characteristic curves for diagnostic tests. Acad Radiol 2012; 19: pp. 1529-1536.