Evaluating Imaging and Computer-aided Detection and Diagnosis Devices at the FDA

This report summarizes the Joint FDA-MIPS Workshop on Methods for the Evaluation of Imaging and Computer-Assist Devices. The purpose of the workshop was to gather information on the current state of the science and facilitate consensus development on statistical methods and study designs for the evaluation of imaging devices to support US Food and Drug Administration submissions. Additionally, participants expected to identify gaps in knowledge and unmet needs that should be addressed in future research. This summary is intended to document the topics that were discussed at the meeting and disseminate the lessons that have been learned through past studies of imaging and computer-aided detection and diagnosis device performance.

In this report, we summarize the Joint FDA-MIPS Workshop on Methods for the Evaluation of Imaging and Computer-Assist Devices, held on July 14, 2010. The purpose of this workshop was to gather information and facilitate consensus on the current state of the science of statistical methods and study designs for the evaluation of imaging devices and computer-aided detection and diagnosis (CAD) systems in the hands of clinicians, the image readers, to support US Food and Drug Administration (FDA) submissions. We hope that this summary serves as a stand-alone starting point and overview for investigators who are interested in reader studies evaluating imaging technology to support FDA submissions.

Additionally, we identified gaps in knowledge and unmet needs that should be addressed in future research. This document summarizes the topics that were discussed and the lessons learned from past studies of readers using imaging devices and CAD systems. Our primary goal is to improve device evaluations submitted to the FDA, that is, to make them more meaningful and powerful, thereby reducing the resources required of the FDA and industry to make important and clinically useful devices and tools available for improved patient care.

The workshop invitations targeted a multidisciplinary group of leaders in the field of medical imaging evaluation who have somehow been involved in the FDA approval and clearance of imaging devices: statisticians, clinicians, imaging physicists, mathematicians, and computer scientists from government, academia, and industry. All told, there were 22 attendees from academia, 11 from industry, 31 from the FDA, five from other government agencies including the National Institutes of Health, and 12 clinicians. Most attendees had either attended or participated in an FDA advisory panel meeting to judge the safety and effectiveness of an imaging device seeking premarket approval.

The meeting was divided into three sessions, each with two invited presenters followed by a panel that included an additional invited expert in the field. The first session was “Statistical Perspectives,” with Alicia Toledano, ScD (Statistics Collaborative, Inc), and Nancy Obuchowski, PhD (Cleveland Clinic Foundation), as the key presenters and Berkman Sahiner, PhD (FDA), on the panel. The second session was “Clinical Perspectives,” with Carl D’Orsi, MD (Emory University), and Margarita Zuley, MD (University of Pittsburgh), as key presenters and Barbara McNeil, MD, PhD (Harvard Medical School), as the third member on the panel. The final session was “Developer Perspectives,” with Heang-Ping Chan, PhD (University of Michigan), and Maryellen Giger, PhD (University of Chicago), as key presenters and David Gur, ScD (University of Pittsburgh), on the panel. The sessions were moderated by Elizabeth Krupinski, PhD (University of Arizona), and Lori Dodd, PhD (National Institute of Allergy and Infectious Diseases), who also provided a summary titled “State of Consensus and Plans for the Future” at the end of the meeting. Kyle Myers, PhD, and Brandon Gallas, PhD, of the FDA organized and provided the introductory background and guidance for the workshop.

Because the main focus of the workshop was the FDA approval and clearance of imaging devices, the views expressed there may have been limited or biased. The discussions at this workshop were focused on getting imaging devices approved and did not define what is or is not a valuable contribution to the scientific literature. Given the high stakes, manufacturers are sometimes reluctant to use cutting-edge study designs and analyses, preferring more established and traditional designs and analyses. In this setting, manufacturers typically report sensitivity and specificity, receiver operating characteristic (ROC) curves and areas under ROC curves (AUCs). However, they are not sure which of these limited measures is the “right” one (the one that will get their devices approved) and which measure they should use to size their studies. Manufacturers also often struggle to balance study biases against study burden (complexity, time, and costs) and clinical reality against experimental tractability and abstraction.

Get Radiology Tree app to read full this article<

Background

Get Radiology Tree app to read full this article<

Diagnostic Decisions and Performance

Get Radiology Tree app to read full this article<

Reader Variability

Get Radiology Tree app to read full this article<

Phases of Evaluation

Get Radiology Tree app to read full this article<

Premarket versus postmarket

Get Radiology Tree app to read full this article<

Area under the curve versus sensitivity and specificity

Get Radiology Tree app to read full this article<

Reader study designs and analyses

Get Radiology Tree app to read full this article<

Enrichment

Get Radiology Tree app to read full this article<

Pairing Cases and Readers Across Modalities

Get Radiology Tree app to read full this article<

Var(A−B)=Var(A)+Var(B)−2Corr(A,B)Var(A)Var(B)−−−−−−−−−−−−−√. Var

(

−

)

Var

(

)

Var

(

)

−

Corr

(

)

Var

(

)

Var

(

)

The higher the correlation, the lower the variance of the difference.

Get Radiology Tree app to read full this article<

Subgroups

Get Radiology Tree app to read full this article<

Sequential Study Design Versus Independent or Crossover Study Design

Get Radiology Tree app to read full this article<

Lesion Localization

Get Radiology Tree app to read full this article<

Topics deserving more research and attention

Get Radiology Tree app to read full this article<

Pilot Studies and Reader Training

Get Radiology Tree app to read full this article<

Sensitivity for a “Clinically Relevant” Specificity or Specificity Range

Get Radiology Tree app to read full this article<

Linking Binary and Multilevel Ratings

Get Radiology Tree app to read full this article<

Surrogate End Points

Get Radiology Tree app to read full this article<

Bench Testing and Simulated Images and Lesions

Get Radiology Tree app to read full this article<

CAD Changes

Get Radiology Tree app to read full this article<

Data Sets

Get Radiology Tree app to read full this article<

Conclusions

Get Radiology Tree app to read full this article<

References

1. Determination of safety and effectiveness. 21 CFR § 860 (2011).
2. Premarket approval application. 21 CFR § 814 (2011).
3. US Food and Drug Administration. Radiological Devices Panel meeting: computer-aided detection devices. Available at: http://www.fda.gov/ohrms/dockets/ac/08/minutes/2008%134349m1.pdf . Accessed November 17, 2011.
4. American College of Radiology: Breast Imaging Reporting and Data System (BI-RADS).2003.American College of RadiologyReston, VA
5. Metz C.E.: ROC methodology in radiologic imaging. Invest Radiol 1986; 21: pp. 720-733.
6. Green D.M., Swets J.A.: Signal detection theory and psychophysics.1966.John WileyNew York
7. Lusted L.B.: Introduction to medical decision making.1968.Charles C. ThomasSpringfield, IL
8. McNeil B.J., Keller E., Adelstein S.J.: Primer on certain elements of medical decision making. N Engl J Med 1975; 293: pp. 211-215.
9. Metz C.E.: Basic principles of ROC analysis. Semin Nucl Med 1978; 8: pp. 283-298.
10. Patton D.D.: Introduction to clinical decision making. Semin Nucl Med 1978; 8: pp. 273-282.
11. Weinstein M.C., Fineberg H.V.: Clinical decision analysis.1980.SaundersPhiladelphia, PA
12. Hanley J.A., McNeil B.J.: The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 1982; 143: pp. 29-36.
13. Metz C.E.: Some practical issues of experimental design and data analysis in radiological ROC studies. Invest Radiol 1989; 24: pp. 234-245.
14. Gur D., Bandos A.I., Rockette H.E.: Comparing areas under receiver operating characteristic curves: potential impact of the “last” experimentally measured operating point. Radiology 2008; 247: pp. 12-15.
15. Pesce L.L., Metz C.E., Berbaum K.S.: On the convexity of ROC curves estimated from radiological test results. Acad Radiol 2010; 17: pp. 960-968.
16. Jiang Y., Metz C.E.: BI-RADS data should not be used to estimate ROC curves. Radiology 2010; 256: pp. 29-31.
17. Metz C.E.: Fundamental ROC analysis.Beutel J.Kundel H.L.Van Metter R.L.Handbook of medical imaging, vol. 1: physics and psychophysics.2000.SPIE PressBellingham, WA:pp. 751-769.
18. Obuchowski N.A.: ROC analysis. AJR Am J Roentgenol 2005; 184: pp. 364-372.
19. International Commission of Radiation Units and Measurements: Receiver operating characteristic analysis in medical imaging. ICRU Report 79.2008.International Commission of Radiation Units and MeasurementsBethesda, MD
20. Elmore J.G., Wells C.K., Lee C.H., et. al.: Variability in radiologists’ interpretations of mammograms. N Engl J Med 1994; 331: pp. 1493-1499.
21. Beam C., Layde P.M., Sullivan D.C.: Variability in the interpretation of screening mammograms by US radiologists. Arch Intern Med 1996; 156: pp. 209-213.
22. D’Orsi C.J., Swets J.A.: Variability in the interpretation of mammograms. N Engl J Med 1995; 332: pp. 1172.
23. Gallas B.D., Pennello G.A., Myers K.J.: Multi-reader multi-case variance analysis for binary data. J Opt Soc Am A 2007; 24: pp. B70-B80.
24. Gallas B.D., Brown D.G.: Reader studies for validation of CAD systems. Neural Networks 2008; 21: pp. 387-397.
25. Obuchowski N.A.: Reducing the number of reader interpretations in MRMC studies. Acad Radiol 2009; 16: pp. 209-217.
26. University of Chicago. Metz ROC software. Available at: http://metz-roc.uchicago.edu . Accessed November 17, 2011.
27. University of Iowa Medical Image Perception Laboratory. Home page. Available at: http://perception.radiology.uiowa.edu . Accessed November 17, 2011.
28. Cleveland Clinic, Quantitative Health Sciences. Research activities: ROC analysis. Available at: http://www.bio.ri.ccf.org/html/rocanalysis.html . Accessed November 17, 2011.
29. Bankier A.A., Levine D., Halpern E.F., et. al.: Consensus interpretation in imaging research: is there a better way?. Radiology 2010; 257: pp. 14-17.
30. Hillman B.J.: ACRIN—lessons learned in conducting multi-center trials of imaging and cancer. Cancer Imaging 2005; 5: pp. S97-S101.
31. Roe C.A., Metz C.E.: Variance-component modeling in the analysis of receiver operating characteristic (ROC) index estimates. Acad Radiol 1997; 4: pp. 587-600.
32. Rockette H.E., Campbell W.L., Britton C.A., et. al.: Empiric assessment of parameters that affect the design of multireader receiver operating characteristic studies. Acad Radiol 1999; 6: pp. 723-729.
33. Wagner R.F., Beiden S.V., Campbell G., et. al.: Assessment of medical imaging and computer-assist systems: lessons from recent experience. Acad Radiol 2002; 9: pp. 1264-1277.
34. Wagner R.F., Beam C.A., Beiden S.V.: Reader variability in mammography and its implications for expected utility over the population of readers and cases. Med Decis Making 2004; 24: pp. 561-572.
35. Zhou X.-H., Obuchowski N.A., McClish D.K.: Statistical methods in diagnostic medicine.2002.John WileyHoboken, NJ
36. Wagner R.F., Metz C.E., Campbell G.: Assessment of medical imaging systems and computer aids: a tutorial review. Acad Radiol 2007; 14: pp. 723-748.
37. Jiang Y., Miglioretti D.L., Metz C.E., et. al.: Breast cancer detection rate: designing imaging trials to demonstrate improvements. Radiology 2007; 243: pp. 360-367.
38. Barlow W.E., Chi C., Carney P.A., et. al.: Accuracy of screening mammography interpretation by characteristics of radiologists. J Natl Cancer Inst 2004; 96: pp. 1840-1850.
39. Taplin S.H., Rutter C.M., Lehman C.D.: Testing the effect of computer-assisted detection on interpretive performance in screening mammography. AJR Am J Roentgenol 2006; 187: pp. 1475-1482.
40. Otero H.J., Rybicki F.J., Greenberg D., et. al.: Cost-effective diagnostic cardiovascular imaging: when does it provide good value for the money?. Int J Cardiovasc Imaging 2010; 26: pp. 605-612.
41. Tosteson A.N.A., Stout N.K., Fryback D.G., et. al.: Cost-effectiveness of digital mammography breast cancer screening. Ann Intern Med 2008; 148: pp. 1-10.
42. Shaw L.J., Wilson P.W.F., Hachamovitch R., et. al.: Improved near-term coronary artery disease risk classification with gated stress myocardial perfusion SPECT. JACC Cardiovasc Imaging 2010; 3: pp. 1139-1148.
43. Niklason L.T., Christian B.T., Niklason L.E., et. al.: Digital tomosynthesis in breast imaging. Radiology 1997; 205: pp. 399-406.
44. Gur D., Abrams G.S., Chough D.M., et. al.: Digital breast tomosynthesis: observer performance study. AJR Am J Roentgenol 2009; 193: pp. 586-591.
45. Pisano E.D., Gatsonis C., Hendrick E., et. al.: Diagnostic performance of digital versus film mammography for breast-cancer screening. N Engl J Med 2005; 353: pp. 1773-1783.
46. Gur D., Rockette H.E., Armfield D.R., et. al.: Prevalence effect in a laboratory environment. Radiology 2003; 228: pp. 10-14.
47. Gur D., Bandos A.I., Fuhrman C.R., et. al.: The prevalence effect in a laboratory environment: changing the confidence ratings. Acad Radiol 2007; 14: pp. 49-53.
48. Franke R.H., Kaul J.D.: The Hawthorne experiments: first statistical interpretation. Am Sociol Rev 1978; 43: pp. 623-643.
49. McCarney R., Warner J., Iliffe S., et. al.: The Hawthorne effect: a randomised, controlled trial. BMC Med Res Methodol 2007; 7: pp. 30.
50. Gur D., Bandos A.I., Cohen C.S., et. al.: The “laboratory” effect: comparing radiologists’ performance and variability during prospective clinical and laboratory mammography interpretations. Radiology 2008; 249: pp. 47-53.
51. US Food and Drug Administration. Summary of safety and effectiveness data: GE FFDM (P990066). Available at: http://www.accessdata.fda.gov/cdrh_docs/pdf/P990066b.pdf . Accessed November 17, 2011.
52. US Food and Drug Administration. Summary of safety and effectiveness data: Fischer FFDM (P010017). Available at: http://www.accessdata.fda.gov/cdrh_docs/pdf/P010017b.pdf . Accessed November 17, 2011.
53. US Food and Drug Administration. Summary of safety and effectiveness data: Hologic FFDM (P010025). Available at: http://www.accessdata.fda.gov/cdrh_docs/pdf/P010025b.pdf . Accessed November 17, 2011.
54. US Food and Drug Administration. Summary of safety and effectiveness data: Fuji FFDM (P050014). Available at: http://www.accessdata.fda.gov/cdrh_docs/pdf5/P050014b.pdf . Accessed November 17, 2011.
55. Samuelson F., Gallas B.D., Myers K.J., et. al.: The importance of ROC data. Acad Radiol 2010; 18: pp. 257-258.
56. Kundel H.L., Revesz G., Stauffer H.M.: Evaluation of a television image processing system. Invest Radiol 1968; 3: pp. 44-50.
57. Dean J.C., Ilvento C.C.: Improved cancer detection using computer-aided detection with diagnostic and screening mammography: prospective study of 104 cancers. AJR Am J Roentgenol 2006; 187: pp. 20-28.
58. Lewin J.M.: Full-field digital mammography. A candid assessment. Diagn Imaging 1999; 21: pp. 40-45.
59. Cole E., Pisano E.D., Brown M., et. al.: Diagnostic accuracy of Fischer Senoscan digital mammography versus screen-film mammography in a diagnostic mammography population. Acad Radiol 2004; 11: pp. 879-886.
60. Pinsky P, Gallas BD. Enriched designs for assessing predictive performance—analysis of bias and variance. Stat Med. In press.
61. Horvitz D.G., Thompson D.J.: A generalization of sampling without replacement from a finite universe. J Am Stat 1952; 47: pp. 663-685.
62. Katz J., Zeger S.L.: Estimation of design effects in cluster surveys. Ann Epidemiol 1994; 4: pp. 295-301.
63. Zhou X.H.: Correcting for verification bias in studies of a diagnostic test’s accuracy. Stat Methods Med Res 1998; 7: pp. 337-353.
64. Alonzo T.A., Pepe M.S.: Assessing accuracy of a continuous screening test in the presence of verification bias. J Roy Stat Soc C-App 2005; 54: pp. 173-190.
65. He H., Lyness J.M., McDermott M.P.: Direct estimation of the area under the receiver operating characteristic curve in the presence of verification bias. Stat Med 2009; 28: pp. 361-376.
66. Swets J.A., Pickett R.M.: Evaluation of diagnostic systems: methods from signal detection theory.1982.Academic PressNew York
67. Dorfman D.D., Berbaum K.S., Metz C.E.: Receiver operating characteristic rating analysis: generalization to the population of readers and patients with the jackknife method. Invest Radiol 1992; 27: pp. 723-731.
68. Roe C.A., Metz C.E.: Dorfman-Berbaum-Metz method for statistical analysis of multireader, multimodality receiver operating characteristic (ROC) data: validation with computer simulation. Acad Radiol 1997; 4: pp. 298-303.
69. Obuchowski N.A., Beiden S.V., Berbaum K.S., et. al.: Multireader, multicase receiver operating characteristic analysis: an empirical comparison of five methods. Acad Radiol 2004; 11: pp. 980-995.
70. Gallas B.D.: One-shot estimate of MRMC variance: AUC. Acad Radiol 2006; 13: pp. 353-362.
71. Obuchowski N.A.: Multireader receiver operating characteristic studies: a comparison of study designs. Acad Radiol 1995; 2: pp. 709-716.
72. US Food and Drug Administration. Summary of safety and effectiveness data: Hologic DBT (P080003). Available at: http://www.accessdata.fda.gov/cdrh_docs/pdf8/P080003b.pdf . Accessed November 17, 2011.
73. Kobayashi T., Xu X.W., MacMahon H., et. al.: Effect of a computer-aided diagnosis scheme on radiologists’ performance in detection of lung nodules on radiographs. Radiology 1996; 199: pp. 843-848.
74. Beiden S.V., Wagner R.F., Doi K., et. al.: Independent versus sequential reading in ROC studies of computer-assist modalities: analysis of components of variance. Acad Radiol 2002; 9: pp. 1036-1043.
75. Hadjiiski L., Chan H.-P., Sahiner B., et. al.: Improvement in radiologists’ characterization of malignant and benign breast masses on serial mammograms with computer-aided diagnosis: an ROC study. Radiology 2004; 233: pp. 255-265.
76. Obuchowski N.A., Meziane M., Dachman A.H., et. al.: What’s the control in studies measuring the effect of computer-aided detection (CAD) on observer performance?. Acad Radiol 2010; 17: pp. 761-767.
77. Obuchowski N.A., Mazzone P.J., Dachman A.H.: Bias, underestimation of risk, and loss of statistical power in patient-level analyses of lesion detection. Eur Radiol 2010; 20: pp. 584-594.
78. Bunch P.C., Hamilton J.F., Sanderson G.K., et. al.: A free-response approach to the measurement and characterization of radiographic-observer performance. J Appl Photographic Eng 1978; 4: pp. 166-171.
79. Egan J.P., Greenberg G.Z., Schulman A.I.: Operating characteristics, signal detectability, and the method of free response. J Acoust Soc Am 1961; 33: pp. 993-1007.
80. Chakraborty D.P.: Maximum likelihood analysis of free-response receiver operating characteristic (FROC) data. Med Phys 1989; 16: pp. 561-568.
81. Starr S.J., Metz C.E., Lusted L.B., et. al.: Visual detection and localization of radiographic images. Radiology 1975; 116: pp. 533-538.
82. Swensson R.G.: Unified measurement of observer performance in detecting and localizing target objects on images. Med Phys 1996; 23: pp. 1709-1725.
83. Obuchowski N.A.: Nonparametric analysis of clustered ROC curve data. Biometrics 1997; 53: pp. 567-578.
84. Chakraborty D.P., Berbaum K.S.: Observer studies involving detection and localization: modeling, analysis and validation. Med Phys 2004; 31: pp. 2313-2330.
85. Chakraborty D.P.: Analysis of location specific observer performance data: validated extensions of the jackknife free-response (JAFROC) method. Acad Radiol 2006; 13: pp. 1187-1193.
86. Edwards D.C., Kupinski M.A., Metz C.E., et. al.: Maximum likelihood fitting of FROC curves under an initial-detection-and-candidate-analysis model. Med Phys 2002; 29: pp. 2861-2870.
87. Obuchowski N.A., Lieber M.L., Powell K.A.: Data analysis for detection and localization of multiple abnormalities with application to mammography. Acad Radiol 2000; 7: pp. 516-525.
88. Zanca F., Chakraborty D.P., Marchal G., et. al.: Consistency of methods for analysing location-specific data. Radiat Prot Dosimetry 2010; 139: pp. 52-56.
89. Chakraborty D.P.: Validation and statistical power comparison of methods for analyzing free-response observer performance studies. Acad Radiol 2008; 15: pp. 1554-1566.
90. Good W.F., Abrams G.S., Catullo V.J., et. al.: Digital breast tomosynthesis: a pilot observer study. AJR Am J Roentgenol 2008; 190: pp. 865-869.
91. Babinski P.J., Babinski B.S.: Adaptability through cross-training in radiology departments. Radiol Manage 2011; 33: pp. 45-49.
92. Yablon C.M., Wu J.S., Slanetz P.J., et. al.: A report on the current status of grand rounds in radiology residency programs in the United States. Acad Radiol 2011; 18: pp. 1593-1597.
93. Gur D., Sumkin J.H., Hardesty L.A., et. al.: Recall and detection rates in screening mammography. Cancer 2004; 100: pp. 1590-1594.
94. Jiang Y., Metz C.E., Nishikawa R.M.: A receiver operating characteristic partial area index for highly sensitive diagnostic tests. Radiology 1996; 201: pp. 745-750.
95. Obuchowski N.A., McClish D.K.: Sample size determination for diagnostic accuracy studies involving binormal ROC Curve indices. Stat Med 1997; 16: pp. 1529-1542.
96. Baker S.G., Pinsky P.F.: A proposed design and analysis for comparing digital and analog mammography: special ROC methods for cancer screening. J Am Stat Assoc 2001; 96: pp. 421-428.
97. Zhang D.D., Zhou X.-H., Freeman D.H., et. al.: A non-parametric method for the comparison of partial areas under ROC curves and its application to large health care data sets. Stat Med 2002; 21: pp. 701-715.
98. Wang F., Gatsonis C.A.: Hierarchical models for ROC curve summary measures: design and analysis of multi-reader, multi-modality studies of medical tests. Stat Med 2008; 27: pp. 243-256.
99. Li C.-R., Liao C.-T., Liu J- P.: A non-inferiority test for diagnostic accuracy based on the paired partial areas under ROC curves. Stat Med 2008; 27: pp. 1762-1776.
100. Pepe M.S.: The statistical evaluation of medical tests for classification and prediction.2003.Oxford University PressOxford, United Kingdom
101. Hadjiiski L., Sahiner B., Helvie M.A., et. al.: Breast masses: computer-aided diagnosis with serial mammograms. Radiology 2006; 240: pp. 343-356.
102. Berbaum K.S., Dorfman D.D., Franken J.E.A., et. al.: An empirical comparison of discrete ratings and subjective probability ratings. Acad Radiol 2002; 9: pp. 756-763.
103. Jiang Y., Nishikawa R.M., Schmidt R.A., et. al.: Improving breast cancer diagnosis with computer-aided diagnosis. Acad Radiol 1999; 6: pp. 22-33.
104. Horsch K., Giger M.L., Vyborny C.J., et. al.: Classification of breast lesions with multimodality computer-aided diagnosis: observer study results on an independent clinical data set. Radiology 2006; 240: pp. 357-368.
105. Horsch K., Giger M.L., Metz C.E.: Potential effect of different radiologist reporting methods on studies showing benefit of CAD. Acad Radiol 2008; 15: pp. 139-152.
106. Armato S.G., Roberts R.Y., Kocherginsky M., et. al.: Assessment of radiologist performance in the detection of lung nodules: dependence on the definition of “truth”. Acad Radiol 2009; 16: pp. 28-38.
107. Miller D.P., O’Shaughnessy K.F., Wood S.A., et. al.: Gold standards and expert panels: a pulmonary nodule case study with challenges and solutions. SPIE Proc Med Imaging 2004; 5372: pp. 173-184.
108. US Food and Drug Administration. Radiological Devices Panel. Available at: http://www.fda.gov/ohrms/dockets/ac/08/transcripts/2008-4349t1-05.pdf . Accessed November 17, 2011.
109. US Food and Drug Administration. Radiological Devices Panel: two draft guidance documents for computer-aided-detection (CAD) devices. Available at: http://www.fda.gov/downloads/AdvisoryCommittees/CommitteesMeetingMaterials/MedicalDevices/MedicalDevicesAdvisoryCommittee/RadiologicalDevicesPanel/UCM197419.pdf . Accessed November 17, 2011.
110. Obuchowski N.A.: Predicting readers’ diagnostic accuracy with a new CAD algorithm. Acad Radiol 2011; 18: pp. 1412-1419.
111. Chan H.-P., Sahiner B., Wagner R.F., et. al.: Classifier design for computer-aided diagnosis: effects of finite sample size on the mean performance of classical and neural network classifiers. Med Phys 1999; 26: pp. 2654-2668.
112. Sahiner B., Chan H.-P., Petrick N., et. al.: Feature selection and classifier performance in computer-aided diagnosis: the effect of finite sample size. Med Phys 2000; 27: pp. 1509-1522.
113. Kupinski M.A., Giger M.L.: Feature selection with limited datasets. Med Phys 1999; 26: pp. 2176-2182.
114. Simon R., Radmacher M.D., Dobbin K., et. al.: Pitfalls in the use of DNA microarray data for diagnostic and prognostic classification. J Natl Cancer Inst 2003; 95: pp. 14-18.

Evaluating Imaging and Computer-aided Detection and Diagnosis Devices at the FDA

Background

Diagnostic Decisions and Performance

Reader Variability

Phases of Evaluation

Premarket versus postmarket

Area under the curve versus sensitivity and specificity

Reader study designs and analyses

Enrichment

Pairing Cases and Readers Across Modalities

Subgroups

Sequential Study Design Versus Independent or Crossover Study Design

Lesion Localization

Topics deserving more research and attention

Pilot Studies and Reader Training

Sensitivity for a “Clinically Relevant” Specificity or Specificity Range

Linking Binary and Multilevel Ratings

Surrogate End Points

Bench Testing and Simulated Images and Lesions

CAD Changes

Data Sets

Conclusions

References

Further Reading

A New Approach to Assess Intracranial White Matter Abnormalities in Glaucoma Patients

Baseline SUVmax at PET-CT in Stage IIIA Non-small-cell Lung Cancer Patients Undergoing Surgery after Neoadjuvant Therapy

Correlations between SUVmax and Expression of GLUT1 and Growth Factors Inducing Lymphangiogenesis