Investigation of Optimal Use of Computer-Aided Detection Systems

Rationale and Objectives

The aim of this study was to explore different computerized models (the “machine”) as a means to achieve optimal use of computer-aided detection (CAD) systems and to investigate whether these models can play a primary role in clinical decision making and possibly replace a clinician’s subjective decision for combining his or her own assessment with that provided by a CAD system.

Materials and Methods

Data previously collected from a fully crossed, multiple-reader, multiple-case observer study with and without CAD by seven observers asked to identify simulated small masses on two separate sets of 100 mammographic images (low-contrast and high-contrast sets; ie, low-contrast and high-contrast simulated masses added to random locations on normal mammograms) were used. This allowed testing two relative sensitivities between the observers and CAD. Seven models that combined detection assessments from CAD standalone, unaided read, and CAD-aided read (second read and concurrent read) were developed using the leave-one-out technique for training and testing. These models were personalized for each observer. Detection performance accuracies were analyzed using the area under a portion of the free-response receiver-operating characteristic curve (AUFC), sensitivity, and number of false-positives per image.

Results

For the low-contrast set, the use of computerized models resulted in significantly higher AUFCs compared to the unaided read mode for all readers, whereas the increased AUFCs between CAD-aided (second and concurrent reads; ie, decisions made by the readers) and unaided read modes were statistically significant for a majority, but not all, of the readers (four and five of the seven readers, respectively). For the high-contrast set, there were no significant trends in the AUFCs whether or not a model was used to combine the original reading modes. Similar results were observed when using sensitivity as the figure of merit. However, the average number of false-positives per image resulting from the computerized models remained the same as that obtained from the unaided read modes.

Conclusions

Individual computerized models (the machine) that combine image assessments from CAD standalone, unaided read, and CAD-aided read can increase detection performance compared to the reading done by the observer. However, relative sensitivity (ie, the difference in sensitivity between CAD standalone and unaided read) was a critical factor that determined incremental improvement in decision making, whether made by the observer or using computerized models.

Over the past two decades, computer-aided detection (CAD) systems have become more pervasive in clinical practice to assist radiologists in their decision making . These systems are intended to help reduce radiologist errors of detection in the identification of early-stage disease and in particular to improve the detection of early-stage cancer. Early-stage cancer typically produces no symptoms but may be revealed through imaging examination. Finding early-stage cancer in patient images or image data is a challenging task and, for radiologists, is akin to looking for a “needle in a haystack.” These challenges in detection are due to the often subtle distinction between early disease and normal tissue, the low disease prevalence in the population, or a combination thereof. Identification (or detection) and interpretation (or diagnosis) of imaging findings indicative of cancer depends on various factors, from technical to psychoperceptual . The human task of image interpretation is a major source of variability in disease detection compared to the technical aspect of imaging patients .

CAD was developed for the detection of a variety of cancers, including breast and lung cancer as well as colonic polyps. CAD devices generally perform automatic assessments of patients’ images or image data (eg, mammograms, computed tomographic scans, magnetic resonance imaging data) and present to radiologists areas that the CAD systems have determined as having the appearance of an abnormality (ie, an image finding). However, CAD system performance is variable and depends on the organ, disease, type of imaging finding, and so on. In mammography, although CAD excels at identifying microcalcifications, it is less effective at identifying masses . In general (ie, independent of the organ of interest), CAD systems also tend to err on the side of caution, presenting images with a large number of false-positive (FP) marks . Some marks are easily dismissed by a radiologist or physician interpreter, but other marks may result in clinical actions being taken (eg, an FP CAD mark on a mammogram may lead to an unnecessary additional imaging examination and/or biopsy). Even with these caveats, CAD systems are commonly used as an additional (or a companion) “pair of eyes” to a radiologist, who is otherwise alone in the interpretation of patient images or image data, unless double reading is performed by a colleague or associate. The value of CAD in actual clinical practice remains ambiguous. The published literature reveals many controversies in the field of CAD , and many authors recognize the need for additional clinical investigations to provide more definitive evidence of the effect of CAD systems on patient care and clinical outcomes.

Get Radiology Tree app to read full this article<

Materials and methods

Study Data

Get Radiology Tree app to read full this article<

Combination Scheme

Get Radiology Tree app to read full this article<

Statistical Analysis

Get Radiology Tree app to read full this article<

Results

Get Radiology Tree app to read full this article<

Table 1

Counts of Readers Presenting Increases (+), Statistically Significant Increases (+∗), Decreases (−), or Statistically Significant Decreases (−∗) in AUFC and Sensitivity per Each Reading Mode and Each Computerized Model

Low-contrast Set High-contrast Set dAUFC dSe dAUFC dSe + +∗ − −∗ + +∗ − −∗ + +∗ − −∗ + +∗ − −∗ C2 - R 6 4 1 7 7 7 7 5 CC - R 7 5 7 7 5 2 5 2 2 g(C,R) - R 7 7 7 6 6 1 6 1 g(C,C2) - R 6 3 1 7 5 6 1 1 2 6 1 g(C,CC) - R 7 7 7 7 7 4 7 3 g(R,C2) - R 7 6 7 5 4 3 4 3 g(R,CC) - R 7 3 7 5 3 4 3 4 g(C,R,C2) - R 7 7 7 6 6 2 1 6 1 g(C,R,CC) - R 7 7 7 7 7 4 7 3 g(C,R) - C2 5 2 2 3 4 3 3 4 3 4 1 g(C,C2) - C2 7 3 3 1 4 1 2 1 5 1 2 5 g(R,C2) - C2 7 3 2 1 5 1 7 4 1 6 1 g(C,R,C2) - C2 7 4 3 1 4 1 2 2 5 2 5 g(C,R) - CC 5 1 2 1 5 7 5 4 1 3 4 3 g(C,CC) - CC 7 4 4 2 3 1 6 5 1 6 1 g(R,CC) - CC 5 1 2 1 4 1 3 7 4 7 2 g(C,R,CC) - CC 7 7 5 1 2 6 5 1 6 1

AUFC, area under the free-response receiver-operating characteristic curve; CC, reader using computer-aided detection (CAD) concurrently; dAUFC, difference in AUFC; dSe, difference in sensitivity; R, reader-unaided read (without CAD); C2, reader using CAD as a second reader.

Get Radiology Tree app to read full this article<

Table 2

Multiple-reader, Multiple-case Analysis for the Low-contrast and High-contrast Sets

Low-contrast Set High-contrast Set Se Average Number of FPs per Image Se Average Number of FPs per Image C 73.5% (66.5–81.1) 1.49 (1.38–1.60) 70.9% (64.0–77.6) 1.34 (1.19–1.50) R 42.2% (29.3–55.5) 0.43 (0.19–0.75) 75.3% (65.8–83.8) 0.16 (0.07–0.27) C2 54.0% (40.4–66.3) 0.56 (0.28–0.88) 78.9% (70.4–86.8 0.22 (0.11–0.34) CC 59.3% (46.5–71.7) 0.50 (0.23–0.81) 79.7% (71.7–86.5) 0.26 (0.13–0.40) g(C,R,C2) 59.5% (49.1–69.8) Identical to above (R) 80.1% (71.2–88.6) Identical to above (R) g(C,R,CC) 65.2% (52.6–75.1) Identical to above (R) 83.6% (75.3–90.8) Identical to above (R)

Overall sensitivity (Se) (per image grid unit) and average number of false-positives (FPs) per image for C (computer-aided detection [CAD]–standalone read mode); for detection decision made by the reader (R, reader-unaided read mode; C2, CAD-aided read mode when CAD is used as a second reader; and CC, CAD-aided read mode when CAD is used concurrently); for detection decision made by the machine (here is presented two of the computerized models; those that combine all reading scores), that is, g(C, R,C2) and g(C, R, CC). Estimates are given with associated 95% confidence intervals.

Get Radiology Tree app to read full this article<

Discussion

General Experimental Findings

Get Radiology Tree app to read full this article<

Implementation and Adoption Challenges

Get Radiology Tree app to read full this article<

References

1. Doi K.: Computer-aided diagnosis in medical imaging: historical review, current status and future potential. Comput Med Imaging Graph 2007; 31: pp. 198-211.
2. Hoffman R., Fiore S.M.: Perceptual (re)learning: a leverage point for human-centered computing. IEEE Intell Syst 2007; 22: pp. 79-83.
3. Robinson P.J.A.: Radiology’s Achilles’ heel: error and variation in the interpretation of the Röntgen image. Br J Radiol 1997; 70: pp. 1085-1098.
4. Petrick N., Sahiner B., Chan H.-P., Helvie M.A., et. al.: Breast cancer detection: evaluation of a mass-detection algorithm for computer-aided diagnosis—experience in 263 patients. Radiology 2002; 224: pp. 217-224.
5. Ellis R.L., Meade A.A., Mathiason M.A., et. al.: Evaluation of computer-aided detection systems in the detection of small invasive breast carcinoma. Radiology 2007; 245: pp. 88-94.
6. Zheng B., Hardesty L.A., Poller W.R., Sumkin J.H., Golla S.: Mammography with computer-aided detection: reproducibility assessment—initial experience. Radiology 2003; 228: pp. 58-62.
7. Astley S.M.: Computer-based detection and prompting of mammographic abnormalities. Br J Radiol 2004; 77: pp. S194-S200.
8. Fenton J.J., Taplin S.H., Carney P.A., et. al.: Influence of computer-aided detection on performance of screening mammography. N Engl J Med 2007; 356: pp. 1399-1409.
9. Vasko C. Utilization and reimbursements and staffing, oh my! Where are we today, and what lies ahead? Available at: http://www.imagingeconomics.com/issues/articles/2008-08_01.asp . Accessed August 19, 2009.
10. Eisenberg R.L.: Radiology and the law: malpractice and other issues.2003.SpringerNew York
11. Smith-Bindman R., Miglioretti D.L., Rosenberg R., et. al.: Physician workload in mammography. AJR Am J Roentgenol 2008; 190: pp. 526-532.
12. Bhargavan M., Kaye A.H., Forman H.P., et. al.: Workload of radiologists in United States in 2006-2007 and trends since 1991-1992. Radiology 2009; 252: pp. 458-467.
13. Zheng B., Chough D., Ronald P., et. al.: Actual versus intended use of CAD systems in the clinical environment. Proc SPIE 2006; 6146: pp. 614602.
14. Paquerault S., Samuelson F.W., Petrick N., et. al.: Investigation of reading mode and relative sensitivity as factors that influence reader performance when using computer-aided detection software. Acad Radiol 2009; 16: pp. 1095-1107.
15. Kobayashi T., Xu X.W., MacMahon H., et. al.: Effect of a computer-aided diagnosis scheme on radiologists’ performance in detection of lung nodules on radiographs. Radiology 1996; 199: pp. 843-848.
16. Jiang Y., Metz C.E.: A quadratic model for combining quantitative diagnostic assessments from radiologist and computer in computer-aided diagnosis. Acad Radiol 2006; 13: pp. 140-151.
17. Fierz W.: Challenge of personalized health care: to what extent is medicine already individualized and what are the future trends?. Med Sci Monit 2004; 10: pp. RA111-RA123.
18. Samuelson F.W., Petrick N.: Comparing image detection algorithms using resampling. Proc IEEE Int Symp Biomed Imaging 2006; pp. 1312-1315.
19. Samuelson F.W., Petrick N., Paquerault S.: Advantages and examples of resampling for CAD evaluation. Proc IEEE Int Symp Biomed Imaging 2007; pp. 492-495.
20. Brem R.F.: Clinical versus research approach to breast cancer detection with CAD: where are we now?. AJR Am J Roentgenol 2007; 188: pp. 234-235.
21. Brem R.F.: Blinded comparison of computer-aided detection with human second reading in screening mammography: the importance of the question and the critical numbers game. AJR Am J Roentgenol 2007; 189: pp. 1142-1144.
22. Gur D., Sumkin J.H.: CAD in screening mammography. AJR Am J Roentgenol 2006; 187: pp. 1474.
23. Miglioretti D.L., Smith-Bindman R., Abraham L., et. al.: Radiologist characteristics associated with interpretive performance of diagnostic mammography. J Natl Cancer Inst 2007; 99: pp. 1854-1863.
24. Taylor P., Potts H.W.: Computer aids and human second reading as interventions in screening mammography: two systematic reviews to compare effects on cancer detection and recall rate. Eur J Cancer 2008; 44: pp. 798-807.
25. Chan H.-P., Sahiner B., Wagner R.F., et. al.: Classifier design for computer-aided diagnosis: effects of finite sample size on the mean performance of classical and neural network classifiers. Med Phys 1999; 26: pp. 2654-2668.
26. Wagner R.F., Beiden S.V., Metz C.E.: Continuous versus categorical data for ROC analysis: some quantitative considerations. Acad Radiol 2001; 8: pp. 328-334.

Investigation of Optimal Use of Computer-Aided Detection Systems

Rationale and Objectives

Materials and Methods

Results

Conclusions

Materials and methods

Study Data

Combination Scheme

Statistical Analysis

Results

Discussion

General Experimental Findings

Implementation and Adoption Challenges

References

Further Reading

14-Mile Hike

Acute Mesenteric Ischemia Induced by Ligation of Porcine Superior Mesenteric Vein

Anatomical Evaluation of the Dural Sinuses in the Region of the Torcular Herophili Using Three Dimensional CT Venography