Rationale and Objectives
Computer-aided diagnosis (CAD) has been developed to ensure that the radiologist considers suspect focal opacities that may represent cancer in chest radiography. Although CAD was not developed to counteract the satisfaction of search (SOS) effect, it may be an effective intervention to do so. The objective of this study is to determine whether an idealized CAD can reduce SOS effects in chest radiography.
Materials and Methods
Fifty-seven chest radiographs, half of which demonstrated diverse, native abnormalities were read twice by 16 observers, once with and once without the addition of a simulated pulmonary nodule. Simulated CAD prompts were provided during the interpretation, which unerringly pointed to the added simulated nodule. Area under the ROC curve for detecting the native abnormalities was estimated for each observer in each treatment condition. In addition to testing for the SOS effect in the presence of CAD prompts, results were compared to those of a previous SOS study.
Results
Significantly more nodules were reported in the SOS with CAD experiment than in the original SOS experiment (49 versus 43, P < .01). An SOS effect was found even when CAD prompts were provided; ROC areas for detecting native abnormalities were reduced with added nodules [0.68 versus 0.65, P (one-tailed) < .05]. Comparison of the current experiment with CAD and the previous SOS experiments failed to show a significant difference of the magnitude of the SOS effect ( P = .52). The threshold for reporting was more conservative with CAD prompts than in SOS studies ( P = .052).
Conclusion
Our results indicate that the CAD prompts, even those that always point to their target lesion without false-positive error, fail to counteract SOS in chest radiography. The stricter decision thresholds with CAD prompts may indicate less visual search for native abnormalities.
Satisfaction of search (SOS) occurs when a lesion is “missed” after detecting another lesion in the same image. An SOS effect in chest radiology, defined operationally as a reduced accuracy in detecting native abnormalities on chest radiographs in the presence of simulated pulmonary nodules, has been demonstrated ( ).
A promising approach to prevent SOS errors in medical imaging diagnosis is computer-aided diagnosis (CAD). CAD has been used to find tumors on chest radiographs, chest computed tomography, and mammograms. The goal of computer-aided diagnosis for nodule detection is to ensure that the interpreting physician considers regions that have a high probability of disease. An obvious radiology examination in which to study the effects of CAD on SOS is chest radiography. Not only have SOS effects on the detection of lung nodules and other abnormalities been quantified ( ), CAD systems for nodule detection in chest imaging are also well developed and commercially available ( ).
Get Radiology Tree app to read full this article<
Materials and methods
Experimental Conditions
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Case Sample
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Procedure
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
“Some of the cases you will read will be completely normal and others will contain one or more clinically significant abnormalities. We are interested only in those features that represent significant abnormality. Avoid “over calling”. A list of features to generally ignore is posted next to the alternator and is also included in your scoring booklet. There is a separate response sheet for each case presented. The top of the response sheet lists the alternator position, the patient’s age and sex, and sometimes a comment about some attribute of the film we would like you to ignore. At the bottom of the response sheet there is a photostat representing a PA view of the case (and sometimes a lateral view). On some PA views there will be a CAD indication of the location of a potential pulmonary nodule (white dot). Please review this area on the film and reject or accept this indication by checking the appropriate box. If you accept the CAD indication, circle your confidence that the feature is abnormal. Otherwise, leave the CAD section blank. Continue describing other abnormal features on the film. As you do, number them and draw their locations on the photostat, give a likely diagnosis if possible and rate your confidence (one to four) that the feature is abnormal. The rating categories are, 1) suspicious, but probably normal; 2) possibly abnormal; 3) probably abnormal; and 4) definitely abnormal. If you find no abnormal features, check the box on the response sheet indicating, “normal features, nothing to report” and proceed to the next case.”
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Scoring
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Comparison With Earlier Experiments
Get Radiology Tree app to read full this article<
Data Analysis
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Results
Effect of Simulated CAD on Reporting Nodules
Get Radiology Tree app to read full this article<
Contaminated Binormal Areas for Detecting Native Abnormalities
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Thresholds for Detecting Native Abnormalities
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Discussion
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
References
1. Berbaum K.S., Franken E.A., Dorfman D.D., et. al.: Satisfaction of search in diagnostic radiology. Invest Radiol 1990; 25: pp. 133-140.
2. Berbaum K.S., Franken E.A., Dorfman D.D., et. al.: Time course of satisfaction of search. Invest Radiol 1991; 26: pp. 640-648.
3. Berbaum K.S., Dorfman D.D., Franken E.A., Caldwell R.T.: Proper ROC analysis and joint ROC analysis of the satisfaction of search effect in chest radiography. Acad Radiol 2000; 7: pp. 945-958.
4. Samuel S., Kundel H.L., Nodine C.F., Toto L.C.: Mechanism of satisfaction of search: Eye position recordings in the reading of chest radiographs. Radiology 1995; 194: pp. 895-902.
5. Abe H., Macmahon H., Shiraishi J., Li Q., Engelmann R., Doi K.: Computer-aided diagnosis in chest radiology. Semin Ultrasound CT MR 2004; 25: pp. 432-437.
6. Kakeda S., Moriya J., Sato H., et. al.: Improved detection of lung nodules on chest radiographs using a commercial computer-aided diagnosis system. AJR Am J Roentgenol 2004; 182: pp. 505-510.
7. Freedman M.: State-of-the-art screening for lung cancer (part 1): The chest radiograph. Thorac Surg Clin 2004; 14: pp. 43-52.
8. Das M., Mühlenbruch G., Mahnken A.H., et. al.: Small pulmonary nodules: Effect of two computer-aided detection systems on radiologist performance. Radiology 2006; 241: pp. 564-571.
9. Sakai S., Soeda H., Takahashi N., et. al.: Computer-aided nodule detection on digital chest radiography: Validation test on consecutive 11 cases of resectable lung cancer. J Digit Imaging 2006; 19: pp. 376-382.
10. Shiraishi J., Abe H., Li F., Engelmann R., MacMahon H., Doi K.: Computer-aided diagnosis for the detection and classification of lung cancers on chest radiographs ROC analysis of radiologists’ performance. Acad Radiol 2006; 13: pp. 995-1003.
11. Li Qiang: Recent progress in computer-aided diagnosis of lung nodules on thin-section CT. Comput Med Imaging Graph 2007; in press. (Epub ahead of print). Available at http://www.sciencedirect.com/science/article/B6T5K-4N9MYKV-3/2/094de96a6b678e72973822b412b31922 .
12. Shiraishi J., Li F., Doi K.: Computer-aided diagnosis for improved detection of lung nodules by use of posterior-anterior and lateral chest radiographs. Acad Radiol 2007; 14: pp. 28-37.
13. Suzuki K., Shiraishi J., Abe H., MacMahon H., Doi K.: False-positive reduction in computer-aided diagnostic scheme for detecting nodules in chest radiographs by means of massive training artificial neural network. Acad Radiol 2005; 12: pp. 191-201.
14. Marten K., Engelke C., Seyfarth T., Grillhosl A., Obenauer S., Rummeny E.J.: Computer-aided detection of pulmonary nodules: Influence of nodule characteristics on detection performance. Clin Radiol 2005; 60: pp. 196-206.
15. Shiraishi J., Li Q., Suzuki K., Engelmann R., Doi K.: Computer-aided diagnostic scheme for the detection of lung nodules on chest radiographs: Localized search method based on anatomical classification. Med Phys 2006; 33: pp. 2642-2653.
16. Berbaum K.S., Franken E.A., Dorfman D.D., et. al.: The role of faulty decision making in the satisfaction of search effect in chest radiology. Acad Radiol 2000; 7: pp. 1098-1106.
17. Berbaum K.S., Franken E.A., Caldwell R.T., Schartz K.M.: Can a checklist reduce SOS errors in chest radiography?. Acad Radiol 2006; 13: pp. 296-304.
18. Dorfman D.D., Berbaum K.S.: A contaminated binormal model for ROC data—Part II. Acad Radiol 2000; 7: pp. 427-437.
19. McNemar Q.: Psychological statistics.1969.John Wiley and SonsNew York, NY:pp. 110-121.
20. Siegel S., Castellan N.J.: Nonparametric statistics for the behavioral sciences.1988.McGraw-Hill Book CompanyNew York:pp. 73-167.
21. Franken E.A., Berbaum K.S., Lu C.H., et. al.: Satisfaction of search in detection of plain film abnormalities in abdominal contrast examinations. Invest Radiol 1994; 29: pp. 403-409.
22. Berbaum K.S., Franken E.A., Dorfman D.D., et. al.: Cause of satisfaction of search effects in contrast studies of the abdomen. Acad Radiol 1996; 3: pp. 815-826.
23. Berbaum K.S., Franken E.A., Dorfman D.D., et. al.: Can order of report prevent satisfaction of search in abdominal contrast studies?. Acad Radiol 2005; 2: pp. 74-84.
24. Alberdi E., Povyakalo A., Strigini L., Ayton P.: Effects of incorrect computer-aided detection (CAD) output on human decision-making in mammography. Acad Radiol 2004; 11: pp. 909-918.
25. Poller W.R., Zheng B., Sumkin J.H., Gur D.: Detecting abnormalities in non-cued areas of digitized mammograms: An observer experience. In Chakraborty DP, Krupinski EA, editors. Medical Imaging 2002: Image Perception, Observer Performance, and Technology Assessment Proc SPIE 2002; 4686: pp. 84-88.