Rational and Objectives
Computer-aided detection (CAD) systems are intended to improve performance. This study investigates how CAD might actually interfere with a visual search task. This is a laboratory study with implications for clinical use of CAD.
Methods
Forty-seven naive observers in two studies were asked to search for a target, embedded in 1/f 2.4 noise while we monitored their eye movements. For some observers, a CAD system marked 75% of targets and 10% of distractors, whereas other observers completed the study without CAD. In experiment 1, the CAD system’s primary function was to tell observers where the target might be. In experiment 2, CAD provided information about target identity.
Results
In experiment 1, there was a significant enhancement of observer sensitivity in the presence of CAD (t(22) = 4.74, P < .001), but there was also a substantial cost. Targets that were not marked by the CAD system were missed more frequently than equivalent targets in no-CAD blocks of the experiment (t(22) = 7.02, P < .001). Experiment 2 showed no behavioral benefit from CAD, but also no significant cost on sensitivity to unmarked targets (t(22) = 0.6, P = NS). Finally, in both experiments, CAD produced reliable changes in eye movements: CAD observers examined a lower total percentage of the search area than the no-CAD observers (experiment 1: t(48) = 3.05, P < .005; experiment 2: t(50) = 7.31, P < .001).
Conclusions
CAD signals do not combine with observers’ unaided performance in a straightforward manner. CAD can engender a sense of certainty that can lead to incomplete search and elevated chances of missing unmarked stimuli.
Computer-aided detection (CAD) algorithms are designed to assist radiologists during medical image interpretation. For instance, in mammography, a typical CAD system marks potential abnormalities on the image to encourage additional evaluation by the radiologist before the radiologist makes a final recommendation. In the United States, CAD is currently used on nearly 75% of all mammograms . Several large studies have assessed the efficacy of CAD . Although most studies show that hit rate increases when CAD is introduced to a practice, false alarm rate also tends to increase, making it unclear whether the benefits of CAD outweigh the costs . From a signal detection perspective, the relatively small benefit of CAD is surprising because the CAD system should be increasing the total amount of information available to the radiologists, yielding increased performance. The size of the hypothetical benefit would be larger if CAD and radiologists were making use of independent signals and smaller if they are using the same noisy signals. Even if CAD and radiologists are not independent, the hypothetical benefit seems to be larger than what is observed . That the use of CAD produces only modest improvement in signal detection measures such as area under the receiver operating characteristic (ROC) curve suggests that radiologists are unable to optimally combine the information conveyed by the CAD system and information they gather from the image itself.
In the current study, we use eye-tracking to study the costs and benefits of the presence of a simultaneous CAD system. The laboratory task we created was designed to emulate critical aspects of a typical radiologic search for a difficult to find target. In both experiments, half of the observers completed the experiment without a CAD system, whereas the other half searched the same trials with the help of our artificial CAD system that marked 75% of all targets and 10% of nontargets. In experiment 1, targets were difficult to find because they were embedded in a field of noise. Here, the CAD system primarily aided target detection (CADe). In experiment 2, we manipulated the appearance of our target “Ts” and distractor “Ls,” making the Ts and Ls more similar to each other. At the same time, we decreased the opacity of the background noise the items so that the items were easier to find. Our intent was to keep the overall difficulty roughly the same across the two experiments. In this case, the CAD system primarily aided target diagnosis (CADx).
Materials and methods
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Differences between Experiments 1 and 2
Get Radiology Tree app to read full this article<
Observers
Get Radiology Tree app to read full this article<
Apparatus
Get Radiology Tree app to read full this article<
Eye-tracking Interest Areas
Get Radiology Tree app to read full this article<
Results
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Eye Movements
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Dwell Time Analysis
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Discussion
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Acknowledgments
Get Radiology Tree app to read full this article<
References
1. Rao V.M., Levin D.C., Parker L., et. al.: How widely is computer-aided detection used in screening and diagnostic mammography?. J Am Coll Radiol 2010; 7: pp. 802-805.
2. Fenton J.J., Abraham L., Taplin S.H., et. al.: Effectiveness of computer-aided detection in community mammography practice. J Nat Cancer Inst 2011; 103: pp. 1152-1161.
3. Dorrius M.D., Weide M.C.J., van Ooijen P.M.A., et. al.: Computer-aided detection in breast MRI: a systematic review and meta-analysis. Eur Radiol 2011; 21: pp. 1600-1608.
4. Philpotts L.E.: Can computer-aided detection be detrimental to mammographic interpretation?. Radiology 2009; 253: pp. 17-22.
5. Birdwell R.L.: The preponderance of evidence supports computer-aided detection for screening mammography. Radiology 2009; 253: pp. 9-16.
6. Burgess A.E., Jacobson F.L., Judy P.F.: Human observer detection experiments with mammograms and power-law noise. Med Phys 2001; 28: pp. 419-437.
7. Brainard D.H.: The psychophysics toolbox. Spatial Vis 1997; 10: pp. 433-436.
8. Pelli D.G.: The VideoToolbox software for visual psychophysics: transforming numbers into movies. Spatial Vis 1997; 10: pp. 437-442.
9. Najemnik J., Geisler W.S.: Eye movement statistics in humans are consistent with an optimal search strategy. J Vis 2008; 8: pp. 1-14.
10. Kundel H.L., Nodine C.F., Krupinski E.A.: Searching for lung nodules-visual dwell indicates locations of false-positive and false-negative decisions. Invest Radiol 1989; 24: pp. 472-478.
11. Taplin S.H., Rutter C.M., Lehman C.D.: Testing the effect of computer-assisted detection on interpretive performance in screening mammography. AJR Am J Roentgenol 2006; 187: pp. 1475-1482.
12. Doi K.: Computer-aided diagnosis in medical imaging: historical review, current status and future potential. Comp Med Imaging and Graphics 2007; 31: pp. 198-211.
13. Nishikawa R.M.: Current status and future directions of computer-aided diagnosis in mammography. Comp Med Imaging and Graphics 2007; 31: pp. 224-235.
14. Pisano E.D., Gatsonis C., Hendrick E., et. al.: Diagnostic performance of digital versus film mammography for breast-cancer screening. New Engl J Med 2005; 353: pp. 1773-1783.
15. Gur D., Bandos A.I., Fuhrman C.R., et. al.: The prevalence effect in a laboratory environment: changing the confidence ratings. Acad Radiol 2007; 14: pp. 49-53.
16. Samulski M., Hupse R., Boetes C., et. al.: Using computer-aided detection in mammography as a decision support. Eur Radiol 2010; 20: pp. 2323-2330.
17. Alberdi E., Povyakalo A., Strigini L., et. al.: Effects of incorrect computer-aided detection (CAD) output on human decision-making in mammography. Acad Radiol 2004; 11: pp. 909-918.
18. Berbaum K., Franken E., Caldwell R.T., et. al.: Satisfaction of search in traditional medical imaging.Samei E.Krupinski E.A.The handbook of medical image perception and techniques.2010.Cambridge University PressCambridge:pp. 107-138.
19. Berbaum K.S., Franken E.A., Dorfman D.D., et. al.: Cause of satisfaction of search effects in contrast studies of the abdomen. Acad Radiol 1996; 3: pp. 815-826.