Rationale and Objectives
For both airport baggage screeners and radiologists, low target prevalence is associated with low detection rate, a phenomenon known as “prevalence effect.” In airport baggage screening, the target prevalence is artificially increased with fictional weapons that are digitally superimposed on real baggage. This strategy improves the detection rate of real weapons and also allows airport supervisors to monitor screener performance. A similar strategy using fictional patients could be applied in radiology. The purpose of this study was twofold: (1) to review the psychophysics literature regarding low target prevalence and (2) to survey radiologists’ attitudes toward using fictional patients as a quality assurance tool.
Materials and Methods
We reviewed the psychophysics literature on low target prevalence and airport x-ray baggage screeners. An online survey was e-mailed to all members of the Association of University Radiologists to determine their attitudes toward using fictional patients in radiology.
Results
Of the 1503 Association of University Radiologists member recipients, there were 153 respondents (10% response rate). When asked whether the use of fictional patients was a good idea, the responses were as follows: disagree (44%), neutral (25%), and agree (31%). The most frequent concern was the time taken away from doing clinical work (89% of the respondents).
Conclusions
The psychophysics literature supports the use of fictional targets to mitigate the prevalence effect. However, the use of fictional patients is not a popular idea among academic radiologists.
Introduction
When looking for a needle in a haystack, how do you know when you have missed the needle? Psychologists define “prevalence effect” as a phenomenon where one is more likely to miss a target that occurs with low frequency. When visually searching for something, decreased target prevalence leads to decreased target detection rate. In simple visual search tasks, a target prevalence of 1% is associated with a detection rate of only 70% . Prevalence effect is a problem for two occupations in particular: airport baggage screeners and radiologists, both of whom “spend the bulk of their time looking for things they rarely see.” Although not all radiology findings are rare, it is true that positive radiology findings are less common than normal findings. The radiology literature on the topic is sparse, but prevalence effect appears to be problematic for radiologists only when prevalence is below 1% . There are no existing radiology workflow strategies to mitigate the prevalence effect. Existing strategies to track radiologist misses are retrospective and therefore likely underestimate the number of misses.
How can airport baggage screeners and radiologists mitigate the prevalence effect? In one psychophysics paper from 2010, the authors concluded, “We want to reduce the error rates in real-life low-prevalence situations, such as airport baggage screening or X-ray examination in medical settings. Naturally, one may hope to do so by giving the workers stricter instructions. The present study suggests that such a method is probably futile. The only effective method is to randomly distribute some ‘pseudo-targets’ into the screening, thereby ensuring that, by gaining experience with such targets, workers will not miss real targets when they show up.”
Get Radiology Tree app to read full this article<
Psychophysics Literature
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Airport Baggage X-Ray Screening
Get Radiology Tree app to read full this article<
Existing Quality Control in Radiology
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Use of Fictional Patients for Monitoring Radiologist Performance
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Methods
Online Survey
Get Radiology Tree app to read full this article<
Survey Distribution
Get Radiology Tree app to read full this article<
Analysis of Survey Responses
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Results
Get Radiology Tree app to read full this article<
TABLE 1
Demographics of Survey Respondents
Characteristics_n_ (%) In-practice academic 111 (72) Radiology resident 20 (13) In-practice multispecialty group and integrated health system 9 (6) Radiology fellow 4 (3) In-practice private practice 4 (3) Researcher 2 (1) Retired 2 (1) In-practice military and government 1 (1)
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
TABLE 2
Questions 1–4
Question Stem (Shortened Text) Choices (Shortened Text) Responses, n (%) Q1: When should the radiologist receive the feedback? Immediately 84 (55) Daily 36 (24) Monthly 23 (15) Other 8 (5) Never 2 (1) Q2: Who should receive the performance feedback? Radiologist 149 (97) Radiology supervisor 88 (57) Nonradiology supervisor 10 (6) Other 10 (6) Q3: What types of cases should be included? Typical difficulty 133 (87) Life threatening 125 (82) Easy difficulty 114 (75) Previously missed by radiologist 108 (71) Common diseases 108 (71) Increased long-term mortality 101 (66) Normal studies 97 (63) Incidental, nonurgent follow-up 83 (54) Hard difficulty 81 (53) Image artifacts 74 (48) Selected by a society 59 (39) Rare diseases 42 (27) Selected by supervisor 34 (22) Selected by radiologist 31 (20) Incidental and insignificant 19 (12) Other 3 (2) Q4: What real-time corrective action is most appropriate? Real-time acknowledgement 117 (76) Take a lesson and pass a test 11 (7) No corrective action 10 (7) Other 8 (5) Take a break 7 (5)
TABLE 3
Questions 5–8
Question Stem (Shortened Text) Choices (Shortened Text) Responses, n (%) Q5: Barriers to implementation? Time away from clinical work 136 (89) Radiologist confusion when trying to call in abnormal results 129 (84) Radiologist’s fear of court liability 110 (72) Radiologist’s fear of disciplinary action 102 (67) Radiologist distracted by constant question of what is real or fictional 97 (63) Difficulty with PACS integration 95 (62) Radiologist feeling upset or patronized 89 (58) Difficulty creating a fictional patient 86 (56) Radiologist’s fear of embarrassment 80 (52) Once implemented, we cannot go back 73 (48) Too much work for cross-sectional studies 62 (41) Unwanted ties with pay for performance 61 (40) Unwanted government oversight 53 (35) Other 7 (5) Q6: How frequently should radiologist performance be monitored? Daily 48 (31) Weekly 41 (27) Monthly 38 (25) Every few hours 14 (9) Other 11 (7) Hourly 1 (1) Q7: “Fictional patients should be used to monitor radiologist performance in real time.” Somewhat disagree 43 (28) Somewhat agree 41 (27) Neither agree nor disagree 38 (25) Strongly disagree 25 (16) Strongly agree 6 (4) Q8: What other radiology quality control measures should be explored? Restrictions on radiologist cases per shift 89 (58) Restrictions on radiologist hours per shift 64 (42) Real-time biometric fatigue monitoring 56 (37) None 36 (24) Other 9 (6)
PACS, picture archiving and communication system.
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Discussion
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Conclusions
Get Radiology Tree app to read full this article<
Acknowledgments
Get Radiology Tree app to read full this article<
Appendix
Survey Text
Demographics
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Survey Explanation
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Survey Questions
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
References
1. Wolfe J.M., Horowitz T.S., Kenner N.M.: Cognitive psychology: rare items often missed in visual searches. Nature 2005; 435: pp. 439-440.
2. Wolfe J.M., Kenner N.M., Horowitz T.S.: Visual search: the perils of rare targets. J Vis 2005; 5: pp. 779.
3. Mitroff S.R., Biggs A.T.: The ultra-rare-item effect: visual search for exceedingly rare items is highly susceptible to error. Psychol Sci 2014; 25: pp. 284-289.
4. Wolfe J.M., Horowitz T.S., Van Wert M.J., et. al.: Low target prevalence is a stubborn source of errors in visual search tasks. J Exp Psychol Gen 2007; 136: pp. 623-638.
5. Hallinan J.T.: Why We Make Mistakes: How We Look Without Seeing, Forget Things in Seconds, and Are All Pretty Sure We Are Way Above Average. Crown/Archetype2009.
6. Evans K.K., Birdwell R.L., Wolfe J.M.: If you don’t find it often, you often don’t find it: why some cancers are missed in breast cancer screening. PLoS ONE 2013; 8: e64366
7. Gur D., Rockette H.E., Armfield D.R., et. al.: Prevalence effect in a laboratory environment. Radiology 2003; 228: pp. 10-14.
8. Lau J.S.-H., Huang L.: The prevalence effect is determined by past experience, not future prospects. Vision Res 2010; 50: pp. 1469-1474.
9. Schwaninger A., Hofer F., Wetter O.E.: Adaptive Computer-Based Training Increases on the Job Performance of X-Ray Screeners. 2007 41st Annual IEEE International Carnahan Conference on Security Technology2007. p. 117–124
10. Grier R.A., Warm J.S., Dember W.N., et. al.: The vigilance decrement reflects limitations in effortful attention, not mindlessness. Hum Factors 2003; 45: pp. 349-359.
11. Fleck M.S., Mitroff S.R.: Rare targets are rarely missed in correctable search. Psychol Sci 2007; 18: pp. 943-947.
12. Pattyn N., Neyt X., Henderickx D., et. al.: Psychophysiological investigation of vigilance decrement: boredom or cognitive fatigue?. Physiol Behav 2008; 93: pp. 369-378.
13. Levin D.T., Angelone B.L., Beck M.R.: Visual search for rare targets: distracter tuning as a mechanism for learning from repeated target-absent searches: visual search for rare targets. Br J Psychol 2011; 102: pp. 313-327.
14. Clark K., Cain M.S., Adamo S.H., et. al.: Overcoming hurdles in translating visual search research between the lab and the field. Nebr Symp Motiv 2012; 59: pp. 147-181.
15. Biggs A.T., Mitroff S.R.: Finding a Needle (and a Thread, and a Thimble, and…) in a Haystack: Multiple-Target Visual Search for Ultra-Rare Items. http://dx.doi.org/10.1037/e633262013-817
16. Thomson D.R., Besner D., Smilek D.: A critical examination of the evidence for sensitivity loss in modern vigilance tasks. Psychol Rev 2016; 123: pp. 70-83.
17. Biggs A.T.: Getting satisfied with “satisfaction of search”: how to measure errors during multiple-target visual search. Atten Percept Psychophys 2017; Springer US; 1–14
18. Chen D.L., Moskowitz T.J., Shue K.: Decision-making under the gambler’s fallacy: evidence from asylum judges, loan officers, and baseball umpires. Q J Econ 2016; 131: pp. 1181-1242. qje.oxfordjournals.org
19. Evans K.K., Tambouret R.H., Evered A., et. al.: Prevalence of abnormalities influences cytologists’ error rates in screening for cervical cancer. Arch Pathol Lab Med 2011; 135: pp. 1557-1560.
20. McCarley J.S., Kramer A.F., Wickens C.D., et. al.: Visual skills in airport-security screening. Psychol Sci 2004; 15: pp. 302-306.
21. Nodine C.F., Mello-Thoms C., Kundel H.L., et. al.: Time course of perception and decision making during mammographic interpretation. AJR Am J Roentgenol 2002; 179: pp. 917-923.
22. Thomson D.R., Smilek D., Besner D.: Reducing the vigilance decrement: the effects of perceptual variability. Conscious Cogn 2015; 33: pp. 386-397.
23. Wolfe J.M., Brunelli D.N., Rubinstein J., et. al.: Prevalence effects in newly trained airport checkpoint screeners: trained observers miss rare targets, too. J Vis 2013; 13: pp. 33.
24. McCarley J.S., Gosney J.: Metacognitive judgments in a simulated luggage screening task. Proceedings of the Human Factors and Ergonomics Society Annual Meeting. SAGE Publications2005. pp. 1620–1624
25. Durso F.T.: Handbook of applied cognition.2007.John Wiley & SonsChichester, England; Hoboken, NJ
26. McCarley J.S.: Effects of speed–accuracy instructions on oculomotor scanning and target recognition in a simulated baggage x-ray screening task. Ergonomics 2009; 52: pp. 325-333.
27. Bolfing A., Halbherr T., Schwaninger A.: How image based factors and human factors contribute to threat detection performance in x-ray aviation security screening. HCI and usability for education and work.2008.SpringerBerlin, Heidelbergpp. 419-438.
28. Hardmeier D., Hofer F., Schwaninger A.: The X-ray object recognition test (X-ray ORT)—a reliable and valid instrument for measuring visual abilities needed in X-ray screening. Security Technology. 2005 CCST ‘05 39th Annual 2005 International Carnahan Conference on Security Technology. IEEE2005. pp. 189–192
29. Schwaninger A., Hardmeler D., Hofer F.: Aviation security screeners visual abilities & visual knowledge measurement. IEEE Aerosp Electron Syst Mag 2005; 20: pp. 29-35.
30. Michel S., Koller S.M., de Ruiter J.C., et. al.: Computer-Based Training Increases Efficiency in X-Ray Image Interpretation by Aviation Security Screeners. 2007 41st Annual IEEE International Carnahan Conference on Security Technology2007. p. 201–206
31. Bradner E., Rene Marsh C.: TSA screeners failed tests to detect explosives, weapons. CNN; Available at: http://www.cnn.com/2015/06/01/politics/tsa-failed-undercover-airport-screening-tests/
32. Wetter O.E., Hardmeier D., Hofer F.: Covert testing at airports: Exploring methodology and results. 2008 42nd Annual IEEE International Carnahan Conference on Security Technology2008. p. 357–363
33. Schwaninger A.: Why do airport security screeners sometimes fail in covert tests?. 43rd Annual 2009 International Carnahan Conference on Security Technology2009. p. 41–45
34. Pinto A., Caranci F., Romano L., et. al.: Learning from errors in radiology: a comprehensive review. Semin Ultrasound CT MR 2012; 33: pp. 379-382.
35. Kelly A.M., Cronin P.: Practical approaches to quality improvement for radiologists. Radiographics 2015; 35: pp. 1630-1642.
36. Borgstede J.P., Lewis R.S., Bhargavan M., et. al.: RADPEER quality assurance program: a multifacility study of interpretive disagreement rates. J Am Coll Radiol 2004; 1: pp. 59-65.
37. Gale A.G.: PERFORMS—a self assessment scheme for radiologists in breast screening. Semin Breast Dis 2003; 6: pp. 148-152.
38. University of Sydney : BreastScreen Reader Assessment Strategy (BREAST). Faculty of Health Science; Available at: http://sydney.edu.au/health-sciences/breastaustralia/index.shtml
39. Barnett R.N.: Importance of quality control in the medical laboratory. Ann N Y Acad Sci 1969; 161: pp. 477-483.
40. Lee F.D.: External quality assessment in histopathology: an overview. J Clin Pathol 1989; 42: pp. 1009-1011.
41. Landsberger H.A.: Hawthorne revisited: management and the worker: its critics, and developments in human relations in industry. Cornell University1958.
42. McCarley J.S., Kramer A.F.: Eye movements as a window on perception and cognition. Neuroergonomics: The brain at work. Oxford University Press New York, NY2006. 95–112
43. Meyer-Delius J., Liebl L.: Evaluation of vigilance related to visual perception.Sheridan T.B.Johannsen G.Monitoring behavior and supervisory control.1976.Springer USBoston, MA:pp. 97-106.
44. Bergasa L.M., Nuevo J., Sotelo M.A., et. al.: Real-time system for monitoring driver vigilance. IEEE Intelligent Vehicles Symposium2004. IEEE; pp. 78–83
45. Hammoud R.I.: Passive eye monitoring: algorithms, applications and experiments.2008.Springer Science & Business MediaBerlin, Germany
46. Samuel S., Kundel H.L., Nodine C.F., et. al.: Mechanism of satisfaction of search: eye position recordings in the reading of chest radiographs. Radiology 1995; 194: pp. 895-902.
47. Burling D., Halligan S., Altman D.G., et. al.: CT colonography interpretation times: effect of reader experience, fatigue, and scan findings in a multi-centre setting. Eur Radiol 2006; 16: pp. 1745-1749.
48. Boland G.W.L., Halpern E.F., Gazelle G.S.: Radiologist report turnaround time: impact of pay-for-performance measures. Am J Transplant 2010; 195: pp. 707-711.
49. Ondategui-Parra S., Bhagwat J.G., Zou K.H., et. al.: Use of productivity and financial indicators for monitoring performance in academic radiology departments: U.S. nationwide survey. Radiology 2005; 236: pp. 214-219.
50. Chan H.-P., Doi K., Vybrony C.J., et. al.: Improvement in radiologists’ detection of clustered microcalcifications on mammograms: the potential of computer-aided diagnosis. Invest Radiol 1990; 25: pp. 1102.
51. van Zelst J.C.M., Tan T., Platel B., et. al.: Improved cancer detection in automated breast ultrasound by radiologists using computer aided detection. Eur J Radiol 2017; 89: pp. 54-59.
52. Lodwick G.S.: Computer-aided diagnosis in radiology. A research plan. Invest Radiol 1966; 1: pp. 72-80.
53. van Ginneken B.: Fifty years of computer analysis in chest imaging: rule-based, machine learning, deep learning. Radiol Phys Technol 2017; 10: pp. 23-32.
54. Rice S., McCarley J.S.: Effects of response bias and judgment framing on operator use of an automated aid in a target detection task. J Exp Psychol Appl 2011; 17: pp. 320-331.