Rationale and Objectives
To measure the effect of explicit prevalence expectation on the performance of experienced radiologists during image interpretation of pulmonary lesions on chest radiographs.
Materials and Methods
Each of 22 experienced radiologists was allocated to one of three groups to interpret a set of 30 (15 abnormal) posteroanterior chest images on two occasions to decide if pulmonary lesions were present. Before each viewing, the radiologists were told that the images contained a specific number of abnormal images: group 1, 9 versus 15; group 2, 22 versus 15; and group 3, not told versus 15, respectively. Eye position metrics and receiver operating characteristics confidence ratings were compared for normal and abnormal images. An analysis of false-positive and false-negative decisions was also performed.
Results
For normal images, at higher prevalence expectation, significant increases were noted for duration of image scrutiny (group 1: P = .0004; group 2: P = .007; and group 3: P = .003) and number of fixations per image (group 1: P = .0006; group 2: P = .0004; and group 3: P = .0001). Also for normal images, group 1 demonstrated a significant increase ( P = .038) in average confidence ratings when prevalence expectation increased. For abnormal images, at higher prevalence expectation, significant increases were noted for duration of image scrutiny in group 1 ( P = .005) and number of fixations per image in group 1 ( P = .01) and group 2 ( P = .003).
Conclusions
Confidence ratings and visual search of the expert radiologists appear to be affected by changing prevalence expectations. The impact of prevalence expectation appears to be more apparent for normal images.
It has been acknowledged that the effects of prevalence on radiologists’ behavior are not well understood . The prevalence phenomenon is reported to alter radiologists’ behavior and this may be because of the actual differences in prevalence levels or in the expectation of the prevalence level (even when the actual normal-to-abnormal ratio remains constant). However, the mechanisms responsible for this change in radiologist behavior remain unknown.
Previous studies on the impact of prevalence on radiologic performance have provided various conclusions with one study suggesting that varying prevalence was unlikely to alter the accuracy of the observers , another demonstrating increased diagnostic efficacy with increasing prevalence , and another showing no significant effect . Also, the impact on radiologic confidence of prevalence remains unclear with previous studies presenting conflicting results . The first article established that observers tend to increase their confidence ratings with increasing prevalence , whereas a later article suggested that observers tend to decrease their confidence ratings with increasing prevalence . With such discrepancy, it is surprising that a greater emphasis has not been placed on understanding the impact of this phenomenon on radiologists’ behavior , particularly because the issue of prevalence is present every time a radiologist enters and is in a reading room. However, it is important to note that these previous studies focused on actual prevalence changes to the image test set rather than only altering the stated prevalence expectation to the radiologists before the reading session begins. To initiate this type of research into prevalence and radiologic behavior, a recent article, although showing no significant impact on reporting accuracy using receiver operating characteristic (ROC) analysis, did show that visual search in terms of interpretation time and the number of visual fixations was significantly changed when higher prevalence was told to be expected . This preliminary study, however, did not investigate other important behavioral issues, such as levels of confidence and the impact of prevalence expectation, on types of radiologic error. Also, although it is a reasonable assumption that increased prevalence expectation could affect quite different visual interactions with abnormal images compared to normal images; the previous study combined all images together as a single group.
Get Radiology Tree app to read full this article<
Materials and methods
Get Radiology Tree app to read full this article<
Participants
Get Radiology Tree app to read full this article<
Table 1
Numbers and Details of Participating Radiologists
Radiology Group_n_ Male/Female ( n ) Mean Number of Years of Postregistration Experience (Minimum and Maximum Years are Given in Parentheses) Chest specialists 5 4/1 22 (8, 42) Nonchest specialists 17 13/4 24 (6, 28) All radiologists 22 17/5 23 (6, 42)
Get Radiology Tree app to read full this article<
Images
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Image Display
Get Radiology Tree app to read full this article<
Image Reading
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Eye Tracking
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Confidence Ratings
Get Radiology Tree app to read full this article<
Error Classification
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Results
Eye Tracking Metrics
Get Radiology Tree app to read full this article<
Table 2
Eye Position Metrics for Normal and Abnormal Images across all Radiologists in Each Subgroup
Comparison Expected Prevalence Median Total Duration of Image Scrutiny per Image—Normal Images (milliseconds) Median Total Duration of Image Scrutiny per Image—Abnormal Images (milliseconds) Median Number of Fixations per Image—Normal Images Median Number of Fixations per Image—Abnormal Images Group 1 (5)11461__14788__24__25__7717__13432__17__27__8602__10428__15__17__7002__9336__15__10__8294.5__14014__19.5__25 98539 (SD: 4855)12004 (SD: 5305)19 (SD: 7.873)20 (SD: 10.35)16158__14581__22__23__12898__24249__27__38__6974__9154__15__15__11770__11223__25__18__16497__13959__33__28 1511844 (SD: 6461)12865 (SD: 6659)23 (SD: 11.82)23 (SD: 12.03) Group 2 (6)31833__34938__37__36__17465__15878__34__22__20843__15748__37__25__10876__11449__21__16__18452__19423__38__41__10929__12803__20__19 1516966 (SD: 9132) 17138 (SD: 8991)32 (SD: 14.07)25 (SD: 12.79)37810__36816__58__51__17179__12543.5__31__25.5__28253__24310__54__42__15941__11387__34__22__13966__16722__29__34__13869__12409__27__19 2218297 (SD: 16361) 16722 (SD: 11784)36 (SD: 18.90)33 (SD: 17.82) Group 3 (5)13466__13121__31__24__20546__18492__32__28__13095__19430__23__28__10399__6341__23__8__14969__15373__20__20 1514503 (SD: 6597) 14564 (SD: 6899)25 (SD: 9.429) 23 (SD: 12.35)16785__19183__37__39__15256__14493__25__18__16979__12180__35__20__23872__9726__42__14.5__23722__15987__34__21 Not told (0)18565 (SD: 6716) 15133 (SD: 6555)34 (SD: 12.25) 20 (SD: 13.12)
SD, standard deviation.
Bold font indicates a significant change . The numbers in parenthesis indicate the number of radiologists in each group. Italics indicate individual reader performance.
Get Radiology Tree app to read full this article<
Confidence Ratings for Normal and Abnormal Images
Get Radiology Tree app to read full this article<
Table 3
The Mean Confidence Ratings of the Observers, When Interpreting Normal and Abnormal Images, When Different Prevalence Expectations Were Told
Comparison Expected Prevalence Mean Confidence Rating—Normal Images Mean Confidence Rating—Abnormal Images Group 1 (7)1.27__4.33__1.00__3.47__1.47__3.80__1.87__4.47__1.07__4.67__1.33__3.93__1.60__4.33 91.37 (SD: 0.9) 4.14 (SD: 1.4)1.73__4.93__1.27__3.73__1.00__3.73__2.33__4.53__2.00__4.67__1.07__3.73__1.80__4.33 151.60 (SD: 1.1) 4.24 (SD: 1.2) Group 2 (7)1.13__3.67__2.87__4.47__3.07__4.07__2.73__4.47__1.73__3.87__2.53__4.40__1.47__3.87 15 2.22 (SD: 1.51) 4.45 (SD: 1.19)1.60__3.80__3.67__4.40__2.60__4.07__2.47__4.40__1.93__3.73__2.87__4.47__1.60__4.00 22 2.39 (SD: 1.53) 4.46 (SD: 1.15) Group 3 (8)1.93__4.53__1.67__4.07__1.47__4.67__2.13__4.87__1.53__4.07__1.47__4.13__1.93__4.47__2.20__4.80 15 1.79 (SD: 1.24) 4.45 (SD: 1.12)1.93__4.27__1.67__3.93__1.67__4.60__2.13__5.00__1.47__4.33__2.27__4.60__1.93__4.67__2.00__4.93 Not told (0) 1.88 (SD: 1.28) 4.54 (SD: 1.05)
SD, standard deviation.
Bold font indicates a significant change . The numbers in parenthesis indicate the number of radiologists in each group. Italics indicate individual reader performance.
Get Radiology Tree app to read full this article<
Error Classification
Get Radiology Tree app to read full this article<
Table 4
Mean, Standard Deviation, and P values for Sensitivity, Specificity, and Total False Positives and False Negatives across All Radiologists within Each Subgroup
Comparison Expected Prevalence Sensitivity Specificity False Positives (Normal Images) False Negatives (Abnormal Images) Group 1 (7)0.93__0.87__2__1__0.73__1.00__0__3__0.80__0.87__2__3__0.93__0.67__5__1__0.93__0.93__1__1__0.93__0.67__5__1__0.87__0.73__4__2 9/30 0.87 (SD: 0.08) 0.82 (SD: 0.13) 2.7 (SD: 1.98) 1.7 (SD: 1.98)1.00__0.73__4__0__0.93__0.80__3__1__0.80__1.00__0__3__1.00__0.47__8__0__1.00__0.47__8__1__0.93__0.73__4__3__0.80__0.93__1__3 15/30 0.92 (SD: 0.09) 0.73 (SD: 0.20) 4 (SD: 3.10) 1.3 (SD: 1.26) Group 2 (7)0.93__0.87__2__1__1.00__0.40__9__0__0.93__0.27__11__1__1.00__0.33__10__0__0.80__0.73__4__3__0.93__0.47__8__1__0.93__0.67__5__1 15/30 0.93 (SD: 0.07) 0.53 (SD: 0.25) 7 (SD: 3.37) 1 (SD: 1.00)0.93__0.60__6__1__0.93__0.20__12__1__0.93__0.47__8__1__1.00__0.27__11__0__0.80__0.67__5__3__1.00__0.27__11__0__0.93__0.73__4__1 22/30 0.93 (SD: 0.07) 0.45 (SD: 0.24) 8.14 (SD: 3.24) 1 (SD: 1.00) Group 3 (8)0.93__0.53__7__1__0.80__0.73__4__3__1.00__0.80__3__0__1.00__0.53__7__0__0.93__0.53__7__1__0.93__0.80__3__1__0.93__0.73__4__1__1.00__0.47__8__0 15/30 0.94 (SD: 0.07) 0.64 (SD: 0.14) 5.38 (SD: 3.37) 0.88 (SD: 0.99)0.93__0.60__6__1__0.80__0.67__5__3__1.00__0.73__4__0__1.00__0.67__5__0__1.00__0.60__6__0__0.93__0.53__7__1__0.93__0.60__6__1__1.00__0.33__10__0 Not told (0) 0.95 (SD: 0.07) 0.59 (SD: 0.12) 6.12 (SD: 1.80) 0.75 (SD: 1.00)
SD, standard deviation.
Italics indicate individual reader performance.
Get Radiology Tree app to read full this article<
Discussion
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Acknowledgments
Get Radiology Tree app to read full this article<
References
1. Wolfe J.: Explicit expectations and the effects of prevalence. Radiology 2011; 261: pp. 328.
2. Larson D.: Changing Radiologists’ expectations: false information versus years of experience. Radiology 2011; 261: pp. 327.
3. Gur D., Rockette H.E., Warfel T., et. al.: From the laboratory to the clinic: the “prevalence effect”. Acad Radiol 2003; 10: pp. 1324-1326.
4. Egglin T.K., Feinstein A.R.: Context bias: a problem in diagnostic radiology. JAMA 1996; 276: pp. 1752-1755.
5. Gur D., Rockette H.E., Armfield D.R., et. al.: Prevalence effect in a laboratory environment. Radiology 2003; 228: pp. 10-14.
6. Gur D., Bandos A.I., Fuhrman C.R., et. al.: The prevalence effect in a laboratory environment: changing the confidence ratings. Acad Radiol 2007; 14: pp. 49-53.
7. Reed W., Ryan J.T., McEntee M., et. al.: The effect of abnormality-prevalence expectation on expert observer performance and visual search. Radiology 2011; 258: pp. 938-943.
8. Manning D, Ethell SC, Donovan T. Categories of observer error from eye tracking and AFROC data. In SPIE medical imaging: image perception, observer performance and technology, pp 90–99, SPIE Library 2004.
9. Ryan JT, McEntee MF, Manning DJ, et al. A software system for the simulation of chest lesions. Proc SPIE 2007; 6515: J1–J8.
10. Samei E., Badano A., Chakraborty D., et. al.: Assessment of display performance for medical imaging systems. American Association of Physicists in Medicine Task Group 18. Available at: http://www.aapm.org/pubs/reports/OR_03.pdf Assessed May 30, 2010
11. Tobii. User manual—Tobii Eye Tracker: ClearView analysis software. 2006.
12. Manning D., Ethell S., Donovan T., et. al.: How do radiologists do it? The influence of experience and training on searching for chest nodules. Radiography 2006; 12: pp. 134-142.
13. Nyce J.M., Steele J.S., Gunderman R.B.: Bridging the knowledge divide in radiology education. Radiology 2006; 239: pp. 629-631.
14. Murray M.: Radiologists as teachers: the value of engaging in conversation. Clin Radiol 2008; 63: pp. 207-219.
15. Nodine C.F., Mello-Thoms C., Kundel H.L., et. al.: Time course of perception and decision making during mammographic interpretation. AJR Am J Roentgenol 2002; 179: pp. 917-923.
16. Wolfe J.M., Van Wert M.J.: Varying target prevalence reveals two dissociable decision criteria in visual search. Curr Biol 2002; 20: pp. 121-124.
17. Ryan J.T., Haygood T.M., Yamal J.M., et. al.: The “memory effect” for repeated radiologic observations. AJR Am J Roentgenol 2011; 197: pp. W985-W991.