Rationale and Objectives
Clinical and contextual information associated with images may influence how radiologists draw diagnostic inferences, highlighting the need to control multiple sources of bias in the methodologic design of investigations involving radiologic interpretation. In the past, manual control methods to mask review films presented in practice have been used to reduce potential interpretive bias associated with differences between viewing images for patient care and reviewing images for the purposes of research, education, and quality improvement. These manual precedents from the film era raise the question whether similar methods to reduce bias can be implemented in the modern digital environment.
Materials and Methods
A prototype application, CreateAPatient, was built for masking review case presentations within one institution’s production radiology information system and picture archiving and communication system. To test whether CreateAPatient could be used to mask review images presented in practice, six board-certified radiologists participated in a pilot study. During pilot testing, seven digital chest radiographs, known to contain lung nodules and associated with fictitious patient identifiers, were mixed into the routine workloads of the participating radiologists while they covered general evening call shifts. The aim was to test whether it was possible to mask the presentation of these review cases, both by probing the interpreting radiologists to report detection and by conducting a forced-choice experiment on a separate cohort of 20 radiologists and information technology professionals.
Results
None of the participating radiologists reported awareness of review activity, and forced-choice detection was less than predicted at chance, suggesting that radiologists were effectively blinded. In addition, no evidence was identified of review reports unsafely propagating beyond their intended scope or otherwise interfering with patient care, despite integration of these records within production electronic work flow systems.
Conclusions
Information technology can facilitate the design of unbiased methods involving professional review of digital diagnostic images.
Clinical and contextual information associated with images may influence how radiologists draw diagnostic inferences. For example, in mammography, when a regularly screened woman receives a diagnosis of breast cancer without suspicion having been raised on her last screening mammogram, a quality-related question naturally arises: Were the results of the last screening mammogram truly negative? Multiple studies of these positive “interval cancer” mammograms suggest that the answer to this question is complex. As an extreme illustrative example, one reviewer in a recent study reported positive findings in three of 20 interval-cancer mammograms when they were deidentified and presented mixed with normal mammograms, but positive findings were reported in 17 of the same 20 mammograms when they were unmixed and associated with additional clinical information. When aggregating these data statistically, the magnitude of design effects remains significant and large, with positive finding rates ranging over an order of magnitude across review conditions in most studies .
The importance of review design is not limited to studies of interval breast cancer. Semelka et al recently reported that of 31 external radiologist reviewers, none documented suspicion of spinal instability when reviewing a computed tomographic scan showing 1-mm symmetric facet widening at T10. The study scan was presented with five control scans in the context of a quality assurance review. Study reviewers were not informed that the computed tomographic examination had also been the subject of a settled lawsuit, in which four of four expert witnesses for the plaintiff had represented the spinal instability findings as expected per standard of care.
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Materials and methods
Get Radiology Tree app to read full this article<
Review Masking Pilot
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Forced-choice Detection Test
Get Radiology Tree app to read full this article<
Results
Blinded Review Pilot
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Forced-choice Detection Test
Get Radiology Tree app to read full this article<
Discussion
Get Radiology Tree app to read full this article<
Scope and Terminology
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Study Limitations and Future Direction
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Conclusions
Get Radiology Tree app to read full this article<
Acknowledgments
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
References
1. Ciatto S., Catarzi S., Lamberini M.P., et. al.: Interval breast cancers in screening: the effect of mammography review method on classification. Breast 2007; 16: pp. 646-652.
2. Semelka R.C., Ryan A.F., Yonkers S., et. al.: Objective determination of standard of care: use of blind readings by external radiologists. AJR Am J Roentgenol 2010; 195: pp. 429-431.
3. Wolfe J.M., Horowitz T.S., Van Wert M.J., et. al.: Low target prevalence is a stubborn source of errors in visual search tasks. J Exp Psychol Gen 2007; 136: pp. 623-638.
4. Gunderman R.B.: Biases in radiologic reasoning. AJR Am J Roentgenol 2009; 192: pp. 561-564.
5. Ryan A., Semelka R., Molina P.: Evaluation of radiologist interpretive performance using blinded reads by multiple external readers. Invest Radiol 2010; 45: pp. 211-216.
6. Liu P.T., Johnson C.D., Miranda R., et. al.: A reference standard-based quality assurance program for radiology. J Am Coll Radiol 2010; 7: pp. 61-66.
7. de Rijke J.M., Schouten L.J., Schreutelkamp J.L., et. al.: A blind review and an informed review of interval breast cancer cases in the Limburg screening programme, the Netherlands. J Med Screen 2000; 7: pp. 19-23.
8. Gordon P.B., Borugian M.J., Warren Burhenne L.J.: A true screening environment for review of interval breast cancers: pilot study to reduce bias. Radiology 2007; 245: pp. 411-415.
9. Evertsz C. Methods and system for in-service monitoring and training for a radiologic workstation. US Patent No. 7366676; 2008.
10. Glyptis P., Givens D.: Developing a true blind review. ACR Bull 2011; 189: pp. 24.
11. Rethans J.-J., Gorter S., Bokken L., et. al.: Unannounced standardised patients in real practice: a systematic literature review. Med Educ 2007; 41: pp. 537-549.
12. Siegel E.: Effect of dual-energy subtraction on performance of a commercial computer-assisted diagnosis system in detection of pulmonary nodules. Proc SPIE 2005; 5748: pp. 392-398.
13. Renshaw A.A., Pinnar N.E., Jiroutek M.R., et. al.: Blinded review as a method for quality improvement in surgical pathology. Arch Pathol Lab Med 2002; 126: pp. 961-963.
14. Tang D, Agarwal A, Brien DO, et al. Overlapping experiment infrastructure: more, better, faster experimentation categories and subject descriptors. In: Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. Washington, DC: Association for Computing Machinery; 2010:17–26.
15. Wetter O, Hardmeier D. Covert testing at airports: exploring methodology and results. In: Proceedings of the 42nd Annual IEEE International Carnahan Conference on Security Technology. Piscataway, NJ: IEEE; 2008:357–363.
16. Rosenhan D.: On being sane in insane places. Science 1973; 179: pp. 250-258.
17. Maiburg B.H.J., Rethans J.-J.E., van Erk I.M., et. al.: Fielding incognito standardised patients as “known” patients in a controlled trial in general practice. Med Educ 2004; 38: pp. 1229-1235.
18. Ozuah P.O., Reznik M.: Residents’ asthma communication skills in announced versus unannounced standardized patient exercises. Ambul Pediatr 2007; 7: pp. 445-448.
19. Hofvind S., Skaane P., Vitak B., et. al.: Influence of review design on percentages of missed interval breast cancers: retrospective study of interval cancers in a population-based screening program. Radiology 2005; 237: pp. 437-443.
20. Kaptchuk T.J.: Intentional ignorance: a history of blind assessment and placebo controls in medicine. Bull Hist Med 1998; 72: pp. 389-433.