Residents' Ability to Interpret Radiology Images

Rationale and Objectives

Despite increasing radiology coverage, nonradiology residents continue to preliminarily interpret basic radiologic studies independently, yet their ability to do so accurately is not routinely assessed.

Materials and Methods

An online test of basic radiologic image interpretation was developed through an iterative process. Educational objectives were established, then questions and images were gathered to create an assessment. The test was administered online to first-year interns (postgraduate year [PGY] 1) from 14 different specialties, as well as a sample of third- and fourth-year radiology residents (PGY3/R2 and PGY4/R3).

Results

Over a 2-year period, 368 residents were assessed, including PGY1 ( n = 349), PGY3/R2 ( n = 14), and PGY4/R3 ( n = 5) residents. Overall, the test discriminated effectively between interns (average score = 66%) and advanced residents (R2 = 86%, R3 = 89%; P < .05). Item analysis indicated discrimination indices ranging from −0.72 to 48.3 (mean = 3.12, median 0.58) for individual questions, including four questions with negative discrimination indices. After removal of the negatively indexed questions, the overall predictive value of the instrument persisted and discrimination indices increased for all but one of the remaining questions (range 0.027–70.8, mean 5.76, median 0.94).

Conclusions

Validation of an initial iteration of an assessment of basic image-interpretation skills led to revisions that improved the test. The results offer a specific test of radiologic reading skills with validation evidence for residents. More generally, results demonstrate a principled approach to test development.

Basic radiology skills are critical to almost all practicing physicians. Graduate-level training in radiology has been established as an important skill with varying pedagogical approaches . With the recent emphasis on competency-based assessment , the need for valid task-based methods of assessment of radiology skills is paramount.

Because first-year residents, in programs other than radiology, must review imaging studies in settings with indirect supervision, it is especially important to assess this skill to establish competence. At times, residents with indirect supervision are asked to interpret radiographs and other imaging studies without input from a radiologist, or provide a preliminary read before receiving a somewhat delayed final radiologist’s interpretation. First-year residents not specializing in radiology are often required to independently interpret basic radiologic studies, in most instances while on overnight service, however their ability to do so accurately is not routinely assessed. Residents’ image-interpretation errors have been documented in various settings and specialties, although reports of the rate and severity of the errors vary . Moreover, with the increasing reliance on electronic medical records and computer-based picture archiving and communication systems, clinical radiologic education is occurring on a more limited basis. Largely gone are radiology rounds when entire clinical teams would visit the radiology department for review of the imaging and discussion of the patients on their service.

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Materials and methods

Examination Development

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Examination Administration

Get Radiology Tree app to read full this article<

Table 1

Sample of Interns (postgraduate year [PGY] 1) Assessed and the Programs They Represent

Program Residents Anesthesiology 48 Emergency medicine 28 Family medicine 17 Internal medicine 90 Internal medicine/pediatrics combined 16 Neurosurgery 6 Obstetrics and gynecology 11 Orthopedic surgery 11 Pathology–anatomic and clinical 15 Pediatrics 40 Psychiatry 21 Surgery 38 Surgery–Plastic 5 Surgery–Vascular 2 Unknown 1 Total 349

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Validity Evidence Analysis

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Results

Get Radiology Tree app to read full this article<

Analysis of Overall Test Discrimination

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Analysis of Individual Examination Questions

Get Radiology Tree app to read full this article<

Table 2

Discrimination Index Values for Questions on the Initial and Revised Radiology Assessment

Question Original Examination Discrimination Index Revised Examination Discrimination Index 1 0.9645 1.3099 2 0.1895 0.2754 3 0.8575 1.1641 4 0.4727 0.6032 5 0.1196 0.2740 6 1.0340 1.3484 7 −0.1641 — 8 0.0451 0.0267 9 −0.6117 — 10 −0.7231 — 11 0.4966 0.7155 12 0.6644 0.9026 13 0.7087 0.9039 14 0.1162 0.2417 15 −0.0597 — 16 1.7784 2.3341 17 48.2723 70.8292 18 6.6817 9.1365 19 0.6996 0.9663 20 0.8570 1.1916

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Analysis of Overall Revised Examination Discrimination

Get Radiology Tree app to read full this article<

Analysis of Individual Revised Examination Item Discrimination

Get Radiology Tree app to read full this article<

Discussion

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Appendix

Get Radiology Tree app to read full this article<

Appendix A

Details of Image-interpretation Examination

Question Number Question Prompt that Accompanies Image Type of Image Correct Diagnosis or Answer 1 22-year-old patient Standard posteroanterior (PA) and lateral chest radiograph Normal female chest x-ray 2 50-year-old patient with cough Standard PA and lateral chest radiograph Pulmonary edema 3 65-year-old patient who is short of breath after bronchoscopy Standard portable AP chest radiograph Pneumothorax 4 72-year-old patient with a cough Standard PA and lateral chest radiograph Pneumonia–right upper lobe 5 59-year-old patient Standard PA and lateral chest radiograph Pneumonia–right lower lobe 6 30-year-old patient who is seen in the “urgent care” facility Two PA chest radiographs separated by time Lung metastases 7 Is the endotracheal tube placement correct? Portable AP chest radiograph Normal endotracheal tube position 8 Is the tip of the right internal jugular central venous line placed correctly? Chest radiograph coned down on mediastinum Normal central line position 9 Is the Swan-Ganz catheter properly placed? Portable AP chest radiograph Normal Swan-Ganz catheter position 10 Is the tip of the Swan-Ganz catheter properly positioned? Portable AP chest radiograph Abnormal Swan-Ganz catheter position 11 Is the feeding tube placement acceptable? Portable abdominal radiograph, supine Acceptable feeding-tube position 12 Is the feeding tube placement acceptable? Chest radiograph coned down on mediastinum Nonacceptable feeding-tube position 13 Is the feeding tube placement acceptable? Portable AP chest radiograph Nonacceptable feeding-tube position 14 75-year-old patient who is seen in the emergency room Axial head computed tomography (CT) CVA–stroke 15 72-year-old patient who is seen in the emergency room Axial head CT Epidural hematoma 16 52-year-old-patient who is seen in the emergency room Abdominal AP radiograph, standing Free air in the abdomen 17 74-year-old patient who is seen in the emergency room Abdominal radiograph, supine Small-bowel obstruction 18 68-year-old patient who is seen in the emergency room CT scout of abdomen Ileus 19 You are taking care of a 35-year-old patient who has shortness of breath and fever, and are considering a chest x-ray or CT to rule out pneumonia or pulmonary embolism as an initial study. How much more radiation (whole-body equivalent dose) is a patient exposed when getting a pulmonary embolism CT rather than a PA chest radiograph (2×, 5×, 10×, 50×, 200×)? No image PE CT is 200× more radiation than PA chest radiograph 20 A 19-year-old patient with inflammatory bowel disease returns to the emergency department with flank pain and hematuria and is referred for what turns out to be her 13th renal stone CT in the past 14 months. About how many patients of this age undergoing this number of renal stone CTs will develop cancer as a result of the radiation exposure from these studies (1/5; 1/100; 1/1000; 1/10,000; 1/1,000,000) No image 1 in 100

CVA, cerebrovascular accident; PE, pulmonary embolism; CT, computed tomography.

Get Radiology Tree app to read full this article<

References

• 1. Shaffer K., Ng J.M., Hirsh D.A.: An integrated model for radiology education: development of a year-long curriculum in imaging with focus on ambulatory and multidisciplinary medicine. Acad Radiol 2009; 16: pp. 1292-1301.

• 2. Chertoff J.: Competency assessment in resident education. Acad Radiol 2008; 15: pp. 1215-1216.

• 3. Cooper V.F., Goodhartz L.A., Nemcek A.A., et. al.: Radiology resident interpretations of on-call imaging studies: the incidence of major discrepancies. Acad Radiol 2008; 15: pp. 1198-1204.

• 4. Miyakoshi A., Nguyen Q.T., Cohen W.A., et. al.: Accuracy of preliminary interpretation of neurologic CT examinations by on-call radiology residents and assessment of patient outcomes at a level I trauma center. J Am Coll Radiol 2009; 6: pp. 864-870.

• 5. Tieng N., Grinberg D., Li S.F.: Discrepancies in interpretation of ED body computed tomographic scans by radiology residents. Am J Emerg Med 2007; 25: pp. 45-48.

• 6. Barlev D.M., Amis E.S.: Termination of a mandatory radiology clerkship at the Albert Einstein College of Medicine: a survey of student opinion. Acad Radiol 1994; 1: pp. 187-190.

• 7. Gay S.B., Hillman B.J.: Evaluation of a mandatory radiology resident research rotation. Acad Radiol 2000; 7: pp. 172-175.

• 8. O’Brien K.E., Cannarozzi M.L., Torre D.M., et. al.: Training and assessment of CXR/basic radiology interpretation skills: results from the 2005 CDIM Survey. Teach Learn Med 2008; 20: pp. 157-162.

• 9. Mirsadraee S., Mankad K., McCoubrie P., et. al.: Radiology curriculum for undergraduate medical studies—a consensus survey. Clin Radiol 2012; 67: pp. 1155-1161.

• 10. Poot J.D., Hartman M.S., Daffner R.H.: Understanding the US medical school requirements and medical students’ attitudes about radiology rotations. Acad Radiol 2012; 19: pp. 369-373.

• 11. McEvoy F.J., McEvoy P.M., Svalastoga E.L.: Web-based teaching tool incorporating peer assessment and self-assessment: example of aligned teaching. AJR Am J Roentgenol 2010; 194: pp. W56-W59.

• 12. Sharpe R.E., Surrey D., Gorniak R.J., et. al.: Radiology Report Comparator: a novel method to augment resident education. J Digit Imaging 2012; 25: pp. 330-336.

• 13. Jang T.B., Ruggeri W., Dyne P., et. al.: The learning curve of resident physicians using emergency ultrasonography for cholelithiasis and cholecystitis. Acad Emerg Med 2010; 17: pp. 1247-1252.

• 14. Khan R., Krupinski E., Graham J.A., et. al.: Assessing first year radiology resident competence pre-call: development and implementation of a computer-based exam before and after the 12 month training requirement. Acad Radiol 2012; 19: pp. 752-758.

• 15. Verweij J.W., Hofstee H.M., Golding R.P., et. al.: Interobserver agreement between on-call radiology residents and radiology specialists in the diagnosis of pulmonary embolism using computed tomography pulmonary angiography. J Comput Assist Tomogr 2009; 33: pp. 952-955.

• 16. Ruma J., Klein K.A., Chong S., et. al.: Cross-sectional examination interpretation discrepancies between on-call diagnostic radiology residents and subspecialty faculty radiologists: analysis by imaging modality and subspecialty. J Am Coll Radiol 2011; 8: pp. 409-414.

• 17. Russo G.S., Olson R.L., Varthi A., et. al.: Physical examination and radiographic interpretation of carpal anatomy in orthopedic residents and emergency medicine physicians. American journal of orthopedics 2012; 41: pp. 262-267.

• 18. Epstein R.M., Hundert E.M.: Defining and assessing professional competence. JAMA 2002; 287: pp. 226-235.

• 19. Lypson M.L., Frohna J.G., Gruppen L.D., et. al.: Assessing residents’ competencies at baseline: identifying the gaps. Acad Med 2004; 79: pp. 564-570.

• 20. Moercke A.M., Eika B.: What are the clinical skills levels of newly graduated physicians? Self-assessment study of an intended curriculum identified by a Delphi process. Med Educ 2002; 36: pp. 472-478.

• 21. Fink A., Kosecoff J., Chassin M., et. al.: Consensus methods: characteristics and guidelines for use. Am J Public Health 1984; 74: pp. 979-983.

• 22. Wagner D., Lypson M.L.: Centralized assessment in graduate medical education: cents and sensibilities. J Grad Med Educ 2009; 1: pp. 21-27.

• 23. Thorpe G.L., Favia A.D.: Data analysis using item response theory methodology: an introduction to selected programs and applications.Psychology Faculty Scholarship.2012.University of Maine Digital Commons http://digitalcommons.library.umaine.edu/psy_facpub/20

• 24. Tublin M., Brown C., Pacella C., et. al.: Overnight subspecialty radiology coverage: review of a practice model and analysis of its impact on CT utilization rates in academic and community emergency departments. AJR Am J Roentgenol 2012; 199: pp. 623-626.