Rationale and Objectives
With the advent of the new core and certifying examinations, a need has arisen to restructure learning and assessment to better reflect the emphasis on continuous learning throughout residency. We developed a multiparametric end-of-rotation (EOR) evaluation tool to assess medical knowledge, oral presentation, and written communication skills administered to the residents at the end of each core subspecialty rotation. The benefit of continual assessment is obvious from a program perspective; the purpose of this article is to evaluate the residents’ perception of the process.
Materials and Methods
All residents ( n = 31, 28 postgraduate years two through five and 3 postgraduate year one) participate in the mandatory EOR evaluation as a required component of the residency program. After receiving Institutional Review Board approval, informed consent was obtained from the residents wishing to participate in quarterly 16-question online surveys assessing their experience. Each survey consists of 15 questions with Likert scale responses (1 through 5 from strongly disagree to strongly agree) and one free text answer. Data are collected quarterly, starting in September 2011.
Results
Overall, the residents’ response has been positive. The new evaluation method is felt to be more meaningful than (average 3.9, standard deviation 0.9) and is favored by the residents over the traditional competency-based evaluation (average 4.0, standard deviation 1.0). However, residents retain neutral attitudes regarding preparation for boards or changes in study habits (average score 3.6, standard deviation 0.9 and 3.6, and standard deviation 1.1, respectively).
Conclusion
Residents rate the EOR evaluation experience positively, although they do not report changes in study habits or increased preparedness for the new certifying examination.
Evaluation and assessment serve as the starting point for curriculum development. The advent of the new American Board of Radiology core and certifying examinations and subsequent Association of Program Directors in Radiology Residency Restructuring Committee recommendations regarding development of a 36-month core curriculum followed by a subspecialty focused year have provided a unique opportunity for development of a new assessment tool. With this in mind, our group created a novel end-of-rotation (EOR) assessment method to evaluate this new educational paradigm.
Computer-based assessment of resident knowledge is long standing and has been shown to improve resident performance . The challenge has been to develop a method that accurately assesses residents, not only from a medical knowledge perspective but also in terms of becoming an independently functioning radiologist well versed in the competencies. Structured assessments do provide data, but are often not concurrent with faculty perception of resident ranking . The shortcomings of multiple-choice testing in assessing knowledge and competence are long standing and although open-ended questions certainly increase accuracy, it does not capture the elements of clinical practice. There is a wealth of literature from emergency medicine documenting assessment techniques for the general competencies and communication . However, to our knowledge, there has not been an evaluation tool in radiology such as ours, which is unique in its scope.
Get Radiology Tree app to read full this article<
Materials and methods
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Table 1
Survey Questions
The end-of-rotation assessment is a fair assessment of learning. I understand the evaluation process used in the end-of-rotation assessment. Feedback helped me better prepare for future rotations. The current end-of-rotation assessment is an improvement over the previous evaluation method (core competency) based. The end-of-rotation feedback is more meaningful than the previous evaluation method. The end-of-rotation assessment provides a better understanding of expectations. Assessment should be performed at the end of the rotation. Assessment should be performed at the midpoint of the rotation. The expectations were appropriate for my level. Performing the sample dictation and receiving written feedback is a valuable exercise. The oral examination is helpful in practicing oral communication skills. The computer-based examination is an appropriate reflection of what I should be expected to know. I feel more prepared for the board examinations because of this testing. Incorporation of end-of-rotation testing has caused me to be more diligent in studying consistently during my rotations.
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Results
Get Radiology Tree app to read full this article<
Table 2
Aggregate Questionnaire Data by Class
PGY 1 PGY 2 PGY 3 PGY 4 PGY5 The end-of-rotation assessment is a fair assessment of learning. 3 3.7 4 4.2 3.4 I understand the evaluation process used in the end-of-rotation assessment. 4.4 4.1 4.4 4.2 3.8 Feedback helped me better prepare for future rotations. 4 3.8 4 4.2 3.4 The current end-of-rotation assessment is an improvement over the previous evaluation method (core competency) based. 3 3.6 3.9 4.7 3.5 The end-of-rotation feedback is more meaningful than the previous evaluation method. 3 3.5 4.2 4.4 3.5 The end-of-rotation assessment provides a better understanding of expectations. 4 3.9 3.9 4.1 3.7 Assessment should be performed at the end of the rotation. 4.2 4 4.4 4.4 3.8 Assessment should be performed at the midpoint of the rotation. 4.2 2 2.6 1.9 2.3 The expectations were appropriate for my level. 4.2 3.8 4.2 4.1 3.9 Performing the sample dictation and receiving written feedback is a valuable exercise. 3.8 3.7 3.8 3.9 2.8 The oral examination is helpful in practicing oral communication skills. 4.4 4 4.4 4.6 4 The computer-based examination is an appropriate reflection of what I should be expected to know. 3.2 3.7 3.6 3.2 3.8 I feel more prepared for the board examinations because of this testing. 3.2 3.6 3.6 4 2.9 Incorporation of end-of-rotation testing has caused me to be more diligent in studying consistently during my rotations. 3 3.3 4 3.8 3.3
PGY, postgraduate year.
Table 3
Aggregate Residency Results for Questionnaire
September 2011 December 2011 March 2012 The end-of-rotation assessment is a fair assessment of learning. 3.8 (SD 0.6) 3.8 (SD 0.7) 3.8 (SD 0.7) I understand the evaluation process used in the end-of-rotation assessment. 4.1 (SD 0.7) 4.2 (SD 0.6) 4.1 (SD 0.6) Feedback helped me better prepare for future rotations. 3.8 (SD 0.6) 3.8 (SD 0.8) 3.8 (SD 0.9) The current end-of-rotation assessment is an improvement over the previous evaluation method (core competency) based. 3.9 (SD 1.1) 4.0 (SD 1.0) 4.0 (SD 1.0) The end of rotation feedback is more meaningful than the previous evaluation method. 3.8 (SD 1.1) 3.8 (SD 1.0) 3.9 (SD 0.9) The end-of-rotation assessment provides a better understanding of expectations. 3.6 (SD 1.0) 4.0 (SD 0.7) 4.0 (SD 0.6) Assessment should be performed at the end of the rotation. 3.9 (SD 1.1) 4.2 (SD 0.9) 4.1 (SD 0.7) Assessment should be performed at the midpoint of the rotation. 2.3 (SD 0.8) 2.2 (SD 0.7) 2.3 (SD 0.9) The expectations were appropriate for my level. 3.8 (SD 0.7) 4.0 (SD 0.6) 4.0 (SD 0.6) Performing the sample dictation and receiving written feedback is a valuable exercise. 3.6 (SD 1.1) 3.6 (SD 1.1) 3.5 (SD 1.0) The oral examination is helpful in practicing oral communication skills. 4.4 (SD 0.8) 4.2 (SD 0.9) 4.2 (SD 1.0) The computer-based examination is an appropriate reflection of what I should be expected to know. 3.9 (SD 0.8) 3.6 (SD 1.0) 3.4 (SD 0.8) I feel more prepared for the board examinations because of this testing. 3.5 (SD 1.0) 3.5 (SD 1.1) 3.5 (SD 0.8) Incorporation of end-of-rotation testing has caused me to be more diligent in studying consistently during my rotations. 3.5 (SD 1.3) 3.5 (SD 1.3) 3.6 (SD 1.0)
Table 4
Aggregate Residency Results for Questionnaire Excluding PGY1 and PGY5 Classes
September 2011 December 2011 March 2012 The end-of-rotation assessment is a fair assessment of learning. 4 3.8 4.1 I understand the evaluation process used in the end-of-rotation assessment. 4.3 4.2 4.2 Feedback helped me better prepare for future rotations. 4.1 4 4 The current end-of-rotation assessment is an improvement over the previous evaluation method (core competency) based. 4.2 4.2 4.1 The end-of-rotation feedback is more meaningful than the previous evaluation method. 4.1 3.9 4 The end of rotation assessment provides a better understanding of expectations. 3.8 4.1 4.1 Assessment should be performed at the end of the rotation. 4.4 4.5 4.1 Assessment should be performed at the midpoint of the rotation. 3.3 2.2 2.1 The expectations were appropriate for my level. 4 4 4.1 Performing the sample dictation and receiving written feedback is a valuable exercise. 4.1 3.8 3.7 The oral examination is helpful in practicing oral communication skills. 4.5 4.3 4.4 The computer-based examination is an appropriate reflection of what I should be expected to know. 4.1 3.5 3.4 I feel more prepared for the board examinations because of this testing. 4.2 3.9 3.6 Incorporation of end-of-rotation testing has caused me to be more diligent in studying consistently during my rotations. 4.1 3.6 3.7
PGY, postgraduate year.
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Table 5
Time Allotted for Testing
Amount of Time Allotted for End-of-Rotation Testing Is: Number of Responses Percentage of Overall Responses Too little 12 18 Adequate 56 82 Too much 0 0
The number of responses represents aggregate responses across the three surveys.
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Discussion
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
References
1. Deitte L., Chertoff J.D., Mainiero M.B., et. al.: Challenges and opportunities in restructuring radiology residencies: the APDR residency restructuring committee report. J Am Coll Radiol 2010; 7: pp. 507-511.
2. Finlay K., Norman G.R., Keane D.R., et. al.: A web-based test of residents’ skills in diagnostic radiology. Can Assoc Radiol J 2006; 57: pp. 106-116.
3. Faulkner H., Regehr G., Martin J., et. al.: Validation of an objective structured assessment of technical skills for surgical residents. Acad Med 1996; 71: pp. 1363-1365.
4. Veloski J.J., Rabinowitz H.K., Robeson M.R., et. al.: Patients don’t present with five choices: an alternative to multiple-choice tests in assessing physicians’ competence. Acad Med 1999; 74: pp. 539-546.
5. Swing S.R.: Assessing the ACGME general competencies: general considerations and assessment methods. Acad Emerg Med 2002; 9: pp. 1278-1288.
6. Hobgood C.D., Riviello R.J., Jouriles N., et. al.: Assessment of communication and interpersonal skills competencies. Acad Emerg Med 2002; 9: pp. 1257-1269.
7. The ACGME competencies. Available at: http://acgme.org/acWebsite/navPages/nav_commonpr.asp . Accessed April 17, 2011.
8. Phelps A., Naeger D.M., MacKenzie J., et. al.: Educating radiology residents in the new era: implementation and evaluation of online end-of-rotation examinations. Acad Radiol 2011; 18: pp. 1442-1446.
9. Ogburn T., Espey E.: The R-I-M-E method for evaluation of medical students on an obstetrics and gynecology clerkship. Am J Obstet Gynecol 2003; 189: pp. 666-669.
10. Chertoff J.: Competency assessment in resident education. Acad Radiol 2008; 15: pp. 1215-1216.
11. Itri J.N., Redfern R.O., Scanlon M.H.: Using a web-based application to enhance resident training and improve performance on-call. Acad Radiol 2010; 17: pp. 917-920.
12. Khan R., Krupinski E., Graham A., et. al.: Assessing first year radiology resident competence pre-call: development and implementation of a computer-based exam before and after the 12 month training requirement. Acad Radiol 2012; 19: pp. 752-758.
13. Ende J.: Feedback in clinical medical education. JAMA 1983; 250: pp. 777-781.
14. Jolly B.C., Ayers B., Macdonald M.M., et. al.: The reproducibility of assessing radiological reporting: studies from the development of the General Medical Council’s Performance Procedures. Med Educ 2001; 35: pp. 36-44.