Home Milestones on a Shoestring
Post
Cancel

Milestones on a Shoestring

Posted Sep 30, 2015 Updated Apr 10, 2023
By J. Eric Schmitt MD PhD , Mary H. Scanlon MD , Sabah Servaes MD , Dayna Levin MD , Tessa S. Cook MD PhD
6 min read

Rationale and Objectives

The advent of the ACGME’s Next Accreditation System represents a significant new challenge for residencies and fellowships, owing to its requirements for more complex and detailed information.

Material and Methods

We developed a system of online assessment tools to provide comprehensive coverage of the twelve ACGME Milestones and digitized them using freely available cloud-based productivity tools. These tools include a combination of point-of-care procedural assessments, electronic quizzes, online modules, and other data entry forms. Using free statistical analytic tools, we also developed an automated system for management, processing, and data reporting.

Results

After one year of use, our Milestones project has resulted in the submission of over 20,000 individual data points. The use of automated statistical methods to generate resident-specific profiles has allowed for dynamic reports of individual residents’ progress. These profiles both summarize data and also allow program directors access to more granular information as needed.

Conclusion

Informatics-driven strategies for data assessment and processing represent feasible solutions to Milestones assessment and analysis, reducing the potential administrative burden for program directors, residents, and staff.

Medical educators have witnessed a sea change in resident assessment over the past 30 years. For most of the history of academic medicine, the determination of resident competency for independent practice was based on the subjective assessment of the academic faculty who trained them. Since the founding of the Accreditation Council for Graduate Medical Education (ACGME) in 1981, resident education has experienced increasing structure, standardization, and accountability requirements with the goal to provide a more consistent level of quality in postgraduate training . Most recently, the ACGME has unveiled its Next Accreditation System (NAS) for resident education and introduced the concept of ACGME Milestones . Citing a public demand for greater accountability in medical education, these new standards strive to focus on outcomes, be learner centered, and prepare trainees for the evolving complexities of 21st century medicine . To accomplish these goals, the NAS and Milestones increase the granularity of information gathered on individual residents; the new standards require both frequent reporting of resident progress over time (every 6 months) and more detailed information about resident performance in the specific domains deemed critical to the practice of their chosen specialty.

In 2013, in collaboration with the American Board of Radiology, the ACGME established specialty-specific Milestones for Radiology , summarized in Table 1 . Starting in July 2013, radiology residency programs have been required to gather data and provide summary reports for individual residents to the ACGME . In 2013–2014, approximately 64% of residency programs provided some form of feedback to their residents about their Milestones progress .

Table 1

The 12 Radiology Milestones

From Vydareny et al. .

Abbreviation Full Name PCTS1 Patient Care and Technical Skills 1:

Consultant PCTS2 Patient Care and Technical Skills 2:

Competence in Procedures MK1 Medical Knowledge 1:

Protocol Selection and Optimization of Images MK2 Medical Knowledge 2:

Interpretation of Images PROF1 Professionalism 1:

Professional Values and Ethics ICS1 Interpersonal and Communication Skills 1:

Effective communication with patients, families, and caregivers. ICS2 Interpersonal and Communication Skills 2:

Effective communication with members of the health care team SBP1 Systems Based Practice 1:

Quality Improvement (QI) SBP2 Systems Based Practice 2:

Healthcare Economics PBL1 Practice Based Learning and Improvement 1:

Contrast Agents, Radiation Safety, MR Safety, Sedation PBL2 Practice Based Learning and Improvement 2:

Self-Directed Learning PBL3 Practice Based Learning and Improvement 3:

Scholarly Activity

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Assessment of the problem

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Table 2

Distribution of Assessment Burden Between Departmental Groups

Category Assessment Authors Faculty Technologists Staff Fellows Resident Procedural Ultrasound • • Image-guided biopsy • Interventional procedures • • Fluoroscopy (GI/GU) • Iodine 131 therapy • Lumbar punctures • • Pediatric fluoroscopy • Pediatric sonography • Handoff/signout • • Consent Procedural consent • • • MRI consent • Protocols Imaging protocols • Professionalism Professionalism 360 ∗ • • Scholarly activity Scholarly activity • CQI checklist • Journal club form • Quizzes Outside modules † • Online quizzes • Self-assessment modules (SAMs) •

CQI, Continuous Quality Improvement; IHI, Institute for Healthcare Improvement; MRI, Magnetic Resonance Imaging; RSNA, Radiological Society of North America.

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Implementation and technical details

Get Radiology Tree app to read full this article<

Table 3

Features and Key Components of Google Drive

Features Key Components Free Drive: 15 Gb of free cloud storage Integrated tools Forms: User-friendly online Web surveys Multiplatform Sheets: Database and analytic tools Mobile devices supported Sites: Free Web hosting Enables file sharing Gmail: E-mail server allowing automation. SSL encryption Script: Java-based scripting language

Get Radiology Tree app to read full this article<

Individual Resident Profiles/Postprocessing

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Assessing and reevaluating

Get Radiology Tree app to read full this article<

Figure 1, Examples of assessment style (a) and module style (b) forms constructed using Google Forms. Note the Milestone (eg, ICS1) key words, which are used to filter and aggregate the acquired data.

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Figure 2, Milestones submissions to date. The daily (a) and cumulative (b) number of Milestones form submissions from the beginning of our pilot study (May 9, 2013) to June 20, 2014. (c) Number of form submissions to date by category.

Figure 3, Example of mean class differences in Milestones as assessed by the Ultrasound Procedural Assessment. Mean scores with standard errors for the R1 ( dark ) and R2 ( light ) classes are shown. The PBL3, SBP2, and MK2 Milestones are not assessed with this specific assessment tool.

Figure 4, Portions of a sample individual resident profile. This montage provides screenshots from the top of a resident profile, cumulative accumulation of PCTS1 Milestones by semester, color-coded images documenting quiz scores and module completion, computer-estimated milestones progress, and 12-month conference attendance. A complete sample profile ( in the original color ) can be accessed at http://bit.ly/1lsPP56 .

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Discussion

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Acknowledgments

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

References

  • 1. Swing S.R., Beeson M.S., Carraccio C., et. al.: Educational Milestone Development in the First 7 Specialties to Enter the Next Accreditation System. J Grad Med Educ 2013; 5: pp. 98-106.

  • 2. Nasca T.J., Philbert I., Brigham T., et. al.: The Next GME Accreditation System—rationale and benefits. New Engl J Med 2012; 366: pp. 1051-1056.

  • 3. Nicholson B.T., Paladin A.M., Oldham S.A., et. al.: The Next Accreditation System in Radiology: a report from the APDR Residency Structure Committee. J Am Coll Radiol 2013; pp. 1-6.

  • 4. Vydareny K., Amis E., Borgstede J., et. al.: Diagnostic radiology milestones. J Grad Med Educ Supplement 2013; 5: pp. 74-78.

  • 5. ACGME. Frequently Asked Questions about the Next Accreditation System [Internet]. Available at: https://www.acgme.org/acgmeweb/Portals/0/PDFs/FAQ/420_Diagnostic_Radiology_FAQs.pdf . Accessed August 4, 2015.

  • 6. Shetty A., Hammer M., Gould J., et. al.: Results of the 2014 survey of the American Alliance of Academic Chief Residents in Radiology. Elsevier Ltd Academic radiology 2014; 21: pp. 1331-1347.

  • 7. Relyea-Chew A., Chew F.S.: Section Editor’s Notebook: using self-assessment modules to document milestone achievement in radiology residency training. Am J Roentgenol 2013; 201: pp. 1184-1185.

  • 8. Google. Google Sites: Storage and File Limits [Internet]. Available at: https://support.google.com/sites/answer/96770 . Accessed August 4, 2015.

  • 9. R Core Development Team: R: A language and environment for statistical computing.2012.R Foundation for Statistical ComputingVienna, Austria

  • 10. Lang D. RGoogleDocs: Primitive interface to Google Documents from R; 2012.

  • 11. Wickam H.: stringr: Make it easier to work with strings.2012.

  • 12. Dahl D.: xtable: Export tables to LaTeX or HTML.2014.

  • 13. Itri J.N., Kim W., Scanlon M.H.: Orion: a Web-based application designed to monitor resident and fellow performance on-call. J Digit Imaging 2011; pp. 897-907.

  • 14. Epstein R.M., Hundert E.M.: Professional competence. JAMA 2014; 287: pp. 226-235.

Academic Radiology, Volume 22, Issue 10
Journals General Radiology
This post is licensed under CC BY 4.0 by the author.
Recently Updated
Contents