Inherent Limitations of Multiple-Choice Testing

We cannot become what we need to be by remaining what we are. —Max DePree

Ironically, as radiology increases its reliance on multiple-choice assessment, other venerable bastions of multiple-choice testing are moving away from it. For example, the College Board, which designs and administers the SAT and Advanced Placement (AP) subject tests, added a free-response writing section to the SAT in 2005 . The number of multiple-choice questions on the AP science exams is being decreased by nearly half, while the number of free-response questions has increased from four to seven . What lessons might such changes offer to learners and educators about the appropriate role of multiple-choice testing in radiology education and certification?

Some in radiology may find it difficult to believe, but the form of educational assessment we know as the multiple-choice question is less than a century old. Before 1917, primary, secondary, undergraduate, graduate, and professional schools assessed the performance of learners without presenting a series of questions that ask them to choose the best response from four or five alternatives . Today by the time contemporary radiology residents sit down to take one of the American Board of Radiology’s exams, they have taken dozens and even hundreds of multiple-choice tests, a testing format that many of them (and their teachers) have come to regard as unavoidable.

Yet like any approach to educational assessment, multiple-choice testing has both strengths and weaknesses, and it is important that educators understand both if they are to do the best job of assessing learner performance. There are some things that the multiple-choice format does well, others it does poorly, and some it does not do at all. In what follows, we will briefly review some of the more important advantages of multiple-choice testing, and then devote the bulk of our attention to its disadvantages. Multiple-choice testing is now so pervasive that hardly anyone needs to argue on its behalf, but its drawbacks frequently go unrecognized.

Why is multiple-choice testing so popular today? For one thing, multiple-choice examinations are quick, making it possible to cover a large range of material in a relatively short time. They are also relatively inexpensive, because the same questions and variations are often used again and again, and computers can be relied on to perform the scoring. And because they are standardized–administered and scored essentially the same way for every learner every time—such tests appear to be objective and fair . Examiner bias is reduced, at least in the sense that evaluators’ personal impressions of examinees are kept out of the picture.

It is important to add that efforts are under way to improve the quality of multiple-choice testing. For example, a number of professional organizations sponsor item-writing workshops, the purpose of which is to craft better questions. Such questions are often pretested, to see how well they discriminate between examinees. Knowledgeable test designers and item writers are aware of the inherent weaknesses of multiple-choice questions, such as the fact that they are relatively good at testing recall and application but not so good at analysis, evaluation, and creativity. Like the College Board, some medical organizations are attempting to compensate for the inherent weaknesses of multiple-choice testing by shifting partially to other methods of evaluation, such as learner portfolios.

Get Radiology Tree app to read full this article<

References

1. Improving paragraphs. In: The official SAT study guide. The College Board. 2009. p. 169.
2. College Board announces changes to AP Biology Course and Exam. The College Board, Released Feb. 1, 2011. http://press.collegeboard.org/releases/2011/content/college-board-announces-changes-ap-biology-course-and-exam . Accessed January 5, 2013.
3. Hakel M.D.: Into the great beyond.Hakel M.D.Beyond multiple choice: evaluating alternatives to traditional testing for selection.1998.Lawrence Erlbaum AssociatesMahwah, NJ: p. 2
4. Wilson R.B., Case S.M.: Extended matching questions: an alternative to multiple-choice or free-response questions. J Vet Med Educ 1993; 20: pp. 3. http://scholar.lib.vt.edu/ejournals/JVME/V20-3/wilson.html Accessed Dec. 20th, 2012
5. Kiong Lau P.N., Hoe Lau S., Hong S.K., et. al.: Guessing, partial knowledge, and misconceptions in multiple choice tests. Educ Tech Soc 2011; 14: pp. 99-110.
6. Roediger H.L., Marsh E.J.: The positive and negative consequences of multiple-choice testing. J Exp Psych Learn Memory Cogn 2005; 31: pp. 1155-1159.
7. Berlin L.: The ABR ‘recalls’ conundrum: an ethical quandary. J Am Coll Radiol 2012; 9: p. 380–383
8. Hall F.M.: The ABR and resident recall ‘cheating.’. Radiology 2012; 263: p. 323–325
9. Applebome P. Insisting it’s nothing, creator says SAT, Not S.A.T. The New York Times April 2, 1997. http://www.nytimes.com/1997/04/02/us/insisting-it-is-nothing-creator-says-sat-not-sat.html . Accessed January 5, 2013.
10. Rothstein J.M.: College performance predictions and the SAT. J Econometr 2004; 121: p. 297–317
11. Callahan C.A., Hojat M., Veloski J., et. al.: The predictive validity of three versions of the MCAT in relation to performance in medical school, residency, and licensing examinations: a longitudinal study of 36 classes of Jefferson Medical College. Acad Med June 2010; 85: p. 980–987
12. Simkin M.G., Kuechler W.L.: Multiple-choice tests and student understanding: what is the connection?. Decis Sci J Innov Educ 2005; 3: p. 73–97
13. Gunderman R.B., Ballenger Z., Heitkamp D.E.: Avoiding testocracy. Radiology 2012; 265: p. 332–335
14. ‘No Child Left Behind’ gets mixed grades. Pew Center Released June 13, 2007. http://www.people-press.org/2007/06/13/no-child-left-behind-gets-mixed-grades/ . Accessed January 5, 2013.

Inherent Limitations of Multiple-Choice Testing

References

Further Reading

A Comparative Analysis of Water-Soluble and Blood-Pool Contrast Agents for in vivo Vascular Imaging with Micro-CT

A Novel Analysis Algorithm for Potential Quantitative Assessment of Myocardial Computed Tomography Perfusion

Academic Radiology in 2020