In a recent critique of our paper on the use of free-response receiver-operating characteristic (FROC)–type studies during technology and practice assessments, Dr Chakraborty went to great lengths to defend his beliefs, and for that we applaud him. Unfortunately, being professionally vested in one approach, or analytic tool, has both advantages and disadvantages. Being both methodology developers and at the same time “in-the-trenches experimentalists” who perform some of the largest , as well as smaller , multireader, multicase FROC, receiver-operating characteristic, and binary response–type studies that are driven by specific clinical questions, we have a different perspective on this topic than Dr Chakraborty, and perhaps others. Our original perspective was not in any way intended to diminish the importance and/or validity of FROC studies in general. On the contrary, the primary messages we intended to convey are as follows:
1. Different clinical questions may be best addressed by different types of studies. Often, seemingly similar approaches, in terms of the type of observer performance study being performed, actually address (answer) different questions.
2. When applicable, the FROC method can be statistically a very powerful approach for addressing specific questions that include correct localization. However, this approach cannot, and should not, be recognized as the best approach to all clinically driven observer performance studies. In addition, like all other approaches, FROC analysis has methodologic advantages and disadvantages that need to be understood when deciding what approach and analytic tool to actually use to best answer a specific clinically relevant question.
3. Experimentalists should always first ask a specific clinically relevant question, then pose an appropriate hypothesis to be tested and, on the basis of the hypothesis, select the best approach to test the hypothesis in question. Hence, FROC-type studies and analyses have to be considered but one of the tools available to experimentalists, not the only tool!
4. Statistical power to demonstrate differences between technologies or practices is an important parameter or variable when selecting the type of study to perform only when the approach selected truly answers the very question being asked and does not represent a “forced” approach “because it is statistically more powerful.”
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
References
1. Gur D., Rockette H.E.: Performance assessments of diagnostic systems under the FROC paradigm: experimental, analytical, and results interpretation issues. Acad Radiol 2008; 15: pp. 1312-1315.
2. Chakraborty D.P.: Counterpoint to “Performance assessment of diagnostic systems under the FROC paradigm” by Gur and Rockette. Acad Radiol 2009; 16: pp. 507-510.
3. Bandos A.I., Rockette H.E., Song T., Gur D.: Area under the free-response ROC curve (FROC) and a related summary index. Biometrics 2009; 65: pp. 247-256.
4. Gur D., Rockette H.E., Armfield D.R., et. al.: Prevalence effect in a laboratory environment. Radiology 2003; 228: pp. 10-14.
5. Gur D., Bandos A.I., Cohen C.S., et. al.: The “laboratory” effect: comparing radiologists’ performance and variability during prospective clinical and laboratory mammography interpretations. Radiology 2008; 249: pp. 47-53.
6. Herron J.M., Bender T.M., Campbell W.L., Sumkin J.H., Rockette H.E., Gur D.: Effects of luminance and resolution on observer performance with chest radiographs. Radiology 2000; 215: pp. 169-174.
7. Good W.F., Abrams G.S., Catullo V.J., et. al.: Digital breast tomosynthesis: a pilot observer study. AJR Am J Roentgenol 2008; 190: pp. 865-869.
8. Gur D, Abrams GS, Chough DM, et al. Digital breast tomosynthesis—an observer performance study. AJR Am J Roentgenol. In press.
9. Gur D., Bandos A.I., Klym A.H., et. al.: Agreement of the order of overall performance levels under different reading paradigms. Acad Radiol 2008; 15: pp. 1567-1573.