Rationale and Objectives
In clinical guideline or criteria development processes, such as those used in developing American College of Radiology Appropriateness Criteria (ACR AC), experts subjectively evaluate benefits and risks associated with imaging tests and make complex decisions about imaging recommendations. The analytic hierarchy process (AHP) decomposes complex decisions into structured smaller decisions, incorporates quantitative evidence and qualitative expert opinion, and promotes structured consensus building. AHP may supplement and/or improve the transparency of expert opinion contributions to developing guidelines or criteria.
Materials and Methods
To conduct an empirical test using health services research tools, we convened a mock ACR AC panel of emergency department radiology and nonradiology physicians to evaluate by multicriteria decision analysis, the relative appropriateness of imaging tests for diagnosing suspected appendicitis. Panel members selected benefit-risk criteria via an online survey and assessed contrast-enhanced computed tomography, magnetic resonance imaging, and ultrasound using an AHP-based software. Participants were asked whether the process was manageable, transparent, and improved shared understanding. Priority scores were converted to rankings and compared to the rank order of ACR AC ratings.
Results
When compared to magnetic resonance and ultrasound imaging, participants agreed with the ACR AC that contrast-enhanced computed tomography is the most appropriate test. Contrary to the ACR AC ratings, study results suggest that magnetic resonance is preferable to ultrasound. When compared to nonradiologists, radiologists’ priority scores reflect a stronger preference for computed tomography.
Conclusions
Study participants addressed decision-making challenges using a relatively efficient data collection mechanism, suggesting that AHP may benefit the ACR AC guideline development process in identifying the relative appropriateness of imaging tests. With additional development, AHP may improve transparency when expert opinion is used in clinical guideline or appropriateness criteria development.
Introduction
The American College of Radiology (ACR) publishes evidence-based and opinion-based criteria outlining appropriate uses of imaging tests . ACR Appropriateness Criteria (ACR AC) are developed and revised every 3 years by panels composed of 10–16 volunteer ACR members . Panel members rate imaging tests on a scale that ranges from 1 to 9 (1–3, inappropriate; 4–6, equivocal; 7–9, appropriate) using the RAND/UCLA Appropriateness Method . The ACR AC quantitative ratings represent qualitative reconciliation of benefits and risks into one measure that supports theultimate decision (or recommendation) of the radiologist expert panel. For a given clinical scenario, referred to as a topic variant, panel members assess the risks of each test against the benefits of performing the procedure. Expert opinion is used to address evidence gaps and supplements existing evidence .
Decision scientists have shown that individuals struggle with complex decisions involving multiple objectives with uncertain trade-offs . As the number of alternatives and criteria judgments increases, individuals’ decision-making capabilities degrade . In this context, ACR AC expert panel members may face several challenges: (1) rating multiple imaging alternatives for any given clinical indication; (2) assignment of importance to multiple potential benefits and risks; (3) as volunteers, working with limited financial resources and time; (4) decision making in an environment of high uncertainty with regard to benefits and risks across alternatives; and (5) given divergent views, arriving at a single metric representing the benefit-risk balance, or appropriateness. While the modified Delphi consensus approach of the RAND/UCLA Appropriateness Method assists panel members with reaching consensus, support for decomposing complex decisions, individually or in groups, is lacking.
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Methods
Selection and Definition of Clinical Use Case
Get Radiology Tree app to read full this article<
Selection of MCDA Method
Get Radiology Tree app to read full this article<
Mock ACR AC Panel Participant Recruitment
Get Radiology Tree app to read full this article<
Selection of Imaging Technologies
Get Radiology Tree app to read full this article<
Premeeting Survey
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Mock ACR AC Meeting
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Results
Mock ACR AC Panel Participants
Get Radiology Tree app to read full this article<
Premeeting Survey: Selection of Relevant Criteria
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Comparison of Mock Panel Criteria Selection to the ACR AC Considerations
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
AHP Voting Results
Criteria Weights
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Imaging Test Priority Scores
Get Radiology Tree app to read full this article<
TABLE 1
Priority Scores by Goal and Criterion
Goals Criteria US MRI ceCT Relative Performance US MRI ceCT Goal 1 Minimize time to diagnosis 0.1859 0.3493 0.4629
Goal 1a Maximize provider utility 0.1364 0.2702 0.3659
Increase diagnostic accuracy consistency 0.0576 0.2304 0.2304
Maximize access to test 0.0349 0.0121 0.1006
Minimize patient-specific exclusions 0.0440 0.0277 0.0349
Goal 1b Decrease burden to patient 0.0494 0.0791 0.0970
Minimize potential for additional confirmatory testing 0.0107 0.0679 0.0714
Minimize incidental findings management 0.0357 0.0083 0.0077
Minimize examination time 0.0030 0.0030 0.0179
Goal 2 Minimize potential harms to patient 0.2454 0.3574 0.3231
Missed cases 0.0454 0.1651 0.3000
Minimize ionizing radiation dose 0.1000 0.0961 0.0116
Minimize contrast reaction potential 0.1000 0.0961 0.0116
Relative performance, by goal, subgoal, and criteria, is depicted in the far right column in the following order: US, MRI, ceCT. Dark gray bars distinguish the highest scoring technology.
ceCT, contrast-enhanced computed tomography; MRI, magnetic resonance imaging; US, ultrasound.
Get Radiology Tree app to read full this article<
Inconsistency Check and Sensitivity Analyses
Get Radiology Tree app to read full this article<
Panel Experience
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Discussion
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Acknowledgments
Get Radiology Tree app to read full this article<
Appendix
Supplementary Data
Get Radiology Tree app to read full this article<
File S1
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
File S2
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Table S1
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Table S2
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Table S3
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
References
1. Cascade P.N.: The American College of Radiology. ACR appropriateness criteria project. Radiology 2000; 214: pp. 3-46.
2. ACR Appropriateness Criteria : ACR Appropriateness Criteria® organization and composition of expert panels. https://www.acr.org/~/media/ACR/Documents/AppCriteria/OrgCompExpertPanel.pdf?la=en Accessed March 23, 2017
3. American College of Radiology : About the ACR appropriateness criteria. http://www.acr.org/Quality-Safety/Appropriateness-Criteria/About-AC Accessed March 23, 2017
4. Brook R.H.: The RAND/UCLA appropriateness method.1994.Public Health Service, U.S. Department of Health and Human ServicesRockville, MD
5. Keeney R.L., Raiffa H.: Decisions with multiple objectives: performances and value trade-offs.1976.John Wiley and SonsNew York
6. Riabacke M., Danielson M., Ekenberg L.: State-of-the-art prescriptive criteria weight elicitation. Adv Decision Sci 2012; 2012: 276584
7. Belton V.S.T.J.: Multiple criteria decision analysis: an integrated approach.2002.Kluwer Academic PublishersBoston
8. Hunink M.G.M., Glasziou P.P., Siegel J.E., et. al.: Decision making in health and medicine: integrating evidence and values.2001.Cambridge University PressCambridge
9. Linkov I., Moberg E.: Multi-criteria decision analysis environmental applications and case studies. http://public.eblib.com/choice/publicfullrecord.aspx?p=952014 Accessed March 23, 2017
10. Adunlin G., Diaby V., Xiao H.: Application of multicriteria decision analysis in health care: a systematic review and bibliometric analysis. Health Expect 2015; 18: pp. 1894-1905.
11. Smith M.P., Katz D.S., Lalani T., et. al.: ACR appropriateness criteria(R) right lower quadrant pain-suspected appendicitis. Ultrasound Q 2015; 31: pp. 85-91.
12. Karul M., Berliner C., Keller S., et. al.: Imaging of appendicitis in adults. Rofo 2014; 186: pp. 551-558.
13. Heverhagen J.T., Pfestroff K., Heverhagen A.E., et. al.: Diagnostic accuracy of magnetic resonance imaging: a prospective evaluation of patients with suspected appendicitis (diamond). J Magn Reson Imaging 2012; 35: pp. 617-623.
14. American College of Radiology : Appropriateness criteria. https://acsearch.acr.org/list Accessed March 23, 2017
15. ACR Appropriateness Criteria : ACR Appropriateness Criteria® evidence table development—diagnostic studies. https://www.acr.org/~/media/ACR/Documents/AppCriteria/EvidenceTableDevelopment.pdf?db=web Accessed March 23, 2017
16. Dong Q., Saaty T.: An analytic hierarchy process model of group consensus. J Syst Sci Syst Eng 2014; 23: pp. 362-374.
17. Aczél J., Saaty T.L.: Procedures for synthesizing ratio judgements. J Math Psychol 1983; 27: pp. 93-102.
18. Hummel J.M., Bridges J.F., IJzerman M.J.: Group decision making with the analytic hierarchy process in benefit-risk assessment: a tutorial. Patient 2014; 7: pp. 129-140.
19. Saaty T.L.: The analytic hierarchy process: planning, priority setting, resource allocation.1980.McGraw-Hill International Book Co.New York
20. Hilgerink M.P., Hummel M.J., Manohar S., et. al.: Assessment of the added value of the Twente Photoacoustic Mammoscope in breast cancer diagnosis. Med Devices (Auckl) 2011; 4: pp. 107-115.
21. Maruthur N.M., Joy S.M., Dolan J.G., et. al.: Use of the analytic hierarchy process for medication decision-making in type 2 diabetes. PLoS ONE 2015; 10: pp. e0126625.
22. Poletti P.A., Platon A., De Perrot T., et. al.: Acute appendicitis: prospective evaluation of a diagnostic algorithm integrating ultrasound and low-dose CT to reduce the need of standard CT. Eur Radiol 2011; 21: pp. 2558-2566.
23. Figueira J., Greco S., Ehrgott M.: Multiple criteria decision analysis: state of the art surveys.2005.SpringerNew York
24. Broekhuizen H., Groothuis-Oudshoorn C.G., van Til J.A., et. al.: A review and classification of approaches for dealing with uncertainty in multi-criteria decision analysis for healthcare decisions. Pharmacoeconomics 2015; 33: pp. 445-455.
25. Millet I., Wedley W.C.: Modelling risk and uncertainty with the analytic hierarchy process. J Multicriteria Decision Anal 2002; 11: pp. 97-107.
26. Dolan J.G.: Involving patients in decisions regarding preventive health interventions using the analytic hierarchy process. Health Expect 2000; 3: pp. 37-45.
27. Bots P.W.G., Hulshof J.A.M.: Designing multi-criteria decision analysis processes for priority setting in health policy. J Multicriteria Decision Anal 2000; 9: pp. 56-75.