Rationale and Objectives
Health agencies making regulatory marketing-authorization decisions use qualitative and quantitative approaches to assess expected benefits and expected risks associated with medical interventions. There is, however, no universal standard approach that regulatory agencies consistently use to conduct benefit-risk assessment (BRA) for pharmaceuticals or medical devices, including for imaging technologies. Economics, health services research, and health outcomes research use quantitative approaches to elicit preferences of stakeholders, identify priorities, and model health conditions and health intervention effects.
Materials and Methods
Challenges to BRA in medical devices are outlined, highlighting additional barriers in radiology. Three quantitative methods—multi-criteria decision analysis, health outcomes modeling and stated-choice survey—are assessed using criteria that are important in balancing benefits and risks of medical devices and imaging technologies.
Results
To be useful in regulatory BRA, quantitative methods need to: aggregate multiple benefits and risks, incorporate qualitative considerations, account for uncertainty, and make clear whose preferences/priorities are being used. Each quantitative method performs differently across these criteria and little is known about how BRA estimates and conclusions vary by approach. While no specific quantitative method is likely to be the strongest in all of the important areas, quantitative methods may have a place in BRA of medical devices and radiology.
Discussion
Quantitative BRA approaches have been more widely applied in medicines, with fewer BRAs in devices. Despite substantial differences in characteristics of pharmaceuticals and devices, BRA methods may be as applicable to medical devices and imaging technologies as they are to pharmaceuticals. Further research to guide the development and selection of quantitative BRA methods for medical devices and imaging technologies is needed.
Weighing expected benefits against expected risks is essential for health agencies making regulatory marketing-authorization decisions. Benefit–risk balance of medical interventions also influences adoption and use after approval, although costs become a major factor as well. The US Food and Drug Administration (FDA) and the European Medicines Agency (EMA) explicitly and implicitly weigh these tradeoffs in determining whether new pharmaceutical products and medical devices, such as diagnostic imaging tests, have met minimum standards for overall quality, that is, safety- (risk) and benefit-related effects. In 1998, a World Health Organization–United Nations Education, Scientific and Cultural Organization encouraged formalizing frameworks and methods for benefit–risk assessment (BRA) in a report calling for improvements in existing approaches to assessing pharmaceutical products . Greater attention has been applied to pharmaceutical products in the implementation of quantitative BRA methods, although initiatives have also focused on processes for medical devices . In 2012 alone, three publications from the FDA, EMA, and the Agency for Health Research and Quality (AHRQ) have addressed BRA methods for pharmaceutical products or devices . Although there are distinct qualities or characteristics associated with diagnostic tests compared to medicines as “economic goods,” quantitative BRA methods may be useful—in terms of improving consistency, reliability, and transparency—via their application to device-related premarket decision-making processes.
The aim of this article is to identify and discuss opportunities and challenges associated with approaches to quantitative BRA applied to medical devices. First, we describe ways in which the environment for BRA assessment for devices differ from that of pharmaceutical products, and we focus on key issues of the device regulatory environment—predicate creep, multiple applications, and behavioral factors. As an important subcategory within medical devices, relevant applications to diagnostic imaging receive special attention. Second, we describe three promising methods by which quantitative BRA assessment may be conducted in the area of diagnostic imaging, pointing out positive features and limitations of each. Researchers and regulators may wish to formalize approaches to this evolving field. We suggest that policy changes and additional research are needed to better assess the expected benefits and expected risks of devices in specific applications, such as diagnostic imaging tests.
Comparing Benefit–Risk assessment for devices and medicines
Regulatory Environment
Get Radiology Tree app to read full this article<
Predicate Creep
Get Radiology Tree app to read full this article<
Multiple Applications of Interventions
Get Radiology Tree app to read full this article<
Behavioral Factors
Get Radiology Tree app to read full this article<
Imaging Procedures
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Potential advantages and limitations of select BRA methods
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Table 1
Potential Advantages and Disadvantage of Three Quantitative Methods: Relative Performance by Four Criteria
Criteria Multicriteria Decision Analysis Health Outcomes Modeling Stated-Choice Surveys Assessing multiple risks and benefits The Simplest and most direct method for weighing multiple criteria or attributes. Easily performed with ranking or rating methods. Multiple benefits and risks are combined into a single metric—net health benefit; results may not be transparent. Sample size requirements increase dramatically as risk and benefit criteria increase; cognitive burden of multiplicity may become problematic. Scoring or prioritizing qualitative criteria (eg, risk tolerance, severity of disease, and so forth) Qualitative criteria can be given weights and then combined with quantitative criteria into a single metric. May require the use of other methods to convert qualitative criteria into quantitative values although, in principle, the framework can be extended. Preference elicitation techniques may be used to quantify preferences for qualitative criteria. Accounting for uncertainty Uncertainty with regard to performance or preference weights of attributes may be explored with sensitivity analyses. Typically, no account is made for quality of information by way of confidence or credible intervals. One-way sensitivity analysis testing the impact of modifying individual parameters is generally a minimum standard. More advanced probabilistic sensitivity analyses incorporate distributions of inputs rather than fixed estimates. Excellent method for estimating tolerance for uncertainty. Respondents may struggle with assigning preferences to low probability events. Accounting for stakeholder preferences Makes explicit the decision makers’ criteria weights of the decision makers. May have poor generalizability because of small sample sizes. Regulatory decision makers have not been receptive to the use of patient health state utilities combined with epidemiological models to project QALYs. A more formal method for eliciting stakeholder preferences via respondents choosing among specific discrete sets of attributes and levels of attributes. Best Applications When time and resources are very constrained and patient preferences and health outcomes are not known and cannot be collected in the time for decision making. When longer term outcomes influence the benefit–harm balance and/or there is substantial uncertainty around point estimates for quantifying benefits and/or harms. When patients’ tolerance for uncertainty strongly influences the benefit–harm balance or when identifying preferences for specific attributes is highly relevant or useful.
Advantages and disadvantages of the methods are subjectively appraised by the authors.
Get Radiology Tree app to read full this article<
Multicriteria Decision Analysis
Get Radiology Tree app to read full this article<
Health Outcomes Modeling
Get Radiology Tree app to read full this article<
Stated-Choice Surveys
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Conclusions
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
References
1. Council for International Organizations of Medical Sciences (CIOMS). Benefit-risk balance for marketed drugs: evaluating safety signals. Report of CIOMS Working Group IV , http://www.cioms.ch/publications/g4-benefit-risk.pdf . (accessed October 12, 2012)
2. FDA: Guidance for industry: premarketing risk assessment.2005.Center for Drug Evaluation and Research (CDER) and the Center for Biologics Evaluation and Research (CBER) (accessed January 2013) http://www.fda.gov/downloads/Drugs/GuidanceComplianceRegulatoryInformation/Guidances/ucm072002.pdf
3. European Medicine Agency (EMEA) CHMP. Benefit-risk methodology project: Work Package 2 report: applicability of current tools and processes for regulatory benefit-risk assessment. 2010; http://www.ema.europa.eu/ema/index.jsp?curl=pages/special_topics/document_listing/document_listing_000314.jsp&mid=WC0b01ac0580223ed6#section2 (accessed October 22, 2012)
4. FDA. Draft guidance for industry and Food and Drug Administration Staff: the 510(k) program: evaluating substantial equivalence in premarket notifications [510(k)]. 2011; http://www.fda.gov/downloads/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/UCM284443.pdf (accessed October 26, 2012)
5. Drummond M., Griffin A., Tarricone R.: Economic evaluation for devices and drugs—same or different?. Value Health 2009; 12: pp. 402-404.
6. Boyd C., Singh S., Varadhan R., et. al.: Methods for benefit and harm assessment in systematic reviews.2012.Agency for Healthcare Research and Quality (AHRQ) Publication No. 12(13)-EHC150-EFRockville, MD
7. Office of Device Evaluation. Fiscal year 2009 ODE annual report. 2009; http://www.fda.gov/downloads/AboutFDA/CentersOffices/CDRH/CDRHReports/UCM223893.pdf (accessed December 7, 2012)
8. Center for Device and Radiological Health. Ten year comparison of new molecular entities. 2011; http://www.fda.gov/downloads/Drugs/DevelopmentApprovalProcess/DrugInnovation/ucm293663.pdf (accessed March 6, 2014)
9. Government Accountability Office. Medical devices: FDA should take steps to ensure that high-risk device types are approved through the most stringent premarket review process. 2009 (Publication No. GAO-09–190). http://www.gao.gov/new.items/d09190.pdf .
10. Tunis S.R., Stryer D.B., Clancy C.M.: Practical clinical trials: increasing the value of clinical research for decision making in clinical and health policy. JAMA 2003; 290: pp. 1624-1632.
11. Hines J.Z., Lurie P., Yu E., et. al.: Left to their own devices: breakdowns in United States medical device premarket review. PLoS Med 2010; 7: pp. e1000280.
12. Matchar D.B.: Chapter 1: Introduction to the methods Guide for Medical Test Reviews. J Gen Intern Med Jun 2012; 27 Suppl 1: pp. S4-S10.
13. Gazelle G.S., McMahon P.M., Siebert U., et. al.: Cost-effectiveness analysis in the assessment of diagnostic imaging technologies. Radiology 2005; 235: pp. 361-370.
14. Brown S.D.: Professional norms regarding how radiologists handle incidental findings. J Am Coll Radiol Apr 2013; 10: pp. 253-257.
15. Mussen F., Salek S., Walker S.: A quantitative approach to benefit-risk assessment of medicines—part 1: the development of a new model using multi-criteria decision analysis. Pharmacoepidemiol Drug Saf 2007; 16: pp. S2-S15.
16. Drummond M.F., O’Brien B., Stoddart G.L., et. al.: Methods for the economic evaluation of health care programmes.2nd ed1997.Oxford University PressOxford
17. Bridges J.P.F., Hauber A.B., Marshall D., et. al.: Conjoint analysis applications in health—a checklist: a report of the ISPOR Good Research Practices for Conjoint Analysis Task Force. Value in Health 2011; 14: pp. 403-413.
18. Johnson F.R., Lancsar E., Marshall D., et. al.: Constructing experimental designs for discrete-choice experiments: report of the ISPOR conjoint analysis experimental design good research practices task force. Value in Health 2013; 16: pp. 3-13.
19. Johnson F.R., Ozdemir S., Mansfield C., et. al.: Crohn’s disease patients’ risk-benefit preferences: serious adverse event risks versus treatment efficacy. Gastroenterology 2007; 133: pp. 769-779.
20. Johnson F.R., Van Houtven G., Ozdemir S., et. al.: Multiple sclerosis patients’ benefit-risk preferences: serious adverse event risks versus treatment efficacy. J Neurol 2009; 256: pp. 554-562.
21. Hauber A.B., Fairchild A.O., Johnson F.R.: Quantifying Benefit-Risk Preferences for Medical Interventions: an overview of a growing empirical literature. Appl Health Econ Health Policy 2013;
22. FDA. Guidance for industry and Food and Drug Administration staff: factors to consider when making benefit-risk determinations in medical device premarket approval and de novo classifications. 2012; http://www.fda.gov/downloads/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/UCM296379.pdf (accessed October 22, 2012)
23. FDA. Structured approach to benefit-risk assessment in drug regulatory decision-making: PDUFA V Plan (FY 2013-2017) Draft of February 2013. http://www.fda.gov/downloads/ForIndustry/UserFees/PrescriptionDrugUserFee/UCM329758.pdf (accessed January 24, 2014)
24. Hughes D, Waddingham EDJ, Mt-Isa S, et al. IMI-PROTECT Benefit-Risk Group RECOMMENDATIONS REPORT (Work Package 5): Recommendations for the methodology and visualisation techniques to be used in the assessment of benefit and risk of medicines. European Medicines Agency and Pharmacoepidemiological Research on Outcomes of Therapeutics by a European ConsorTium (PROTECT). 2013. http://www.imi-protect.eu/benefitsRep.shtml (accessed March 1, 2014)