Rationale and Objectives
A gold standard is often an imperfect diagnostic test, falling short of achieving 100% accuracy in clinical practice. Using an imperfect gold standard without fully comprehending its limitations and biases can lead to erroneous classification of patients with and without disease. This will ultimately affect treatment decisions and patient outcomes. Therefore, validation is essential before implementing a reference standard into practice. Performing a comprehensive validation process is discussed, along with its advantages and challenges. The different types of validation methods are reviewed. An example from our work in developing a new reference standard for vasospasm diagnosis in aneurysmal subarachnoid hemorrhage patients is provided.
Conclusion
Employing a new reference standard may result in a definitional shift of the disease and classification scheme of patients; therefore, it is important to also assess the impact of a new reference standard on patient outcomes and its clinical effectiveness.
The term gold standard is used to describe a diagnostic test that is regarded as definitive for a particular disease, and thereby becomes the ultimate measure for comparison. However, these gold standard methods are often imperfect and do not have 100% accuracy in practice. A “perfect” gold standard may only exist in theory and caution must be taken when classifying patients with a diagnosis based on a current gold standard. For example, colposcopy-directed biopsy of the cervix is the current gold standard for detection of cervical neoplasia. However, its sensitivity is only 60%, which would seem that it is far from being a definitive test for this patient population . In practice, diagnostic tests are sometimes assigned the status of gold standard, or as the “most sensitive test,” without verification . Using these imperfect gold standards without understanding their limitations may lead to erroneous classification of patients, affecting treatment decisions and patient outcomes. Therefore, validation of a new reference standard is an important step before its implementation in practice.
Validation is a term that has become synonymous with “accuracy.” To measure validity, or accuracy, is to assess how closely the results from a new diagnostic method approximate the current gold standard . However, a diagnostic test is not validated based on accuracy statistics alone. It is important to assess that the reference standard does indeed do what it is intended to do in the target population . The interpretation of these test results in the reference standard and the impact this has on patient outcomes are necessary factors to also consider in the validation process. Comprehensive evaluation of a reference standard includes assessing it clinical credibility, accuracy for diagnosis, generalizability to other target populations, and ideally assessing its clinical effectiveness .
Validation processes can be helpful in identifying flaws and bias in the reference standard that may result in misleading findings . These include selection bias, poorly defined inclusion and exclusion criteria, missing data, and unclear rationale for treatment decisions. Selection bias may exist in a reference standard that is only applicable to a subgroup of the target population. For example, digital subtraction angiography (DSA) is considered the gold standard for the diagnosis of vasospasm in aneurysmal subarachnoid hemorrhage (A-SAH) patients. Being an invasive technique with a complication rate of approximately 5% and a permanent stroke rate estimating 0.5%–1% , DSA is not performed in all patients. Because of these associated risks, patients with high suspicion of vasospasm based on symptoms or other imaging tests are more likely to have a DSA performed compared to patients without symptoms, resulting in selection bias. The performance of the gold standard in this subgroup of high-risk patients may not be the same for all patients in the population, thereby limiting its generalizability.
Our goal is to develop a reference standard that is applicable to the entire A-SAH population, including patients with and without symptoms, and, importantly, not to exclude patients that did not receive a particular imaging test. We hope that this novel approach in developing a reference standard, using both clinical and imaging criteria with consideration of treatment effects, will improve the accuracy in the classification of patients with and without vasospasm. This new reference standard will result in a definitional shift of the diagnosis of vasospasm, affecting the classification scheme and treatment decisions of A-SAH patients.
Development of a new reference standard for vasospasm
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Validation methods
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Discussion
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
References
1. Pfeiffer R.M., Castle P.E.: With or without a gold standard. Epidemiology 2005; 15: pp. 595-597.
2. Streiner D.L., Norman G.R.: “Precision” and “accuracy”: two terms that are neither. J Clin Epidemiol 2006; 59: pp. 327-330.
3. Altman D.G., Royston P.: What do we mean by validating a prognostic model?. Stat Med 2000; 19: pp. 453-473.
4. Simon R., Altman D.G.: Statistical aspects of prognostic factor studies in oncology. Br J Cancer 1994; 6: pp. 979-985.
5. Earnst F., Forbes G., Sandok B.A., et. al.: Complications of diagnostic cerebral angiography: prospective assessment of risk. Am J Roentgenol AJR 1984; 142: pp. 247-253.
6. Pryor J.C., Setton A., Nelson P.K., et. al.: Complications of diagnostic cerebral angiography and tips on avoidance. Neuroimaging Clin N Am 1996; 6: pp. 751-758.
7. Alonzo T.A., Pepe M.S.: Assessing the accuracy of a new diagnostic test when a gold standard does not exist. UW Biostatistics Working Paper Series 1998; pp. 3-32.
8. Reichman M., Greenberg E., Gold R., et. al.: Developing patient-centered outcome measures for evaluating vasospasm in aneurysmal subarachnoid hemorrhage. Acad Radiol 2009; 16: pp. 541-545.
9. Rabinstein A.A., Weigand S., Atkinson J.L.D., et. al.: Patterns of cerebral infarction in aneurysmal subarachnoid hemorrhage. Stroke 2005; 36: pp. 992-997.
10. Shimoda M., Takeuchi M., Tominaga J., et. al.: Asymptomatic versus symptomatic infarcts from vasospasm in patients with subarachnoid hemorrhage: serial magnetic resonance imaging. Neurosurgery 2001; 49: pp. 1341-1350.
11. Macdonald LR., Weir B.: Medical aspects of vasospasm. Cerebral vasospasm.2001.Academic Press 353–458
12. ten Bosch J.J., Angmar-Mansson B.: Characterization and validation of diagnostic methods. Monogr Oral Sci 2000; 17: pp. 174-189.
13. Glasziou P., Irwig L., Deeks J.J.: When should a new test become the current reference standard?. Ann Intern Med 2008; 149: pp. 816-821.
14. Rutjes A.W.S., Reitsma J.B., Coomarasamy A., et. al.: Evaluation of diagnostic tests when there is no gold standard. A review of methods. Health Technol Assess 2007; 50: pp. 11-12.
15. Fraile M., Rull M., Julian F.J., et. al.: Sentinel node biopsy as a practical alternative to axillary lymph node dissection in breast cancer patients: an approach to its validity. Ann Oncol 2000; 11: pp. 701-705.