Assessing Diagnostic Confidence

Rationale and Objectives

The ability of a test to influence diagnostic confidence is used as a measure of its efficacy. Our aim was to compare analytic methods that evaluate changes in confidence.

Materials and Methods

The approaches compared were “basic,” “retained diagnoses,” “Omary,” “Tsushima,” and “score-based” methods. For illustration, data from a clinical study assessing changes in diagnostic confidence (0%–100%) before and after abdominopelvic computed tomography (CT) in patients with acute abdominal pain were used.

Results

The basic, retained diagnoses and Omary methods all ignore whether the test yields a correct diagnosis (confident, but incorrect, diagnoses are regarded positively). Although the Tsushima method takes some account of diagnostic accuracy, all misdiagnoses are considered equal. The score-based method addresses some of the fundamental limitations in the other analyticl methods, such as diagnostic accuracy and the varying nature of different misdiagnoses. In the case study, mean (SD) diagnostic confidence for the cohort as a whole (n = 62) increased following CT: 50.7% (20.8%) to 73.2% (20.9%). Pretest diagnoses were changed following CT in 43% (27 of 62) of patients. Pretest diagnoses proved to be incorrect in 52% (32 of 62), and post-test diagnoses incorrect in as many as 19% (12 of 62) of patients. All five analytic methods indicated a positive contribution for CT (all P ≤ .003).

Conclusion

Although our illustrative case study revealed no consequential differences across the five methods, there remain substantial differences in the fundamental principles underlying them that should affect choice of analytic method when assessing diagnostic confidence.

The degree of confidence in a diagnosis is an important factor in patient management. If confidence in a particular diagnosis is high, management usually proceeds on that basis; if there is diagnostic uncertainty, other diagnostic possibilities are generally pursued.

The effect that a diagnostic test has on diagnostic confidence is used as one assessment of its efficacy ( ). A number of analytic methods have been used to assess how changes in pretest compared to post-test confidences can be used to assess such diagnostic tests.

Get Radiology Tree app to read full this article<

Materials and methods

Basic Analytic Method

Get Radiology Tree app to read full this article<

Analyses Incorporating Changes in Diagnoses

Get Radiology Tree app to read full this article<

Retained diagnoses

Get Radiology Tree app to read full this article<

“Omary” correction

Get Radiology Tree app to read full this article<

Analyses Incorporating Diagnostic Changes and Diagnostic Accuracy

Get Radiology Tree app to read full this article<

“Tsushima” method

Get Radiology Tree app to read full this article<

“Score-based” method

Get Radiology Tree app to read full this article<

Illustrative Case Study

Get Radiology Tree app to read full this article<

Statistical Analyses

Get Radiology Tree app to read full this article<

Results

Patients and Diagnoses in the Case Study

Get Radiology Tree app to read full this article<

Numerical Comparison of Analytic Methods in the Case Study

Get Radiology Tree app to read full this article<

Table 1

Summary of Evaluation of Changes in Diagnostic Confidence, by Analytic Method, in the Illustrative Case Study

Analytical Method No. of Patients Contributing to Analysis Mean Change in Confidence (SD) Estimated Difference (95% CI)t -Test Value_P_ -Value Basic method 62 22.6% (24.7%) (16.3–28.9) 7.20 <.0001 Retained diagnosis 35 17.7% (23.1%) (9.8–25.7) 4.53 <.0001 Omary method 62 26.5% (23.4%) (20.5–32.4) 8.90 <.0001 Tsushima method 62 37.7% (69.6%) (20.1–55.4) 4.27 <.0001 Score-based method 62 0.68 (1.73) (0.24–1.12) 3.10 .003

Get Radiology Tree app to read full this article<

Discussion

Get Radiology Tree app to read full this article<

Analytic Methods That Do Not Take Diagnostic Accuracy Into Consideration

Get Radiology Tree app to read full this article<

Analytic Methods That Take Diagnostic Accuracy Into Consideration

Get Radiology Tree app to read full this article<

Fundamental Differences Between Tsushima and Score-Based Methods

Get Radiology Tree app to read full this article<

Similarities and Differences Across All Methods

Get Radiology Tree app to read full this article<

Table 2

Scenarios Illustrating the Changes in Confidence as Assessed by the Five Different Analytic Methods

Scenario Diagnosis Diagnostic Confidence Analytic Method Derivation of Confidence Change Estimated Confidence Change Direction of Confidence Change 1 Pretest Appendicitis 70% Basic method 90%−70% +20% + Post-test Renal colic 90% Retained NA NA NA Actual Appendicitis Omary 90% − (100% − 70%) +60% + Tsushima Type (e) case ⁎ : (−90%) − (70%) −160% − Score-based CT confidently overturns initial correct severe diagnosis −3.83 † − 2 Pretest Renal colic 70% Basic method 50%−70% −20% − Post-test Renal colic 50% Retained 50%−70% −20% − Actual Appendicitis Omary 50% − (100% − 70%) +20% + Tsushima Type (c) case ⁎ : (−50%) − (−70%) +20% + Score-based CT wrongly confirms undercalling a severe diagnosis −1.17 † − 3 Pretest Appendicitis 70% Basic method 50%−70% −20% − Post-test Appendicitis 50% Retained 50%−70% −20% − Actual Renal colic Omary 50% − (100% − 70%) +20% + Tsushima Type (c) case ⁎ : (−50%) − (−70%) +20% + Score-based CT wrongly confirms overcalling a less severe diagnosis −0.33 † − 4 Pretest Renal colic 70% Basic method 50%−70% −20% − Post-test Ovarian cyst 50% Retained NA NA NA Actual Appendicitis Omary 50% − (100% − 70%) +20% + Tsushima Type (c) case ⁎ : (−50%) − (−70%) +20% + Score-based CT changes diagnosis but wrongly undercalls a severe diagnosis −2.83 † −

NA, not applicable; +, the test is considered to be positive or beneficial (increases confidence); −, the test is considered to be negative, harmful, or detrimental (decreases confidence).

Get Radiology Tree app to read full this article<

References

1. Fryback D.G., Thornbury J.R.: The efficacy of diagnostic imaging. Med Decis Making 1991; 11: pp. 88-94.
2. Mackenzie R., Dixon A.K.: Measuring the effects of imaging: An evaluative framework. Clin Radiol 1995; 50: pp. 513-518.
3. Anzilotti K., Schweitzer M.E., Hecht P., Wapner K., Kahn M., Ross M.: Effect of foot and ankle MR imaging on clinical decision making. Radiology 1996; 201: pp. 515-517.
4. Blanchard T.K., Bearcroft P.W., Constant C.R., Griffin D.R., Dixon A.K.: Diagnostic and therapeutic impact of MRI and arthrography in the investigation of full-thickness rotator cuff tears. Eur Radiol 1999; 9: pp. 638-642.
5. Dhillon S., Halligan S., Goh V., Matravers P., Chambers A., Remedios D.: The therapeutic impact of abdominal ultrasound in patients with acute abdominal symptoms. Clin Radiol 2002; 57: pp. 268-271.
6. Stacul F., Cova M., Pravato M., Floriani I.: Comparison between the efficacy of dimeric and monomeric non-ionic contrast media (iodixanol vs iopromide) in urography in patients with macroscopic haematuria. Eur Radiol 2003; 13: pp. 810-814.
7. Chambers A., Halligan S., Goh V., Dhillon S., Hassan A.: Therapeutic impact of abdominopelvic computed tomography in patients with acute abdominal symptoms. Acta Radiol 2004; 45: pp. 248-253.
8. Esses D., Birnbaum A., Bijur P., Shah S., Gleyzer A., Gallagher E.J.: Ability of CT to alter decision making in elderly patients with acute abdominal pain. Am J Emerg Med 2004; 22: pp. 270-272.
9. Bearcroft P.W., Guy S., Bradley M., Robinson F.: MRI of the ankle: Effect on diagnostic confidence and patient management. AJR Am J Roentgenol 2006; 187: pp. 1327-1331.
10. Dixon A.K., Southern J.P., Teale A., Freer C.E., Hall L.D., Williams A., Sims C.: Magnetic resonance imaging of the head and spine: Effective for the clinician or the patient?. BMJ 1991; 302: pp. 79-82.
11. Omary R.A., Kaplan P.A., Dussault R.G., Hornsby P.P., Carter C.T., Kahler D.M., Hillman B.J.: The impact of ankle radiographs on the diagnosis and management of acute ankle injuries. Acad Radiol 1996; 3: pp. 758-765.
12. Neish A.S., Taylor G.A., Lund D.P., Atkinson C.C.: Effect of CT information on the diagnosis and management of acute abdominal injury in children. Radiology 1998; 206: pp. 327-331.
13. Carrico C.W., Fenton L.Z., Taylor G.A., DiFiore J.W., Soprano J.V.: Impact of sonography on the diagnosis and treatment of acute lower abdominal pain in children and young adults. AJR Am J Roentgenol 1999; 172: pp. 513-516.
14. Abramson S., Walders N., Applegate K.E., Gilkeson R.C., Robbin M.R.: Impact in the emergency department of unenhanced CT on diagnostic confidence and therapeutic efficacy in patients with suspected renal colic: A prospective survey. 2000 ARRS President’s Award. American Roentgen Ray Society. AJR Am J Roentgenol 2000; 175: pp. 1689-1695.
15. Tsushima Y., Aoki J., Endo K.: Contribution of the diagnostic test to the physician’s diagnostic thinking: New method to evaluate the effect. Acad Radiol 2003; 10: pp. 751-755.
16. Ng C.S., Palmer C.R.: Analysis of diagnostic confidence and diagnostic accuracy: A unified framework. Br J Radiol 2007; 80: pp. 152-160.
17. Ng C.S., Watson C.J., Palmer C.R., See T.C., Beharry N.A., Housden B.A., Bradley J.A., Dixon A.K.: Evaluation of early abdominopelvic computed tomography in patients with acute abdominal pain of unknown cause: Prospective randomised study. BMJ 2002; 325: pp. 1387.
18. Tsushima Y., Yamada S., Aoki J., Motojima T., Endo K.: Effect of contrast-enhanced computed tomography on diagnosis and management of acute abdomen in adults. Clin Radiol 2002; 57: pp. 507-513.
19. Sala E., Watson C.J., Beadsmoore C., Groot-Wassink T., Fanshawe T.R., Smith J.C., Bradley A., Palmer C.R., Shaw A., Dixon A.K.: A randomized, controlled trial of routine early abdominal computed tomography in patients presenting with non-specific acute abdominal pain. Clin Radiol 2007; 62: pp. 961-969.

Assessing Diagnostic Confidence

Rationale and Objectives

Materials and Methods

Results

Conclusion

Materials and methods

Basic Analytic Method

Analyses Incorporating Changes in Diagnoses

Retained diagnoses

“Omary” correction

Analyses Incorporating Diagnostic Changes and Diagnostic Accuracy

“Tsushima” method

“Score-based” method

Illustrative Case Study

Statistical Analyses

Results

Patients and Diagnoses in the Case Study

Numerical Comparison of Analytic Methods in the Case Study

Discussion

Analytic Methods That Do Not Take Diagnostic Accuracy Into Consideration

Analytic Methods That Take Diagnostic Accuracy Into Consideration

Fundamental Differences Between Tsushima and Score-Based Methods

Similarities and Differences Across All Methods

References

Further Reading

3D TOF MRA of Intracranial Aneurysms at 1.5 T and 3 T

Abstracts of Funded National Institutes of Health Grants

Bad news