Home CAD-associated Reader Error in CT Colonography
Post
Cancel

CAD-associated Reader Error in CT Colonography

Posted Jun 30, 2012 Updated Apr 10, 2023
By Vadim S. Koshkin BA , J. Louis Hinshaw MD , Kristen Wroblewski MS , Abraham H. Dachman MD
15 min read

Rationale and Objectives

Computed tomographic colonographic interpretation with computer-aided detection (CAD) may be superior to unaided viewing, although polyp characteristics may influence accuracy. Reader error due to polyp characteristics was evaluated in a multiple-case, multiple-reader trial of computed tomographic colonography with CAD.

Materials and Methods

Two experts retrospectively reviewed 52 positive cases (74 polyps) and categorized them as hard, moderate, or easy to detect. Each case was evaluated without and with CAD. Features that may influence a reader’s ability to detect a polyp or to accept or reject a CAD mark were tabulated. The association between polyp characteristics and detection rates in the trial was assessed. The difference in detection rates (CAD vs unassisted) was calculated, and regression analysis was performed.

Results

Of 64 polyps found by CAD, experts categorized 20 as hard, 28 as moderate, and 16 as easy to detect. Reader characterization errors predominated (47.3%) over other errors. Factors associated with lower detection rates included small size, flat morphology, and resemblance to a thickened fold. CAD was superior for polyps resembling lipomas compared to those that did not resemble lipomas (average increase in detection rate with CAD, 12.8% vs 5.5%; P < .05).

Conclusions

Polyp characteristic may impair computed tomographic colonographic interpretation augmented by CAD. Readers can avoid errors of measurement by evaluating diminutive polyp candidates with sample measurements. Caution should be taken when evaluating focally thick folds and when using visual impression to dismiss a polyp candidate as a lipoma when it is submerged in densely tagged fluid.

Computed tomographic (CT) colonography (or “virtual colonoscopy”) is an examination of the colon to detect polyps and masses that is endorsed by the American Cancer Society for colorectal cancer screening. Substantial training is required to learn how to interpret CT colonographic studies , and despite the reported high sensitivity for polyps ≥6 mm in size in some large screening trials , sensitivity has been lower in other trials . This variability has been attributed to many factors, but the largest factor seems to be observer errors due to missing polyps that could potentially have been recognized . However, certain errors, such as those due to immobile stool that is untagged by oral contrast, are unavoidable. Other sources of error in reading CT colonographic studies involve technique-related or patient-related factors (eg, collapsed bowel, polyps obscured by untagged residual fluid, poor scanning technique, respiratory motion). Small lesions and plaquelike flat lesions are also potential sources of error .

Computer-aided detection (CAD) has been developed and tested as a means to improve reader sensitivity. CAD systems detect raised areas suspicious for polyps and then use a set of classifiers to reduce false-positives (eg, analyzing the texture of a polyp candidate to determine if it is stool) . CAD has high sensitivity in stand-alone trials (ie, comparing optical colonoscopic truth to CAD software output without a human reader) . When CAD is used by a human reader, the reader may accept or reject a CAD hit, so the ultimate benefit of CAD depends on whether it helps or hinders the radiologist interpreting an exam . For example, polyps with irregular surface shapes have been shown to be more likely to be dismissed as not polyps even when detected by CAD . Consequently, CAD observer trials are important in proving the benefit of CAD-assisted reading .

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Materials and methods

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Study Design

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Table 1

Characteristics of Polyps Recorded in the MRMC Trial

Polyp Characteristic Unassisted Detection Rate (%) ∗ P CAD-assisted Detection Rate (%) ∗ P CAD hit <.01 <.001 No ( n = 10) 6.8 ± 9.9 7.4 ± 7.5 Yes ( n = 64) 47.7 ± 35.6 54.0 ± 33.5 Morphology <.001 <.001 Sessile ( n = 47) 44.5 ± 32.8 52.0 ± 31.5 Pedunculated ( n = 12) 75.9 ± 34.0 78.1 ± 30.3 Flat ( n = 5) 10.5 ± 12.3 15.8 ± 12.9 Adenoma .016 .020 No ( n = 23) 38.2 ± 32.3 42.6 ± 33.6 Yes ( n = 41) 53.0 ± 36.6 60.5 ± 32.1 Segment .073 .037 Rectum ( n = 10) 38.9 ± 40.3 41.6 ± 35.6 Sigmoid ( n = 12) 49.1 ± 33.4 57.0 ± 29.3 Descending ( n = 10) 64.2 ± 30.2 72.6 ± 29.5 Transverse ( n = 11) 44.0 ± 35.9 51.7 ± 27.7 Ascending ( n = 15) 52.6 ± 36.5 61.1 ± 33.5 Cecum ( n = 6) 26.3 ± 35.7 24.6 ± 38.8 Size (mm) <.001 <.001 6–9 ( n = 44) 35.9 ± 30.8 43.8 ± 29.3 ≥10 ( n = 20) 73.7 ± 31.9 76.6 ± 31.7

CAD, computer-aided detection; MRMC, multiple-reader, multiple-case.

Data are expressed as mean ± standard deviation. The mean is the average detection rate among all polyps with that characteristic. The P values reported are from the comparison of mean detection rates across categories of each polyp characteristic using a generalized estimating equation model.

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Evaluation of Potential Sources of Error

Get Radiology Tree app to read full this article<

Figure 1, Male patient, aged 61 years. Error of detection. Computer-aided detection (CAD) miss of 8-mm flat polyp in the cecum. Consensus expert opinion established difficulty rating as hard. Detected by none of 19 readers during unassisted read and none of 19 readers during CAD-assisted read. Highly magnified axial view from a prone scan showing white, densely tagged residual fluid outlining a small, sessile, lobular mucosal, soft tissue density irregularity consistent with a polyp ( arrows ). This polyp was not visible on a standard 120° endoluminal fly-through and was categorized in retrospective review as difficult to detect.

Figure 2, Male patient, aged 58 years. Error of characterization. Computer-aided detection (CAD) hit of a 10-mm sessile polyp in the descending colon. Consensus expert opinion established difficulty rating as moderate. Detected by 15 of 19 readers on unassisted read and 18 of 19 readers on CAD-assisted read. Highly magnified axial prone view shows dense residual fluid partially coating a soft tissue lesion consistent with a polyp ( arrows ). This lesion was categorized as having surface tagging in retrospective review, ordinarily a helpful sign in detecting polyps. Note that because of a partial volume effect, some of the high density appears to be within the polyp, rather than just on the surface. This can mislead the reader to judge the polyp candidate as stool.

Figure 3, (a) Axial image and (b) 3D endoluminal view. Female patient, aged 55 years. Error of measurement. Computer-aided detection (CAD) hit of a 7-mm sessile polyp in the sigmoid colon. Consensus expert opinion established difficulty rating as moderate. Detected by 14 of 19 readers on unassisted read and 13 of 19 readers on CAD-assisted read. The measurement of lesions near the 6-mm threshold is more subjective and subject to error. Choosing the axial image to create a measurement results in a long axis of 4.1 mm, yet an optimized three-dimensional view measurement results in a 6.1-mm long-axis measurement.

Table 2

Characteristics of Polyps on the Basis of the Retrospective Analysis

Polyp Characteristic ∗ Unassisted Detection Rate (%) † P CAD-assisted Detection Rate (%) † P Position relative to fold .738 .598 Between folds ( n = 21) 49.1 ± 36.1 59.9 ± 31.3 On a fold ( n = 33) 48.0 ± 33.3 52.3 ± 33.5 Far from fold ( n = 10) 43.7 ± 44.6 47.4 ± 39.2 Resembles thick fold .002 <.001 No ( n = 56) 51.2 ± 35.7 58.8 ± 32.2 In at least one view ( n = 8) 23.0 ± 24.2 20.4 ± 21.3 Polyp changed position .872 .976 No ( n = 41) 50.7 ± 35.0 56.2 ± 34.8 Yes ( n = 20) 48.2 ± 35.9 55.5 ± 30.0 Difficulty <.001 Easy ( n = 16) 86.2 ± 14.5 89.5 ± 12.0 Moderate ( n = 28) 50.0 ± 29.8 56.4 ± 27.2 Hard ( n = 20) 13.7 ± 17.3 22.4 ± 20.8 Lipoma like .257 .143 No ( n = 57) 48.4 ± 35.8 53.9 ± 33.6 Yes ( n = 7) 42.1 ± 36.2 54.9 ± 35.7 Tagging .020 .011 No ( n = 25) 36.2 ± 35.3 44.0 ± 31.1 Yes ( n = 39) 55.1 ± 34.2 60.5 ± 33.8 Visible on supine FOV .336 .721 No ( n = 10) 41.1 ± 37.3 51.6 ± 34.5 In at least one view ( n = 54) 48.9 ± 35.5 54.5 ± 33.6 Visible on prone FOV .598 .312 No ( n = 12) 34.2 ± 40.2 39.5 ± 39.8 In at least one view ( n = 52) 50.8 ± 34.1 57.4 ± 31.4 Submerged supine .391 .611 No ( n = 49) 49.4 ± 35.2 55.4 ± 33.1 Partially ( n = 7) 34.6 ± 38.1 41.4 ± 36.5 Completely ( n = 7) 54.1 ± 37.6 61.7 ± 35.1 Submerged prone .007 .125 No ( n = 47) 48.5 ± 35.6 55.0 ± 32.6 Partially ( n = 7) 75.9 ± 16.9 79.7 ± 21.4 Completely ( n = 8) 30.3 ± 31.5 36.8 ± 34.5 Main error <.001 <.001 Characterization ( n = 32) 62.3 ± 32.9 67.8 ± 32.0 Detection ( n = 20) 21.8 ± 25.1 30.0 ± 24.2 Measurement ( n = 9) 36.3 ± 25.4 43.3 ± 18.6

CAD, computer-aided detection; FOV, field of view.

Data are expressed as mean ± standard deviation. The mean is the average detection rate among all polyps with that characteristic. The P values reported are from the comparison of mean detection rates across categories of each polyp characteristic using a generalized estimating equation model.

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Statistical Analyses

Get Radiology Tree app to read full this article<

Figure 4, Flowchart illustrating the number of patients and polyps from the original multiple-reader, multiple-case trial and the final retrospective experts' consensus categorization of the overall degree of difficulty for detecting the polyps. CAD, computer-aided detection.

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Results

Associations between Polyp Characteristics and Detection Rates

Get Radiology Tree app to read full this article<

Table 3

Details on 10 CAD Misses

Case Polyp Segment Seen on Size Morphology CRADS Adenoma CAD-assisted Detection Rate ∗ (%) Unassisted Detection Rate ∗ (%) 4 1 Ascending Supine and prone 8 Sessile C3 Yes 10.5 5.3 4 2 Ascending Supine and prone 6 Sessile C3 No 21.1 31.6 21 1 Descending Supine and prone 6 Pedunculated C2 Yes 5.3 5.3 34 1 Rectum Supine and prone 9 Sessile C2 Yes 15.8 15.8 48 1 Cecum Supine and prone 8 Flat C2 Yes 0 0 49 1 Sigmoid Prone 6 Sessile C2 Yes 0 0 76 2 Sigmoid Supine and prone 7 Flat C2 No 10.5 5.3 76 1 Transverse Supine 6 Flat C2 No 0 0 139 3 Ascending Supine and prone 7 Sessile C3 Yes 10.5 5.3 144 1 Sigmoid Supine and prone 6 Sessile C2 No 0 0

CAD, computer-aided detection; CRADS, computed tomographic colonography reporting and data systems.

Polyp characteristics were as determined by the multiple-reader, multiple-case reference standard.

Get Radiology Tree app to read full this article<

Figure 5, Male patient, aged 64 years. Error of characterization. Computer-aided detection (CAD) hit of 6-mm flat polyp. Consensus expert opinion established difficulty rating as hard. Detected by one of 19 readers on unassisted read and one of 19 readers on CAD-assisted read. Missed polyp can be mistaken for a thick fold when evaluated on a 120° endoluminal fly-through. (a) Supine axial view shows a bulbous fold ( arrow ). (b) Supine endoluminal view as seen on the automated fly-though along the centerline ( arrows ). Similar pitfall ( arrows ) is seen on (c) prone axial and (d) prone endoluminal views. (e) Corresponding fly-through movie from the supine view, without CAD on and (f) with CAD on to identify the polyp candidate in the cecum (near the center of the viewing field during fly-though and on the final frame in the seven o'clock position), shown along the automated centerline at a 120° viewing angle. The polyp is easily confused for a normal slightly thick fold. Parts (e) and (f) are available online at www.academicradiology.org .

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Effect of CAD on Detection Rates

Get Radiology Tree app to read full this article<

Figure 6, Male patient, aged 68 years. Lipoma-like polyp. Error of characterization. Computer-aided detection (CAD) hit of 11-mm pedunculated polyp in the ascending colon. Consensus expert opinion established difficulty rating as easy. Detected by 16 of 19 readers on unassisted read and 17 of 19 readers on CAD-assisted read. Prone axial view shown in (a) standard computed tomographic colonographic view (window, 1500 Hounsfield units; level, 0 Hounsfield units) and (b) wide soft tissue (window, 1300 Hounsfield units; level, 400 Hounsfield units), and (c) with a region of interest showing the polyp density of 28 Hounsfield units. The polyp ( arrow ) is submerged in well-tagged residual fluid. Note that the polyp more closely resembles the density of subcutaneous fat (F) than soft tissue density of the abdominal wall muscle (M).

Get Radiology Tree app to read full this article<

Main Sources of Error Evaluated in the MRMC Trial

Get Radiology Tree app to read full this article<

Table 4

Main Source of Error Stratified by Difficulty, Size, Histology, and CAD Detection

Variable Main Error_P_ ∗ Characterization

( n = 35) Detection

( n = 27) Measurement

( n = 9) Unknown

( n = 3) Difficulty .010 Easy 12 (75%) 0 1 (6%) 3 (19%) Moderate 16 (55%) 8 (28%) 5 (17%) 0 Hard 7 (24%) 19 (66%) 3 (10%) 0 Size (mm) .004 6–9 18 (33%) 26 (48%) 9 (17%) 1 (2%) ≥10 17 (85%) 1 (5%) 0 2 (10%) Histology .567 Adenoma 21 (45%) 16 (34%) 7 (15%) 3 (6%) Not adenoma 14 (52%) 11 (41%) 2 (7%) 0 CAD detection .315 Hit 32 (50%) 20 (31%) 9 (14%) 3 (5%) Miss 3 (30%) 7 (70%) 0 0

CAD, computer-aided detection.

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Discussion

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Study Limitations

Get Radiology Tree app to read full this article<

Conclusions

Get Radiology Tree app to read full this article<

Supplementary data

Get Radiology Tree app to read full this article<

Video 1

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Video 2

Get Radiology Tree app to read full this article<

References

  • 1. Dachman A.H., Kelly K.B., Zintsmaster M.P., et. al.: Formative evaluation of standardized training for CT colonographic image interpretation by novice readers. Radiology 2008; 249: pp. 167-177.

  • 2. Fletcher J.G., Chen M.H., Herman B.A., et. al.: Can radiologist training and testing ensure high performance in CT colonography? Lessons from the National CT Colonography Trial. AJR Am J Roentgenol 2010; 195: pp. 117-125.

  • 3. Pickhardt P.J., Choi J.R., Hwang I., et. al.: Computed tomographic virtual colonoscopy to screen for colorectal neoplasia in asymptomatic adults. N Engl J Med 2003; 349: pp. 2191-2200.

  • 4. Johnson C.D., Harmsen W.S., Wilson L.A., et. al.: Prospective blinded evaluation of computed tomographic colonography for screen detection of colorectal polyps. Gastroenterology 2003; 125: pp. 311-319.

  • 5. Johnson C.D., Chen M.H., Toledano A.Y., et. al.: Accuracy of CT colonography for detection of large adenomas and cancer N. Engl J Med 2008; 359: pp. 1207-1217.

  • 6. Regge D., Laudi C., Galatola G., et. al.: Diagnostic accuracy of computed tomographic colonography for the detection of advanced neoplasia in individuals at increased risk of colorectal cancer. JAMA 2009; 301: pp. 2453-2461.

  • 7. Knudsen A.B., Lansdorp-Vogelaar I., Rutter C.M., et. al.: Cost-effectiveness of computed tomographic colonography screening for colorectal cancer in the Medicare population. J Natl Cancer Inst 2010; 102: pp. 1238-1252.

  • 8. Cotton P.B., Durkalski V.L., Benoit P.C., et. al.: Computed tomographic colonography (virtual colonoscopy): a multicenter comparison with standard colonoscopy for detection of colorectal neoplasia. JAMA 2004; 291: pp. 1713-1719.

  • 9. Rockey D.C., Paulson E., Davis W., et. al.: Analysis of air contrast barium enema, computed tomographic colonography, and colonoscopy: prospective comparison. Lancet 2005; 365: pp. 305-311.

  • 10. Doshi T., Rusinak D., Halvorsen R.A., et. al.: CT colonography: false-negative interpretations. Radiology 2007; 244: pp. 165-173.

  • 11. Park S.H., Kim S.Y., Lee S.S., et. al.: Sensitivity of CT colonography for nonpolyploid colorectal lesions interpreted by human readers and with computer-aided detection. AJR Am J Roentgenol 2009; 193: pp. 1-9.

  • 12. Krupinski E.A.: What can the radiologist teach CAD: lessons from CT colonoscopy. Acad Radiol 2009; 16: pp. 1-3.

  • 13. Bogoni L., Cathier P., Dundar M., et. al.: Computer-aided detection (CAD) for CT colonography: a tool to address a growing need. Br J Radiol 2005; 78: pp. S57-S62.

  • 14. Summers R.M., Yao J., Pickhardt P.J., et. al.: Computed tomographic virtual colonoscopy computer-aided polyp detection in a screening population. Gastroenterology 2005; 129: pp. 1832-1844.

  • 15. Yoshida H., Nappi J., MacEneaney P., et. al.: Computer-aided diagnosis scheme for detection of polyps at CT colonography. Radiographics 2002; 22: pp. 963-979.

  • 16. Lawrence E.M., Pickhardt P.J., Kim D.H., et. al.: Computer-aided detection (CAD) of colorectal polyps: standalone performance in a large asymptomatic screening population. Radiology 2010; 256: pp. 791-798.

  • 17. Taylor S.A., Robinson C., Boone D., et. al.: Polyp characteristics correctly annotated by computer-aided detection software but ignored by reporting radiologists during CT colonography. Radiology 2009; 243: pp. 715-723.

  • 18. Taylor S.A., Greenhalgh R., Ilangovan R., et. al.: CT colonography and computer-aided detection: effect of false-positive results on reader specificity and reading efficiency in a low-prevalence screening population. Radiology 2008; 247: pp. 133-140.

  • 19. Taylor S.A., Charman S.C., Lefere P., et. al.: CT colonography: investigation of the optimum reader paradigm by using computer-aided detection software. Radiology 2008; 246: pp. 462-471.

  • 20. Taylor S.A., Brittenden J., Lenton J.: Influence of computer-aided detection false-positives on reader performance and diagnostic confidence for CT colonography. AJR Am J Roentgenol 2009; 192: pp. 1682-1689.

  • 21. Petrick N., Haider M., Summers R.M., et. al.: CT colonography with computer-aided detection as a second reader: observer performance study. Radiology 2007; 246: pp. 148-156.

  • 22. Dachman A.H., Obuchowski N.A., Hoffmeister J.W., et. al.: Impact of computer aided detection for CT colonography in a multireader, multicase trial. Radiology 2010; 256: pp. 827-835.

  • 23. Zeger S.L., Liang K.Y.: Longitudinal data analysis for discrete and continuous outcomes. Biometrics 1986; 42: pp. 121-130.

  • 24. Zhang H., Guo W., Liu G., et. al.: Colonic polyps: application value of computer-aided detection in computed tomographic colonography. Chin Med J 2011; 124: pp. 380-384.

  • 25. Fidler J.L., Fletcher J.G., Johnson C.D.: Understanding interpretive errors in radiologists learning computed tomography colonography. Acad Radiol 2004; 11: pp. 750-756.

  • 26. Summers R.M., Frentz S.M., Liu J., et. al.: Conspicuity of colorectal polyps at CT colonography. Acad Radiol 2009; 16: pp. 4-14.

  • 27. Nappi J.J., Nagata K.: Sources of false positives in computer-assisted CT colonography. Abdom Imaging 2011; 36: pp. 153-164.

  • 28. Summers R.M., Liu J., Rehani B., et. al.: CT colonography computer-aided polyp detection: effect on radiologist observers of polyp identification by CAD on both the supine and prone scans. Acad Radiol 2010; 17: pp. 948-959.

  • 29. Gluecker T.M., Fletcher J.G., Welch T.J., et. al.: Assessment of vasculature using combined MRI and MR angiography. AJR Am J Roentgenol 2004; 182: pp. 881-889.

  • 30. Robinson C., Halligan S., Iinuma G., et. al.: CT colonography: computer-assisted detection of colorectal cancer. Br J Radiol 2011; 84: pp. 435-440.

  • 31. Pickhardt P.J., Kim D.H., Taylor A.J., et. al.: Extracolonic tumors of the gastrointestinal tract detected incidentally at screening CT colonography. Dis Colon Rectum 2006; 50: pp. 56-63.

Academic Radiology, Volume 19, Issue 7
Journals General Radiology
This post is licensed under CC BY 4.0 by the author.
Recently Updated
Contents