Home Reliability, Validity, and Reader Acceptance of LI-RADS—An In-depth Analysis
Post
Cancel

Reliability, Validity, and Reader Acceptance of LI-RADS—An In-depth Analysis

Posted Aug 31, 2016 Updated Apr 10, 2023
By Borna K. Barth MD , Olivio F. Donati MD , Michael A. Fischer MD , Erika J. Ulbrich MD , Christoph A. Karlo MD , Anton Becker MD , Burkhard Seifert PhD , Caecilia S. Reiner MD
12 min read

Rationale and Objectives

This study aimed to analyze interreader agreement and diagnostic accuracy of Liver Imaging Reporting and Data System (LI-RADS) in comparison to a nonstandardized 5-point scale and to assess reader acceptance of LI-RADS for clinical routine.

Materials and Methods

Eighty-four consecutive patients at risk for hepatocellular carcinoma who underwent liver magnetic resonance imaging were included in this Health Insurance Portability and Accountability Act-compliant retrospective study. Four readers rated the likelihood of hepatocellular carcinoma for 104 liver observations using LI-RADS criteria and a 5-point Likert scale (LIKERT) based on subjective impression in two separate reading sessions. Interreader agreement was assessed using kappa statistics (κ). Diagnostic accuracy was assessed with receiver operating characteristic analysis. Reader acceptance was evaluated with a questionnaire. A sub-analysis of LI-RADS’s major features (arterial phase hyper-enhancement, washout, capsule appearance, and threshold growth) and scores for lesions </>1.5 cm was performed.

Results

LI-RADS showed similar overall interreader agreement compared to LIKERT (κ, 0.44 [95%CI: 0.37, 0.52] and 0.35 [95%CI: 0.27, 0.43]) with a tendency toward higher interreader agreement for LI-RADS. Interreader agreement (κ) was 0.51 (95%CI: 0.38, 0.65) for arterial phase hyper-enhancement, 0.52 (95%CI: 0.39, 0.65) for washout, 0.37 (95%CI: 0.23, 0.52) for capsule appearance, and 0.50 (95%CI: 0.38, 0.61) for threshold growth. Overall interreader agreement for LI-RADS categories was similar between observations <1.5 cm and observations >1.5 cm. Overall diagnostic accuracy for LIKERT and LI-RADS was comparable (area under the receiver operating characteristic curve, 0.86 and 0.87). Readers fully agreed with the statement “A short version of LI-RADS would facilitate the use in clinical routine” (median, 5.0; interquartile range, 2.25).

Conclusions

LI-RADS showed similar interreader agreement and diagnostic accuracy compared to nonstandardized reporting. However, further reduction of complexity and refinement of imaging features may be needed.

Introduction

Hepatocellular carcinoma (HCC) is a major health issue, being the sixth most common cancer worldwide . Patients with liver cirrhosis, especially with underlying hepatitis B or C infection or chronic alcohol abuse, have an increased risk of developing HCC . The current clinical practice guidelines of the American Association for the Study of Liver Disease and the European Association for the Study of the Liver on the management of HCC recommend routine imaging surveillance for these patients at increased risk for HCC .

Given the widespread use of cross-sectional imaging in HCC surveillance, reliable liver lesion reporting is of utmost importance to facilitate the comparison and communication of findings within and between institutions . Most often, image reporting is descriptive using free text, and is varying by experience level and personal preference, factors which have been identified as potential confounders for consistent and reliable reporting . One approach toward standardization of reporting has been the use of a Likert scale (LIKERT), where the degree of suspicion of a diagnosis is expressed on a 5-point scale without fixed criteria, based only on the radiologist’s subjective impression. However, further standardization with the use of standardized terminology and fixed criteria for lesion scoring has been advocated. One of the first efforts to address this issue was made by Petruzzi et al. , who developed a reporting system for categorization of liver observations in patients at risk for HCC. The system revealed solid reliability and validity data. Following these first attempts to standardize reporting in liver imaging, an expert panel of the American College of Radiology (ACR) introduced the “Liver Imaging Reporting and Data System” (LI-RADS), a system of standardized terminology and criteria for interpretation and reporting of liver observations in cirrhotic livers . The system consists of five major categories based on a sequential decision tree, each of them featuring specific criteria, allowing stratifying a liver observation according to the level of concern for HCC. Although adopted by several institutions worldwide , there is still limited evidence if the elaborated LI-RADS improves interreader agreement. Uncertainty arises since Davenport et al. and Zhang et al. presented data on repeatability of LI-RADS, showing fair to moderate agreement between readers . The use of a 5-point LIKERT without fixed criteria may be more straightforward and easier to use, although potentially more prone to interreader variability, which has not been investigated yet. So far, Zhang et al. recently demonstrated differences in diagnostic accuracy between LI-RADS and a nonstandardized reporting approach for detection of HCC (accuracy, 78.6% and 87.2%, P < 0.001), but did not evaluate potential differences in interreader agreement. Moreover, they showed substantial discordance between computed tomography (CT) and magnetic resonance (MR) for stratification of hepatic nodules using LI-RADS .

Get Radiology Tree app to read full this article<

Materials and Methods

Get Radiology Tree app to read full this article<

Patient Selection

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

MRI Protocol

Get Radiology Tree app to read full this article<

Preparations for Image Analysis

Get Radiology Tree app to read full this article<

Image Analysis

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Reader Acceptance

Get Radiology Tree app to read full this article<

Observation Characteristics

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Statistical Analysis

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Results

Distribution of Scores

Get Radiology Tree app to read full this article<

Table 1

Frequency Distributions of LIKERT, LI-RADS, and LI-RADS mc for 104 Observations Stratified by System and Reader

System Reader 1 Reader 2 Reader 3 Reader 4 Overall LIKERT 1 28(26.9) 22(21.2) 18(17.3) 16(15.4) 84(20.2) 2 18(17.3) 9(8.7) 21(20.2) 14(13.5) 62(14.9) 3 8(7.7) 15(14.4) 12(11.5) 10(9.6) 45(10.8) 4 13(12.5) 23(22.1) 31(29.8) 29(27.9) 96(23.1) 5 37(35.6) 35(33.7) 22(21.2) 35(33.7) 129(31.0) LI-RADS 1 20(19.2) 21(20.2) 15(14.4) 9(8.7) 65(15.6) 2 15(14.4) 12(11.5) 15(14.4) 7(6.7) 49(11.8) 3 17(16.3) 15(14.4) 9(8.7) 24(23.1) 65(15.6) 4A 14(13.5) 20(19.2) 24(23.1) 21(20.2) 79(19.0) 4B 5(4.8) 4(3.8) 11(10.6) 7(6.7) 27(6.5) 5A 16(15.4) 14(13.5) 20(19.2) 15(14.4) 65(15.6) 5B 17(16.3) 18(17.3) 10(9.6) 21(20.2) 66(15.9) LI-RADS mc 4 19(18.3) 24(23.1) 35(33.7) 28(26.9) 106(25.5) 5 33(31.7) 32(30.8) 30(28.8) 36(34.6) 131(31.5)

LI-RADS, Liver Imaging Reporting and Data System; LI-RADS mc , Liver Imaging Reporting and Data System major category.

A total of 104 observations are in the dataset. The column “overall” includes the number of eligible observations multiplied by the number of readers. Numbers in parentheses are percentages of the number of times the readers assigned the LIKERT or the LI-RADS score divided by the total number of observations.

Get Radiology Tree app to read full this article<

Interreader Agreement

Get Radiology Tree app to read full this article<

Table 2

Interreader Agreement (κ) for LIKERT and LI-RADS mc Stratified by Reader

Reader System Reader 2 Reader 3 Reader 4 All Readers All observations ( n = 104) Reader 1 LIKERT 0.41 (0.30, 0.53) 0.37 (0.26, 0.49) 0.34 (0.23, 0.45) LI-RADS mc 0.58 (0.47, 0.70) 0.58 (0.47, 0.70) 0.33 (0.21, 0.45) Reader 2 LIKERT — 0.43 (0.31, 0.55) 0.28 (0.17, 0.40) LI-RADS mc — 0.44 (0.32, 0.56) 0.44 (0.32, 0.56) Reader 3 LIKERT — 0.25 (0.13, 0.36) LI-RADS mc — 0.39 (0.27, 0.51) All readers LIKERT — 0.35 (0.27, 0.43) LI-RADS mc — 0.44 (0.37, 0.52)

LI-RADS mc , Liver Imaging Reporting and Data System major category.

A total of 104 observations are in the dataset. Data are κ-values (Cohen κ for inter-rater; Fleiss κ for multi-rater). Data in parentheses are 95% confidence intervals.

Figure 1, 35-year-old woman 35-year-old women with liver cirrhosis Child-Pugh Score C undergoing Gadoterate meglumine (Gd-DOTA)-enhanced liver magnetic resonance imaging (MRI) for preoperative evaluation before liver transplantation. The arrows highlight an 8-mm benign observation (ie, cirrhotic nodule) in segment VI, verified with histopathology after explantation 2 months after the MRI. The attributed LIKERT scores were LK2 (three readers) and LK3 (one reader), and Liver Imaging Reporting and Data System (LI-RADS) categories were LR2 (one reader) and LR3 (three readers), demonstrating equal interreader agreement for both approaches for this case. Images show a non-enhanced (a) and dynamic contrast-enhanced T1-weighted fat-saturated three-dimensional gradient echo sequence in arterial (b) , 2-minute delayed (d) , and 4-minute delayed (d) phase, a T2-weighted (e) , in-phase (f) , and opposed-phase (g) image, respectively.

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Figure 2, 67-year-old woman with liver cirrhosis Child-Pugh Score A undergoing Gadoterate meglumine (Gd-DOTA)-enhanced liver magnetic resonance imaging (MRI) for preoperative evaluation before liver transplantation. The arrow highlights a 1.5-cm hepatocellular carcinoma (HCC) observation confirmed by histopathology in liver segment VI. The assigned LIKERT scores were LK2, LK3 (one reader each), and LK4 (two readers), and the Liver Imaging Reporting and Data System (LI-RADS) categories were LR4 (three readers) and LR5 (one reader), demonstrating superior interreader agreement for LI-RADS. For this particular observation, three readers agreed on three major features (positive arterial hyper-enhancement, no washout, and no capsule), and only one reader interpreted a positive washout and capsule. There was no prior cross-sectional imaging available and therefore “threshold growth” could not be assessed. The figure emphasizes the potential variability in assigning major LI-RADS features. The figure shows a non-enhanced (a) and dynamic contrast-enhanced T1-weighted fat-saturated three-dimensional gradient echo sequence in arterial (b) , 2-minute delayed (d) , and 4-minute delayed (d) phase and a T2-weighted (e) , in-phase (f) , and opposed-phase (g) image.

Get Radiology Tree app to read full this article<

Diagnostic Accuracy

Get Radiology Tree app to read full this article<

Figure 3, Receiver operating characteristic (ROC) curves with area under the ROC curve (AUC) as an indicator of diagnostic performance. The curve “Overall” shows pooled data of all readers.

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

In-depth Analysis of Interreader Agreement

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

LI-RADS Reader Acceptance

Get Radiology Tree app to read full this article<

Table 3

Reader Acceptance of LI-RADS

Questions Score I can assess a liver observation more reliably with LI-RADS than with LIKERT scale 3.5 [IQR 1.00] I would use LI-RADS in clinical routine reporting 4.0 [IQR 0.75] Ancillary features were useful for assigning definitive LI-RADS categories 2.0 [IQR 0.75] The use of ancillary features is too time-consuming 3.0 [IQR 1.75] The use of LI-RADS requires a high level of radiological expertise 3.5 [IQR 1.75] LI-RADS would be helpful in particular for less-experienced radiologists 3.0 [IQR 2.25] The use of standardized reporting of liver observations should be promoted 5.0 [IQR 1.50] A short version of LI-RADS would facilitate the use in clinical routine 5.0 [IQR 2.25]

IQR, interquartile range; LI-RADS, Liver Imaging Reporting and Data System.

For assessment of reader acceptance of LI-RADS, all readers completed a questionnaire after the second readout. Answers were given on an ordinal 5-point scale (1, I strongly disagree; 2, I disagree; 3, I neither agree nor disagree; 4; I agree; 5, I strongly agree). The scores are presented as median and IQR.

Get Radiology Tree app to read full this article<

Discussion

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

Get Radiology Tree app to read full this article<

References

  • 1. Bray F., Ren J.S., Masuyer E., et. al.: Global estimates of cancer prevalence for 27 sites in the adult population in 2008. Int J Cancer 2013; 132: pp. 1133-1145.

  • 2. Llovet J.M., Burroughs A., Bruix J.: Hepatocellular carcinoma. Lancet 2003; 362: pp. 1907-1917.

  • 3. European Association for the Study of the Liver , European Organisation for Research and Treatment of Cancer : EASL-EORTC clinical practice guidelines: management of hepatocellular carcinoma. J Hepatol 2012; 56: pp. 908-943.

  • 4. Tan C.H., Low S.C., Thng C.H.: APASL and AASLD consensus guidelines on imaging diagnosis of hepatocellular carcinoma: a review. Int J Hepatol 2011; 2011: pp. 519783.

  • 5. Krinsky G.: Terminology of hepatocellular nodules in cirrhosis: plea for consistency. Radiology 2002; 224: pp. 638.

  • 6. Lee J.M., Trevisani F., Vilgrain V., et. al.: Imaging diagnosis and staging of hepatocellular carcinoma. Liver Transpl 2011; 17: pp. S34-S43.

  • 7. Pomfret E.A., Washburn K., Wald C., et. al.: Report of a national conference on liver allocation in patients with hepatocellular carcinoma in the United States. Liver Transpl 2010; 16: pp. 262-278.

  • 8. Wald C., Russo M.W., Heimbach J.K., et. al.: New OPTN/UNOS policy for liver transplant allocation: standardization of liver imaging, diagnosis, classification, and reporting of hepatocellular carcinoma. Radiology 2013; 266: pp. 376-382.

  • 9. Petruzzi N., Mitchell D., Guglielmo F., et. al.: Hepatocellular carcinoma likelihood on MRI exams: evaluation of a standardized categorization system. Acad Radiol 2013; 20: pp. 694-698.

  • 10. American College of Radiology : Liver imaging reporting and data system, version 2013.1. Available at: http://www.acr.org/Quality-Safety/Resources/LIRADS Accessed April 2014

  • 11. Darnell A., Forner A., Rimola J., et. al.: Liver imaging reporting and data system with MR imaging: evaluation in nodules 20 mm or smaller detected in cirrhosis at screening US. Radiology 2015; 275: pp. 698-707.

  • 12. Mitchell D.G., Bruix J., Sherman M., et. al.: LI-RADS (Liver Imaging Reporting and Data System): summary, discussion, and consensus of the LI-RADS Management Working Group and future directions. Hepatology 2015; 61: pp. 1056-1065.

  • 13. Davenport M.S., Khalatbari S., Liu P.S., et. al.: Repeatability of diagnostic features and scoring systems for hepatocellular carcinoma by using MR imaging. Radiology 2014; 272: pp. 132-142.

  • 14. Zhang Y.D., Zhu F.P., Xu X., et. al.: Classifying CT/MR findings in patients with suspicion of hepatocellular carcinoma: comparison of liver imaging reporting and data system and criteria-free Likert scale reporting models. J Magn Reson Imaging 2016; 43: pp. 373-383.

  • 15. Zhang Y.D., Zhu F.P., Xu X., et. al.: Liver imaging reporting and data system: substantial discordance between CT and MR for imaging classification of hepatic nodules. Acad Radiol 2016; 23: pp. 344-352.

  • 16. Viera A.J., Garrett J.M.: Understanding interobserver agreement: the kappa statistic. Fam Med 2005; 37: pp. 360-363.

  • 17. Bashir M.R., Huang R., Mayes N., et. al.: Concordance of hypervascular liver nodule characterization between the organ procurement and transplant network and liver imaging reporting and data system classifications. J Magn Reson Imaging 2014; 42: pp. 305-314.

  • 18. D’Orsi C., Sickles E., Mendelson E., et. al.: ACR BI-RADS Atlas, Breast Imaging Reporting and Data System.2013.American College of RadiologyReston, VA

  • 19. Berg W.A., Campassi C., Langenberg P., et. al.: Breast Imaging Reporting and Data System: inter- and intraobserver variability in feature analysis and final assessment. AJR Am J Roentgenol 2000; 174: pp. 1769-1777.

  • 20. Kerlikowske K., Grady D., Barclay J., et. al.: Variability and accuracy in mammographic interpretation using the American College of Radiology Breast Imaging Reporting and Data System. J Natl Cancer Inst 1998; 90: pp. 1801-1809.

  • 21. Berg W.A., D’Orsi C.J., Jackson V.P., et. al.: Does training in the Breast Imaging Reporting and Data System (BI-RADS) improve biopsy recommendations or feature analysis agreement with experienced breast imagers at mammography?. Radiology 2002; 224: pp. 871-880.

  • 22. American College of Radiology : Liver imaging reporting and data system, version 2014. Available at: http://www.acr.org/Quality-Safety/Resources/LIRADS Accessed September 2014

  • 23. Clark T.J., McNeeley M.F., Maki J.H.: Design and implementation of handheld and desktop software for the structured reporting of hepatic masses using the LI-RADS schema. Acad Radiol 2014; 21: pp. 491-506.

Academic Radiology, Volume 23, Issue 9
Journals General Radiology
This post is licensed under CC BY 4.0 by the author.
Recently Updated
Contents