Volume-based Response Evaluation with Consensual Lesion Selection

Rationale and Objectives

Lesion volume is considered as a promising alternative to Response Evaluation Criteria in Solid Tumors (RECIST) to make tumor measurements more accurate and consistent, which would enable an earlier detection of temporal changes. In this article, we report the results of a pilot study aiming at evaluating the effects of a consensual lesion selection on volume-based response (VBR) assessments.

Materials and Methods

Eleven patients with lung computed tomography scans acquired at three time points were selected from Reference Image Database to Evaluate Response to therapy in lung cancer (RIDER) and proprietary databases. Images were analyzed according to RECIST 1.1 and VBR criteria by three readers working in different geographic locations. Cloud solutions were used to connect readers and carry out a consensus process on the selection of lesions used for computing response. Because there are not currently accepted thresholds for computing VBR, we have applied a set of thresholds based on measurement variability (−35% and +55%). The benefit of this consensus was measured in terms of multiobserver agreement by using Fleiss kappa (κ fleiss ) and corresponding standard errors (SE).

Results

VBR after consensual selection of target lesions allowed to obtain κ fleiss = 0.85 (SE = 0.091), which increases up to 0.95 (SE = 0.092), if an extra consensus on new lesions is added. As a reference, the agreement when applying RECIST without consensus was κ fleiss = 0.72 (SE = 0.088). These differences were found to be statistically significant according to a z-test.

Conclusions

An agreement on the selection of lesions allows reducing the inter-reader variability when computing VBR. Cloud solutions showed to be an interesting and feasible strategy for standardizing response evaluations, reducing variability, and increasing consistency of results in multicenter clinical trials.

According to recent statistics of the World Health Organization, cancer is the leading cause of death worldwide accounting for 8.2 million deaths in 2012 . Lung cancer accounted for 1.6 million deaths (19.4%), which makes it the first cause of cancer death even ahead of liver (9.1%) and stomach (8.8%) cancers. Computed tomography (CT) is currently the standard imaging modality for assessing the response to treatment in patients with solid tumors. In general, the quantification of the response is performed by using Response Evaluation Criteria in Solid Tumors (RECIST) . In summary, this standard establishes the way of measuring lesions and provides a set of thresholds to classify the response into partial response, stable disease, and progressive disease. In the context of this article, the term RECIST refers to its revised version (1.1) .

Even when RECIST criteria have been broadly adopted in clinical trials, they present some drawbacks that are a source of measurement variability. This is quite inconvenient because the consistency in the production of trial results is necessary for comparison purposes . For example, the lesion size is measured as its longest axial diameter, which is not a robust measure in case of complex lesions and creates problems of accuracy and precision . To cope with this drawback, the use of volume is currently being considered as a promising direction to make tumor measurements more accurate and consistent, which would enable an earlier detection of temporal changes . The benefit of using volume as biomarker has already been reported in the literature ; however, the use of volume also presents some drawbacks like the long segmentation time in case of big lesions (in particular for thin-slice acquisitions) and the lack of accepted thresholds for computing response. Regarding this last point, it is worth mentioning the intensive efforts performed by the Quantitative Imaging Biomarkers Alliance to better understand volumetric biomarkers and their sources of variability. In the context of this article, the term volume-based response (VBR) refers to the response estimated exactly as established by RECIST, except for the use of volume of lesions and volume-specific thresholds instead of diameters. The use of VBR was preferred to names like 3D-RECIST to avoid confusion with the application of RECIST with 3D-extended thresholds and also because currently there is no formal definition of the volumetric version of RECIST.

Get Radiology Tree app to read full this article<

Materials and methods

Data Set

Get Radiology Tree app to read full this article<

Cloud-based Setup System

Get Radiology Tree app to read full this article<

Volume Measurement

Get Radiology Tree app to read full this article<

Response Agreement

Target Lesions

Get Radiology Tree app to read full this article<

New Lesions

Get Radiology Tree app to read full this article<

Volume Thresholds

Get Radiology Tree app to read full this article<

Data Analysis

Get Radiology Tree app to read full this article<

Results

Response Agreement

Get Radiology Tree app to read full this article<

Table 1

Fleiss Kappa and Standard Errors (Between Parentheses) According to Different Types of Consensus and Criteria

Consensus TL Only Response RECIST VBR TL NL — — X 0.72 (0.808)/SuA 0.76 (0.090)/SuA — — — 0.72 (0.090)/SuA 0.76 (0.090)/SuA X — X 0.72 (0.088)/SuA 0.85 (0.091)/APA X — — 0.72 (0.088)/SuA 0.85 (0.092)/APA X X — 0.81 (0.089)/APA0.95 (0.093)/APA

APA, almost perfect; NL, new lesion criteria; RECIST, Response Evaluation Criteria in Solid Tumors; SuA, substantial; TL, target lesion; VBR, volume-based response.

Consensus on TL, NL, and TL only response has been marked with “X” when they are applied and “—” when they are not. The interpretation of values according to Landis and Koch has been added as follows: slight/fair/moderate/substantial/almost perfect agreement. The highest value is shown in bold.

Get Radiology Tree app to read full this article<

Table 2

Reproducibility Coefficient for ln(SLV), ln(ΔSLV BL ), and ln(ΔSLV NADIR ) for Consensual and Nonconsensual Selection of target lesions

Consensus ln(SLV) ln(ΔSLV BL ) ln(ΔSLV NADIR ) No 2.081 1.167 0.915 Yes 0.593 0.678 0.677

SLV, sum of lesion volumes; ΔSLV BL , change in SLV with respect to baseline (BL); ΔSLV NADIR , change in SLV with respect to nadir.

Get Radiology Tree app to read full this article<

Comparison Between VBR and RECIST

Get Radiology Tree app to read full this article<

Table 3

Contingency Table of Responses Provided by RECIST and VBR*

VBR* RECIST PR SD PD Total PR 14 15 0 29 SD 0 8 2 10 PD 2 2 23 27 Total 16 25 25 66

PD, progressive disease; PR, partial response; RECIST, Response Evaluation Criteria in Solid Tumors; SD, stable disease; VBR, volume-based response.

VBR ∗ was computed after consensus on target lesion and new lesion, whereas RECIST was calculated in the classical way (without consensus).

Get Radiology Tree app to read full this article<

Number of Lesions

Get Radiology Tree app to read full this article<

Table 4

Response Agreement (as a Percent of Matches) According to the Number of Lesions Used for Computing Response

1L 2L 4L DIFF(1-2) DIFF(2-4) 93% 75%100% 0% 83%

1 L/2 L/4L, 1/2/4 lesion(s) chosen by both raters; DIFF(1-2), different number of lesions chosen varying between 1 and 2; DIFF(2-4), different number of lesions chosen varying between 2 and 4.

The best value is shown in bold.

Get Radiology Tree app to read full this article<

RECIST Compliance

Get Radiology Tree app to read full this article<

Discussion

Get Radiology Tree app to read full this article<

Conclusions

Get Radiology Tree app to read full this article<

References

1. 2012.International Agency for Research on Cancer Available from: http://globocan.iarc.fr
2. Therasse P., Arbuck S.G., Eisenhauer E.A., et. al.: New guidelines to evaluate the response to treatment in solid tumors. J Natl Cancer Inst 2000 Feb; 92: pp. 205-216. Available from: http://jnci.oxfordjournals.org/content/92/3/205.abstract
3. Eisenhauera E.A., Therasse P., Bogaerts J., et. al.: New response evaluation criteria in solid tumours: revised RECIST guideline (version 1.1). Eur J Cancer 2009 Jan; 45: pp. 228-247. Available from: http://www.ncbi.nlm.nih.gov/pubmed/19097774
4. Erasmus J.J., Gladish G.W., Broemeling L., et. al.: Interobserver and intraobserver variability in measurement of non–small-cell carcinoma lung lesions: implications for assessment of tumor response. J Clin Oncol 2003 Jan; 21: pp. 2574-2582. Available from http://jco.ascopubs.org/content/21/13/2574
5. Suzuki C, Jacobsson H, Hatschek T, Torkzad MR, Bodén K, Eriksson-Alm Y, et al. Radiologic measurements of tumor response to treatment: practical approaches and limitations. Radiographics [Internet]. [cited 2013 Nov 27];28(2):329–344. Available from: http://www.ncbi.nlm.nih.gov/pubmed/18349443
6. Nishino M., Jagannathan J.P., Ramaiya N.H., et. al.: Revised RECIST guideline version 1.1: what oncologists want to know and what radiologists need to know. American Roentgen Ray Society AJR Am J Roentgenol 2010 Aug 23; 195: pp. 281-289. Available from: http://www.ajronline.org/doi/full/10.2214/AJR.09.4110
7. Buckler A.J., Mozley P.D., Schwartz L., et. al.: Volumetric CT in lung cancer: an example for the qualification of imaging as a biomarker. Acad Radiol 2010 Jan; 17: pp. 107-115. Available from: http://www.ncbi.nlm.nih.gov/pubmed/19969254
8. Buckler A.J., Mulshine J.L., Gottlieb R., et. al.: The use of volumetric CT as an imaging biomarker in lung cancer. Acad Radiol 2010 Jan 1; 17: pp. 100-106. Available from: http://www.academicradiology.org/article/S1076-6332(09)00574-1/abstract
9. Seyal A.R., Parekh K., Velichko Y.S., et. al.: Tumor growth kinetics versus recist to assess response to locoregional therapy in breast cancer liver metastases. Acad Radiol 2014 May 12; Available from: http://www.ncbi.nlm.nih.gov/pubmed/24833565
10. Mozley P.D., Bendtsen C., Zhao B., et. al.: Measurement of tumor volumes improves RECIST-based response assessments in advanced lung cancer. Transl Oncol 2012 Feb; 5: pp. 19-25. Available from: http://www.ncbi.nlm.nih.gov/pubmed/22348172
11. Quantitative Imaging Biomarkers Alliance [Internet]. Available from: https://www.rsna.org/QIBA.aspx
12. Khul C.K., Barabasch A., Dirrichs T., et. al.: Target lesion selection as a source of variability of response classification by RECIST 1.1. 2013 ASCO Annual Meeting J Clin Oncol 2013; 31: abstr 11077
13. Zhao B., Lee S.M., Lee H.-J., et. al.: Variability in assessing treatment response: metastatic colorectal cancer as a paradigm. Clin Cancer Res 2014 Jul 1; 20: pp. 3560-3568. Available from: http://www.ncbi.nlm.nih.gov/pubmed/24780294
14. Padhani A.R., Ollivier L.: The RECIST criteria: implications for diagnostic radiologists. Br J Radiol 2001 Nov 1; 74: pp. 983-986. Available from: http://bjr.birjournals.org/content/74/887/983.full
15. Schuetze S.M., Baker L.H., Benjamin R.S., et. al.: Selection of response criteria for clinical trials of sarcoma treatment. Oncologist 2008 Jan 1; 13: pp. 32-40. Available from: http://theoncologist.alphamedpress.org/content/13/suppl_2/32.full
16. Dodd L.E., Korn E.L., Freidlin B., et. al.: Blinded independent central review of progression-free survival in phase III clinical trials: important design element or unnecessary expense?. J Clin Oncol 2008 Aug 1; 26: pp. 3791-3796. Available from: http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=2654812&tool=pmcentrez&rendertype=abstract
17. Shamsi K., Patt R.H.: Onsite image evaluations and independent image blinded reads: close cousins or distant relatives?. J Clin Oncol 2009 Apr 20; 27: pp. 2103-2104. author reply 2104–5. Available from http://jco.ascopubs.org/content/27/12/2103
18. Ford R., Schwartz L., Dancey J., et. al.: Lessons learned from independent central review. Eur J Cancer 2009 Jan; 45: pp. 268-274. Available from: http://www.ncbi.nlm.nih.gov/pubmed/19101138
19. Zhang J.J., Chen H., He K., et. al.: Evaluation of blinded independent central review of tumor progression in oncology clinical trials: a meta-analysis. Ther Innov Regul Sci 2012 Sep 7; 47: pp. 167-174. Available from: http://dij.sagepub.com/content/47/2/167.short
20. Dodd L.E., Korn E.L., Freidlin B., et. al.: An audit strategy for progression-free survival. Biometrics 2011 Sep; 67: pp. 1092-1099. Available from: http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=3160504&tool=pmcentrez&rendertype=abstract
21. Amit O., Mannino F., Stone A.M., et. al.: Blinded independent central review of progression in cancer clinical trials: results from a meta-analysis. Eur J Cancer 2011 Aug; 47: pp. 1772-1778. Available from: http://www.ncbi.nlm.nih.gov/pubmed/21429737
22. Armato S.G., Meyer C.R., Mcnitt-Gray M.F., et. al.: The Reference Image Database to Evaluate Response to therapy in lung cancer (RIDER) project: a resource for the development of change-analysis software. Clin Pharmacol Ther 2008 Oct; 84: pp. 448-456. Available from: http://www.ncbi.nlm.nih.gov/pubmed/18754000
23. Herman G.T., Zheng J., Bucholtz C.A.: Shape-based interpolation. IEEE Comput Graph Appl 1992; 12:
24. Fleiss J.L.: Measuring nominal scale agreement among many raters. Psychol Bull 1971; 76: pp. 378-382. Available from: http://content.apa.org/journals/bul/76/5/378
25. Barnhart H.X., Haber M.J., Lin L.I.: An overview on assessing agreement with continuous measurements. J Biopharm Stat 2007 Jan; 17: pp. 529-569. Available from: http://www.ncbi.nlm.nih.gov/pubmed/17613641
26. Royston J.P.: An extension of Shapiro and Wilks W test for normality to large samples. Appl Stat 1982; 31: pp. 115-124.
27. Beaumont H., Labatte J.-M., Souchet S., et. al.: Determination of thresholds for the assessment of response to therapy relying on volume-based tumor burden in pulmonary CT images. ASCO Meet Abstr 2013; 31: pp. e18508. Available from: http://meeting.ascopubs.org/cgi/content/abstract/31/15_suppl/e18508
28. R Development Core Team: R: a language and environment for statistical computing.2013.R Foundation for Statistical Computingpp. 409. Available from: http://www.r-project.org
29. Gamer M.: Package irr.2012. Available from: http://cran.r-project.org/web/packages/irr/
30. Landis J.R., Koch G.G.: The measurement of observer agreement for categorical data. Biometrics 1977 Mar; 33: pp. 159-174. Available from: http://www.ncbi.nlm.nih.gov/pubmed/843571
31. Moskowitz C.S., Jia X., Schwartz L.H., et. al.: A simulation study to evaluate the impact of the number of lesions measured on response assessment. Eur J Cancer 2009 Jan; 45: pp. 300-310. Available from: http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=2652848&tool=pmcentrez&rendertype=abstract
32. Darkeh M.H.S.E., Suzuki C., Torkzad M.R.: The minimum number of target lesions that need to be measured to be representative of the total number of target lesions (according to RECIST). Br J Radiol 2009 Aug; 82: pp. 681-686. Available from: http://www.ncbi.nlm.nih.gov/pubmed/19366735
33. Skougaard K., McCullagh M.J.D., Nielsen D., et. al.: Observer variability in a phase II trial—assessing consistency in RECIST application. Acta Oncol 2012 Jul; 51: pp. 774-780. Available from: http://www.ncbi.nlm.nih.gov/pubmed/22432439

Volume-based Response Evaluation with Consensual Lesion Selection

Rationale and Objectives

Materials and Methods

Results

Conclusions

Materials and methods

Data Set

Cloud-based Setup System

Volume Measurement

Response Agreement

Target Lesions

New Lesions

Volume Thresholds

Data Analysis

Results

Response Agreement

Comparison Between VBR and RECIST

Number of Lesions

RECIST Compliance

Discussion

Conclusions

References

Further Reading

Assessing the Appropriateness of Outpatient Abdominopelvic CT and MRI Examinations Using the American College of Radiology Appropriateness Criteria

Automated Breast Segmentation of Fat and Water MR Images Using Dynamic Programming

Automated Tissue Segmentation in Breast MRI