Rationale and Objectives
The International Labour Office (ILO) system for classifying chest radiographic changes related to inhalation of pathogenic dusts is predicated on film-screen radiography. Digital radiography has replaced film in many centers. Digital images can be printed on film (“hard copy”) or can be viewed at a computer workstation (“soft copy”). The goal of the present investigation was to compare the inter-reader and intra-reader agreement of ILO classifications for pneumoconiosis across image formats.
Materials and Methods
Traditional film radiographs, hard copy digital images, and soft copy digital images from 107 subjects were read by six B readers. A multiple reader version of the inter-reader kappa statistic was compared across image formats. Intra-reader kappa comparisons were carried out using an iterative least-squares approach (unadjusted analysis) as well as a two-stage regression model adjusting for readers and subject-level covariates.
Results
There were few significant differences in the inter-reader and intra-reader agreement across formats. For parenchymal abnormalities, inter-reader and intra-reader kappa values ranged from 0.536 to 0.646, and 0.65 to 0.77, respectively. In the covariate-adjusted analysis film-screen radiography was generally associated with a numerically greater reliability (ie, higher kappa values) than the other image formats, although differences were rarely statistically significant.
Conclusion
Film-screen radiographs, hard copy digital images, and soft copy digital images yielded similar reliability measures. These findings provide further support to the recommendation that soft copy digital images can be used for the recognition and classification of dust-related parenchymal abnormalities using the ILO system.
Since the early 20th century, standard posteroanterior (PA) film-screen chest radiography (FSR) has been the primary method for screening, diagnosis, medical monitoring, and epidemiological study of the pneumoconioses. In the 1930s, the International Labour Office (ILO) developed a scoring system for standardizing the classification of radiographs of pneumoconioses . The system has undergone several revisions, most recently in 2000 . The ILO system remains the most widely used method for classifying chest radiographs for pleural and parenchymal abnormalities related to inhalation of pathogenic dusts .
During the past three decades, numerous medical centers around the globe have adopted different forms of digital x-ray imaging into clinical practice. The widespread adoption of digital x-ray technology has numerous implications for use of the ILO system. The ILO system is dependent on the use of traditional FSR that is no longer available in many parts of United States or some other countries. Thus a need and preference for moving towards digital technology is sharply on the rise.
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Materials and methods
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
k=P0−Pc1−Pc, k
=
P
0
−
P
c
1
−
P
c
,
where P0=∑mi=1pii P
0
=
∑
i
=
1
m
p
i
i denotes the observed probability of agreement and Pc=∑mi=1pi.p.i P
c
=
∑
i
=
1
m
p
i
.
p
.
i is the probability of agreement that would be expected purely by chance. Thus kappa measures agreement in excess of what would be expected by chance alone and higher kappa values imply greater reliability. Values of kappa greater than 0.75 are considered excellent, values between 0.40 and 0.75 are fair to good, and values less than 0.40 represent poor agreement beyond chance alone . Cohen recommended using the above definition for kappa for ratings in a nominal scale and suggested using weighted kappa when ordinal rating categories are used . For dichotomous rating scales, all weighted versions reduce to the unweighted version of kappa.
Get Radiology Tree app to read full this article<
Analyzing Inter-reader Reliability
Get Radiology Tree app to read full this article<
Analysis of Intra-reader Reliability
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Regression Analysis of Intra-reader Reliability
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
kappa=β0∑βkWK kappa
=
β
0
∑
β
k
W
K
where kappa is a subject-specific version of intra-reader kappa statistics, the β k are regression coefficients, and the W k are independent covariates, such as age, gender, body mass index, pack-years of smoking and individual B readers. Because models of kappa are not directly implementable in standard software packages, a multistage approach was adopted to fit such a model, as described in the following section.
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Poi=Pci+(1−Pci)ki, P
o
i
=
P
c
i
+
(
1
-
P
c
i
)
k
i
,
where P oi = Pr(X 1 = 0, X 2 = 0) + Pr(X 1 = 1, X 2 = 1) is the probability of exact agreement in rating, and P ci = Pr(X 1 = 0)Pr(X 2 = 0) + Pr(X 1 = 1)Pr(X 2 = 1), X 1 and X 2 being the ratings by the same reader in the two rounds, respectively. Here κ i is the subject-level version of the intra-reader agreement statistic kappa as defined in equation 1 . The basic strategy consists of a two-stage process as follows.
Get Radiology Tree app to read full this article<
Stage 1
Get Radiology Tree app to read full this article<
Stage 2
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Results
Get Radiology Tree app to read full this article<
Table 1
Subject Characteristics
Frequency Percent Gender Male 86 80 Female 21 20 Body mass index (BMI, kg/m 2 ) BMI <25 (normal) 28 26 25 ≤BMI <30 (overweight) 45 42 30 ≤BMI (obese) 34 32 Ever smoked No 39 36 Yes 68 64 Current smoking No 97 91 Yes 10 9 History of dust exposure No 47 44 Yes 60 56 Dust exposure type ( n = 60) ∗ Silica 34 57 Asbestos 28 47 Other/unknown 12 20 Mean (SD) Median (range) Age (y) 64.7 (11.9) 65 (31–91) BMI (kg/m 2 ) 28.5 (5.2) 28.1 (19.5–48.8) Pack-years All subjects ( n = 107) 19.5 (24.1) 12 (0–96) Ever smoked ( n = 68) 30.7 (23.8) 23.5 (1–96)
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Inter-reader Reliability
Get Radiology Tree app to read full this article<
Table 2A
Inter-reader Kappa Values of Reader Agreement by Image Format
Inter-reader Kappa
Round 1 Inter-reader Kappa
Round 2 FSR HC SC FSR HC SC 1.A: Image Quality (IQ: 4-point ordinal scale) 0.318 0.163 0.206 0.225 0.191 0.193 1.A: IQ (Category 1 versus 2–4) 0.338 0.270 0.242 0.245 0.266 0.244 1.A: IQ (Categories 1–2 versus 3–4) 0.432 0.215 0.182 0.305 0.218 0.087 2.A: Any parenchymal abnormalities (yes/no) 0.595 0.553 0.629 0.623 0.536 0.646 2.B: Small opacities (12-point scale) 0.288 0.225 0.271 0.258 0.240 0.256 2.B: Small opacities (4-point scale) 0.468 0.425 0.443 0.452 0.449 0.458 2.C: Large opacities (4-point scale) 0.463 0.506 0.463 0.453 0.531 0.447 2.C: Large opacities (yes/no) 0.622 0.701 0.621 0.597 0.718 0.635 3.A: Pleural abnormalities (yes/no) 0.415 0.524 0.499 0.462 0.421 0.437 3.C: Costophrenic angle obliteration (yes/no) 0.566 0.567 0.500 0.531 0.370 0.367 3.D: Diffuse pleural thickening (yes/no) 0.599 0.557 0.559 0.619 0.416 0.454
Table 2B
Significant Differences Among Inter-reader Kappa Values for Round 1 ∗
Kappa
Difference SE Lower
CI Upper
CI Kappa difference image quality (4-point ordinal scale): FSR versus HC 0.155 0.052 0.046 0.250 Kappa difference image quality (categories 1–2 versus 3–4): FSR versus HC 0.217 0.096 0.010 0.376 Kappa difference image quality (categories 1–2 versus 3–4): FSR versus SC 0.250 0.100 0.032 0.417
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Intra-reader Reliability
Get Radiology Tree app to read full this article<
Table 3
Intra-reader Kappa Values within and between Formats ∗
Variables Within Format † Between Format ‡ FSR HC SC FSR and HC HC and SC FSR and SC 1.A: Image quality (IQ: 4-point ordinal) 0.49 (0.31, 0.68) 0.41 (0.29, 0.52) 0.42 (0.21, 0.65) 0.13 (-0.11, 0.45) 0.28 (0.18, 0.50) 0.18 (0.05, 0.41) 1.A: IQ (Category 1 vs. 2–4) 0.54 (0.30, 0.76) 0.45 (0.07, 0.64) 0.48 (0.23, 0.67) 0.16 (-0.16, 0.53) 0.34 (0.17, 0.59) 0.21 (0.03, 0.48) 1.A: IQ (Categories 1–2 vs. 3–4) 0.51 (0.34, 0.65) 0.38 (-0.01, 0.52) 0.29 (-0.03, .49) 0.16 (-0.02, 0.38) 0.23 (-0.02, 0.51) 0.13 (-0.04, 0.36) 2.A: Parenchymal abnormalities (yes/no) 0.77 (0.71, 0.82) 0.75 (0.66, 0.79) 0.72 (0.61, 0.86) 0.65 (0.54, 0.75) 0.68 (0.52, 0.81) 0.70 (0.57, 0.80) 2.B: Small opacities (12-point scale) 0.39 (0.29, 0.46) 0.36 (0.26, 0.48) 0.37 (0.26, 0.51) 0.29 (0.22, 0.38) 0.33 (0.25, 0.40) 0.35 (0.27, 0.41) 2.B: Small opacities (4-point scale) 0.61 (0.49, 0.67) 0.60 (0.47, 0.70) 0.59 (0.50, 0.68) 0.48 (0.40, 0.59) 0.53 (0.47, 0.56) 0.56 (0.41, 0.64) 2.C: Large opacities (4-point scale) 0.70 (0.60, 0.86) 0.70 (0.62, 0.77) 0.61 (0.48, 0.83) 0.63 (0.55, 0.83) 0.58 (0.50, 0.77) 0.58 (0.44, 0.76) 2.C: Large opacities (yes/no) 0.81 (0.67, 0.96) 0.81 (0.77, 0.85) 0.75 (0.69, 0.88) 0.74 (0.60, 0.92) 0.74 (0.64, 0.83) 0.72 (0.57, 0.88) 3.A: Pleural abnormalities (yes/no) 0.69 (0.54, 0.83) 0.66 (0.47, 0.85) 0.69 (0.59, 0.84) 0.59 (0.49, 0.64) 0.66 (0.58, 0.73) 0.56 (0.49, 0.68) 3.C: Costophrenic angle Obliteration (yes/no) 0.73 (0.48, 0.85) 0.52 (0, 0.85) 0.59 (0, 0.79) 0.46 (-0.02, 0.65) 0.69 (0.49, 0.93) 0.48 (0.19, 0.62) 3.D: Diffuse pleural thickening (yes/no) 0.71 (0.48, 0.92) 0.58 (0, 1) 0.65 (0, 1) 0.54 (-0.02, 0.79) 0.72 (0.49, 1) 0.57 (0.19, 0.79)
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Table 4
Intra-Reader Comparison of Kappas (Unadjusted for Covariates)
FSR
Kappa FSR
SE HC
Kappa HC
SE SC
Kappa SC
SE FSR vs. HC
P Value FSR vs. SC
P Value HC vs. SC
P Value Reader 1 Image quality (Category 1 vs. 2–4) 0.30 (0.11) 0.41 (0.12) 0.61 (0.09) .485.020 .158 Image quality (Categories 1–2 vs. 3–4) 0.43 (0.18) 0.52 (0.10) 0.29 (0.18) .626 .609 .248 Parenchymal abnormalities (yes/no) 0.70 (0.08) 0.79 (0.08) 0.67 (0.09) .416 .780 .285 Large opacities (yes/no) 0.70 (0.11) 0.77 (0.10) 0.68 (0.10) .673 .910 .476 Pleural abnormalities (yes/no) 0.55 (0.09) 0.48 (0.08) 0.62 (0.08) .549 .467 .162 Costophrenic angle obliteration (yes/no) 0.48 (0.22) 0 NA 0 NA NA NA NA Diffuse pleural thickening (yes/no) 0.48 (0.22) 0 NA 0 NA NA NA NA Reader 2 Image quality (Category 1 vs. 2–4) 0.76 (0.07) 0.63 (0.08) 0.68 (0.07) .233 .431 .580 Image quality (Categories 1–2 vs. 3–4) 0.43 (0.16) 0.49 (0.11) 0.43 (0.14) .731 .995 .661 Parenchymal abnormalities (yes/no) 0.74 (0.07) 0.77 (0.06) 0.74 (0.07) .727 .990 .641 Large opacities (yes/no) 0.96 (0.04) 0.85 (0.07) 0.81 (0.09) .113 .119 .723 Pleural abnormalities (yes/no) 0.54 (0.07) 0.64 (0.08) 0.67 (0.08) .287 .174 .781 Costophrenic angle obliteration (yes/no) 0.73 (0.09) 0.72 (0.11) 0.72 (0.10) .910 .920 .986 Diffuse pleural thickening (yes/no) 0.66 (0.12) 0.63 (0.13) 0.67 (0.13) .843 .937 .813 Reader 3 Image quality (Category 1 vs. 2–4) 0.56 (0.08) 0.55 (0.08) 0.23 (0.09) .894.007.007 Image quality (Categories 1–2 vs. 3–4) 0.64 (0.15) 0.38 (0.12) 0.19 (0.19) .154.023 .340 Parenchymal abnormalities (yes/no) 0.84 (0.06) 0.79 (0.07) 0.87 (0.05) .479 .632 .295 Large opacities (yes/no) 0.88 (0.06) 0.84 (0.06) 0.88 (0.06) .394 .917 .669 Pleural abnormalities (yes/no) 0.67 (0.08) 0.85 (0.06) 0.60 (0.10).046 .502.021 Costophrenic angle obliteration (yes/no) 0.85 (0.11) 0.59 (0.17) 0.65 (0.19) .083 .259 .766 Diffuse pleural thickening (yes/no) 0.92 (0.08) 0.65 (0.19) 0.74 (0.18) .113 .345 .740 Reader 4 Image quality (Category 1 vs. 2–4) 0.49 (0.11) 0.07 (0.11) 0.25 (0.11).008 .098 .232 Image quality (Categories 1–2 vs. 3–4) 0.34 (0.11) 0.50 (0.10) 0.39 (0.10) .315 .764 .336 Parenchymal abnormalities (yes/no) 0.75 (0.07) 0.72 (0.09) 0.71 (0.07) .755 .605 .952 Large opacities (yes/no) 0.84 (0.08) 0.84 (0.07) 0.72 (0.10) .948 392 .316 Pleural abnormalities (yes/no) 0.75 (0.07) 0.65 (0.08) 0.67 (0.09) .254 .423 .830 Costophrenic angle obliteration (yes/no) 0.75 (0.14) 0.85 (0.15) 0.74 (0.18) .625 .946 .305 Diffuse pleural thickening (yes/no) 0.74 (0.18) 1.00 NA 0.79 (0.20) NA .836 NA Reader 5 Image quality (Category 1 vs. 2–4) 0.55 (0.08) 0.61 (0.09) 0.67 (0.07) .632 .284 .606 Image quality (Categories 1–2 vs. 3–4) 0.65 (0.19) 0.42 (0.14) 0.49 (0.31) .268 .657 .807 Parenchymal abnormalities (yes/no) 0.77 (0.06) 0.77 (0.06) 0.79 (0.06) .998 .822 .807 Large opacities (yes/no) 0.73 (0.13) 0.80 (0.10) 0.70 (0.14) .689 .879 .533 Pleural abnormalities (yes/no) 0.83 (0.06) 0.59 (0.09) 0.84 (0.06).008 .868.016 Costophrenic angle obliteration (yes/no) 0.79 (0.14) 0.20 (0.18) 0.79 (0.14).002 .998.002 Diffuse pleural thickening (yes/no) 0.88 (0.11) 0.23 (0.20) 0.74 (0.18).001 .524.021 Reader 6 Image quality (Category 1 vs. 2–4) 0.55 (0.10) 0.46 (0.12) 0.48 (0.10) .550 .599 .921 Image quality (Categories 1–2 vs. 3–4) 0.58 (0.19) 0.17 (0.11) 0.07 (0.05) .059.007 .379 Parenchymal abnormalities (yes/no) 0.81 (0.06) 0.65 (0.09) 0.61 (0.09) .074 .053 .678 Large opacities (yes/no) 0.85 (0.06) 0.80 (0.07) 0.72 (0.10) .532 .140 .513 Pleural abnormalities (yes/no) 0.79 (0.06) 0.74 (0.08) 0.76 (0.07) .669 .734 .911 Costophrenic angle obliteration (yes/no) 0.83 (0.10) 0.78 (0.12) 0.68 (0.13) .778 .321 .508 Diffuse pleural thickening (yes/no) 0.65 (0.19) 1.00 NA 1.00 NA NA NA NA
FSR, film-screen chest radiography; SE, standard error; HC, hard copy; SC, soft copy.
Table values show kappa estimates for each image format (FSR, SC, and HC), the respective SE, and the P values for the pairwise compar-isons for each image format separately for each of the six B readers.
NA = not available due to the fact that either (a) kappa = 1.00 (boundary value) or (b) all ratings in one of the rounds fell in a single category.
Table 5
Difference in Intra-reader Reliability across Image Formats (Covariate Adjusted) ∗
FSR vs. HC FSR vs. SC HC vs. SC 1.A: Image quality (4-pt scale) 0.0180 (0.89) −0.0009 (0.99) −0.0189 (0.89) 1.A: Image quality (Category 1 vs. 2–4) 0.0351 (0.58)0.1103 ( 0.05) 0.0752 (0.19) 1.A: Image quality (Categories 1–2 vs. 3–4) −0.0100 (0.93)0.2086 ( 0.07)0.2186 ( 0.01) 2.A: Parenchy abnormalities (yes/no) 0.0165 (0.74) 0.0451 (0.17) 0.0286 (0.51) 2.B: Small opacities (4-pt scale) 0.0044 (0.90) 0.0261 (0.36) 0.0218 (0.49) 2.C: Large opacities (yes/no) 0.0768 (0.15) 0.1161 (0.11) 0.0393 (0.61) 2.C: Large opacities (4-pt scale)0.0982 ( 0.046)0.1502 ( 0.06) 0.0520 (0.48) 3.A: Pleural abnormalities (yes/no) 0.0428 (0.37) −0.0738 (0.16)−0.1166 ( 0.03) 3.C: Costophrenic angle obliteration (yes/no) 0.0851 (0.34) 0.0693 (0.39) −0.0158 (0.81) 3.D: Diffuse pleural thickening (yes/no) 0.0596 (0.63) 0.0307 (0.78) −0.0290 (0.83)
The model for small opacities based on the 12-point scale is not presented due to the sparsity of the underlying 12 × 12 matrix.
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Discussion
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Acknowledgments
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
References
1. Musch D.C.: Interobserver variation in classifying radiographs for the pneumoconioses.1981.University of Michigan Doctoral DissertationAnn Arbor, MI
2. International Labour Organization (ILO): Guidelines for the use of ILO international classification of radiographs of pneumoconioses, Revised edition 2000.2002.International Labour OfficeGeneva
3. Pham Q.T.: Chest radiography in the diagnosis of pneumoconiosis. Int J Tuberc Lung Dis 2001; 5: pp. 478-482.
4. Mulloy K.B., Coultas D.B., Samet J.M.: Use of chest radiographs in epidemiological investigations of pneumoconioses. Br J Indust Med 1993; 50: pp. 273-275.
5. Zähringer M., Piekarski C., Saupe M., et. al.: Comparison of digital selenium radiography with an analog screen-film system in the diagnostic process of pneumoconiosis according to ILO classification. [in German] Fortschr Röntgenstr 2001; 173: pp. 942-948.
6. Takashima Y., Suganuma N., Sakurazawa H., et. al.: A flat-panel detector digital radiography and a storage phosphor computed radiography: screening for pneumoconioses. J Occup Health 2007; 49: pp. 39-45.
7. Franzblau A., Kazerooni E.A., Sen A., et. al.: Comparison of digital radiographs with film radiographs for the classification of pneumoconiosis. Acad Radiol 2009; 16: pp. 669-677.
8. Rosenman K.D., Reilly M.J., Kalinowski D.J., et. al.: Silicosis in the 1990’s. Chest 1997; 111: pp. 779-786.
9. Reilly M.J., Rosenman K.D., Watt F.C., et. al.: Silicosis surveillance - Michigan, New Jersey, Ohio, and Wisconsin, 1987–1990. MMWR 1993; 42: pp. 23-28.
10. American Thoracic Society: Epidemiology standardization project: recommended respiratory disease questionnaires for use with adults and children in epidemiological research. Am Rev Resp Dis 1978; 118: pp. 7-53.
11. SAS Institute Inc: SAS/STAT user’s guide, version 9.1.2002.SAS Institute IncCary, NC
12. Cohen J.: A coefficient of agreement for nominal scales. Educ Psychol Measure 1960; 20: pp. 37-46.
13. Fleiss J.L., Levin B., Paik M.C.: Statistical methods for rates and proportions.3rd ed2003.John Wiley & SonsNew York
14. Barnhart H.X., Williamson J.M.: Weighted least-squares approach for comparing correlated kappa. Biometrics 2002; 58: pp. 1012-1019.
15. Lipsitz S.R., Williamson J., Klar N., et. al.: A simple method for estimating a regression model for κ between a pair of readers. J Royal Stat Soc Series A 2001; 164: pp. 449-465.
16. Huuskonen O., Kivisaari L., Zitting A., et. al.: High-resolution computed tomography classification of lung fibrosis for patients with asbestos-related disease. Scand J Work Environ Health 2001; 27: pp. 106-112.
17. Impivaara O., Zitting A.J., Kuusela T., et. al.: Observer variation in classifying chest radiographs for small lung opacities and pleural abnormalities in a population sample. Am J Indust Med 1998; 34: pp. 261-265.
18. Lawson C.C., LeMasters M.K., Lemasters G.K., et. al.: Reliability and validity of chest radiograph surveillance programs. Chest 2001; 120: pp. 64-68.
19. Musch D.C., Landis R., Higgins I.T.T., et. al.: An application of kappa-type analyses to interobserver variation in classifying chest radiographs for pneumoconiosis. Stat Med 1984; 3: pp. 73-83.
20. Musch D.C., Higgins I.T.T., Landis J.R.: Some factors influencing interobserver variation in classifying simple pneumoconiosis. Brit J Ind Med 1985; 42: pp. 346-349.
21. Naidoo R.N., Robins T.G., Soloman A., et. al.: Radiographic outcomes among South African coal miners. Int Arch Occup Environ Health 2004; 77: pp. 471-481.
22. Welch L.S., Hunting K.L., Balmes J., et. al.: Variability in classification of radiographs using the 1980 International Labor Organization Classification for Pneumoconioses. Chest 1998; 114: pp. 1740-1748.