Estimating Screening-Mammography Receiver Operating Characteristic (ROC) Curves from Stratified Random Samples of Screening Mammograms

Rationale and Objectives

To evaluate stratified random sampling (SRS) of screening mammograms by (1) Breast Imaging Reporting and Data System (BI-RADS) assessment categories, and (2) the presence of breast cancer in mammograms, for estimation of screening-mammography receiver operating characteristic (ROC) curves in retrospective observer studies.

Materials and Methods

We compared observer study case sets constructed by (1) random sampling (RS); (2) SRS with proportional allocation (SRS-P) with BI-RADS 1 and 2 noncancer cases accounting for 90.6% of all noncancer cases; (3) SRS with disproportional allocation (SRS-D) with BI-RADS 1 and 2 noncancer cases accounting for 10%–80%; and (4) SRS-D and multiple imputation (SRS-D + MI) with missing BI-RADS 1 and 2 noncancer cases imputed to recover the 90.6% proportion. Monte Carlo simulated case sets were drawn from a large case population modeled after published Digital Mammography Imaging Screening Trial data. We compared the bias, root-mean-square error, and coverage of 95% confidence intervals of area under the ROC curve (AUC) estimates from the sampling methods (200–2000 cases, of which 25% were cancer cases) versus from the large case population.

Results

AUC estimates were unbiased from RS, SRS-P, and SRS-D + MI, but biased from SRS-D. AUC estimates from SRS-P and SRS-D + MI had 10% smaller root-mean-square error than RS.

Conclusions

Both SRS-P and SRS-D + MI can be used to obtain unbiased and 10% more efficient estimate of screening-mammography ROC curves.

Screening mammography for early detection of breast cancer leads to reduction in breast cancer mortality . Radiologists’ interpretation of screening mammograms is paramount to the effectiveness of breast cancer screening. Receiver operating characteristic (ROC) analysis, which summarizes inherent trade-offs between sensitivity and specificity as the decision threshold is made more or less stringent, is an established method for the assessment of diagnostic performance. However, reliable estimation of ROC curves requires both the diagnostic “truth” (ie, whether breast cancer is present in the mammogram) and an ordinal response for every patient from the radiologist of an estimated “likelihood of malignancy.” Therefore, ROC curves are usually estimated only in retrospective observer-performance studies (hereafter simply observer studies), in which readers provide likelihood of malignancy responses to cases of which diagnostic truth has been independently verified.

It is impractical to estimate screening-mammography ROC curves in observer studies by simple random sampling (RS) of clinical cases because of the low prevalence of breast cancer, approximately five per 1000 screening mammograms . This low cancer prevalence implies large uncertainty in the ROC curve estimate even if the total number of cases is large . Furthermore, 90% or more of all cases will be interpreted as either Breast Imaging Reporting and Data System (BI-RADS) assessment category 1 (negative) or 2 (benign finding), leading to repetitive and uninteresting studies for the observers . Investigators often increase the prevalence of cancer cases in observer studies by including fewer noncancer cases then seen in clinical practice. This approach can greatly alleviate the difficulty caused by low cancer prevalence and increase the efficiency of the observer study by decreasing the uncertainty of ROC curve estimates without increasing the number of cases in the study.

Get Radiology Tree app to read full this article<

Materials and methods

Get Radiology Tree app to read full this article<

Stratified Random Sampling

Get Radiology Tree app to read full this article<

Table 1

Observer Study Case Set Sampling Methods

Method Abbreviation Explanation Random sampling RS ∗ Random sample from large clinical case population Stratified random sampling with proportional allocation SRS-P ∗ Random sample within each stratum of (1) noncancer cases assigned BI-RADS 1 clinically, (2) noncancer cases assigned BI-RADS 2 clinically, (3) noncancer cases assigned BI-RADS 0 clinically, and (4) cancer cases, with fixed number-of-case ratios between noncancer-case strata identical to those of large clinical case population Stratified random sampling with disproportional allocation SRS-D Random sample within each stratum as in SRS-P, with fixed number-of-case ratios between noncancer-case strata different from those of large clinical case population Stratified random sampling with disproportional allocation with multiple imputation SRS-D + MI Random sample within each stratum as in SRS-P, with fixed number-of-case ratios between noncancer-case strata different from those of large clinical case population. After multiple imputation, the number-of-case ratios between noncancer-case strata becomes identical to those of large clinical case population

The proportion of cancer cases was fixed at 25% for all methods (before multiple imputation).

Get Radiology Tree app to read full this article<

MI of BI-RADS 1 and 2 Noncancer Cases

Get Radiology Tree app to read full this article<

Table 2

The DMIST Case Population by BI-RADS Assessment Categories and Likelihood of Malignancy Scores

Cases BI-RADS Assessment Categories ∗ Total 1 2 0 Noncancer cases Number 30,232 8538 3546 42,316 Proportion (%) 71.4 20.2 8.4 100.0 Cancer cases Number 51 26 177 254 Proportion (%) 20.1 10.2 69.7 100.0

Likelihood of Malignancy Scores † Total 1 2 3 4 5 6 7 Noncancer cases Number 32,466 6563 2175 976 44 11 1 42,236 Proportion (%) 76.868 15.539 5.150 2.311 0.104 0.026 0.002 100.000 Cancer cases Number 122 25 49 85 25 18 10 334 Proportion (%) 36.5 7.5 14.7 25.4 7.5 5.4 3.0 100.0

BI-RADS, Breast Imaging Reporting and Data System; DMIST, Digital Mammography Imaging Screening Trial.

Adapted from Ref. , Tables 3 and 4 , data on digital mammography. The two case populations in the top and bottom halves of the table are not identical because of different lengths of follow-up.

Get Radiology Tree app to read full this article<

Characteristics of Screening-Mammography Case Population

Get Radiology Tree app to read full this article<

We used the likelihood of malignancy scores, not the BI-RADS assessment data, to estimate the ROC curve . For cancer cases, we sampled randomly (by proportion) from the DMIST cancer-case likelihood of malignancy score data ( Table 2 ). However, for noncancer cases, we needed to simulate the likelihood of malignancy scores according to BI-RADS assessment categories, 1, 2, and 0, which were not available in the published DMIST data . Instead, we sampled noncancer cases (by proportion) presented in Table 2 according to three levels of correlation strength between the BI-RADS assessment categories and likelihood of malignancy scores: Spearman correlation coefficients 0.89, 0.80, and 0.69, denoted as high, medium, and low correlation, respectively. These magnitudes of the correlation strength were based on two laboratory observer studies in which observers reported both the BI-RADS assessment and likelihood of malignancy score [Jiang Y. et al., unpublished data 2003, and Ref. ]. Given differences between retrospective laboratory studies and a prospective clinical trial, and given differences in the proportions of case mix, we chose a fairly wide range in the magnitudes of the correlation strength in an effort to capture the true correlation strength in DMIST, which could not be calculated from the partially published data . Our results (described subsequently) showed little dependence on these choices of the correlation strength.

Get Radiology Tree app to read full this article<

Example Case Sets

Get Radiology Tree app to read full this article<

Table 3

The Number of Cases in Four Example Observer Study Case Sets by BI-RADS Assessment Categories and Likelihood of Malignancy Scores

Case sets BI-RADS Assessment Categories Total 1 2 0 Noncancer cases Case set 1: RS 64 25 11 100 Case set 2: SRS-P ∗ 71 20 9 100 Case set 3: SRS-D ∗ 39 11 50 100 Case set 4: SRS-D ∗ + MI 427 † (388) ‡ 120 † (109) ‡ 50 597 † (497) ‡ Cancer cases Case sets 1–4 7 3 23 33

Likelihood of Malignancy Scores Total 1 2 3 4 5 6 7 Noncancer cases Case set 1: RS 78 15 3 4 0 0 0 100 Case set 2: SRS-P ∗ 77 16 5 2 0 0 0 100 Case set 3: SRS-D ∗ 41 12 28 17 1 1 0 100 Case set 4: SRS-D ∗ + MI 438 † (397) ‡ 112 † (100) ‡ 28 17 1 1 0 597 † (497) ‡ Cancer cases Case sets 1–4 12 3 5 8 2 2 1 33

BI-RADS, Breast Imaging Reporting and Data System; RS, random sampling; SRS-D, stratified random sampling with disproportional allocation; SRS-D + MI, stratified random sampling with disproportional allocation with multiple imputation; SRS-P, stratified random sampling with proportional allocation.

Get Radiology Tree app to read full this article<

Simulation Study and Data Analysis

Get Radiology Tree app to read full this article<

Table 4

Summary of Monte Carlo Simulated Observer Study Case Sets by Clinical BI-RADS Assessment Categories

Case Set Sampling Method ∗ Total Number of Cases (Cancer + Noncancer) to be Read by Observer Total Number of Cancer Cases to be Read by Observer ∗ Total Number of BI-RADS 0 Noncancer Cases to be Read by Observer Total Number of BI-RADS 1 and 2 Noncancer Cases to be Read by Observer Total Number of Imputed BI-RADS 1 and 2 Noncancer Cases Effective Cancer Prevalence (Read + Imputed Cases) Proportion of BI-RADS 1 and 2 Noncancer Cases (Read + Imputed) RS N 0.25N 0.063N † 0.687N † 0 25% ‡ 91.6% † , § SRS-P N 0.25N 0.063N ‡ 0.687N ‡ 0 25% ‡ 91.6% ‡ , § SRS-D N 0.25N 0.150N, 0.225N, 0.300N, 0.375N, 0.450N, 0.525N, 0.600N, 0.675N ‡ 0.600N, 0.525N, 0.450N, 0.375N, 0.300N, 0.225N, 0.150N, 0.075N ‡ 0 25% ‡ 80%, 70%, 60%, 50%, 40%, 30%, 20%, 10% ‡ SRS-D + MI N 0.25N 0.150N, 0.225N, 0.300N, 0.375N, 0.450N, 0.525N, 0.600N, 0.675N ‡ 0.600N, 0.525N, 0.450N, 0.375N, 0.300N, 0.225N, 0.150N, 0.075N ‡ 1.04N, 1.93N, 2.82N, 3.71N, 4.61N, 5.50N, 6.39N, 7.29N ‡ 12.3%, 8.5%, 6.5%, 5.3%, 4.5%, 3.8%, 3.4%, 3.0% ‡ 91.6% ‡ , §

N = 200, 400, 800, 1200, or 2000. Imputed cases are not to be read by the observer.

Get Radiology Tree app to read full this article<

Results

Get Radiology Tree app to read full this article<

Table 5

Comparison of Bias and RMSE of AUC Estimates Between RS and SRS-P

N ∗ RS SRS-P RMSE Ratio (SRS-P/RS) Low correlation Bias RMSE Bias RMSE 200 0.0000 0.0409 0.0004 0.0376 0.919 400 0.0002 0.0291 0.0004 0.0264 0.907 800 0.0000 0.0206 0.0003 0.0187 0.907 1200 −0.0003 0.0168 0.0002 0.0153 0.911 2000 −0.0002 0.0131 0.0000 0.0119 0.908 Medium correlation 200 0.0005 0.0414 −0.0001 0.0370 0.894 400 0.0001 0.0292 0.0001 0.0263 0.901 800 0.0001 0.0205 0.0003 0.0187 0.912 1200 0.0000 0.0168 0.0003 0.0151 0.898 2000 0.0000 0.0130 0.0002 0.0117 0.900 High correlation 200 −0.0004 0.0414 −0.0003 0.0368 0.889 400 0.0000 0.0295 0.0004 0.0262 0.888 800 0.0000 0.0208 0.0002 0.0184 0.885 1200 −0.0002 0.0167 0.0004 0.0152 0.910 2000 0.0000 0.0130 0.0000 0.0116 0.892

AUC, area under the curve; RMSE, root-mean-square error; RS, random sampling; SRS-P, stratified random sampling with proportional allocation.

Get Radiology Tree app to read full this article<

Table 6

Comparison of Bias and RMSE of AUC Estimates Between RS and SRS-D

N ∗ RS † SRS-D 10% BI-RADS 1 and 2 ‡ 30% BI-RADS 1 and 2 ‡ 50% BI-RADS 1 and 2 ‡ 70% BI-RADS 1 and 2 ‡ Bias RMSE Bias RMSE RMSE Ratio (SRS-D/RS) Bias RMSE RMSE Ratio (SRS-D/RS) Bias RMSE RMSE Ratio (SRS-D/RS) Bias RMSE RMSE Ratio (SRS-D/RS) Low correlation 200 0.0000 0.0409 −0.1337 0.1420 3.472 −0.0997 0.1095 2.677 −0.0682 0.0811 1.983 −0.0365 0.0545 1.333 400 0.0002 0.0291 −0.1346 0.1389 4.773 −0.1004 0.1054 3.622 −0.0675 0.0741 2.546 −0.0351 0.0452 1.553 800 0.0000 0.0206 −0.1325 0.1347 6.539 −0.1004 0.1029 4.995 −0.0677 0.0712 3.456 −0.0348 0.0403 1.956 1200 −0.0003 0.0168 −0.1339 0.1354 8.060 −0.1004 0.1021 6.077 −0.0684 0.0705 4.196 −0.0356 0.0393 2.339 2000 −0.0002 0.0131 −0.1337 0.1346 10.275 −0.1007 0.1018 7.771 −0.0677 0.0691 5.275 −0.0353 0.0375 2.863 Medium correlation 200 0.0005 0.0414 −0.2293 0.2339 5.650 −0.1724 0.1780 4.300 −0.1165 0.1233 2.978 −0.0595 0.0716 1.729 400 0.0001 0.0292 −0.2314 0.2337 8.003 −0.1736 0.1762 6.034 −0.1163 0.1198 4.103 −0.0603 0.0662 2.267 800 0.0001 0.0205 −0.2294 0.2305 11.244 −0.1736 0.1749 8.532 −0.1162 0.1180 5.756 −0.0608 0.0637 3.107 1200 0.0000 0.0168 −0.2313 0.2320 13.810 −0.1734 0.1743 10.375 −0.1174 0.1186 7.060 −0.0610 0.0630 3.750 2000 0.0000 0.0130 −0.2305 0.2310 17.769 −0.1743 0.1748 13.446 −0.1170 0.1178 9.062 −0.0604 0.0617 4.746 High correlation 200 −0.0004 0.0414 −0.2805 0.2847 6.877 −0.2119 0.2167 5.234 −0.1445 0.1502 3.628 −0.0730 0.0824 1.990 400 0.0000 0.0295 −0.2816 0.2837 9.617 −0.2123 0.2149 7.285 −0.1423 0.1453 4.925 −0.0747 0.0793 2.688 800 0.0000 0.0208 −0.2809 0.2820 13.558 −0.2125 0.2137 10.274 −0.1431 0.1446 6.952 −0.0739 0.0765 3.678 1200 −0.0002 0.0167 −0.2837 0.2844 17.030 −0.2130 0.2138 12.802 −0.1443 0.1453 8.701 −0.0749 0.0765 4.581 2000 0.0000 0.0130 −0.2818 0.2822 21.708 −0.2130 0.2135 16.423 −0.1443 0.1448 11.138 −0.0744 0.0754 5.800

AUC, area under the curve; BI-RADS, Breast Imaging Reporting and Data System; RMSE, root-mean-square error; RS, random sampling; SRS-D, stratified random sampling with disproportional allocation.

Results for four of the eight different types (proportion of BI-RADS 1 and 2 cases) of SRS-D case sets are shown.

Get Radiology Tree app to read full this article<

Table 7

Comparison of Bias and RMSE of AUC Estimates Between RS and SRS-D + MI

N ∗ RS † SRS-D + MI 10% BI-RADS 1 and 2 ‡ 30% BI-RADS 1 and 2 ‡ 50% BI-RADS 1 and 2 ‡ 70% BI-RADS 1 and 2 ‡ Bias RMSE Bias RMSE RMSE Ratio (SRS-D + MI/RS) Bias RMSE RMSE Ratio (SRS-D + MI/RS) Bias RMSE RMSE Ratio (SRS-D + MI/RS) Bias RMSE RMSE Ratio (SRS-D + MI/RS) Low correlation 200 0.0000 0.0409 0.0003 0.0369 0.902 0.0000 0.0370 0.905 0.0003 0.0367 0.897 0.0007 0.0364 0.890 400 0.0002 0.0291 −0.0008 0.0253 0.869 0.0019 0.0270 0.928 0.0009 0.0258 0.887 0.0008 0.0256 0.880 800 0.0000 0.0206 0.0006 0.0185 0.898 0.0005 0.0188 0.913 0.0006 0.0183 0.888 0.0000 0.0184 0.893 1200 −0.0003 0.0168 −0.0003 0.0153 0.911 0.0004 0.0147 0.875 −0.0003 0.0151 0.899 −0.0004 0.0151 0.899 2000 −0.0002 0.0131 0.0005 0.0118 0.901 0.0006 0.0116 0.886 0.0006 0.0115 0.878 0.0006 0.0115 0.878 Medium correlation 200 0.0005 0.0414 0.0006 0.0367 0.886 0.0004 0.0374 0.903 −0.0001 0.0367 0.886 0.0003 0.0366 0.884 400 0.0001 0.0292 0.0011 0.0260 0.890 0.0011 0.0260 0.890 −0.0002 0.0264 0.904 0.0024 0.0260 0.890 800 0.0001 0.0205 0.0002 0.0185 0.902 0.0009 0.0183 0.893 0.0000 0.0181 0.883 0.0002 0.0185 0.902 1200 0.0000 0.0168 0.0000 0.0146 0.869 −0.0001 0.0147 0.875 0.0003 0.0151 0.899 0.0002 0.0151 0.899 2000 0.0000 0.0130 0.0002 0.0116 0.892 0.0003 0.0116 0.892 0.0005 0.0117 0.900 0.0001 0.0118 0.908 High correlation 200 −0.0004 0.0414 0.0004 0.0359 0.867 0.0004 0.0381 0.920 0.0002 0.0366 0.884 0.0007 0.0364 0.879 400 0.0000 0.0295 0.0012 0.0257 0.871 0.0011 0.0257 0.871 0.0012 0.0264 0.895 0.0021 0.0261 0.885 800 0.0000 0.0208 0.0007 0.0180 0.865 0.0000 0.0183 0.880 −0.0004 0.0188 0.904 0.0004 0.0187 0.899 1200 −0.0002 0.0167 0.0002 0.0154 0.922 0.0003 0.0152 0.910 0.0001 0.0152 0.910 −0.0005 0.0153 0.916 2000 0.0000 0.0130 −0.0001 0.0116 0.892 0.0000 0.0114 0.877 0.0005 0.0115 0.885 0.0008 0.0117 0.900

AUC, area under the curve; BI-RADS, Breast Imaging Reporting and Data System; RMSE, root-mean-square error; RS, random sampling; SRS-D + MI, stratified random sampling with disproportional allocation with multiple imputation.

Results for four of the eight different types (proportion of BI-RADS 1 and 2 cases) of SRS-D + MI case sets are shown.

Get Radiology Tree app to read full this article<

Table 8

Comparison of Coverage of 95% CIs of AUC Estimates Among RS, SRS-P, and SRS-D + MI

N ∗ RS SRS-P SRS-D + MI 10% BI-RADS 1 and 2 † 30% BI-RADS 1 and 2 † 50% BI-RADS 1 and 2 † 70% BI-RADS 1 and 2 † Low correlation 200 0.955 0.970 0.962 0.981 0.969 0.980 400 0.951 0.970 0.977 0.977 0.979 0.979 800 0.951 0.971 0.978 0.979 0.975 0.977 1200 0.950 0.969 0.972 0.969 0.982 0.979 2000 0.951 0.969 0.978 0.974 0.980 0.984 Medium correlation 200 0.952 0.971 0.968 0.977 0.979 0.967 400 0.951 0.971 0.978 0.978 0.976 0.980 800 0.953 0.969 0.975 0.982 0.978 0.987 1200 0.951 0.970 0.983 0.983 0.981 0.981 2000 0.952 0.973 0.982 0.981 0.979 0.976 High correlation 200 0.949 0.973 0.968 0.971 0.975 0.972 400 0.947 0.971 0.975 0.973 0.981 0.983 800 0.949 0.973 0.977 0.981 0.982 0.983 1200 0.954 0.970 0.980 0.978 0.979 0.986 2000 0.951 0.975 0.983 0.983 0.980 0.981

AUC, area under the curve; BI-RADS, Breast Imaging Reporting and Data System; CIs, confidence intervals; RS, random sampling; SRS-D + MI, stratified random sampling with disproportional allocation with multiple imputation; SRS-P, stratified random sampling with proportional allocation.

Results for four of the eight different types (proportion of BI-RADS 1 and 2 cases) of SRS-D + MI case sets are shown.

Get Radiology Tree app to read full this article<

Discussion

Get Radiology Tree app to read full this article<

Acknowledgments

Get Radiology Tree app to read full this article<

Appendix

Multiple Imputation

Get Radiology Tree app to read full this article<

f(x1,…,xk;n,q1,…,qk)=n!x1!,…,xk!qx11,…,qxkk,∑ki=1xi=n. f

(

…

;

…

)

…

∑

Get Radiology Tree app to read full this article<

p(q1,…,qk|x1,…,xk)∝f(x1,…,xk;n,q1,…,qk)p(q1,…,qk), p

(

…

)

∝

(

…

;

…

)

(

…

)

where p(q1,…,qk) p

(

…

) is the prior distribution.

Get Radiology Tree app to read full this article<

g(q1,…,qk−1;α1,…,αk)=1B(α)∏ki=1qαi−1i, g

(

…

−

;

…

)

(

)

∏

−

where {qi>0,i=1,…,k} {

…

} , ∑ki=1qi=1 ∑

1 , B(α)=∏ki=1Γ(αi)/Γ(∑ki=1αi) B

(

)

∏

(

)

(

∑

) , and Γ(⋅) Γ

(

⋅

) is the gamma function.

Get Radiology Tree app to read full this article<

Dirichlet(qi,xi+αi)∝Multinom(xi,qi)Dirichlet(qi,αi). D

(

)

∝

(

)

(

)

Get Radiology Tree app to read full this article<

References

1. Bjurstam N., Bjorneld L., Warwick J., et. al.: The Gothenburg breast screening trial. Cancer 2003; 97: pp. 2387-2396.
2. Shapiro S., Venet W., Strax P., et. al.: Selection, follow-up, and analysis in the Health Insurance Plan Study: a randomized trial with breast cancer screening. Natl Cancer Inst Monogr 1985; 67: pp. 65-74.
3. Tabar L., Fagerberg G., Duffy S.W., et. al.: Update of the Swedish two-county program of mammographic screening for breast cancer. Radiol Clin North Am 1992; 30: pp. 187-210.
4. Obuchowski N.A., Zhou X.H.: Prospective studies of diagnostic test accuracy when disease prevalence is low. Biostatistics 2002; 3: pp. 477-492.
5. Metz C.E.: Fundamental ROC analysis.Beutel J.Kundel H.L.Van Metter R.L.Handbook of medical imaging Vol. 1 Physics and psychophysics.2000.SPIE PressBellingham, WA:pp. 751-769.
6. Wagner R.F., Metz C.E., Campbell G.: Assessment of medical imaging systems and computer aids: a tutorial review. Acad Radiol 2007; 14: pp. 723-748.
7. D’Orsi C.J., Bassett L.W., Berg W.A., et. al.: Breast imaging reporting and data system: ACR BI-RADS-mammography.4 ed.2003.American College of RadiologyReston, VA
8. Metz C.E.: Some practical issues of experimental-design and data-analysis in radiological ROC studies. Invest Radiol 1989; 24: pp. 234-245.
9. Pisano E.D., Gatsonis C., Hendrick E., et. al.: Diagnostic performance of digital versus film mammography for breast-cancer screening. N Engl J Med 2005; 353: pp. 1773-1783.
10. Scheaffer R.L., Mendenhall W., Ott L.: Elementary survey sampling.6th ed.2006.Thomson Brooks/ColeBelmont, CA
11. Little R.J.A., Rubin D.B.: Statistical analysis with missing data.2nd ed.2002.WileyHoboken, NJ
12. Schafer J.L.: Analysis of incomplete multivariate data.2000.Chapman and Hall/CRCNew York, NY
13. Rubin D.B.: Multiple imputation for nonresponse in surveys.2004.Wiley-InterscienceHoboken, NJ
14. Jiang Y., Metz C.E.: BI-RADS data should not be used to estimate ROC curves. Radiology 2010; 256: pp. 29-31.
15. Nishikawa R.M., Edwards A., Schmidt R.A., et. al.: Can radiologists recognize that a computer has identified cancers that they have overlooked? Proc. SPIE 2006; 614601: pp. 1-8.
16. Dorfman D.D., Berbaum K.S., Metz C.E.: Receiver operating characteristic rating analysis. Generalization to the population of readers and patients with the jackknife method. Invest Radiol 1992; 27: pp. 723-731.
17. Metz C.E., Pan X.: “Proper” binormal ROC curves: theory and maximum-likelihood estimation. J Math Psychol 1999; 43: pp. 1-33.
18. Pesce L.L., Metz C.E.: Reliable and computationally efficient maximum-likelihood estimation of “proper” binormal ROC curves. Acad Radiol 2007; 14: pp. 814-829.
19. Gur D., Bandos A.I., Fuhrman C.R., et. al.: The prevalence effect in a laboratory environment: changing the confidence ratings. Acad Radiol 2007; 14: pp. 49-53.
20. Gur D., Rockette H.E., Warfel T., et. al.: From the laboratory to the clinic: the “prevalence effect”. Acad Radiol 2003; 10: pp. 1324-1326.
21. Gur D., Bandos A.I., Cohen C.S., et. al.: The “laboratory” effect: comparing radiologists’ performance and variability during prospective clinical and laboratory mammography interpretations. Radiology 2008; 249: pp. 47-53.
22. Gur D., Rockette H.E., Armfield D.R., et. al.: Prevalence effect in a laboratory environment. Radiology 2003; 228: pp. 10-14.

Estimating Screening-Mammography Receiver Operating Characteristic (ROC) Curves from Stratified Random Samples of Screening Mammograms

Rationale and Objectives

Materials and Methods

Results

Conclusions

Materials and methods

Stratified Random Sampling

MI of BI-RADS 1 and 2 Noncancer Cases

Characteristics of Screening-Mammography Case Population

Example Case Sets

Simulation Study and Data Analysis

Results

Discussion

Acknowledgments

Appendix

Multiple Imputation

References

Further Reading

A Pilot Trial to Examine the Effect of High-Dose Niacin on Arterial Wall Inflammation Using Fluorodeoxyglucose Positron Emission Tomography

Automated Tumor Volumetry Using Computer-Aided Image Segmentation

Comparison of Myocardial Perfusion Evaluation with Single Versus Dual-Energy CT and Effect of Beam-Hardening Artifacts