Rationale and objectives

The Dorfman-Berbaum-Metz (DBM) method has been one of the most popular methods for analyzing multireader receiver-operating characteristic (ROC) studies since it was proposed in 1992. Despite its popularity, the original procedure has several drawbacks: it is limited to jackknife accuracy estimates, it is substantially conservative, and it is not based on a satisfactory conceptual or theoretical model. Recently, solutions to these problems have been presented in three papers. Our purpose is to summarize and provide an overview of these recent developments.

Materials and Methods

We present and discuss the recently proposed solutions for the various drawbacks of the original DBM method.

Results

We compare the solutions in a simulation study and find that they result in improved performance for the DBM procedure. We also compare the solutions using two real data studies and find that the modified DBM procedure that incorporates these solutions yields more significant results and clearer interpretations of the variance component parameters than the original DBM procedure.

Conclusions

We recommend using the modified DBM procedure that incorporates the recent developments.

There are several different statistical methods for analyzing multireader receiver-operating characteristic (ROC) studies, with the Dorfman-Berbaum-Metz (DBM) method ( ) being one of the most frequently used methods. The DBM method involves an analysis of variance (ANOVA) of pseudovalues computed with the Quenouille-Tukey jackknife ( ). The basic data for the analysis are pseudovalues corresponding to test-reader ROC accuracy measures, such as the area under the ROC curve (AUC), computed by jackknifing cases separately for each test-reader combination. Throughout we use the term test to refer to a diagnostic test, modality, or treatment. A mixed-effects ANOVA is performed on the pseudovalues to test the null hypothesis that the average accuracy of readers is the same for all of the diagnostic tests studied. Accuracy can be characterized using any accuracy measure, such as sensitivity, specificity, area under the ROC curve, partial area under the ROC curve, sensitivity at a fixed specificity, or specificity at a fixed sensitivity. Furthermore, these measures of accuracy can be estimated parametrically, semi-parametrically or nonparametrically; the DBM method accuracy estimates are the corresponding jackknife estimates.

Although the DBM method may be the most frequently used analysis method for multireader ROC studies since it was proposed in 1992, having been used in over 100 published studies ( ), the original procedure has several drawbacks: it requires that the analysis be based on jackknife accuracy estimates, it is substantially conservative, and it is not based on a satisfactory conceptual or theoretical model. Recently, solutions to these problems were presented in three papers ( ). We summarize these recent developments and compare the solutions in a simulation study and in two examples.

Materials and methods

Original DBM Method

Get Radiology Tree app to read full this article<

Yijk=cˆij−(c−1)ˆij(k) Y

−

(

−

)

(

)

where c denotes the number of cases, ˆij

j denotes the AUC estimate based on all of the data for the i th test and j th reader, and ˆij(k)

(

) denotes the AUC estimate based on the same data but with data for the k th case removed. Thus, in effect, Y__ijk represents the contribution of the k th case to the accuracy estimate for the i th test and j th reader, ˆij

j . Then using the Y__ijk as the data to be evaluated by conventional statistical analysis, the DBM procedure tests for a test effect using a fully crossed three-factor ANOVA with test treated as a fixed factor and reader and case as random factors. A “jackknife estimate” of AUC for the i th test and j th reader is given by the mean of the corresponding pseudovalues:

Y¯¯¯ij⋅=1c∑ck=1Yijk. Y

⋅

∑

We refer to ˆij

j as the original AUC estimate, Y¯¯¯ij⋅ Y

⋅ as the jackknife AUC estimate, and the Y__ijk as the raw pseudovalues .

Get Radiology Tree app to read full this article<

Yijk=μ+τi+Rj+Ck+(τR)ij+(τC)ik+(RC)jk+(τRC)ijk+εijk, Y

(

)

(

)

(

)

(

)

i=1,…,t;j=1,…,r;k=1,…,c; i

…

;

…

;

…

; where τ i denotes the fixed effect of test i , R__j denotes the random effect of reader j , C__k denotes the random effect of case k , the multiple symbols in parentheses denote interactions, and ε ijk is the error term. The interaction terms are all random effects. The random effects are assumed to be mutually independent and normally distributed with zero means and respective variances σ R 2 , σ C 2 , σ τ R 2 , σ τ C 2 , σ RC 2 , σ τ RC 2 and σ ε 2 . Because there are no replications, σ τ RC 2 and σ ε 2 are inseparable. The DBM F statistic for testing for a test effect is the conventional mixed-model ANOVA F statistic based on the pseudovalues. Letting MS(T), MS(T*R), MS(T*C), and MS(T*R*C) denote the mean squares corresponding to the test, test × reader, test × case, and test × reader × case effects, respectively, the F statistic for testing for a test effect for the model is given by

F=MS(T)MS(T*R)+MS(T*C)−MS(T*R*C) F

(

)

(

T*R

)

(

T*C

)

−

(

T*R*C

)

Under the null hypothesis of no test effect, F has an approximate F df 1 , df 2 distribution, where df 1 = t − 1 and df 2 is the Satterthwaite ( ) degrees of freedom approximation given by

df2=[MS(T*R)+MS(T*C)−MS(T*R*C)]2MS(T*R)2(t−1)(r−1)+MS(T*C)2(t−1)(c−1)+MS(T*R*C)2(t−1)(r−1)(c−1). df

[

(

T*R

)

(

T*C

)

−

(

T*R*C

)

]

(

T*R

)

(

−

)

(

−

)

(

T*C

)

(

−

)

(

−

)

(

T*R*C

)

(

−

)

(

−

)

(

−

)

Get Radiology Tree app to read full this article<

σˆ2τR=1c[MS(T*R)−MS(T*R*C)]σˆ2τC=1r[MS(T*C)−MS(T*R*C)]. σ

[

(

T*R

)

−

(

T*R*C

)

]

[

(

T*C

)

−

(

T*R*C

)

]

Taking into account possible model simplification, the F statistic and ddf for the original DBM method are given by

Forig=⎧⎩⎨⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪MS(T)MS(T*R)+MS(T*C)−MS(T*R*C)MS(T)/MS(T*R)MS(T)/MS(T*C)MS(T)/MS(T*R*C) F

orig

{

(

)

(

T*R

)

(

T*C

)

−

(

T*R*C

)

(

)

(

T*R

)

(

)

(

T*C

)

(

)

(

T*R*C

)

σˆ2τR>0,σˆ2τR>0,σˆ2τR≤0,σˆ2τR≤0,σˆ2τCσˆ2τCσˆ2τCσˆ2τC>0≤0>0≤0 σ

⁢

≤

⁢

≤

⁢

≤

and

ddforig=⎧⎩⎨⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪Equation4(t−1)(r−1)(t−1)(c−1)(t−1)(r−1)(c−1)σˆ2τR>0,σˆ2τC>0σˆ2τR>0,σˆ2τC≤0σˆ2τR≤0,σˆ2τC>0σˆ2τR≤0,σˆ2τC≤0. ddf

orig

{

Equation

(

−

)

(

−

)

≤

(

−

)

(

−

)

≤

(

−

)

(

−

)

(

−

)

≤

The numerator degrees of freedom for F in Equation 6 is t − 1. We refer to this approach, using F orig and ddf orig , as original DBM . Note that the conditions in Equations 6 and 7 can also be written in terms of the mean squares; eg, σˆ2τR>0,σˆ2τC>0 σ

0 is equivalent to MS(T*R) > MS(T*R*C), MS(T*C) > MS(T*R*C).

Get Radiology Tree app to read full this article<

FDBM=⎧⎩⎨MS(T)MS(T*R)+MS(T*C)−MS(T*R*C)MS(T)/MS(T*R)σˆ2τC>0σˆ2τC≤0. F

DBM

{

(

)

(

T*R

)

(

T*C

)

−

(

T*R*C

)

(

)

(

T*R

)

≤

Because σˆ2τC≤0 σ

≤

0 is equivalent to MS(T*C) − MS(T*R*C) ≤ 0, this F statistic can be succinctly written in the following form that takes model simplification into account:

FDBM=MS(T)MS(T*R)+max[MS(T*C)−MS(T*R*C),0]. F

DBM

(

)

(

T*R

)

max

⁡

[

(

T*C

)

−

(

T*R*C

)

]

The corresponding conventional ANOVA ddf is given by

ddfD={Equation4(t−1)(r−1)σˆ2τC>0σˆ2τC≤0. ddf

{

Equation

(

−

)

(

−

)

≤

Thus, new model simplification uses F DBM and ddf D .

Get Radiology Tree app to read full this article<

ddfH=⎧⎩⎨⎪⎪{MS(T*R)+MS(T*C)−MS(T*R*C)}2MS(T*R)2/[(t−1)(r−1)](t−1)(r−1)σˆ2τC>0σˆ2τC≤0 ddf

{

(

T*R

)

(

T*C

)

−

(

T*R*C

)

}

(

T*R

)

[

(

−

)

(

−

)

]

(

−

)

(

−

)

≤

Equation 11 can be written more compactly in the form

ddfH={MS(T*R)+max[MS(T*C)−MS(T*R*C),0]}2MS(T*R)2(t−1)(r−1). ddf

{

(

T*R

)

max

⁡

[

(

T*C

)

−

(

T*R*C

)

]

}

(

T*R

)

(

−

)

(

−

)

The quantity ddf H is derived by assuming that new model simplification is used—that is, it is to be used with F DBM (Eq. 9 ). We refer to this approach, using F DBM and ddf H , as new model simplification plus ddf H .

Get Radiology Tree app to read full this article<

Table 1

Summary of the Different DBM Approaches

a) Original DBM F orig ddf orig ConditionMS(T)MS(T*R)+MS(T*C)−MS(T*R*C) MS

(

)

(

T*R

)

(

T*C

)

−

(

T*R*C

) Equation 4 σˆ2τR>0,σˆ2τC>0 σ

0 MS(T)/MS(T*R) ( t − 1) ( r − 1)σˆ2τR>0,σˆ2τC≤0 σ

≤

0 MS(T)/MS(T*C) ( t − 1) ( c − 1)σˆ2τR≤0,σˆ2τC>0 σ

≤

0 MS(T)/MS(T*R*C) ( t − 1) ( r − 1) ( c − 1)σˆ2τR≤0,σˆ2τC≤0 σ

≤

0 b) New model simplificationFDBM=MS(T)MS(T*R)+max[MS(T*C)−MS(T*R*C),0] F

DBM

(

)

(

T*R

)

max

⁡

[

(

T*C

)

−

(

T*R*C

)

] ddfD={Equation4σˆ2τC>0(t−1)(r−1)σˆ2τC≤0 ddf

{

Equation

(

−

)

(

−

)

≤

0 c) New model simplification plus ddf H FDBM=MS(T)MS(T*R)+max[MS(T*C)−MS(T*R*C),0] F

DBM

(

)

(

T*R

)

max

⁡

[

(

T*C

)

−

(

T*R*C

)

] (same as in (b))ddfH={MS(T*R)+max[MS(T*C)−MS(T*R*C),0]}2MS(T*R)2(t−1)(r−1) ddf

{

(

T*R

)

max

⁡

[

(

T*C

)

−

(

T*R*C

)

]

}

(

T*R

)

(

−

)

(

−

)

These approaches can be used with raw, normalized, or quasi pseudovalues. See Table 6 for computational formulas for σˆ2τR σ

τR

2 and σˆ2τC σ

2 .

Table 2

Relationships Between the DBM F Statistics and Between the DBM Denominator Degrees of Freedom

σ2τR σ

2 σ2τC σ

2 F Relationship Ddf Relationship >0 >0Forig=FDBM F

orig

DBM ddforig=ddfD<ddfH ddf

orig

ddf

H ≤0 >0Forig≤FDBM F

orig

≤

DBM (equality iff σ2τR=0 σ

0 )ddfD<ddforig,ddfD<ddfH ddf

ddf

orig

ddf

0 ≤0Forig=FDBM F

orig

DBM ddforig=ddfD=ddfH ddf

orig

ddf

H ≤0 ≤0Forig≤FDBM F

orig

≤

DBM (equality iff σ2τR=0 σ

0 )ddfD=ddfH<ddforig ddf

ddf

orig

These relationships are derived in Appendix A . Iff: if and only if.

Get Radiology Tree app to read full this article<

ˆij=μ˜+τ˜i+Rj+(τR)ij+εij,

(

)

i=1,…,t;j=1,…,r; i

…

;

…

; where ˆij

j is the AUC estimate (or other accuracy estimate) for the i th test and j th reader, τ˜i τ

i denotes the fixed effect of test i, R__j denotes the random effect of reader j , (τR)ij (

)

j denotes the random test × reader interaction, and ε ij is the error term having mean zero and variance σ˜2ε σ

2 . The random effects R__j and (τR)ij (

)

j are assumed independent and normally distributed with zero means and variances σ˜2R σ

2 and σ˜2τR σ

2 , respectively, and are assumed independent of the ε ij . We use the tilde symbol “∼” to distinguish OR model parameters from analogous DBM model parameters. Because the same cases are read by each reader using each test, the error terms are not assumed to be independent. Instead, equi-covariance of the errors between readers and tests is assumed, resulting in three possible covariances given by

Cov⎛⎝⎜εij,εi′j′⎞⎠⎟=⎧⎩⎨⎪⎪Cov1Cov2Cov3i≠i′,j=j′(different test, same reader)i=i′,j≠j′(same test, different reader)i≠i′,j≠j′(different test, different reader). Cov

(

′

)

{

Cov

≠

′

(

different test, same reader

)

Cov

′

≠

′

(

same test, different reader

)

Cov

≠

′

≠

′

(

different test, different reader

)

Obuchowski and Rockette ( ) suggest the following ordering: Cov 1 ≥ Cov 2 ≥ Cov 3 . Conditional on the reader and test × reader effects (that is, treating readers as fixed), it follows from Equation 13 that Cov 1 , Cov 2 , and Cov 3 are also the corresponding covariances of the AUC estimates; for example, Cov 2 is the covariance between the AUCs for two fixed readers using the same test, whereas Cov 3 is the covariance between the AUCs for two fixed readers using different modalities.

Get Radiology Tree app to read full this article<

FOR=MS(T)ˆijMS(T*R)ˆij+max[r(Covˆ2−Covˆ3),0], F

(

)

(

T*R

)

max

⁡

[

(

Cov

−

Cov

)

]

where MS(T)ˆij MS

(

)

j and MS(T*R)ˆij MS

(

T*R

)

j are the test and test × reader mean squares corresponding to the OR model, and where Covˆ2 Cov

2 and Covˆ3 Cov

3 are covariance estimates; the subscript “ ˆij

j ” is used here to indicate that the mean squares are computed from the AUCs rather than the pseudovalues. The quantities Covˆ2 Cov

2 and Covˆ3 Cov

3 are estimated by averaging corresponding covariance estimates for pairs of AUCs, estimated using covariance estimation methods that treat readers as fixed. For example,

Covˆ2=2tr(r−1)∑ti=1∑j<j’Covˆ(ˆij,ˆij’), Cov

(

−

)

∑

′

Cov

(

′

)

where Covˆ(ˆij,ˆij’) Cov

(

′

) is an estimate of the covariance between AUCs for fixed readers j and j ′ using test i , estimated using a fixed reader method such as bootstrapping or jackknifing.

Get Radiology Tree app to read full this article<

ddfH={MS(T*R)+max[r(Covˆ2−Covˆ3),0]}2MS(T*R)2(t−1)(r−1). ddf

{

(

T*R

)

max

⁡

[

(

Cov

−

Cov

)

]

}

(

T*R

)

(

−

)

(

−

)

Under any of the conditions described above that result in F DBM = F OR , the same value for ddf H is obtained using either Equation 12 or 16 , and there is a one-to-one correspondence between the DBM and OR computed quantities, as shown in Table 4 .

Table 3

Conditions That Result in F DBM = F OR as Defined by Equations 9 and 15

(1) Normalized pseudovalues are used with DBM and σ˜ˆ2ε σ

2 , Covˆ1 Cov

1 , Covˆ2 Cov

2 and Covˆ3 Cov

3 are jackknife variance and covariance estimates.or (2) Raw pseudovalues are used with DBM, σ˜ˆ2ε σ

2 , Covˆ1 Cov

1 , Covˆ2 Cov

2 and Covˆ3 Cov

3 are jackknife variance and covariance estimates, and ˆij

j are jackknife accuracy estimates.or (3) Quasi pseudovalues are used with DBM.

Note: Any one of the above conditions results in F DBM = F OR .

Table 4

Relationship Between DBM and OR Computed Quantities

OR Computed Quantity Equivalent Function of DBM Computed QuantitiesMS(T)ˆij MS(T)

j =1cMS(T) =

MS(T) MS(R)ˆij MS(R)

j =1cMS(R) =

MS(R) MS(T*R)ˆij MS(T*R)

j =1cMS(T*R) =

MS(T*R) σ˜ˆ2ε σ

2 =1trc[MS(C)+(t−1)MS(T*C)+(r−1)MS(R*C)+(t−1)(r−1)MS(T*R*C)] =

[

MS(C)

(

−

)

MS(T*C)

(

−

)

MS(R*C)

(

−

)

(

−

)

MS(T*R*C)] Cov1ˆ Cov

=1trc{MS(C)−MS(TC)+(r−1)[MS(RC)−MS(TRC)]}

{MS(C)

−

MS(T*C)

(

−

)

[MS(R*C)

−

MS(T*R*C)]} Cov2ˆ Cov

=1trc{MS(C)−MS(RC)+(t−1)[MS(TC)−MS(TRC)]}

{

MS(C)

−

MS(R*C)

(

−

)

[

MS(T*C)

−

MS(T*R*C)]} Cov3ˆ Cov

=1trc[MS(C)−MS(TC)−MS(RC)+MS(TRC)]

[

MS(C)

−

MS(T*C)

−

MS(R*C)

MS(T*R*C)] DBM Computed Quantity Equivalent Function of OR Computed Quantities MS(T)=cMS(T)ˆij =

MS(T)

j MS(R)=cMS(R)ˆij =

MS(R)

j MS(T*R)=cMS(T*R)ˆij =

MS(T*R)

j MS(C)=c[σ˜ˆ2ε−(t−1)Covˆ1+(r−1)Covˆ2+(t−1)(r−1)Covˆ3)] =

[

−

(

−

)

Cov

(

−

)

Cov

(

−

)

(

−

)

Cov

)

] MS(T*C)=c[σ˜ˆ2ε−Covˆ1+(r−1)(Covˆ2−Covˆ3)] =

[

−

Cov

(

−

)

(

Cov

−

Cov

)

] MS(R*C)=c[σ˜ˆ2ε+(t−1)Covˆ1−Covˆ2−(t−1)Covˆ3)] =

[

(

−

)

Cov

−

Cov

−

(

−

)

Cov

)

] MS(T*R*C)=c[σ˜ˆ2ε−Covˆ1−Covˆ2+Covˆ3] =

[

−

Cov

−

Cov

]

These relationships assume one of the three conditions given in Table 3 .

Get Radiology Tree app to read full this article<

Table 5

Relationship Between DBM and OR Model Parameters

OR Model Parameter Equivalent Function of DBM Model Parametersμ˜ μ

˜ = μτ˜i τ

i = τ _i_σ˜2R σ

2 = σ R 2 σ˜2τR σ

2 = σ τ R 2 σ˜2ε σ

2 =(σ2C+σ2τC+σ2RC+σ2τRC+σ2ε)/c =

(

)

c Cov 1 =(σ2C+σ2RC)/c =

(

)

c Cov 2 =(σ2C+σ2τc)/c =

(

)

c Cov 3 =σ2C/c =

c DBM Model Parameter Equivalent Function of OR Model Parameters μ=μ˜ =

˜ τ i=τ˜i =

i σ R 2 =σ˜2R =

2 σ τ R 2 =σ˜2τR =

2 σ C 2 =cCov3 =

Cov

3 σ τ C 2 =c(Cov2−Cov3) =

(

Cov

−

Cov

) σ RC 2 =c(Cov1−Cov3) =

(

Cov

−

Cov

) σ τ RC 2 + σ ε 2 =c(σ˜2ε−Cov1−Cov2+Cov3) =

(

−

Cov

−

Cov

)

These relationships assume that the constraints for the OR model parameters are those implied by the DBM model: σ2ε≥Cov1+Cov2−Cov3 σ

≥

Cov

−

Cov

3 , Cov1≥Cov3 Cov

≥

Cov

3 , Cov2≥Cov3 Cov

≥

Cov

3 , and Cov3≥0 Cov

≥

0 . They also assume the same linear constraint for the τi τ

i (eg, Στi=0 Σ

0 ) for both models and that either (1) normalized or quasi pseudovalues are used; or (2) if raw pseudovalues are used, then the OR model outcome is the jackknife accuracy estimate.

Adapted and reprinted, with permission, from Hillis et al. ( ).

Get Radiology Tree app to read full this article<

Table 6

ANOVA Estimates for DBM Variance Components

DBM Model Parameter Estimate σ R 2 1tc[MS(R)−MS(T*R)−MS(R*C)+MS(T*R*C)] 1

[

(

)

−

(

T*R

)

−

(

R*C

)

(

T*R*C

)

] σ C 2 1tr[MS(C)−MS(T*C)−MS(R*C)+MS(T*R*C)] 1

[

(

)

−

(

T*C

)

−

(

R*C

)

(

T*R*C

)

] σ τ R 2 1c[MS(T*R)−MS(T*R*C)] 1

[

(

T*R

)

−

(

T*R*C

)

] σ τ C 2 1r[MS(T*C)−MS(T*R*C)] 1

[

(

T*C

)

−

(

T*R*C

)

] σ RC 2 1t[MS(R*C)−MS(T*R*C)] 1

[

(

R*C

)

−

(

T*R*C

)

] σ τ RC 2 + σ ε 2 MS(T*R*C)

These estimates, except for the last, can be negative.

Get Radiology Tree app to read full this article<

Results

Simulation Study

Get Radiology Tree app to read full this article<

Table 7

Semi-parametric Estimation Results of the Simulation Study for Discrete Rating Data

Type I Error Rates Approach Pseudovalues_N_ Mean Min Max Range SD CI Width Mean Original Raw 144 0.036 0.009 0.063 0.054 0.0124 0.196 Normalized 144 0.036 0.011 0.062 0.052 0.0111 0.188 New Raw 144 0.042 0.011 0.070 0.060 0.0123 4.05E+121 Normalized 144 0.043 0.017 0.067 0.050 0.0108 2.74E+121 New plus ddf H Raw 144 0.049 0.016 0.075 0.060 0.0124 0.192 Normalized 144 0.051 0.025 0.077 0.052 0.0105 0.184

Original, original DBM; New, new model simplification; New plus ddf H , new model simplification plus ddf H ; Min, minimum; Max, maximum; SD, standard deviation; CI width, width of a 95% confidence interval for the difference of the AUC estimates.

Table 8

Nonparametric Estimation Results of the Simulation Study for Discrete Rating Data

Type I Error Rates Approach_N_ Mean Min Max Range SD CI Width Mean Original 144 0.041 0.014 0.069 0.055 0.0098 0.177 New 144 0.046 0.024 0.072 0.049 0.0100 4.55E+121 New plus ddf H 144 0.053 0.029 0.079 0.050 0.0097 0.174

No distinction is made between raw and normalized pseudovalues since the trapezoid estimate is the same for either type of pseudovalues. Original, original DBM; new, new model simplification; new plus ddf H , new model simplification plus ddf H ; min, minimum; max, maximum; SD, standard deviation; CI width, width of a 95% confidence interval for the difference of the AUC estimates.

Get Radiology Tree app to read full this article<

Table 9

DBM Procedure Analyses for Van Dyke et al. (20) Data for Semi-parametric and Corresponding Jackknife AUC Estimates

Test 1 (CINE) 2 (Spin Echo) Reader (j)ˆ1j

j (semiparametric)Y 1 j (jackknife)ˆ2j

j (semiparametric)Y 2 j. (jackknife) 1 0.933 0.947 0.951 0.950 2 0.890 0.909 0.935 0.933 3 0.929 0.929 0.928 0.928 4 0.970 0.981 1.000 0.999 5 0.833 0.836 0.945 0.943ˆ1⋅=.911

⋅

.911 Y1⋅⋅=.920 Y

⋅

.920 ˆ2⋅=.952

⋅

.952 Y2⋅⋅=.951 Y

⋅

.951

ANOVA Table Source ddf Raw Pseudovalue Mean Square Normalized Pseudovalue Mean Square T 1 0.264166 0.468996 R 4 0.315637 0.297310 C 113 0.392538 0.392538 T × R 4 0.112560 0.108062 T × C 113 0.143095 0.143095 R × C 452 0.098771 0.098771 T × R × C 452 0.072068 0.072068

T, tests; R, readers; C, cases.

Raw pseudovalues results:

a) Original DBM: F orig = 1.439, ddf orig = 10.03, p = 0.2579

b) New model simplification: F DBM = 1.439, ddf D = 10.03, p = 0.2579

c) New model simplification plus ddf H : F DBM = 1.439, ddf H = 10.64, p = 0.2563

Normalized pseudovalues results:

a) Original DBM: F orig = 2.619, ddf orig = 10.31, p = 0.1358

b) New model simplification: F DBM = 2.619, ddf D = 10.31, p = 0.1358

c) New model simplification plus ddf H : F DBM = 2.619, ddf H = 10.99, p = 0.1339

Get Radiology Tree app to read full this article<

Table 10

Variance Component Estimates for Van Dyke et al. ( ) Data Based on Normalized Pseudovalues

DBM OR Variance Component Estimate Variance Component Estimate σ R 2 0.000713σ˜2R σ

2 0.000713 σ τ R 2 0.000316σ˜2τR σ

2 0.000316 σ C 2 0.022274 Cov 1 0.000313 σ τ C 2 0.014205 Cov 2 0.000320 σ RC 2 0.013351 Cov 3 0.000195 σ τ RC 2 + σ ε 2 0.072068σ˜2ε σ

2 0.001069

Get Radiology Tree app to read full this article<

Table 11

DBM Procedure Analyses for Franken et al. ( ) Data for ANOVA Table

Source ddf Raw Pseudovalue Mean Square Normalized Pseudovalue Mean Square T 1 0.063574 0.066606 R 3 0.088782 0.097686 C 99 0.547734 0.547734 T×R 3 0.007781 0.007494 T×C 99 0.078071 0.078071 R×C 297 0.127582 0.127582 T×R×C 297 0.083643 0.083643

T, tests; R, readers; C, cases.

Raw pseudovalues results:

a) Original DBM: F orig = 0.760, ddf orig = 297, p = 0.3840

b) New model simplification: F DBM = 8.171, ddf D = 3, p = 0.0647

c) New model simplification plus ddf H : F DBM = 8.171, ddf H = 3, p = 0.0647

Normalized pseudovalues results:

a) Original DBM: F orig = 0.796, ddf orig = 297, p = 0.3729

b) New model simplification: F DBM = 8.888, ddf D = 3, p = 0.0585

c) New model simplification plus ddf H : F DBM = 8.888, ddf H = 3, p = 0.0585

Get Radiology Tree app to read full this article<

Discussion

Get Radiology Tree app to read full this article<

Acknowledgments

Get Radiology Tree app to read full this article<

Appendix A

Get Radiology Tree app to read full this article<

ddfD=(t−1)(r−1)=ddfH. ddf

(

−

)

(

−

)

ddf

Thus, ddf D < ddf H if σˆ2τC>0 σ

0 and ddf D = ddf H if σˆ2τC≤0 σ

≤

0 . These relationships hold regardless of the value of σˆ2τR σ

2 . Now we consider each of the four situations separately for the other relationships.

Get Radiology Tree app to read full this article<

Forig=FDBM=MS(T)MS(T*R)+MS(T*C)−MS(T*R*C) F

orig

DBM

(

)

(

T*R

)

(

T*C

)

−

(

T*R*C

)

ddforig=ddfD=[MS(T*R)+MS(T*C)−MS(T*R*C)]2MS(T*R)2(t−1)(r−1)+MS(T*C)2(t−1)(c−1)+MS(T*R*C)(t−1)(r−1)(c−1). ddf

orig

ddf

[

(

T*R

)

(

T*C

)

−

(

T*R*C

)

]

(

T*R

)

(

−

)

(

−

)

(

T*C

)

(

−

)

(

−

)

(

T*R*C

)

(

−

)

(

−

)

(

−

)

_Situation 2:_σˆ2τR≤0,σˆ2τC>0 σ

≤

0 . From Equation 5 , we have cσˆ2τR=MS(T*R)−MS(T*R*C) c

(

T*R

)

−

(

T*R*C

) . Hence with Forig=FDBM F

M if and only if σˆ2τR=0 σ

0 . Also, That is, ddfD<ddforig ddf

ddf

orig . In the proof we have utilized the relationship MS(T*R)−MS(T*R*C)+MS(T*C)>0 MS

(

T*R

)

−

(

T*R*C

)

(

T*C

)

0 , because from Equation 5 we have MS(T*C)−MS(T*R*C)=rσˆ2τC>0 MS

(

T*C

)

−

(

T*R*C

)

0 .

Get Radiology Tree app to read full this article<

Forig=FDBM=MS(T)MS(T*R) F

orig

DBM

(

)

(

T*R

)

ddforig=ddfD=(t−1)(r−1) ddf

orig

ddf

(

−

)

(

−

)

_Situation 4:_σˆ2τR≤0,σˆ2τC≤0 σ

≤

0 . From Equation 5 , it follows that MS(T*R)≤MS(T*R*C) MS

(

T*R

)

≤

(

T*R*C

) , with equality if and only if σˆ2τR=0 σ

0 . Thus,

Forig=MS(T)MS(T*R*C)≤MS(T)MS(T*R)=FDBM, F

orig

(

)

(

T*R*C

)

≤

(

)

(

T*R

)

DBM

with equality if and only if σˆ2τR=0 σ

0 . Also,

ddforig=(t−1)(r−1)(c−1)>(t−1)(r−1)=ddfDBM. ddf

orig

(

−

)

(

−

)

(

−

)

(

−

)

(

−

)

ddf

DBM

Note that we require the assumption that c > 2 for this last relationship.

Get Radiology Tree app to read full this article<

Appendix B

Get Radiology Tree app to read full this article<

Corr(AUCˆij,AUCˆi’j’)=Cov(AUCˆij,AUCˆi’j’)Var(AUCˆij)Var(AUCˆi’j’)√ Corr

(

AUC

′

)

Cov

(

AUC

′

)

Var

(

AUC

)

Var

(

AUC

′

)

where Cov(AUCˆij,AUCˆi′j′) Cov

(

AUC

′

) is the covariance. To find the covariance and variances, we write AUCˆij AUC

j and AUCˆi′j′ AUC

′

′ as functions of random and fixed effects using the OR model (Eq. ). It follows from well known statistical properties that the variance for each AUC estimate is the sum of the OR model variance components corresponding to the random effects, and Cov(AUCˆij,AUCˆi′j′) Cov

(

AUC

′

) is the sum of the variance components corresponding to the reader or test × reader random effects that the AUC estimates have in common (ie, they have the same subscript values for each AUC estimate), plus the covariance between the error terms.

Get Radiology Tree app to read full this article<

ρBR=Corr(AUCˆij,AUCˆij’)=Cov(AUCˆij,AUCˆij’)Var(AUCˆij)Var(AUCˆij’)√ ρ

Corr

(

AUC

′

)

Cov

(

AUC

′

)

Var

(

AUC

)

Var

(

AUC

′

)

where j ≠ j ′. From Equation 13 , with AUCˆij AUC

j taking the place of ˆij

j , we have

AUCˆij=μ˜+τ˜i+Rj+(τR)ij+εijAUCˆij′=μ˜+τ˜i+Rj′+(τR)ij′+εij′. AUC

(

)

AUC

′

(

)

′

Each AUC estimate has the same variance, equal to the sum of all of the variance components corresponding to the random effects; that is,

Var(AUCijˆ)=Var(AUCij’ˆ)=σ˜2R+σ˜2τR+σ˜2e. Var

(

AUC

)

Var

(

AUC

′

)

Examination of Equation 18 shows that the AUCs do not have any reader or test × reader random effects in common because j ≠ j ′. Thus, the covariance is equal to Cov 2 , the covariance between the error terms for different readers using the same test:

Cov(AUCijˆ,AUCij’ˆ)=Cov2. Cov

(

AUC

′

)

Cov

Get Radiology Tree app to read full this article<

ρBR=Cov2σ˜2R+σ˜2τR+σ˜2e. ρ

Cov

Now we derive the between-reader correlation between AUC estimates for two different readers using the same test, but this time treating readers as fixed . In this case, the correlation is a measure of the association between the deviation of one reader’s AUC estimate from that reader’s underlying AUC, due to case variation and reader error, with the deviation of the other reader’s AUC estimate from that reader’s underlying AUC. In contrast, ρBR ρ

R is a measure of association between deviations of randomly chosen readers’ AUC estimates from the reader population AUC.

Get Radiology Tree app to read full this article<

ρBR|readers=Corr(AUCˆij,AUCˆij’∣∣Ri(τR)ij,(τR)ij’)=Cov(AUCˆij,AUCˆij’∣∣Ri(τR)ij,(τR)ij’)Var(AUCˆij∣∣Ri,(τR)ij,(τR)ij’)Var(AUCˆij’∣∣Ri,(τR)ij,(τR)ij’)√ ρ

readers

Corr

(

AUC

′

(

)

(

)

′

)

Cov

(

AUC

′

(

)

(

)

′

)

Var

(

AUC

(

τR

)

(

)

′

)

Var

(

AUC

′

(

)

(

)

′

)

When we condition on the reader and test × reader random effects, the only random effects in Equations 18 are the error terms. Thus, each AUC has the same variance, equal to σ˜2ε σ

2 :

Var(AUCˆij∣∣∣Ri,(τR)ij,(τR)ij’)= Var(AUCˆij’∣∣∣Ri,(τR)ij,(τR)ij’)=σ˜2ε. Var

(

AUC

(

)

(

)

′

)

= Var

(

AUC

′

(

)

(

)

′

)

Similarly, the covariance is equal to Cov 2 , the covariance between the error terms:

Cov(AUCˆij,AUCˆij’∣∣Ri,(τR)ij,(τR)ij’)=Cov2 Cov

(

′

(

)

(

)

′

)

Cov

It follows from Equations 20, 21, and 22 that

ρBR|readers=Cov2σ˜2ε. ρ

Cov

These correlations can be written in terms of the DBM model parameters using the relationships in Table 5 . For example, since Cov2=(σ2C+σ2τC) Cov

(

) and σ˜2ε=σ2C+σ2τC+σ2RC+σ2tRC+σ2ε σ

2 , where σ C 2 , σ τ C 2 , σ RC 2 , σ τ RC 2 and σ ε 2 denote the DBM model variance components, then ρBR|readers=(σ2C+σ2τC)/(σ2C+σ2τC+σ2RC+σ2τRC+σ2ε) ρ

(

)

(

) in terms of the DBM variance components. This last expression is also given in Equation 4 of Roe and Metz ( ).

Get Radiology Tree app to read full this article<

References

1. Dorfman D.D., Berbaum K.S., Metz C.E.: Receiver operating characteristic rating analysis: Generalization to the population of readers and patients with the jackknife method. Investig Radiol 1992; 27: pp. 723-731.
2. Roe C.A., Metz C.E.: Dorfman-Berbaum-Metz method for statistical analysis of multireader, multimodality receiver operating characteristic data: Validation with computer simulation. Acad Radiol 1997; 4: pp. 298-303.
3. Dorfman D.D., Berbaum K.S., Lenth R.V., Chen Y.F., Donaghy B.A.: Monte Carlo validation of a multireader method for receiver operating characteristic discrete rating data: Factorial experimental design. Acad Radiol 1998; 5: pp. 591-602.
4. Quenoille M.H.: Approximate tests of correlation in time series. J R Stat Soc Ser B 1949; 11: pp. 68-84.
5. Quenoille M.H.: Notes on bias in estimation. Biometrika 1956; 43: pp. 353-360.
6. Tukey J.W.: Bias and confidence in not quite large samples. Ann Math Stat 1958; 29: pp. 614. (abstract)
7. Berbaum K.S.: God, like the devil, is in the details. Acad Radiol 2006; 13: pp. 1311-1316.
8. Hillis S.L., Obuchowski N.A., Schartz K.M., Berbaum K.S.: A comparison of the Dorfman-Berbaum-Metz and Obuchowski-Rockette Methods for receiver operating characteristic (ROC) data. Stat Med 2005; 24: pp. 1579-1607.
9. Hillis S.L., Berbaum K.S.: Monte Carlo validation of the Dorfman-Berbaum-Metz method using normalized pseudovalues and less data-based model simplification. Acad Radiol 2005; 12: pp. 1534-1542.
10. Hillis S.L.: A comparison of denominator degrees of freedom methods for multiple observer ROC analysis. Stat Med 2007; 26: pp. 596-619.
11. Satterthwaite F.E.: Synthesis of variance. Psychometrika 1941; 6: pp. 309-316.
12. Satterthwaite F.E.: An approximate distribution of estimates of variance components. Biometric Bull 1946; 2: pp. 110-114.
13. Hanley J.A., Mcneil B.J.: The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 1982; 143: pp. 29-36.
14. Hillis S.L., Berbaum K.S.: Power estimation for the Dorfman-Berbaum-Metz method. Acad Radiol 2004; 11: pp. 1260-1273.
15. Obuchowski N.A., Rockette H.E.: Hypothesis testing of diagnostic accuracy for multiple readers and multiple tests: An ANOVA approach with dependent observations. Commun Stat Simul Comput 1995; 24: pp. 285-308.
16. Obuchowski N.A.: Multireader, multimodality receiver operating characteristic curve studies: Hypothesis testing and sample size estimation using an analysis of variance approach with dependent observations. Acad Radiol 1995; 2: pp. S22-S29.
17. Dorfman D.D., Alf E.: Maximum likelihood estimation of parameters of signal-detection theory and determination of confidence intervals: Rating method data. J Math Psychol 1969; 6: pp. 487-496.
18. Dorfman D.D.: RSCORE II.Swets J.A.Pickett R.M.Evaluation of Diagnostic Systems: Methods From Signal Detection Theory.1982.Academic PressSan Diego, CA:pp. 212-232.
19. The SAS System for Windows, Version 9.1.2002.SAS InstituteCary, NC
20. Van Dyke CW, White RD, Obuchowski NA, Geisinger MA, Lorig RJ, Meziane MA. Cine MRI in the diagnosis of thoracic aortic dissection. Presented at the 79th RSNA Meeting, Chicago, IL, 1993.
21. Franken E.A., Berbaum K.S., Marley S.M., Smith W.L., Sato Y., Kao S.C., Milam S.G.: Evaluation of a digital workstation for interpreting neonatal examinations: A receiver operating characteristic study. Invest Radiol 1992; 27: pp. 732-737.
22. Berbaum K.S., Schartz K.M., Pesce L.L., Hillis S.L.: DBM MRMC 2.1 (computer software).2006. http://perception.radiology.uiowa.edu
23. Berbaum K.S., Metz C.E., Pesce L.L., Schartz K.M.: DBM MRMC 2.1 User’s Guide (software manual).2006. http://perception.radiology.uiowa.edu
24. Hillis S.L., Schartz K.M., Pesce L.L., Berbaum K.S.: DBM MRMC 2.1 for SAS (computer software).2007. http://perception.radiology.uiowa.edu

Recent Developments in the Dorfman-Berbaum-Metz Procedure for Multireader ROC Study Analysis

Rationale and objectives

Materials and Methods

Results

Conclusions

Materials and methods

Original DBM Method

=1trc{MS(C)−MS(T*C)+(r−1)[MS(R*C)−MS(T*R*C)]}

=1trc{MS(C)−MS(R*C)+(t−1)[MS(T*C)−MS(T*R*C)]}

=1trc[MS(C)−MS(T*C)−MS(R*C)+MS(T*R*C)]

Results

Simulation Study

Discussion

Acknowledgments

Appendix A

Appendix B

References

Further Reading

3D TOF MRA of Intracranial Aneurysms at 1.5 T and 3 T

Abstracts of Funded National Institutes of Health Grants

Assessing Diagnostic Confidence

=1trc{MS(C)−MS(TC)+(r−1)[MS(RC)−MS(TRC)]}

=1trc{MS(C)−MS(RC)+(t−1)[MS(TC)−MS(TRC)]}

=1trc[MS(C)−MS(TC)−MS(RC)+MS(TRC)]