### Abstract

Current approaches to ROC analysis use the MRMC (multiple-reader, multiple-case) paradigm in which several readers read each case and their ratings are used to construct an estimate of the area under the ROC curve or some other ROC-related parameter. Standard practice is to decompose the parameter of interest according to a linear model into terms that depend in various ways on the readers, cases and modalities. It is assumed that the terms are statistically independent (or at least uncorrelated). Bootstrap methods are then used to estimate the variance of the estimate and the contributions from the individual terms in the assumed expansion. Though the methodological aspects of MRMC analysis have been studied in detail, the literature on the probabilistic basis of the individual terms is sparse. In particular, few papers state what probability law applies to each term and what underlying assumptions are needed for the assumed independence. This paper approaches the MRMC problem from a mechanistic perspective. For a single modality, three sources of randomness are included: the images, the reader skill and the reader uncertainty. The probability law on the parameter estimate is written in terms of three nested conditional probabilities, and random variables associated with this probability are referred to as triply stochastic. The triply stochastic probability is used to define the overall average of any ROC parameter as well as certain partial averages of utility in MRMC analysis. When this theory is applied to estimates of an ROC parameter for a single modality, it is shown that the variance of the estimate can be written as a sum of three terms, rather than the four that would be expected in MRMC analysis. The usual terms in MRMC expansions do not appear naturally in multiply-stochastic theory. A rigorous MRMC expansion can be constructed by adding and subtracting partial averages to the parameter of interest in a tautological manner. In this approach the parameter is decomposed into a sum of four random uncorrelated, zero-mean random variables, with each term clearly defined in terms of conditional probabilities. When the variance of the expansion is computed, however, numerous subtractions occur, and there is no apparent advantage to computing the variance term by term; the final result is the same as one gets from the triply stochastic decomposition, at least for the Wilcoxon estimator. No other nontrivial MRMC expansion appears to be possible.

Original language | English (US) |
---|---|

Title of host publication | Progress in Biomedical Optics and Imaging - Proceedings of SPIE |

Editors | M.P. Eckstein, Y. Jiang |

Pages | 21-31 |

Number of pages | 11 |

Volume | 5749 |

DOIs | |

State | Published - 2005 |

Event | Medical Imaging 2005 - Image Perception, Observer Performance, and Technology Assessment - San Diego, CA, United States Duration: Feb 15 2005 → Feb 17 2005 |

### Other

Other | Medical Imaging 2005 - Image Perception, Observer Performance, and Technology Assessment |
---|---|

Country | United States |

City | San Diego, CA |

Period | 2/15/05 → 2/17/05 |

### Fingerprint

### Keywords

- Multiple case
- Multiple reader
- Psychophysics
- Receiver operating characteristic

### ASJC Scopus subject areas

- Engineering(all)
- Applied Mathematics
- Computer Science Applications
- Electrical and Electronic Engineering
- Electronic, Optical and Magnetic Materials
- Condensed Matter Physics

### Cite this

*Progress in Biomedical Optics and Imaging - Proceedings of SPIE*(Vol. 5749, pp. 21-31). [04] https://doi.org/10.1117/12.595685

**Probabilistic foundations of the MRMC method.** / Barrett, Harrison H; Kupinski, Matthew A; Clarkson, Eric W.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

*Progress in Biomedical Optics and Imaging - Proceedings of SPIE.*vol. 5749, 04, pp. 21-31, Medical Imaging 2005 - Image Perception, Observer Performance, and Technology Assessment, San Diego, CA, United States, 2/15/05. https://doi.org/10.1117/12.595685

}

TY - GEN

T1 - Probabilistic foundations of the MRMC method

AU - Barrett, Harrison H

AU - Kupinski, Matthew A

AU - Clarkson, Eric W

PY - 2005

Y1 - 2005

N2 - Current approaches to ROC analysis use the MRMC (multiple-reader, multiple-case) paradigm in which several readers read each case and their ratings are used to construct an estimate of the area under the ROC curve or some other ROC-related parameter. Standard practice is to decompose the parameter of interest according to a linear model into terms that depend in various ways on the readers, cases and modalities. It is assumed that the terms are statistically independent (or at least uncorrelated). Bootstrap methods are then used to estimate the variance of the estimate and the contributions from the individual terms in the assumed expansion. Though the methodological aspects of MRMC analysis have been studied in detail, the literature on the probabilistic basis of the individual terms is sparse. In particular, few papers state what probability law applies to each term and what underlying assumptions are needed for the assumed independence. This paper approaches the MRMC problem from a mechanistic perspective. For a single modality, three sources of randomness are included: the images, the reader skill and the reader uncertainty. The probability law on the parameter estimate is written in terms of three nested conditional probabilities, and random variables associated with this probability are referred to as triply stochastic. The triply stochastic probability is used to define the overall average of any ROC parameter as well as certain partial averages of utility in MRMC analysis. When this theory is applied to estimates of an ROC parameter for a single modality, it is shown that the variance of the estimate can be written as a sum of three terms, rather than the four that would be expected in MRMC analysis. The usual terms in MRMC expansions do not appear naturally in multiply-stochastic theory. A rigorous MRMC expansion can be constructed by adding and subtracting partial averages to the parameter of interest in a tautological manner. In this approach the parameter is decomposed into a sum of four random uncorrelated, zero-mean random variables, with each term clearly defined in terms of conditional probabilities. When the variance of the expansion is computed, however, numerous subtractions occur, and there is no apparent advantage to computing the variance term by term; the final result is the same as one gets from the triply stochastic decomposition, at least for the Wilcoxon estimator. No other nontrivial MRMC expansion appears to be possible.

AB - Current approaches to ROC analysis use the MRMC (multiple-reader, multiple-case) paradigm in which several readers read each case and their ratings are used to construct an estimate of the area under the ROC curve or some other ROC-related parameter. Standard practice is to decompose the parameter of interest according to a linear model into terms that depend in various ways on the readers, cases and modalities. It is assumed that the terms are statistically independent (or at least uncorrelated). Bootstrap methods are then used to estimate the variance of the estimate and the contributions from the individual terms in the assumed expansion. Though the methodological aspects of MRMC analysis have been studied in detail, the literature on the probabilistic basis of the individual terms is sparse. In particular, few papers state what probability law applies to each term and what underlying assumptions are needed for the assumed independence. This paper approaches the MRMC problem from a mechanistic perspective. For a single modality, three sources of randomness are included: the images, the reader skill and the reader uncertainty. The probability law on the parameter estimate is written in terms of three nested conditional probabilities, and random variables associated with this probability are referred to as triply stochastic. The triply stochastic probability is used to define the overall average of any ROC parameter as well as certain partial averages of utility in MRMC analysis. When this theory is applied to estimates of an ROC parameter for a single modality, it is shown that the variance of the estimate can be written as a sum of three terms, rather than the four that would be expected in MRMC analysis. The usual terms in MRMC expansions do not appear naturally in multiply-stochastic theory. A rigorous MRMC expansion can be constructed by adding and subtracting partial averages to the parameter of interest in a tautological manner. In this approach the parameter is decomposed into a sum of four random uncorrelated, zero-mean random variables, with each term clearly defined in terms of conditional probabilities. When the variance of the expansion is computed, however, numerous subtractions occur, and there is no apparent advantage to computing the variance term by term; the final result is the same as one gets from the triply stochastic decomposition, at least for the Wilcoxon estimator. No other nontrivial MRMC expansion appears to be possible.

KW - Multiple case

KW - Multiple reader

KW - Psychophysics

KW - Receiver operating characteristic

UR - http://www.scopus.com/inward/record.url?scp=24644470046&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=24644470046&partnerID=8YFLogxK

U2 - 10.1117/12.595685

DO - 10.1117/12.595685

M3 - Conference contribution

AN - SCOPUS:24644470046

VL - 5749

SP - 21

EP - 31

BT - Progress in Biomedical Optics and Imaging - Proceedings of SPIE

A2 - Eckstein, M.P.

A2 - Jiang, Y.

ER -