Probabilistic foundations of the MRMC method

Research output: Chapter in Book/Report/Conference proceedingConference contribution

19 Citations (Scopus)

Abstract

Current approaches to ROC analysis use the MRMC (multiple-reader, multiple-case) paradigm in which several readers read each case and their ratings are used to construct an estimate of the area under the ROC curve or some other ROC-related parameter. Standard practice is to decompose the parameter of interest according to a linear model into terms that depend in various ways on the readers, cases and modalities. It is assumed that the terms are statistically independent (or at least uncorrelated). Bootstrap methods are then used to estimate the variance of the estimate and the contributions from the individual terms in the assumed expansion. Though the methodological aspects of MRMC analysis have been studied in detail, the literature on the probabilistic basis of the individual terms is sparse. In particular, few papers state what probability law applies to each term and what underlying assumptions are needed for the assumed independence. This paper approaches the MRMC problem from a mechanistic perspective. For a single modality, three sources of randomness are included: the images, the reader skill and the reader uncertainty. The probability law on the parameter estimate is written in terms of three nested conditional probabilities, and random variables associated with this probability are referred to as triply stochastic. The triply stochastic probability is used to define the overall average of any ROC parameter as well as certain partial averages of utility in MRMC analysis. When this theory is applied to estimates of an ROC parameter for a single modality, it is shown that the variance of the estimate can be written as a sum of three terms, rather than the four that would be expected in MRMC analysis. The usual terms in MRMC expansions do not appear naturally in multiply-stochastic theory. A rigorous MRMC expansion can be constructed by adding and subtracting partial averages to the parameter of interest in a tautological manner. In this approach the parameter is decomposed into a sum of four random uncorrelated, zero-mean random variables, with each term clearly defined in terms of conditional probabilities. When the variance of the expansion is computed, however, numerous subtractions occur, and there is no apparent advantage to computing the variance term by term; the final result is the same as one gets from the triply stochastic decomposition, at least for the Wilcoxon estimator. No other nontrivial MRMC expansion appears to be possible.

Original languageEnglish (US)
Title of host publicationProgress in Biomedical Optics and Imaging - Proceedings of SPIE
EditorsM.P. Eckstein, Y. Jiang
Pages21-31
Number of pages11
Volume5749
DOIs
StatePublished - 2005
EventMedical Imaging 2005 - Image Perception, Observer Performance, and Technology Assessment - San Diego, CA, United States
Duration: Feb 15 2005Feb 17 2005

Other

OtherMedical Imaging 2005 - Image Perception, Observer Performance, and Technology Assessment
CountryUnited States
CitySan Diego, CA
Period2/15/052/17/05

Fingerprint

readers
Term
Random variables
Estimate
Modality
estimates
expansion
Conditional probability
random variables
ROC Analysis
Stochastic Decomposition
Associated Random Variables
Partial
Bootstrap Method
Decomposition
Receiver Operating Characteristic Curve
Subtraction
Randomness
ratings
Linear Model

Keywords

  • Multiple case
  • Multiple reader
  • Psychophysics
  • Receiver operating characteristic

ASJC Scopus subject areas

  • Engineering(all)
  • Applied Mathematics
  • Computer Science Applications
  • Electrical and Electronic Engineering
  • Electronic, Optical and Magnetic Materials
  • Condensed Matter Physics

Cite this

Barrett, H. H., Kupinski, M. A., & Clarkson, E. W. (2005). Probabilistic foundations of the MRMC method. In M. P. Eckstein, & Y. Jiang (Eds.), Progress in Biomedical Optics and Imaging - Proceedings of SPIE (Vol. 5749, pp. 21-31). [04] https://doi.org/10.1117/12.595685

Probabilistic foundations of the MRMC method. / Barrett, Harrison H; Kupinski, Matthew A; Clarkson, Eric W.

Progress in Biomedical Optics and Imaging - Proceedings of SPIE. ed. / M.P. Eckstein; Y. Jiang. Vol. 5749 2005. p. 21-31 04.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Barrett, HH, Kupinski, MA & Clarkson, EW 2005, Probabilistic foundations of the MRMC method. in MP Eckstein & Y Jiang (eds), Progress in Biomedical Optics and Imaging - Proceedings of SPIE. vol. 5749, 04, pp. 21-31, Medical Imaging 2005 - Image Perception, Observer Performance, and Technology Assessment, San Diego, CA, United States, 2/15/05. https://doi.org/10.1117/12.595685
Barrett HH, Kupinski MA, Clarkson EW. Probabilistic foundations of the MRMC method. In Eckstein MP, Jiang Y, editors, Progress in Biomedical Optics and Imaging - Proceedings of SPIE. Vol. 5749. 2005. p. 21-31. 04 https://doi.org/10.1117/12.595685
Barrett, Harrison H ; Kupinski, Matthew A ; Clarkson, Eric W. / Probabilistic foundations of the MRMC method. Progress in Biomedical Optics and Imaging - Proceedings of SPIE. editor / M.P. Eckstein ; Y. Jiang. Vol. 5749 2005. pp. 21-31
@inproceedings{76354daf00a34d21b0920bc608dcad53,
title = "Probabilistic foundations of the MRMC method",
abstract = "Current approaches to ROC analysis use the MRMC (multiple-reader, multiple-case) paradigm in which several readers read each case and their ratings are used to construct an estimate of the area under the ROC curve or some other ROC-related parameter. Standard practice is to decompose the parameter of interest according to a linear model into terms that depend in various ways on the readers, cases and modalities. It is assumed that the terms are statistically independent (or at least uncorrelated). Bootstrap methods are then used to estimate the variance of the estimate and the contributions from the individual terms in the assumed expansion. Though the methodological aspects of MRMC analysis have been studied in detail, the literature on the probabilistic basis of the individual terms is sparse. In particular, few papers state what probability law applies to each term and what underlying assumptions are needed for the assumed independence. This paper approaches the MRMC problem from a mechanistic perspective. For a single modality, three sources of randomness are included: the images, the reader skill and the reader uncertainty. The probability law on the parameter estimate is written in terms of three nested conditional probabilities, and random variables associated with this probability are referred to as triply stochastic. The triply stochastic probability is used to define the overall average of any ROC parameter as well as certain partial averages of utility in MRMC analysis. When this theory is applied to estimates of an ROC parameter for a single modality, it is shown that the variance of the estimate can be written as a sum of three terms, rather than the four that would be expected in MRMC analysis. The usual terms in MRMC expansions do not appear naturally in multiply-stochastic theory. A rigorous MRMC expansion can be constructed by adding and subtracting partial averages to the parameter of interest in a tautological manner. In this approach the parameter is decomposed into a sum of four random uncorrelated, zero-mean random variables, with each term clearly defined in terms of conditional probabilities. When the variance of the expansion is computed, however, numerous subtractions occur, and there is no apparent advantage to computing the variance term by term; the final result is the same as one gets from the triply stochastic decomposition, at least for the Wilcoxon estimator. No other nontrivial MRMC expansion appears to be possible.",
keywords = "Multiple case, Multiple reader, Psychophysics, Receiver operating characteristic",
author = "Barrett, {Harrison H} and Kupinski, {Matthew A} and Clarkson, {Eric W}",
year = "2005",
doi = "10.1117/12.595685",
language = "English (US)",
volume = "5749",
pages = "21--31",
editor = "M.P. Eckstein and Y. Jiang",
booktitle = "Progress in Biomedical Optics and Imaging - Proceedings of SPIE",

}

TY - GEN

T1 - Probabilistic foundations of the MRMC method

AU - Barrett, Harrison H

AU - Kupinski, Matthew A

AU - Clarkson, Eric W

PY - 2005

Y1 - 2005

N2 - Current approaches to ROC analysis use the MRMC (multiple-reader, multiple-case) paradigm in which several readers read each case and their ratings are used to construct an estimate of the area under the ROC curve or some other ROC-related parameter. Standard practice is to decompose the parameter of interest according to a linear model into terms that depend in various ways on the readers, cases and modalities. It is assumed that the terms are statistically independent (or at least uncorrelated). Bootstrap methods are then used to estimate the variance of the estimate and the contributions from the individual terms in the assumed expansion. Though the methodological aspects of MRMC analysis have been studied in detail, the literature on the probabilistic basis of the individual terms is sparse. In particular, few papers state what probability law applies to each term and what underlying assumptions are needed for the assumed independence. This paper approaches the MRMC problem from a mechanistic perspective. For a single modality, three sources of randomness are included: the images, the reader skill and the reader uncertainty. The probability law on the parameter estimate is written in terms of three nested conditional probabilities, and random variables associated with this probability are referred to as triply stochastic. The triply stochastic probability is used to define the overall average of any ROC parameter as well as certain partial averages of utility in MRMC analysis. When this theory is applied to estimates of an ROC parameter for a single modality, it is shown that the variance of the estimate can be written as a sum of three terms, rather than the four that would be expected in MRMC analysis. The usual terms in MRMC expansions do not appear naturally in multiply-stochastic theory. A rigorous MRMC expansion can be constructed by adding and subtracting partial averages to the parameter of interest in a tautological manner. In this approach the parameter is decomposed into a sum of four random uncorrelated, zero-mean random variables, with each term clearly defined in terms of conditional probabilities. When the variance of the expansion is computed, however, numerous subtractions occur, and there is no apparent advantage to computing the variance term by term; the final result is the same as one gets from the triply stochastic decomposition, at least for the Wilcoxon estimator. No other nontrivial MRMC expansion appears to be possible.

AB - Current approaches to ROC analysis use the MRMC (multiple-reader, multiple-case) paradigm in which several readers read each case and their ratings are used to construct an estimate of the area under the ROC curve or some other ROC-related parameter. Standard practice is to decompose the parameter of interest according to a linear model into terms that depend in various ways on the readers, cases and modalities. It is assumed that the terms are statistically independent (or at least uncorrelated). Bootstrap methods are then used to estimate the variance of the estimate and the contributions from the individual terms in the assumed expansion. Though the methodological aspects of MRMC analysis have been studied in detail, the literature on the probabilistic basis of the individual terms is sparse. In particular, few papers state what probability law applies to each term and what underlying assumptions are needed for the assumed independence. This paper approaches the MRMC problem from a mechanistic perspective. For a single modality, three sources of randomness are included: the images, the reader skill and the reader uncertainty. The probability law on the parameter estimate is written in terms of three nested conditional probabilities, and random variables associated with this probability are referred to as triply stochastic. The triply stochastic probability is used to define the overall average of any ROC parameter as well as certain partial averages of utility in MRMC analysis. When this theory is applied to estimates of an ROC parameter for a single modality, it is shown that the variance of the estimate can be written as a sum of three terms, rather than the four that would be expected in MRMC analysis. The usual terms in MRMC expansions do not appear naturally in multiply-stochastic theory. A rigorous MRMC expansion can be constructed by adding and subtracting partial averages to the parameter of interest in a tautological manner. In this approach the parameter is decomposed into a sum of four random uncorrelated, zero-mean random variables, with each term clearly defined in terms of conditional probabilities. When the variance of the expansion is computed, however, numerous subtractions occur, and there is no apparent advantage to computing the variance term by term; the final result is the same as one gets from the triply stochastic decomposition, at least for the Wilcoxon estimator. No other nontrivial MRMC expansion appears to be possible.

KW - Multiple case

KW - Multiple reader

KW - Psychophysics

KW - Receiver operating characteristic

UR - http://www.scopus.com/inward/record.url?scp=24644470046&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=24644470046&partnerID=8YFLogxK

U2 - 10.1117/12.595685

DO - 10.1117/12.595685

M3 - Conference contribution

AN - SCOPUS:24644470046

VL - 5749

SP - 21

EP - 31

BT - Progress in Biomedical Optics and Imaging - Proceedings of SPIE

A2 - Eckstein, M.P.

A2 - Jiang, Y.

ER -