A Probabilistic Model for the MRMC Method, Part 1: Theoretical Development

Research output: Contribution to journalArticle

22 Citations (Scopus)

Abstract

Rationale and Objectives: Current approaches to receiver operating characteristic (ROC) analysis use the MRMC (multiple-reader, multiple-case) paradigm in which several readers read each case and their ratings (or scores) are used to construct an estimate of the area under the ROC curve or some other ROC-related parameter. Standard practice is to decompose the parameter of interest according to a linear model into terms that depend in various ways on the readers, cases, and modalities. Though the methodologic aspects of MRMC analysis have been studied in detail, the literature on the probabilistic basis of the individual terms is sparse. In particular, few articles state what probability law applies to each term and what underlying assumptions are needed for the assumed independence. When probability distributions are specified for these terms, these distributions are assumed to be Gaussians. Materials and Methods: This article approaches the MRMC problem from a mechanistic perspective. For a single modality, three sources of randomness are included: the images, the reader skill, and the reader uncertainty. The probability law on the reader scores is written in terms of three nested conditional probabilities, and random variables associated with this probability are referred to as triply stochastic. Results: In this article, we present the probabilistic MRMC model and apply this model to the Wilcoxon statistic. The result is a seven-term expansion for the variance of the figure of merit. Conclusion: We relate the terms in this expansion to those in the standard, linear MRMC model. Finally, we use the probabilistic model to derive constraints on the coefficients in the seven-term expansion.

Original languageEnglish (US)
Pages (from-to)1410-1421
Number of pages12
JournalAcademic Radiology
Volume13
Issue number11
DOIs
StatePublished - Nov 2006

Fingerprint

Statistical Models
ROC Curve
Uncertainty
Linear Models

Keywords

  • multiple reader multiple case
  • ROC analysis
  • Wilcoxon statistic

ASJC Scopus subject areas

  • Radiology Nuclear Medicine and imaging

Cite this

A Probabilistic Model for the MRMC Method, Part 1 : Theoretical Development. / Clarkson, Eric W; Kupinski, Matthew A; Barrett, Harrison H.

In: Academic Radiology, Vol. 13, No. 11, 11.2006, p. 1410-1421.

Research output: Contribution to journalArticle

@article{4eda86a80aec4c5eb2648aa604bc26fe,
title = "A Probabilistic Model for the MRMC Method, Part 1: Theoretical Development",
abstract = "Rationale and Objectives: Current approaches to receiver operating characteristic (ROC) analysis use the MRMC (multiple-reader, multiple-case) paradigm in which several readers read each case and their ratings (or scores) are used to construct an estimate of the area under the ROC curve or some other ROC-related parameter. Standard practice is to decompose the parameter of interest according to a linear model into terms that depend in various ways on the readers, cases, and modalities. Though the methodologic aspects of MRMC analysis have been studied in detail, the literature on the probabilistic basis of the individual terms is sparse. In particular, few articles state what probability law applies to each term and what underlying assumptions are needed for the assumed independence. When probability distributions are specified for these terms, these distributions are assumed to be Gaussians. Materials and Methods: This article approaches the MRMC problem from a mechanistic perspective. For a single modality, three sources of randomness are included: the images, the reader skill, and the reader uncertainty. The probability law on the reader scores is written in terms of three nested conditional probabilities, and random variables associated with this probability are referred to as triply stochastic. Results: In this article, we present the probabilistic MRMC model and apply this model to the Wilcoxon statistic. The result is a seven-term expansion for the variance of the figure of merit. Conclusion: We relate the terms in this expansion to those in the standard, linear MRMC model. Finally, we use the probabilistic model to derive constraints on the coefficients in the seven-term expansion.",
keywords = "multiple reader multiple case, ROC analysis, Wilcoxon statistic",
author = "Clarkson, {Eric W} and Kupinski, {Matthew A} and Barrett, {Harrison H}",
year = "2006",
month = "11",
doi = "10.1016/j.acra.2006.07.016",
language = "English (US)",
volume = "13",
pages = "1410--1421",
journal = "Academic Radiology",
issn = "1076-6332",
publisher = "Elsevier USA",
number = "11",

}

TY - JOUR

T1 - A Probabilistic Model for the MRMC Method, Part 1

T2 - Theoretical Development

AU - Clarkson, Eric W

AU - Kupinski, Matthew A

AU - Barrett, Harrison H

PY - 2006/11

Y1 - 2006/11

N2 - Rationale and Objectives: Current approaches to receiver operating characteristic (ROC) analysis use the MRMC (multiple-reader, multiple-case) paradigm in which several readers read each case and their ratings (or scores) are used to construct an estimate of the area under the ROC curve or some other ROC-related parameter. Standard practice is to decompose the parameter of interest according to a linear model into terms that depend in various ways on the readers, cases, and modalities. Though the methodologic aspects of MRMC analysis have been studied in detail, the literature on the probabilistic basis of the individual terms is sparse. In particular, few articles state what probability law applies to each term and what underlying assumptions are needed for the assumed independence. When probability distributions are specified for these terms, these distributions are assumed to be Gaussians. Materials and Methods: This article approaches the MRMC problem from a mechanistic perspective. For a single modality, three sources of randomness are included: the images, the reader skill, and the reader uncertainty. The probability law on the reader scores is written in terms of three nested conditional probabilities, and random variables associated with this probability are referred to as triply stochastic. Results: In this article, we present the probabilistic MRMC model and apply this model to the Wilcoxon statistic. The result is a seven-term expansion for the variance of the figure of merit. Conclusion: We relate the terms in this expansion to those in the standard, linear MRMC model. Finally, we use the probabilistic model to derive constraints on the coefficients in the seven-term expansion.

AB - Rationale and Objectives: Current approaches to receiver operating characteristic (ROC) analysis use the MRMC (multiple-reader, multiple-case) paradigm in which several readers read each case and their ratings (or scores) are used to construct an estimate of the area under the ROC curve or some other ROC-related parameter. Standard practice is to decompose the parameter of interest according to a linear model into terms that depend in various ways on the readers, cases, and modalities. Though the methodologic aspects of MRMC analysis have been studied in detail, the literature on the probabilistic basis of the individual terms is sparse. In particular, few articles state what probability law applies to each term and what underlying assumptions are needed for the assumed independence. When probability distributions are specified for these terms, these distributions are assumed to be Gaussians. Materials and Methods: This article approaches the MRMC problem from a mechanistic perspective. For a single modality, three sources of randomness are included: the images, the reader skill, and the reader uncertainty. The probability law on the reader scores is written in terms of three nested conditional probabilities, and random variables associated with this probability are referred to as triply stochastic. Results: In this article, we present the probabilistic MRMC model and apply this model to the Wilcoxon statistic. The result is a seven-term expansion for the variance of the figure of merit. Conclusion: We relate the terms in this expansion to those in the standard, linear MRMC model. Finally, we use the probabilistic model to derive constraints on the coefficients in the seven-term expansion.

KW - multiple reader multiple case

KW - ROC analysis

KW - Wilcoxon statistic

UR - http://www.scopus.com/inward/record.url?scp=33750372592&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=33750372592&partnerID=8YFLogxK

U2 - 10.1016/j.acra.2006.07.016

DO - 10.1016/j.acra.2006.07.016

M3 - Article

C2 - 17070460

AN - SCOPUS:33750372592

VL - 13

SP - 1410

EP - 1421

JO - Academic Radiology

JF - Academic Radiology

SN - 1076-6332

IS - 11

ER -