Task-based evaluation of segmentation algorithms for diffusion-weighted MRI without using a gold standard

Abhinav K. Jha, Matthew A Kupinski, Jeffrey J Rodriguez, Renu M. Stephen, Alison T Stopeck

Research output: Contribution to journalArticle

11 Citations (Scopus)

Abstract

In many studies, the estimation of the apparent diffusion coefficient (ADC) of lesions in visceral organs in diffusion-weighted (DW) magnetic resonance images requires an accurate lesion-segmentation algorithm. To evaluate these lesion-segmentation algorithms, region-overlap measures are used currently. However, the end task from the DW images is accurate ADC estimation, and the region-overlap measures do not evaluate the segmentation algorithms on this task. Moreover, these measures rely on the existence of gold-standard segmentation of the lesion, which is typically unavailable. In this paper, we study the problem of task-based evaluation of segmentation algorithms in DW imaging in the absence of a gold standard. We first show that using manual segmentations instead of gold-standard segmentations for this task-based evaluation is unreliable. We then propose a method to compare the segmentation algorithms that does not require gold-standard or manual segmentation results. The no-gold-standard method estimates the bias and the variance of the error between the true ADC values and the ADC values estimated using the automated segmentation algorithm. The method can be used to rank the segmentation algorithms on the basis of both the ensemble mean square error and precision. We also propose consistency checks for this evaluation technique.

Original languageEnglish (US)
Pages (from-to)4425-4446
Number of pages22
JournalPhysics in Medicine and Biology
Volume57
Issue number13
DOIs
StatePublished - Jul 7 2012

Fingerprint

Diffusion Magnetic Resonance Imaging
Gold
Magnetic Resonance Spectroscopy

ASJC Scopus subject areas

  • Radiology Nuclear Medicine and imaging
  • Radiological and Ultrasound Technology

Cite this

Task-based evaluation of segmentation algorithms for diffusion-weighted MRI without using a gold standard. / Jha, Abhinav K.; Kupinski, Matthew A; Rodriguez, Jeffrey J; Stephen, Renu M.; Stopeck, Alison T.

In: Physics in Medicine and Biology, Vol. 57, No. 13, 07.07.2012, p. 4425-4446.

Research output: Contribution to journalArticle

@article{e23473af912949bc84f17ee4a0819973,
title = "Task-based evaluation of segmentation algorithms for diffusion-weighted MRI without using a gold standard",
abstract = "In many studies, the estimation of the apparent diffusion coefficient (ADC) of lesions in visceral organs in diffusion-weighted (DW) magnetic resonance images requires an accurate lesion-segmentation algorithm. To evaluate these lesion-segmentation algorithms, region-overlap measures are used currently. However, the end task from the DW images is accurate ADC estimation, and the region-overlap measures do not evaluate the segmentation algorithms on this task. Moreover, these measures rely on the existence of gold-standard segmentation of the lesion, which is typically unavailable. In this paper, we study the problem of task-based evaluation of segmentation algorithms in DW imaging in the absence of a gold standard. We first show that using manual segmentations instead of gold-standard segmentations for this task-based evaluation is unreliable. We then propose a method to compare the segmentation algorithms that does not require gold-standard or manual segmentation results. The no-gold-standard method estimates the bias and the variance of the error between the true ADC values and the ADC values estimated using the automated segmentation algorithm. The method can be used to rank the segmentation algorithms on the basis of both the ensemble mean square error and precision. We also propose consistency checks for this evaluation technique.",
author = "Jha, {Abhinav K.} and Kupinski, {Matthew A} and Rodriguez, {Jeffrey J} and Stephen, {Renu M.} and Stopeck, {Alison T}",
year = "2012",
month = "7",
day = "7",
doi = "10.1088/0031-9155/57/13/4425",
language = "English (US)",
volume = "57",
pages = "4425--4446",
journal = "Physics in Medicine and Biology",
issn = "0031-9155",
publisher = "IOP Publishing Ltd.",
number = "13",

}

TY - JOUR

T1 - Task-based evaluation of segmentation algorithms for diffusion-weighted MRI without using a gold standard

AU - Jha, Abhinav K.

AU - Kupinski, Matthew A

AU - Rodriguez, Jeffrey J

AU - Stephen, Renu M.

AU - Stopeck, Alison T

PY - 2012/7/7

Y1 - 2012/7/7

N2 - In many studies, the estimation of the apparent diffusion coefficient (ADC) of lesions in visceral organs in diffusion-weighted (DW) magnetic resonance images requires an accurate lesion-segmentation algorithm. To evaluate these lesion-segmentation algorithms, region-overlap measures are used currently. However, the end task from the DW images is accurate ADC estimation, and the region-overlap measures do not evaluate the segmentation algorithms on this task. Moreover, these measures rely on the existence of gold-standard segmentation of the lesion, which is typically unavailable. In this paper, we study the problem of task-based evaluation of segmentation algorithms in DW imaging in the absence of a gold standard. We first show that using manual segmentations instead of gold-standard segmentations for this task-based evaluation is unreliable. We then propose a method to compare the segmentation algorithms that does not require gold-standard or manual segmentation results. The no-gold-standard method estimates the bias and the variance of the error between the true ADC values and the ADC values estimated using the automated segmentation algorithm. The method can be used to rank the segmentation algorithms on the basis of both the ensemble mean square error and precision. We also propose consistency checks for this evaluation technique.

AB - In many studies, the estimation of the apparent diffusion coefficient (ADC) of lesions in visceral organs in diffusion-weighted (DW) magnetic resonance images requires an accurate lesion-segmentation algorithm. To evaluate these lesion-segmentation algorithms, region-overlap measures are used currently. However, the end task from the DW images is accurate ADC estimation, and the region-overlap measures do not evaluate the segmentation algorithms on this task. Moreover, these measures rely on the existence of gold-standard segmentation of the lesion, which is typically unavailable. In this paper, we study the problem of task-based evaluation of segmentation algorithms in DW imaging in the absence of a gold standard. We first show that using manual segmentations instead of gold-standard segmentations for this task-based evaluation is unreliable. We then propose a method to compare the segmentation algorithms that does not require gold-standard or manual segmentation results. The no-gold-standard method estimates the bias and the variance of the error between the true ADC values and the ADC values estimated using the automated segmentation algorithm. The method can be used to rank the segmentation algorithms on the basis of both the ensemble mean square error and precision. We also propose consistency checks for this evaluation technique.

UR - http://www.scopus.com/inward/record.url?scp=84862734500&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84862734500&partnerID=8YFLogxK

U2 - 10.1088/0031-9155/57/13/4425

DO - 10.1088/0031-9155/57/13/4425

M3 - Article

C2 - 22713231

AN - SCOPUS:84862734500

VL - 57

SP - 4425

EP - 4446

JO - Physics in Medicine and Biology

JF - Physics in Medicine and Biology

SN - 0031-9155

IS - 13

ER -