Interpreting the ROC curve from the radiologist's point of view: The "Diagnostician Operating Choice" (DOC) curve

C. A. Beam, E. F. Conant, Elizabeth A Krupinski, H. L. Kundel, E. A. Sickles

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

We introduce an interesting interpretation of the ROC Curve that, subsequently, opens a new research paradigm. We define the "Diagnostician Operating Choice" (DOC) Curve to be the set of all (True Positive Probability/True Negative Probability) or ("skill in diseased population"/"skill in non-diseased population" when considered from the diagnostician's perspective) options made available to a particular radiologist when interpreting a particular diagnostic technology. The DOC Curve is, thus, the choice set presented to the diagnostician by their interaction with the technology. This new paradigm calls for tools that can measure the particular choice set of any particular individual radiologist interpreting a particular technology when applied in a particular clinical setting. Fundamental requirements for this paradigm are for the DOC Curve to be unique to individuals and constant across similar experimental conditions. To investigate constancy, we analyzed data from a reading study of 10 radiologists. Each radiologist interpreted the same set of 148 screening mammograms twice using a modified version of BI-RADS. ROC Curves for each radiologist were computed and compared between the two reading occasions with the CORROC2 program. None of the areas were statistically significantly different (p<0.05), providing confirmation (but not proof) of constancy across the two reading conditions. The DOC Curve paradigm suggests new areas of research focusing on the behavior in individuals interacting with technology. A clear need is for more efficient estimation of individual DOC Curves based on limited case sets. Paradoxically, the answer to this last problem might lie in using large population-based ("MRMC") studies to develop highly efficient and externally validated standardized testing tools for assessment of the individual.

Original languageEnglish (US)
Title of host publicationProgress in Biomedical Optics and Imaging - Proceedings of SPIE
EditorsM.P. Eckstein, Y. Jiang
Pages8-14
Number of pages7
Volume5749
DOIs
Publication statusPublished - 2005
EventMedical Imaging 2005 - Image Perception, Observer Performance, and Technology Assessment - San Diego, CA, United States
Duration: Feb 15 2005Feb 17 2005

Other

OtherMedical Imaging 2005 - Image Perception, Observer Performance, and Technology Assessment
CountryUnited States
CitySan Diego, CA
Period2/15/052/17/05

    Fingerprint

Keywords

  • Diagnosis
  • Observer Variability
  • ROC Curve

ASJC Scopus subject areas

  • Engineering(all)
  • Applied Mathematics
  • Computer Science Applications
  • Electrical and Electronic Engineering
  • Electronic, Optical and Magnetic Materials
  • Condensed Matter Physics

Cite this

Beam, C. A., Conant, E. F., Krupinski, E. A., Kundel, H. L., & Sickles, E. A. (2005). Interpreting the ROC curve from the radiologist's point of view: The "Diagnostician Operating Choice" (DOC) curve. In M. P. Eckstein, & Y. Jiang (Eds.), Progress in Biomedical Optics and Imaging - Proceedings of SPIE (Vol. 5749, pp. 8-14). [02] https://doi.org/10.1117/12.595198