New measures for the evaluation of interactive information retrieval systems: Normalized task completion time and normalized user effectiveness

Jing Cheng, Xiao Hu, Patrick B Heidorn

Research output: Contribution to journalArticle

6 Citations (Scopus)

Abstract

User satisfaction, though difficult to measure, is the main goal of Information Retrieval (IR) systems. In recent years, as Interactive Information Retrieval (IIR) systems have become increasingly popular, user effectiveness also has become critical in evaluating IIR systems. However, existing measures in IR evaluation are not particularly suitable for gauging user satisfaction and user effectiveness. In this paper, we propose two new measures to evaluate IIR systems, the Normalized Task Completion Time (NT) and the Normalized User Effectiveness (NUE). The two measures overcome limitations of existing measures and are efficient to calculate in that they do not need a large pool of search tasks. A user study was conducted to investigate the relationships between the two measures and the user satisfaction and effectiveness of a given IR system. The learning effects described by NT, NUE, and the task completion time were also studied and compared. The results show that NT is strongly correlated with user satisfaction, NUE is a better indicator of system effectiveness than task completion time, and both new measures are superior to task completion time in describing the learning effect of the given IR system.

Original languageEnglish (US)
JournalProceedings of the ASIS Annual Meeting
Volume47
DOIs
StatePublished - Nov 2010

Fingerprint

Information retrieval systems
information retrieval
evaluation
learning success
Gaging
Information retrieval
time
system of indicators

Keywords

  • Experimentation
  • Human factors
  • Measurement

ASJC Scopus subject areas

  • Information Systems
  • Library and Information Sciences

Cite this

@article{0647ea02ab2b4f89acbe331c1fc5a993,
title = "New measures for the evaluation of interactive information retrieval systems: Normalized task completion time and normalized user effectiveness",
abstract = "User satisfaction, though difficult to measure, is the main goal of Information Retrieval (IR) systems. In recent years, as Interactive Information Retrieval (IIR) systems have become increasingly popular, user effectiveness also has become critical in evaluating IIR systems. However, existing measures in IR evaluation are not particularly suitable for gauging user satisfaction and user effectiveness. In this paper, we propose two new measures to evaluate IIR systems, the Normalized Task Completion Time (NT) and the Normalized User Effectiveness (NUE). The two measures overcome limitations of existing measures and are efficient to calculate in that they do not need a large pool of search tasks. A user study was conducted to investigate the relationships between the two measures and the user satisfaction and effectiveness of a given IR system. The learning effects described by NT, NUE, and the task completion time were also studied and compared. The results show that NT is strongly correlated with user satisfaction, NUE is a better indicator of system effectiveness than task completion time, and both new measures are superior to task completion time in describing the learning effect of the given IR system.",
keywords = "Experimentation, Human factors, Measurement",
author = "Jing Cheng and Xiao Hu and Heidorn, {Patrick B}",
year = "2010",
month = "11",
doi = "10.1002/meet.14504701144",
language = "English (US)",
volume = "47",
journal = "Proceedings of the ASIST Annual Meeting",
issn = "0044-7870",
publisher = "Learned Information",

}

TY - JOUR

T1 - New measures for the evaluation of interactive information retrieval systems

T2 - Normalized task completion time and normalized user effectiveness

AU - Cheng, Jing

AU - Hu, Xiao

AU - Heidorn, Patrick B

PY - 2010/11

Y1 - 2010/11

N2 - User satisfaction, though difficult to measure, is the main goal of Information Retrieval (IR) systems. In recent years, as Interactive Information Retrieval (IIR) systems have become increasingly popular, user effectiveness also has become critical in evaluating IIR systems. However, existing measures in IR evaluation are not particularly suitable for gauging user satisfaction and user effectiveness. In this paper, we propose two new measures to evaluate IIR systems, the Normalized Task Completion Time (NT) and the Normalized User Effectiveness (NUE). The two measures overcome limitations of existing measures and are efficient to calculate in that they do not need a large pool of search tasks. A user study was conducted to investigate the relationships between the two measures and the user satisfaction and effectiveness of a given IR system. The learning effects described by NT, NUE, and the task completion time were also studied and compared. The results show that NT is strongly correlated with user satisfaction, NUE is a better indicator of system effectiveness than task completion time, and both new measures are superior to task completion time in describing the learning effect of the given IR system.

AB - User satisfaction, though difficult to measure, is the main goal of Information Retrieval (IR) systems. In recent years, as Interactive Information Retrieval (IIR) systems have become increasingly popular, user effectiveness also has become critical in evaluating IIR systems. However, existing measures in IR evaluation are not particularly suitable for gauging user satisfaction and user effectiveness. In this paper, we propose two new measures to evaluate IIR systems, the Normalized Task Completion Time (NT) and the Normalized User Effectiveness (NUE). The two measures overcome limitations of existing measures and are efficient to calculate in that they do not need a large pool of search tasks. A user study was conducted to investigate the relationships between the two measures and the user satisfaction and effectiveness of a given IR system. The learning effects described by NT, NUE, and the task completion time were also studied and compared. The results show that NT is strongly correlated with user satisfaction, NUE is a better indicator of system effectiveness than task completion time, and both new measures are superior to task completion time in describing the learning effect of the given IR system.

KW - Experimentation

KW - Human factors

KW - Measurement

UR - http://www.scopus.com/inward/record.url?scp=84861422067&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84861422067&partnerID=8YFLogxK

U2 - 10.1002/meet.14504701144

DO - 10.1002/meet.14504701144

M3 - Article

AN - SCOPUS:84861422067

VL - 47

JO - Proceedings of the ASIST Annual Meeting

JF - Proceedings of the ASIST Annual Meeting

SN - 0044-7870

ER -