Evaluation of an integrated multi-task machine learning system with humans in the loop

Aaron Steinfeld, S. Rachael Bennett, Kyle Cunningham, Matt Lahut, Pablo Alejandro Quinones, Django Wexler, Dan Siewiorek, Jordan Hayes, Paul Cohen, Julie Fitzgerald, Othar Hansson, Mike Pool, Mark Drummond

Research output: Contribution to conferencePaper

15 Scopus citations

Abstract

Performance of a cognitive personal assistant, RADAR, consisting of multiple machine learning components, natural language processing, and optimization was examined with a test explicitly developed to measure the impact of integrated machine learning when used by a human user in a real world setting. Three conditions (conventional tools, Radar without learning, and Radar with learning) were evaluated in a large-scale, between-subjects study. The study revealed that integrated machine learning does produce a positive impact on overall performance. This paper also discusses how specific machine learning components contributed to human-system performance.

Original languageEnglish (US)
Pages182-188
Number of pages7
StatePublished - Dec 1 2007
Event2007 Performance Metrics for Intelligent Systems Workshop, PerMIS'07 - Gaithersburg, MD, United States
Duration: Aug 28 2007Aug 30 2007

Other

Other2007 Performance Metrics for Intelligent Systems Workshop, PerMIS'07
CountryUnited States
CityGaithersburg, MD
Period8/28/078/30/07

Keywords

  • Evaluation
  • Intelligent systems
  • Machine learning
  • Mixed-initiative assistants

ASJC Scopus subject areas

  • Artificial Intelligence
  • Computer Science Applications
  • Control and Systems Engineering

Fingerprint Dive into the research topics of 'Evaluation of an integrated multi-task machine learning system with humans in the loop'. Together they form a unique fingerprint.

  • Cite this

    Steinfeld, A., Bennett, S. R., Cunningham, K., Lahut, M., Quinones, P. A., Wexler, D., Siewiorek, D., Hayes, J., Cohen, P., Fitzgerald, J., Hansson, O., Pool, M., & Drummond, M. (2007). Evaluation of an integrated multi-task machine learning system with humans in the loop. 182-188. Paper presented at 2007 Performance Metrics for Intelligent Systems Workshop, PerMIS'07, Gaithersburg, MD, United States.