Evaluation of an integrated multi-task machine learning system with humans in the loop

Aaron Steinfeld, S. Rachael Bennett, Kyle Cunningham, Matt Lahut, Pablo Alejandro Quinones, Django Wexler, Dan Siewiorek, Jordan Hayes, Paul R Cohen, Julie Fitzgerald, Othar Hansson, Mike Pool, Mark Drummond

Research output: Chapter in Book/Report/Conference proceedingConference contribution

15 Citations (Scopus)

Abstract

Performance of a cognitive personal assistant, RADAR, consisting of multiple machine learning components, natural language processing, and optimization was examined with a test explicitly developed to measure the impact of integrated machine learning when used by a human user in a real world setting. Three conditions (conventional tools, Radar without learning, and Radar with learning) were evaluated in a large-scale, between-subjects study. The study revealed that integrated machine learning does produce a positive impact on overall performance. This paper also discusses how specific machine learning components contributed to human-system performance.

Original languageEnglish (US)
Title of host publicationPerformance Metrics for Intelligent Systems (PerMIS) Workshop
Pages182-188
Number of pages7
StatePublished - 2007
Externally publishedYes
Event2007 Performance Metrics for Intelligent Systems Workshop, PerMIS'07 - Gaithersburg, MD, United States
Duration: Aug 28 2007Aug 30 2007

Other

Other2007 Performance Metrics for Intelligent Systems Workshop, PerMIS'07
CountryUnited States
CityGaithersburg, MD
Period8/28/078/30/07

Fingerprint

Learning systems
Radar
Processing

Keywords

  • Evaluation
  • Intelligent systems
  • Machine learning
  • Mixed-initiative assistants

ASJC Scopus subject areas

  • Artificial Intelligence
  • Computer Science Applications
  • Control and Systems Engineering

Cite this

Steinfeld, A., Bennett, S. R., Cunningham, K., Lahut, M., Quinones, P. A., Wexler, D., ... Drummond, M. (2007). Evaluation of an integrated multi-task machine learning system with humans in the loop. In Performance Metrics for Intelligent Systems (PerMIS) Workshop (pp. 182-188)

Evaluation of an integrated multi-task machine learning system with humans in the loop. / Steinfeld, Aaron; Bennett, S. Rachael; Cunningham, Kyle; Lahut, Matt; Quinones, Pablo Alejandro; Wexler, Django; Siewiorek, Dan; Hayes, Jordan; Cohen, Paul R; Fitzgerald, Julie; Hansson, Othar; Pool, Mike; Drummond, Mark.

Performance Metrics for Intelligent Systems (PerMIS) Workshop. 2007. p. 182-188.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Steinfeld, A, Bennett, SR, Cunningham, K, Lahut, M, Quinones, PA, Wexler, D, Siewiorek, D, Hayes, J, Cohen, PR, Fitzgerald, J, Hansson, O, Pool, M & Drummond, M 2007, Evaluation of an integrated multi-task machine learning system with humans in the loop. in Performance Metrics for Intelligent Systems (PerMIS) Workshop. pp. 182-188, 2007 Performance Metrics for Intelligent Systems Workshop, PerMIS'07, Gaithersburg, MD, United States, 8/28/07.
Steinfeld A, Bennett SR, Cunningham K, Lahut M, Quinones PA, Wexler D et al. Evaluation of an integrated multi-task machine learning system with humans in the loop. In Performance Metrics for Intelligent Systems (PerMIS) Workshop. 2007. p. 182-188
Steinfeld, Aaron ; Bennett, S. Rachael ; Cunningham, Kyle ; Lahut, Matt ; Quinones, Pablo Alejandro ; Wexler, Django ; Siewiorek, Dan ; Hayes, Jordan ; Cohen, Paul R ; Fitzgerald, Julie ; Hansson, Othar ; Pool, Mike ; Drummond, Mark. / Evaluation of an integrated multi-task machine learning system with humans in the loop. Performance Metrics for Intelligent Systems (PerMIS) Workshop. 2007. pp. 182-188
@inproceedings{1cf1412b585a4791b3692dd817ac868a,
title = "Evaluation of an integrated multi-task machine learning system with humans in the loop",
abstract = "Performance of a cognitive personal assistant, RADAR, consisting of multiple machine learning components, natural language processing, and optimization was examined with a test explicitly developed to measure the impact of integrated machine learning when used by a human user in a real world setting. Three conditions (conventional tools, Radar without learning, and Radar with learning) were evaluated in a large-scale, between-subjects study. The study revealed that integrated machine learning does produce a positive impact on overall performance. This paper also discusses how specific machine learning components contributed to human-system performance.",
keywords = "Evaluation, Intelligent systems, Machine learning, Mixed-initiative assistants",
author = "Aaron Steinfeld and Bennett, {S. Rachael} and Kyle Cunningham and Matt Lahut and Quinones, {Pablo Alejandro} and Django Wexler and Dan Siewiorek and Jordan Hayes and Cohen, {Paul R} and Julie Fitzgerald and Othar Hansson and Mike Pool and Mark Drummond",
year = "2007",
language = "English (US)",
pages = "182--188",
booktitle = "Performance Metrics for Intelligent Systems (PerMIS) Workshop",

}

TY - GEN

T1 - Evaluation of an integrated multi-task machine learning system with humans in the loop

AU - Steinfeld, Aaron

AU - Bennett, S. Rachael

AU - Cunningham, Kyle

AU - Lahut, Matt

AU - Quinones, Pablo Alejandro

AU - Wexler, Django

AU - Siewiorek, Dan

AU - Hayes, Jordan

AU - Cohen, Paul R

AU - Fitzgerald, Julie

AU - Hansson, Othar

AU - Pool, Mike

AU - Drummond, Mark

PY - 2007

Y1 - 2007

N2 - Performance of a cognitive personal assistant, RADAR, consisting of multiple machine learning components, natural language processing, and optimization was examined with a test explicitly developed to measure the impact of integrated machine learning when used by a human user in a real world setting. Three conditions (conventional tools, Radar without learning, and Radar with learning) were evaluated in a large-scale, between-subjects study. The study revealed that integrated machine learning does produce a positive impact on overall performance. This paper also discusses how specific machine learning components contributed to human-system performance.

AB - Performance of a cognitive personal assistant, RADAR, consisting of multiple machine learning components, natural language processing, and optimization was examined with a test explicitly developed to measure the impact of integrated machine learning when used by a human user in a real world setting. Three conditions (conventional tools, Radar without learning, and Radar with learning) were evaluated in a large-scale, between-subjects study. The study revealed that integrated machine learning does produce a positive impact on overall performance. This paper also discusses how specific machine learning components contributed to human-system performance.

KW - Evaluation

KW - Intelligent systems

KW - Machine learning

KW - Mixed-initiative assistants

UR - http://www.scopus.com/inward/record.url?scp=45349083234&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=45349083234&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:45349083234

SP - 182

EP - 188

BT - Performance Metrics for Intelligent Systems (PerMIS) Workshop

ER -