Evaluation of an integrated multi-task machine learning system with humans in the loop

Aaron Steinfeld; S. Rachael Bennett; Kyle Cunningham; Matt Lahut; Pablo Alejandro Quinones; Django Wexler; Dan Siewiorek; Jordan Hayes; Paul Cohen; Julie Fitzgerald; Othar Hansson; Mike Pool; Mark Drummond

Evaluation of an integrated multi-task machine learning system with humans in the loop

Aaron Steinfeld, S. Rachael Bennett, Kyle Cunningham, Matt Lahut, Pablo Alejandro Quinones, Django Wexler, Dan Siewiorek, Jordan Hayes, Paul Cohen, Julie Fitzgerald, Othar Hansson, Mike Pool, Mark Drummond

Research output: Contribution to conference › Paper › peer-review

15 Scopus citations

Abstract

Performance of a cognitive personal assistant, RADAR, consisting of multiple machine learning components, natural language processing, and optimization was examined with a test explicitly developed to measure the impact of integrated machine learning when used by a human user in a real world setting. Three conditions (conventional tools, Radar without learning, and Radar with learning) were evaluated in a large-scale, between-subjects study. The study revealed that integrated machine learning does produce a positive impact on overall performance. This paper also discusses how specific machine learning components contributed to human-system performance.

Original language	English (US)
Pages	182-188
Number of pages	7
State	Published - 2007
Externally published	Yes
Event	2007 Performance Metrics for Intelligent Systems Workshop, PerMIS'07 - Gaithersburg, MD, United States Duration: Aug 28 2007 → Aug 30 2007

Other

Other	2007 Performance Metrics for Intelligent Systems Workshop, PerMIS'07
Country/Territory	United States
City	Gaithersburg, MD
Period	8/28/07 → 8/30/07

Keywords

Evaluation
Intelligent systems
Machine learning
Mixed-initiative assistants

ASJC Scopus subject areas

Artificial Intelligence
Computer Science Applications
Control and Systems Engineering

Cite this

Steinfeld, A., Bennett, S. R., Cunningham, K., Lahut, M., Quinones, P. A., Wexler, D., Siewiorek, D., Hayes, J., Cohen, P., Fitzgerald, J., Hansson, O., Pool, M., & Drummond, M. (2007). Evaluation of an integrated multi-task machine learning system with humans in the loop. 182-188. Paper presented at 2007 Performance Metrics for Intelligent Systems Workshop, PerMIS'07, Gaithersburg, MD, United States.

Evaluation of an integrated multi-task machine learning system with humans in the loop. / Steinfeld, Aaron; Bennett, S. Rachael; Cunningham, Kyle; Lahut, Matt; Quinones, Pablo Alejandro; Wexler, Django; Siewiorek, Dan; Hayes, Jordan; Cohen, Paul; Fitzgerald, Julie; Hansson, Othar; Pool, Mike; Drummond, Mark.

2007. 182-188 Paper presented at 2007 Performance Metrics for Intelligent Systems Workshop, PerMIS'07, Gaithersburg, MD, United States.

Research output: Contribution to conference › Paper › peer-review

Steinfeld, A, Bennett, SR, Cunningham, K, Lahut, M, Quinones, PA, Wexler, D, Siewiorek, D, Hayes, J, Cohen, P, Fitzgerald, J, Hansson, O, Pool, M & Drummond, M 2007, 'Evaluation of an integrated multi-task machine learning system with humans in the loop', Paper presented at 2007 Performance Metrics for Intelligent Systems Workshop, PerMIS'07, Gaithersburg, MD, United States, 8/28/07 - 8/30/07 pp. 182-188.

Steinfeld, Aaron ; Bennett, S. Rachael ; Cunningham, Kyle ; Lahut, Matt ; Quinones, Pablo Alejandro ; Wexler, Django ; Siewiorek, Dan ; Hayes, Jordan ; Cohen, Paul ; Fitzgerald, Julie ; Hansson, Othar ; Pool, Mike ; Drummond, Mark. / Evaluation of an integrated multi-task machine learning system with humans in the loop. Paper presented at 2007 Performance Metrics for Intelligent Systems Workshop, PerMIS'07, Gaithersburg, MD, United States.7 p.

@conference{1cf1412b585a4791b3692dd817ac868a,

title = "Evaluation of an integrated multi-task machine learning system with humans in the loop",

abstract = "Performance of a cognitive personal assistant, RADAR, consisting of multiple machine learning components, natural language processing, and optimization was examined with a test explicitly developed to measure the impact of integrated machine learning when used by a human user in a real world setting. Three conditions (conventional tools, Radar without learning, and Radar with learning) were evaluated in a large-scale, between-subjects study. The study revealed that integrated machine learning does produce a positive impact on overall performance. This paper also discusses how specific machine learning components contributed to human-system performance.",

keywords = "Evaluation, Intelligent systems, Machine learning, Mixed-initiative assistants",

author = "Aaron Steinfeld and Bennett, {S. Rachael} and Kyle Cunningham and Matt Lahut and Quinones, {Pablo Alejandro} and Django Wexler and Dan Siewiorek and Jordan Hayes and Paul Cohen and Julie Fitzgerald and Othar Hansson and Mike Pool and Mark Drummond",

year = "2007",

language = "English (US)",

pages = "182--188",

note = "2007 Performance Metrics for Intelligent Systems Workshop, PerMIS'07 ; Conference date: 28-08-2007 Through 30-08-2007",

}

TY - CONF

T1 - Evaluation of an integrated multi-task machine learning system with humans in the loop

AU - Steinfeld, Aaron

AU - Bennett, S. Rachael

AU - Cunningham, Kyle

AU - Lahut, Matt

AU - Quinones, Pablo Alejandro

AU - Wexler, Django

AU - Siewiorek, Dan

AU - Hayes, Jordan

AU - Cohen, Paul

AU - Fitzgerald, Julie

AU - Hansson, Othar

AU - Pool, Mike

AU - Drummond, Mark

PY - 2007

Y1 - 2007

N2 - Performance of a cognitive personal assistant, RADAR, consisting of multiple machine learning components, natural language processing, and optimization was examined with a test explicitly developed to measure the impact of integrated machine learning when used by a human user in a real world setting. Three conditions (conventional tools, Radar without learning, and Radar with learning) were evaluated in a large-scale, between-subjects study. The study revealed that integrated machine learning does produce a positive impact on overall performance. This paper also discusses how specific machine learning components contributed to human-system performance.

AB - Performance of a cognitive personal assistant, RADAR, consisting of multiple machine learning components, natural language processing, and optimization was examined with a test explicitly developed to measure the impact of integrated machine learning when used by a human user in a real world setting. Three conditions (conventional tools, Radar without learning, and Radar with learning) were evaluated in a large-scale, between-subjects study. The study revealed that integrated machine learning does produce a positive impact on overall performance. This paper also discusses how specific machine learning components contributed to human-system performance.

KW - Evaluation

KW - Intelligent systems

KW - Machine learning

KW - Mixed-initiative assistants

UR - http://www.scopus.com/inward/record.url?scp=45349083234&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=45349083234&partnerID=8YFLogxK

M3 - Paper

AN - SCOPUS:45349083234

SP - 182

EP - 188

T2 - 2007 Performance Metrics for Intelligent Systems Workshop, PerMIS'07

Y2 - 28 August 2007 through 30 August 2007

ER -

Evaluation of an integrated multi-task machine learning system with humans in the loop

Abstract

Other

Keywords

ASJC Scopus subject areas

Other files and links

Fingerprint

Cite this