Abstract
Performance of a cognitive personal assistant, RADAR, consisting of multiple machine learning components, natural language processing, and optimization was examined with a test explicitly developed to measure the impact of integrated machine learning when used by a human user in a real world setting. Three conditions (conventional tools, Radar without learning, and Radar with learning) were evaluated in a large-scale, between-subjects study. The study revealed that integrated machine learning does produce a positive impact on overall performance. This paper also discusses how specific machine learning components contributed to human-system performance.
| Original language | English (US) |
|---|---|
| Title of host publication | Performance Metrics for Intelligent Systems (PerMIS) Workshop |
| Pages | 182-188 |
| Number of pages | 7 |
| State | Published - 2007 |
| Externally published | Yes |
| Event | 2007 Performance Metrics for Intelligent Systems Workshop, PerMIS'07 - Gaithersburg, MD, United States Duration: Aug 28 2007 → Aug 30 2007 |
Other
| Other | 2007 Performance Metrics for Intelligent Systems Workshop, PerMIS'07 |
|---|---|
| Country | United States |
| City | Gaithersburg, MD |
| Period | 8/28/07 → 8/30/07 |
Fingerprint
Keywords
- Evaluation
- Intelligent systems
- Machine learning
- Mixed-initiative assistants
ASJC Scopus subject areas
- Artificial Intelligence
- Computer Science Applications
- Control and Systems Engineering
Cite this
Evaluation of an integrated multi-task machine learning system with humans in the loop. / Steinfeld, Aaron; Bennett, S. Rachael; Cunningham, Kyle; Lahut, Matt; Quinones, Pablo Alejandro; Wexler, Django; Siewiorek, Dan; Hayes, Jordan; Cohen, Paul R; Fitzgerald, Julie; Hansson, Othar; Pool, Mike; Drummond, Mark.
Performance Metrics for Intelligent Systems (PerMIS) Workshop. 2007. p. 182-188.Research output: Chapter in Book/Report/Conference proceeding › Conference contribution
}
TY - GEN
T1 - Evaluation of an integrated multi-task machine learning system with humans in the loop
AU - Steinfeld, Aaron
AU - Bennett, S. Rachael
AU - Cunningham, Kyle
AU - Lahut, Matt
AU - Quinones, Pablo Alejandro
AU - Wexler, Django
AU - Siewiorek, Dan
AU - Hayes, Jordan
AU - Cohen, Paul R
AU - Fitzgerald, Julie
AU - Hansson, Othar
AU - Pool, Mike
AU - Drummond, Mark
PY - 2007
Y1 - 2007
N2 - Performance of a cognitive personal assistant, RADAR, consisting of multiple machine learning components, natural language processing, and optimization was examined with a test explicitly developed to measure the impact of integrated machine learning when used by a human user in a real world setting. Three conditions (conventional tools, Radar without learning, and Radar with learning) were evaluated in a large-scale, between-subjects study. The study revealed that integrated machine learning does produce a positive impact on overall performance. This paper also discusses how specific machine learning components contributed to human-system performance.
AB - Performance of a cognitive personal assistant, RADAR, consisting of multiple machine learning components, natural language processing, and optimization was examined with a test explicitly developed to measure the impact of integrated machine learning when used by a human user in a real world setting. Three conditions (conventional tools, Radar without learning, and Radar with learning) were evaluated in a large-scale, between-subjects study. The study revealed that integrated machine learning does produce a positive impact on overall performance. This paper also discusses how specific machine learning components contributed to human-system performance.
KW - Evaluation
KW - Intelligent systems
KW - Machine learning
KW - Mixed-initiative assistants
UR - http://www.scopus.com/inward/record.url?scp=45349083234&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=45349083234&partnerID=8YFLogxK
M3 - Conference contribution
AN - SCOPUS:45349083234
SP - 182
EP - 188
BT - Performance Metrics for Intelligent Systems (PerMIS) Workshop
ER -