An evaluation of the chat and knowledge delivery components of a low-level dialog system

The AZ-ALICE experiment

Robert P. Schumaker, Mark Ginsburg, Hsinchun Chen, Ying Liu

Research output: Contribution to journalArticle

11 Citations (Scopus)

Abstract

An effective networked knowledge delivery platform is one of the Holy Grails of Web computing. Knowledge delivery approaches range from the heavy and narrow to the light and broad. This paper explores a lightweight and flexible dialog framework based on the ALICE system, and evaluates its performance in chat and knowledge delivery using both a conversational setting and a specific telecommunications knowledge domain. Metrics for evaluation are presented, and the evaluations of three experimental systems (a pure dialog system, a domain knowledge system, and a hybrid system combining dialog and domain knowledge) are presented and discussed. Our study of 257 subjects shows approximately a 20% user correction rate on system responses. Certain error classes (such as nonsense replies) were particular to the dialog system, while others (such as mistaking opinion questions for definition questions) were particular to the domain system. A third type of error, wordy and awkward responses, is a basic system property and spans all three experimental systems. We also show that the highest response satisfaction results are obtained when coupling domain-specific knowledge together with conversational dialog.

Original languageEnglish (US)
Pages (from-to)2236-2246
Number of pages11
JournalDecision Support Systems
Volume42
Issue number4
DOIs
StatePublished - Jan 2007

Fingerprint

Hybrid systems
Telecommunication
Experiments
Telecommunications
Light
Dialogue Systems
Experiment
Evaluation
Experimental System
Domain knowledge
World Wide Web
Knowledge System
Nonsense
Holy

Keywords

  • AIML
  • ALICE
  • chatterbot
  • Dialog platform
  • Domain-specific knowledge
  • Knowledge delivery evaluation
  • XML

ASJC Scopus subject areas

  • Management Information Systems
  • Information Systems
  • Information Systems and Management

Cite this

An evaluation of the chat and knowledge delivery components of a low-level dialog system : The AZ-ALICE experiment. / Schumaker, Robert P.; Ginsburg, Mark; Chen, Hsinchun; Liu, Ying.

In: Decision Support Systems, Vol. 42, No. 4, 01.2007, p. 2236-2246.

Research output: Contribution to journalArticle

@article{49bae897ce794216b57ac1aff0646b20,
title = "An evaluation of the chat and knowledge delivery components of a low-level dialog system: The AZ-ALICE experiment",
abstract = "An effective networked knowledge delivery platform is one of the Holy Grails of Web computing. Knowledge delivery approaches range from the heavy and narrow to the light and broad. This paper explores a lightweight and flexible dialog framework based on the ALICE system, and evaluates its performance in chat and knowledge delivery using both a conversational setting and a specific telecommunications knowledge domain. Metrics for evaluation are presented, and the evaluations of three experimental systems (a pure dialog system, a domain knowledge system, and a hybrid system combining dialog and domain knowledge) are presented and discussed. Our study of 257 subjects shows approximately a 20{\%} user correction rate on system responses. Certain error classes (such as nonsense replies) were particular to the dialog system, while others (such as mistaking opinion questions for definition questions) were particular to the domain system. A third type of error, wordy and awkward responses, is a basic system property and spans all three experimental systems. We also show that the highest response satisfaction results are obtained when coupling domain-specific knowledge together with conversational dialog.",
keywords = "AIML, ALICE, chatterbot, Dialog platform, Domain-specific knowledge, Knowledge delivery evaluation, XML",
author = "Schumaker, {Robert P.} and Mark Ginsburg and Hsinchun Chen and Ying Liu",
year = "2007",
month = "1",
doi = "10.1016/j.dss.2006.07.001",
language = "English (US)",
volume = "42",
pages = "2236--2246",
journal = "Decision Support Systems",
issn = "0167-9236",
publisher = "Elsevier",
number = "4",

}

TY - JOUR

T1 - An evaluation of the chat and knowledge delivery components of a low-level dialog system

T2 - The AZ-ALICE experiment

AU - Schumaker, Robert P.

AU - Ginsburg, Mark

AU - Chen, Hsinchun

AU - Liu, Ying

PY - 2007/1

Y1 - 2007/1

N2 - An effective networked knowledge delivery platform is one of the Holy Grails of Web computing. Knowledge delivery approaches range from the heavy and narrow to the light and broad. This paper explores a lightweight and flexible dialog framework based on the ALICE system, and evaluates its performance in chat and knowledge delivery using both a conversational setting and a specific telecommunications knowledge domain. Metrics for evaluation are presented, and the evaluations of three experimental systems (a pure dialog system, a domain knowledge system, and a hybrid system combining dialog and domain knowledge) are presented and discussed. Our study of 257 subjects shows approximately a 20% user correction rate on system responses. Certain error classes (such as nonsense replies) were particular to the dialog system, while others (such as mistaking opinion questions for definition questions) were particular to the domain system. A third type of error, wordy and awkward responses, is a basic system property and spans all three experimental systems. We also show that the highest response satisfaction results are obtained when coupling domain-specific knowledge together with conversational dialog.

AB - An effective networked knowledge delivery platform is one of the Holy Grails of Web computing. Knowledge delivery approaches range from the heavy and narrow to the light and broad. This paper explores a lightweight and flexible dialog framework based on the ALICE system, and evaluates its performance in chat and knowledge delivery using both a conversational setting and a specific telecommunications knowledge domain. Metrics for evaluation are presented, and the evaluations of three experimental systems (a pure dialog system, a domain knowledge system, and a hybrid system combining dialog and domain knowledge) are presented and discussed. Our study of 257 subjects shows approximately a 20% user correction rate on system responses. Certain error classes (such as nonsense replies) were particular to the dialog system, while others (such as mistaking opinion questions for definition questions) were particular to the domain system. A third type of error, wordy and awkward responses, is a basic system property and spans all three experimental systems. We also show that the highest response satisfaction results are obtained when coupling domain-specific knowledge together with conversational dialog.

KW - AIML

KW - ALICE

KW - chatterbot

KW - Dialog platform

KW - Domain-specific knowledge

KW - Knowledge delivery evaluation

KW - XML

UR - http://www.scopus.com/inward/record.url?scp=33845703020&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=33845703020&partnerID=8YFLogxK

U2 - 10.1016/j.dss.2006.07.001

DO - 10.1016/j.dss.2006.07.001

M3 - Article

VL - 42

SP - 2236

EP - 2246

JO - Decision Support Systems

JF - Decision Support Systems

SN - 0167-9236

IS - 4

ER -