Validation of expert systems for innovation management: Issues, methodology, and empirical assessment

Sundaresan Ram, Sudha Ram

Research output: Contribution to journalArticle

10 Citations (Scopus)

Abstract

Faced with the complexities of managing new product development, most of us would welcome the support of a computer-based system that captures the knowledge and the reasoning capabilities of experts in our field. Considerable effort has been focused on the design and development of expert systems for applications such as new product management. However, design and development are only two steps on the path to successful implementation of a useful expert system. A rigorous validation process is essential for ensuring that the expert system performs as intended. Using the INNOVATOR expert system as an example, Sundaresan Ram and Sudha Ram propose and test a framework for validating expert systems designed for new product management. The proposed validation framework considers three aspects of the expert system: its knowledge acquisition methodology, its performance, and its utility. Validation of an expert system's knowledge acquisition methodology involves assessment of the knowledge sources used, the criteria for selecting human experts, and the methods used for knowledge acquisition. Using multiple sources improves the likelihood that the expert system will capture the necessary core knowledge. Similarly, selection of the experts who are to supply the knowledge used by the expert system should be based on reliable measures of new product expertise rather than ad hoc measures. The system's performance is evaluated through formal tests of the accuracy and the completeness of the knowledge base, the consistency and the accuracy of the decisions made by the system, and the reasoning process by which the system reaches its decisions. Such tests may involve direct examination of the system by experts, and Turing tests, which compare both the recommendations and the reasoning process of the system with those of selected experts. Both types of tests may involve experts from whom knowledge was acquired during the development of the system as well as experts who were not involved in the design and development of the system. Assessment of an expert system's utility focuses on user perceptions of system performance and utility as well as the design of the user interface. First, end-users must evaluate the relevance of the chosen problem domain. In other words, the validation process must verify that the expert system addresses an important problem that requires decision support tools. Second, the expert system must provide a logical, systematic approach to solving the problem. Finally, the expert system must provide a consistent, intuitive user interface.

Original languageEnglish (US)
Pages (from-to)53-68
Number of pages16
JournalJournal of Product Innovation Management
Volume13
Issue number1
DOIs
StatePublished - Jan 1996
Externally publishedYes

Fingerprint

Expert systems
Innovation
Knowledge acquisition
Expert system
Methodology
Innovation management
User interfaces
Product development

ASJC Scopus subject areas

  • Marketing

Cite this

Validation of expert systems for innovation management : Issues, methodology, and empirical assessment. / Ram, Sundaresan; Ram, Sudha.

In: Journal of Product Innovation Management, Vol. 13, No. 1, 01.1996, p. 53-68.

Research output: Contribution to journalArticle

@article{523555f438b6464f91a142901002cc89,
title = "Validation of expert systems for innovation management: Issues, methodology, and empirical assessment",
abstract = "Faced with the complexities of managing new product development, most of us would welcome the support of a computer-based system that captures the knowledge and the reasoning capabilities of experts in our field. Considerable effort has been focused on the design and development of expert systems for applications such as new product management. However, design and development are only two steps on the path to successful implementation of a useful expert system. A rigorous validation process is essential for ensuring that the expert system performs as intended. Using the INNOVATOR expert system as an example, Sundaresan Ram and Sudha Ram propose and test a framework for validating expert systems designed for new product management. The proposed validation framework considers three aspects of the expert system: its knowledge acquisition methodology, its performance, and its utility. Validation of an expert system's knowledge acquisition methodology involves assessment of the knowledge sources used, the criteria for selecting human experts, and the methods used for knowledge acquisition. Using multiple sources improves the likelihood that the expert system will capture the necessary core knowledge. Similarly, selection of the experts who are to supply the knowledge used by the expert system should be based on reliable measures of new product expertise rather than ad hoc measures. The system's performance is evaluated through formal tests of the accuracy and the completeness of the knowledge base, the consistency and the accuracy of the decisions made by the system, and the reasoning process by which the system reaches its decisions. Such tests may involve direct examination of the system by experts, and Turing tests, which compare both the recommendations and the reasoning process of the system with those of selected experts. Both types of tests may involve experts from whom knowledge was acquired during the development of the system as well as experts who were not involved in the design and development of the system. Assessment of an expert system's utility focuses on user perceptions of system performance and utility as well as the design of the user interface. First, end-users must evaluate the relevance of the chosen problem domain. In other words, the validation process must verify that the expert system addresses an important problem that requires decision support tools. Second, the expert system must provide a logical, systematic approach to solving the problem. Finally, the expert system must provide a consistent, intuitive user interface.",
author = "Sundaresan Ram and Sudha Ram",
year = "1996",
month = "1",
doi = "10.1016/0737-6782(95)00087-9",
language = "English (US)",
volume = "13",
pages = "53--68",
journal = "Journal of Product Innovation Management",
issn = "0737-6782",
publisher = "Wiley-Blackwell",
number = "1",

}

TY - JOUR

T1 - Validation of expert systems for innovation management

T2 - Issues, methodology, and empirical assessment

AU - Ram, Sundaresan

AU - Ram, Sudha

PY - 1996/1

Y1 - 1996/1

N2 - Faced with the complexities of managing new product development, most of us would welcome the support of a computer-based system that captures the knowledge and the reasoning capabilities of experts in our field. Considerable effort has been focused on the design and development of expert systems for applications such as new product management. However, design and development are only two steps on the path to successful implementation of a useful expert system. A rigorous validation process is essential for ensuring that the expert system performs as intended. Using the INNOVATOR expert system as an example, Sundaresan Ram and Sudha Ram propose and test a framework for validating expert systems designed for new product management. The proposed validation framework considers three aspects of the expert system: its knowledge acquisition methodology, its performance, and its utility. Validation of an expert system's knowledge acquisition methodology involves assessment of the knowledge sources used, the criteria for selecting human experts, and the methods used for knowledge acquisition. Using multiple sources improves the likelihood that the expert system will capture the necessary core knowledge. Similarly, selection of the experts who are to supply the knowledge used by the expert system should be based on reliable measures of new product expertise rather than ad hoc measures. The system's performance is evaluated through formal tests of the accuracy and the completeness of the knowledge base, the consistency and the accuracy of the decisions made by the system, and the reasoning process by which the system reaches its decisions. Such tests may involve direct examination of the system by experts, and Turing tests, which compare both the recommendations and the reasoning process of the system with those of selected experts. Both types of tests may involve experts from whom knowledge was acquired during the development of the system as well as experts who were not involved in the design and development of the system. Assessment of an expert system's utility focuses on user perceptions of system performance and utility as well as the design of the user interface. First, end-users must evaluate the relevance of the chosen problem domain. In other words, the validation process must verify that the expert system addresses an important problem that requires decision support tools. Second, the expert system must provide a logical, systematic approach to solving the problem. Finally, the expert system must provide a consistent, intuitive user interface.

AB - Faced with the complexities of managing new product development, most of us would welcome the support of a computer-based system that captures the knowledge and the reasoning capabilities of experts in our field. Considerable effort has been focused on the design and development of expert systems for applications such as new product management. However, design and development are only two steps on the path to successful implementation of a useful expert system. A rigorous validation process is essential for ensuring that the expert system performs as intended. Using the INNOVATOR expert system as an example, Sundaresan Ram and Sudha Ram propose and test a framework for validating expert systems designed for new product management. The proposed validation framework considers three aspects of the expert system: its knowledge acquisition methodology, its performance, and its utility. Validation of an expert system's knowledge acquisition methodology involves assessment of the knowledge sources used, the criteria for selecting human experts, and the methods used for knowledge acquisition. Using multiple sources improves the likelihood that the expert system will capture the necessary core knowledge. Similarly, selection of the experts who are to supply the knowledge used by the expert system should be based on reliable measures of new product expertise rather than ad hoc measures. The system's performance is evaluated through formal tests of the accuracy and the completeness of the knowledge base, the consistency and the accuracy of the decisions made by the system, and the reasoning process by which the system reaches its decisions. Such tests may involve direct examination of the system by experts, and Turing tests, which compare both the recommendations and the reasoning process of the system with those of selected experts. Both types of tests may involve experts from whom knowledge was acquired during the development of the system as well as experts who were not involved in the design and development of the system. Assessment of an expert system's utility focuses on user perceptions of system performance and utility as well as the design of the user interface. First, end-users must evaluate the relevance of the chosen problem domain. In other words, the validation process must verify that the expert system addresses an important problem that requires decision support tools. Second, the expert system must provide a logical, systematic approach to solving the problem. Finally, the expert system must provide a consistent, intuitive user interface.

UR - http://www.scopus.com/inward/record.url?scp=0029770608&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0029770608&partnerID=8YFLogxK

U2 - 10.1016/0737-6782(95)00087-9

DO - 10.1016/0737-6782(95)00087-9

M3 - Article

AN - SCOPUS:0029770608

VL - 13

SP - 53

EP - 68

JO - Journal of Product Innovation Management

JF - Journal of Product Innovation Management

SN - 0737-6782

IS - 1

ER -