Automatic, multimodal evaluation of human interaction

Matthew L. Jensen, Thomas O. Meservy, Judee K Burgoon, Jay F Nunamaker

Research output: Contribution to journalArticle

19 Citations (Scopus)

Abstract

This article outlines an approach for automatically extracting behavioral indicators from video, audio, and text and explores the possibility of using those indicators to predict human-interpretable judgments of involvement, dominance, tension, and arousal. We utilized two-dimensional spatial inputs extracted from video, acoustic properties extracted from audio and verbal content transcribed from face-to-face interactions to construct a set of multimodal features. Multiple predictive models were created using the extracted features as predictors and human-coded perceptions of involvement, tenseness, and arousal as the criterion. These predicted perceptions were then used as independent variables in classifying truth and deception. Though the predicted values for perceptions performed comparably to human-coded perceptions in detecting deception, the results were not satisfying. Thus, the extracted multimodal features were used to predict deception directly. Classification accuracy was substantially higher than typical human deception detection performance. Through this research, we consider the feasibility and validity of the approach and identify how such an approach could contribute to the broader community.

Original languageEnglish (US)
Pages (from-to)367-389
Number of pages23
JournalGroup Decision and Negotiation
Volume19
Issue number4
DOIs
StatePublished - Jul 2010

Fingerprint

Acoustic properties
interaction
evaluation
video
predictive model
acoustics
Interaction
Deception
Evaluation
community
performance
Arousal

Keywords

  • Credibility
  • Credibility assessment
  • Deception
  • Deception detection
  • Multimodal behavior analysis

ASJC Scopus subject areas

  • Management of Technology and Innovation
  • Strategy and Management
  • Social Sciences(all)
  • Decision Sciences(all)

Cite this

Automatic, multimodal evaluation of human interaction. / Jensen, Matthew L.; Meservy, Thomas O.; Burgoon, Judee K; Nunamaker, Jay F.

In: Group Decision and Negotiation, Vol. 19, No. 4, 07.2010, p. 367-389.

Research output: Contribution to journalArticle

Jensen, Matthew L. ; Meservy, Thomas O. ; Burgoon, Judee K ; Nunamaker, Jay F. / Automatic, multimodal evaluation of human interaction. In: Group Decision and Negotiation. 2010 ; Vol. 19, No. 4. pp. 367-389.
@article{fa3abe5f8cd1440081a0462d9421890a,
title = "Automatic, multimodal evaluation of human interaction",
abstract = "This article outlines an approach for automatically extracting behavioral indicators from video, audio, and text and explores the possibility of using those indicators to predict human-interpretable judgments of involvement, dominance, tension, and arousal. We utilized two-dimensional spatial inputs extracted from video, acoustic properties extracted from audio and verbal content transcribed from face-to-face interactions to construct a set of multimodal features. Multiple predictive models were created using the extracted features as predictors and human-coded perceptions of involvement, tenseness, and arousal as the criterion. These predicted perceptions were then used as independent variables in classifying truth and deception. Though the predicted values for perceptions performed comparably to human-coded perceptions in detecting deception, the results were not satisfying. Thus, the extracted multimodal features were used to predict deception directly. Classification accuracy was substantially higher than typical human deception detection performance. Through this research, we consider the feasibility and validity of the approach and identify how such an approach could contribute to the broader community.",
keywords = "Credibility, Credibility assessment, Deception, Deception detection, Multimodal behavior analysis",
author = "Jensen, {Matthew L.} and Meservy, {Thomas O.} and Burgoon, {Judee K} and Nunamaker, {Jay F}",
year = "2010",
month = "7",
doi = "10.1007/s10726-009-9171-0",
language = "English (US)",
volume = "19",
pages = "367--389",
journal = "Group Decision and Negotiation",
issn = "0926-2644",
publisher = "Springer Netherlands",
number = "4",

}

TY - JOUR

T1 - Automatic, multimodal evaluation of human interaction

AU - Jensen, Matthew L.

AU - Meservy, Thomas O.

AU - Burgoon, Judee K

AU - Nunamaker, Jay F

PY - 2010/7

Y1 - 2010/7

N2 - This article outlines an approach for automatically extracting behavioral indicators from video, audio, and text and explores the possibility of using those indicators to predict human-interpretable judgments of involvement, dominance, tension, and arousal. We utilized two-dimensional spatial inputs extracted from video, acoustic properties extracted from audio and verbal content transcribed from face-to-face interactions to construct a set of multimodal features. Multiple predictive models were created using the extracted features as predictors and human-coded perceptions of involvement, tenseness, and arousal as the criterion. These predicted perceptions were then used as independent variables in classifying truth and deception. Though the predicted values for perceptions performed comparably to human-coded perceptions in detecting deception, the results were not satisfying. Thus, the extracted multimodal features were used to predict deception directly. Classification accuracy was substantially higher than typical human deception detection performance. Through this research, we consider the feasibility and validity of the approach and identify how such an approach could contribute to the broader community.

AB - This article outlines an approach for automatically extracting behavioral indicators from video, audio, and text and explores the possibility of using those indicators to predict human-interpretable judgments of involvement, dominance, tension, and arousal. We utilized two-dimensional spatial inputs extracted from video, acoustic properties extracted from audio and verbal content transcribed from face-to-face interactions to construct a set of multimodal features. Multiple predictive models were created using the extracted features as predictors and human-coded perceptions of involvement, tenseness, and arousal as the criterion. These predicted perceptions were then used as independent variables in classifying truth and deception. Though the predicted values for perceptions performed comparably to human-coded perceptions in detecting deception, the results were not satisfying. Thus, the extracted multimodal features were used to predict deception directly. Classification accuracy was substantially higher than typical human deception detection performance. Through this research, we consider the feasibility and validity of the approach and identify how such an approach could contribute to the broader community.

KW - Credibility

KW - Credibility assessment

KW - Deception

KW - Deception detection

KW - Multimodal behavior analysis

UR - http://www.scopus.com/inward/record.url?scp=77953542013&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=77953542013&partnerID=8YFLogxK

U2 - 10.1007/s10726-009-9171-0

DO - 10.1007/s10726-009-9171-0

M3 - Article

VL - 19

SP - 367

EP - 389

JO - Group Decision and Negotiation

JF - Group Decision and Negotiation

SN - 0926-2644

IS - 4

ER -