Word sense disambiguation with pictures

Jacobus J Barnard, Matthew Johnson

Research output: Contribution to journalArticle

32 Citations (Scopus)

Abstract

We introduce using images for word sense disambiguation, either alone, or in conjunction with traditional text based methods. The approach is based on a recently developed method for automatically annotating images by using a statistical model for the joint probability for image regions and words. The model itself is learned from a data base of images with associated text. To use the model for word sense disambiguation, we constrain the predicted words to be possible senses for the word under consideration. When word prediction is constrained to a narrow set of choices (such as possible senses), it can be quite reliable. We report on experiments using the resulting sense probabilities as is, as well as augmenting a state of the art text based word sense disambiguation algorithm. In order to evaluate our approach, we developed a new corpus, ImCor, which consists of a substantive portion of the Corel image data set associated with disambiguated text drawn from the SemCor corpus. Our experiments using this corpus suggest that visual information can be very useful in disambiguating word senses. It also illustrates that associated non-textual information such as image data can help ground language meaning.

Original languageEnglish (US)
Pages (from-to)13-30
Number of pages18
JournalArtificial Intelligence
Volume167
Issue number1-2
DOIs
StatePublished - Sep 2005

Fingerprint

Experiments
experiment
Disambiguation
Word Sense
language
Statistical Models
Experiment
Data Base
Language
Statistical Model
Prediction

Keywords

  • Image auto-annotation
  • Region labeling
  • Statistical models
  • Word sense disambiguation

ASJC Scopus subject areas

  • Artificial Intelligence
  • Computational Theory and Mathematics

Cite this

Word sense disambiguation with pictures. / Barnard, Jacobus J; Johnson, Matthew.

In: Artificial Intelligence, Vol. 167, No. 1-2, 09.2005, p. 13-30.

Research output: Contribution to journalArticle

Barnard, Jacobus J ; Johnson, Matthew. / Word sense disambiguation with pictures. In: Artificial Intelligence. 2005 ; Vol. 167, No. 1-2. pp. 13-30.
@article{deeaa61e6a624110a3e7fad055e5a4ee,
title = "Word sense disambiguation with pictures",
abstract = "We introduce using images for word sense disambiguation, either alone, or in conjunction with traditional text based methods. The approach is based on a recently developed method for automatically annotating images by using a statistical model for the joint probability for image regions and words. The model itself is learned from a data base of images with associated text. To use the model for word sense disambiguation, we constrain the predicted words to be possible senses for the word under consideration. When word prediction is constrained to a narrow set of choices (such as possible senses), it can be quite reliable. We report on experiments using the resulting sense probabilities as is, as well as augmenting a state of the art text based word sense disambiguation algorithm. In order to evaluate our approach, we developed a new corpus, ImCor, which consists of a substantive portion of the Corel image data set associated with disambiguated text drawn from the SemCor corpus. Our experiments using this corpus suggest that visual information can be very useful in disambiguating word senses. It also illustrates that associated non-textual information such as image data can help ground language meaning.",
keywords = "Image auto-annotation, Region labeling, Statistical models, Word sense disambiguation",
author = "Barnard, {Jacobus J} and Matthew Johnson",
year = "2005",
month = "9",
doi = "10.1016/j.artint.2005.04.009",
language = "English (US)",
volume = "167",
pages = "13--30",
journal = "Artificial Intelligence",
issn = "0004-3702",
publisher = "Elsevier",
number = "1-2",

}

TY - JOUR

T1 - Word sense disambiguation with pictures

AU - Barnard, Jacobus J

AU - Johnson, Matthew

PY - 2005/9

Y1 - 2005/9

N2 - We introduce using images for word sense disambiguation, either alone, or in conjunction with traditional text based methods. The approach is based on a recently developed method for automatically annotating images by using a statistical model for the joint probability for image regions and words. The model itself is learned from a data base of images with associated text. To use the model for word sense disambiguation, we constrain the predicted words to be possible senses for the word under consideration. When word prediction is constrained to a narrow set of choices (such as possible senses), it can be quite reliable. We report on experiments using the resulting sense probabilities as is, as well as augmenting a state of the art text based word sense disambiguation algorithm. In order to evaluate our approach, we developed a new corpus, ImCor, which consists of a substantive portion of the Corel image data set associated with disambiguated text drawn from the SemCor corpus. Our experiments using this corpus suggest that visual information can be very useful in disambiguating word senses. It also illustrates that associated non-textual information such as image data can help ground language meaning.

AB - We introduce using images for word sense disambiguation, either alone, or in conjunction with traditional text based methods. The approach is based on a recently developed method for automatically annotating images by using a statistical model for the joint probability for image regions and words. The model itself is learned from a data base of images with associated text. To use the model for word sense disambiguation, we constrain the predicted words to be possible senses for the word under consideration. When word prediction is constrained to a narrow set of choices (such as possible senses), it can be quite reliable. We report on experiments using the resulting sense probabilities as is, as well as augmenting a state of the art text based word sense disambiguation algorithm. In order to evaluate our approach, we developed a new corpus, ImCor, which consists of a substantive portion of the Corel image data set associated with disambiguated text drawn from the SemCor corpus. Our experiments using this corpus suggest that visual information can be very useful in disambiguating word senses. It also illustrates that associated non-textual information such as image data can help ground language meaning.

KW - Image auto-annotation

KW - Region labeling

KW - Statistical models

KW - Word sense disambiguation

UR - http://www.scopus.com/inward/record.url?scp=24144458606&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=24144458606&partnerID=8YFLogxK

U2 - 10.1016/j.artint.2005.04.009

DO - 10.1016/j.artint.2005.04.009

M3 - Article

AN - SCOPUS:24144458606

VL - 167

SP - 13

EP - 30

JO - Artificial Intelligence

JF - Artificial Intelligence

SN - 0004-3702

IS - 1-2

ER -