Improving and aligning speech with presentation slides

Ranjini Swaminathan, Michael E. Thompson, Sandiway Fong, Alon Efrat, Arnon Amir, Kobus Barnard

Research output: Chapter in Book/Report/Conference proceedingConference contribution

3 Scopus citations

Abstract

We present a novel method to correct automatically generated speech transcripts of talks and lecture videos using text from accompanying presentation slides. The approach finesses the challenges of dealing with technical terms which are often outside the vocabulary of speech recognizers. Further, we align the transcript to the slide word sequence so that we can improve the organization of closed captioning for hearing impaired users, and improve automatic highlighting or magnification for visually impaired users. For each speech segment associated with a slide, we construct a sequential Hidden Markov Model for the observed phonemes that follows slide word order, interspersed with text not on the slide. Incongruence between slide words and mistaken transcript words is accounted for using phoneme confusion probabilities. Hence, transcript words different from aligned high probability slide words can be corrected. Experiments on six talks show improvement in transcript accuracy and alignment with slide words.

Original languageEnglish (US)
Title of host publicationProceedings - 2010 20th International Conference on Pattern Recognition, ICPR 2010
Pages3280-3283
Number of pages4
DOIs
StatePublished - Nov 18 2010
Event2010 20th International Conference on Pattern Recognition, ICPR 2010 - Istanbul, Turkey
Duration: Aug 23 2010Aug 26 2010

Publication series

NameProceedings - International Conference on Pattern Recognition
ISSN (Print)1051-4651

Other

Other2010 20th International Conference on Pattern Recognition, ICPR 2010
CountryTurkey
CityIstanbul
Period8/23/108/26/10

ASJC Scopus subject areas

  • Computer Vision and Pattern Recognition

Fingerprint Dive into the research topics of 'Improving and aligning speech with presentation slides'. Together they form a unique fingerprint.

  • Cite this

    Swaminathan, R., Thompson, M. E., Fong, S., Efrat, A., Amir, A., & Barnard, K. (2010). Improving and aligning speech with presentation slides. In Proceedings - 2010 20th International Conference on Pattern Recognition, ICPR 2010 (pp. 3280-3283). [5597494] (Proceedings - International Conference on Pattern Recognition). https://doi.org/10.1109/ICPR.2010.802