Using the Hammer only on Nails: A Hybrid Method for Representation-Based Evidence Retrieval for Question Answering

Zhengzhong Liang, Yiyun Zhao, Mihai Surdeanu

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Evidence retrieval is a key component of explainable question answering (QA). We argue that, despite recent progress, transformer network-based approaches such as universal sentence encoder (USE-QA) do not always outperform traditional information retrieval (IR) methods such as BM25 for evidence retrieval for QA. We introduce a lexical probing task that validates this observation: we demonstrate that neural IR methods have the capacity to capture lexical differences between questions and answers, but miss obvious lexical overlap signal. Learning from this probing analysis, we introduce a hybrid approach for representation-based evidence retrieval that combines the advantages of both IR directions. Our approach uses a routing classifier that learns when to direct incoming questions to BM25 vs. USE-QA for evidence retrieval using very simple statistics, which can be efficiently extracted from the top candidate evidence sentences produced by a BM25 model. We demonstrate that this hybrid evidence retrieval generally performs better than either individual retrieval strategy on three QA datasets: OpenBookQA, ReQA SQuAD, and ReQA NQ. Furthermore, we show that the proposed routing strategy is considerably faster than neural methods, with a runtime that is up to 5 times faster than USE-QA.

Original languageEnglish (US)
Title of host publicationAdvances in Information Retrieval - 43rd European Conference on IR Research, ECIR 2021, Proceedings
EditorsDjoerd Hiemstra, Marie-Francine Moens, Josiane Mothe, Raffaele Perego, Martin Potthast, Fabrizio Sebastiani
PublisherSpringer Science and Business Media Deutschland GmbH
Pages327-341
Number of pages15
ISBN (Print)9783030721121
DOIs
StatePublished - 2021
Event43rd European Conference on Information Retrieval Research, ECIR 2021 - Virtual, Online
Duration: Mar 28 2021Apr 1 2021

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume12656 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference43rd European Conference on Information Retrieval Research, ECIR 2021
CityVirtual, Online
Period3/28/214/1/21

Keywords

  • BM25
  • Neural information retrieval
  • Representation-based

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)

Fingerprint

Dive into the research topics of 'Using the Hammer only on Nails: A Hybrid Method for Representation-Based Evidence Retrieval for Question Answering'. Together they form a unique fingerprint.

Cite this