Extending the language modeling framework for sentence retrieval to include local context

Ronald T. Fernández, David E. Losada, Leif A. Azzopardi

Research output: Contribution to journalArticle

16 Citations (Scopus)

Abstract

Employing effective methods of sentence retrieval is essential for many tasks in Information Retrieval, such as summarization, novelty detection and question answering. The best performing sentence retrieval techniques attempt to perform matching directly between the sentences and the query. However, in this paper, we posit that the local context of a sentence can provide crucial additional evidence to further improve sentence retrieval. Using a Language Modeling Framework, we propose a novel reformulation of the sentence retrieval problem that extends previous approaches so that the local context is seamlessly incorporated within the retrieval models. In a series of comprehensive experiments, we show that localized smoothing and the prior importance of a sentence can improve retrieval effectiveness. The proposed models significantly and substantially outperform the state of the art and other competitive sentence retrieval baselines on recall-oriented measures, while remaining competitive on precision-oriented measures. This research demonstrates that local context plays an important role in estimating the relevance of a sentence, and that existing sentence retrieval language models can be extended to utilize this evidence effectively.
Original languageEnglish
Pages (from-to)355-389
Number of pages35
JournalInformation Retrieval
Volume14
Issue number4
DOIs
Publication statusPublished - 1 Aug 2011
Externally publishedYes

Keywords

  • context
  • sentence retrieval
  • language models
  • information retrieval

Fingerprint Dive into the research topics of 'Extending the language modeling framework for sentence retrieval to include local context'. Together they form a unique fingerprint.

  • Cite this