CLEF 2017 dynamic search evaluation lab overview

Evangelos Kanoulas, Leif Azzopardi

Research output: Chapter in Book/Report/Conference proceedingConference contribution book

1 Citation (Scopus)
8 Downloads (Pure)

Abstract

In this paper we provide an overview of the first edition of the CLEF Dynamic Search Lab. The CLEF Dynamic Search lab ran in the form of a workshop with the goal of approaching one key question: how can we evaluate dynamic search algorithms? Unlike static search algorithms, which essentially consider user request’s independently, and which do not adapt the ranking w.r.t the user’s sequence of interactions, dynamic search algorithms try to infer from the user’s intentions from their interactions and then adapt the ranking accordingly. Personalized session search, contextual search, and dialog systems often adopt such algorithms. This lab provides an opportunity for researchers to discuss the challenges faced when trying to measure and evaluate the performance of dynamic search algorithms, given the context of available corpora, simulations methods, and current evaluation metrics. To seed the discussion, a pilot task was run with the goal of producing search agents that could simulate the process of a user, interacting with a search system over the course of a search session. Herein, we describe the overall objectives of the CLEF 2017 Dynamic Search Lab, the resources created for the pilot task and the evaluation methodology adopted.

Original languageEnglish
Title of host publicationExperimental IR Meets Multilinguality, Multimodality, and Interaction
Subtitle of host publication8th International Conference of the CLEF Association, CLEF 2017, Proceedings
EditorsGareth J.F. Jones, Séamus Lawless, Julio Gonzalo, Liadh Kelly, Lorraine Goeuriot, Thomas Mandl, Linda Cappellato, Nicola Ferro
Place of PublicationBerlin
PublisherSpringer-Verlag
Pages361-366
Number of pages6
Volume10456
ISBN (Print)9783319658124, 9783319658131
DOIs
Publication statusPublished - 17 Aug 2017
Event8th International Conference of the CLEF Association, CLEF 2017 - Dublin, Ireland
Duration: 11 Sep 201714 Sep 2017

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume10456 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference8th International Conference of the CLEF Association, CLEF 2017
CountryIreland
CityDublin
Period11/09/1714/09/17

Keywords

  • CLEF dynamic search lab
  • dynamic search algorithms
  • performance
  • assessment

Fingerprint Dive into the research topics of 'CLEF 2017 dynamic search evaluation lab overview'. Together they form a unique fingerprint.

  • Cite this

    Kanoulas, E., & Azzopardi, L. (2017). CLEF 2017 dynamic search evaluation lab overview. In G. J. F. Jones, S. Lawless, J. Gonzalo, L. Kelly, L. Goeuriot, T. Mandl, L. Cappellato, & N. Ferro (Eds.), Experimental IR Meets Multilinguality, Multimodality, and Interaction: 8th International Conference of the CLEF Association, CLEF 2017, Proceedings (Vol. 10456, pp. 361-366). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 10456 LNCS). Springer-Verlag. https://doi.org/10.1007/978-3-319-65813-1_31