CLEF 2017 dynamic search evaluation lab overview

Evangelos Kanoulas, Leif Azzopardi

Research output: Chapter in Book/Report/Conference proceedingConference contribution book

1 Citation (Scopus)

Abstract

In this paper we provide an overview of the first edition of the CLEF Dynamic Search Lab. The CLEF Dynamic Search lab ran in the form of a workshop with the goal of approaching one key question: how can we evaluate dynamic search algorithms? Unlike static search algorithms, which essentially consider user request’s independently, and which do not adapt the ranking w.r.t the user’s sequence of interactions, dynamic search algorithms try to infer from the user’s intentions from their interactions and then adapt the ranking accordingly. Personalized session search, contextual search, and dialog systems often adopt such algorithms. This lab provides an opportunity for researchers to discuss the challenges faced when trying to measure and evaluate the performance of dynamic search algorithms, given the context of available corpora, simulations methods, and current evaluation metrics. To seed the discussion, a pilot task was run with the goal of producing search agents that could simulate the process of a user, interacting with a search system over the course of a search session. Herein, we describe the overall objectives of the CLEF 2017 Dynamic Search Lab, the resources created for the pilot task and the evaluation methodology adopted.

LanguageEnglish
Title of host publicationExperimental IR Meets Multilinguality, Multimodality, and Interaction
Subtitle of host publication8th International Conference of the CLEF Association, CLEF 2017, Proceedings
EditorsGareth J.F. Jones, Séamus Lawless, Julio Gonzalo, Liadh Kelly, Lorraine Goeuriot, Thomas Mandl, Linda Cappellato, Nicola Ferro
Place of PublicationBerlin
PublisherSpringer-Verlag
Pages361-366
Number of pages6
Volume10456
ISBN (Print)9783319658124, 9783319658131
DOIs
Publication statusPublished - 17 Aug 2017
Event8th International Conference of the CLEF Association, CLEF 2017 - Dublin, Ireland
Duration: 11 Sep 201714 Sep 2017

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume10456 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference8th International Conference of the CLEF Association, CLEF 2017
CountryIreland
CityDublin
Period11/09/1714/09/17

Fingerprint

Evaluation
Search Algorithm
Dynamic Algorithms
Ranking
Dialogue Systems
Seed
Evaluate
Interaction
Simulation Methods
Metric
Resources
Methodology

Keywords

  • CLEF dynamic search lab
  • dynamic search algorithms
  • performance
  • assessment

Cite this

Kanoulas, E., & Azzopardi, L. (2017). CLEF 2017 dynamic search evaluation lab overview. In G. J. F. Jones, S. Lawless, J. Gonzalo, L. Kelly, L. Goeuriot, T. Mandl, L. Cappellato, ... N. Ferro (Eds.), Experimental IR Meets Multilinguality, Multimodality, and Interaction: 8th International Conference of the CLEF Association, CLEF 2017, Proceedings (Vol. 10456, pp. 361-366). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 10456 LNCS). Berlin: Springer-Verlag. https://doi.org/10.1007/978-3-319-65813-1_31
Kanoulas, Evangelos ; Azzopardi, Leif. / CLEF 2017 dynamic search evaluation lab overview. Experimental IR Meets Multilinguality, Multimodality, and Interaction: 8th International Conference of the CLEF Association, CLEF 2017, Proceedings. editor / Gareth J.F. Jones ; Séamus Lawless ; Julio Gonzalo ; Liadh Kelly ; Lorraine Goeuriot ; Thomas Mandl ; Linda Cappellato ; Nicola Ferro. Vol. 10456 Berlin : Springer-Verlag, 2017. pp. 361-366 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).
@inproceedings{b59b64623c5b4c8591af784d9e596d6f,
title = "CLEF 2017 dynamic search evaluation lab overview",
abstract = "In this paper we provide an overview of the first edition of the CLEF Dynamic Search Lab. The CLEF Dynamic Search lab ran in the form of a workshop with the goal of approaching one key question: how can we evaluate dynamic search algorithms? Unlike static search algorithms, which essentially consider user request’s independently, and which do not adapt the ranking w.r.t the user’s sequence of interactions, dynamic search algorithms try to infer from the user’s intentions from their interactions and then adapt the ranking accordingly. Personalized session search, contextual search, and dialog systems often adopt such algorithms. This lab provides an opportunity for researchers to discuss the challenges faced when trying to measure and evaluate the performance of dynamic search algorithms, given the context of available corpora, simulations methods, and current evaluation metrics. To seed the discussion, a pilot task was run with the goal of producing search agents that could simulate the process of a user, interacting with a search system over the course of a search session. Herein, we describe the overall objectives of the CLEF 2017 Dynamic Search Lab, the resources created for the pilot task and the evaluation methodology adopted.",
keywords = "CLEF dynamic search lab, dynamic search algorithms, performance, assessment",
author = "Evangelos Kanoulas and Leif Azzopardi",
year = "2017",
month = "8",
day = "17",
doi = "10.1007/978-3-319-65813-1_31",
language = "English",
isbn = "9783319658124",
volume = "10456",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
publisher = "Springer-Verlag",
pages = "361--366",
editor = "Jones, {Gareth J.F.} and S{\'e}amus Lawless and Julio Gonzalo and Liadh Kelly and Lorraine Goeuriot and Thomas Mandl and Linda Cappellato and Nicola Ferro",
booktitle = "Experimental IR Meets Multilinguality, Multimodality, and Interaction",

}

Kanoulas, E & Azzopardi, L 2017, CLEF 2017 dynamic search evaluation lab overview. in GJF Jones, S Lawless, J Gonzalo, L Kelly, L Goeuriot, T Mandl, L Cappellato & N Ferro (eds), Experimental IR Meets Multilinguality, Multimodality, and Interaction: 8th International Conference of the CLEF Association, CLEF 2017, Proceedings. vol. 10456, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 10456 LNCS, Springer-Verlag, Berlin, pp. 361-366, 8th International Conference of the CLEF Association, CLEF 2017, Dublin, Ireland, 11/09/17. https://doi.org/10.1007/978-3-319-65813-1_31

CLEF 2017 dynamic search evaluation lab overview. / Kanoulas, Evangelos; Azzopardi, Leif.

Experimental IR Meets Multilinguality, Multimodality, and Interaction: 8th International Conference of the CLEF Association, CLEF 2017, Proceedings. ed. / Gareth J.F. Jones; Séamus Lawless; Julio Gonzalo; Liadh Kelly; Lorraine Goeuriot; Thomas Mandl; Linda Cappellato; Nicola Ferro. Vol. 10456 Berlin : Springer-Verlag, 2017. p. 361-366 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 10456 LNCS).

Research output: Chapter in Book/Report/Conference proceedingConference contribution book

TY - GEN

T1 - CLEF 2017 dynamic search evaluation lab overview

AU - Kanoulas, Evangelos

AU - Azzopardi, Leif

PY - 2017/8/17

Y1 - 2017/8/17

N2 - In this paper we provide an overview of the first edition of the CLEF Dynamic Search Lab. The CLEF Dynamic Search lab ran in the form of a workshop with the goal of approaching one key question: how can we evaluate dynamic search algorithms? Unlike static search algorithms, which essentially consider user request’s independently, and which do not adapt the ranking w.r.t the user’s sequence of interactions, dynamic search algorithms try to infer from the user’s intentions from their interactions and then adapt the ranking accordingly. Personalized session search, contextual search, and dialog systems often adopt such algorithms. This lab provides an opportunity for researchers to discuss the challenges faced when trying to measure and evaluate the performance of dynamic search algorithms, given the context of available corpora, simulations methods, and current evaluation metrics. To seed the discussion, a pilot task was run with the goal of producing search agents that could simulate the process of a user, interacting with a search system over the course of a search session. Herein, we describe the overall objectives of the CLEF 2017 Dynamic Search Lab, the resources created for the pilot task and the evaluation methodology adopted.

AB - In this paper we provide an overview of the first edition of the CLEF Dynamic Search Lab. The CLEF Dynamic Search lab ran in the form of a workshop with the goal of approaching one key question: how can we evaluate dynamic search algorithms? Unlike static search algorithms, which essentially consider user request’s independently, and which do not adapt the ranking w.r.t the user’s sequence of interactions, dynamic search algorithms try to infer from the user’s intentions from their interactions and then adapt the ranking accordingly. Personalized session search, contextual search, and dialog systems often adopt such algorithms. This lab provides an opportunity for researchers to discuss the challenges faced when trying to measure and evaluate the performance of dynamic search algorithms, given the context of available corpora, simulations methods, and current evaluation metrics. To seed the discussion, a pilot task was run with the goal of producing search agents that could simulate the process of a user, interacting with a search system over the course of a search session. Herein, we describe the overall objectives of the CLEF 2017 Dynamic Search Lab, the resources created for the pilot task and the evaluation methodology adopted.

KW - CLEF dynamic search lab

KW - dynamic search algorithms

KW - performance

KW - assessment

UR - http://www.scopus.com/inward/record.url?scp=85029418703&partnerID=8YFLogxK

U2 - 10.1007/978-3-319-65813-1_31

DO - 10.1007/978-3-319-65813-1_31

M3 - Conference contribution book

SN - 9783319658124

SN - 9783319658131

VL - 10456

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 361

EP - 366

BT - Experimental IR Meets Multilinguality, Multimodality, and Interaction

A2 - Jones, Gareth J.F.

A2 - Lawless, Séamus

A2 - Gonzalo, Julio

A2 - Kelly, Liadh

A2 - Goeuriot, Lorraine

A2 - Mandl, Thomas

A2 - Cappellato, Linda

A2 - Ferro, Nicola

PB - Springer-Verlag

CY - Berlin

ER -

Kanoulas E, Azzopardi L. CLEF 2017 dynamic search evaluation lab overview. In Jones GJF, Lawless S, Gonzalo J, Kelly L, Goeuriot L, Mandl T, Cappellato L, Ferro N, editors, Experimental IR Meets Multilinguality, Multimodality, and Interaction: 8th International Conference of the CLEF Association, CLEF 2017, Proceedings. Vol. 10456. Berlin: Springer-Verlag. 2017. p. 361-366. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/978-3-319-65813-1_31