TY - GEN
T1 - CLEF 2017 dynamic search evaluation lab overview
AU - Kanoulas, Evangelos
AU - Azzopardi, Leif
PY - 2017/8/17
Y1 - 2017/8/17
N2 - In this paper we provide an overview of the first edition of the CLEF Dynamic Search Lab. The CLEF Dynamic Search lab ran in the form of a workshop with the goal of approaching one key question: how can we evaluate dynamic search algorithms? Unlike static search algorithms, which essentially consider user request’s independently, and which do not adapt the ranking w.r.t the user’s sequence of interactions, dynamic search algorithms try to infer from the user’s intentions from their interactions and then adapt the ranking accordingly. Personalized session search, contextual search, and dialog systems often adopt such algorithms. This lab provides an opportunity for researchers to discuss the challenges faced when trying to measure and evaluate the performance of dynamic search algorithms, given the context of available corpora, simulations methods, and current evaluation metrics. To seed the discussion, a pilot task was run with the goal of producing search agents that could simulate the process of a user, interacting with a search system over the course of a search session. Herein, we describe the overall objectives of the CLEF 2017 Dynamic Search Lab, the resources created for the pilot task and the evaluation methodology adopted.
AB - In this paper we provide an overview of the first edition of the CLEF Dynamic Search Lab. The CLEF Dynamic Search lab ran in the form of a workshop with the goal of approaching one key question: how can we evaluate dynamic search algorithms? Unlike static search algorithms, which essentially consider user request’s independently, and which do not adapt the ranking w.r.t the user’s sequence of interactions, dynamic search algorithms try to infer from the user’s intentions from their interactions and then adapt the ranking accordingly. Personalized session search, contextual search, and dialog systems often adopt such algorithms. This lab provides an opportunity for researchers to discuss the challenges faced when trying to measure and evaluate the performance of dynamic search algorithms, given the context of available corpora, simulations methods, and current evaluation metrics. To seed the discussion, a pilot task was run with the goal of producing search agents that could simulate the process of a user, interacting with a search system over the course of a search session. Herein, we describe the overall objectives of the CLEF 2017 Dynamic Search Lab, the resources created for the pilot task and the evaluation methodology adopted.
KW - CLEF dynamic search lab
KW - dynamic search algorithms
KW - performance
KW - assessment
UR - http://www.scopus.com/inward/record.url?scp=85029418703&partnerID=8YFLogxK
U2 - 10.1007/978-3-319-65813-1_31
DO - 10.1007/978-3-319-65813-1_31
M3 - Conference contribution book
AN - SCOPUS:85029418703
SN - 9783319658124
SN - 9783319658131
VL - 10456
T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
SP - 361
EP - 366
BT - Experimental IR Meets Multilinguality, Multimodality, and Interaction
A2 - Jones, Gareth J.F.
A2 - Lawless, Séamus
A2 - Gonzalo, Julio
A2 - Kelly, Liadh
A2 - Goeuriot, Lorraine
A2 - Mandl, Thomas
A2 - Cappellato, Linda
A2 - Ferro, Nicola
PB - Springer-Verlag
CY - Berlin
T2 - 8th International Conference of the CLEF Association, CLEF 2017
Y2 - 11 September 2017 through 14 September 2017
ER -