Overview of the CLEF dynamic search evaluation lab 2018

Evangelos Kanoulas; Leif Azzopardi; Grace Hui Yang

doi:10.1007/978-3-319-98932-7_31

Overview of the CLEF dynamic search evaluation lab 2018

Evangelos Kanoulas^*, Leif Azzopardi, Grace Hui Yang

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution book

5 Citations (Scopus)

Abstract

In this paper we provide an overview of the CLEF 2018 Dynamic Search Lab. The lab ran for the first time in 2017 as a workshop. The outcomes of the workshop were used to define the tasks of this year’s evaluation lab. The lab strives to answer one key question: how can we evaluate, and consequently build, dynamic search algorithms? Unlike static search algorithms, which consider user request’s independently, and consequently do not adapt their ranking with respect to the user’s sequence of interactions and the user’s end goal, dynamic search algorithms try to infer the user’s intentions based on their interactions and adapt their ranking accordingly. Session personalization, contextual search, conversational search, dialog systems are some examples of dynamic search. Herein, we describe the overall objectives of the CLEF 2018 Dynamic Search Lab, the resources created, and the evaluation methodology designed.

Original language	English
Title of host publication	Experimental IR Meets Multilinguality, Multimodality, and Interaction - 9th International Conference of the CLEF Association, CLEF 2018, Proceedings
Editors	Eric SanJuan, Fionn Murtagh, Jian Yun Nie, Laure Soulier, Linda Cappellato, Patrice Bellot, Josiane Mothe, Chiraz Trabelsi, Nicola Ferro
Publisher	Springer-Verlag
Pages	362-371
Number of pages	10
ISBN (Electronic)	978-3-319-98932-7
ISBN (Print)	978-3-319-98931-0
DOIs	https://doi.org/10.1007/978-3-319-98932-7_31
Publication status	Published - 15 Aug 2018
Event	9th International Conference of the CLEF Association, CLEF 2018 - Avignon, France Duration: 10 Sept 2018 → 14 Sept 2018

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	11018 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	9th International Conference of the CLEF Association, CLEF 2018
Country/Territory	France
City	Avignon
Period	10/09/18 → 14/09/18

Funding

This work was partially supported by the Google Faculty Research Award program.

Keywords

Conversational search
Dynamic search
Evaluation
Information retrieval
Interactive search
Meta-search
Query generation
Query suggestion
Result re-ranking
User simulations

Access to Document

10.1007/978-3-319-98932-7_31

Cite this

Kanoulas, E., Azzopardi, L., & Yang, G. H. (2018). Overview of the CLEF dynamic search evaluation lab 2018. In E. SanJuan, F. Murtagh, J. Y. Nie, L. Soulier, L. Cappellato, P. Bellot, J. Mothe, C. Trabelsi, & N. Ferro (Eds.), Experimental IR Meets Multilinguality, Multimodality, and Interaction - 9th International Conference of the CLEF Association, CLEF 2018, Proceedings (pp. 362-371). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11018 LNCS). Springer-Verlag. https://doi.org/10.1007/978-3-319-98932-7_31

Kanoulas, Evangelos ; Azzopardi, Leif ; Yang, Grace Hui. / Overview of the CLEF dynamic search evaluation lab 2018. Experimental IR Meets Multilinguality, Multimodality, and Interaction - 9th International Conference of the CLEF Association, CLEF 2018, Proceedings. editor / Eric SanJuan ; Fionn Murtagh ; Jian Yun Nie ; Laure Soulier ; Linda Cappellato ; Patrice Bellot ; Josiane Mothe ; Chiraz Trabelsi ; Nicola Ferro. Springer-Verlag, 2018. pp. 362-371 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{7c186d54e04e4b978220e67c3da5311b,

title = "Overview of the CLEF dynamic search evaluation lab 2018",

abstract = "In this paper we provide an overview of the CLEF 2018 Dynamic Search Lab. The lab ran for the first time in 2017 as a workshop. The outcomes of the workshop were used to define the tasks of this year{\textquoteright}s evaluation lab. The lab strives to answer one key question: how can we evaluate, and consequently build, dynamic search algorithms? Unlike static search algorithms, which consider user request{\textquoteright}s independently, and consequently do not adapt their ranking with respect to the user{\textquoteright}s sequence of interactions and the user{\textquoteright}s end goal, dynamic search algorithms try to infer the user{\textquoteright}s intentions based on their interactions and adapt their ranking accordingly. Session personalization, contextual search, conversational search, dialog systems are some examples of dynamic search. Herein, we describe the overall objectives of the CLEF 2018 Dynamic Search Lab, the resources created, and the evaluation methodology designed.",

keywords = "Conversational search, Dynamic search, Evaluation, Information retrieval, Interactive search, Meta-search, Query generation, Query suggestion, Result re-ranking, User simulations",

author = "Evangelos Kanoulas and Leif Azzopardi and Yang, \{Grace Hui\}",

year = "2018",

month = aug,

day = "15",

doi = "10.1007/978-3-319-98932-7\_31",

language = "English",

isbn = "978-3-319-98931-0",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer-Verlag",

pages = "362--371",

editor = "Eric SanJuan and Fionn Murtagh and Nie, \{Jian Yun\} and Laure Soulier and Linda Cappellato and Patrice Bellot and Josiane Mothe and Chiraz Trabelsi and Nicola Ferro",

booktitle = "Experimental IR Meets Multilinguality, Multimodality, and Interaction - 9th International Conference of the CLEF Association, CLEF 2018, Proceedings",

note = "9th International Conference of the CLEF Association, CLEF 2018 ; Conference date: 10-09-2018 Through 14-09-2018",

}

Kanoulas, E, Azzopardi, L & Yang, GH 2018, Overview of the CLEF dynamic search evaluation lab 2018. in E SanJuan, F Murtagh, JY Nie, L Soulier, L Cappellato, P Bellot, J Mothe, C Trabelsi & N Ferro (eds), Experimental IR Meets Multilinguality, Multimodality, and Interaction - 9th International Conference of the CLEF Association, CLEF 2018, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 11018 LNCS, Springer-Verlag, pp. 362-371, 9th International Conference of the CLEF Association, CLEF 2018, Avignon, France, 10/09/18. https://doi.org/10.1007/978-3-319-98932-7_31

Overview of the CLEF dynamic search evaluation lab 2018. / Kanoulas, Evangelos; Azzopardi, Leif; Yang, Grace Hui.
Experimental IR Meets Multilinguality, Multimodality, and Interaction - 9th International Conference of the CLEF Association, CLEF 2018, Proceedings. ed. / Eric SanJuan; Fionn Murtagh; Jian Yun Nie; Laure Soulier; Linda Cappellato; Patrice Bellot; Josiane Mothe; Chiraz Trabelsi; Nicola Ferro. Springer-Verlag, 2018. p. 362-371 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11018 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution book

TY - GEN

T1 - Overview of the CLEF dynamic search evaluation lab 2018

AU - Kanoulas, Evangelos

AU - Azzopardi, Leif

AU - Yang, Grace Hui

PY - 2018/8/15

Y1 - 2018/8/15

N2 - In this paper we provide an overview of the CLEF 2018 Dynamic Search Lab. The lab ran for the first time in 2017 as a workshop. The outcomes of the workshop were used to define the tasks of this year’s evaluation lab. The lab strives to answer one key question: how can we evaluate, and consequently build, dynamic search algorithms? Unlike static search algorithms, which consider user request’s independently, and consequently do not adapt their ranking with respect to the user’s sequence of interactions and the user’s end goal, dynamic search algorithms try to infer the user’s intentions based on their interactions and adapt their ranking accordingly. Session personalization, contextual search, conversational search, dialog systems are some examples of dynamic search. Herein, we describe the overall objectives of the CLEF 2018 Dynamic Search Lab, the resources created, and the evaluation methodology designed.

AB - In this paper we provide an overview of the CLEF 2018 Dynamic Search Lab. The lab ran for the first time in 2017 as a workshop. The outcomes of the workshop were used to define the tasks of this year’s evaluation lab. The lab strives to answer one key question: how can we evaluate, and consequently build, dynamic search algorithms? Unlike static search algorithms, which consider user request’s independently, and consequently do not adapt their ranking with respect to the user’s sequence of interactions and the user’s end goal, dynamic search algorithms try to infer the user’s intentions based on their interactions and adapt their ranking accordingly. Session personalization, contextual search, conversational search, dialog systems are some examples of dynamic search. Herein, we describe the overall objectives of the CLEF 2018 Dynamic Search Lab, the resources created, and the evaluation methodology designed.

KW - Conversational search

KW - Dynamic search

KW - Evaluation

KW - Information retrieval

KW - Interactive search

KW - Meta-search

KW - Query generation

KW - Query suggestion

KW - Result re-ranking

KW - User simulations

UR - https://www.scopus.com/pages/publications/85052809008

UR - https://link.springer.com/chapter/10.1007/978-3-319-98932-7_31

U2 - 10.1007/978-3-319-98932-7_31

DO - 10.1007/978-3-319-98932-7_31

M3 - Conference contribution book

AN - SCOPUS:85052809008

SN - 978-3-319-98931-0

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 362

EP - 371

BT - Experimental IR Meets Multilinguality, Multimodality, and Interaction - 9th International Conference of the CLEF Association, CLEF 2018, Proceedings

A2 - SanJuan, Eric

A2 - Murtagh, Fionn

A2 - Nie, Jian Yun

A2 - Soulier, Laure

A2 - Cappellato, Linda

A2 - Bellot, Patrice

A2 - Mothe, Josiane

A2 - Trabelsi, Chiraz

A2 - Ferro, Nicola

PB - Springer-Verlag

T2 - 9th International Conference of the CLEF Association, CLEF 2018

Y2 - 10 September 2018 through 14 September 2018

ER -

Kanoulas E, Azzopardi L, Yang GH. Overview of the CLEF dynamic search evaluation lab 2018. In SanJuan E, Murtagh F, Nie JY, Soulier L, Cappellato L, Bellot P, Mothe J, Trabelsi C, Ferro N, editors, Experimental IR Meets Multilinguality, Multimodality, and Interaction - 9th International Conference of the CLEF Association, CLEF 2018, Proceedings. Springer-Verlag. 2018. p. 362-371. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-319-98932-7_31

Overview of the CLEF dynamic search evaluation lab 2018

Abstract

Publication series

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this