Towards a living lab for information retrieval research and development: a proposal for a living lab for product search tasks

Leif Azzopardi, Krisztian Balog

Research output: Chapter in Book/Report/Conference proceedingConference contribution book

12 Citations (Scopus)


The notion of having a 'living lab' to undertaken evaluations has been proposed by a number of proponents within the field of Information Retrieval (IR). However, what such a living lab might look like and how it might be setup has not been discussed in detail. Living labs have a number of appealing points such as realistic evaluation contexts where tasks are directly linked to user experience and the closer integration of research/academia and development. Industry facilitating more efficient knowledge transfer. However, operationalizing a living lab opens up a number of concerns regarding security, privacy, etc. as well as challenges regarding the design, development and maintenance of the infrastructure required to support such evaluations. However, we aim to further the discussion on living labs for IR evaluation and propose one possible architecture to create such an evaluation environment. To focus discussion, we put forward a proposal for a living lab on product search tasks within the context of an online shop.
Original languageEnglish
Title of host publicationMultilingual and Multimodal Information Access Evaluation
Subtitle of host publicationSecond International Conference of the Cross-Language Evaluation Forum, CLEF 2011, Amsterdam, The Netherlands, September 19-22, 2011, Proceedings
EditorsP. Forner, J. Gonzalo, J. Kekäläinen, M. Lalmas, M. de Rijke
Place of PublicationBerlin, Heidelberg
Number of pages12
ISBN (Print)978-3-642-23707-2
Publication statusPublished - 2011
Externally publishedYes

Publication series



  • living lab
  • information retrieval
  • IR research

Cite this