TREC iKAT 2023: A Test Collection for Evaluating Conversational and Interactive Knowledge Assistants

Mohammad Aliannejadi, Zahra Abbasiantaeb, Shubham Chatterjee, Jeffrey Dalton, Leif Azzopardi

Research output: Chapter in Book/Report/Conference proceedingConference contribution book

Abstract

Conversational information seeking has evolved rapidly in the last few years with the development of Large Language Models (LLMs), providing the basis for interpreting and responding in a naturalistic manner to user requests. The extended TREC Interactive Knowledge Assistance Track (iKAT) collection aims to enable researchers to test and evaluate their Conversational Search Agent (CSA). The collection contains a set of 36 personalized dialogues over 20 different topics each coupled with a Personal Text Knowledge Base (PTKB) that defines the bespoke user personas. A total of 344 turns with approximately 26,000 passages are provided as assessments on relevance, as well as additional assessments on generated responses over four key dimensions: relevance, completeness, groundedness, and naturalness. The collection challenges CSAs to efficiently navigate diverse personal contexts, elicit pertinent persona information, and employ context for relevant conversations.
The integration of a PTKB and the emphasis on decisional search tasks contribute to the uniqueness of this test collection, making it an essential benchmark for advancing research in conversational and interactive knowledge assistants.
Original languageEnglish
Title of host publicationSIGIR '24: Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval
PublisherAssociation for Computing Machinery (ACM)
Pages819 - 829
Number of pages11
ISBN (Electronic)979-8-4007-0431-4
DOIs
Publication statusPublished - 10 Jul 2024
EventThe 47th International ACM SIGIR Conference on Research and Development in Information Retrieval: SIGIR' 24 - Washington DC, United States
Duration: 14 Jul 202418 Jul 2024
Conference number: 47
https://sigir-2024.github.io
https://sigir-2024.github.io/

Conference

ConferenceThe 47th International ACM SIGIR Conference on Research and Development in Information Retrieval
Abbreviated titleSIGIR 2024
Country/TerritoryUnited States
CityWashington DC
Period14/07/2418/07/24
Internet address

Keywords

  • conversational information seeking
  • conversational search agents
  • evaluation
  • test collection

Fingerprint

Dive into the research topics of 'TREC iKAT 2023: A Test Collection for Evaluating Conversational and Interactive Knowledge Assistants'. Together they form a unique fingerprint.

Cite this