Abstract
Conversational information seeking has evolved rapidly in the last few years with the development of Large Language Models (LLMs), providing the basis for interpreting and responding in a naturalistic manner to user requests. The extended TREC Interactive Knowledge Assistance Track (iKAT) collection aims to enable researchers to test and evaluate their Conversational Search Agent (CSA). The collection contains a set of 36 personalized dialogues over 20 different topics each coupled with a Personal Text Knowledge Base (PTKB) that defines the bespoke user personas. A total of 344 turns with approximately 26,000 passages are provided as assessments on relevance, as well as additional assessments on generated responses over four key dimensions: relevance, completeness, groundedness, and naturalness. The collection challenges CSAs to efficiently navigate diverse personal contexts, elicit pertinent persona information, and employ context for relevant conversations.
The integration of a PTKB and the emphasis on decisional search tasks contribute to the uniqueness of this test collection, making it an essential benchmark for advancing research in conversational and interactive knowledge assistants.
The integration of a PTKB and the emphasis on decisional search tasks contribute to the uniqueness of this test collection, making it an essential benchmark for advancing research in conversational and interactive knowledge assistants.
Original language | English |
---|---|
Title of host publication | SIGIR '24: Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval |
Publisher | Association for Computing Machinery (ACM) |
Pages | 819 - 829 |
Number of pages | 11 |
ISBN (Electronic) | 979-8-4007-0431-4 |
DOIs | |
Publication status | Published - 10 Jul 2024 |
Event | The 47th International ACM SIGIR Conference on Research and Development in Information Retrieval: SIGIR' 24 - Washington DC, United States Duration: 14 Jul 2024 → 18 Jul 2024 Conference number: 47 https://sigir-2024.github.io https://sigir-2024.github.io/ |
Conference
Conference | The 47th International ACM SIGIR Conference on Research and Development in Information Retrieval |
---|---|
Abbreviated title | SIGIR 2024 |
Country/Territory | United States |
City | Washington DC |
Period | 14/07/24 → 18/07/24 |
Internet address |
Keywords
- conversational information seeking
- conversational search agents
- evaluation
- test collection