Usage based effectiveness measures: monitoring application performance in information retrieval

Research output: Chapter in Book/Report/Conference proceedingConference contribution book

14 Citations (Scopus)


The aim of an Information Retrieval (IR) application is to support the user accessing relevant information effectively and efficiently. It is well known that system performance, in terms of finding relevant information is heavily dependent upon the IR application (i.e. the IR system exposed through the application's interface), as well as how the application is used by the user (i.e. how the user interacts with the system through the interface). Thus, a very pragmatic evaluation question that arises at the application level is: what is the effectiveness experienced by the user during the usage of the application? To be able to answer this question, we represent the usage of an application by the stream of documents the user encounters while interacting with the application. This representation enables us to monitor and track the performance over time and usage. By taking a stream-based, time-centric view of the IR process, instead of a rank-list, topic/task centric view, the evaluation can be performed on any IR based application. To illustrate the difference and the utility of this approach, we demonstrate how a new suite of usage based effectiveness measures can be applied. This work provides the conceptual foundations for measuring, monitoring and modeling the performance of any IR application which needs to be evaluated over time and in context.
Original languageEnglish
Title of host publicationCIKM '09 Proceedings of the 18th ACM Conference on Information and Knowledge Management
Place of PublicationNew York, NY, USA
Number of pages10
Publication statusPublished - 2 Nov 2009
Externally publishedYes


  • application monitoring
  • user experience
  • usage performance


Dive into the research topics of 'Usage based effectiveness measures: monitoring application performance in information retrieval'. Together they form a unique fingerprint.

Cite this