Beyond actions: exploring the discovery of tactics from user logs

Jiyin He, Pernilla Qvarfordt, Martin Halvey, Gene Golovchinsky

Research output: Contribution to journalArticle

7 Citations (Scopus)

Abstract

Search log analysis has become a common practice to gain insights into user search behaviour; it helps gain an understanding of user needs and preferences, as well as an insight into how well a system supports such needs. Currently, log analysis is typically focused on low-level user actions, i.e. logged events such as issued queries and clicked results, and often only a selection of such events are logged and analysed. However, types of logged events may differ widely from interface to interface, making comparison between systems difficult. Further, the interpretation of the meaning of and subsequent analysis of a selection of events may lead to conclusions out of context—e.g. the statistics of observed query reformulations may be influenced by the existence of a relevance feedback component. Alternatively, in lab studies user activities can be analysed at a higher level, such as search tactics and strategies, abstracted away from detailed interface implementation. Unfortunately, until now the required manual codings that map logged events to higher-level interpretations have prevented large-scale use of this type of analysis. In this paper, we propose a new method for analysing search logs by (semi-)automatically identifying user search tactics from logged events, allowing large-scale analysis that is comparable across search systems. In addition, as the resulting analysis is at a tactical level we reduce potential issues surrounding the need for interpretation of low-level user actions for log analysis. We validate the efficiency and effectiveness of the proposed tactic identification method using logs of two reference search systems of different natures: a product search system and a video search system. With the identified tactics, we perform a series of novel log analyses in terms of entropy rate of user search tactic sequences, demonstrating how this type of analysis allows comparisons of user search behaviours across systems of different nature and design. This analysis provides insights not achievable with traditional log analysis.
LanguageEnglish
Pages1200–1226
Number of pages27
JournalInformation Processing and Management
Volume52
Issue number6
Early online date11 Jun 2016
DOIs
Publication statusPublished - 30 Nov 2016

Fingerprint

tactics
Entropy
Statistics
Feedback
event
interpretation
system comparison
Tactics
entropy
coding
video
statistics
efficiency

Keywords

  • search behaviour
  • search tactics
  • log analysis

Cite this

He, Jiyin ; Qvarfordt, Pernilla ; Halvey, Martin ; Golovchinsky, Gene. / Beyond actions : exploring the discovery of tactics from user logs. In: Information Processing and Management. 2016 ; Vol. 52, No. 6. pp. 1200–1226.
@article{5573f111efd0458d839ae9ec2d4ba1c5,
title = "Beyond actions: exploring the discovery of tactics from user logs",
abstract = "Search log analysis has become a common practice to gain insights into user search behaviour; it helps gain an understanding of user needs and preferences, as well as an insight into how well a system supports such needs. Currently, log analysis is typically focused on low-level user actions, i.e. logged events such as issued queries and clicked results, and often only a selection of such events are logged and analysed. However, types of logged events may differ widely from interface to interface, making comparison between systems difficult. Further, the interpretation of the meaning of and subsequent analysis of a selection of events may lead to conclusions out of context—e.g. the statistics of observed query reformulations may be influenced by the existence of a relevance feedback component. Alternatively, in lab studies user activities can be analysed at a higher level, such as search tactics and strategies, abstracted away from detailed interface implementation. Unfortunately, until now the required manual codings that map logged events to higher-level interpretations have prevented large-scale use of this type of analysis. In this paper, we propose a new method for analysing search logs by (semi-)automatically identifying user search tactics from logged events, allowing large-scale analysis that is comparable across search systems. In addition, as the resulting analysis is at a tactical level we reduce potential issues surrounding the need for interpretation of low-level user actions for log analysis. We validate the efficiency and effectiveness of the proposed tactic identification method using logs of two reference search systems of different natures: a product search system and a video search system. With the identified tactics, we perform a series of novel log analyses in terms of entropy rate of user search tactic sequences, demonstrating how this type of analysis allows comparisons of user search behaviours across systems of different nature and design. This analysis provides insights not achievable with traditional log analysis.",
keywords = "search behaviour, search tactics, log analysis",
author = "Jiyin He and Pernilla Qvarfordt and Martin Halvey and Gene Golovchinsky",
year = "2016",
month = "11",
day = "30",
doi = "10.1016/j.ipm.2016.05.007",
language = "English",
volume = "52",
pages = "1200–1226",
journal = "Information Processing and Management",
issn = "0306-4573",
number = "6",

}

Beyond actions : exploring the discovery of tactics from user logs. / He, Jiyin; Qvarfordt, Pernilla; Halvey, Martin; Golovchinsky, Gene.

In: Information Processing and Management, Vol. 52, No. 6, 30.11.2016, p. 1200–1226.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Beyond actions

T2 - Information Processing and Management

AU - He, Jiyin

AU - Qvarfordt, Pernilla

AU - Halvey, Martin

AU - Golovchinsky, Gene

PY - 2016/11/30

Y1 - 2016/11/30

N2 - Search log analysis has become a common practice to gain insights into user search behaviour; it helps gain an understanding of user needs and preferences, as well as an insight into how well a system supports such needs. Currently, log analysis is typically focused on low-level user actions, i.e. logged events such as issued queries and clicked results, and often only a selection of such events are logged and analysed. However, types of logged events may differ widely from interface to interface, making comparison between systems difficult. Further, the interpretation of the meaning of and subsequent analysis of a selection of events may lead to conclusions out of context—e.g. the statistics of observed query reformulations may be influenced by the existence of a relevance feedback component. Alternatively, in lab studies user activities can be analysed at a higher level, such as search tactics and strategies, abstracted away from detailed interface implementation. Unfortunately, until now the required manual codings that map logged events to higher-level interpretations have prevented large-scale use of this type of analysis. In this paper, we propose a new method for analysing search logs by (semi-)automatically identifying user search tactics from logged events, allowing large-scale analysis that is comparable across search systems. In addition, as the resulting analysis is at a tactical level we reduce potential issues surrounding the need for interpretation of low-level user actions for log analysis. We validate the efficiency and effectiveness of the proposed tactic identification method using logs of two reference search systems of different natures: a product search system and a video search system. With the identified tactics, we perform a series of novel log analyses in terms of entropy rate of user search tactic sequences, demonstrating how this type of analysis allows comparisons of user search behaviours across systems of different nature and design. This analysis provides insights not achievable with traditional log analysis.

AB - Search log analysis has become a common practice to gain insights into user search behaviour; it helps gain an understanding of user needs and preferences, as well as an insight into how well a system supports such needs. Currently, log analysis is typically focused on low-level user actions, i.e. logged events such as issued queries and clicked results, and often only a selection of such events are logged and analysed. However, types of logged events may differ widely from interface to interface, making comparison between systems difficult. Further, the interpretation of the meaning of and subsequent analysis of a selection of events may lead to conclusions out of context—e.g. the statistics of observed query reformulations may be influenced by the existence of a relevance feedback component. Alternatively, in lab studies user activities can be analysed at a higher level, such as search tactics and strategies, abstracted away from detailed interface implementation. Unfortunately, until now the required manual codings that map logged events to higher-level interpretations have prevented large-scale use of this type of analysis. In this paper, we propose a new method for analysing search logs by (semi-)automatically identifying user search tactics from logged events, allowing large-scale analysis that is comparable across search systems. In addition, as the resulting analysis is at a tactical level we reduce potential issues surrounding the need for interpretation of low-level user actions for log analysis. We validate the efficiency and effectiveness of the proposed tactic identification method using logs of two reference search systems of different natures: a product search system and a video search system. With the identified tactics, we perform a series of novel log analyses in terms of entropy rate of user search tactic sequences, demonstrating how this type of analysis allows comparisons of user search behaviours across systems of different nature and design. This analysis provides insights not achievable with traditional log analysis.

KW - search behaviour

KW - search tactics

KW - log analysis

UR - http://www.sciencedirect.com/science/journal/03064573

U2 - 10.1016/j.ipm.2016.05.007

DO - 10.1016/j.ipm.2016.05.007

M3 - Article

VL - 52

SP - 1200

EP - 1226

JO - Information Processing and Management

JF - Information Processing and Management

SN - 0306-4573

IS - 6

ER -