An eye-tracking approach to the analysis of relevance judgments on the web: the case of Google search engine

Panagiotis Balatsoukas, Ian Ruthven

Research output: Contribution to journalArticle

33 Citations (Scopus)

Abstract

Eye movement data can provide an in-depth view of human reasoning and the decision-making process, and modern information retrieval (IR) research can benefit from the analysis of this type of data. The aim of this research was to examine the relationship between relevance criteria use and visual behavior in the context of predictive relevance judgments. To address this objective, a multimethod research design was employed that involved observation of participants’ eye movements, talk-aloud protocols, and postsearch interviews. Specifically, the results reported in this article came from the analysis of 281 predictive relevance judgments made by 24 participants using the Google search engine. We present a novel stepwise methodological framework for the analysis of relevance judgments and eye movements on the Web and show new patterns of relevance criteria use during predictive relevance judgment. For example, the findings showed an effect of ranking order and surrogate components (Title, Summary, and URL) on the use of relevance criteria. Also, differences were observed in the cognitive effort spent between very relevant and not relevant judgments. We conclude with the implications of this study for IR research.
LanguageEnglish
Pages1728-1746
Number of pages19
JournalJournal of the American Society for Information Science and Technology
Volume63
Issue number9
Early online date17 Aug 2012
DOIs
Publication statusPublished - Sep 2012

Fingerprint

Search engines
World Wide Web
Eye movements
Information retrieval
Websites
Decision making
Network protocols

Keywords

  • information seeking
  • search engines
  • eye-tracking
  • judgement
  • user studies
  • web
  • google
  • search engine

Cite this

@article{f1289c9ea848446ca53d1b4391de9b40,
title = "An eye-tracking approach to the analysis of relevance judgments on the web: the case of Google search engine",
abstract = "Eye movement data can provide an in-depth view of human reasoning and the decision-making process, and modern information retrieval (IR) research can benefit from the analysis of this type of data. The aim of this research was to examine the relationship between relevance criteria use and visual behavior in the context of predictive relevance judgments. To address this objective, a multimethod research design was employed that involved observation of participants’ eye movements, talk-aloud protocols, and postsearch interviews. Specifically, the results reported in this article came from the analysis of 281 predictive relevance judgments made by 24 participants using the Google search engine. We present a novel stepwise methodological framework for the analysis of relevance judgments and eye movements on the Web and show new patterns of relevance criteria use during predictive relevance judgment. For example, the findings showed an effect of ranking order and surrogate components (Title, Summary, and URL) on the use of relevance criteria. Also, differences were observed in the cognitive effort spent between very relevant and not relevant judgments. We conclude with the implications of this study for IR research.",
keywords = "information seeking, search engines, eye-tracking, judgement, user studies, web, google, search engine",
author = "Panagiotis Balatsoukas and Ian Ruthven",
year = "2012",
month = "9",
doi = "10.1002/asi.22707",
language = "English",
volume = "63",
pages = "1728--1746",
journal = "Journal of the American Society for Information Science and Technology",
issn = "1532-2890",
number = "9",

}

TY - JOUR

T1 - An eye-tracking approach to the analysis of relevance judgments on the web

T2 - Journal of the American Society for Information Science and Technology

AU - Balatsoukas, Panagiotis

AU - Ruthven, Ian

PY - 2012/9

Y1 - 2012/9

N2 - Eye movement data can provide an in-depth view of human reasoning and the decision-making process, and modern information retrieval (IR) research can benefit from the analysis of this type of data. The aim of this research was to examine the relationship between relevance criteria use and visual behavior in the context of predictive relevance judgments. To address this objective, a multimethod research design was employed that involved observation of participants’ eye movements, talk-aloud protocols, and postsearch interviews. Specifically, the results reported in this article came from the analysis of 281 predictive relevance judgments made by 24 participants using the Google search engine. We present a novel stepwise methodological framework for the analysis of relevance judgments and eye movements on the Web and show new patterns of relevance criteria use during predictive relevance judgment. For example, the findings showed an effect of ranking order and surrogate components (Title, Summary, and URL) on the use of relevance criteria. Also, differences were observed in the cognitive effort spent between very relevant and not relevant judgments. We conclude with the implications of this study for IR research.

AB - Eye movement data can provide an in-depth view of human reasoning and the decision-making process, and modern information retrieval (IR) research can benefit from the analysis of this type of data. The aim of this research was to examine the relationship between relevance criteria use and visual behavior in the context of predictive relevance judgments. To address this objective, a multimethod research design was employed that involved observation of participants’ eye movements, talk-aloud protocols, and postsearch interviews. Specifically, the results reported in this article came from the analysis of 281 predictive relevance judgments made by 24 participants using the Google search engine. We present a novel stepwise methodological framework for the analysis of relevance judgments and eye movements on the Web and show new patterns of relevance criteria use during predictive relevance judgment. For example, the findings showed an effect of ranking order and surrogate components (Title, Summary, and URL) on the use of relevance criteria. Also, differences were observed in the cognitive effort spent between very relevant and not relevant judgments. We conclude with the implications of this study for IR research.

KW - information seeking

KW - search engines

KW - eye-tracking

KW - judgement

KW - user studies

KW - web

KW - google

KW - search engine

U2 - 10.1002/asi.22707

DO - 10.1002/asi.22707

M3 - Article

VL - 63

SP - 1728

EP - 1746

JO - Journal of the American Society for Information Science and Technology

JF - Journal of the American Society for Information Science and Technology

SN - 1532-2890

IS - 9

ER -