An eye-tracking approach to the analysis of relevance judgments on the web: the case of Google search engine

Panagiotis Balatsoukas, Ian Ruthven

Research output: Contribution to journalArticlepeer-review

44 Citations (Scopus)


Eye movement data can provide an in-depth view of human reasoning and the decision-making process, and modern information retrieval (IR) research can benefit from the analysis of this type of data. The aim of this research was to examine the relationship between relevance criteria use and visual behavior in the context of predictive relevance judgments. To address this objective, a multimethod research design was employed that involved observation of participants’ eye movements, talk-aloud protocols, and postsearch interviews. Specifically, the results reported in this article came from the analysis of 281 predictive relevance judgments made by 24 participants using the Google search engine. We present a novel stepwise methodological framework for the analysis of relevance judgments and eye movements on the Web and show new patterns of relevance criteria use during predictive relevance judgment. For example, the findings showed an effect of ranking order and surrogate components (Title, Summary, and URL) on the use of relevance criteria. Also, differences were observed in the cognitive effort spent between very relevant and not relevant judgments. We conclude with the implications of this study for IR research.
Original languageEnglish
Pages (from-to)1728-1746
Number of pages19
JournalJournal of the American Society for Information Science and Technology
Issue number9
Early online date17 Aug 2012
Publication statusPublished - Sept 2012


  • information seeking
  • search engines
  • eye-tracking
  • judgement
  • user studies
  • web
  • google
  • search engine


Dive into the research topics of 'An eye-tracking approach to the analysis of relevance judgments on the web: the case of Google search engine'. Together they form a unique fingerprint.
  • The use of relevance criteria during predictive judgment: an eye tracking approach

    Balatsoukas, P. & Ruthven, I., 22 Oct 2010, ASIS&T '10: Proceedings of the 73rd ASIS&T Annual Meeting on Navigating Streams in an Information Ecosystem. New York, NY, Vol. 47. p. 1-10 10 p. 73

    Research output: Chapter in Book/Report/Conference proceedingConference contribution book

    7 Citations (Scopus)

Cite this