Evaluating the effort involved in relevance assessments for images

Martin Halvey, Robert Villa

Research output: Chapter in Book/Report/Conference proceedingChapter (peer-reviewed)

4 Citations (Scopus)

Abstract

How assessors and end users judge the relevance of images has been studied in information science and information retrieval for a considerable time. The criteria by which assessors' judge relevance has been intensively studied, and there has been a large amount of work which has investigated how relevance judgments for test collections can be more cheaply generated, such as through crowd sourcing. Relatively little work has investigated the process individual assessors go through to judge the relevance of an image. In this paper, we focus on the process by which relevance is judged for images, and in particular, the degree of effort a user must expend to judge relevance for different topics. Results suggest that topic difficulty and how semantic/visual a topic is impact user performance and perceived effort.
LanguageEnglish
Title of host publicationProceedings of the 37th international ACM SIGIR Conference on Research & Development in Information Retrieval (SIGIR '14)
Place of PublicationNew York
Pages887-890
Number of pages4
DOIs
Publication statusPublished - Mar 2014
Event37th International ACM SIGIR Conference on Research and Development in Information Retrieval - Gold Coast, Australia
Duration: 6 Jul 201411 Jul 2014

Conference

Conference37th International ACM SIGIR Conference on Research and Development in Information Retrieval
Abbreviated titleSIGIR '14
CountryAustralia
Period6/07/1411/07/14

Fingerprint

Information science
Information retrieval
Semantics

Keywords

  • information retrieval
  • relevance judgements
  • image retrieval

Cite this

Halvey, M., & Villa, R. (2014). Evaluating the effort involved in relevance assessments for images. In Proceedings of the 37th international ACM SIGIR Conference on Research & Development in Information Retrieval (SIGIR '14) (pp. 887-890). New York. https://doi.org/10.1145/2600428.2609466
Halvey, Martin ; Villa, Robert. / Evaluating the effort involved in relevance assessments for images. Proceedings of the 37th international ACM SIGIR Conference on Research & Development in Information Retrieval (SIGIR '14). New York, 2014. pp. 887-890
@inbook{fda6c4978ff046c987eff714ad7bbfd3,
title = "Evaluating the effort involved in relevance assessments for images",
abstract = "How assessors and end users judge the relevance of images has been studied in information science and information retrieval for a considerable time. The criteria by which assessors' judge relevance has been intensively studied, and there has been a large amount of work which has investigated how relevance judgments for test collections can be more cheaply generated, such as through crowd sourcing. Relatively little work has investigated the process individual assessors go through to judge the relevance of an image. In this paper, we focus on the process by which relevance is judged for images, and in particular, the degree of effort a user must expend to judge relevance for different topics. Results suggest that topic difficulty and how semantic/visual a topic is impact user performance and perceived effort.",
keywords = "information retrieval, relevance judgements, image retrieval",
author = "Martin Halvey and Robert Villa",
year = "2014",
month = "3",
doi = "10.1145/2600428.2609466",
language = "English",
isbn = "9781450322577",
pages = "887--890",
booktitle = "Proceedings of the 37th international ACM SIGIR Conference on Research & Development in Information Retrieval (SIGIR '14)",

}

Halvey, M & Villa, R 2014, Evaluating the effort involved in relevance assessments for images. in Proceedings of the 37th international ACM SIGIR Conference on Research & Development in Information Retrieval (SIGIR '14). New York, pp. 887-890, 37th International ACM SIGIR Conference on Research and Development in Information Retrieval, Australia, 6/07/14. https://doi.org/10.1145/2600428.2609466

Evaluating the effort involved in relevance assessments for images. / Halvey, Martin; Villa, Robert.

Proceedings of the 37th international ACM SIGIR Conference on Research & Development in Information Retrieval (SIGIR '14). New York, 2014. p. 887-890.

Research output: Chapter in Book/Report/Conference proceedingChapter (peer-reviewed)

TY - CHAP

T1 - Evaluating the effort involved in relevance assessments for images

AU - Halvey, Martin

AU - Villa, Robert

PY - 2014/3

Y1 - 2014/3

N2 - How assessors and end users judge the relevance of images has been studied in information science and information retrieval for a considerable time. The criteria by which assessors' judge relevance has been intensively studied, and there has been a large amount of work which has investigated how relevance judgments for test collections can be more cheaply generated, such as through crowd sourcing. Relatively little work has investigated the process individual assessors go through to judge the relevance of an image. In this paper, we focus on the process by which relevance is judged for images, and in particular, the degree of effort a user must expend to judge relevance for different topics. Results suggest that topic difficulty and how semantic/visual a topic is impact user performance and perceived effort.

AB - How assessors and end users judge the relevance of images has been studied in information science and information retrieval for a considerable time. The criteria by which assessors' judge relevance has been intensively studied, and there has been a large amount of work which has investigated how relevance judgments for test collections can be more cheaply generated, such as through crowd sourcing. Relatively little work has investigated the process individual assessors go through to judge the relevance of an image. In this paper, we focus on the process by which relevance is judged for images, and in particular, the degree of effort a user must expend to judge relevance for different topics. Results suggest that topic difficulty and how semantic/visual a topic is impact user performance and perceived effort.

KW - information retrieval

KW - relevance judgements

KW - image retrieval

UR - http://dl.acm.org/citation.cfm?id=2600428

UR - http://dl.acm.org/citation.cfm?id=2609466

U2 - 10.1145/2600428.2609466

DO - 10.1145/2600428.2609466

M3 - Chapter (peer-reviewed)

SN - 9781450322577

SP - 887

EP - 890

BT - Proceedings of the 37th international ACM SIGIR Conference on Research & Development in Information Retrieval (SIGIR '14)

CY - New York

ER -

Halvey M, Villa R. Evaluating the effort involved in relevance assessments for images. In Proceedings of the 37th international ACM SIGIR Conference on Research & Development in Information Retrieval (SIGIR '14). New York. 2014. p. 887-890 https://doi.org/10.1145/2600428.2609466