Video test collection with graded relevance assessments

Weng Qiying, Martin Halvey, Robert Villa

Research output: Chapter in Book/Report/Conference proceedingConference contribution book

Abstract

Relevance is a complex, but core, concept within the field of Information Retrieval. In order to allow system comparisons the many factors that influence relevance are often discarded to allow abstraction to a single score relating to relevance. This means that a great wealth of information is often discarded. In this paper we outline the creation of a video test collection with graded relevance assessments, to the best of our knowledge the first example of such a test collection for video retrieval. To directly address the shortcoming above we also gathered behavioural and perceptual data from assessors during the assessment process. All of this information along with judgements are available for download. Our intention is to allow other researchers to supplement the judgements to help create an adaptive test collection which contains supplementary information rather than a completely static collection with binary judgements.
LanguageEnglish
Title of host publicationCHIIR'16
Subtitle of host publicationProceedings of the 2016 ACM on Conference on Human Information Interaction and Retrieval
Place of PublicationNew York
Pages309-312
Number of pages4
DOIs
Publication statusPublished - 13 Mar 2016
EventACM SIGIR Conference on Human Information Interaction and Retrieval - North Carolina, Chapel Hill, United States
Duration: 13 Mar 201617 Mar 2016

Conference

ConferenceACM SIGIR Conference on Human Information Interaction and Retrieval
CountryUnited States
CityChapel Hill
Period13/03/1617/03/16

Fingerprint

Information retrieval

Keywords

  • video
  • variable
  • graded
  • relevance
  • information retrieval
  • test collection

Cite this

Qiying, W., Halvey, M., & Villa, R. (2016). Video test collection with graded relevance assessments. In CHIIR'16: Proceedings of the 2016 ACM on Conference on Human Information Interaction and Retrieval (pp. 309-312). New York. https://doi.org/10.1145/2854946.2854980
Qiying, Weng ; Halvey, Martin ; Villa, Robert. / Video test collection with graded relevance assessments. CHIIR'16: Proceedings of the 2016 ACM on Conference on Human Information Interaction and Retrieval. New York, 2016. pp. 309-312
@inproceedings{63c0626ac8854abab4ec4e8bbaca83ec,
title = "Video test collection with graded relevance assessments",
abstract = "Relevance is a complex, but core, concept within the field of Information Retrieval. In order to allow system comparisons the many factors that influence relevance are often discarded to allow abstraction to a single score relating to relevance. This means that a great wealth of information is often discarded. In this paper we outline the creation of a video test collection with graded relevance assessments, to the best of our knowledge the first example of such a test collection for video retrieval. To directly address the shortcoming above we also gathered behavioural and perceptual data from assessors during the assessment process. All of this information along with judgements are available for download. Our intention is to allow other researchers to supplement the judgements to help create an adaptive test collection which contains supplementary information rather than a completely static collection with binary judgements.",
keywords = "video, variable, graded, relevance, information retrieval, test collection",
author = "Weng Qiying and Martin Halvey and Robert Villa",
year = "2016",
month = "3",
day = "13",
doi = "10.1145/2854946.2854980",
language = "English",
isbn = "9781450337519",
pages = "309--312",
booktitle = "CHIIR'16",

}

Qiying, W, Halvey, M & Villa, R 2016, Video test collection with graded relevance assessments. in CHIIR'16: Proceedings of the 2016 ACM on Conference on Human Information Interaction and Retrieval. New York, pp. 309-312, ACM SIGIR Conference on Human Information Interaction and Retrieval, Chapel Hill, United States, 13/03/16. https://doi.org/10.1145/2854946.2854980

Video test collection with graded relevance assessments. / Qiying, Weng; Halvey, Martin; Villa, Robert.

CHIIR'16: Proceedings of the 2016 ACM on Conference on Human Information Interaction and Retrieval. New York, 2016. p. 309-312.

Research output: Chapter in Book/Report/Conference proceedingConference contribution book

TY - GEN

T1 - Video test collection with graded relevance assessments

AU - Qiying, Weng

AU - Halvey, Martin

AU - Villa, Robert

PY - 2016/3/13

Y1 - 2016/3/13

N2 - Relevance is a complex, but core, concept within the field of Information Retrieval. In order to allow system comparisons the many factors that influence relevance are often discarded to allow abstraction to a single score relating to relevance. This means that a great wealth of information is often discarded. In this paper we outline the creation of a video test collection with graded relevance assessments, to the best of our knowledge the first example of such a test collection for video retrieval. To directly address the shortcoming above we also gathered behavioural and perceptual data from assessors during the assessment process. All of this information along with judgements are available for download. Our intention is to allow other researchers to supplement the judgements to help create an adaptive test collection which contains supplementary information rather than a completely static collection with binary judgements.

AB - Relevance is a complex, but core, concept within the field of Information Retrieval. In order to allow system comparisons the many factors that influence relevance are often discarded to allow abstraction to a single score relating to relevance. This means that a great wealth of information is often discarded. In this paper we outline the creation of a video test collection with graded relevance assessments, to the best of our knowledge the first example of such a test collection for video retrieval. To directly address the shortcoming above we also gathered behavioural and perceptual data from assessors during the assessment process. All of this information along with judgements are available for download. Our intention is to allow other researchers to supplement the judgements to help create an adaptive test collection which contains supplementary information rather than a completely static collection with binary judgements.

KW - video

KW - variable

KW - graded

KW - relevance

KW - information retrieval

KW - test collection

U2 - 10.1145/2854946.2854980

DO - 10.1145/2854946.2854980

M3 - Conference contribution book

SN - 9781450337519

SP - 309

EP - 312

BT - CHIIR'16

CY - New York

ER -

Qiying W, Halvey M, Villa R. Video test collection with graded relevance assessments. In CHIIR'16: Proceedings of the 2016 ACM on Conference on Human Information Interaction and Retrieval. New York. 2016. p. 309-312 https://doi.org/10.1145/2854946.2854980