Building a large-scale corpus for evaluating event detection on twitter

Andrew J. McMinn, Yashar Moshfeghi, Joemon M. Jose

Research output: Chapter in Book/Report/Conference proceedingConference contribution book

90 Citations (Scopus)

Abstract

Despite the popularity of Twitter for research, there are very few publicly available corpora, and those which are available are either too small or unsuitable for tasks such as event detection. This is partially due to a number of issues associated with the creation of Twitter corpora, including restrictions on the distribution of the tweets and the difficultly of creating relevance judgements at such a large scale. The difficulty of creating relevance judgements for the task of event detection is further hampered by ambiguity in the definition of event. In this paper, we propose a methodology for the creation of an event detection corpus. Specifically, we first create a new corpus that covers a period of 4 weeks and contains over 120 million tweets, which we make available for research. We then propose a definition of event which fits the characteristics of Twitter, and using this definition, we generate a set of relevance judgements aimed specifically at the task of event detection. To do so, we make use of existing state-of-the-art event detection approaches and Wikipedia to generate a set of candidate events with associated tweets. We then use crowdsourcing to gather relevance judgements, and discuss the quality of results, including how we ensured integrity and prevented spam. As a result of this process, along with our Twitter corpus, we release relevance judgements containing over 150,000 tweets, covering more than 500 events, which can be used for the evaluation of event detection approaches.
Original languageEnglish
Title of host publicationProceedings of the 22nd ACM International Conference on Information Knowledge Management
Place of PublicationNew York, NY
Pages409-418
Number of pages10
DOIs
Publication statusPublished - 27 Oct 2013
Event22nd ACM International Conference on Information & Knowledge Management - San Francisco, United States
Duration: 27 Oct 20131 Nov 2013

Conference

Conference22nd ACM International Conference on Information & Knowledge Management
CountryUnited States
CitySan Francisco
Period27/10/131/11/13

Keywords

  • test collection
  • twitter
  • event detection
  • social media
  • crowdsourcing
  • reproducibility

Cite this

McMinn, A. J., Moshfeghi, Y., & Jose, J. M. (2013). Building a large-scale corpus for evaluating event detection on twitter. In Proceedings of the 22nd ACM International Conference on Information Knowledge Management (pp. 409-418). New York, NY. https://doi.org/10.1145/2505515.2505695