1 Search Results for "Inel, Oana"

Document
DOI: 10.4230/OASIcs.LDK.2019.12
Validation Methodology for Expert-Annotated Datasets: Event Annotation Case Study

Authors: Oana Inel and Lora Aroyo

Published in: OASIcs, Volume 70, 2nd Conference on Language, Data and Knowledge (LDK 2019)

Abstract
Event detection is still a difficult task due to the complexity and the ambiguity of such entities. On the one hand, we observe a low inter-annotator agreement among experts when annotating events, disregarding the multitude of existing annotation guidelines and their numerous revisions. On the other hand, event extraction systems have a lower measured performance in terms of F1-score compared to other types of entities such as people or locations. In this paper we study the consistency and completeness of expert-annotated datasets for events and time expressions. We propose a data-agnostic validation methodology of such datasets in terms of consistency and completeness. Furthermore, we combine the power of crowds and machines to correct and extend expert-annotated datasets of events. We show the benefit of using crowd-annotated events to train and evaluate a state-of-the-art event extraction system. Our results show that the crowd-annotated events increase the performance of the system by at least 5.3%.
Cite as

Oana Inel and Lora Aroyo. Validation Methodology for Expert-Annotated Datasets: Event Annotation Case Study. In 2nd Conference on Language, Data and Knowledge (LDK 2019). Open Access Series in Informatics (OASIcs), Volume 70, pp. 12:1-12:15, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2019)

Copy BibTex To Clipboard

@InProceedings{inel_et_al:OASIcs.LDK.2019.12,
  author =	{Inel, Oana and Aroyo, Lora},
  title =	{{Validation Methodology for Expert-Annotated Datasets: Event Annotation Case Study}},
  booktitle =	{2nd Conference on Language, Data and Knowledge (LDK 2019)},
  pages =	{12:1--12:15},
  series =	{Open Access Series in Informatics (OASIcs)},
  ISBN =	{978-3-95977-105-4},
  ISSN =	{2190-6807},
  year =	{2019},
  volume =	{70},
  editor =	{Eskevich, Maria and de Melo, Gerard and F\"{a}th, Christian and McCrae, John P. and Buitelaar, Paul and Chiarcos, Christian and Klimek, Bettina and Dojchinovski, Milan},
  publisher =	{Schloss Dagstuhl -- Leibniz-Zentrum f{\"u}r Informatik},
  address =	{Dagstuhl, Germany},
  URL =		{https://drops-dev.dagstuhl.de/entities/document/10.4230/OASIcs.LDK.2019.12},
  URN =		{urn:nbn:de:0030-drops-103762},
  doi =		{10.4230/OASIcs.LDK.2019.12},
  annote =	{Keywords: Crowdsourcing, Human-in-the-Loop, Event Extraction, Time Extraction}
}
  • Refine by Author
  • 1 Aroyo, Lora
  • 1 Inel, Oana

  • Refine by Classification
  • 1 Computing methodologies → Machine learning
  • 1 Human-centered computing → Empirical studies in HCI
  • 1 Information systems → Crowdsourcing

  • Refine by Keyword
  • 1 Crowdsourcing
  • 1 Event Extraction
  • 1 Human-in-the-Loop
  • 1 Time Extraction

  • Refine by Type
  • 1 document

  • Refine by Publication Year
  • 1 2019

Questions / Remarks / Feedback
X

Feedback for Dagstuhl Publishing


Thanks for your feedback!

Feedback submitted

Could not send message

Please try again later or send an E-mail
© 2023-2024 Schloss Dagstuhl – LZI GmbH Imprint Privacy Contact