PaperlayerView exact revision on arXiv

Source-linked AI summary

MAVEN: A Massive General Domain Event Detection Dataset

Xiaozhi Wang, Ziqi Wang, Xu Han, Wangyi Jiang, Rong Han, Zhiyuan Liu, Juanzi Li, Peng Li, Yankai Lin, Jie Zhou

arXiv:2004.13590v2cs.CL

TL;DR

Existing event-detection datasets are too small and limited in event-type coverage for robust general-domain research. The paper introduces MAVEN, a large human-annotated Wikipedia dataset with broad event coverage, evaluates representative ED models, and finds that general-domain ED remains challenging.

  • Problem

    Existing ED datasets lack sufficient scale for training and stable benchmarking and cover too few event types for general-domain events.

  • Method

    The paper constructs MAVEN from English Wikipedia and FrameNet, using human annotation and a hierarchical event-type schema, then evaluates representative state-of-the-art ED models.

  • Results

    MAVEN contains 4,480 documents, 118,732 event mentions, and 168 event types; existing ED methods show significant performance drops on MAVEN compared with existing benchmarks.

  • Takeaways & Limitations

    General-domain ED remains challenging, and MAVEN may facilitate further research through analyses of event correlations, hierarchical schemas, and transfer learning.

  • Takeaways & Limitations

    Reproduced-model results may differ from original papers because hyperparameters and training details were manually tuned.

Abstract

from arXiv · show

Event detection (ED), which means identifying event trigger words and classifying event types, is the first and most fundamental step for extracting event knowledge from plain text. Most existing datasets exhibit the following issues that limit further development of ED: (1) Data scarcity. Existing small-scale datasets are not sufficient for training and stably benchmarking increasingly sophisticated modern neural methods. (2) Low coverage. Limited event types of existing datasets cannot well cover general-domain events, which restricts the applications of ED models. To alleviate these problems, we present a MAssive eVENt detection dataset (MAVEN), which contains 4,480 Wikipedia documents, 118,732 event mention instances, and 168 event types. MAVEN alleviates the data scarcity problem and covers much more general event types. We reproduce the recent state-of-the-art ED models and conduct a thorough evaluation on MAVEN. The experimental results show that existing ED methods cannot achieve promising results on MAVEN as on the small datasets, which suggests that ED in the real world remains a challenging task and requires further research efforts. We also discuss further directions for general domain ED with empirical analyses. The source code and dataset can be obtained from https://github.com/THU-KEG/MAVEN-dataset.

1 Introduction

Event detection identifies event triggers and classifies their types, but existing datasets are too small and narrow for general-domain evaluation. MAVEN addresses these gaps with a large, broad-coverage dataset and evaluates current ED models, revealing that general-domain ED remains challenging.

  • Motivation: Event detection identifies trigger words or phrases and classifies the event types they evoke.It is described as an important information-extraction task and a first stage in extracting event knowledge from text.
  • Motivation: Existing ED datasets are insufficient for training and stable benchmarking because they are small and cover limited event types.The cited datasets provide limited coverage of general-domain events, hindering transfer to general applications.
  • MAVEN: MAVEN contains 4,480 annotated documents and 118,732 event mentions, making it twenty times larger than ACE 2005.The dataset also contains 111,611 different events and is described as the largest human-annotated ED dataset at the time.
  • MAVEN: MAVEN contains 168 event types derived from FrameNet and organizes them in a tree-structured hierarchical schema.The schema aims to preserve broad general-event coverage while supporting annotation and future modeling with hierarchy information.
  • Evaluation: The paper reproduces recent state-of-the-art ED models and conducts a thorough evaluation on MAVEN.The introduction presents this evaluation as part of assessing ED performance on the new dataset.
  • Evaluation: Existing ED methods show significant performance drops on MAVEN compared with existing ED benchmarks, indicating that general-domain ED remains challenging.The paper also discusses modeling multiple events, hierarchical schemas, and transfer learning as promising directions.

2 Event Detection Definition

MAVEN adopts event-detection terminology centered on events, event mentions, and triggers. ED is divided into identifying triggers and classifying their event types.

  • Terminology: An event is a specific occurrence involving participants, while an event mention is a sentence describing that event.The same event may appear in multiple mentions within a document.
  • Terminology: An event trigger is the key word or phrase in an event mention that most clearly expresses the event occurrence.Triggers are the textual cues that ED systems identify.
  • Subtasks: ED identifies event triggers from possible candidates and classifies the event types of the identified triggers.These are conventionally treated as Trigger Identification and Trigger Classification.
  • Evaluation: Both ED subtasks are evaluated using micro precision, recall, and F-1 scores.Recent neural methods typically formulate ED as token-level multiclass classification.

3 Data Collection of MAVEN

MAVEN is constructed as a large, broadly covered, human-annotated event detection dataset using Wikipedia, FrameNet-derived event types, and multi-stage annotation. Its design combines scalable candidate generation with hierarchical labeling and quality control.

  • Event Schema Construction: MAVEN derives a hierarchical event schema from FrameNet to improve general-domain event coverage while simplifying annotation.The schema is organized as a tree, with fine-grained types preferred and coarse types used for rare events.
  • Document Selection: 4,480 Wikipedia documents covering 90 of 95 EventWiki topics form MAVEN’s annotated corpus.Documents are selected from event-related articles and filtered for sufficient length and event-related frames.
  • Candidate Selection and Automatic Labeling: Heuristic candidate selection and automatic frame-based labeling reduce annotation effort before human annotators assign event types and merge mentions.The automatic labeling process achieves 52.4% precision and 49.7% recall, saving about half of the overall annotation effort.
  • Human Annotation: Two-stage annotation uses 121 crowd annotators followed by 17 experienced annotators and experts to improve consistency and accuracy.On sampled documents, second-stage event-type and mention-merging accuracies are estimated at 90.1% and 86.0%.

4 Data Analysis of MAVEN

MAVEN substantially expands event coverage and data volume while retaining a real-world long-tail distribution. Its broader schema also makes multiple events and event-type associations within sentences more common and complex than in earlier datasets.

  • Data Size: MAVEN’s scale and coverage are intended to accelerate research on general-domain event detection.The dataset contains substantially more events than existing event detection datasets.
  • Data Distribution: MAVEN’s large scale alleviates event-type data scarcity: 41% of types exceed 500 instances and 82% exceed 100 instances.Only 39% of ACE 2005 event types have more than 100 instances.
  • Data Distribution: MAVEN preserves an inherently long-tail real-world distribution rather than applying data augmentation or balancing during construction.Its hierarchical schema is intended to support future transfer from coarse-grained types to rare fine-grained types.
  • Multiple Events in One Sentence: Multiple events in one sentence are much more common in MAVEN than in ACE 2005 and Rich ERE.The larger number of event types also creates more complex associations between event types.

5 Experiments

Experiments show that MAVEN provides a more stable and demanding benchmark for general-domain event detection, while revealing challenges in data scale, event-type coverage, transfer, and error patterns.

  • 5.2 Experimental Setting: MAVEN’s benchmark setting uses official negative instances and averages results over 10 runs with standard trigger-classification metrics.The reproduced models are evaluated on MAVEN and ACE 2005 using micro precision, recall, and F-1; span-based metrics support sequence-labeling comparisons.
  • 5.3 Overall Experimental Results: Existing ED models perform significantly worse on MAVEN than on ACE 2005, indicating that general-domain event detection remains challenging.The authors report that models perform well on ACE 2005 but achieve unsatisfying performance on MAVEN.
  • 5.3 Overall Experimental Results: CRF-based models improve performance on MAVEN, consistent with the greater prevalence of multiple events in one sentence.BERT+CRF also outperforms DMBERT on MAVEN, supporting further exploration of modeling event correlations.
  • 5.4 Analyses on Data Size and #Event Types: MAVEN’s massive scale enables stable benchmarking, whereas comparisons at existing dataset sizes can be unreliable.The data-size analysis varies the proportion of training documents and compares models trained with different amounts of data.
  • 5.4 Analyses on Data Size and #Event Types: Increasing the number of event types causes significant performance drops, demonstrating the challenge of MAVEN’s broad event coverage.The analysis trains models to classify randomly selected subsets of event types and observes performance as coverage increases.
  • 5.5 Analyses on Transferability: Intermediate pre-training on MAVEN improves ACE 2005 event detection, while direct data augmentation harms performance across the different domains and schemas.The transfer experiments use DMBERT and compare adding MAVEN instances with pre-training on MAVEN before fine-tuning on ACE 2005.
  • 5.6 Error Analysis: Identification mistakes are the most common errors, while models also confuse parent-child, sibling, and frequent event types.These patterns motivate better semantic identification, hierarchical-type handling, and resistance to data imbalance.

6 Related Work

Related work frames event detection as a task with multiple dataset traditions, differing in whether events use predefined types or open-domain representations.

  • ACE-style Event Detection: ACE-style event detection identifies triggers and classifies them into specific event types, using the Rich ERE ontology to broaden but not eliminate coverage limits.The paper contrasts this predefined-schema tradition with other formulations of event extraction.
  • Alternative Dataset Formulations: MUC treats event extraction as slot filling, while TDT and newer datasets use open-domain formulations without predefined event-type classification.Open-domain coverage improves, but the paper notes limits for downstream applications.

7 Conclusion and Future work

The paper concludes that MAVEN alleviates existing data scarcity and coverage problems while exposing the continuing difficulty of general-domain event detection and motivating future research.

  • Conclusion: MAVEN is a massive general-domain event detection dataset designed to alleviate the data scarcity and low coverage of existing datasets.The paper evaluates state-of-the-art models on MAVEN and uses the dataset to support further research.
  • Future Work: Analyses examine modeling multiple event correlations, using hierarchical schemas to distinguish close types, and transfer learning for other event-detection tasks.The paper identifies these as promising directions and plans to extend MAVEN to event argument extraction and event sequencing.

A Hyperparameter Settings and Training Details

This appendix describes the hyperparameter settings and training details used for the implemented event-detection models.

  • Training Details: The section introduces hyperparameter settings and training details for the event-detection models implemented in the experiments.

A.1 BERT-based Models

The appendix documents implementations and hyperparameter choices for BERT-based, MOGANED, DMCNN, and BiLSTM-based event detection models.

  • A.1 BERT-based Models: DMBERT and BERT-CRF use BERTBASE checkpoints, HuggingFace Transformers, AdamW, and mostly shared hyperparameters.Their settings are reported in Table 8.
  • A.1 BERT-based Models: DMBERT marks trigger candidates with [unused0] and [unused1] tokens and uses a larger batch size than the original implementation.The authors attribute the higher reproduced results to these choices.
  • A.1 BERT-based Models: BERT-CRF uses BIO tagging, with B-, I-, and O labels denoting event beginnings, interiors, and other tokens.The corresponding hyperparameters are listed in Table 8.
  • A.1 BERT-based Models: MOGANED is reimplemented with Adam instead of Adam plus L2 norm, and λ is set to 1 rather than 5 after manual tuning.Other settings mostly follow Yan et al. (2019), with additional manual tuning for unspecified hyperparameters; Table 9 reports the settings.
  • A.1 BERT-based Models: DMCNN is reimplemented with Adam instead of ADADELTA and uses alternative word embeddings because the original pretrained embeddings are unavailable.The word-embedding dimension and learning rate were also set because they were unspecified originally; Table 10 reports the settings.
  • A.1 BERT-based Models: BiLSTM and BiLSTM-CRF use Chen et al. (2018) pretrained embeddings and Adam, while BiLSTM-CRF also uses BIO tagging.Their hyperparameters are reported in Table 11.

A.5 Overall Training Details

The appendix reports reproducibility details, model selection procedures, validation results, and mappings used for data augmentation and event-schema construction.

  • A.5 Overall Training Details: Training details include computing infrastructure, model parameter counts, and average runtimes for the evaluated models.These details are summarized in Table 12.
  • A.5 Overall Training Details: Models mostly follow original hyperparameters, with some manual tuning, and are selected using development-set F-1 scores on both datasets.Table 13 presents the validation performance of the models.
  • A.5 Overall Training Details: Table 14 maps event types from parts of ACE 2005 to MAVEN event types for the data augmentation experiments in Section 5.5.The mapping connects source event types with the MAVEN schema used in those experiments.
  • A.5 Overall Training Details: MAVEN organizes its event types in a tree-structured hierarchical schema containing virtual types without annotated instances.Figure 6 presents the schema, while the eight virtual types organize similar event types and are excluded from classification and the count of 168 types.

D Event Types and their corresponding frames

MAVEN defines 168 event types from FrameNet’s event-related frames and organizes them in a hierarchy, with virtual types used only for grouping.

  • D Event Types and their corresponding frames: 168 event types are manually induced from 598 FrameNet event-related frames, but the mapping is not strict because a MAVEN type may cover broader semantics.Table 15 documents the event-type-to-frame correspondences.
  • D Event Types and their corresponding frames: Eight red virtual event types have no annotated instances and only organize similar event types in the hierarchical schema.They do not participate in classification and are excluded from the reported total of 168 event types.
  • D Event Types and their corresponding frames: Several MAVEN event types correspond to multiple FrameNet frames, including Placing, Change, Commerce buy, and Filling.The listed correspondences include Placing scenario, Placing, and Being located; Cause change and Cause change of phase; and Filling with Container focused placing.
  • D Event Types and their corresponding frames: The schema also contains fine-grained types for interpersonal, institutional, and procedural events, such as Defending, Ratification, Submitting documents, and Criminal investigation.These types illustrate the breadth of the event inventory beyond a single event domain.
  • D Event Types and their corresponding frames: The schema includes event types spanning actions, changes, communication, bodily states, commerce, social interaction, and object placement.Examples include Military operation, Statement, Death, Commerce buy, Collaboration, and Placing.
Loading 2004.13590v2…