PaperlayerView exact revision on arXiv

Source-linked AI summary

MAVEN-ERE: A Unified Large-scale Dataset for Event Coreference, Temporal, Causal, and Subevent Relation Extraction

Xiaozhi Wang, Yulin Chen, Ning Ding, Hao Peng, Zimu Wang, Yankai Lin, Xu Han, Lei Hou, Juanzi Li, Zhiyuan Liu, Peng Li, Jie Zhou

arXiv:2211.07342v1cs.CL

TL;DR

Existing event-relation datasets are limited in scale and relation coverage, although the relation types interact. The paper constructs MAVEN-ERE, a unified large-scale dataset with staged annotation, and reports that ERE remains challenging while joint learning can improve performance.

  • Problem

    Existing human-annotated ERE datasets are small and often cover only limited relation types, despite interactions among event relations.

    Source: §1 · pp. 1–2

  • Method

    The authors construct MAVEN-ERE by annotating four event-relation types in shared documents and staging annotation, including a timeline scheme for temporal relations.

    Source: §1 · p. 2

  • Results

    Experiments find ERE on MAVEN-ERE challenging, while jointly considering relation types and better modeling long-range dependency may improve performance.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The findings motivate further research on models for diverse and complex event relations.

    Source: §4.2 · p. 7

    MAVEN-ERE covers only English and within-document event relations, limiting its linguistic coverage and application scope.

    Source: Limitations · p. 9

Abstract

from arXiv · show

The diverse relationships among real-world events, including coreference, temporal, causal, and subevent relations, are fundamental to understanding natural languages. However, two drawbacks of existing datasets limit event relation extraction (ERE) tasks: (1) Small scale. Due to the annotation complexity, the data scale of existing datasets is limited, which cannot well train and evaluate data-hungry models. (2) Absence of unified annotation. Different types of event relations naturally interact with each other, but existing datasets only cover limited relation types at once, which prevents models from taking full advantage of relation interactions. To address these issues, we construct a unified large-scale human-annotated ERE dataset MAVEN-ERE with improved annotation schemes. It contains 103,193 event coreference chains, 1,216,217 temporal relations, 57,992 causal relations, and 15,841 subevent relations, which is larger than existing datasets of all the ERE tasks by at least an order of magnitude. Experiments show that ERE on MAVEN-ERE is quite challenging, and considering relation interactions with joint learning can improve performances. The dataset and source codes can be obtained from https://github.com/THU-KEG/MAVEN-ERE.

1 Introduction

Existing event-relation datasets are limited in scale and relation coverage, while the relations themselves interact. MAVEN-ERE addresses these gaps with a large-scale dataset and staged annotation, and the paper reports benefits from joint learning.

1 Introduction

  • Existing datasets are small and often cover only limited relation types, despite interactions such as causal ordering and temporal containment between related events.The introduction notes that some temporal datasets also restrict annotation to event pairs in adjacent sentences.Source: §1 · pp. 1–2
  • MAVEN-ERE annotates four event-relation types in shared documents, with over an order of magnitude more data than existing datasets in every ERE task.It contains 103,193 coreference chains, 1,216,217 temporal relations, 57,992 causal relations, and 15,841 subevent relations.Source: §1 · p. 2
  • The annotation methodology stages the work: coreference comes first, temporal links use a timeline scheme, and temporal and transitivity constraints narrow later annotation scopes.The timeline scheme produces over six times as many temporal relations per 100 words as MATRES.Source: §1 · p. 2
  • The paper reports that straightforward joint training across the four tasks improves performance.The introduction also frames ERE as a challenging task for data-driven models.Source: §1 · pp. 2–3

2 Dataset Construction

MAVEN-ERE annotates event coreference, temporal, causal, and subevent relations based on event triggers in MAVEN, with a task-specific process for each relation type.

2.1 Coreference Relation

  • Three annotators label each coreference document and majority voting resolves their outputs; average B-Cubed F1 between annotator pairs is 91%.Annotators are trained and qualified, and may flag mentions that do not express events for removal when all annotators agree.Source: §2.1 · p. 3

2 Dataset Construction

2.2 Temporal Relation

  • Temporal annotation sorts event and TIMEX beginnings and endings on timelines, inferring relations from relative positions rather than labeling every event pair.Sub-timelines handle contextually indeterminate relations; this scheme supports all-pair annotation without restricting events to adjacent sentences. The distribution is seriously unbalanced and retained as observed.Source: §2.2 · p. 4

2.3 Causal Relation

  • Causal labels distinguish CAUSE from PRECONDITION, and annotation is limited to event pairs labeled BEFORE or OVERLAP in temporal annotation.Causal and subevent relations are annotated together, with transitivity used to minimize manual labels and restore discarded relations automatically.Source: §2.3 · pp. 4–5

2.4 Subevent Relation

  • A SUBEVENT B means A is a component part of B and spatiotemporally contained by B; annotation is limited to pairs with temporal CONTAINS relations.The containment restriction reduces annotation overhead; three annotators label each document, with majority voting and 75.1% average Cohen’s kappa.Source: §2.4 · p. 5

3 Data Analysis

MAVEN-ERE includes more long-distance temporal and causal relations than existing datasets, while coreference and subevent relations show different distance patterns. Many temporal and causal relations can also be inferred through transitivity.

3.1 Distance between Related Events

  • MAVEN-ERE has much more long-distance temporal and causal relations than existing datasets, supporting real-world applications while posing new challenges for ERE models.Source: §3.1 · p. 6
  • Coreference distances are shorter, with 8.8 event mentions per 100 words versus 1.8 for ACE 2005 and 4.2 for TAC KBP; HiEve and MAVEN-ERE subevent distributions are similar.The paper attributes MAVEN-ERE’s denser short-distance coreference pattern to MAVEN’s broader event coverage and denser event mentions.Source: §3.1 · p. 6

3.2 Relation Transitivity

  • 88.8% of temporal relations and 23.9% of causal relations are transitivity-inferable; more than 60% of temporal relations are inferable in most documents.Source: §3.2 · p. 6, Figure 2 · p. 6

4 Experiments and Analyses

Experiments use RoBERTaBASE baselines to assess MAVEN-ERE and compare independent with joint training. Results show that performance varies with data scale and event distance, while relation identification remains a major error source.

4.1 Experiment Setup

  • RoBERTaBASE encodes whole documents and classifies event-pair triggers; a joint model combines losses across all four tasks alongside independently trained models.The study also evaluates prior datasets; it uses cross-validation for Causal-TB and EventStoryLine and retains causal direction labels.Source: §4.1 · pp. 6–7

4.2 Experimental Result

  • MAVEN-ERE results exceed prior datasets for coreference, causal, and subevent extraction, while temporal performance trails MATRES and TCR, whose local, four-type annotations make the task easier.The authors attribute TB-Dense's lower performance to its 36-document scale; they describe the other three tasks as far from practically usable.Source: §4.2 · p. 7, Table 7 · p. 7, Table 8 · p. 7
  • Joint training brings improvements, especially for causal and subevent extraction, while the authors describe overall ERE performance as challenging and far from practically usable outside coreference.The authors suggest that leveraging interactions among relation types is promising for complex ERE.Source: §4.2 · p. 7

4.3 Analysis on Data Scale

  • Increasing training data yields higher, more stable test performance, with marginal gains at MAVEN-ERE's scale, which the authors consider generally sufficient for training ERE models.Figure 3 tracks RoBERTaBASE test F-1 and coreference MUC across training-data scales; error bars show standard deviations over five runs.Source: Figure 3 · p. 7, §4.3 · p. 8

4.4 Analysis on Distance between Events

  • Longer event distances lower causal and subevent performance, but make temporal relations easier, which the authors attribute to longer temporal distances.Coreference is excluded because its cluster-based evaluation cannot be divided by distance.Source: §4.4 · p. 8, Table 9 · p. 8

4.5 Error Analysis

  • Relation-identification false positives and false negatives account for most errors; only small proportions are fixable by applying transitivity rules.The authors infer that models have learned transitivity rules imperfectly but substantially from large-scale data.Source: §4.5 · p. 8, Table 10 · p. 8

5 Related Work

Prior ERE datasets cover different relation types, but often have limited scale or annotation coverage. Earlier unified schemes span multiple event relations, yet their datasets remain small.

5 Related Work

  • Existing work includes datasets for coreference, temporal, causal, and subevent relations, but datasets are limited in scale and rarely integrate relation types.Some datasets annotate two or three relation kinds.Source: §5 · p. 8
  • O’Gorman et al. (2016) and Hong et al. (2016) offer unified schemes for within-document and cross-document event relations, respectively, but their datasets are small.Source: §5 · p. 9
  • MAVEN-ERE follows O’Gorman et al. (2016)'s guidelines for its construction.Source: §5 · p. 9

6 Conclusion and Future Work

The paper presents MAVEN-ERE as a unified, large-scale dataset for four event-relation tasks. Experiments identify event-relation extraction as challenging, with joint relation modeling and better long-range dependency modeling as potential improvements.

6 Conclusion and Future Work

  • MAVEN-ERE is a unified large-scale dataset for event coreference, temporal, causal, and subevent relations.Source: §6 · p. 9
  • Experiments indicate that real-world event-relation extraction is challenging and may improve through joint consideration of relation types and better long-range dependency modeling.Source: §6 · p. 9
  • Future work will extend the dataset to more scenarios, including additional event-related information and languages.Source: §6 · p. 9

Limitations

MAVEN-ERE is limited to English documents and within-document event relations. The paper identifies both as boundaries for future dataset extensions.

Limitations

  • MAVEN-ERE covers English documents only, limiting the linguistic features represented and the scope of applications built on it.The English-only scope is inherited from the original MAVEN dataset.Source: Limitations · p. 9
  • The dataset covers within-document event relations, not cross-document relations.Source: Limitations · p. 9
  • The authors suggest multilingual transfer learning or native low-resource-language datasets, and extending annotation to cross-document relations.Source: Limitations · p. 9

Ethical Considerations

The authors describe licensing and worker-treatment arrangements for MAVEN-ERE, and report checks for potential risks.

Ethical Considerations

  • MAVEN-ERE uses the CC BY-SA 4.0 MAVEN dataset and CC BY-SA 3.0 Wikipedia corpus, with consent from MAVEN’s authors.Both sources are free for research use.Source: Ethical Considerations · p. 9
  • Annotators were hired through professional data annotation companies and paid agreed salaries for contracted workloads compliant with local regulations.Source: Ethical Considerations · p. 9
  • The authors report no additional risks, citing the absence of private information and the limited need for judgments about social issues; sampled data checks found no risky issues.Source: Ethical Considerations · p. 9

A Dataset Statistics Comparison

The appendix describes the scope used for dataset-statistics comparisons and lists temporal and causal relation transitivity rules.

A Dataset Statistics Comparison

  • Statistics for multilingual datasets are reported only for their English portions, while the TAC KBP comparison follows a setup including 2015–2017 data and two LDC datasets.The detailed statistics are presented in Table 12.Source: A Dataset Statistics Comparison · p. 13

B Transitivity Rules

C Implementation Details

The experiments use RoBERTaBASE-based classifiers, with a separate coreference design and distinct training settings for independent and joint models.

C Implementation Details

  • RoBERTaBASE has 110M parameters and is paired with a two-layer, 150-hidden-dimension perceptron using 0.2 dropout for classification.The other relation tasks use standard cross-entropy; coreference follows Joshi et al. (2019).Source: C Implementation Details · p. 13
  • Independent models use learning rates of 1×10−4 for the classification head and 1×10−5 for the encoder; joint models use 3 × 10−4 and 2×10−5, respectively.The joint loss factors are 0.4, 2.0, 4.0, and 4.0 for coreference, temporal, causal, and subevent relations.Source: C Implementation Details · p. 13
  • The authors manually tune hyperparameters over 10 runs, select them using F-1 scores, and report runtimes ranging from 0.5 hours for subevent ERE to 3.4 hours for joint training.Evaluation uses standard precision, recall, and F-1 scores; coreference metrics follow the cited implementation.Source: C Implementation Details · p. 13

D Data Split Statistics

The appendix documents dataset split choices and discusses MAVEN-ERE’s topic coverage and cross-dataset transfer results.

D Data Split Statistics

  • MAVEN-ERE experiments use the original MAVEN split, while coreference experiments follow prior splits and substitute available LDC datasets for unavailable catalog numbers.TAC KBP 2015 and 2016 data are used for training, 2017 for testing, and 82 training documents are randomly sampled for development.Source: D Data Split Statistics · pp. 13–14
  • Temporal splits follow prior work, causal experiments use 5-fold cross-validation on Causal-TB and EventStoryLine, and HiEve follows prior subevent splits.Split statistics are reported in Tables 14–16 for the relevant experiments.Source: D Data Split Statistics · p. 14, Table 14 · p. 16, Table 15 · p. 16

E Discussions on Genre Diversity

  • MAVEN-ERE inherits 4,480 Wikipedia documents spanning 90 topics, and fine-tuning its pretrained RoBERTaBASE models raises MUC F1 across eight external datasets.Reported increases range from 0.5% on TAC KBP, MATRES, and TCR to 16.0% on Causal-TB; the authors encourage further study of genre gaps.Source: E Discussions on Genre Diversity · p. 14

Related papers

Loading 2211.07342v1…