Source-linked AI summary
Document-Level Event Argument Extraction by Conditional Generation
Sha Li, Heng Ji, Jiawei Han
TL;DR
Sentence-level event extraction does not match document-spanning events and can yield incomplete or uninformative arguments. The paper addresses this with template-conditioned document-level generation, introduces WikiEvents and informative argument extraction, and reports gains across supervised and zero-shot settings. Its framework also demonstrates portability to unseen event types.
Problem
Sentence-level event extraction is limited for events described across documents, producing incomplete and uninformative extractions relative to document-level information seeking.
Method
The paper uses conditional generation from event templates and context to produce filled argument templates without entity recognition or coreference preprocessing.
Results
The model surpasses CRF- and QA-based baselines on conventional and informative argument extraction, while also supporting zero-shot event extraction for unseen types.
Takeaways & Limitations
Template conditioning provides a portable document-level framework that supports long-range informative argument extraction and zero-shot transfer across event types.
Takeaways & Limitations
The authors identify QA-based document-level extraction as needing further investigation because additional context can distract models from the target event.
Abstract
from arXiv · showhide
Event extraction has long been treated as a sentence-level task in the IE community. We argue that this setting does not match human information-seeking behavior and leads to incomplete and uninformative extraction results. We propose a document-level neural event argument extraction model by formulating the task as conditional generation following event templates. We also compile a new document-level event extraction benchmark dataset WikiEvents which includes complete event and coreference annotation. On the task of argument extraction, we achieve an absolute gain of 7.6% F1 and 5.7% F1 over the next best model on the RAMS and WikiEvents datasets respectively. On the more challenging task of informative argument extraction, which requires implicit coreference reasoning, we achieve a 9.3% F1 gain over the best baseline. To demonstrate the portability of our model, we also create the first end-to-end zero-shot event extraction framework and achieve 97% of fully supervised model's trigger extraction performance and 82% of the argument extraction performance given only access to 10 out of the 33 types on ACE.
1 Introduction
The paper argues that sentence-level event extraction misses cross-sentence information and proposes document-level conditional generation with templates, supported by the WikiEvents benchmark and zero-shot framework.
- Motivation: Sentence-level event extraction can produce incomplete and uninformative results because real-world events and information seeking cross sentence boundaries.Cross-sentence examples include arguments separated from triggers and pronouns requiring resolution to informative name mentions.
- Method: The proposed model performs document-level argument extraction by conditionally generating filled event templates from unfilled templates and context.It does not require entity recognition or coreference resolution as preprocessing and can handle long contexts beyond single sentences.
- Motivation: Only 34.5% of arguments detected in the same sentence as the trigger are considered informative.The informative argument extraction task instead seeks the most informative mention in the entire document context.
- Zero-Shot Framework: The template-conditioned model is portable to unseen event types and supports zero-shot transfer when paired with a keyword-based trigger extraction model.The ontology supplies event-role information through templates as an external condition.
- Dataset and Task: WikiEvents is a document-level benchmark with complete event and coreference annotation, introducing informative argument extraction over long-range entity-event relations.The benchmark supports evaluation beyond the nearest argument mention.
2 Method
The method performs document-level argument extraction by conditionally generating filled event templates from document context, then supports zero-shot trigger transfer using keyword-based sequence labeling.
- Argument Extraction Model: Input sequences concatenate the template and document, while generated tokens are copied from either source before argument spans are extracted.Special tokens mark the target event trigger, and the decoder models token probabilities conditioned on prior outputs and encoder input.
- Argument Extraction Model: The model takes an unfilled event template and document context, then generates a filled template with concrete arguments.Templates use argument placeholders and are supplied by the event ontology, avoiding additional question-design effort.
- Argument Extraction Model: The model handles missing and multiple arguments without entity extraction or coreference-resolution preprocessing.When multiple arguments fill one slot, the ground-truth sequence connects them with “and”; generation is restricted to input tokens to prevent hallucination.
- Training and Zero-Shot Transfer: Training minimizes negative log likelihood, while the trigger model learns reference vectors and transition parameters with an orthonormality regularizer.In zero-shot transfer, high-confidence pseudo-labels from cosine similarity supervise a model whose parameters are not class-specific.
- Argument Extraction Model: Clarification statements encode ontology-based entity-type constraints, and candidate outputs are reranked using the probability of their filled templates and type statements.This addresses fluent but type-incompatible generations such as assigning “tax plan” to a person-only Participant role.
- Keyword-Based Trigger Extraction Model: The keyword-based trigger module adapts TapNet and uses class representations derived from event-type keywords within an IO-tag sequence-labeling framework.The simplified IO scheme uses fewer parameters because consecutive triggers of the same event are rare; fully supervised trigger models may be preferable for optimal performance.
3 Benchmark Dataset WIKIEVENTS
WIKIEVENTS is a document-level benchmark with event, argument, and coreference annotation, designed to evaluate both argument extraction and informative argument extraction. Its analysis shows that informative arguments often require document-level inference rather than local sentence-based extraction.
- Evaluation Tasks: WIKIEVENTS evaluates argument extraction and informative argument extraction.The dataset includes coreferential mention clusters and the most informative mention for each cluster.
- Evaluation Tasks: Head F1 measures correctly identified argument head words, while Coref F1 gives full credit to coreferential gold arguments.Argument correctness requires matching reference offsets, and classification additionally requires the correct argument role.
- Evaluation Tasks: Informative argument extraction ranks name mentions above nominal mentions and pronouns, selecting the longest mention when mention types match.The model receives credit only for extracting the most informative mention in the entire document.
- Data Collection: The dataset was built from English Wikipedia articles describing real-world events and related news articles.Articles were collected by following reference links from manually identified event pages.
- Dataset Statistics: WIKIEVENTS follows the KAIROS ontology with 67 event types in a three-level hierarchy, compared with ACE's 33 types in two levels.The dataset therefore has a richer event ontology, especially for argument roles.
- Dataset Analysis: Informative arguments average 68.82 words from their triggers, versus 4.75 words for nearest mentions; only 34.5% of same-sentence arguments are informative.Nearest-mention distances are concentrated near zero, whereas informative-mention distances have a relatively flat long tail.
4 Experiments
The experiments evaluate document-level argument extraction, informative argument extraction, and zero-shot event extraction across RAMS, WIKIEVENTS, ACE, and SM-KBP2020. Results highlight challenges for baseline models and show strong qualitative and transfer behavior for the proposed model.
- Experimental settings: The experiments cover document-level argument extraction, informative argument extraction, and zero-shot event extraction.Evaluation uses RAMS, WIKIEVENTS, ACE, and SM-KBP2020 settings.
- Informative argument extraction: Informative argument extraction is substantially harder than nearest argument extraction for all evaluated models.The task requires selecting the most informative mention across the document rather than the nearest trigger-associated mention.
- Informative argument extraction: CRF baselines struggle with rare roles because separately learned argument tags cannot exploit similarities across roles.The long-tail argument distribution particularly harms performance on rarely seen roles.
- Informative argument extraction: QA models suffer from poor argument identification when multiple answers for one role are close or overlapping.Applying BERT-QA directly at document level also distracts the model with additional context and event ambiguity.
- Qualitative analysis: The proposed model captures all arguments on WIKIEVENTS examples where CRF struggles with rare event types and QA over-generates answers.A qualitative table example shows successful extraction for a relatively rare IdentifyCategorize event, while BERT-QA over-generates for Contact.
5 Related Work
Document-level event extraction extends role filling beyond sentence-level processing, with templates enabling shared modeling across event types and zero-shot transfer. The paper situates its approach against earlier datasets, generative table filling, preprocessing-based methods, trigger-only classification, and question answering.
- Document-Level Event Extraction: Document-level argument extraction remains under-explored because existing datasets are scarce, small, narrowly scoped, or annotate only one event per document.RAMS covers cross-sentence implicit arguments across many event types but annotates one event per document, motivating WIKIEVENTS.
- Generative Event Extraction: The paper’s template-conditioned formulation differs from GRIT’s predefined-table filling by supplying the template with document context as model input.This design supports parameter sharing across event types.
- Zero-Shot Event Extraction: Shared parameters across event types enable zero-shot transfer to new event types.The template is included as part of the model input rather than treated only as an output table.
- Zero-Shot Event Extraction: Earlier zero-shot or few-shot approaches used semantic role labeling or abstract meaning representation to detect mentions before classification.These methods preprocess trigger and argument mentions rather than performing the full extraction directly.
- Zero-Shot Event Extraction: Other prior work reduces event extraction to trigger-only few-shot classification using prototype networks.This line of work examines trigger detection rather than the complete argument extraction task.
- Zero-Shot Event Extraction: Recent zero-shot event extraction work formulates the problem as question answering with different question-design strategies.
6 Conclusion & Future Work
The paper advocates document-level event extraction through a neural argument extraction model and releases WIKIEVENTS with complete event and coreference annotation. It reports broad gains over CRF- and QA-based baselines and demonstrates zero-shot portability, while identifying ontological knowledge as future work.
- Conclusion: The paper proposes the first document-level neural event argument extraction model and releases WIKIEVENTS with complete event and coreference annotation.
- Conclusion: The proposed model surpasses CRF-based and QA-based baselines on both conventional and informative argument extraction tasks.
- Future Work: The authors demonstrate model portability by applying the approach in the zero-shot setting.
- Future Work: Future work will incorporate more ontological knowledge to produce more accurate extractions.