Source-linked AI summary
Event Extraction by Answering (Almost) Natural Questions
Xinya Du, Claire Cardie
TL;DR
Event argument extraction traditionally relies on entity recognition, which can introduce error propagation and limits transfer across related roles. This paper recasts extraction as end-to-end question answering with BERT-based trigger and argument models, outperforming prior methods on ACE 2005 and handling roles unseen during training.
Problem
Existing event argument extraction depends heavily on entity recognition and a multi-step process, with error propagation remaining a concern.
Method
The paper formulates event extraction as question answering using separate BERT-based models for trigger detection and argument extraction.
Results
The framework outperforms prior methods on the ACE 2005 benchmark and extracts event arguments for roles not seen during training.
Takeaways & Limitations
Question answering permits transfer of argument-extraction knowledge across semantically related roles.
Takeaways & Limitations
The framework’s error analysis identifies missing document-level context, argument-boundary errors, and data or lexical sparsity as limitations.
Abstract
from arXiv · showhide
The problem of event extraction requires detecting the event trigger and extracting its corresponding arguments. Existing work in event argument extraction typically relies heavily on entity recognition as a preprocessing/concurrent step, causing the well-known problem of error propagation. To avoid this issue, we introduce a new paradigm for event extraction by formulating it as a question answering (QA) task that extracts the event arguments in an end-to-end manner. Empirical results demonstrate that our framework outperforms prior methods substantially; in addition, it is capable of extracting event arguments for roles not seen at training time (zero-shot learning setting).
1 Introduction
Event extraction identifies event triggers and the arguments filling event-specific roles, but existing approaches depend heavily on entity recognition and can suffer error propagation. The paper formulates event extraction as end-to-end question answering to avoid these weaknesses and support transfer across roles, including unseen roles.
- Task and motivation: Event extraction identifies what is happening, its trigger, and text spans filling semantic roles such as BUYER, SELLER, and ARTIFACT.The paper illustrates this with an ownership-transfer event triggered by “sale.”
- Limitations of prior work: Existing argument extraction commonly identifies entities before assigning roles, so errors can propagate through the multi-step process.Joint models attempt to mitigate this issue, but error propagation still occurs.
- Limitations of prior work: Neural approaches also struggle to exploit similarities between related argument roles across event types, especially when roles have few or no training examples.CONFLICT.ATTACK TARGET and JUSTICE.EXECUTE PERSON both refer to humans affected by an action.
- Proposed paradigm: The paper proposes formulating event extraction as a question answering or machine reading comprehension task.This is presented as a new paradigm for the task.
- Proposed paradigm: The framework uses separate BERT-based QA models for trigger detection and argument extraction, with question templates converting sentences into QA inputs.Trigger detection identifies an action or verb and its event type; argument extraction identifies argument spans.
- Reported contributions: Experiments on ACE 2005 report better trigger and argument extraction than prior methods, while the framework also extracts arguments for roles unseen during training.The authors describe this capability as zero-shot event argument extraction.
2 Methodology
The methodology maps event extraction to two BERT-based QA models driven by question templates. One model detects triggers and event types; the other iteratively extracts argument spans and filters them with a dynamic threshold.
- Framework overview: Two question-template sets prepare BERT inputs for separate trigger-detection and argument-extraction models.The shared format is [CLS] <question> [SEP] <sentence> [SEP].
- Framework overview: BERT_QA_Trigger extracts a single-token event trigger and assigns one event type from a fixed set, while BERT_QA_Arg identifies argument text spans.The argument model receives the input sequence, extracted trigger, and event type, then iteratively identifies candidate arguments.
- Question generation strategies: Trigger questions use fixed templates such as “what is the trigger,” “trigger,” “action,” and “verb.”The framework aims for question-generation strategies that work well and can be implemented quickly across domains.
- Question generation strategies: Argument questions use role names, WH questions based on semantic types, or descriptions from ACE annotation guidelines.Questions can additionally encode trigger information by appending “in <trigger>.”
- Question answering models: The trigger model predicts event types per token, whereas the argument model predicts span start and end offsets using separate output matrices.Trigger inference applies argmax over the event-type distribution.
- Question answering models: Argument training minimizes start-token and end-token losses, assigning no-answer examples to the sequence’s [CLS] token.Both QA models are trained by minimizing negative log-likelihood.
- Argument-span inference: At inference, the system enumerates candidate spans, removes invalid or low-probability spans, computes no-answer scores, and applies a development-set dynamic threshold.The threshold determines how many arguments are retained without adding a hard hyperparameter for that purpose.
3 Experiments
Experiments on ACE 2005 evaluate trigger detection, argument extraction, question-generation strategies, and unseen argument roles. The QA framework performs competitively or better than prior systems while avoiding entity-recognition resources and supporting zero-shot argument extraction.
- Trigger Detection: BERT_QA_Trigger with the best trigger question reaches comparable or better performance than baseline models.A BERT fine-tuning baseline achieves nearly the same performance as its DYGIE++ counterpart.
- Argument Extraction: BERT_QA_Arg with annotation-guideline-based questions significantly outperforms prior work without using entity-recognition resources.The drop from argument identification to argument ID plus classification is around 1%, compared with around 3% for prior multi-step systems.
- Ablation and Ensembling: Dynamic thresholding substantially improves precision and general F1, while ensembling two stronger templates improves both precision and recall.The ensemble combines BERT_QA_Arg models trained with template 2 and template 3 questions.
- Unseen Argument Roles: The framework is evaluated on sentences containing six argument roles seen only at test time, including Vehicle, Artifact, Target, Victim, Recipient, and Buyer.The unseen-role setting keeps 16 of 22 roles for training and reserves 6 roles for testing.
- Question Generation: Adding more semantic information to question templates substantially increases both precision and recall for argument extraction.The comparison progresses from question template 1 to template 2 and then template 3.
4 Further Analysis
Further analysis finds that natural, trigger-aware questions improve event extraction, while errors remain concentrated in missed arguments, complex structures, missing document context, span boundaries, and sparse lexical evidence.
- Question generation strategies: Using “verb” as the trigger-detection question gives BERT_QA_Trigger the best F1, while alternatives show no substantial performance gap.The fixed question and sentence interact semantically, explaining its advantage over BERT FineTune.
- Question generation strategies: Adding “in <trigger>” consistently improves argument-extraction performance by identifying the trigger’s location in the sentence.With predicted triggers, removing this phrase lowers F1 by around 3 percent, and the drop is larger with gold triggers.
- Question generation strategies: Template 3, which uses annotation-guideline descriptions, achieves the best argument-extraction performance in both trigger-information settings.Its questions encode more semantic information about argument roles than templates 1 and 2.
- Error analysis: 54.37% of argument cases produce fewer extracted arguments than the gold data, while only around 14% produce more.This pattern corresponds to higher model precision in the reported results.
- Error analysis: The framework struggles with document-level reasoning, exact argument-span boundaries, lexical sparsity, and complex nested clause structures.Examples include incorrect recipient inference without prior context, merged entity spans, unseen or rare triggers, and incomplete higher-order event extraction.
5 Related Work
Related work spans neural, joint, transition-based, machine-reading-comprehension, and question-generation approaches, while this framework applies natural QA to both trigger and argument extraction without an entity-recognition stage.
- Event extraction: Earlier event-extraction systems use convolutional or recurrent representations, structural dependencies, multitask sharing, adversarial training, or transition-based decoding.Several still require entity recognition during decoding or rely on substantial feature or transition engineering.
- Event extraction: This framework bypasses entity recognition and directly extracts event argument spans through question answering.It differs from approaches that model entity and argument spans while retaining an entity-recognition stage.
- Machine reading comprehension: BERT supplies contextualized representations and attention-based interactions between question tokens and input-sequence tokens.These representations support the framework’s semantic relationships for event extraction and question answering.
- Machine reading comprehension: Prior QA formulations in information extraction often use less diverse or natural templates, whereas this work addresses the more complex combination of trigger detection and argument extraction.The framework’s questions are designed for event-specific roles and triggers.
- Question generation: The paper uses rule-based question generation, including WH-word selection and role descriptions, and notes machine-learning generation as future work for larger ontologies.Its rules use “who” for persons, “where” for places, and “what” for other entity types.
6 Conclusion
The conclusion presents question answering as a new event-extraction paradigm whose natural question strategies improve performance and support roles unseen during training, while broader context remains future work.
- Question answering is introduced as a new paradigm for event extraction, covering trigger detection and argument span extraction.
- More natural questions lead to better performance on both trigger detection and argument span extraction.
- The framework outperforms prior work on ACE 2005 and extracts arguments for roles not seen during training.
- Future work includes incorporating paragraph- or document-level context to improve prediction accuracy.
A Questions Based on Annotation Guidelines
The appendix presents questions derived from annotation-guideline descriptions for argument roles, illustrated with a Business.Merge-Org example.
- Annotation-guideline descriptions are used to formulate questions for each argument role.
- For Business.Merge-Org, the role-specific question is “What is merged?”