Source-linked AI summary
Entity, Relation, and Event Extraction with Contextualized Span Representations
David Wadden, Ulme Wennberg, Yi Luan, Hannaneh Hajishirzi
TL;DR
Information extraction systems need to capture both local context and long-range dependencies across sentences. DYGIE++ addresses this with shared span representations built from contextualized encodings and dynamic task-specific graph updates, achieving state-of-the-art results across three extraction tasks and four domains. Its results indicate complementary roles for BERT context and graph propagation, while event extraction remains constrained by the use of predicted entity mentions.
Problem
Information extraction tasks benefit from global cross-sentence context and non-local dependencies, but a unified framework must model these relationships across entities, relations, and events.
Method
DYGIE++ enumerates document spans, encodes them with multi-sentence BERT, and refines shared representations through task-specific message passing on dynamic span graphs.
Results
State-of-the-art results were obtained across named entity, relation, and event extraction on four domains, with relative error reductions of 0.2–27.9%.
Takeaways & Limitations
BERT captures within- and adjacent-sentence context, while graph propagation adds cross-sentence structure and can improve entity disambiguation through coreference links.
Takeaways & Limitations
Event extraction uses predicted entity mentions as argument candidates because gold entity labels are unavailable in the realistic evaluation setting.
Abstract
from arXiv · showhide
We examine the capabilities of a unified, multi-task framework for three information extraction tasks: named entity recognition, relation extraction, and event extraction. Our framework (called DyGIE++) accomplishes all tasks by enumerating, refining, and scoring text spans designed to capture local (within-sentence) and global (cross-sentence) context. Our framework achieves state-of-the-art results across all tasks, on four datasets from a variety of domains. We perform experiments comparing different techniques to construct span representations. Contextualized embeddings like BERT perform well at capturing relationships among entities in the same or adjacent sentences, while dynamic span graph updates model long-range cross-sentence relationships. For instance, propagating span representations via predicted coreference links can enable the model to disambiguate challenging entity mentions. Our code is publicly available at https://github.com/dwadden/dygiepp and can be easily adapted for new tasks or datasets.
1 Introduction
DYGIE++ is a unified multi-task framework that uses contextualized span representations and dynamic graph updates to model local and global context for information extraction. It achieves state-of-the-art performance across named entity, relation, and event extraction.
- Motivation: Global context helps information extraction by connecting non-local dependencies, such as coreference links and entities associated with event triggers.These relationships can disambiguate difficult entity mentions and inform event-trigger prediction.
- Context modeling: BERT can capture relationships within and near sentence boundaries, while dynamic span graphs represent longer-range cross-sentence dependencies.The framework combines both contextualization mechanisms.
- Framework: DYGIE++ enumerates candidate spans, encodes them with contextual language models, and applies task-specific message updates over a span graph.The resulting representations are scored for three information extraction tasks.
- Results: 0.2–27.9% relative error reductions were achieved across tasks and domains, with state-of-the-art results on all tasks and all but one subtask.The experiments cover named entity, relation, and event extraction across four text domains.
2 Task and Model
The model represents within-sentence spans using contextualized token encodings, then dynamically propagates task-specific information among related spans before scoring predictions. Its architecture supports entity, relation, coreference, and event extraction in a shared framework.
- Architecture: DYGIE++ extends a span-based entity and relation model with event extraction, trigger–argument propagation, and multi-sentence BERT encodings.These additions let the framework share contextualized span representations across tasks.
- Span representation: The model constructs all possible within-sentence spans up to a threshold length from document tokens.Named entity labels are predicted for spans, while relations are predicted for span pairs.
- Event extraction: Event extraction predicts token-level triggers and argument roles for candidate spans in each trigger’s sentence.Argument candidates are predicted entity mentions rather than gold entity labels.
- Contextual encoding: BERT encodes each sentence with a sliding window containing a fixed-size neighborhood of surrounding sentences.Span vectors concatenate endpoint token representations with a learned span-width embedding.
- Graph propagation: A dynamically generated span graph updates each representation using likely coreference, relation, or event neighbors.The procedure is trained end-to-end, with task-specific message passing between connected spans.
- Prediction: Task-specific feedforward scoring functions predict labels for individual spans or concatenated span pairs.The same re-contextualized representations support the end tasks.
3 Experimental Setup
The experiments evaluate DYGIE++ across four datasets and compare contextualization variants against prior systems under established task-specific setups. Evaluation uses span- and label-matching criteria, with test-set F1 reported for the principal benchmark tables.
- Datasets: Four datasets—ACE05, SciERC, GENIA, and WLPC—cover news, scientific abstracts, biomedical literature, and wet-lab protocols.The datasets provide annotations for different combinations of entities, relations, events, and coreference.
- Evaluation: Evaluation follows prior state-of-the-art setups, requiring correct labels and spans for entities, span pairs and labels for relations, and offsets or roles for events.The reported benchmark metric is test-set F1.
- Reporting: Table 1 reports test-set F1 for the best models across tasks and datasets, including a four-model trigger-detection ensemble where marked.Statistical significance was tested on a subset of results, with exceptions reported for two ACE05-Event argument metrics.
- Model variations: The study compares BERT plus LSTM, BERT fine-tuning, and task-specific message propagation variants.BERTBASE is used for entity and relation extraction, while BERTLARGE is used for event extraction.
- Comparisons: DYGIE++ is compared with DYGIE for entity and relation extraction and with Zhang et al. (2019) for event extraction.DYGIE uses ELMo with dynamic span graphs, while the event baseline uses ELMo and inverse reinforcement learning.
4 Results and Analyses
DyGIE++ achieves state-of-the-art performance across entity, relation, and event extraction, while analyses show complementary benefits from BERT context and graph propagation. Results also identify task-specific limits and resource trade-offs.
- State-of-the-art Results: The framework establishes new state-of-the-art results on all three high-level extraction tasks and all subtasks except event argument identification.Relative error reductions over previous state-of-the-art models range from 0.2–27.9%.
- Benefits of Graph Propagation: Coreference propagation improves named entity recognition across three domains and also improves relation extraction on SciERC.The largest gains occur on SciERC, which contains frequent long-range coreferences, acronyms, and abbreviations.
- Benefits of Graph Propagation: Relation propagation improves relation extraction over pretrained BERT but not fine-tuned BERT, consistent with relations being confined to single sentences.The authors attribute this pattern to fine-tuned BERT learning within-sentence relationships effectively.
- Benefits of Cross-Sentence Context with BERT: A three-sentence BERT context window produces the best performance across all relation and event extraction tasks.Both BERT model variants benefit from wider context windows.
- Benefits of Graph Propagation: The best event extraction results do not use propagation techniques, and the authors identify higher-order trigger–argument interactions as future work.They hypothesize that propagation is less helpful because trigger–argument relationships are asymmetric.
- Pre-training or Fine Tuning BERT Under Limited Resources: The BERT + LSTM + propagation approach stores gradients on 15 million parameters versus 100 million in BERTBASE.The authors suggest this lower-memory approach may suit limited-resource settings or long-document encoding.
- Importance of In-Domain Pretraining: SciBERT significantly improves performance on scientific datasets including SciERC and GENIA compared with BERT.The results indicate that in-domain unlabeled text can improve pre-training for similar domains.
5 Conclusion
The paper presents DyGIE++ as a plug-and-play framework for information extraction and combines contextual embeddings with graph propagation. The authors report improved performance from combining these approaches and identify higher-order interaction modeling as future work.
- DyGIE++ is a plug-and-play framework intended for application across many information extraction tasks.
- Combining BERT embeddings with graph propagation improves performance compared with using either approach alone.BERT supplies robust multi-sentence representations, while graph propagation adds task- and domain-relevant structure.
- Future work could extend the framework to other NLP tasks and model higher-order interactions such as those in event extraction.
A.1 Dataset statistics
The paper summarizes the datasets used for joint information extraction experiments and reports their dataset statistics.
- Table 8 provides summary statistics for all datasets used in the paper.
- The dataset statistics include counts of entity, relation, event trigger, and event argument categories.
A.2 ACE event data preprocessing and evaluation
The ACE event setup adopts preprocessing that ignores time and value arguments and uses an offsets-plus-event-type criterion for argument identification. Event propagation alternates information exchange between trigger and argument nodes, using role-likelihood-weighted updates and gated representations.
- Data preprocessing: ACE preprocessing ignores time and value arguments, making the paper’s named-entity and event-argument results incomparable with some prior works.The authors release preprocessing code to support exact reproduction and comparison.
- Evaluation: An event argument is counted as correctly identified only when both its offsets and event type are correct.Some prior work requires only correct offsets, so those evaluations are not compared.
- Event propagation: The event graph contains trigger and argument nodes that exchange information about likely event participants.Triggers receive information from likely arguments, while arguments receive information from probable triggers.
- Event propagation: One event-propagation iteration updates triggers and arguments analogously, producing contextualized representations used by the final trigger scorer.The final trigger scorer uses contextualized surrogate spans rather than original token embeddings.
- Event propagation: The model scores candidate argument roles, averages candidate argument embeddings using those likelihoods, and projects the resulting update before applying a gate.The gate controls how much of the update is incorporated into the trigger representation.
C CorefProp visualizations
CorefProp changes named-entity predictions by transferring information across coreferent mentions. Its corrections include resolving missed or overly specific labels, while an antecedent update correctly identifies a difficult acronym as a protein.
- SciERC corrections: CorefProp corrects SciERC errors involving missed entities and overly specific Material or Method predictions.The correct replacement in the latter cases is the more general OtherScientificTerm label.
- SciERC corrections: CorefProp can also introduce overly general labels or missed predictions, although its confusion matrix shows more corrections than mistakes.The figure compares cases changed from incorrect to correct against cases changed from correct to incorrect.
- GENIA example: A coreference update from “v-erbA oncoprotein” enables the model to classify the later “v-erbA” mention as a protein.The visualization shows attention flowing from the antecedent to the acronym.
D Statistical significance of results
The authors estimate result variability by retraining selected configurations across multiple random seeds and reporting means with standard errors. Most tested results are significant, but event argument results are not.
- Significance testing: 5 random seeds provide mean and standard error estimates for a subset of Table 1 results.Trigger-detection ensembles use 3 ensembles because training them is computationally demanding.
- Significance testing: Most evaluated results place the mean more than two standard errors above the previous state of the art and are considered significant.The authors did not perform these tests for every experiment because of the large number of experiments.
- Significance testing: Event argument results are not significant, while trigger classification shows moderate significance.The trigger-classification mean is slightly less than two standard errors above the state of the art.
E Implementation Details
Implementation choices include task-specific optimization schedules, early stopping, model-size selection, and optional constrained decoding. The reported setup generally uses BERTBASE, while BERTLARGE helps one event-extraction configuration.
- Optimization: BERT finetuning uses separate learning rates for task-specific layers and BERT, followed by linear decay after warmup.The task-specific layers use 1 × 10^-3, BERT uses 5 × 10^-5, and warmup lasts 20,000 and 40,000 batches respectively.
- Optimization: Event extraction without finetuning uses SGD with momentum, weight decay, batch size 15, and learning-rate halving when dev F1 stagnates.The initial learning rate is 0.02, momentum is 0.9, and weight decay is 1 × 10^-6.
- Model selection: All models use early stopping based on dev-set loss, while event-model selection also addresses trigger overfitting and ACE05-E domain shift.The final ACE05-E predictions use four models because dev-set stopping was unreliable under these conditions.
- Model selection: BERTLARGE improves event extraction with a final LSTM layer but not other tasks or fine-tuned BERT configurations.The final experiments therefore use BERTBASE except in that event-extraction setting.
- Event decoding: Constrained event decoding does not improve performance because the model learns compatibility between event types and argument roles.The constraint was intended to prevent incompatible assignments such as Prosecutor in a Personnel.Nominate event.
- Reporting: Table 9 reports means and standard errors across repeated models, whereas Table 1 reports the best single run for consistency with prior literature.Arg-ID and Arg-C do not improve state of the art when averaged across five models.