PaperlayerView exact revision on arXiv

Source-linked AI summary

DEGREE: A Data-Efficient Generation-Based Event Extraction Model

I-Hung Hsu, Kuan-Hao Huang, Elizabeth Boschee, Scott Miller, Prem Natarajan, Kai-Wei Chang, Nanyun Peng

arXiv:2108.12724v3cs.CLcs.AI

TL;DR

Expensive expert annotations make data-efficient low-resource event extraction an important challenge. DEGREE formulates end-to-end extraction as conditional generation using prompts and natural templates, and reports superior low-resource performance over prior approaches. Its applicability assumes accessible weak supervision, while inference enumerates all event types.

  • Problem

    High-quality event annotations are expensive, so learning event extraction from only a few annotated examples is a crucial challenge.

  • Method

    DEGREE generates natural template-conforming sentences from passages and prompts, using weak supervision and jointly predicting triggers and arguments.

  • Results

    DEGREE achieves significantly better performance than prior approaches on low-resource event extraction.

  • Takeaways & Limitations

    DEGREE demonstrates strong performance for low-resource event extraction by better utilizing label semantics, weak supervision, and dependencies between triggers and arguments.

  • Takeaways & Limitations

    DEGREE assumes weakly supervised information and human-written templates are accessible or inexpensive, and template construction remains manual.

Abstract

from arXiv · show

Event extraction requires high-quality expert human annotations, which are usually expensive. Therefore, learning a data-efficient event extraction model that can be trained with only a few labeled examples has become a crucial challenge. In this paper, we focus on low-resource end-to-end event extraction and propose DEGREE, a data-efficient model that formulates event extraction as a conditional generation problem. Given a passage and a manually designed prompt, DEGREE learns to summarize the events mentioned in the passage into a natural sentence that follows a predefined pattern. The final event predictions are then extracted from the generated sentence with a deterministic algorithm. DEGREE has three advantages to learn well with less training data. First, our designed prompts provide semantic guidance for DEGREE to leverage DEGREE and thus better capture the event arguments. Moreover, DEGREE is capable of using additional weakly-supervised information, such as the description of events encoded in the prompts. Finally, DEGREE learns triggers and arguments jointly in an end-to-end manner, which encourages the model to better utilize the shared knowledge and dependencies among them. Our experimental results demonstrate the strong performance of DEGREE for low-resource event extraction.

1 Introduction

Event extraction identifies event triggers and argument roles, but conventional systems depend on costly expert annotations. DEGREE addresses low-resource end-to-end extraction by generating natural template-conforming summaries from passages and prompts.

  • Event extraction identifies triggers and participants with specific roles, supporting applications such as knowledge graphs and question answering.
  • High-quality event annotations are expensive, making models difficult to extend to new domains and event types.The ACE 2005 corpus requires two annotation rounds by linguistics experts.
  • DEGREE takes a passage and manually designed prompt, then generates a natural sentence following a predefined template.The generated sentence is parsed deterministically to extract event triggers and arguments.
  • Prompt templates provide label semantics and guide predictions by indicating argument entity types and relationships between roles.For example, template words such as “somewhere” guide location predictions, while sentence structure depicts relations between Attacker and Target.
  • DEGREE incorporates event descriptions and keywords as weak supervision and jointly predicts triggers and arguments in an end-to-end framework.
  • Experiments on ACE 2005 and ERE-EN demonstrate strong low-resource performance, with comprehensive ablations examining model strengths and weaknesses.

2 Data-Efficient Event Extraction

DEGREE formulates low-resource event extraction as conditional generation, using designed prompts and natural templates to jointly predict triggers and arguments. Generated outputs are deterministically parsed into event predictions, while template construction and inference over many event types remain practical constraints.

  • The DEGREE Model: DEGREE formulates event extraction as conditional generation, producing a formatted output for each event type from a passage and designed prompt.The generated sentences are compared with E2E templates to identify triggers and arguments, which are then converted to passage span offsets by string matching.
  • The DEGREE Model: Prompt components provide event definitions, keywords, and natural-sentence templates that encode label semantics and relationships among argument roles.The templates also define the expected output format, while natural sentences are intended to help the model leverage pretrained decoder knowledge.
  • The DEGREE Model: E2E templates combine an event-detection template with event-type-specific argument templates whose placeholders represent trigger and argument roles.For example, placeholders such as “some attacker” and “somewhere” correspond to Attacker and Place roles in a Conflict:Attack template.
  • The DEGREE Model: DEGREE trains by replacing template placeholders with gold triggers and arguments, concatenating multiple arguments and retaining placeholders when roles have no prediction.For multiple triggers of one event type, the model generates multiple E2E templates, one for each trigger.
  • Discussion: DEGREE requires human-designed templates and enumerates all event types during inference, creating effort and efficiency concerns for large-scale or many-type applications.Template construction is reported to take about one minute for non-experts, while the efficiency issue is minor on ACE 2005 and ERE-EN but lacks thorough large-scale evaluation.

3 Experiments

Experiments evaluate DEGREE in low- and high-resource event extraction using trigger and argument classification F1-scores, with prompt ablations and additional comparisons. DEGREE is strongest in low-resource settings, while remaining competitive in high-resource settings and achieving state-of-the-art event argument extraction.

  • Experimental Setup: The experiments vary training data proportions from 1% to 50% and evaluate trigger and argument identification and classification using established F1-based criteria.Argument classification F1 is treated as the more important metric for end-to-end comparisons.
  • Low-Resource Results: Both DEGREE and DEGREE(PIPE) outperform all other baselines when using less than 10% of the training data.At 1% of training data, they improve trigger classification F1 by more than 15 points and argument classification F1 by more than 5 points.
  • Low-Resource Results: DEGREE and DEGREE(PIPE) benefit argument prediction more than trigger prediction because argument prediction requires capturing stronger dependencies among arguments.OneIE needs about 50% of the training data to become competitive in argument prediction, compared with 20% for trigger prediction.
  • Low-Resource Results: DEGREE is slightly better than DEGREE(PIPE) under low-resource conditions, providing empirical evidence for jointly predicting triggers and arguments.The comparison supports the reported benefit of end-to-end joint prediction in low-resource settings.
  • High-Resource Results: In high-resource event extraction, DEGREE and DEGREE(PIPE) outperform all baselines except OneIE on trigger classification and slightly outperform OneIE on argument classification in two of three datasets.The advantage of DEGREE over DEGREE(PIPE) becomes less obvious when enough training examples are available.
  • High-Resource Results: DEGREE(EAE) achieves state-of-the-art performance for event argument extraction when gold triggers are provided, while prompt ablations show all three prompt components are necessary.Removing the event type definition, event keywords, or event-detection template decreases event-detection performance on ACE05-E.

4 Ablation Studies

Ablation studies show that prompt components and natural-language templates materially affect DEGREE’s event extraction performance, especially in low-resource settings. Template quality matters, although DEGREE remains superior to the cited baselines.

  • Impacts of components in prompts: Removing any event type definition, event keywords, or event-detection template decreases performance, indicating all three prompt components are necessary.
  • Impacts of components in prompts: Removing any event type definition, query trigger, or argument-extraction template causes performance drops, with query trigger being the most important component.The advantage of these weakly supervised signals becomes more pronounced with less training data.
  • Effects of different template designs: Natural-language templates perform better than alternatives, particularly with 10% of the data, while replacing placeholders with special tokens causes over 1 F1 score drop.This supports leveraging label semantics through natural-language role placeholders.
  • Sensitivity to template design: Different template-construction rules produce performance fluctuations, showing that template quality affects performance to a certain degree.The study varies word choices and role ordering, motivating caution in template design.
  • Sensitivity to template design: Despite sensitivity to template design, DEGREE still outperforms OneIE and BART-Gen in the reported comparison.

5 Related Work

Related work includes supervised, generation-based, low-resource, and meta-learning approaches to event extraction. DEGREE differs by targeting low-resource end-to-end extraction with natural-language generation and joint trigger-argument modeling.

  • Fully supervised event extraction: Traditional event extraction commonly uses fully supervised, classification-based, pipeline-style models, although some incorporate global features to jointly decide triggers and arguments.
  • Generation-based event extraction: TANL, TempGen, and BART-Gen are generation-based approaches, but their unnatural outputs or pipeline designs limit label-semantic utilization or knowledge sharing.
  • Low-resource event extraction: Low-resource approaches include machine-reading-comprehension and sequence-to-structure generation, with Text2Event’s unnatural output format hindering full use of label semantics.
  • Low-resource event extraction: Meta-learning methods address limited labels but are applicable only to event detection, unlike DEGREE’s focus on end-to-end event extraction.

6 Conclusion & Future Work

The paper concludes that DEGREE is a data-efficient generation-based event extraction model whose use of label semantics, weak supervision, and joint prediction supports low-resource extraction. Its deployment assumes accessible prompt information, and automating template construction remains future work.

  • DEGREE requires less training data by using label semantics and weakly supervised information while jointly predicting triggers and arguments.
  • Experimental results and ablation studies show DEGREE’s superiority for low-resource event extraction.
  • DEGREE assumes that event descriptions, similar keywords, and human-written templates are accessible or inexpensive for users to craft.
  • Automating template construction is left for future work to reduce deployment effort in large-scale corpora.

Ethics Considerations

The ethics discussion highlights risks associated with fine-tuning a pretrained generative language model. Generated outputs may reflect pretraining-corpus influences and could include malicious, counterfactual, or biased sentences.

  • Because DEGREE fine-tunes a pretrained generative language model, its generated output may be affected by the pretraining corpus.
  • The model could accidentally generate malicious, counterfactual, or biased sentences, creating potential ethics concerns before real-world deployment.

B Training Details of Proposed Model

The training setup samples unrelated event types as negative examples while fine-tuning DEGREE variants with predefined event-role constraints.

  • Training treats annotated event types as positive examples and samples m unrelated event types as negative examples.In the experiments, m is usually set to 13 or 15.
  • DEGREE, DEGREE(ED), and DEGREE(EAE) are fine-tuned from pre-trained BART-large.The model has approximately 406 million parameters and uses the Huggingface package.
  • The experiments use AdamW with learning rate 10^-5 and weight decay 10^-5.Batch size is 6 for DEGREE(EAE) and 32 for DEGREE(ED) and DEGREE.
  • The negative-example hyperparameter m is searched over {3, 5, 7, 10, 13, 15, 18, 21}, with values below 10 usually less useful.Learning rate and weight decay are tuned from {10^-5, 10^-4} for event argument extraction and fixed to 10^-5 otherwise.
  • Valid argument roles for each event type are predefined by the dataset ontology or selected by the user.

C Datasets

The study evaluates event extraction on English subsets of ACE 2005 and ERE, using document-based training splits and leaving potentially offensive words and real names unchanged.

  • The experiments use ACE 2005 and ERE, both licensed under the LDC User Agreement for Non-Members.
  • Only English data are used from both ACE 2005 and ERE, although the original datasets also contain Chinese data.
  • The datasets are created for entity, relation, and event extraction, while this paper focuses only on event extraction.
  • The datasets are not modified to protect or anonymize offensive words and real names because identifying such information is part of the task.
  • Training data are split by documents rather than instances, and each proportion is designed to contain as many event types as possible.The authors describe this as a more realistic setup.

D Implementation Details

Implementation details cover baseline execution, released resources, and the event-extraction templates and dataset-statistics tables used in the experiments.

  • Experiments with all baselines use three random seeds and report the best value.
  • DyGIE++ is evaluated with its released pre-trained model, while OneIE and BERT_QA are trained using their provided code and default parameters.
  • TANL is evaluated with two settings, and the result from the better-performing setting with more training epochs is reported.
  • Text2Event uses its official code and provided parameters, while dbRNN and Joint3EE results are taken directly from their papers.
  • Tables provide the ACE05-E and ACE05-E+ EAE templates, the ERE-EN EAE templates, and dataset statistics.
  • The dataset-statistics table defines columns for documents, sentences, events, event types, arguments, and argument-role types.

E Few-Shot and Zero-Shot Event Extraction

Zero- and few-shot experiments evaluate DEGREE on unseen or rare ACE05-E event types, with separate trigger, argument, and end-to-end extraction comparisons.

  • Settings: The zero-shot setting removes unseen event types from training, while the k-shot setting retains k examples for each unseen type.Evaluation uses micro F1 only for unseen or rare types.
  • Compared baselines: The compared baselines include BERT_QA, OneIE, matching, and lemmatization, with the latter two used only for event detection.
  • Experimental results: DEGREE(ED) outperforms BERT_QA trained with 10 shots and OneIE trained with 5 shots in the zero-shot trigger-detection setting.
  • Experimental results: DEGREE(ED) exceeds the matching baseline by over 4.7% absolute trigger classification F1 in both n = 5 and n = 10 zero-shot cases.
  • Experimental results: With one training instance per unseen type, DEGREE(ED) outperforms both proposed string-matching baselines.
  • Experimental results: When gold triggers are supplied, DEGREE performs much better than all baselines on event argument extraction.
  • Experimental results: For joint trigger-and-argument extraction, DEGREE shows strong generalizability to unseen event types and outperforms BERT_QA and OneIE.
  • Experimental results: Figure 4 reports trigger classification F1 for detection, argument classification under gold triggers, and argument classification for full event extraction.
Loading 2108.12724v3…