Source-linked AI summary
Meta-Learning with Dynamic-Memory-Based Prototypical Network for Few-Shot Event Detection
Shumin Deng, Ningyu Zhang, Jiaojian Kang, Yichi Zhang, Wei Zhang, Huajun Chen
TL;DR
Event detection typically requires large labeled datasets, although real applications may contain few examples and new event types. The paper reformulates this setting as few-shot event detection and introduces DMB-PN, which uses dynamic memory to improve prototypes and sentence encodings. Experiments report better performance under sample scarcity, especially with many event types and extremely few instances.
Problem
Conventional event detection relies on large labeled datasets, but real applications may have few examples and frequently emerging event types.
Method
DMB-PN combines a prototypical network with a Dynamic Memory Network to learn event prototypes and robust sentence encodings through multi-hop contextual processing.
Results
DMB-PN outperforms baseline models and is especially robust when event-type variety is large and the instance quantity is extremely small.
Takeaways & Limitations
Dynamic memory enables the model to distill contextual information from event instances multiple times rather than relying on one-time prototype averaging.
Takeaways & Limitations
The paper limits its scope by excluding data augmentation with external pretrained knowledge and leaves application to other few-shot tasks for future work.
Abstract
from arXiv · showhide
Event detection (ED), a sub-task of event extraction, involves identifying triggers and categorizing event mentions. Existing methods primarily rely upon supervised learning and require large-scale labeled event datasets which are unfortunately not readily available in many real-life applications. In this paper, we consider and reformulate the ED task with limited labeled data as a Few-Shot Learning problem. We propose a Dynamic-Memory-Based Prototypical Network (DMB-PN), which exploits Dynamic Memory Network (DMN) to not only learn better prototypes for event types, but also produce more robust sentence encodings for event mentions. Differing from vanilla prototypical networks simply computing event prototypes by averaging, which only consume event mentions once, our model is more robust and is capable of distilling contextual information from event mentions for multiple times due to the multi-hop mechanism of DMNs. The experiments show that DMB-PN not only deals with sample scarcity better than a series of baseline models but also performs more robustly when the variety of event types is relatively large and the instance quantity is extremely small.
1 INTRODUCTION
The paper reformulates event detection with limited labeled data as few-shot event detection and proposes DMB-PN to learn robust event prototypes and sentence encodings. It reports stronger robustness under scarce instances and many event types than baseline approaches.
- Motivation: Few-shot event detection addresses event types that have few labeled examples and may emerge after training, challenging conventional supervised methods.Existing ED systems rely on large labeled datasets, while some ACE-2005 event types have fewer than 20 instances.
- Problem Formulation: The paper formulates ED with limited labels as a meta-learning problem in which models adapt from support examples to classify query instances, including unseen event types.The formulation uses an Event Type Learner trained on meta-training event types and adapted with only a few examples for new types.
- Task Illustration: Few-shot event detection examples identify trigger words in support and query sets, with labels for support mentions and predictions for query mentions.The example uses a 3-shot setting to illustrate the task.
- Proposed Method: DMB-PN integrates a Dynamic Memory Network with a prototypical network to learn event-type prototypes and more robust sentence encodings.Its multi-hop mechanism repeatedly distills contextual information from event mentions instead of averaging each encoding only once.
- Contributions: The experiments report that DMB-PN outperforms baseline models, particularly when event-type variety is large and the number of instances is extremely small.The reported advantage is attributed to repeatedly and incrementally distilling contextual information from event instances.
2 RELATED WORK
Prior event-extraction methods commonly depend on engineered features, NLP tools, or large labeled datasets, motivating few-shot approaches for data-scarce settings. Related few-shot NLP work is limited and mainly uses metric-based methods such as prototypical networks.
- Event Extraction: Traditional event-extraction systems rely on elaborately designed features, complex NLP tools, or large-scale labeled event datasets.
- Few-Shot NLP: Few-shot NLP research remains limited and is mostly based on metric-based methods, including prototypical networks for relation classification and adaptive metric learning for text classification.
- Positioning: This paper uses a prototypical network as a metric-based method for few-shot event detection without external pretrained knowledge or data augmentation.The stated scope focuses on few-shot models rather than augmentation with external pretrained knowledge.
3 METHOD
DMB-PN formulates few-shot event detection through trigger identification and event classification, using dynamic-memory encoders and a memory-based prototypical network for support and query representations.
- Problem Formulation: Few-shot event detection uses N-way-K-shot tasks with labeled support examples and unlabeled query instances, including unseen event types at testing.Each event type has K labeled support samples, and prediction relies on observing those samples.
- General Architecture: DMB-PN divides event detection into trigger identification and few-shot event classification, addressing whether words are triggers and how mentions contribute to prototypes.The model overview explicitly separates these two subtasks and assigns each a question-like interpretation.
- Trigger Identification: The trigger-identification encoder vectorizes words, uses dynamic memory to identify triggers, and produces sentence encodings for subsequent processing.Word representations combine pretrained semantic embeddings with relative-position embeddings before dynamic-memory processing.
- Few-Shot Event Classification: For few-shot classification, DMB-PN generates support and query encodings with a dynamic-memory architecture, then predicts event types by comparing query encodings with prototypes.The prototypical network serves as the DMN answer module, and each query type receives a probability.
- Few-Shot Event Classification: Unlike traditional prototypical networks that average support embeddings, the memory-based mechanism produces event prototypes through repeated contextual processing.The design uses event mentions as questions and combines support-set sentence encodings through a Bi-GRU before memory-based prototype construction.
4 EXPERIMENTS
The experiments evaluate DMB-PN for few-shot event detection using the newly created FewEvent dataset, with varying event-type and shot configurations.
- The experiments compare models across K-shot and N-way settings to assess memory-based approaches under different event-type and instance quantities.
- FewEvent contains 70,852 instances spanning 19 event types and 100 event subtypes, with about 700 instances per event type on average.
- The dataset was constructed by expanding event types from ACE-2005 and TAC-KBP-2017 and adding types from automatically labeled data, Freebase, and Wikipedia.
- Training, validation, and testing use 80, 10, and 10 event types respectively, with no overlap between training and testing types.
4.2 Baselines and Settings
The study compares combinations of sentence encoders and metric-learning models under fixed training settings, evaluating event detection with accuracy and F1 score.
- Sentence-encoder baselines include CNN, Bi-LSTM, Self-Attention, and DMN, while metric models include Matching Networks, Prototypical Networks, and memory-based Prototypical Networks.
- Training uses stochastic gradient descent for 30,000 iterations, with testing performed for 2,000 iterations.
- The models use 50-dimensional memory units and word embeddings, 30-dimensional position embeddings, and three memory-module passes.
- Event-detection performance is evaluated using Accuracy and F1 Score.
4.3 General Comparisons
General comparisons show that Prototypical Networks generally outperform Matching Networks, while DMB-PN achieves the best result among the compared models.
- With the same sentence encoder, Prototypical Networks outperform Matching Networks in almost all N-way-K-shot settings.The paper relates this difference to instance-to-prototype matching being less susceptible to support-set outlier noise than instance-to-instance matching.
- Prototype learning reduces noise from instance randomness by comparing queries with event prototypes instead of directly matching support and query instances.
- DMB-PN achieves the best result among the compared combinations of sentence encoders and metric models.The paper attributes this to repeatedly distilling contextual information from support instances rather than simply averaging them once.
4.4 K-Shot Evaluations
K-shot evaluations indicate that DMB-PN becomes especially advantageous as the number of examples decreases, while DMN-based encoders remain robust in low-shot settings.
- DMB-PN achieves the best accuracy margin in both 5-way and 10-way settings, and its margin increases as the shot number decreases.
- +15.38 is the reported accuracy margin for DMB-PN with a DMN-based encoder and the same metric model.
- DMN-based models show increasing margins as the shot number decreases, unlike models using other sentence encoders.The paper associates this robustness with the multi-hop mechanism's ability to exploit limited instances repeatedly.
- Table 1 reports Accuracy, F1 Score, and accuracy margins for combinations of sentence encoders and metric models.
- Figure 4 examines accuracy as the number of event types increases under fixed 5-shot and 15-shot settings.
4.5 N-Way Evaluations
As the number of ways increases with shot count fixed, accuracy decreases, but memory-based prototypical models remain more robust than vanilla prototypical and matching networks.
- 4.5 N-Way Evaluations: Accuracy decreases as the way number increases when the shot number is fixed.
- 4.5 N-Way Evaluations: CNN-MPN outperforms vanilla prototypical networks, which outperform matching networks in the evaluated N-way settings.
- 4.5 N-Way Evaluations: Memory-based prototypical networks are more robust to increasing way numbers because multi-hop memory learning produces more distinguishable event prototypes.The margins between memory-based models and other metric-based baselines increase as the way number grows.
- 4.5 N-Way Evaluations: Figure 5 visualizes five event prototypes, same-type events by color, support instances, and a red query instance in a 5-way-15-shot task.
4.6 Case Study
The case study examines DMB-PN’s event prototypes and attention patterns, showing distinguishable prototype distances and high attention to event triggers.
- 4.6 Case Study: DMB-PN makes a query instance’s distance to its correct event prototype distinguishable even when surrounding instance distances are similar.The visualization illustrates why the query can be assigned to Die through its closer distance to the Die prototype.
- 4.6 Case Study: Figure 6 compares word attentions for support and query instances across training and testing, with lighter green indicating lower attention.
- 4.6 Case Study: DMB-PN assigns higher attention to event triggers than to other words during both training and testing.Only the five words with the largest attention values are colorized in the visualization.
4.7 Parameter Analysis
The parameter analysis evaluates the trigger-identification loss ratio λ and finds that performance peaks at λ=0.5 before declining.
- 4.7 Parameter Analysis: Figure 7 reports DMB-PN trigger-identification accuracy across different few-shot tasks as λ varies.
- 4.7 Parameter Analysis: Trigger-identification performance increases and then decreases as λ rises, achieving its best performance at λ=0.5.The authors therefore select 0.5 as the hyperparameter value.
- 4.7 Parameter Analysis: DMB-PN jointly trains trigger identification and few-shot event classification, allowing the two objectives to influence one another.The final results combine both objectives, motivating a median loss ratio for trigger identification.
5 CONCLUSION
The paper proposes DMB-PN for few-shot event detection, combining trigger identification with dynamic-memory-based event classification. Experiments support its advantage under limited labeled data, especially with many event types and few instances.
- 5 CONCLUSION: DMB-PN combines trigger identification with dynamic-memory-based prototypical classification using multi-hop contextual encodings.The first stage locates triggers and produces memory-augmented sentence encodings; the second classifies query event types.
- 5 CONCLUSION: Experiments show dynamic-memory-based approaches are more feasible than other sentence-encoding baselines when labeled instances are limited.The reported advantage is especially pronounced when event-type variety is large and instance quantity is small.
- 5 CONCLUSION: Future work will apply DMB-PN to other few-shot tasks, including few-shot relation extraction.