Source-linked AI summary
Induction Networks for Few-Shot Text Classification
Ruiying Geng, Binhua Li, Yongbin Li, Xiaodan Zhu, Ping Jian, Jian Sun
TL;DR
Few-shot text classification must adapt to unseen classes from sparse labeled data, but sample-wise comparison is disrupted by varied expressions within each class. The paper proposes Induction Networks, which combine class-wise induction, dynamic routing, and meta-learning. Across sentiment and dialogue intent datasets, the method outperforms existing state-of-the-art approaches.
Problem
Few-shot text classification needs to recognize unseen classes from small support sets, while sample-wise comparison can be disturbed by varied expressions within a class.
Method
Induction Networks induce generalized class-level representations from support sets by combining dynamic routing with a meta-learning framework.
Results
Induction Networks outperform existing state-of-the-art few-shot text classification models on sentiment and dialogue intent datasets.
Takeaways & Limitations
The experiments support class-wise generalization as an effective approach for few-shot text classification.
Abstract
from arXiv · showhide
Text classification tends to struggle when data is deficient or when it needs to adapt to unseen classes. In such challenging scenarios, recent studies have used meta-learning to simulate the few-shot task, in which new queries are compared to a small support set at the sample-wise level. However, this sample-wise comparison may be severely disturbed by the various expressions in the same class. Therefore, we should be able to learn a general representation of each class in the support set and then compare it to new queries. In this paper, we propose a novel Induction Network to learn such a generalized class-wise representation, by innovatively leveraging the dynamic routing algorithm in meta-learning. In this way, we find the model is able to induce and generalize better. We evaluate the proposed model on a well-studied sentiment classification dataset (English) and a real-world dialogue intent classification dataset (Chinese). Experiment results show that on both datasets, the proposed model significantly outperforms the existing state-of-the-art approaches, proving the effectiveness of class-wise generalization in few-shot text classification.
1 Introduction
Few-shot text classification addresses data scarcity and unseen classes by learning generalized class-level representations from small support sets. Induction Networks combine dynamic routing with meta-learning and outperform existing state-of-the-art models on two datasets.
- Supervised deep learning requires large labeled datasets, making adaptation to new classes costly when annotations are limited.
- Sample-wise comparison is vulnerable to noise from varied expressions within the same class, with larger support sets offering diminishing gains.
- Class-wise induction ignores irrelevant details and encapsulates general semantic information across linguistically varied samples.
- Induction Networks explicitly induce class-level representations from small support sets to address sample-wise diversity in few-shot learning.
- The Induction Module combines dynamic routing with meta-learning, using matrix transformation and routing to generalize to unseen classes.
- Induction Networks outperform current state-of-the-art models on a sentiment benchmark and a real-world dialogue intent classification dataset.
2 Related Work
Prior few-shot learning work includes optimization-based and distance-metric approaches, while capsule networks motivate dynamic routing for generalized class-level representations. Induction Networks apply this idea to few-shot text classification.
- Meta-learning approaches transfer knowledge across auxiliary tasks to help solve target few-shot problems without overfitting.
- Optimization-based Methods: Optimization-based methods learn to update model parameters from few-shot gradients, including Meta Network and SNAIL architectures.
- Distance Metric Learning: Distance-metric methods classify queries using relationships to support examples, including cosine, squared Euclidean, and learned deep distance measures.
- Few-shot text classification research includes lifelong domain embeddings, task clustering, and multi-label classification with known structure.
- Capsule Network: Capsule networks use dynamic routing to encode relationships between parts and wholes, inspiring class-level representation learning from support samples.
3 Problem Definition
Few-shot classification adapts classifiers to unseen classes from only a few labeled examples. Episode-based meta-training simulates this setting by learning from support sets and minimizing loss on query sets.
- 3.1 Few-Shot Classification: Few-shot classification adapts a classifier to new classes unseen during training using only a few labeled examples per class.
- 3.1 Few-Shot Classification: A C-way K-shot problem contains K labeled support examples for each of C unique classes.
- 3.1 Few-Shot Classification: Meta-learning extracts transferable knowledge from training classes so classifiers can use small support sets to classify test examples.
- 3.2 Training Procedure: Episode-based meta-training randomly selects classes, divides their examples into support set S and query set Q, then updates parameters using query loss.
- 3.2 Training Procedure: The training procedure matches test-time few-shot inference by repeatedly constructing meta-episodes from sampled classes and examples.
- 3.2 Training Procedure: The number of possible meta-tasks can be exponentially large, making the training procedure difficult to overfit.
4 The Models
Induction Networks process few-shot text through Encoder, Induction, and Relation Modules. The model encodes texts, induces class representations from support samples through dynamic routing, and compares those class vectors with query vectors.
- The architecture contains Encoder, Induction, and Relation Modules operating within each meta-episode.Figure 1 illustrates a 3-way 2-shot problem with one query example.
- Encoder Module: The Encoder Module uses a bidirectional LSTM with self-attention to convert variable-length text into a fixed-size embedding.The text representation is a weighted sum of LSTM hidden states.
- Induction Module: The Induction Module applies shared transformations and dynamic routing to convert support-sample vectors into class-level representations.Routing iteratively adjusts coupling strengths so class vectors encapsulate the relevant sample features.
- Induction Module: A non-linear squashing function preserves vector direction while reducing magnitude, ensuring routing outputs have length no greater than 1.The function is applied to class candidates during induction.
- Relation Module: The Relation Module measures each query–class correlation using a neural tensor layer, then produces a sigmoid-activated scalar relation score.The model trains these scores with mean square error, assigning 1 to matched pairs and 0 to mismatched pairs.
- Training: All three modules are trained jointly by backpropagation, and the model requires no fine-tuning on unseen classes.Induction and comparison abilities accumulate across training episodes.
5 Experiments
Experiments evaluate Induction Networks on English sentiment and Chinese intent classification, comparing them with established few-shot baselines and analyzing their components and representations.
- Datasets: The evaluation covers the English ARSC sentiment dataset and the real-world Chinese ODIC dialogue-intent dataset.ARSC experiments use multiple product-domain tasks, while ODIC has non-overlapping training and testing labels.
- Experiment Setup: The baselines include metric-based models, SNAIL, and ROBUSTTC-FSL, with accuracy used as the evaluation metric.The metric-based baselines include Matching, Prototypical, Graph, and Relation Networks.
- Overall Performance: Induction Networks achieved the best classification performance in all four ODIC experiments, with a statistically significant advantage over SNAIL at the 99% significance level.The comparison targets few-shot intent classification under the reported ODIC experimental settings.
- Ablation Study: Three dynamic-routing iterations produced the best ARSC ablation result, while replacing the induction or relation modules reduced performance.Additional iterations did not further improve performance, and the authors report that routing automatically adjusts coupling coefficients from the input support set.
- Further Analysis: Matrix transformation made support vectors more separable, and Induction Networks learned more semantically separated query vectors than Relation Networks.The visualizations use t-SNE under the 5-way 10-shot setting and compare transformed versus untransformed support vectors and the two encoders.
6 Conclusion
The Induction Networks induce class-level representations from support sets to address sample-wise diversity in few-shot text classification. Their dynamic-routing module improves generalization to unseen classes and outperforms existing state-of-the-art models.
- Induction Networks induce class-level representations from support sets to address sample-wise diversity in few-shot learning.
- The Induction Module combines dynamic routing with a meta-learning framework to make recognition of unseen classes more general.
- The proposed model outperforms existing state-of-the-art few-shot text classification models.
- Both matrix transformation and the routing procedure contribute consistently to few-shot learning tasks.