Source-linked AI summary
Few-shot Text Classification with Distributional Signatures
Yujia Bao, Menghua Wu, Shiyu Chang, Regina Barzilay
TL;DR
Few-shot text classification is difficult because lexical features that matter for one task may not transfer to another. The paper maps distributional signatures to attention scores within a meta-learning framework, then uses that attention to weight lexical representations. Across five text-classification and one relation-classification dataset, the model consistently outperforms baselines, including prototypical networks by 20.6% on average in one-shot text classification.
Problem
Lexical features can be highly informative for one text classification task but insignificant for another, limiting direct transfer of meta-learned representations.
Method
An attention generator maps distributional signatures to word-importance scores, and a per-episode ridge regressor uses the weighted lexical representations for few-shot prediction.
Results
The model achieves significant gains over baselines, including 20.6% average improvement over prototypical networks in one-shot text classification and 17.3% in one-shot relation classification.
Takeaways & Limitations
The results support meta-learning with distributional signatures as a way to identify important words for new classes in few-shot text and relation classification.
Takeaways & Limitations
The robustness guarantee assumes word substitutions preserve unigram probabilities and form a bijection over the vocabulary.
Abstract
from arXiv · showhide
In this paper, we explore meta-learning for few-shot text classification. Meta-learning has shown strong performance in computer vision, where low-level patterns are transferable across learning tasks. However, directly applying this approach to text is challenging--lexical features highly informative for one task may be insignificant for another. Thus, rather than learning solely from words, our model also leverages their distributional signatures, which encode pertinent word occurrence patterns. Our model is trained within a meta-learning framework to map these signatures into attention scores, which are then used to weight the lexical representations of words. We demonstrate that our model consistently outperforms prototypical networks learned on lexical knowledge (Snell et al., 2017) in both few-shot text classification and relation classification by a significant margin across six benchmark datasets (20.0% on average in 1-shot classification).
1 INTRODUCTION
Meta-learning transfers well in vision but struggles in text because task-specific lexical features are not reliably transferable. The paper addresses this by transferring attention through distributional signatures, achieving strong few-shot classification gains.
- Motivation: Vision meta-learning succeeds in low-resource settings, including 99.6% accuracy with one example per class on character recognition.The approach recreates few-shot training episodes so models learn to generalize to new classes.
- Motivation: Text poses a transfer challenge because words highly informative for one class may be irrelevant for another, causing lexical meta-learners to underperform.In HuffPost headlines, traditional prototypical networks may attend to words such as “date” while downplaying predictive words such as “grandma.”
- Approach: The proposed method transfers attention through distributional signatures rather than directly through words, then uses that attention to weight lexical representations.These signatures capture characteristics of underlying word distributions and are intended to behave consistently across tasks.
- Approach: The model combines an attention generator with a ridge regressor that adapts from a few examples, while the generator is shared across episodes.The regressor is trained from scratch per episode, and its prediction loss supervises the attention generator.
- Results: 20.6% average improvement over prototypical networks is reported for one-shot text classification, alongside 17.3% for one-shot relation classification.The evaluation covers five text classification datasets and one relation classification dataset.
2 RELATED WORK
Prior meta-learning methods exploit transferable visual features, but lexical features in language do not generalize reliably across classes. This work instead transfers attention using shared distributional behavior and learns that transfer from downstream performance.
- Existing approaches: Meta-learning methods developed for vision include metric learning, optimization priors, and relations between classes, with some adaptations to NLP.Their success in vision relies on low-level features that transfer across classes.
- Limitation: Lexicon-aware meta-learners fail to generalize on standard multi-class text classification datasets because image and language data differ in transferable knowledge.The passage characterizes this as an innate difference between the modalities.
- Proposed perspective: Although salient text features may not transfer, their distributional behaviors are alike, allowing the model to identify important features for novel classes.The work focuses on learning the connection between word importance and distributional signatures.
- Proposed perspective: Unlike approaches with pre-designed transfer mechanisms, this method learns to transfer attention according to downstream-task performance.It uses distributional statistics to transfer attention across tasks.
3 BACKGROUND
The few-shot framework trains models through episodes that mimic adaptation to new classes. Each episode separates support, query, and source examples, while the proposed extension uses a source pool to exploit broader training data.
- Problem statement: The problem is to learn from labeled Ytrain classes and predict for disjoint Ytest classes with only a few annotations.The target classes are new but related to the training classes.
- Meta-training: During meta-training, each episode samples N classes, K support examples, and L query examples, then updates the model using query loss.This setup is called N-way K-shot classification.
- Meta-testing: Meta-testing applies the same episode mechanism to N new classes and evaluates average query-set performance across testing episodes.The support set supplies the few annotations used for adaptation.
- Episode structure: A single illustrated episode uses N = 3, K = 1, and L = 2, with separate support, query, and source-pool examples.Rectangles represent input examples and their labels.
- Model extension: The extension augments episodes with a source pool containing examples outside the selected training classes, or all Ytrain examples during meta-testing.It uses distributional statistics over broader training data for more robust inference.
4 METHOD
The method combines distributional signatures from the source pool and support set with meta-learned attention to emphasize task-relevant lexical information. A ridge regressor uses these representations for few-shot prediction, while episode-level loss trains the shared attention generator.
- Attention generator: The model extracts distributional statistics from the source pool and support set, then uses an attention generator to produce word-level attention.The generated attention provides an inductive bias for word importance and is intended to improve few-shot classification representations.
- Model components: The attention generator is shared across episodes, whereas the ridge regressor is trained from scratch for each episode using support-set examples.The ridge regressor's prediction loss supervises the attention generator during meta-training.
- Distributional signatures: Distributional signatures include general word importance from source-pool unigram statistics and class-specific importance estimated from the support set.The class-specific statistic uses conditional label likelihood and entropy, assigning higher weight to words with skewed class distributions.
- Attention generator: A bi-directional LSTM fuses the two noisy, complementary signatures before dot-product attention produces the final word weights.The attention score is computed from the biLSTM output and a learnable vector.
- Ridge regressor: The ridge regressor constructs attention-weighted lexical representations, fits the support set, predicts on the query set, and backpropagates query loss to the attention generator.Its closed-form, regularized solution supports end-to-end differentiation through the model.
- Theoretical analysis: Theoretical analysis shows attention-generator outputs remain invariant under word substitutions that preserve source-pool unigram probabilities.The theorem applies to the support and query inputs within an episode under the specified perturbation.
5 EXPERIMENTAL SETUP
The study evaluates few-shot text and relation classification across six datasets using 5-way 1-shot and 5-way 5-shot episodes. It compares the proposed model with supervised and meta-learning baselines under fixed experimental procedures and reports results with repeated testing episodes.
- Datasets: FewRel contains sentences annotated with head entities, tail entities, and relations, with 80 relation types in the public dataset.The task is to predict the relation between the head and tail entities.
- Baselines: The experiments use average, inverse-document-frequency, and CNN representations, together with nearest-neighbor, fine-tuning, MAML, prototypical-network, and ridge-regression algorithms.The proposed model is denoted OUR and is compared with different representation-and-learning-algorithm combinations.
- Evaluation protocol: Results are reported for 5-way 1-shot and 5-way 5-shot classification on six datasets, with ablations included in the bottom four rows of Table 1.Complete results with standard deviations are reported in additional appendix tables.
- Implementation: The experiments use pre-trained fastText embeddings for all models, with additional BERT experiments for sentence-level datasets and positional embeddings for FewRel.Training uses fixed word embeddings in the main experiments.
- Evaluation protocol: The protocol samples 100 training episodes per epoch, evaluates 1000 test episodes, averages accuracy over five random seeds, and applies validation-based early stopping.Early stopping occurs when validation loss fails to improve for 20 epochs.
6 RESULTS
Across few-shot classification experiments, the model consistently outperforms baselines and generalizes better to unseen classes. Analyses attribute this performance to task-specific attention learned from distributional signatures rather than lexical information alone.
- Classification results: 20.0% average improvement over CNN+PROTO in 1-shot classification accompanies consistent gains across datasets and better generalization to unseen classes.The model improves 5-way 1-shot accuracy by 7.5% and 5-way 5-shot accuracy by 3.9% against the best baseline for each dataset.
- Classification results: 3.9% average improvement in 5-shot accuracy is reported against the best baseline for each dataset.
- Ablation study: Both distributional statistics contribute to performance, with s(·) having the larger impact; an biLSTM also slightly improves over independent word scoring.
- Ablation study: Restricting the attention generator to distributional signatures is essential: adding lexical word embeddings causes consistent performance drops.
- Contextualized representations: BERT significantly improves FewRel classification but provides no performance boost on HuffPost.The authors associate this discrepancy with FewRel being highly contextual and news classification being mostly keyword-based.
- Representation analysis: The attention-weighted representation produces better separation than general or class-specific word importance, supporting learning from few examples.The analysis attributes the benefit to representations learned on top of distributional signatures.
- Attention analysis: Task-specific attention downweights “statistical” among economics-related classes but upweights it when jobs is compared with distant classes.
7 CONCLUSION
The paper proposes meta-learning that converts distributional statistics into attention for fast adaptation in few-shot text and relation classification. Experiments validate that the approach identifies important words for new classes and demonstrates the promise of distributional signatures.
- The model learns an attention generator that translates distributional statistics into attention guiding fast adaptation to new classification tasks.
- Experimental results on text and relation classification validate that the model identifies important words for new classes.
- The results demonstrate the promise of meta-learning with distributional signatures for few-shot classification.
A.1 REGULARIZED LINEAR CLASSIFIER
The regularized linear classifier uses a simple, quickly trainable model to estimate task-specific word importance from few-shot support examples. Ridge regression and regularization address the small-support-set setting.
- The classifier approximates task-specific word importance from an N-way K-shot support set.
- Averaged word embeddings provide the input representation for each example.
- A simple linear classifier reduces overfitting when the support set contains only a few examples.
- The model predicts with softmax(Wψ(x)) while minimizing cross-entropy loss with Frobenius-norm regularization.
- The classifier converges very quickly in practice because the support set has fewer than 25 examples and the loss is strongly convex.
A.2 PROOF OF THEOREM 1
Theorem 1 establishes that the attention generator is invariant to bijective word perturbations that preserve unigram probabilities. The proof shows invariance for both general and class-specific word-importance statistics.
- The attention generator gives identical outputs for an input and its perturbation when the perturbation is a bijection preserving word probabilities.
- General word importance remains invariant because the perturbation preserves unigram probability estimated over the source pool.
- Class-specific word importance is invariant because conditional word-label probabilities estimated from the support set are preserved under the bijection.
A.3 LEARNING PROCEDURE
The learning procedure applies early stopping when validation loss fails to improve for 20 epochs.
- Early stopping is applied when validation loss does not improve for 20 epochs.
A.4 DATASETS
The paper evaluates generalization across six datasets using easy and hierarchy-based hard splits, compares multiple meta-learning baselines, and analyzes why distributional signatures improve transfer across mismatched lexical distributions.
- Dataset splits: Two split mechanisms test cross-class generalization: random easy splits and hierarchy-based hard splits with distant train, validation, and test classes.HuffPost and Reuters use easy splits; the other four datasets use hard splits.
- Hard-split construction: 20 Newsgroups, Amazon, RCV1, and FewRel use dataset-specific class or entity-type structure to construct hard splits.20 Newsgroups separates top-level categories, Amazon uses spectral clustering, and FewRel isolates relations whose head entity type is WORK OF ART.
- Datasets and baselines: The evaluation covers five standard text-classification datasets and one relation-classification dataset, with comparisons against induction networks and P-MAML.P-MAML is evaluated only on shorter-document datasets because longer documents exceed BERT’s 512-token limit after tokenization.
- Baseline comparison: Distributional signatures significantly outperform lexical-representation baselines across all datasets, while induction networks overfit meta-training classes and BERT adapts poorly to meta-test lexical mismatch.The paper attributes the baseline gap to differences between seen and unseen lexical distributions.
- Attention analysis: Attention-weighted representations separate classes better than unweighted or directly distributional representations and identify meaningful keywords without seeing target-class examples during meta-training.The generated attention is reported as closer to an oracle attention than the compared distributional representations.
- Downstream classifiers: Distributional signatures improve 5-way 1-shot accuracy by 9.9% for prototypical networks and 6.1% with logistic regression, while ridge regression remains slightly better and more efficient.Ridge and logistic regression achieve 78.0 versus 77.1 on 5-way 5-shot and 60.1 versus 58.5 on 5-way 1-shot, respectively.