Diverse Few-Shot Text Classification with Multiple Metrics

Mo Yu; Xiaoxiao Guo; Jinfeng Yi; Shiyu Chang; Saloni Potdar; Yu Cheng; Gerald Tesauro; Haoyu Wang; Bowen Zhou

Source-linked AI summary

Diverse Few-Shot Text Classification with Multiple Metrics

Mo Yu, Xiaoxiao Guo, Jinfeng Yi, Shiyu Chang, Saloni Potdar, Yu Cheng, Gerald Tesauro, Haoyu Wang, Bowen Zhou

arXiv:1805.07513v1May 19, 2018cs.CLcs.LG

TL;DR

Few-shot text classification is challenging because real-world tasks are diverse and a single metric may not capture their variations. The paper clusters related meta-training tasks, learns multiple metrics, and combines them for new tasks. Experiments on sentiment and dialog intent classification report favorable predictive accuracy, while the pipeline design and cross-task vocabulary differences define important scope boundaries.

Problem
Real-world few-shot text-classification tasks are diverse, so a single metric may be insufficient to capture their complex variations.
Method
The method clusters meta-training tasks, learns cluster-specific embedding metrics, and assigns each target task a linear combination of those metrics.
Results
The proposed method reports significant gains on few-shot sentiment and dialog intent classification and favorable predictive accuracy against state-of-the-art few-shot algorithms.
Takeaways & Limitations
Multiple metrics and task clustering support few-shot learning when tasks come from diverse domains.
Takeaways & Limitations
End-to-end joint optimization is not directly applicable in the described setting, so the method uses pipeline training and validation sets for task clustering.

Abstract

from arXiv · show

We study few-shot learning in natural language domains. Compared to many existing works that apply either metric-based or optimization-based meta-learning to image domain with low inter-task variance, we consider a more realistic setting, where tasks are diverse. However, it imposes tremendous difficulties to existing state-of-the-art metric-based algorithms since a single metric is insufficient to capture complex task variations in natural language domain. To alleviate the problem, we propose an adaptive metric learning approach that automatically determines the best weighted combination from a set of metrics obtained from meta-training tasks for a newly seen few-shot task. Extensive quantitative evaluations on real-world sentiment analysis and dialog intent classification datasets demonstrate that the proposed method performs favorably against state-of-the-art few shot learning algorithms in terms of predictive accuracy. We make our code and data available for further study.

1 Introduction

Few-shot text classification involves diverse tasks whose optimal meta-models may differ, making a single shared metric insufficient. The proposed framework clusters related meta-training tasks, learns cluster-specific metrics, and combines them for new tasks, achieving gains on sentiment and dialog intent classification.

Motivation: Few-shot learning adapts classifiers from few examples per class using meta-models learned across many meta-training tasks.Prior approaches include metric-based models that learn task-specific similarities and optimization-based models that predict parameters or updates from task gradients.
Motivation: Image-domain benchmarks often use one common meta-model because their tasks are sampled from a related single domain.The paper contrasts this simplified setting with real-world text classification, where clients submit varied business-specific tasks.
Problem: Diverse text tasks can require different optimal meta-models, so a single metric may not capture their task variation.Tasks may differ in labels and domain relationships, as in sentiment and opinion classification for different products or services.
Approach: The method clusters meta-training tasks, trains one embedding metric per cluster, and assigns each target task a linear combination of cluster metrics.Clustering makes metrics shared within related task groups while allowing target tasks to derive different metrics from prior experience.
Approach: A matrix-completion task-clustering algorithm estimates cross-task transfer relationships and applies spectral clustering to produce task partitions.The transfer matrix handles missing and unreliable accuracy estimates before clustering.
Results: The experiments report significant gains on few-shot sentiment and dialog intent classification, supporting multiple metrics for diverse tasks.The paper describes the work as addressing diverse few-shot learning on real-world text classification problems.

2 Problem Definition

The paper formulates metric-based few-shot learning as meta-training multiple similarity metrics and meta-testing them on new few-shot tasks. Its base models use neural encoders to compare inputs or class support sets, with performance averaged across target-task test sets.

Few-Shot Learning Formulation: Meta-training learns a set of K metrics M={Λ1,···,ΛK} across N tasks, with each metric mapping two inputs to a scalar similarity score.Each task contains training, validation, and testing data, and K is usually much smaller than N.
Few-Shot Learning Formulation: Meta-testing applies the learned metrics to new few-shot tasks containing training and testing data.The labeled training set generates a prediction model for each target task.
Few-Shot Learning Formulation: The target-task predictors are kNN-based models built from the learned metrics, and overall performance is the macro-average accuracy across testing sets.The formulation can be generalized to other meta-learning approaches.
Few-Shot Learning Formulation: Multiple metrics are motivated by task diversity, while previous metric-based methods appear as the special case where M contains one metric.This formulation separates the number of available metrics from the number of meta-training tasks.
Matching Networks: Matching Networks use an encoder f to map inputs to d-length vectors, defining similarity as Λ(x1,x2)=f(x1)^T f(x2).The model combines the encoder with augmented memory storing the support set.
Matching Networks: For a test instance, Matching Networks predict labels through similarity to the support set, with attention weights defined by a softmax over encoder similarities.The weighted support labels form a valid predictive distribution.
Text Encoder: For text classification, the encoder is implemented as a CNN with convolution and max-pooling over the sentence.The figure distinguishes a CNN classifier, which predicts from sentence embeddings, from a Matching Network, which uses k-nearest-neighbor prediction.
Prototypical Networks: Prototypical Networks are presented as a metric-learning variation of Matching Networks that constructs the classifier differently.Their class-specific support sets are used in the prototype-based formulation.

3 Methodology

The method clusters diverse tasks using robust transfer-based similarity estimation, then learns cluster-specific metrics and combines them for each target task. It addresses unreliable and costly cross-task evaluations through score filtering, matrix completion, and spectral clustering.

3 Methodology: The framework first clusters meta-training tasks, trains one encoder per cluster, and uses those encoders to support few-shot prediction.The overall pipeline combines robust task clustering with cluster-model training and target-task adaptation.
3.1 Task Clustering: Transfer performance forms an asymmetric task-pair matrix because adapting a model from task i to task j can differ from the reverse direction.Each entry measures how representations learned on one task transfer to another task.
3.1.1 Estimation of Cross-Task Transfer Performance: Random task-pair sampling reduces the O(n^2) cost of evaluating all source-target transfers, while retaining only consistent bidirectional scores removes unreliable entries.The filtered scores produce a partially observed symmetric similarity matrix for clustering.
3.1.1 Estimation of Cross-Task Transfer Performance: Dynamic task-specific thresholds classify sufficiently high or low transfer scores as binary similarities, leaving uncertain pairs unobserved.Thresholds use each target task’s mean and standard deviation rather than a fixed global cutoff.
3.1.2 Task Clustering Method: Matrix completion reconstructs a full low-rank similarity matrix while a sparse error matrix captures incorrect observed entries, after which spectral clustering obtains task groups.The formulation uses the observed binary matrix, low-rank structure, and sparse corruption model.
3.1.2 Task Clustering Method: Under mild conditions, matrix completion can recover the underlying similarity matrix from at least O(n log^2 n) correct observed entries.This theoretical guarantee implies that only a small fraction of task pairs may be needed as the number of tasks grows.
3.2 Few-Shot Learning on Cluster-models: Each cluster receives a shared embedding metric, and target-task prediction linearly combines the frozen cluster encoders using few-shot-trained weights.This allows a target task to derive a task-specific metric from metrics learned by different clusters.
Discussion: The approach uses pipeline training and validation sets because joint training loss can fit arbitrary task partitions and produce metrics that do not generalize.The paper identifies reinforcement learning combined with meta-learning as a possible direction for end-to-end training.

4 Tasks and Data Sets

The evaluation uses sentiment and dialog-intent classification tasks organized into meta-training and meta-testing sets. The dialog dataset spans many clients and label counts, while target tasks are constructed with limited labeled examples.

Datasets and Experimental Setup: Experiments use two text classification datasets, dividing tasks into meta-training tasks for clustering and encoder training and meta-testing target tasks for evaluation.The target tasks are evaluated in the few-shot setting described by the method.
4.1 Sentiment Classification: The sentiment dataset contains Amazon reviews from 23 product domains, with three binary tasks per domain defined by rating thresholds.The positive-review buckets are 5 stars, at least 4 stars, or at least 2 stars.
4.2 Dialog Intent Classification: The dialog dataset contains conversations from an online service that trains and serves intent classifiers for clients across domains.The scenarios range from personal assistants to service-ordering and customer-service requests.
4.2 Dialog Intent Classification: The dialog benchmark includes 175 client tasks, from 2 to 100 labels, with 10 tasks randomly selected as targets.Meta-training data use 64% for training, 16% for validation, and the remainder for testing.
4.2 Dialog Intent Classification: Target dialog tasks use one example per label plus 20 randomly selected labeled examples to approximate the data a client could readily provide.This construction adapts the variable-label dataset to a few-shot scenario.
Additional Evaluation: The paper also evaluates an additional larger-task dataset, but reports those results in a multi-task learning setting outside the paper’s main few-shot focus.The authors refer readers to a non-archive version for those results.

5 Experiments

Experiments evaluate ROBUSTTC-FSL on few-shot sentiment and dialog intent tasks, showing consistent gains over baselines and benefits from adaptive clustering for more diverse intent tasks.

Experimental Results: Table 1 evaluates 12 few-shot product sentiment tasks and 10 few-shot dialog intent tasks using knowledge from previously observed tasks.The sentiment experiments leverage 57 prior tasks, while intent experiments leverage 165 tasks from other clients’ data.
Experimental Results: Single-metric approaches underperform because one metric is insufficient for diverse tasks, especially dialog intent classification.On intent classification, single-metric or single-model methods perform worse than the single-task CNN baseline.
Experimental Results: ROBUSTTC-FSL outperforms prior baselines by more than 6% on sentiment classification and more than 3% on intent classification.It also significantly outperforms the proposed baseline, indicating benefits from task clustering.
Adaptive ROBUSTTC-FSL: Adaptive ROBUSTTC-FSL improves the best ROBUSTTC-FSL result on intent classification by more than 5%.It treats a target task as out-of-clusters when no cluster exceeds 20% accuracy on its training data, then falls back to a single-task CNN.
Effect of the Number of Clusters: ROBUSTTC performs best with 5 clusters for sentiment analysis and 20 clusters for intent classification, while all clustering results beat the single-metric baseline.The single-metric baseline corresponds to one cluster.
Cluster Analysis: Task clusters commonly group reviews with the same rating threshold or similar product domains, and learned weights adapt multiple metrics for target tasks.Table 2 also reports a representative clustering comparison of 83.12 versus 82.65 at five clusters.

6 Related Work

Previous few-shot learning methods typically use one metric across related tasks sampled from a single dataset. Real-world text classification tasks can differ in domains and label counts, making a single metric insufficient.

Metric-based approaches learn generalizable metrics and matching functions from multiple training tasks, effectively using one metric across tasks.
Earlier few-shot benchmarks commonly use k-shot, N-way tasks with the same number of labels, sampled from one large related dataset.
Real-world few-shot tasks may vary in label counts and relatedness, so a single meta-model or metric-model is usually insufficient.

7 Conclusion

The paper proposes task clustering for diverse few-shot learning, combining multiple metrics rather than relying on a single metric. It reports significantly better performance on diverse tasks and identifies extensions for future work.

The proposed approach uses task clustering to support multiple metrics for few-shot learning on diverse tasks.
The method performs significantly better than previous single-metric methods when few-shot tasks come from diverse domains.
Future work includes extending task clustering to other few-shot algorithms and exploring composition methods beyond linear combination.

Appendix A: Perfect Recovery Guarantee for the Problem (5)

The appendix gives a perfect-recovery guarantee for matrix completion under incoherence, random sampling, and bounded corruption assumptions. It concludes that sufficiently many correct observations allow recovery with high probability.

The theorem assumes a rank-k matrix whose row and column spaces have bounded coherence and whose singular-vector product has bounded maximum entry.
Among uniformly sampled observed entries, a randomly sampled subset may be corrupted while the partially observed matrix remains suitable for recovery.
Under the stated conditions and a specified range for λ, the underlying matrix is perfectly recovered with probability at least 1 − n^-3.
The guarantee implies that at least O(n log^2 n) observed correct entries suffice for perfect recovery despite some incorrect observations.
The completed similarity matrix is symmetric because the input matrix Y is symmetric.

Appendix B: Proof of Low-rankness of Matrix X

The appendix proves that the task-similarity matrix is low-rank by expressing it as a sum of cluster-specific rank-one matrices. Its rank is therefore bounded by the number of clusters.

Let A contain cluster membership vectors; the similarity matrix X is computed from these underlying perfect clustering assignments.
Each cluster contribution B_i = a_i a_i^⊤ is rank one.
Because X is a sum of k rank-one cluster contributions, rank(X) ≤ k.
When the number of clusters is small, the similarity matrix X is correspondingly low-rank.

Appendix C: Proof of Theorem 7.1

The appendix establishes recovery and uniqueness results for the underlying matrices under incoherence, sampling, corruption, and dual-certificate conditions. It proves the main theorem by constructing a suitable dual certificate and showing that any competing optimizer must coincide with the truth.

Assumptions: The proof assumes a singular-value decomposition of X and two incoherence conditions on its singular spaces and UV⊤.A1 bounds row and column-space coherence, while A2 bounds the maximum absolute entry of UV⊤ by µ1√r/n.
Sampling proposition: With more than m0 uniformly sampled entries, a matrix in T whose observed entries are all zero is zero with probability 1 − 3n^−β.This proposition supplies the injectivity property used later in the uniqueness argument.
Dual certificate theorem: Theorem 1 states that if m1 − m2 > m0 and a dual matrix Q satisfies conditions (a)–(e), then (X,E) is the unique optimizer of (9) with probability at least 1 − 3n^−β.The conditions constrain Q’s support, tangent-space projection, complementary norm, and behavior on corrupted and uncorrupted entries.
Main theorem proof: Theorem 3.1 is proved by constructing Q through the inverse of PT PΩ PT and bounding certificate errors so conditions (b)–(e) hold.The construction proceeds by solving for εT and ε∆, then controlling εΩ, εT, PT⊥(H), and PT⊥(F).
Main theorem proof: The proof concludes under the sufficient condition ξ(X)µ(E) ≤ 1/(4k + 5).This bound is obtained after checking the certificate norm conditions and choosing λ appropriately.

Appendix D: Data Statistics

The appendix reports statistics for the sentiment and user-intent datasets used in the experiments, including domain information and target-task statistics.

Sentiment classification: The sentiment dataset statistics are presented for the multi-domain sentiment classification data.The dataset contains Amazon product reviews organized by product domains; the paper removes musical-instrument and tools-hardware domains because they have too few labeled examples.
Intent classification: The user-intent dataset statistics are presented for the user intent classification data.The statistics concern the 10 target intent-classification tasks listed in Table 4.