PaperlayerView exact revision on arXiv

Source-linked AI summary

Hierarchically Structured Meta-learning

Huaxiu Yao, Ying Wei, Junzhou Huang, Zhenhui Li

arXiv:1905.05301v2cs.LGstat.ML

TL;DR

Meta-learning must learn quickly from few samples despite uncertainty and heterogeneity across tasks, which global knowledge sharing does not handle well. HSML hierarchically clusters tasks to customize transferable knowledge by cluster while preserving within-cluster generalization, and experiments report effectiveness across toy regression and few-shot classification.

  • Problem

    Meta-learning faces task uncertainty and heterogeneity that cannot be handled effectively by globally sharing knowledge across tasks.

  • Method

    HSML uses hierarchical task clustering to tailor gradient-based parameter initializations to task clusters while preserving knowledge generalization within similar tasks.

  • Results

    HSML empirically outperforms state-of-the-art meta-learning algorithms in toy regression and few-shot image classification problems.

  • Takeaways & Limitations

    HSML provides a meta-learning framework that simultaneously customizes task knowledge and preserves generalization, with interpretable task relationships.

  • Takeaways & Limitations

    The continual-learning version is simple and does not model more complex task-relation evolution such as relationship forgetting.

Abstract

from arXiv · show

In order to learn quickly with few samples, meta-learning utilizes prior knowledge learned from previous tasks. However, a critical challenge in meta-learning is task uncertainty and heterogeneity, which can not be handled via globally sharing knowledge among tasks. In this paper, based on gradient-based meta-learning, we propose a hierarchically structured meta-learning (HSML) algorithm that explicitly tailors the transferable knowledge to different clusters of tasks. Inspired by the way human beings organize knowledge, we resort to a hierarchical task clustering structure to cluster tasks. As a result, the proposed approach not only addresses the challenge via the knowledge customization to different clusters of tasks, but also preserves knowledge generalization among a cluster of similar tasks. To tackle the changing of task relationship, in addition, we extend the hierarchical structure to a continual learning environment. The experimental results show that our approach can achieve state-of-the-art performance in both toy-regression and few-shot image classification problems.

1. Introduction

Meta-learning transfers knowledge from prior tasks, but global sharing struggles with heterogeneous task distributions while task-specific customization can sacrifice generalization. HSML addresses this tension by hierarchically clustering tasks and tailoring parameter initializations to clusters.

  • Motivation: Meta-learning uses transferable knowledge from previous tasks to improve learning from few examples.Gradient-based methods represent this knowledge as parameter initialization.
  • Motivation: Globally shared knowledge struggles with sequences of tasks from different distributions, while task-specific customization risks losing generalization.The introduction frames the goal as balancing customization with generalization.
  • HSML: HSML uses hierarchical soft clustering to customize transferable knowledge across task clusters while preserving generalization within similar tasks.Task representations come from pooling or recurrent autoencoder aggregators and feed the clustering structure.
  • HSML: A parameter gate tailors a globally shared parameter initialization to each cluster for tasks assigned to that cluster.The cluster-specific initialization serves as the starting point for those tasks.
  • Contributions: HSML is reported to outperform state-of-the-art meta-learning algorithms on toy regression and few-shot image classification, with interpretability and theoretical superiority over existing gradient-based methods.The paper also identifies better balance between generalization and customization as a contribution.

2. Related Work

Meta-learning methods differ in how they represent transferable knowledge, while gradient-based approaches commonly use a globally shared initialization. HSML instead customizes that initialization at the task-cluster level for changing task distributions.

  • Meta-learning approaches: Meta-learning approaches represent transferable knowledge through recurrent memories, learned optimizers, nearest-neighbor metric spaces, or parameter initialization.These correspond to recurrent-, optimizer-, nearest-neighbor-, and gradient-based methods.
  • Gradient-based methods: Gradient-based meta-learning commonly assumes one initialization shared across all tasks.This is illustrated as the globally shared initialization setting.
  • Gradient-based methods: Recent methods adapt the shared initialization to individual tasks using probabilistic models or task-specific information.These approaches target dynamically changing tasks.
  • HSML: HSML customizes the global initialization to each task cluster through hierarchical clustering.This design occupies an intermediate level between global sharing and per-task customization.

3. Preliminaries

Meta-learning trains a meta-learner and task-specific base learners across historical tasks, then applies the learned meta-learner to future tasks with few examples. In gradient-based meta-learning, the meta-learner is a parameter initialization adapted by gradient descent.

  • Meta-learning problem: Tasks are sampled from an environment distribution, and each task provides a small training set with examples and labels.Training and testing examples are treated as separate sets in the task formulation.
  • Meta-learning problem: The base learner uses parameters θTi to make predictions for task Ti, with mean square error for regression or log-likelihood for classification.The supplied preliminaries describe these task-level prediction losses.
  • Meta-learning objective: Meta-learning seeks a well-generalized meta-learner that facilitates training a base learner on a future task using few examples.Meta-training and meta-testing form the two stages of the process.
  • Meta-learning objective: During meta-training, base-learner parameters are optimized over historical task training sets while the meta-learner is optimized over task test sets.The two optimization objectives are described as alternating updates.
  • Gradient-based meta-learning: MAML represents the meta-learner as a generalizable initialization from which a few gradient-descent steps reach task-specific parameters.The preliminaries present one gradient step as an example of the adaptation process.

4. Methodology

HSML represents tasks, assigns them softly within a hierarchical clustering structure, and adapts global transferable knowledge to cluster-specific initializations. Its design combines task-level representation, online hierarchical clustering, and gated knowledge adaptation for rapid task optimization.

  • Knowledge Adaptation: HSML adapts the global initialization θ0 to cluster-specific parameters so few gradient steps can reach task-optimal parameters.The adaptation path is determined by the hierarchical task-clustering structure.
  • Task Representation Learning: Task representations aggregate all training examples using permutation-invariant pooling or recurrent autoencoder aggregators.The proposed candidates are pooling autoencoder aggregation and recurrent autoencoder aggregation.
  • Task Representation Learning: The pooling aggregator learns example representations with an autoencoder and aggregates them using max or mean pooling.The resulting task representation gi is intended to be permutation invariant.
  • Task Representation Learning: The recurrent aggregator sequentially encodes and reconstructs examples, then aggregates their representations into a task representation.Random permutations are used because sequential feeding otherwise makes the representation permutation sensitive.
  • Hierarchical Task Clustering: HSML uses hierarchical rather than flat clustering because one task-group level may not model complex task relationships.Cluster centers are learned online as tasks arrive, and the structure can expand when an incoming task does not fit existing clusters.
  • Hierarchical Task Clustering: Soft assignments are computed across levels and remain differentiable, while weighted updates transform representations between hierarchical levels.Assignments are based on distances to learnable cluster centers, and each level receives a task representation from the preceding level.
  • Knowledge Adaptation: A parameter gate combines the hierarchical cluster representation with the task representation to adapt θ0 as θ0i = θ0 ◦ oi.The gate is intended to preserve and reinforce cluster-specific adaptation.
  • Optimization: HSML jointly optimizes test empirical risk and reconstruction error, balancing them with ξ over global initialization, clustering, and adaptation parameters.The learnable parameter set Θ includes θ0, clustering parameters, and Wg.

5. Analysis

The analysis models HSML as cluster-specific initialization adaptation and derives generalization bounds under convex and non-convex losses. It concludes that HSML can achieve a tighter bound than MAML under the stated assumptions.

  • Setup: HSML is analyzed as adapting a globally shared SGD initialization through hierarchical clustering into cluster-specific initializations.The analysis focuses on the generalization-bound advantage of this adaptation.
  • Assumptions: The bounds assume training and testing examples are independently and identically distributed within each task distribution Si.This assumption is used to define stability and relate empirical and expected risk.
  • Generalization Bounds: The paper derives generalization bounds for both convex and non-convex losses when the base learner is optimized with SGD.The non-convex result assumes bounded, smooth loss and a Lipschitz Hessian, with a constrained decreasing step size.
  • Comparison with MAML: MAML is a special case of HSML when every cluster transformation matrix equals the identity matrix.This embeds globally shared initialization within the HSML formulation.
  • Comparison with MAML: HSML achieves a tighter generalization bound than MAML under the paper’s analyzed conditions.The comparison is based on the existence of suitable cluster-specific transformation matrices.

6. Experiments

Experiments evaluate HSML on toy regression and heterogeneous few-shot image classification, testing performance, task-cluster discovery, and continual adaptation. Results indicate that hierarchical clustering supports interpretable task relationships and improves performance over global and task-specific baselines.

  • Experimental goals: The evaluation tests HSML against global and task-specific meta-learning baselines on toy regression and few-shot image classification.The experiments also examine task-cluster discovery and continual updates to the clustering structure.
  • Experimental settings: The recurrent autoencoder aggregator is on average better than the pooling autoencoder aggregator and is therefore used by default.All baselines use the same base-learner neural-network structure.
  • Toy regression: Toy regression samples tasks from sinusoidal, linear, cubic, and quadratic function families under 5-shot and 10-shot settings.Table 1 reports MSE ± 95% confidence intervals averaged over 4,000 tasks.
  • Toy regression: HSML clusters toy-regression tasks primarily by function shape, including similarities across different underlying function families.Quadratic, cubic, and linear tasks activate related clusters when their shapes are similar, and HSML fits better than the compared baselines qualitatively.
  • Few-shot classification: The few-shot classification benchmark contains Bird, Texture, Aircraft, and Fungi datasets, making its tasks more heterogeneous than single-dataset benchmarks.Tasks use N-way, K-shot classification with a standard four-block convolutional base learner and a three-layer clustering structure.
  • Few-shot classification: Different image datasets mainly activate different clusters, while secondary activations capture cross-task relationships associated with shape, environment, and surface texture.Table 2 compares 5-way 1-shot and 5-shot accuracy ± 95% confidence intervals averaged over 1,000 tasks per dataset.
  • Cluster analysis: Too few clusters may fail to learn task-clustering characteristics, whereas increasing layers can achieve similar performance with more parameters.Table 3 reports 5-way 1-shot classification accuracy for different cluster-number configurations.

7. Conclusion and Discussion

The paper concludes that HSML combines task-specific customization with within-cluster generalization through hierarchical clustering, with effectiveness and interpretability demonstrated across two problem types. Its continual-learning extension remains limited because it models only a simple form of task-group addition.

  • Conclusion: HSML customizes task knowledge while preserving knowledge generalization through a hierarchical clustering structure.Experiments demonstrate effectiveness and interpretability in toy regression and few-shot classification.
  • Limitations and future directions: The continual-learning version adds new underlying function groups but does not model more complex task-evolution relations such as relationship forgetting.The authors identify richer lifelong-learning dynamics as a future direction.

A. Detailed Theoretical Analysis

The paper analyzes HSML’s generalization behavior under convex and non-convex losses, using stability-based bounds and assumptions on the loss landscape. It also describes the theoretical initialization representation and experimental settings used for image classification.

  • Theoretical Analysis: The analysis derives generalization bounds for the base learner when the loss is convex or non-convex.The bounds rely on prior stability results and Hoeffding-style concentration arguments.
  • Theoretical Analysis: The existence analysis shows that the product of transformations can be related to an identity-minus-gradient expression, with a better parameter reachable through a negative-gradient direction.The paper illustrates this behavior using a contour plot in Figure 8.
  • Experimental Settings: The few-shot image-classification experiments use resized 84 × 84 × 3 images and standard convolutional base learners, with hyperparameters summarized in Table 4.The implementation uses full Hessian-vector products for MAML and adds one cluster at a time in continual scenarios.

E. Leave-one-out Experiments on Few-shot Image Classification

The leave-one-out experiment evaluates cross-dataset generalization by training on three image datasets and testing on a fourth, making the task more challenging than conventional class-held-out evaluation. Despite lower overall performance, HSML outperforms MAML and MUMOMAML in 5-way 1-shot classification.

  • Experimental Design: Leave-one-out training uses three datasets for meta-training and the remaining dataset for meta-testing, rather than merely holding out classes.For example, texture, bird, and aircraft datasets train the model while fungi provides the test dataset.
  • Results: All leave-one-out results are significantly worse than results without leave-one-out evaluation, confirming the greater difficulty of cross-dataset testing.The comparison is reported for 5-way 1-shot classification in Table 6.
  • Results: HSML achieves better performance than MAML and MUMOMAML in the difficult leave-one-out setting.The paper attributes this comparison to HSML’s ability to capture task-clustering structure.

F. Additional Results of Few-shot Classification

Additional experiments examine online updates, cluster-count choices, and task-embedding aggregators in few-shot classification. The recurrent autoencoder aggregator performs best overall, especially in the 1-shot setting, while the two aggregators become more similar with more training samples.

  • Additional Results: The additional results include online-update comparisons and evaluations across different numbers of clusters in few-shot image classification.Full results are provided in Tables 7 and 8.
  • Aggregator Comparison: The recurrent autoencoder aggregator performs best, significantly outperforming mean pooling in the 1-shot scenario.The compared variants are HSML-RAA and HSML-MPAA.
  • Aggregator Comparison: As the number of training samples increases, recurrent and mean-pooling aggregators achieve more similar performance.The paper suggests that training a stronger mean-pooling aggregator may require more data.

H. Ablation Studies

Ablation studies isolate the roles of task representation, hierarchical clustering, and knowledge adaptation in 5-way 1-shot image classification. The results support jointly learned embeddings, hierarchical structure, and parameter gating over simpler alternatives.

  • Task Clustering: HSML outperforms MAML trained with ground-truth task clusters, indicating benefits from soft hierarchical clustering and cross-cluster knowledge transfer.The ground-truth-clustered MAML variant is described as an upper-bound application unavailable in real-world settings.
  • Task Representation: Meta-learning gradients further optimize task embeddings, while replacing the proposed embedding with an encoder’s last hidden state performs worse than HSML.The alternative with the last hidden state still exceeds MUMOMAML, supporting a contribution from hierarchical clustering.
  • Hierarchical Clustering: Removing hierarchical clustering performs worse than using flat clustering, supporting knowledge generalization within task clusters.This comparison is reported as ablation A3.
  • Knowledge Adaptation: Combining task and cluster representations yields the best knowledge-adaptation performance, while directly learning initialization without a parameter gate performs poorly.The paper proposes that parameter gating can constrain optimization and warm-start new continual-learning clusters.

I. Additional Task Clustering Results of Toy Regression Tasks

Additional toy-regression analyses show that tasks can be clustered by shape, separating fluctuation, arc, and linear-slope patterns.

  • Toy-regression tasks are clustered according to their shapes.The additional analysis further verifies shape-based task clustering.
  • Cluster 1 captures fluctuation-mode curves, including Sin a1-a4 and Cubic a1-a4 tasks.
  • Cluster 2 captures arc-shaped tasks, such as Quad a2-a4.
  • Clusters 3 and 4 separate mainly linear shapes by slope direction: positive for cluster 3 and negative for cluster 4.Cluster 3 includes Line a1, Line a2, Quad a1, Quad a2, and Cubic a1; cluster 4 includes Line a3, Line a4, and Cubic a4.
Loading 1905.05301v2…