PaperlayerView exact revision on arXiv

Source-linked AI summary

Meta-Learning without Memorization

Mingzhang Yin, George Tucker, Mingyuan Zhou, Sergey Levine, Chelsea Finn

arXiv:1912.03820v3cs.LGcs.AIstat.ML

TL;DR

Meta-learning can memorize non-mutually-exclusive training tasks, ignoring task data and failing to adapt to novel tasks. The paper introduces an information-theoretic meta-regularizer that encourages data-driven adaptation, and reports successful application across contextual and gradient-based methods and practical settings. The approach substantially outperforms standard meta-learning algorithms in those settings.

  • Problem

    Meta-learning requires carefully designed mutually-exclusive tasks because otherwise models may solve training tasks zero-shot instead of adapting, limiting applicable domains.

  • Method

    The paper proposes an algorithm-agnostic information-theoretic meta-regularizer that softly restricts information flow from meta-parameters to test-label predictions and encourages use of task training data.

  • Results

    The approach substantially outperforms standard meta-learning algorithms and applies successfully to contextual and gradient-based methods in practical settings.

  • Takeaways & Limitations

    Meta-regularization enables efficient adaptation from non-mutually-exclusive tasks without restricting the task distribution.

  • Takeaways & Limitations

    The regularizer can fail when the network stores the test-label prediction in part of z∗, incurring only a small information penalty.

Abstract

from arXiv · show

The ability to learn new concepts with small amounts of data is a critical aspect of intelligence that has proven challenging for deep learning methods. Meta-learning has emerged as a promising technique for leveraging data from previous tasks to enable efficient learning of new tasks. However, most meta-learning algorithms implicitly require that the meta-training tasks be mutually-exclusive, such that no single model can solve all of the tasks at once. For example, when creating tasks for few-shot image classification, prior work uses a per-task random assignment of image classes to N-way classification labels. If this is not done, the meta-learner can ignore the task training data and learn a single model that performs all of the meta-training tasks zero-shot, but does not adapt effectively to new image classes. This requirement means that the user must take great care in designing the tasks, for example by shuffling labels or removing task identifying information from the inputs. In some domains, this makes meta-learning entirely inapplicable. In this paper, we address this challenge by designing a meta-regularization objective using information theory that places precedence on data-driven adaptation. This causes the meta-learner to decide what must be learned from the task training data and what should be inferred from the task testing input. By doing so, our algorithm can successfully use data from non-mutually-exclusive tasks to efficiently adapt to novel tasks. We demonstrate its applicability to both contextual and gradient-based meta-learning algorithms, and apply it in practical settings where applying standard meta-learning has been difficult. Our approach substantially outperforms standard meta-learning algorithms in these settings.

1 INTRODUCTION

Meta-learning targets few-shot generalization but can memorize meta-training tasks instead of adapting, creating a need for carefully designed mutually-exclusive tasks. The paper proposes an information-theoretic meta-regularizer to mitigate this problem without restricting task distributions.

  • Meta-learning optimizes few-shot generalization so models can predict accurately after seeing a small task training set.
  • Current methods can ignore task training data when test inputs identify the task, achieving low meta-training loss through zero-shot decisions but failing on novel tasks.
  • Existing algorithms avoid memorization by constructing mutually-exclusive tasks that prevent one model from solving all tasks without adaptation.
  • For N-way classification, randomly permuting each task’s class-to-label assignment prevents the assignment from being inferred from test inputs alone.
  • The proposed meta-regularizer uses information theory to mitigate memorization without imposing restrictions on the task distribution.

2 PRELIMINARIES

The paper formalizes supervised meta-learning as learning meta-parameters that support task-specific adaptation from small training sets. It describes both gradient-based and contextual algorithms within a shared conditional-likelihood framework.

  • Tasks provide training and test datasets sampled from a task distribution, and meta-training learns parameters for adapting to a new task from limited data.
  • Meta-parameters θ summarize meta-training, while task-specific parameters φ are computed from task training data.
  • The framework maximizes q(ŷ∗=y∗|x∗,θ,D), combining meta-parameter, task-parameter, and predictive distributions.
  • Gradient-based methods: MAML computes φ from θ and task data through gradient optimization before using the resulting predictor for test predictions.
  • Contextual methods: CNP maps task training data to a summary statistic φ, which a predictor combines with x∗ to produce a predictive distribution.

3 THE MEMORIZATION PROBLEM IN META-LEARNING

The memorization problem occurs when meta-learners solve meta-training tasks without using task training data, producing poor adaptation to novel tasks. Mutual exclusivity can prevent it, but requires task engineering that is unavailable in some domains.

  • Meta-learners can overfit tasks by making predictions independent of task training data, which may cause poor generalization to novel tasks.
  • In pose prediction, memorizing each training object’s canonical pose enables accurate predictions without adaptation, whereas unseen objects require task data to infer that pose.
  • Medical prescription tasks illustrate a domain where symptoms and prescriptions are closely related, preventing arbitrary label reassignment for task construction.
  • Complete memorization is defined by I(ŷ∗;D|x∗,θ)=0, meaning predictions contain no additional information from task training data.
  • Memorization can generalize to new datapoints from training-like tasks while failing on novel tasks, distinguishing it from typical supervised-learning overfitting.
  • MAML and CNP can converge to memorization because their predictive pathways may achieve low training error without using task-specific information.
  • Mutually-exclusive tasks use per-task random label permutations to prevent test inputs alone from revealing task-specific class-to-label assignments.
  • The mutually-exclusive-task requirement burdens users with task engineering and cannot be applied across all desired meta-learning domains.

4 META REGULARIZATION USING INFORMATION THEORY

Meta-regularization uses information-theoretic constraints to reduce task information encoded in predictions or meta-parameters, encouraging reliance on task training data while preserving adaptation capacity. The approach includes activation and weight regularization and is linked theoretically to improved meta-test generalization, though activation regularization can sometimes be bypassed.

  • Meta-regularization on activations: Meta-regularization targets the memorization problem by encouraging predictions to use task training data rather than task information encoded in the predictive network.The proposed strategy controls information flow between test inputs, predictions, and task data.
  • Meta-regularization on activations: An intermediate stochastic bottleneck z∗ constrains information from x∗ while retaining a predictive pathway through task-specific parameters.The model factorizes prediction through q(z∗|x∗, θ) and q(ŷ∗|z∗, φ, θ).
  • Meta-regularization on activations: The activation objective combines a predictive loss with a KL penalty, where β controls regularization strength and r(z∗) may be N(z∗; 0, I).The KL term penalizes deviation of q(z∗|x∗, θ) from the variational marginal approximation.
  • Limitations: The activation regularizer performs well overall but can fail when the network stores the prediction of y∗ in part of z∗ at a small information penalty.This provides a specific failure mode for the information constraint.
  • Meta-regularization on weights: Weight regularization limits task information stored in non-adapting meta-parameters θ while leaving parameters that process task training data unrestricted.For MAML, regularization is applied to parameters uninvolved in task adaptation; for CNP, it is applied to encoder parameters.
  • Generalization analysis: A PAC-Bayes analysis relates weight regularization to a generalization bound, suggesting that meta-regularization on weights improves generalization on the meta-test set.The bound accounts for finite numbers of observed tasks and per-task validation examples, with β trading off bound tightness and confidence.

5 RELATED WORK

Prior meta-learning work addresses several forms of overfitting and latent-task complexity, but these approaches largely assume mutually-exclusive task distributions. This paper distinguishes memorization as task-level function memorization and studies generalization with PAC-Bayes theory.

  • Prior meta-learning methods: Existing methods mitigate meta-learning overfitting by reducing adapted parameters, compressing task embeddings, augmenting data, adding gradient objectives, or regularizing entropy.These approaches are described as addressing generalization in mutually-exclusive task settings.
  • Bayesian meta-learning: Bayesian meta-learning approaches constrain inferred task variables toward priors, corresponding to information bottlenecks on latent task representations.The paper situates its derivation within Bayesian interpretations of meta-learning.
  • Generalization analysis: PAC-Bayes analyses quantify the gap between empirical error on training tasks and expected error on new tasks, including extensions to multitask learning.Related work also uses tightened bounds as training objectives for mutually-exclusive tasks.
  • Memorization versus overfitting: Meta-learning memorization occurs at the task level, stores functions rather than labels, and can preserve within-task generalization while failing on new tasks.Its information source includes both meta-training data and task-specific training data, unlike conventional supervised overfitting.

6 EXPERIMENTS

Experiments test memorization across regression and classification tasks, showing that meta-regularization promotes data-driven adaptation and improves performance on non-mutually-exclusive tasks.

  • Experimental setup: The evaluation examines memorization across non-mutually-exclusive tasks using MAML, CNP, and their meta-regularized variants.Experiments include sinusoid regression, pose prediction, and few-shot Omniglot and MiniImagenet classification.
  • 6.1 SINUSOID REGRESSION: Adding task-identity information causes unregularized MAML and CNP to memorize, whereas MR-MAML and MR-CNP adapt to the task data and greatly outperform them.This result is reported for the non-mutually-exclusive sinusoid regression problem.
  • 6.2 POSE PREDICTION: Meta-regularization on weights outperforms competing methods by a large margin on pose prediction, while β tunes reliance on task data versus prior information.The comparison includes fine-tuning, and test error varies with the regularization coefficient β.
  • 6.2 POSE PREDICTION: Standard weight decay and Bayes-by-Backprop do not solve the memorization problem, indicating that it differs from conventional supervised-learning overfitting.The comparison evaluates these methods against meta-regularized CNP on pose prediction.
  • 6.3 OMNIGLOT AND MINIIMAGENET CLASSIFICATION: On non-mutually-exclusive classification, MR-MAML significantly outperforms previous methods and controls pre-update accuracy near chance so adaptation uses the task training data.High pre-update accuracy for MAML and TAML is presented as evidence that they ignore task data.
  • 6.3 OMNIGLOT AND MINIIMAGENET CLASSIFICATION: On standard mutually-exclusive Omniglot, small β values slightly improve over MAML, so meta-regularization does not degrade performance in that setting.The paper reports this result for the standard 20-way 1-shot task.

7 CONCLUSION AND DISCUSSION

The paper identifies memorization on non-mutually-exclusive tasks as a function-level overfitting problem and proposes information-theoretic meta-regularization to address it without restricting task distributions. The approach is presented as a step toward applying meta-learning across broader real-world domains.

  • The paper formalizes memorization on non-mutually-exclusive tasks as function-level overfitting, distinct from conventional label-level overfitting.
  • Meta-regularization uses an information-theoretic perspective to mitigate memorization without imposing restrictions on the task distribution.
  • The method softens information flow from meta-parameters to test-label predictions, encouraging use of task training data during meta-training.
  • The memorization issue is identified as potentially relevant to personalized speech recognition, adaptive robots, and goal-conditioned manipulation.
  • The paper suggests that this challenge may extend beyond meta-learning to other conditional prediction problems.

A.1 ALGORITHM

The algorithms implement meta-regularization by restricting pre-adaptation parameters while leaving adaptation-related parameters unrestricted. Separate procedures apply this design to CNP and MAML, using sampled parameter distributions and task-specific adaptation.

  • A.1 ALGORITHM: For CNP, regularization is applied to encoder weights θ while feature-extractor and decoder weights ˜θ remain unrestricted.
  • A.1 ALGORITHM: For MAML, weights θ from the input to an intermediate hidden layer are regularized, while adaptation weights ˜θ remain unrestricted.
  • A.1 ALGORITHM: Meta-Regularized CNP samples θ from a Gaussian weight distribution, computes task context from training data, and outputs network parameters τ and ˜θ.
  • A.1 ALGORITHM: Meta-Regularized MAML samples θ from a Gaussian distribution and computes task-specific parameters through gradient-based adaptation.
  • A.1 ALGORITHM: At meta-testing, the procedure uses the test input with task-specific parameters computed from the task training data.

A.2 META REGULARIZATION ON ACTIVATIONS

The activation-based analysis bounds information that test inputs provide about predictions by information that task data provides, supporting regularization that promotes data-driven adaptation.

  • A.2 META REGULARIZATION ON ACTIVATIONS: The analysis establishes I(x∗; ˆy∗|z∗, θ) ≤ I(ˆy∗; D|z∗, θ), relating test-input information about predictions to task-training-data information.
  • A.2 META REGULARIZATION ON ACTIVATIONS: Because I(ˆy∗; x∗|θ, D, z∗) = 0, the chain rule yields the stated information inequality.

A.3 META REGULARIZATION ON WEIGHTS

The weight-based analysis decomposes predictive cross entropy into uncertainty, mismatch, memorization, and adaptation-related information terms. It identifies memorization as information about meta-training data retained in meta-parameters and proposes controlling it through an upper bound.

  • A.3 META REGULARIZATION ON WEIGHTS: The meta-training loss in Eq. 1 upper-bounds the predictive cross entropy.
  • A.3 META REGULARIZATION ON WEIGHTS: The decomposition includes data-generating uncertainty, predictive-distribution mismatch, meta-parameter information about meta-training data, and an adaptation-related mutual-information term.
  • A.3 META REGULARIZATION ON WEIGHTS: The term I(D1:N; θ|x∗ 1:N, ξ) quantifies information that meta-parameters contain about meta-training data beyond the data-generating parameters.
  • A.3 META REGULARIZATION ON WEIGHTS: Without proper regularization, cross entropy can be minimized by maximizing the memorization term.
  • A.3 META REGULARIZATION ON WEIGHTS: The memorization term can be controlled by upper bounding it, yielding the regularization used in the paper.

A.4 PROOF OF THE PAC-BAYES GENERALIZATION BOUND

The appendix derives a PAC-Bayes generalization bound for meta-learning by separating task-level and within-task generalization. It also connects this analysis to the paper’s experimental settings and memorization results.

  • Setup: The meta-learner seeks a distribution Q over models that performs well on novel tasks using task data and a learned prior.The learned model distribution is parameterized by θ, while task adaptation produces q(φ|D, θ).
  • Setup: The target error on novel tasks is approximated empirically because computing the population error er(Q) is intractable from finite data.The empirical estimate averages performance across observed tasks and their task test data.
  • Proof strategy: The proof separates generalization error into finite-task and finite-example-per-task components, then bounds both with PAC-Bayes arguments.A union bound combines the task-level and within-task bounds into an overall meta-learner generalization bound.
  • Bound: The resulting complexity term includes a KL divergence between the learned parameter distribution Q and a prior P, with coefficient β after approximation.Taylor expansion for large n yields the stated meta-regularization form when Q is modeled as a Gaussian distribution.
  • Experiments: The experiments apply contextual and gradient-based meta-learning to pose prediction and few-shot classification, including MR-CNP and MR-MAML implementations.The reported settings include Pascal 3D pose prediction, Omniglot, and MiniImagenet classification.
  • Results: On mutually-non-exclusive tasks, MR-MAML and MR-CNP avoid memorization and achieve strong sinusoid performance, while MAML can largely ignore task training data.For 20-way 1-shot Omniglot, MAML has 99.2% pre-update training accuracy versus 5% for MR-MAML, indicating substantially more adaptation for MR-MAML.
Loading 1912.03820v3…