PaperlayerView exact revision on arXiv

Source-linked AI summary

PACOH: Bayes-Optimal Meta-Learning with PAC-Guarantees

Jonas Rothfuss, Vincent Fortuin, Martin Josifoski, Andreas Krause

arXiv:2002.05551v5stat.MLcs.LG

TL;DR

Meta-learning's generalization to unseen tasks is poorly understood, especially when few meta-training tasks are available. PACOH derives PAC-Bayesian bounds and a tractable, regularized meta-learning approach; its GP and BNN instantiations perform strongly and are used in sequential decision problems.

  • Problem

    Meta-learning generalization to unseen tasks is poorly understood, particularly when the number of meta-training tasks is small.

    Source: §1 · p. 1

  • Method

    The paper derives PAC-Bayesian meta-learning bounds and uses them to develop PAC-optimal hyper-posteriors with principled regularization and tractable optimization.

    Source: §1 · p. 1

  • Results

    PACOH with Gaussian processes and Bayesian neural networks achieves state-of-the-art predictive accuracy, improves uncertainty calibration, and alleviates meta-overfitting with as few as five tasks.

    Source: §1 · p. 1

  • Takeaways & Limitations

    PACOH-based learners are also used for sequential decision problems, including a real-world Bayesian optimization task concerning vaccine development.

    Source: §1 · p. 2

    For Bayesian neural network priors, the estimator of the log partition function is consistent but not unbiased.

    Source: §5.3 · p. 6

Abstract

from arXiv · show

Meta-learning can successfully acquire useful inductive biases from data. Yet, its generalization properties to unseen learning tasks are poorly understood. Particularly if the number of meta-training tasks is small, this raises concerns about overfitting. We provide a theoretical analysis using the PAC-Bayesian framework and derive novel generalization bounds for meta-learning. Using these bounds, we develop a class of PAC-optimal meta-learning algorithms with performance guarantees and a principled meta-level regularization. Unlike previous PAC-Bayesian meta-learners, our method results in a standard stochastic optimization problem which can be solved efficiently and scales well. When instantiating our PAC-optimal hyper-posterior (PACOH) with Gaussian processes and Bayesian Neural Networks as base learners, the resulting methods yield state-of-the-art performance, both in terms of predictive accuracy and the quality of uncertainty estimates. Thanks to their principled treatment of uncertainty, our meta-learners can also be successfully employed for sequential decision problems.

1. Introduction

Meta-learning can extract prior knowledge, but with few meta-training tasks it risks overfitting and generalizing poorly to unseen tasks. PACOH addresses this with PAC-Bayesian bounds and a tractable, regularized approach that performs strongly across experiments and is also used for sequential decision problems.

Motivation

  • Small numbers of meta-training tasks raise a risk of overfitting that can impair performance on unseen target tasks.The challenge is to regularize the meta-learner so it generalizes to unseen tasks.Source: §1 · p. 1
  • Earlier PAC-Bayesian meta-learner analyses assume bounded losses and require computationally expensive nested optimization.These assumptions preclude applications such as regression and probabilistic inference, where losses are typically unbounded.Source: §1 · p. 1

Contributions

  • The paper derives a PAC-Bayesian bound for unbounded losses and a Bayesian-learner bound tied to marginal log-likelihood, avoiding nested optimization.The resulting PAC-optimal hyper-posterior supports performance guarantees, principled meta-level regularization, and scalable algorithms using standard variational methods.Source: §1 · p. 1

Experiments

  • PACOH with Gaussian processes and Bayesian neural networks achieves state-of-the-art predictive accuracy, improves uncertainty calibration, and alleviates meta-overfitting with as few as five tasks.The experiments span regression and classification environments.Source: §1 · p. 1

Application

  • PACOH is showcased in a real-world Bayesian optimization task for developing vaccines.The task concerns discovering peptides that bind to MHC-I molecules.Source: §1 · p. 2

2. Related Work

Prior meta-learning approaches learn recurrent programs, shared embeddings, or adaptable neural-network initializations, while probabilistic methods add uncertainty quantification. The paper identifies limited task-data guarantees and difficult optimization as gaps addressed by its PAC-Bayesian analysis.

Meta-learning

  • Meta-learning methods learn recurrent learning programs, shared task embeddings, or neural-network initializations that adapt to new tasks.Probabilistic approaches also seek uncertainty quantification.Source: §2 · p. 2
  • Existing approaches often rely on abundant meta-training tasks and lack performance guarantees, while meta-overfitting has been noted without rigorous analysis under unbounded losses.The paper studies meta-learner generalization with PAC-Bayesian methods and principled meta-level regularization.Source: §2 · p. 2

PAC-Bayesian theory

  • Previous PAC-Bayesian meta-generalization bounds leave both hyper-posterior and posterior unspecified, producing difficult nested optimization problems.The paper's bounds also hold for unbounded losses and yield a tractable meta-learning objective without nested optimization.Source: §2 · p. 2

3. Background: PAC-Bayesian Framework

The PAC-Bayesian framework studies randomized predictors by relating empirical error to generalization error while accounting for the prior and posterior. Gibbs posteriors minimize a PAC-Bayesian bound, and with negative log-likelihood loss they connect to generalized Bayesian inference.

Learning setup

  • A learning task provides a finite sample from an unknown data distribution, and the learner seeks hypotheses with low expected prediction error.The empirical error on the sample is used because the data distribution is unknown.Source: §3 · p. 2

Randomized predictors

  • PAC-Bayesian analysis uses randomized predictors represented by prior P and posterior Q, with Q's Gibbs error averaging hypothesis errors.The prior must be independent of observed data, and the framework can reason about epistemic uncertainty from finite training data.Source: §3 · p. 2

PAC-Bayesian bounds

  • PAC-Bayesian bounds address the unknown generalization error using empirical error, with tractability depending on assumptions about the loss.For unbounded losses, the passage describes a sub-gamma condition based on bounded moments.Source: §3 · p. 2

Gibbs posterior

  • The bound-minimizing Gibbs posterior has density proportional to the prior times an exponential of negative empirical loss.With negative log-likelihood loss it coincides with a generalized Bayesian posterior; setting β = m recovers the standard Bayesian posterior.Source: §3 · p. 3

4. PAC-Bayesian Bounds for Meta-Learning

The section extends PAC-Bayesian analysis to meta-learning, bounding transfer error using empirical multi-task error and complexity terms. For Bayesian base learners, the bound can be optimized through a PAC-optimal hyper-posterior, yielding a tractable stochastic objective with meta-level regularization.

PAC-Bayesian meta-learning bounds

  • Theorem 2 bounds transfer error using empirical multi-task error and complexity terms for a fixed hyper-prior and base learner.The theorem's conditions include bounded losses or sub-gamma losses under the stated moment assumptions.Source: §4 · p. 3

Proof strategy

  • The proof bounds finite-sample generalization within tasks and the meta-level gap from observing finitely many tasks, then controls the cumulant-generating function.It uses a change-of-measure inequality and either bounded-loss or sub-gamma assumptions.Source: §4 · pp. 3–4

Bound properties

  • For bounded losses, Theorem 2 gives a tighter bound than Pentina & Lampert (2014), and it also provides guarantees for unbounded losses under moment constraints.The unbounded-loss result is useful for probabilistic models whose loss is negative log-likelihood.Source: §4 · p. 4

Parameter choices

  • With λ = √n and β = √m, the bounds are consistent; choosing λ = n and β = m keeps a gap but makes KL terms decay faster.The latter choice can be advantageous for smaller sample sizes.Source: §4 · p. 4

Bayesian base learner

  • The bound's PAC-optimal Gibbs base learner removes explicit dependence on the posterior and turns nested optimization into standard stochastic optimization.With negative log-likelihood and λ = n, β_i = m_i, the log partition function coincides with marginal log-likelihood, tractable for models such as Gaussian processes.Source: §4 · p. 4

Meta-level regularization

  • The bound's KL-divergence term regularizes at the meta-level, with its relative weight shrinking as the number of training tasks grows.This matches the stated principle that regularization should be stronger with little data and vanish asymptotically as n, m →∞.Source: §4 · p. 4

Meta-learning the hyper-posterior

  • The hyper-posterior minimizing the meta-learning bound is PAC-optimal, providing the best possible meta-generalization guarantees among meta-learners in the sense of Theorem 2.Its expression is tractable up to a level-II partition function constant with respect to the hyper-prior.Source: §4 · p. 4

5. Meta-Learning using the PACOH

The section translates PACOH into practical GP and BNN meta-learners, approximating the hyper-posterior with SVGD particles and specifying estimators for each base-learner family.

5.1. Approximating the PACOH

  • SVGD approximates the PACOH hyper-posterior Q* with prior-parameter particles updated using kernel-based score gradients.The procedure maintains K particles and uses a kernel matrix and step size for the updates.Source: §5.1 · p. 5

5.2. Meta-Learning Gaussian Process Priors

  • For GP priors, neural-network mean and kernel functions are meta-learned, with λ = n, β = m, negative log-likelihood, and a Gaussian hyper-prior over φ.The GP prior is parameterized by a mean function and kernel; the hyper-prior is zero-centered with variance σ²_PI.Source: §5.2 · p. 5
  • PACOH-GP has a tractable score for SVGD because its marginal log-likelihood is available in closed form.For this setup, ln Z_m(S_i, P_φ) = ln p(y_i|X_i, φ).Source: §5.2 · p. 5

5.3. Meta-Learning Bayesian Neural Network Priors

  • For BNNs, the prior over network weights is diagonal Gaussian, parameterized by its mean and log standard deviation, with a zero-centered spherical Gaussian hyper-prior.The log-space standard deviation avoids additional positivity constraints; the text also notes that other reparameterizable distributions with tractable log-density could be used.Source: §5.3 · p. 5
  • For BNNs, high-dimensional expectations and numerical instability make the generalized marginal log-likelihood intractable; LogSumExp and reparameterized Monte Carlo yield stable, consistent but not unbiased estimates.The estimator draws L reparameterized network-weight samples to estimate the gradient of the generalized marginal log-likelihood.Source: §5.3 · pp. 5–6
  • Mini-batched PACOH-NN uses space O(|Θ|K+L) and per-iteration computation O(K2 + KLnbs), replacing nested optimization with a standard stochastic problem whose complexity does not depend on task count n.Unlike MLAP, whose memory footprint is O(|Θ|n), PACOH-NN can mini-batch tasks without explicitly computing task posteriors.Source: §5.3 · p. 6

6. Experiments

Across regression and classification benchmarks, PACOH-based methods improve predictive accuracy, uncertainty calibration, scalability, and resistance to meta-overfitting. The experiments also demonstrate PACOH-NN priors in a real-world peptide-vaccine design task.

6.1. Experiment Setup

  • Regression experiments use two synthetic and four real-world environments; Omniglot classification uses 30 meta-training and 20 meta-test tasks without data augmentation or cross-alphabet recombination.Baselines include standard GP and BNN models and meta-learning methods including MAML, BMAML, MLAP, neural processes, and MLL-GP.Source: §6.1 · pp. 6–7

6.2. Experiment Results

  • PACOH achieves the best test RMSE across five regression environments, while PACOH-NN makes the most accurate Omniglot class predictions.PACOH-GP and PACOH-NN consistently perform best or rank among the best regression methods.Source: Table 1 · p. 7, §6.2 · p. 7, Table 3 · p. 7
  • PACOH yields the best uncertainty calibration in most regression environments, with substantial improvements on SwissFEL and Berkeley-Sensor.PACOH-NN consistently improves Vanilla BNN uncertainty estimates; PACOH-GP beats Vanilla GP in most environments.Source: Table 2 · p. 7, §6.2 · p. 8
  • With a standard Gaussian prior, BNN predictions and 95% confidence intervals are poor; a PACOH-NN prior learned from 20 sinusoid tasks encodes sinusoidal inductive bias.The meta-learned prior supports better predictions and uncertainty estimates even with minimal training data.Source: §6.2 · p. 7
  • PACOH-NN maintains constant memory and compute load as task counts grow, while MLAP’s resource use grows linearly and becomes prohibitive below 100 tasks.PACOH-NN avoids maintaining task posteriors and supports task-level mini-batching.Source: §6.2 · p. 8, Figure 3 · p. 8
  • PACOH-GP consistently outperforms MLL-GP and barely overfits, especially with fewer than 100 meta-training tasks.MLL-GP’s meta-training performance exceeds its meta-test performance; the gap vanishes as task counts grow.Source: §6.2 · p. 8, Figure 4 · p. 8

6.3. Meta-Learning for Sequential Decision Making

  • In peptide-design Bayesian optimization, PACOH-UCB/TS reach near-optimal regret in fewer than 10 iterations and retain a significant advantage after 50.The BNN prior was meta-learned using five alleles, with the most genetically dissimilar allele held out for the bandit task.Source: Figure 5 · p. 9, §6.3 · p. 9

7. Conclusion

The conclusion presents PACOH as a scalable PAC-Bayesian meta-learning approach with strong predictive performance and uncertainty calibration. The proof of Theorem 2 bounds transfer error by treating task-specific and task-environment generalization separately.

7. Conclusion

  • PACOH outperforms existing methods in predictive performance and uncertainty calibration, provides PAC-Bayesian guarantees without costly nested optimization, and supports positive transfer with as few as five meta-tasks.The conclusion also reports its use for sequential decision making through a vaccine-design application.Source: §7 · p. 9

A.1. Proof of Theorem 2

  • Theorem 2 bounds transfer error by combining bounds on task-specific and task-environment generalization gaps with a union bound.The proof first introduces expected multi-task error as an intermediate quantity.Source: §A.1 · pp. 12–13

Case I: bounded loss

  • For bounded loss, the proof applies Hoeffding’s lemma and Markov’s inequality to obtain a bound holding with probability at least 1 −δ.The bounded-loss case applies Hoeffding’s lemma to each factor before using Markov’s inequality.Source: Case I: bounded loss · pp. 13–14

Case II: sub-gamma loss

  • For sub-gamma loss, the proof assumes sub-gamma task-level and environment-level generalization variables, then uses their expectation bound and Markov’s inequality.The resulting inequality holds with probability at least 1 −δ.Source: Case II: sub-gamma loss · p. 14

A.2. Proof of Corollary 1

For Gibbs-optimal base learners, the resulting PAC-Bayes bound is tighter than the bound that holds uniformly over all Q.

A.2. Proof of Corollary 1

  • The bound for Gibbs-optimal base learners is tighter than Theorem 2’s bound, which holds uniformly for all Q.The proof identifies the Gibbs posterior Q* as the minimizer of the relevant objective for each prior and task.Source: §A.2 · p. 15

A.3. Proof of Proposition 1: PAC-Optimal Hyper-Posterior

The proof obtains the PAC-optimal hyper-posterior by preserving an objective’s minimizer under monotone transformations and applying a Gibbs-posterior result. The GP discussion connects posterior loss to marginal likelihood and describes the parameterized priors used by PACOH-GP.

A.3. Proof of Proposition 1: PAC-Optimal Hyper-Posterior

  • Monotone transformations of the PAC-Bayes objective preserve its minimizer, allowing Lemma 1 to yield the optimal hyper-posterior.The proof then expresses the bound for Q = Q* in terms of the meta-level partition function ZII.Source: §A.3 · pp. 15–16

B. Gaussian process regression

  • For GP regression, empirical loss under the GP posterior equals negative log-likelihood, and a Gaussian likelihood gives a closed-form marginal log-likelihood.The expression uses the kernel matrix, observation-noise variance, and mean vector.Source: B. Gaussian process regression · p. 16
  • The marginal-likelihood log-determinant acts as a complexity penalty, but may be insufficient to prevent meta-overfitting with expressive GP priors.The text identifies priors with neural-network mean and kernel functions as an example of an expressive class.Source: B. Gaussian process regression · p. 16

C. PACOH-GP: Meta-Learning of GP priors

  • PACOH-GP parameterizes both the GP mean and kernel with neural networks, using a feature map followed by a squared exponential kernel to ensure positive-definiteness.The network parameters are meta-learned.Source: C. PACOH-GP: Meta-Learning of GP priors · p. 16

C.1. Meta-training with PACOH-GP

PACOH-GP uses SVGD to approximate the optimal hyper-posterior, then forms target-task predictions by marginalizing over sampled priors. The PACOH-NN material describes analogous meta-training and target-task inference stages for neural-network priors.

C.1. Meta-training with PACOH-GP

  • SVGD meta-training estimates the hyper-posterior score with mini-batches of datasets, and the score’s independence from ZII makes SVGD tractable.The GP algorithm initializes prior particles and updates them iteratively.Source: §C.1 · pp. 16–17

C.2. Meta-Testing / Target-Training with PACOH-GP

  • For an unseen task, PACOH-GP applies a meta-learned prior and forms a predictive distribution by equally weighting predictions from sampled hyper-posterior priors.The resulting GP mixture is a mixture of Gaussians; PACOH-MAP uses one prior particle and gives a single Gaussian.Source: §C.2 · p. 17

D. PACOH-NN: Meta-Learning BNN priors

D.1. Meta-training

  • PACOH-NN meta-training minimizes the respective meta-objective by SVGD on Q*, with mini-batching available at both task and dataset levels.The passage allows a different βi for each task, set to mi or √mi.Source: §D.1 · p. 18

D.2. Meta-testing

  • In PACOH-NN target training, the base learner uses a meta-learned prior and an unseen-task dataset to approximate the generalized Bayesian posterior, with predictions evaluated on held-out data from the same task.The prediction is given as a probability distribution.Source: §D.2 · p. 18

D.3. Properties of the score estimator

The score estimator replaces an intractable marginal log-likelihood with a Monte Carlo estimate while retaining an expected upper bound on transfer error. The appendix also specifies five varied task environments, from synthetic functions to clinical and sensor data.

D.3. Properties of the score estimator

  • The biased Monte Carlo score estimator still minimizes a valid upper bound on transfer error, and its bound is tighter in expectation than the naive estimator’s.As the number of Monte Carlo samples L grows, the estimator becomes asymptotically unbiased; L = 1 removes task boundaries, while the experiments use L = 5.Source: §D.3 · pp. 19–20

E.1.1. SINUSOIDS

  • Sinusoid tasks combine affine and sinusoidal functions with parameters sampled from specified distributions, using noisy observations at inputs drawn uniformly from [-5, 5].Observation noise has standard deviation 0.1.Source: §E.1.1 · p. 20

E.1.2. CAUCHY

  • Cauchy tasks combine a two-dimensional Cauchy mixture with a zero-mean Gaussian-process function, with noisy samples drawn using truncated-normal inputs.The displayed task functions are a one-dimensional projection of the two-dimensional environment; observation noise has standard deviation 0.05.Source: §E.1.2 · pp. 20–21

E.1.3. SWISSFEL

  • SwissFEL tasks are recurrent pulse-energy optimization runs with 12 input dimensions; five runs train the meta-learner and four test it.To limit dependence among online-optimization observations, each run uses its first 400 points, split into 200 training and 200 test points.Source: §E.1.3 · p. 21

E.1.4. PHYSIONET

  • PhysioNet treats each patient as a task and GCS and HCT as environments, with 100 patients for meta-training and 500 each for validation and testing.Patients with fewer than four measurements of the relevant variable are excluded; task training-set sizes range from 4 to 24 points.Source: §E.1.4 · pp. 21–22

E.1.5. BERKELEY-SENSOR

  • Berkeley sensor tasks predict the next temperature from the previous 10 readings, separating meta-training and testing by both sensor and time.Thirty-six sensors train the meta-learner, while 10 other sensors test it on later data; the third day supplies context for target training.Source: §E.1.5 · p. 22

E.2. Experimental Methodology

The experiments evaluate predictions on held-out data from unseen tasks, reporting predictive error separately from uncertainty calibration. They also describe calibration measures, validation-based hyperparameter search, and a vaccine-development bandit setting.

E.2. Experimental Methodology

  • Evaluation tests the learned posterior on held-out data from an unseen task, reporting RMSE for mean predictions and calibration error for uncertainty quality.Results average performance and standard deviations across five random seeds.Source: §E.2 · pp. 22–23

E.2.1. CALIBRATION ERROR

  • Regression calibration error averages absolute residuals between empirical and nominal coverage at 20 equally spaced confidence levels; classification ECE compares accuracy and confidence across 20 bins.Absolute residuals preserve units, unlike the squared residuals used by Kuleshov et al. (2018).Source: §E.2.1 · p. 23

E.3. Hyper-Parameter Selection

  • Each environment-algorithm pair receives a separate hyperparameter search, using Bayesian optimization based on regression trees and average log-likelihood on validation tasks.The selected PACOH-GP settings and detailed evaluation results are provided with the reproduction materials.Source: §E.3 · pp. 23–24

E.4. Meta-Learning for Bandits - Vaccine Development

  • We meta-learn a BNN prior from 5 alleles, then test on the most genetically dissimilar allele, A-6901, in an 813-arm bandit selecting among 813 peptide candidates and receiving negative log-IC50 rewards.The reward is negative log-scale IC50 normalized so that 500 nM maps to zero. GP-UCB and GP-TS priors are meta-learned by minimizing marginal log-likelihood on five meta-training tasks.Source: §E.4 · p. 24

E.5. Further Experimental Results

MAML and Neural Processes also show meta-level overfitting: their meta-train and meta-test errors differ, with the gap shrinking as the number of tasks increases.

E.5. Further Experimental Results

  • The meta-train–meta-test RMSE gap for MAML and NPs vanishes as the number of meta-training tasks grows, supporting meta-overfitting as a relevant issue.The plotted comparison reports test RMSE on meta-training and meta-testing tasks across different numbers of tasks.Source: §E.5 · p. 24, Figure S3 · p. 25

Related papers

Loading 2002.05551v5…