Source-linked AI summary
Neural Processes
Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J. Rezende, S. M. Ali Eslami, Yee Whye Teh
TL;DR
Neural Processes address the tension between neural networks’ efficient computation and Gaussian processes’ flexible, uncertainty-aware function inference. They use neural latent-variable models to learn distributions over functions and adapt predictions from context observations, then demonstrate them across regression, image completion, optimization, and contextual-bandit tasks. The paper’s scope remains primarily diverse but relatively low-dimensional tasks.
Problem
Neural networks are efficient but cannot update outputs after training, while Gaussian processes provide flexible uncertainty-aware inference but are computationally expensive and kernel-restricted.
Method
Neural Processes use neural networks and latent variables to learn distributions over functions, estimate uncertainty from context observations, and learn an implicit data-driven kernel.
Results
The paper demonstrates Neural Processes across 1-D regression, image completion, Bayesian optimization, and contextual bandits, while comparing them with related models.
Takeaways & Limitations
Neural Processes combine flexible test-time predictions and uncertainty estimation with computationally efficient inference and data-driven representations.
Takeaways & Limitations
The demonstrated tasks are diverse but relatively low-dimensional, leaving scaling to higher-dimensional problems for future work.
Abstract
from arXiv · showhide
A neural network (NN) is a parameterised function that can be tuned via gradient descent to approximate a labelled collection of data with high precision. A Gaussian process (GP), on the other hand, is a probabilistic model that defines a distribution over possible functions, and is updated in light of data via the rules of probabilistic inference. GPs are probabilistic, data-efficient and flexible, however they are also computationally intensive and thus limited in their applicability. We introduce a class of neural latent variable models which we call Neural Processes (NPs), combining the best of both worlds. Like GPs, NPs define distributions over functions, are capable of rapid adaptation to new observations, and can estimate the uncertainty in their predictions. Like NNs, NPs are computationally efficient during training and evaluation but also learn to adapt their priors to data. We demonstrate the performance of NPs on a range of learning tasks, including regression and optimisation, and compare and contrast with related models in the literature.
1. Introduction
Neural Processes (NPs) combine neural networks’ computational efficiency with stochastic processes’ ability to model uncertainty and adapt predictions from context observations. The paper introduces NPs and evaluates them across diverse learning tasks.
- Neural networks efficiently evaluate learned functions but cannot update outputs after training, motivating models that adapt at test time.
- Gaussian processes support inference conditioned on observations and represent uncertainty, but classic inference scales as O((n+m)^3) and approximations as O((n+m)^2).
- NP inference with n context points and m target points uses a trained neural-network forward pass that scales as O(n+m).
- NPs learn implicit kernels from data rather than relying on restricted handcrafted kernel forms and separate hyperparameter optimisation.
- Neural Processes combine benefits of neural networks and stochastic processes in a single model class.
- The paper compares NPs with related work and applies them to 1-D regression, image completion, Bayesian optimization, and contextual bandits.
2. Model
Neural Processes represent stochastic processes with neural latent-variable models that combine function-distribution modeling, uncertainty estimation, context-based adaptation, and efficient neural computation. Their construction uses exchangeable and consistent finite-dimensional distributions, latent global function representations, and permutation-invariant context aggregation.
- Stochastic-process foundations: A stochastic process is defined through finite-dimensional marginal distributions over function values at finite input sequences.For Gaussian processes, these marginals are multivariate Gaussians parameterised by mean and covariance functions.
- Stochastic-process foundations: Finite exchangeability and consistency are sufficient to define a stochastic process from its collection of joint distributions.Exchangeability requires permutation invariance, while consistency requires unchanged marginals after integrating out part of a sequence.
- Latent-variable representation: Neural Processes approximate a random function by parameterising it as F(x) = g(x, z), where z is a high-dimensional random vector and g is learnable.The latent variable supplies the randomness of the function, while the decoder neural network captures model complexity.
- Training and prediction: NP training uses multiple datasets concurrently so the model can learn variability across random functions rather than fit a single function.At test time, context points are used to predict target values at target positions.
- Global uncertainty: The latent variable z captures global uncertainty, enabling sampling of whole functions rather than independent output values.Conditioning its learned prior on context produces a posterior over functions, and different context sizes encourage flexibility to context number and position.
- Neural process architecture: The NP implementation combines an encoder h, an order-invariant aggregator a, and a decoder g to process context points efficiently.The encoder maps each (x, y) pair to a representation, while the aggregator summarizes these representations into a global latent-distribution parameterisation.
3. Related work
The related work positions Neural Processes among neural, Gaussian-process, meta-learning, Bayesian, and conditional latent-variable models. It highlights shared goals and distinguishes NPs by their function-level distributions, targeted sampling, and trade-off between guarantees and efficiency.
- Conditional neural processes: Neural Processes generalise Conditional Neural Processes by adding a latent variable for global sampling of different functions from the same context.
- Neural networks and Gaussian processes: Neural Processes and Gaussian processes support few-shot function estimation, whereas neural networks learn a single function directly from much larger datasets.
- Neural networks and Gaussian processes: Matching networks and deep kernel learning use neural representations with explicit distance kernels, while Neural Processes learn their comparison mechanism implicitly.
- Neural networks and Gaussian processes: Neural Processes remain closer to neural networks than most related models, trading Gaussian-process mathematical guarantees for data-driven priors and computational efficiency.
- Conditional latent variable models: Neural statisticians model distributions over sets, whereas Neural Processes model distributions over functions with pair-wise x-y relations.
- Conditional latent variable models: Conditional latent variable models learn p(yT | yC, z), but Neural Processes add an x variable for targeted latent-distribution sampling and broader applications.
4. Results
Neural Processes are evaluated on regression, image completion, Bayesian optimisation, and contextual bandits. Across these tasks, they adapt predictions to context and provide uncertainty, while remaining competitive with established methods.
- 1-D function regression: In 1-D regression, NP samples capture continuity, GP-like shape, context conditioning, and higher uncertainty where observations are absent.More context reduces uncertainty and improves agreement with the ground-truth function, although samples may only approach observations rather than pass exactly through them.
- 2-D function regression: Pixel-wise image completion treats pixel coordinates as inputs and pixel brightness as outputs, with separate models trained on MNIST and CelebA.Increasing context reduces variability and brings predicted pixels closer to the underlying images.
- 2-D function regression: CelebA reconstructions remain imperfect and variable even with all pixels observed because the latent variable z forms a strong bottleneck.The latent variance is clipped to a small value to avoid collapsing, yet samples can still differ when uncertainty is absent.
- Black-box optimisation with Thompson sampling: In Bayesian optimisation, NPs require four times fewer iterations than random search on functions generated from a GP prior.A GP with the correct kernel remains an upper bound, while NP samples are noisier but faster to evaluate through a forward pass.
- Contextual bandits: Neural Processes are highly competitive on the wheel bandit problem, performing similarly to MAML and the NeuralLinear baseline.NeuralLinear is consistently among the best of 20 compared algorithms in the cited evaluation.
5. Discussion
Neural Processes combine stochastic-process and neural-network benefits by learning distributions over functions and making flexible context-conditioned predictions. The paper demonstrates this flexibility on diverse regression tasks while leaving higher-dimensional applications for future work.
- Neural Processes combine stochastic processes and neural networks while learning an implicit measure directly from data instead of using a handcrafted kernel.
- NPs represent distributions over functions and make flexible predictions at test time conditioned on context inputs.
- The paper applies NPs to diverse regression tasks, but these tasks are relatively low-dimensional and higher-dimensional scaling remains future work.