Kernels for Vector-Valued Functions: a Review

Mauricio A. Alvarez; Lorenzo Rosasco; Neil D. Lawrence

Source-linked AI summary

Kernels for Vector-Valued Functions: a Review

Mauricio A. Alvarez, Lorenzo Rosasco, Neil D. Lawrence

arXiv:1106.6251v2Jun 30, 2011stat.MLcs.AImath.ST

TL;DR

Multi-output learning seeks to exploit dependencies among related prediction problems, particularly when data are limited. The survey reviews valid vector-valued kernels across regularization and Gaussian-process frameworks, finding separable and nonseparable kernel families with established constructions but open model-selection issues.

Problem
Multi-output learning needs methods that exploit dependencies among related prediction problems rather than solving each output independently, especially when data are limited.
Method
The survey reviews vector-valued kernel constructions in regularization theory and Gaussian processes, emphasizing their connections and common latent-function or output-coupling representations.
Results
The survey organizes multiple-output kernels into separable and nonseparable families, including separable kernels, linear models of coregionalization, and process convolutions.
Takeaways & Limitations
Multiple-output models represent cross-task relationships through regularization couplings or correlated generative processes, providing a shared framework for kernel methods and Gaussian processes.
Takeaways & Limitations
Model-selection questions about output correlation, latent-function number, coregionalization rank, and input-space configuration remain open issues.

Abstract

from arXiv · show

Kernel methods are among the most popular techniques in machine learning. From a frequentist/discriminative perspective they play a central role in regularization theory as they provide a natural choice for the hypotheses space and the regularization functional through the notion of reproducing kernel Hilbert spaces. From a Bayesian/generative perspective they are the key in the context of Gaussian processes, where the kernel function is also known as the covariance function. Traditionally, kernel methods have been used in supervised learning problem with scalar outputs and indeed there has been a considerable amount of work devoted to designing and learning kernels. More recently there has been an increasing interest in methods that deal with multiple outputs, motivated partly by frameworks like multitask learning. In this paper, we review different methods to design or learn valid kernel functions for multiple outputs, paying particular attention to the connection between probabilistic and functional methods.

1 Introduction

Multiple-output learning uses dependencies among related prediction problems to improve performance, especially when data are limited. The survey reviews kernel and Gaussian-process approaches while connecting Bayesian and regularization perspectives.

Applications use cross-output dependencies to predict missing sensor signals, estimate expensive measurements from proxies, and complete correlated motion sequences.
Multiple-output learning represents predictions as vector-valued functions, equivalently treating each output component as a related supervised problem.
Exploiting relationships among outputs can improve on solving each problem separately and help cope with limited data.
The survey reviews recent kernel-method and Gaussian-process results for multi-output learning and analyzes their Bayesian–regularization connections.
The paper introduces scalar kernel methods before extending related concepts to vector-valued functions and discussing multiple-output kernel construction.

2 Learning Scalar Outputs with Kernel Methods

Scalar kernel learning uses RKHS regularization and Gaussian processes to construct estimators and quantify uncertainty. The survey also explains their connection through feature representations and maximum a posteriori estimation.

Kernel methods: Kernel methods build scalar estimators from training data using symmetric positive-definite kernels.
A Regularization Perspective: RKHS regularization balances empirical squared error against a function-complexity penalty, stabilizing learning when data are scarce or noisy.
A Regularization Perspective: The representer theorem reduces the minimizer to a linear combination of kernels centered at training points.
A Bayesian Perspective: A Gaussian process specifies a function prior through its mean and covariance kernel, which combine with a likelihood to produce a predictive posterior.
A Bayesian Perspective: Under a Gaussian likelihood, posterior prediction is analytic, while non-Gaussian likelihoods require approximations such as Laplace or variational methods.
A Connection Between Bayesian and Regularization Point of Views: A finite-dimensional feature-map construction links Gaussian-process priors with RKHS functions, and maximum a posteriori estimation yields Tikhonov regularization.

3 Learning Multiple Outputs with Kernels Methods

Multi-output learning extends scalar supervised learning to vector-valued functions, with kernels represented as matrix-valued functions that encode relationships among outputs. Both vector-valued RKHS regularization and vector-valued Gaussian processes provide learning frameworks, and their predictive means coincide.

3.1 Multi-output Learning: Multi-output learning estimates a vector-valued function from several related prediction problems, including settings with shared or distinct input spaces and training-set sizes.The paper distinguishes multi-output learning broadly from multi-task learning, where components may have different inputs.
3.2 Reproducing Kernel for Vector Valued Function: A vector-valued reproducing kernel is a positive semi-definite matrix-valued function K: X × X → R^D×D.The kernel acts on output components as well as inputs and satisfies the reproducing property in the corresponding vector-valued RKHS.
3.2 Reproducing Kernel for Vector Valued Function: Vector-valued RKHS functions are represented as linear combinations of matrix-valued kernels centered at training inputs, with the RKHS norm measuring function complexity.Regularized empirical-error minimization yields a representer-theorem solution whose coefficients satisfy a linear system.
3.3 Gaussian Processes for Vector Valued Functions: In vector-valued Gaussian processes, each output corresponds to a process, while K(x,x′) contains covariances expressing cross-output correlation or similarity.For Gaussian likelihoods, predictive distributions and marginal likelihoods can be derived analytically.
3.3 Gaussian Processes for Vector Valued Functions: The Gaussian-process predictive mean coincides with the estimator obtained from the regularization framework.This establishes a direct correspondence between the probabilistic and functional approaches for vector-valued learning.

4 Separable Kernels and Sum of Separable Kernels

Separable kernels decouple input and output contributions by multiplying an input-space scalar kernel with an output-interaction kernel; sums of such products provide a more general class. The central design problem is selecting scalar kernels and learning output-interaction matrices, with equivalent regularization and Bayesian interpretations.

4 Separable Kernels and Sum of Separable Kernels: Separable kernels are products of an input-space kernel and an output kernel encoding interactions among outputs.Their defining structure decouples input and output contributions.
4 Separable Kernels and Sum of Separable Kernels: Sum of separable kernels generalizes this construction by combining multiple input-output kernel products.The associated kernel matrix has a Kronecker-product form.
4 Separable Kernels and Sum of Separable Kernels: Setting the output kernel to the Kronecker delta treats outputs as unrelated, whereas a general matrix B encodes dependencies among them through off-diagonal terms.With unrelated outputs, the kernel matrix becomes block diagonal.
4 Separable Kernels and Sum of Separable Kernels: Choosing the scalar input kernels and designing or learning the output-interaction matrices are the key modeling questions.These choices determine how input similarity and output dependencies are represented.
4 Separable Kernels and Sum of Separable Kernels: The same separable-kernel class can be approached through regularizers or Bayesian generative models, and the two perspectives are equivalent.They provide different interpretations of the same models.

4.1 Kernels and Regularizers

Regularizers for vector-valued functions induce separable matrix-valued kernels whose output-coupling matrices encode relationships among tasks. Examples impose common similarity, cluster-based similarity, or graph-based relationships, with corresponding Bayesian interpretations.

4.1 Kernels and Regularizers: For kernels of the separable form, the RKHS norm becomes a regularizer coupling task components through inner products weighted by the pseudoinverse B†.Thus, the matrix B specifies how different tasks are coupled in the regularization functional.
4.1 Kernels and Regularizers: A mixed-effect regularizer combines component-wise norm control with attraction toward the mean estimator across outputs.Its parameter ω controls the strength of the shared similarity structure.
4.1 Kernels and Regularizers: At ω = 0, components are treated independently; at ω = 1, they are assumed identical and explained by the same function.Intermediate values interpolate between these two cases.
4.1 Kernels and Regularizers: Cluster-based regularization groups components into r clusters and enforces similarity among components assigned to the same cluster.The cluster-membership matrix E defines the resulting output-similarity matrix M.
4.1 Kernels and Regularizers: These regularizer-induced models can be derived from suitable generative models in the Bayesian framework.The paper uses this connection to relate functional and probabilistic constructions.
4.1 Kernels and Regularizers: Graph regularization combines standard component-wise regularization with similarity constraints determined by a D × D positive weight matrix M.The resulting kernel has the form k(x,x′)L†, where k is a scalar input kernel.

4.2 Coregionalization Models

Coregionalization models construct valid multi-output covariance functions by combining output-dependent coregionalization matrices with input-dependent covariance functions. The ICM is a restrictive separable special case, while the LMC supports richer output behavior through multiple latent-function groups and covariance scales.

Linear model of coregionalization: The LMC expresses outputs as linear combinations of independent latent random functions, producing a valid positive semidefinite joint covariance.Latent functions may be grouped by shared covariance functions, with coefficients determining output relationships.
Linear model of coregionalization: The resulting covariance is a sum of separable kernels, combining coregionalization matrices Bq for output dependence with covariance functions kq for input dependence.Each Bq captures relations among outputs, while kq captures dependence on the inputs.
Intrinsic coregionalization model: The ICM sets Q = 1 and uses a single separable kernel K(x, x′) = k(x, x′)B, making it a special case of the LMC and regularization kernels.The rank of B is determined by the number of latent functions contributing to the model.
Intrinsic coregionalization model: The ICM is more restrictive than the LMC because each basic covariance contributes equally to output autocovariances and cross covariances, although inference is greatly simplified.The simplification is associated with properties of the Kronecker product.
Comparison between ICM and LMC: For ICM rank one, outputs have the same form and length-scale but different variances, whereas rank two permits different output forms despite a shared length-scale.Rank-one samples are scaled versions of a latent function; rank-two samples are weighted sums of two latent functions sharing one covariance.

4.3 Extensions

Extensions broaden separable multi-output kernels through learned task structure, grouped regularization, and correlated latent functions. These approaches include convex relaxations, stochastic optimization, and cross-coregionalization matrices that allow negative cross-covariances while preserving positive semidefiniteness.

Extensions within the regularization framework: Extensions within regularization learn output-coupling matrices jointly with predictors, with convex relaxations making otherwise intractable optimization problems tractable.The learned matrix can be interpreted from a Bayesian perspective as a covariance matrix for response variables.
Extensions within the regularization framework: Grouped-task models learn both a partition of tasks and the variables selected for each group, reflecting that different task subgroups may share different features.The formulation is motivated by the assumption that a response variable need not be important for all tasks.
Extensions within the regularization framework: The grouped minimization problem is nonconvex, but stochastic gradient descent can find local minimizers that appear to perform well in practice.The objective includes matrices that couple otherwise disjoint component-wise problems.
Extensions from the Gaussian processes perspective: A Gaussian-process extension uses nonorthogonal latent functions and cross-coregionalization matrices Bq,q′ to represent correlated attributes.This generalizes the covariance construction beyond independent latent processes.
Extensions from the Gaussian processes perspective: Cross-covariances kq,q′(x, x′) may be negative while the full output covariance remains positive semidefinite.The paper connects this flexibility to applications combining correlated attributes in mining, hydrology, and the oil industry.

5 Beyond Separable Kernels

Beyond separable kernels, the survey reviews constructions that couple inputs and outputs nontrivially, including invariant, transformable, spatially varying, and process-convolution kernels. These approaches support output-specific structure such as divergence, curl, smoothness, and length-scale differences.

Invariant kernels: Matrix-valued exponentiated quadratic kernels model divergence-free or curl-free vector fields and cannot generally be reduced to k(x, x′)B.Their input-dependent component similarities distinguish them from separable kernels.
Invariant kernels: A convex combination of divergence-free and curl-free kernels can model general vector fields while separately reconstructing their two components.The construction combines the two kernel types within the matrix-valued framework.
Transformable kernels: Transformable kernels define matrix entries as k(T_d x, T_d′ x′), using output-specific transformations before applying a continuous scalar kernel.This construction permits output interactions through the transformation maps rather than a fixed output matrix.
Spatially varying kernels: The spatially varying LMC allows each coregionalization matrix B_q(x, x′) to depend on the input variables, with inference performed by Markov chain Monte Carlo.This extends the standard LMC by making output interactions input-dependent.
Process convolutions: Process convolutions dynamically transform latent Gaussian processes with smoothing kernels, producing non-separable covariances and output-specific variation.Each output is associated with a different smoothing kernel, which can control its smoothness and length-scale; process convolutions are described as a dynamic counterpart to instantaneous LMC mixing.
Comparison with LMC: Process convolutions can represent different output length-scales directly, whereas LMC may require more components or a compound model with questionable ability to capture relevant output correlations.In the example, process convolution produces long-scale behavior for f1(x) and short-scale behavior for f2(x); larger LMC Q increases expressiveness but also computational burden.

6 Inference and Computational Considerations

The paper discusses parameter estimation and computational complexity for multiple-output kernel and Gaussian-process models. Separable kernels, spectral transformations, and approximation strategies can reduce the burden of large matrix inversions.

Parameter estimation: Hyperparameters can be tuned by cross-validation or marginal-likelihood maximization, while diagonalization and reduced-rank approximations address computational complexity.These choices apply to practical use of multiple-output kernels.
Computational complexity: General multiple-output systems can require O(D3N 3) time and O(D2N 2) storage in the worst case.Cholesky decomposition suits a single regularization value, whereas SVD supports solutions across multiple values such as during cross-validation.
Separable kernels: For separable kernels K(x, x′) = k(x, x′)B, output-space eigendecomposition transforms the problem into D essentially independent rescaled problems.The transformation rotates the outputs into the eigenvector coordinates of B and rescales each kernel matrix or regularization parameter.
Computational complexity: The transformed computation costs O(D3) + O(N 3), compared with O(D3N 3) in the general case.The coupling among tasks is represented as a rotation and rescaling of output points.
Gaussian-process inference: Gaussian-process parameter estimation commonly maximizes the marginal likelihood, while full Bayesian inference assigns priors to hyperparameters and computes their posterior.For non-Gaussian likelihoods, Laplace, variational Bayes, or expectation-propagation approximations can replace unavailable closed-form quantities.
Computational limitations: Process-convolution kernels may require solving a double integral, which is not feasible for every choice of smoothing kernels.Analytically tractable cases and discrete process convolutions provide alternatives.

7 Applications of Multivariate Kernels

Multivariate kernels are applied across geostatistics, computer emulation, time series, sensor networks, robotics, preference elicitation, imaging, environmental monitoring, and biological systems. These applications use shared structure among outputs to reconstruct, predict, or infer otherwise difficult-to-measure quantities.

Computer emulation: Multivariate Gaussian-process emulators model outputs such as structural vibration responses and time-series values across varying physical inputs.In time-series emulation, output dimensions can correspond to time points, while distinct input values represent different series.
Sensor networks: The intrinsic coregionalization model captures dependencies among environmental sensor measurements so faulty readings can be interpolated using healthy sensors.Nearby sensors are described as producing similar readings.
Robotics: The intrinsic coregionalization model is used to infer robotic joint torques across angles, velocities, accelerations, and end-effector loadings.The outputs are torques at different joints under multiple dynamic contexts.
Process-convolution applications: Process convolutions model dependencies in gene regulatory networks and motion-capture data through latent functions and smoothing kernels.The examples include transcription-factor dynamics and body-part angle time courses for dancing movements.
Process-convolution applications: Discretized process convolutions are used for image reconstruction and information fusion from multiple sensors.These applications illustrate process-convolution methods beyond continuous covariance constructions.
Geostatistics: Geostatistical models use inexpensive, oversampled variables as proxies for predicting expensive pollutant measurements.Examples include extracting information about fine particulate matter from abundant coarse-particle readings.

8 Discussion

The discussion organizes vector-valued kernels into separable and nonseparable families and relates their regularization and Gaussian-process interpretations. It identifies model selection for output dependence, latent structure, rank, and input configuration as an open research direction.

Frameworks: The survey presents multiple-output kernels for both regularization theory and Gaussian processes.The regularization view constrains learned vector-valued functions, whereas the Gaussian-process view uses common latent functions to express output correlations.
Kernel families: Separable kernels factor into an output kernel and an input-space kernel, while nonseparable kernels include process convolutions.The linear model of coregionalization is described as the most general separable model, and process convolution as a generalization of it.
Open problems: Model selection remains an open issue involving latent-function number, coregionalization rank, output dependence, and isotopic versus heterotopic inputs.The paper notes that these questions have been studied to some extent in geostatistics but remain unresolved.

Notation

The notation defines dimensions for inputs, outputs, and observations, then specifies scalar and matrix-valued kernels, latent functions, vector-valued functions, and block kernel matrices.

Dimensions: p denotes input dimensionality, D the number of outputs, N_d data points for output d, and Q the number of latent functions.The notation distinguishes output-specific training inputs X_d from the common input space X.
Functions and kernels: k is a scalar kernel, K is a matrix-valued kernel, k_q is a latent-function kernel, and f is the vector-valued output function.The d-th output is denoted f_d(x).
Evaluated quantities: k_q(X, X) and f_d(X_d) denote latent-kernel matrices and output vectors evaluated on training inputs.The output vector f_d(X_d) stacks evaluations of f_d over X_d.
Matrix notation: K(X, X) is the block matrix whose blocks K(X_d, X_d′) contain cross-output kernel evaluations, and I_N is the N-sized identity matrix.The block structure accommodates potentially distinct training inputs for different outputs.