Source-linked AI summary
Operator-valued Kernels for Learning from Functional Response Data
Hachem Kadri, Emmanuel Duflos, Philippe Preux, Stéphane Canu, Alain Rakotomamonjy, Julien Audiffren
TL;DR
The paper addresses supervised learning when both inputs and outputs are functions, a setting inadequately covered by existing nonlinear methods. It extends RKHS learning with operator-valued kernels and a functional learning algorithm, then applies the framework to speech inversion and sound recognition with reported performance improvements over baselines.
Problem
Nonlinear supervised learning with function-valued outputs is insufficiently developed, despite applications where inputs and labels are functions.
Method
The paper develops function-valued RKHS theory, defines operator-valued kernels, and introduces an algorithm for learning operators without discretizing covariate and target functions.
Results
The framework is successfully applied to speech inversion and sound recognition, improving over supervised baselines and generally outperforming a discrete multi-task method in speech inversion.
Takeaways & Limitations
Functional representations and operator-valued kernels provide a framework for nonlinear learning from continuous function-valued data.
Takeaways & Limitations
The concurrent functional model is limited because a response at time t depends only on the covariate value at the same time t.
Abstract
from arXiv · showhide
In this paper we consider the problems of supervised classification and regression in the case where attributes and labels are functions: a data is represented by a set of functions, and the label is also a function. We focus on the use of reproducing kernel Hilbert space theory to learn from such functional data. Basic concepts and properties of kernel-based learning are extended to include the estimation of function-valued functions. In this setting, the representer theorem is restated, a set of rigorously defined infinite-dimensional operator-valued kernels that can be valuably applied when the data are functions is described, and a learning algorithm for nonlinear functional data analysis is introduced. The methodology is illustrated through speech and audio signal processing experiments.
1. Introduction
The paper develops nonlinear supervised learning for data whose attributes and labels are functions, extending kernel methods to function-valued functions through operator-valued kernels.
- Functional supervised learning maps function-valued attributes to function-valued labels, generalizing scalar and finite-dimensional response settings.The paper frames this as learning an operator between spaces of functions.
- Functional data arise in applications such as economic curves, server loads, sounds, and hyperspectral images, where increasingly fine measurements create very large discrete representations.
- Prior work largely addressed discrete multiple-response learning, while nonlinear learning with functional outputs remained insufficiently studied.
- The proposed framework uses operator-valued kernels to map between function spaces and restate kernel-learning concepts for functional data.
- The methodology is motivated and illustrated by speech inversion and sound recognition in audio signal processing.
2. The Interplay of FDA and ML Research
The paper connects functional data analysis with machine learning by treating continuous, dependent measurements as functions and developing nonlinear kernel methods for them.
- Functional representations occur naturally in speech, lip-motion, EMG, and other signal-processing data whose samples are related along time.
- Functional data analysis represents curves or images as single function observations rather than collections of individual measurements.
- Nonlinear modeling for functional response data has been less developed than scalar- or vector-valued kernel learning.
- The paper introduces operator-valued kernels that model sample dependencies while incorporating functional properties such as curve smoothness.
- Its contributions include a spectral algorithm for operators, an algorithmic-stability analysis, and applications to speech inversion and sound recognition.
3. Notations and Conventions
This section establishes the Hilbert-space notation, function spaces, operator conventions, and block-operator terminology used throughout the paper.
- The section summarizes the notation and conventions used for Hilbert spaces, products, operators, and related matrices.
- The input and output spaces are separable Hilbert spaces of real-valued functions, with the output space taken as L2(Ωy).
- Adjoints, self-adjointness, positivity, and order for operators and block operator matrices follow the corresponding bounded-operator definitions.
- A block operator matrix consists of operator-valued entries acting on a product space of copies of Y.
4. Reproducing Kernel Hilbert Spaces of Function-valued Functions
The paper extends RKHS theory to operators mapping function-valued inputs to function-valued outputs, with nonnegative operator-valued kernels providing the construction and reproducing structure.
- The target setting treats function-valued functions as operators between infinite-dimensional Hilbert spaces.
- Extending RKHS theory to infinite-dimensional ranges requires additional restrictions because standard topological properties may not persist.
- An operator-valued kernel maps pairs of inputs to operators on the output space, and its block kernel matrix contains those operators as blocks.
- A function-valued RKHS is characterized by a nonnegative operator-valued reproducing kernel satisfying the reproducing property.
- The reproducing kernel is uniquely determined by its Hilbert space, and the associated pre-Hilbert space can be completed to construct the function-valued RKHS.
- A nonnegative Mercer operator-valued kernel is the reproducing kernel of a function-valued RKHS, extending the scalar-valued RKHS correspondence.
5. Operator-valued Kernels for Functional Data
The paper constructs operator-valued kernels to extend nonlinear kernel learning from vector outputs to functional response data. It develops building schemes, positivity-preserving combinations, and examples based on operators used in functional models.
- Motivation: Operator-valued kernels generalize matrix-valued kernels to infinite-dimensional output spaces and map between spaces of functions for functional response data.They are intended to support nonlinear functional data analysis.
- Linear Functional Response Models: Functional data analysis converts discretely measured values into functions, while functional input-output models treat the learning map as an operator between infinite-dimensional Hilbert spaces.The concurrent model is pointwise, whereas the extended model allows y(t) to depend on x(s) over an interval.
- Kernel Construction: The paper builds operator-valued kernels either by combining two input functions before mapping them to operators or by combining operators constructed from each input.The operator-combination scheme is particularly suited to nonseparable operator-valued kernels.
- Kernel Combinations: Positive operator-valued kernels remain nonnegative under addition, commuting products, and transformations of the form K = THT* for suitable operator-valued functions.These rules extend algebraic kernel-construction principles to block operator kernel matrices.
- Examples: Examples use multiplication and integral self-adjoint operators, while a composition-operator example demonstrates construction from non-self-adjoint operators.The examples are designed as alternatives for extending linear functional models to nonlinear settings.
6. Function-valued Function Learning
The paper formulates functional response learning as operator estimation in function-valued RKHSs, extending kernel methods and representer-theorem results to infinite-dimensional outputs. It develops computational solutions based on operator-valued kernels, spectral decompositions, and regularization, with stability-based generalization guarantees under stated assumptions.
- Problem: Functional response learning estimates an operator F mapping function-valued inputs to function-valued targets.The training inputs and targets lie in spaces of square-integrable functions, so the unknown predictor is function-valued.
- Regularized estimation: Regularization makes the potentially ill-posed operator-estimation problem well posed and guarantees a minimizer for λ > 0.Existence follows from coercivity and lower semicontinuity under the stated Hilbert-space conditions.
- Representer theorem: The representer theorem reduces estimation of F to a finite expansion with operator-valued kernels and function-valued weights.The resulting optimization is over functions u_i in the output space rather than directly over the operator F.
- Solution strategies: Three solution routes are described: discretization, scalar-kernel approximation of output functions, and an analytic system of linear operator equations.Discretization can overlook relationships between sampled functional observations, whereas basis-based and operator-equation approaches retain functional structure differently.
- Learning algorithm: Spectral decomposition of block operator kernel matrices selects a computationally feasible basis for representing input and output functions.The basis is implicitly determined by the reproducing operator-valued kernel rather than fixed parametrically in advance, and different operators can adapt the method to applications.
- Generalization analysis: 1/n scaling of the stability coefficient yields a generalization-error bound, while the analysis applies to kernels whose diagonal operators need not have finite trace.The latter scope is important because finite-trace assumptions can restrict generalization results in infinite-dimensional output spaces.
7. Experiments
The experiments apply operator-valued kernels to functional regression for speech inversion and functional classification for sound recognition. Results show improvements over supervised and discrete multi-task baselines, while smoothness assumptions limit some speech variables.
- 7.1 Speech Inversion: The framework is evaluated on speech inversion, which estimates vocal-tract time functions from acoustic speech signals.The experiments use acoustic-to-articulatory mapping and target vocal-tract trajectories.
- 7.1 Speech Inversion: Functional data analysis is well suited to speech inversion because articulator movements are continuous and relatively smooth over time.The method incorporates this smoothness through an integral operator-valued kernel.
- 7.1 Speech Inversion: The proposed functional approach consistently improves on hierarchical ε-SVR and generally outperforms the discrete multi-task method for vocal-tract prediction.It is weaker for LP and GLO, whose trajectories are not completely smooth; the kernel favors smooth predictions.
- 7.2 Sound Recognition: The sound-recognition pipeline represents each signal with 14 functional parameters: 13 MFCC functions and one energy function.The representation preserves dependencies among feature samples along the time axis.
- 7.2 Sound Recognition: Functional RLSC achieves consistently high classification accuracy for most sound classes compared with the standard RLSC algorithm.Performance is measured as the percentage of correctly recognized sounds.
8. Conclusion
The paper extends kernel-based learning to functional responses by modeling functional supervised learning as operator learning between infinite-dimensional Hilbert spaces. It introduces operator-valued kernels and applies the framework to nonlinear functional data analysis, including speech inversion and sound recognition.
- The methodology extends scalar- and matrix-valued kernel methods to functional response data.
- Functional supervised learning is formalized as learning an operator between two infinite-dimensional scalar-valued Hilbert spaces.
- The paper introduces rigorously defined operator-valued kernels for nonparametric learning with continuous smooth input and output functions.
- Future work includes operator-valued kernels for collaborative filtering and structured output prediction, plus online functional learning.
Appendix A. Proof of Theorem 2 - Completion of F0
The appendix constructs the function-valued RKHS by completing a pre-Hilbert space of finite kernel expansions. It establishes well-defined limits, completeness, and the reproducing property of the resulting space.
- The pre-Hilbert space consists of finite sums of kernel sections K(w_i, ·)u_i with inputs w_i and outputs u_i.
- Cauchy sequences in the pre-Hilbert space converge pointwise to Y-valued functions, using bounded evaluation and completeness of Y.
- The completed space norm is defined through limits of Cauchy sequences and is independent of the sequence representing a function.
- The original pre-Hilbert space is dense in the completion, and the completed space is complete.
- The resulting function-valued space is a reproducing kernel Hilbert space with K as its reproducing kernel.
Appendix B. Representer Theorem
The appendix proves a representer theorem for function-valued RKHS optimization. Under a nonnegative Mercer operator-valued kernel, the regularized solution is characterized through functional derivatives and kernel sections.
- The appendix establishes an analogue of the representer theorem for function-valued reproducing kernel Hilbert spaces.
- The theorem applies to a regularized optimization problem in the RKHS generated by a nonnegative Mercer operator-valued kernel.
- The proof characterizes the optimizer by setting the Fréchet derivative of the objective to zero.
- The Gateaux derivative is used to compute directional changes in the objective, including the RKHS regularization term.
- For the loss term, the derivative is expressed through kernel sections and the residual u_i = y_i − F(x_i).
Appendix C. Proof of Theorem 7
The appendix proves stability of the learning algorithm by comparing solutions trained with and without one example. Convexity and Lipschitz continuity of the loss yield the bound β = σ^2κ^2/(2nλ).
- The proof compares hypotheses learned from the full training set Z and the set Z_i obtained by removing one example.
- Convexity of the loss bounds the loss change along the interpolation between the two learned functions.
- Summing inequalities over the retained training examples and combining the resulting bounds controls the difference between the two solutions.
- The σ-Lipschitz property of the loss with respect to F(x) converts hypothesis differences into a loss bound.
Appendix D. Proof of Lemma 2
The proof establishes the least-squares conditions needed for Lemma 2 by using convexity, the vector-space structure of F, and a uniform bound derived from the boundedness assumption on Y.
- Assumption 4 is shown to hold for the least-squares loss when Assumption 5 is satisfied.This result is used to prove Lemma 2.
- Convexity of the loss in its second argument and 0 ∈ H are used in the proof.The latter follows because F is a vector space.
- The inequality derived from the definition of FZ and the bound on Y holds uniformly over Z, and therefore also holds for FZi.
- Lemma 1 and inequality (21) are then invoked to obtain the proof's concluding bound.
Appendix E. Proof of Remark 1
The proof connects the finite-trace condition for operator-valued kernels with the paper's kernel assumption under the stated Hilbert-Schmidt and trace-boundedness hypotheses.
- The paper shows that the finite trace assumption from Caponnetto and De Vito (2006) implies Assumption 1 on the kernel.
- The argument assumes Kx is Hilbert-Schmidt and supx∈X Tr(K(x, x)) < +∞.
- Under these hypotheses, the proof asserts the existence of η > 0 satisfying a condition for every x ∈ X.