Source-linked AI summary
Second-order optimization of variable projection SVM models and road abnormality detection
Andrea Angino, Matthias Voigt, Rolf Krause, Tamás Dózsa
TL;DR
VP-SVMs have learnable kernels that complicate training, while earlier reduced-objective methods may compromise classification and interpretability. The paper derives second-order optimization formulas and applies trust-region training to road-abnormality detection. The authors report reduced training time and improved generalization compared with gradient descent.
Problem
VP-SVM training is difficult because its kernel matrix scales with the training signals and depends on learnable parameters, while reduced-objective stochastic subgradient training may yield suboptimal classification and reduced interpretability.
Source: §1 · p. 2Method
Results
Takeaways & Limitations
The experiments demonstrate the effectiveness of second-order trust-region training for VP-SVMs on road-abnormality recognition.
Source: §1 · p. 2, §6 · p. 5The stated setup assumes the basis functions remain linearly independent for all parameter vectors.
Source: §2 · p. 2
Abstract
from arXiv · showhide
We introduce a novel second-order optimization framework for minimizing so-called variable projection functionals. We demonstrate that the proposed framework is especially usefulfor the training of variable projection based kernel methods. In particular, the problem of efficiently training variable projection support vector machines (VP-SVMs) is considered. We show the effectiveness of the proposed training methodology in a real-world application, namely we demonstrate how second-order trust region algorithms can be used to train VPSVM models to recognize road surface abnormalities based on 1D signals obtained from a tire sensor.
1. INTRODUCTION
The paper applies second-order trust-region training to interpretable VP-SVMs, addressing their difficult optimization and demonstrating road-abnormality classification with reduced training time and improved generalization.
1. INTRODUCTION
- VP-SVMs combine interpretable parameters with an adaptive feature transformation that is optimized alongside the SVM classifier weights.The transformation maps samples to a low-dimensional space.Source: §1 · p. 2
- VP-SVM training is especially challenging because its kernel matrix scales with training signals and also depends on learnable parameters.The reduced objective and stochastic subgradient methods proposed earlier may yield suboptimal classification and reduced interpretability.Source: §1 · p. 2
- Trust-region training reduced VP-SVM training time while significantly improving generalization for road-surface abnormality classification.The application uses 1D signals from a tire sensor.Source: §1 · p. 2
- The framework derives analytic second-order formulas for Moore–Penrose pseudoinverses, variable-projection operators, and primal quadratic VP-SVM objectives.The authors state that the approach can extend to kernel models whose kernels depend on learnable parameters.Source: §1 · p. 2
2. VARIABLE PROJECTION OPERATORS
Variable projection projects a signal onto a parameter-dependent basis and measures the residual; minimizing that residual is a separable nonlinear least-squares problem.
2. VARIABLE PROJECTION OPERATORS
- For a fixed parameter vector, the projection is the closest element in the basis functions’ span to the signal.The stated setup assumes the basis functions remain linearly independent, including at the sampling points.Source: §2 · pp. 2–3
- The variable-projection operator maps a signal to its orthogonal projection using the basis matrix and its Moore–Penrose pseudoinverse.The projection depends on the parameter vector.Source: §2 · p. 3
- The feature transformation returns the projection’s coordinates in the basis, providing dimension reduction and feature extraction.The basis functions may be chosen using prior information about the signal’s behavior.Source: §2 · p. 3
- Minimizing the residual functional is a separable nonlinear least-squares problem, for which analytic Jacobian formulas enable gradient-based optimization.The cited formulas concern the pseudoinverse and projection operator.Source: §2 · p. 3
3. VP-SVM MODELS
The VP-SVM model uses a kernel classifier with a jointly learned feature transformation and classifier weights, trained by choosing its model parameters.
3. VP-SVM MODELS
- The nonlinear SVM classifies signals using kernel similarities and weights that define a separating hyperplane in the lifted feature space.The kernel maps samples into a reproducing-kernel Hilbert space.Source: §3 · p. 3
- The model uses an RBF kernel and jointly learns the feature transformation with the underlying classifier weights.The transformation is intended to improve classification performance when appropriately chosen.Source: §3 · p. 3
- Training seeks suitable transformation parameters and classifier weights; the quadratic loss term is a classical SVM loss.The loss is twice differentiable except when a sample lies exactly on the separating hyperplane.Source: §3 · p. 3
4. SECOND-ORDER DIFFERENTIATION OF VP-OPERATORS
The section develops derivatives needed to incorporate second-order curvature into VP-SVM training. It gives formulas for the regularizer and kernel-dependent objective terms under stated smoothness and active-set assumptions.
4. SECOND-ORDER DIFFERENTIATION OF VP-OPERATORS
- The regularizer Hessian requires first- and second-order derivatives of the variable projection operator and the pseudoinverse.The section supplies second-order formulas for both, assuming the relevant pseudoinverse derivatives exist and are known.Source: §4 · pp. 3–4
- Kernel-dependent objective terms A and B are differentiated assuming the kernel is twice differentiable.The section also gives the mixed derivatives needed to express the Hessian.Source: §4 · p. 4
- The active-set assumption identifies the first r labels as those satisfying 1 − y_iβ^⊤K(η)_i > 0.This ordering is used in Eq. (6).Source: §4 · p. 4
5. EXPERIMENTS
The experiments compare gradient descent with a second-order trust-region method for training VP-SVMs on tire-sensor signals. Trust-region training performs better in the reported classification tests while using less training time.
5. EXPERIMENTS
- The dataset contains 517 equal-length, zero-padded tire-revolution signals recorded by a tire-implanted force sensor on Budapest public roads.The signals represent tire deformation and are used to recognize road-surface abnormalities.Source: §5 · p. 4
- The model uses five Hermite-Fourier coefficients, compared with 11 adaptive Hermite functions in prior VP-based neural networks that achieved 97% average accuracy.The authors report that their training scheme achieves performance close to the state of the art.Source: §5 · p. 5
- TR uses second-order derivatives to choose an approximate trust-region step, while GD uses first-order information and a fixed learning rate of 10^-3.After a rejected step, TR needs only one kernel evaluation and does not require recomputing the Hessian or kernel derivatives.Source: §5 · p. 5
- In 5-fold cross-validation, TR consistently achieves higher accuracy with fewer iterations and less time than GD; GD takes more than twice as long and still fails to match TR’s accuracy.Table 1 reports mean gradient norms and accuracies, with minimum and maximum values across tests.Source: Table 1 · p. 5, §5 · p. 5
- Figure 1 shows faster loss reduction and higher accuracy for TR on both training and test sets, with faster gradient-norm decay on training data.The figure’s top and middle plots show objective and accuracy; its bottom plot shows training gradient norm over time.Source: Figure 1 · p. 5, §5 · p. 5
6. CONCLUSION
The paper proposes second-order training schemes for VP-SVM classifiers and demonstrates the effectiveness of trust-region training with second-order information.
6. CONCLUSION
- The paper proposes second-order training schemes for VP-SVM classifiers and demonstrates the effectiveness of trust-region training with second-order information.Source: §6 · p. 5