Source-linked AI summary
Convex losses and their applications to SVM, SVR, and Shallow Neural Networks
Filippo Portera
TL;DR
Existing generalized loss formulations have promising theory but unresolved numerical use, so this paper introduces correlation-aware convex losses for SVMs and neural networks. Experiments on small datasets find generalized losses are rarely worse and generally better than standard losses, with performance comparable to neural-network baselines.
Problem
The numerical solution of generalized quadratic losses with pattern-correlation matrices remains poorly understood, limiting evaluation of their sound dual formulations.
Method
The paper introduces convex losses that weight errors through pattern-correlation matrices and applies them to primal SVMs and neural networks.
Results
Generalized losses were rarely worse and generally better than standard losses on the tested small problems, with performance comparable to neural-network baselines.
Takeaways & Limitations
Within these experiments, correlation-aware generalized losses can match or exceed standard-loss performance across several machine-learning tasks.
Takeaways & Limitations
The experiments mostly use small problems, motivating evaluation on bigger datasets and more efficient, effective matrices for deep neural networks.
Abstract
from arXiv · showhide
We propose multiple new convex losses for SVM and Neural Networks, applied to binary classification tasks. While there are practical limitations in exploiting them with the dual SVM models, we are able to use them with SVM primal formulation and Neural Networks. In detail, the primal SVM problem with the modified losses has been solved with the Particle Swarm Optimization algorithm. We prove that the proposed losses are a generalization of the standard loss, and we experiment them with several small data-sets. This preliminary study shows that using pattern correlations inside the loss function could in theory enhance the generalization performances on some data-sets. To evaluate the performance of each loss, we adopt a Nested Cross-Validation procedure. Results show that generalization measures are the same with or without the new losses.
1 Introduction
The section introduces novel convex loss functions for evaluating machine-learning generalization, based on a modification of the generalized quadratic loss. Because the proposed loss generalizes the standard loss, it is theoretically expected to perform equally well or better, with dual SVM objectives derivable for kernel computation.
- The study measures the generalization performance of several machine-learning algorithms using novel loss functions.
- The proposed convex loss modifies the generalized quadratic loss described in prior work.
- Because it generalizes the standard loss, the proposed loss is theoretically expected to perform equally well or better.
- The authors derive two dual objective functions that can be computed using kernel functions for SVM.
2 Related work
The study builds on prior work on modified SVM and SVR problems and uses autograd to evaluate arbitrary loss functions in shallow Neural Networks. It also evaluates these losses on small UCI binary-classification datasets.
- Prior work: The study is partially focused on prior work on modified SVM and SVR problems and cites the original introduction of SVM.The cited works are [A J02],, and [C C95].
- Neural Network methodology: Autograd enables shallow Neural Networks to evaluate arbitrary loss functions by automatically computing their gradients.This capability is attributed to.
- Evaluation: The losses are evaluated with Neural Networks on small UCI datasets for binary classification tasks.The experiments use some small UCI datasets.
3 Some Machine Learning models for binary classification and regression
The section formulates the proposed convex losses for SVM, SVR, and shallow neural networks, while noting that the resulting dual problems remain difficult to solve. It therefore uses primal optimization with PSO for SVM and precomputed loss matrices in shallow neural networks.
- SVM formulation: The proposed loss formulation uses training samples, binary targets, RBF functions, and an l × l matrix parameterized by γF.For sufficiently large γF, the matrix becomes the identity and the problem reduces to standard Support Vector Classification.
- Dual limitations: The SVM dual depends on dual slack variables ξ and is not easy to solve, while the analogous SVR dual depends on ξ and ξ* and is also unsolved.Both models are derived through Lagrangian and KKT conditions, with estimator functions obtained from their respective dual formulations.
- Primal optimization: PSO combined with a standard-SVM dual solver is used to solve the primal SVM problem with the new losses.The dual optimizer pretrains PSO, which then optimizes α and b using 20 particles.
- Experiments: The proposed losses are tested with several losses and various F matrices, including settings where the identity matrix recovers the standard case.The identity matrix must be included among the available F matrices to retain the standard formulation.
- Shallow neural networks: For shallow neural networks, the F matrix is precomputed over the whole training set and implemented in PyTorch with Adam optimization.The models are generally UCI-data-set networks with max 4 layers, DropOut, and Batch Normalisation.
4 Data-sets
The study uses binary-classification data-sets drawn from the UCI repository, with descriptions provided in Table 1.
- The experiments use binary-classification data-sets from the UCI Data-sets repository.
- All data-sets used in the study are described in Table 1.
- The data-set selection is presented as the basis for the paper’s binary-classification experiments.
5 Method
The method defines new losses by separately specifying error weighting and weighting matrices, then evaluates them for SVM, PSO, and shallow neural networks. Model selection uses nested cross-validation with task-specific validation procedures.
- Loss construction: New losses are defined along two dimensions: how errors are weighted and how error-weighting matrices are defined.The first dimension is addressed in one subsection and the second in another.
- Loss construction: The evaluated losses include binary cross-entropy with logits, with L0 identified as the standard BCE loss.The loss formulation is defined using model-output and target-value errors.
- Loss construction: Several F matrices are tested to capture pattern similarities or dissimilarities, each parameterized by the γF hyper-parameter.The matrices can be normalized to match the identity matrix’s norm, using Manhattan distance in their construction.
- Validation: SVM and PSO models use nested cross-validation with 5 outer loops and 3 inner loops, tuning C, γK, and γS.The procedure is used for model validation and hyper-parameter tuning.
- Validation: Shallow neural networks use nested cross-validation with 5 outer loops and 3 inner loops, repeated 10 times to mitigate random-initialization effects.The parameter grid includes hidden-neuron count, learning rate, and γF.
6 Results
Section 6 defines the evaluated algorithms and baselines, then analyzes their computational complexity and experimental reporting procedure. The study reports mean accuracy, standard deviation, and running time across 10 nested-cross-validation runs.
- Complexity: SVM requires an l×l Gram matrix, resulting in O(l^2) kernel evaluation and O(l^2) memory, while training also depends on C, γ_K, and γ_F.Certain SVM hyper-parameter configurations could prevent training from completing.
- Complexity: Neural-network complexity is driven by epochs e, examples l, features d, weights w, batch size b, and the adopted loss.For the standard loss, the four per-batch operations are batch computation, criterion computation, back-propagation, and weight optimization.
- Complexity: The standard-loss neural-network algorithm has complexity O(e l w), whereas L(γ_F) changes criterion computation to O(b^2 d).The other three listed per-batch functions remain O(w b), and the criterion function is smaller than those functions in the comparison described.
- Experimental setup: SVM, Adaboost, and neural-network variants are evaluated, with standard Binary Cross Entropy with Logits used as the baseline for modified losses.SVM uses linear, polynomial, and RBF kernels, while Adaboost uses decision trees with varied depths, estimator counts, and learning rates.
7 Conclusions
The paper introduces novel loss functions and evaluates them with SVMs and Neural Networks, finding generally comparable or improved performance, though experiments remain mostly small-scale. It identifies larger-data evaluation and improved similarity matrices, optimization, and kernel strategies as future directions.
- The study introduces several novel loss functions for Machine Learning problems.
- Particle Swarm Optimization supports the theoretical SVM treatment, and the same procedure may be applicable to SVR.
- Neural Network experiments found generalized losses rarely worse and generally better than the standard case, although testing focused mostly on small problems.
- SVM achieved better generalization than Neural Networks on 3 of 7 data-sets in less time, while Adaboost ranked first on Breast with negligible training time.
- Losses using the described matrices were comparable or better than the Neural Network baseline, except that standard BCE performed best on Iono.
- Future work should investigate more efficient F matrices with bigger data-sets, Muon optimization, Anisotropic Kernels, data-set-specific schemes, and graph kernels.The proposed directions target faster minima, improved similarity matrices, better generalization, and broader graph-classification applications.