Source-linked AI summary
Statistical Mechanics of Support Vector Networks
Rainer Dietrich, Manfred Opper, Haim Sompolinsky
TL;DR
The paper analyzes how SVMs generalize beyond linear rules, especially when nonlinear coefficients are difficult to estimate from limited examples. Using Statistical Mechanics, it finds weak overfitting on simple rules and much faster generalization when inputs have a gap near the decision boundary.
Problem
The paper asks how SVM generalization behaves for nonlinear classification rules and whether complex feature spaces cause overfitting when data are limited.
Source: p. 1Method
The authors analyze SVMs with Statistical Mechanics, using kernel eigenfunctions, replica averaging, and large-N scaling across example regimes.
Source: p. 2Results
Nonlinear rules produce generalization plateaus at insufficient sample scales, while simple rules cause only weak overfitting and input gaps strongly accelerate generalization.
Source: pp. 3–4Takeaways & Limitations
SVM generalization depends on which kernel components are learnable at a given sample scale and is enhanced when the input distribution excludes points near the decision boundary.
Source: pp. 3–4The analysis assumes permutation-symmetric kernels and, for the input average, uniformly distributed binary inputs with k(0)=0.
Source: p. 2
Abstract
from arXiv · showhide
Using methods of Statistical Physics, we investigate the generalization performance of support vector machines (SVMs), which have been recently introduced as a general alternative to neural networks. For nonlinear classification rules, the generalization error saturates on a plateau, when the number of examples is too small to properly estimate the coefficients of the nonlinear part. When trained on simple rules, we find that SVMs overfit only weakly. The performance of SVMs is strongly enhanced, when the distribution of the inputs has a gap in feature space.