Source-linked AI summary
Ridge Regularization: an Essential Concept in Data Science
Trevor Hastie
TL;DR
Ill-conditioned designs and wide-data applications make stable fitting and careful λ selection important. The paper explains ridge through SVD-based shrinkage and its connections to other methods. In a reported simulation, tuned ridge predictions outperform OLS, while the discussion also identifies a bias boundary when p > n.
Problem
Method
Ridge uses λ to shrink coordinates in the principal-component basis, and the SVD provides solutions across λ values.
Source: §2 · p. 5Results
At log(λ) ≈ −4, ridge predictions attain minimum mean-squared error from the true linear function and outperform OLS in the reported simulation.
Source: §3 · p. 7Takeaways & Limitations
Ridge and its extensions are pervasive in applied statistics, and ridge includes correlated variables that variable-selection methods may omit.
Source: §10 · p. 16When p > n, any component of β orthogonal to X's row space contributes to ridge-estimation bias.
Source: §3 · p. 6
Abstract
from arXiv · showhide
Ridge or more formally $\ell_2$ regularization shows up in many areas of statistics and machine learning. It is one of those essential devices that any good data scientist needs to master for their craft. In this brief ridge fest I have collected together some of the magic and beauty of ridge that my colleagues and I have encountered over the past 40 years in applied statistics.
1 Ridge and linear regression
Ridge regularization stabilizes regression when the design matrix is ill-conditioned by adding λ to the diagonal of X⊤X. Across regression settings, it trades bias for lower variance, while alternatives such as lasso can also select variables.
1 Ridge and linear regression
- Adding λI to X⊤X raises its eigenvalues and addresses the instability caused by zero or near-zero eigenvalues.When X is ill-conditioned, X⊤X inherits a poor condition number and may be singular or nearly singular.Source: §1 · p. 1
- Ridge shrinks coefficients toward zero, introducing bias while reducing variance; choosing λ balances these effects.A small λ may suffice when the goal is only to resolve numerical issues.Source: §1 · p. 2
- In logistic regression, a small ridge penalty can resolve an undefined maximum-likelihood estimate when classes are separated in feature space.The same remedy applies to other models linear in their parameters, including Cox and multiclass logistic models.Source: §1 · p. 2
- In wide-data settings such as genomics and document classification, regularization is essential and λ requires careful tuning.Source: §1 · p. 2
- Unlike ridge, lasso uses an ℓ1 penalty to shrink coefficients and set some to zero, enabling variable selection.The elastic net mixes ridge and lasso penalties; group lasso selects predefined groups of variables.Source: §1 · pp. 2–3
- Under a Gaussian prior on β, ridge is the posterior mean, and reducing the prior variance increases shrinkage toward zero.The corresponding penalty parameter is λ = σ²_ϵ/σ²_β.Source: §1 · p. 3
2 Ridge computations and the SVD
Ridge solutions across λ values can be computed efficiently from the singular-value decomposition of X. The SVD also makes clear how shrinkage acts across principal components and how ridge behaves when p exceeds n.
2 Ridge computations and the SVD
- Candidate λ values can be assessed using a validation set, cross-validation including LOO, or Cp and unbiased risk estimates.Source: §2 · p. 4
- Once X is decomposed by SVD, the ridge solution can be computed for all λ values along the regularization path.This provides an efficient way to obtain the multiple solutions needed for tuning λ.Source: §2 · p. 4
- Ridge assigns identical coefficients to identical variables, while correlated variables have coefficients shrunk toward each other.The latter is the mechanism exploited by the elastic net.Source: §2 · p. 4
- With λ > 0, fitted-value coordinates on successive principal components are shrunk by increasing amounts.When X is centered, the SVD basis vectors are its principal components.Source: §2 · pp. 4–5
- When p > n, least squares has infinitely many zero-residual solutions, while the minimum-ℓ2-norm solution is unique.The minimum-norm solution is obtained by setting the free component θ2 to zero.Source: §2 · p. 5
3 Ridge and the bias-variance trade-off
Ridge trades coefficient bias for reduced variance, with the amount and pattern of shrinkage depending on λ and the principal components. In the reported simulation, appropriately tuned ridge predictions outperform OLS, while James–Stein shrinkage performs poorly on that example.
3 Ridge and the bias-variance trade-off
- Ridge differentially shrinks true-coefficient coordinates toward zero, with more shrinkage for smaller principal components; bias increases in every component as λ increases.Source: §3 · p. 5
- When p > n, ridge estimates lie in X's at-most n-dimensional row space, so any true-coefficient component orthogonal to that space contributes to bias.Source: §3 · p. 6
- As λ increases, the covariance decreases in the positive-semidefinite sense, contributing to the bias–variance trade-off in prediction.Source: §3 · p. 6
- At log(λ) ≈ −4, ridge coefficients are closer to truth and predictions attain minimum mean-squared error from the true linear function, outperforming OLS in the simulation.The simulation uses n = 100, p = 54, and SNR = 3.3; LOO cross-validation finds a reasonable λ.Source: Figure 2 · p. 6, §3 · p. 7
- James–Stein estimates have MSE above 15 and perform poorly in this simulation.Source: §3 · p. 7
4 Ridge and data augmentation
Ridge can be implemented through data augmentation: adding points or perturbing observations can approximate ridge fitting. In image classification, natural image transformations create additional labeled examples and are described as a form of ridge regularization.
4 Ridge and data augmentation
- Adding p points around the origin, each λ units along a coordinate axis, yields the ridge coefficient estimate for centered data.Random draws with mean zero and covariance λI, weighted against the original observations, approximate the same result when their empirical covariance is close to λI.Source: §4 · p. 7
- Zero-mean perturbations with the appropriate scalar covariance, assigned the original response, approximate ridge as the number of copies grows.Each perturbed observation receives the same response; the zero-mean cross-terms average away as the number of copies increases.Source: §4 · pp. 7–8, Figure 3 · p. 8
- Natural image transformations create labeled variants that remain recognizable, reducing over-training on original images and serving by analogy as ridge regularization.Examples include small changes to rotation, scale, location, hue, and contrast.Source: §4 · p. 8
5 Ridge and dropout regularization
The section connects ridge-like regularization to random forests and dropout, where predictors or activations share predictive work. For linear regression, dropout's expected score equations yield generalized ridge, which becomes ordinary ridge when variables are standardized.
5 Ridge and dropout regularization
- Random forests consider only m of p candidate variables at each split, making variables share predictive burden; smaller m means a more regularized fit..Source: §5 · p. 9
- Dropout zeros activations with probability ϕ and inflates survivors by 1/(1 −ϕ), so surviving units stand in for omitted ones; ϕ is a tuning parameter.For deep networks, different layers may work better with different values of ϕ.Source: §5 · p. 9
- For linear regression, dropout's expected score equations produce generalized ridge; with standardized variables, the solution is identical to ridge regression.The generalized ridge penalty uses D, the diagonal matrix of predictor-column norms.Source: §5 · p. 9
6 Ridge and the kernel trick
For wide data, ridge can be solved through an n-dimensional formulation using the Gram matrix rather than p-dimensional coefficients. The same formulation extends to quadratically penalized linear models and kernel-based function fitting.
6 Ridge and the kernel trick
- When λ > 0, the ridge solution lies in X's n-dimensional row space, allowing the wide-data problem to be expressed using the n × n Gram matrix K = XX⊤.The resulting solution requires n-dimensional calculations even when p is very large, although forming K costs O(pn^2).Source: §6 · p. 10
- The same dual formulation applies to GLMs and other linear models with quadratically penalized objectives, optimizing over α through the fit vector η = Kα.The linear kernel corresponds to ridged linear models.Source: §6 · p. 10
- Positive-definite kernels extend the formulation to functions represented as sums of kernel evaluations at training inputs; the two-class support-vector machine is one example.The kernel can be viewed as computing inner products in an implicit feature space.Source: §6 · p. 10
- The wide-data GLM formulation reparametrizes the problem in n dimensions, supports cross-validation for λ there, and maps back to β for new wide vectors when needed.Using the QR decomposition reduces the n × p matrix to an n × n matrix for the ridge MLE.Source: §6 · p. 11
7 Ridge and leave-one-out cross validation
Ridge’s linear-operator structure makes leave-one-out (LOO) residuals computable from the full-data fit, with special handling needed for an unpenalized intercept and for the minimum-norm limit when p > n.
7 Ridge and leave-one-out cross validation
- For each omitted observation, the LOO fit is defined on the n − 1 observations excluding its pair (x_i, y_i).Source: §7 · pp. 11–12
- As λ ↓ 0 with p > n, the standard LOO formula appears to break down, but an equally elegant minimum-norm solution exists.Source: §7 · p. 12
- The stated LOO formula does not apply with an unpenalized intercept; the corresponding formula uses a doubly-centered kernel matrix and its pseudoinverse.Source: §7 · p. 12
8 Ridge, minimum norm and double descent
Minimum-norm least squares can show double descent as model dimension increases, while optimally tuned ridge remains comparatively flat. In the spline example, increasing dimension beyond interpolation can also raise small singular values and permit smaller-norm solutions.
8 Ridge, minimum norm and double descent
- With p ≫ n and squared-error loss, gradient descent from β = 0 to zero training error gives the minimum-ℓ2-norm solution [Zhang et al., 2016].Source: §8 · p. 13
- Around dimension 100, minimum-norm test error rises dramatically before declining as dimension grows, while optimally tuned ridge is fairly flat with a minimum near dimension 20.Figure 4 demonstrates double descent; the model interpolates at dimension 100.Source: Figure 4 · p. 13, §8 · pp. 13–14
- The example uses an additive natural-spline model with 9d + 1 parameters, reaching the sample size n = 100 at d = 11.Data use n = 100, a nonlinear nonadditive signal, SNR 3, and a 10K test set.Source: §8 · p. 13
- The test-error increase near interpolation is associated with large solution norms and wiggly fits between training points; beyond interpolation, zero error can be achieved more smoothly with decreasing norm.Thus, dimension alone does not determine model complexity; the minimum-norm objective combines fit and complexity.Source: §8 · p. 14
- As the spline-basis dimension increases, its small singular values rise, potentially allowing more stable solutions with smaller ℓ2 norms.Figure 5 compares natural-spline basis singular values across dimensions 100 to 270.Source: Figure 5 · p. 14, §8 · p. 15
9 Ridge + ridge = lasso
The section connects low-rank matrix approximation to ridge and lasso regularization: truncated SVD solves a rank-constrained problem, while nuclear-norm thresholding and doubly ridged factorization yield related solutions.
9 Ridge + ridge = lasso
- For rank-constrained matrix approximation, retaining the top q singular values of X and reconstructing gives the stated solution despite the problem’s non-convex rank constraint.Source: §9 · p. 15
- Nuclear-norm regularization soft-thresholds singular values, yielding the lasso version of matrix rank selection.The nuclear norm is the sum of a matrix’s singular values.Source: §9 · p. 15
- A biconvex ℓ2-regularized factorization has a solution coinciding with the convex ℓ1-regularized solution, linking doubly ridged and nuclear-norm problems.Source: §9 · p. 15
- For massive sparse matrices and matrices with missing values, the section proposes alternating ridge regressions or updates to obtain low-rank approximations or completion solutions.Source: §9 · p. 15
10 Discussion
The discussion presents ridge and its extensions as broadly useful applied-statistics tools, including in wide-data settings where lasso is also popular.
10 Discussion
- Ridge and elastic net share territory with lasso in wide-data applications, and ridge can be less affected by synonymous features in bag-of-words and n-gram classification.Source: §10 · p. 16
- The discussion says classifiers are hurt less by bias than quantitative regressors.Source: §10 · p. 16
- Ridge has grown from a fix for ill-behaved linear regression into a broader collection of data-modeling tools.Source: §10 · p. 16