PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Optimal Weighted $\ell_2$ Regularization in Overparameterized Linear Regression

Denny Wu, Ji Xu

arXiv:2006.05800v4stat.MLcs.LGmath.ST

TL;DR

The paper asks how ridge regression behaves in overparameterized linear models with anisotropic features and coefficient priors, especially when standard isotropic assumptions fail. It derives exact proportional-asymptotic risk for weighted ridge and PCR, finding conditions for negative optimal regularization and weighted objectives that outperform standard ridge and PCR. The results characterize optimal weighting in ridgeless and tuned regimes while relying on covariance information that may be difficult to estimate.

  • Problem

    Prior high-dimensional analyses largely assume isotropic features or signals and do not fully explain negative optimal ridge parameters or weighted regularization in overparameterized models.

  • Method

    The paper derives proportional-asymptotic prediction risk for generalized ridge and PCR under anisotropic covariances, joint spectral assumptions, and weighted objectives.

  • Results

    The analysis identifies conditions for negative λ_opt, characterizes PCR double descent, and determines optimal weighting matrices for ridgeless and optimally regularized estimators.

  • Takeaways & Limitations

    Weighted ℓ2 regularization can improve over standard ridge regression and PCR, while overparameterization provides an implicit ℓ2 regularization effect.

  • Takeaways & Limitations

    The main results generally require knowledge of Σ_x and the signal covariance, although Σ_x can be estimated from unlabeled data while the signal covariance is typically difficult to estimate directly.

Abstract

from arXiv · show

We consider the linear model $\mathbf{y} = \mathbf{X} \mathbfβ_\star + \mathbfε$ with $\mathbf{X}\in \mathbb{R}^{n\times p}$ in the overparameterized regime $p>n$. We estimate $\mathbfβ_\star$ via generalized (weighted) ridge regression: $\hat{\mathbfβ}_λ= \left(\mathbf{X}^T\mathbf{X} + λ\mathbfΣ_w\right)^\dagger \mathbf{X}^T\mathbf{y}$, where $\mathbfΣ_w$ is the weighting matrix. Under a random design setting with general data covariance $\mathbfΣ_x$ and anisotropic prior on the true coefficients $\mathbb{E}\mathbfβ_\star\mathbfβ_\star^T = \mathbfΣ_β$, we provide an exact characterization of the prediction risk $\mathbb{E}(y-\mathbf{x}^T\hat{\mathbfβ}_λ)^2$ in the proportional asymptotic limit $p/n\rightarrow γ\in (1,\infty)$. Our general setup leads to a number of interesting findings. We outline precise conditions that decide the sign of the optimal setting $λ_{\rm opt}$ for the ridge parameter $λ$ and confirm the implicit $\ell_2$ regularization effect of overparameterization, which theoretically justifies the surprising empirical observation that $λ_{\rm opt}$ can be negative in the overparameterized regime. We also characterize the double descent phenomenon for principal component regression (PCR) when both $\mathbf{X}$ and $\mathbfβ_\star$ are anisotropic. Finally, we determine the optimal weighting matrix $\mathbfΣ_w$ for both the ridgeless ($λ\to 0$) and optimally regularized ($λ= λ_{\rm opt}$) case, and demonstrate the advantage of the weighted objective over standard ridge regression and PCR.

1 Introduction

This work studies generalized ridge regression for overparameterized linear models with anisotropic data covariance and coefficient priors. It derives risk and optimality results that explain negative ridge parameters and establish advantages of weighted regularization.

  • Motivation: Overparameterization is connected to implicit ℓ2 regularization and to empirical observations that negative ridge parameters can improve generalization.The paper motivates this connection through double descent and high-dimensional least-squares behavior.
  • Setup: The paper analyzes generalized ridge regression in the proportional overparameterized limit p/n → γ ∈ (1, ∞) under anisotropic data and coefficient covariances.The estimator uses a positive-definite weighting matrix Σ_w, with Σ_w = I recovering standard ridge regression.
  • Motivation: Anisotropic Σ_x and Σ_β allow the analysis to identify when the optimal ridge parameter is negative, unlike prior isotropic analyses.The paper links negative λ_opt to large signal-to-noise ratio and alignment between the large directions of Σ_x and Σ_β.
  • Contributions: The work derives exact asymptotic prediction risk, including bias–variance decompositions and principal component regression risk under general covariance structures.It also characterizes double descent for PCR in settings more general than prior work.
  • Contributions: The optimal weighting matrix is characterized for both ridgeless and optimally regularized estimators, showing benefits over standard ridge regression and PCR.For optimally regularized ridge, many settings yield Σ_w = Σ_β^-1; the ridgeless optimum can interpolate between Σ_x and Σ_β^-1.

2 Related Works

The related-work discussion positions the paper at the intersection of high-dimensional ridge asymptotics, weighted regularization, overparameterization, and concurrent analyses. It emphasizes that prior results generally impose more restrictive covariance or parameter-regime assumptions.

  • Asymptotics of Ridge Regression: Earlier ridge analyses primarily treated isotropic data or isotropic coefficient priors, whereas this work studies their joint anisotropic structure.The paper builds on general random-matrix results to extend proportional-asymptotic risk calculations.
  • Weighted Regularization: Weighted regularization has a long history, but its effectiveness in high-dimensional overparameterized regression remained largely unknown.Related work includes weighted estimators in classical regression and semisupervised settings.
  • Benefit of Overparameterization: The paper relates its setting to double descent and implicit regularization results developed for least squares, random features, and classification.It notes that translating these explanations to neural networks can be nuanced.
  • Concurrent Works: Concurrent work analyzed asymptotic risk under narrower covariance relations, while other work studied negative regularization in a specific spike model.The paper distinguishes its alignment-based characterization from those more restrictive settings.

3 Setup and Assumptions

The setup defines weighted ridge and PCR under random designs with joint asymptotic structure for transformed feature and signal spectra. It also identifies methodological limitations of existing AMP and CGMT approaches for negative regularization and covariance alignment.

  • Weighted Ridge Setup: The generalized ridge estimator transforms the design by Σ_w^-1/2, yielding effective covariance Σ_x/w and signal covariance Σ_wβ for risk analysis.The resulting risk separates variance, which is signal-independent, from bias, which is noise-independent.
  • Principal Component Regression: PCR retains the leading θp eigenvectors of Σ_x before fitting least squares in the reduced feature space.The estimator is parameterized by θ ∈ [0, 1].
  • Limitations of Existing Tools: Existing AMP and CGMT frameworks may have difficulty establishing convergence for λ < 0 and handling the joint relation between Σ_x and Σ_β.That joint relation is central to the paper’s negative-ridge analysis.
  • Assumptions: The analysis emphasizes that alignment between Σ_x and β⋆ determines the sign of the optimal regularization parameter beyond spike covariance models.This extends the negative-ridge discussion beyond the specific spike setup considered in related work.
  • Assumptions: The asymptotic assumption requires the paired eigenvalues of effective feature and signal covariances to converge jointly with bounded spectra.This joint relation supports analysis beyond cases where either covariance is isotropic.

4 Risk Characterization

The paper derives asymptotic prediction-risk formulas for generalized ridge regression and connects them to ridgeless regression and PCR. The analysis explains how anisotropic feature–signal relationships shape risk, optimal regularization, and double descent.

  • Risk characterization: Theorem 1 characterizes the asymptotic prediction risk of generalized ridge regression, including its bias and variance components.The variance term is linked to the Stieltjes transform, while the paper’s main technical contribution is a less restrictive characterization of the bias term.
  • Risk characterization: The derived asymptotic risk matches finite-sample experiments for standard ridge regression, including aligned, misaligned, and randomly related anisotropic designs.The experiments use γ = 2 and (n, p) = (300, 600), with discrete and continuous component distributions.
  • Risk characterization: Alignment means feature and coefficient components share the same ordering, whereas misalignment reverses that ordering and changes whether learning is easy or difficult.Aligned large-magnitude features tend to correspond to large-magnitude signal components; misalignment makes large features contribute less to labels.
  • Risk characterization: The risk formula also covers the ridgeless limit, which produces the minimum weighted norm solution and reduces to the minimum ℓ2-norm solution when Σw = I.This connects generalized ridge regression with the interpolating estimator used in prior analyses.
  • Risk characterization: PCR exhibits double descent under anisotropic features and signal: risk spikes as θγ approaches 1 from below and decreases with further overparameterization when θγ > 1.Under the stated monotonicity condition, PCR risk decreases with θ in the θγ > 1 regime.
  • Risk characterization: The general anisotropic setting can produce multiple descent for ridgeless regression as γ increases, while PCR can decrease monotonically in the overparameterized regime.These behaviors distinguish changing model size in ridgeless regression from increasing the retained principal-component fraction in PCR.

5 Analysis of Optimal λopt

The analysis decomposes optimal ridge selection into variance and bias effects, yielding conditions for positive or negative λopt and showing how overparameterization changes this tradeoff. It also identifies special-case optima, confirms implicit regularization, and reports improved risk from optimal ridge over ridgeless regression.

  • Special cases: For isotropic features, isotropic signals, or randomly ordered feature-signal relations, λopt has the closed forms c/ξ, ˜σ2/c, and ˜σ2/E[g], respectively.These cases recover earlier results and provide tractable benchmarks for the general analysis.
  • Sign of λopt: The variance derivative is negative, while the bias derivative changes sign with alignment, producing a bias-variance tradeoff in λopt.As noise increases and variance dominates, λopt eventually becomes positive.
  • Sign of λopt: λopt is negative when feature and signal directions are aligned, but positive when they are misaligned, under the stated conditional-expectation conditions.Alignment makes shrinkage increase bias, whereas misalignment makes shrinkage beneficial even without label noise.
  • Implicit regularization: In the overparameterized regime, λopt can remain negative at lower SNR as γ increases, whereas for γ < 1 it is always non-negative.This supports the interpretation that overparameterization supplies an implicit ℓ2 regularization effect.
  • Empirical validation: In the Figure 4 experiment, optimal ridge improves prediction risk over ridgeless regression, with larger gains as γ or the condition numbers increase.The noiseless experiment varies alignment between feature and signal eigenvalues and compares optimal λ with the ridgeless estimator.
  • Limitations: A general characterization for anisotropic β⋆ remains open because λopt may lack a convenient closed form.The paper leaves this analysis for future work.

6 Optimal Weighting Matrix

The paper characterizes how weighting matrices trade off bias and variance in ridgeless and optimally tuned generalized ridge regression. The optimal choices depend on whether bias, variance, or jointly optimized risk is targeted, and weighted objectives can outperform standard ridge and PCR.

  • 6.1 Minimum ∥β̂∥Σw solution: Σw = Σx minimizes variance, while Σw = Σ̄β^-1 minimizes bias; these objectives generally differ, creating a bias-variance tradeoff.The tradeoff disappears in the special case ΣxΣ̄β = I.
  • 6.1 Minimum ∥β̂∥Σw solution: In the noiseless setting, the minimum ∥β̂∥Σw solution outperforms PCR for all θ ∈ [0, 1) under the stated monotonicity assumptions.This holds for Σw = Σ̄β^-1 and for Σw = (fv(Σx))^-1.
  • 6.2 Optimal weighted ridge estimator: For optimally tuned ridge, Σw = Σ̄β^-1 minimizes minλ R(r, λ) over the admissible class, and Σw = (fv(Σx))^-1 is optimal among choices depending only on Σx.The optimal weighting for tuned ridge does not depend on the noise level, only on signal strength across eigendirections.
  • 6.2 Optimal weighted ridge estimator: Figure 5 evaluates optimally tuned prediction risk across γ for several weighting matrices, including Σx, Σβ, I, and powers of Σβ around Σ̄β^-1.The figure uses aligned discrete constructions for dx and dβ, with noiseless solid curves and fixed-SNR dashed curves.
  • 6.2 Optimal weighted ridge estimator: Heuristic choices based only on Σx can outperform standard isotropic regularization, although they may not match the signal-informed optimum.Polynomial or power transformations of Σx are proposed when fv is unknown.

7 Conclusion

The paper concludes with an exact asymptotic theory for generalized ridge regression in overparameterized linear models. This theory explains negative ridge and implicit regularization, identifies optimal weighted penalties, and motivates data-driven weighting as a future direction.

  • 7 Conclusion: The paper provides a precise asymptotic characterization of generalized ridge prediction risk in the overparameterized regime.The framework supports the analysis of negative ridge, implicit regularization, and optimal weighted penalties.
  • 7 Conclusion: The analysis theoretically justifies negative or vanishing regularization as beneficial in settings including certain two-layer neural networks.Figure 7 illustrates this phenomenon for selected two-layer neural nets.
  • 7 Conclusion: The paper identifies constructing a weighting matrix solely from training data that outperforms isotropic shrinkage as a future research direction.The conclusion notes that current derivations rely on assumptions that may be relaxed, including bounded eigenvalues.

A.2 Proof of Corollary 3

This proof establishes the asymptotic prediction-risk result by analyzing contributions from retained and omitted eigendirections. Monotonicity and conditional-expectation arguments determine the signs of the relevant terms.

  • A.2 Proof of Corollary 3: The asymptotic prediction risk follows after replacing conditional expectations with ordinary expectations.The proof invokes the limiting characterization of the relevant quantities before this substitution.
  • A.2 Proof of Corollary 3: The proof decomposes the risk change into three parts and shows the first two are nonnegative under the stated monotonicity conditions.Part (i) is positive when E[g|h] decreases with h, while part (ii) is nonnegative.
  • A.2 Proof of Corollary 3: The third part is nonnegative because h mθ/(h mθ + 1) is increasing in h.This completes the sign analysis needed for the corollary.
  • A.2 Proof of Corollary 3: For the simplified single-point-mass case, the fixed-point relation becomes 1 = λm(−λ) + γζ/(1 + ζ).Here ζ = h · m(−λ), and the simplification is used to obtain the desired result.

B.2 Proof of Theorem 4

This proof determines the sign of the derivative governing the optimal regularization by separating universally negative terms from a monotonicity-dependent term. The sign changes according to how conditional signal strength varies with feature strength.

  • B.2 Proof of Theorem 4: Part 3 is always negative for λ > −c0 because 1 − γE[ζ^2/(1 + ζ)^2] is positive.The positivity follows from the fixed-point relations and m(−λ), m′(−λ) > 0.
  • B.2 Proof of Theorem 4: When E[g|h] increases with h, the remaining term has the sign associated with nonnegative regularization; when it decreases, the sign reverses.At λ = 0, equality occurs only when E[g|h] is almost surely constant.
  • B.2 Proof of Theorem 4: The proof concludes Theorem 4 after combining the sign analysis with the derived fixed-point identities.The final argument uses the monotonicity-dependent characterization of Part 4.
  • B.2 Proof of Theorem 4: The proof shows that λopt < 0 is sufficient when the relevant inequality involving ζ and conditional signal strength is positive.Under the distributional assumptions, this condition yields an explicit upper bound on the noise variance.

B.4 Proof of Proposition 6

The proof extends the risk characterization to γ < 1 and establishes monotonicity properties used to analyze optimal regularization. In the noiseless case, the optimal parameter is zero below the interpolation threshold, while risk increases with γ.

  • Extension to γ < 1: The risk expression from the overparameterized proof extends to γ < 1 for nonzero λ, with λ = 0 handled by continuity.The argument requires replacing m(−λ) with the Stieltjes transform s(−λ) at λ = 0.
  • Extension to γ < 1: For γ < 1, m(−λ) need not be positive, so the proof must rederive the key inequality rather than reuse the γ > 1 argument.The positivity of s(z) remains available, but positivity of m(−λ) does not.
  • Sign analysis: The contradiction argument establishes 1 + h·m(−λ) < 0 almost surely when λ < 0 and m(−λ) < 0.This yields the required sign condition for the derivative analysis.
  • Special cases: For isotropic features, the optimal ridge parameter is λ_opt = c/ξ, while isotropic signals yield λ_opt = σ̃^2/c.These special cases provide closed-form benchmarks for the general optimality analysis.
  • Optimal risk: In the noiseless underparameterized regime, λ_opt = 0 and R(λ_opt) = 0; consequently, prediction risk increases with γ.This conclusion applies for γ < 1 and is combined with the optimal-risk expression analyzed in the proof.

C.1 Proof of Theorem 8

The proof identifies weighting choices that minimize variance and establishes that the inverse conditional signal covariance is optimal for weighted ridge regression. It also shows that the resulting minimum-norm solution can outperform PCR in the relevant regime.

  • Variance-optimal weighting: The minimum variance is achieved when r is a single point mass, corresponding to Σ_w = Σ_x^-1.The proof uses the equality condition in the variance bound to identify this weighting.
  • Conditional reduction: The proof reduces general admissible relations by replacing v with E[v|s], leaving the risk unchanged within the specified class.This conditional-expectation reduction supports the stated optimal weighting form.
  • Comparison with PCR: The minimum-||β̂||_Σw solution outperforms PCR when θγ < 1.The comparison follows by relating PCR risk to the full-model risk of the minimum weighted-norm estimator.
  • Joint optimization: The optimal regularization parameter exists for the interpolated risk function, enabling joint optimization over the weighting and ridge parameters.The proof establishes existence through the limiting behavior of the risk equation as λ varies.
  • Optimal weighted ridge: For optimally regularized weighted ridge regression, the optimal weighting uses the inverse conditional relation between signal and feature covariance.The theorem’s optimization over admissible weights attains its minimum at the conditional weighting choice.

D.1 Experiment Setup

The experiments vary discrete and continuous feature and signal spectra under aligned, misaligned, and mixed relationships. They compare theoretical and finite-sample risks and examine ridgeless, PCR, and optimally weighted ridge behavior.

  • Spectral designs: The experiment varies spectral distributions using discrete-to-discrete, discrete-to-continuous, continuous-to-continuous, and continuous-to-discrete constructions.These designs independently control the feature spectrum d_x and signal spectrum d_β.
  • Aligned and misaligned designs: Aligned and misaligned setups use three-point feature spectra with signal covariance proportional to Σ_x or Σ_x^-1, respectively.The aligned case sets Σ_β = 18/5 · Σ_x, while the misaligned case sets Σ_β = 4/9 · Σ_x^-1.
  • Risk validation: Finite-sample and asymptotic risks match for standard ridge regression across discrete, continuous, aligned, misaligned, and random spectral combinations.Figure 8 uses γ = 2, (n, p) = (300, 600), and SNR ξ = 5.
  • PCR and ridgeless comparison: Ridgeless regression exhibits multiple risk peaks under anisotropic covariances, especially when feature and signal spectra are misaligned.PCR is largely decreasing with θ in the same comparison, although it is not always monotone when θγ > 1.
  • Optimal λ versus overparameterization: Figure 10 varies γ from 1.1 to 4 and compares optimal λ in noiseless and fixed-SNR settings for uniform and two-point feature spectra.The zero level is marked by a solid green line.
Loading 2006.05800v4…