Source-linked AI summary
Optimization over covariance matrices with a parameterized metric
Yibang Li, Bamdev Mishra, Pratik Jawanpuria, Cyrus Mostajeran
TL;DR
Choosing a Riemannian metric is an optimization and preconditioning question because it determines descent directions, while existing metrics can behave differently across objectives. The paper introduces and analyzes a two-parameter family containing standard metrics, finding a conditioning floor governed by r = p+q and benefits from tuning exponent and shape in supported covariance problems.
Problem
Metric choice affects gradient-based optimization on covariance matrices, but the relative effectiveness of common geometries depends on the objective.
Method
The paper introduces a two-parameter metric family defined by a Sylvester-type equation, analyzes Riemannian-Hessian conditioning, and develops rules for tuning its exponent and shape.
Results
The Hessian conditioning floor depends on r = p+q, while experiments confirm predicted conditioning and improvements from tuning r and shape.
Takeaways & Limitations
Metric selection can be treated as objective-specific preconditioning, with diagonal members attaining the conditioning floor in the pure-power regime.
Takeaways & Limitations
The fitted exponent cannot reach the Bures–Wasserstein distance objective because its optimum lies off the diagonal p = q.
Abstract
from arXiv · showhide
The choice of Riemannian metric can strongly influence the convergence of gradient-based optimization over covariance matrices. Euclidean, Bures-Wasserstein and affine-invariant metrics are common choices, but their relative effectiveness depends on the objective. We introduce a two-parameter family defined by $X^{p}LX^{q}+X^{q}LX^{p}=U$, solved for $L$ at each tangent vector $U$, that contains all three as exact members, at $(0,0)$, $(1,0)$ and $(1,1)$, and extends past them. We treat the choice of member as a particular way of preconditioning for a given problem. To this end, we analyze the conditioning of the Riemannian Hessian at the solution. We show that it obeys a lower bound that depends on $(p,q)$ only through the exponent $r=p+q$. When the Euclidean Hessian is a pure power that mixes no eigendirections, the member $p=q=r/2$ attains that bound, and a closed-form criterion identifies the other members that do. We discuss ways to tune $r$ for a given problem. Experiments on real covariance data confirm the predicted conditioning and the benefit of tuning $r$. A task covariance example shows a further gain from tuning the shape.
1. INTRODUCTION
Metric choice determines the descent direction on the SPD manifold and therefore serves as an algorithmic preconditioning decision. The paper introduces a two-parameter family containing Euclidean, Bures–Wasserstein, and affine-invariant geometries, then studies how tuning it affects Hessian conditioning.
- The Riemannian metric converts the Euclidean gradient into a descent direction, so choosing it is an intrinsic preconditioning decision.
- Common metrics are motivated geometrically, but their optimization effectiveness can depend on the objective class.
- The proposed two-parameter family contains Euclidean, Bures–Wasserstein, and affine-invariant geometry as exact members.
- The paper analyzes how the Riemannian Hessian condition number depends on the family parameters and develops approximations for choosing them.
- Experiments evaluate whether tuning the metric improves optimization over covariance-matrix objectives.
2. A PARAMETERIZED METRIC
The paper defines a two-parameter Riemannian metric by solving a Sylvester-type equation for each tangent vector. Its exponent controls homogeneity, while the parameter difference controls shape, yielding a family that includes and extends standard metrics.
- For tangent vector U, L is obtained by solving X^p L X^q + X^q L X^p = U, and the resulting operator defines the metric on tangent vectors.
- Positive eigenvalue coefficients make the operator self-adjoint, positive definite, and invertible for every real p and q.
- The weight is symmetric in (p,q) and positively homogeneous of degree r = p+q, which the paper calls the member’s exponent.
- At (0,0), (1,0), and (1,1), the family gives Euclidean, Bures–Wasserstein, and affine-invariant metrics, respectively, while arbitrary real parameters extend beyond them.
- At fixed degree r, the exponent determines homogeneity while p−q determines shape, separating conditioning effects from within-degree variation.
- The metric gradient is computed entrywise in the iterate’s eigenbasis, and one implementation covers the family without extra cost beyond the eigendecomposition already required for the objectives.
3. THE ROLE OF r = p + q IN HESSIAN CONDITIONING
The Riemannian Hessian condition number has a degree-dependent lower bound, minimized when the metric exponent r matches the objective’s pure-power exponent. Under stronger pure-power assumptions, the diagonal member is optimal at fixed r, while a closed-form condition identifies when off-diagonal shapes also attain the bound.
- Hessian conditioning: The Riemannian Hessian at a critical point is obtained by preconditioning the Euclidean Hessian with the inverse metric, without derivative-of-metric terms.In the eigenbasis of the solution, the metric rescales Hessian rows and columns by the square root of the weight; diagonal entries become weighted Euclidean-Hessian entries.
- Conditioning floor: For any homogeneous metric of degree r, the Hessian condition number is bounded below by κ^|r−r⋆| when the Euclidean Hessian has diagonal pure power degree γ−2 and r⋆=2−γ.The bound is independent of off-diagonal Euclidean-Hessian entries and couplings between distinct frames.
- Attaining the floor: For a Schur-multiplier Euclidean Hessian with full pure power, the bound is attained when the weighted spectrum is monotone in both eigenvalue arguments.The diagonal member p=q=r/2 always satisfies this monotonicity condition, so it attains the floor at every exponent.
- Conditioning floor: The lower bound is minimized at r=r⋆, where it equals one, although minimizing the bound and the actual condition number coincide only when the bound is attained.The degree r is therefore the central tuning variable for matching the metric to the objective’s local Hessian scaling.
- Off-diagonal shape: The monotonicity criterion requires the shifted exponents to have the same sign, and the diagonal member satisfies it for every r; monotonicity is sufficient but not necessary.Thus off-diagonal members can sometimes attain the floor beyond the monotonicity region, but their admissible range depends on the degree error and κ.
- Exact conditioning: In the pure-power Schur regime, degree error and shape separate: at fixed r the diagonal member minimizes conditioning, while shape is flat within a κ-dependent band and grows outside it.Changing the degree by t costs κ^t, whereas changing the shape by t costs cosh(t log κ/2), so degree mismatch is more expensive.
4. SELECTION RULES FOR r, p AND q
The selection rules reduce metric tuning to estimating the exponent r, with p = q = r/2 optimal in the pure-power Schur regime. Least-squares and range-based criteria estimate r from frame data, while objective-specific Hessian structure determines the target exponent and whether named metrics suffice.
- Selecting p and q: In the pure-power Schur regime, p = q = r/2 attains the conditioning floor for every r and is the unique admissible member at r = r⋆.The same diagonal choice is also the cheapest member, so tuning r alone suffices in this regime.
- Selecting r: The target exponent is r⋆ = 2 − γ for Hessians acting by congruence with a power of X, but mixed-degree Hessians require estimating r.The Euclidean least-squares Hessian has r⋆ = 0, the log-determinant Hessian has r⋆ = 2, and the inverse-fit Hessian has r⋆ = 4.
- Estimating r: The least-squares criterion fits log λij = r uij + log bij, yielding a closed-form estimate from the slope of log bij against uij.The estimate is computable in one pass over frame data, provided all bij > 0 and the uij are not all equal.
- Estimating r: The range criterion minimizes the range of log λk through a one-dimensional convex problem that depends only on the extreme frames and can be solved as a linear program.Unlike least squares, it uses an ℓ∞ criterion rather than an ℓ2 fit.
- Computational cost: Both criteria require one eigendecomposition and m Hessian-vector products, making tuning a one-time cost relative to many optimization iterations.The products pair frame directions with Hessian actions without explicitly forming the Hessian.
5. EXPERIMENTS
Experiments show that tuning the metric exponent improves Hessian conditioning and optimization when objectives demand degrees beyond standard metrics, while shape tuning provides additional gains on task covariance data.
- 5. EXPERIMENTS: All experiments used the same algorithmic settings except the metric weight, with conditioning measured at a common reference solution.Runs used Algorithm 1, the same start and stopping tolerance, and an Armijo line search.
- 5.1. The exponent read off the Hessian: For pure power-congruence objectives, conditioning follows κϕ = κ^|r−r⋆|, and two Hessian-vector products recover each target exponent exactly when the residual vanishes.The covariance experiments use n = 36, κ(X⋆) = 9836, and confirm the predicted degree dependence.
- 5.2. An exponent past every named metric: Both exponent criteria reached machine precision on the barrier objectives, whereas no named metric did; their returned exponents were near 3 or 4.The range criterion returned 3.45 and 4.15 on Wine and 3.10 and 4.05 on Breast under the two barriers.
- 5.2. An exponent past every named metric: Reading more frames was unnecessary for the range criterion and worsened the least-squares fit; the n diagonal frames sufficed in both regimes.Each subset retained the pair S0, and no sampled subset violated the stated fitting conditions.
- 5.2. An exponent past every named metric: The log-determinant benefited little from tuning because affine-invariant geometry already has its exponent, while barrier gains were widest at weaker regularization.Across regularization weights, the two barriers improved in every setting, but log-determinant sometimes lost.
- 5.3. Tuning the shape: Median κϕ fell from 7.70 to 3.53 with least-squares shape tuning and from 7.64 to 3.28 with range tuning across seven SARCOS cases.These correspond to median reductions of 55.80% and 54.54% versus the matching exponent-only baselines.
6. CONCLUSION
The paper’s metric family unifies and extends three standard covariance geometries while reducing conditioning selection to the exponent r in the pure-power regime. Experiments show that tuning r helps, with additional gains from tuning shape in the SARCOS example.
- The two-parameter family contains Euclidean, Bures–Wasserstein, and affine-invariant metrics as exact members and extends beyond them.
- The Riemannian Hessian conditioning floor depends on p and q only through r = p + q, and the diagonal member attains it in the pure-power Schur regime.
- Tuning r improves optimization on the studied covariance problems, while the SARCOS task covariance example shows further benefit from tuning the metric shape.