PaperlayerView exact revision on arXiv

Source-linked AI summary

Lightweight Adaptive ReduNet via Hyperspherical Manifold Learning

Zhenglin Huang, Qifa Yan, Bin Dai, Xiaohu Tang

arXiv:2608.20668v1cs.LGcs.AI

TL;DR

ReduNet's Euclidean updates under a unit-sphere constraint motivate LA-ReduNet's normalized, truncated, sample-adaptive Riemannian updates for forward layer-wise MCR2 construction. LA-ReduNet reaches stable MCR2 values with substantially fewer layers, comparable classification accuracy, and approximately 1/29 of the parameter storage under the considered settings.

  • Problem

    ReduNet's optimization still relies on a Euclidean gradient update under a unit-sphere constraint.

  • Method

    LA-ReduNet develops a normalized, truncated, sample-adaptive Riemannian update tailored to forward layer-wise MCR2 construction.

  • Results

    Approximately 1/29 parameter storage is required, while LA-ReduNet reaches stable MCR2 values with substantially fewer layers and comparable classification accuracy.

  • Takeaways & Limitations

    LA-ReduNet is a lightweight ReduNet architecture requiring substantially fewer layers for the MCR2 objective to reach a stable value.

  • Takeaways & Limitations

    The LA-ReduNet setting does not satisfy sufficient conditions derived from global analysis, and the conditions can be conservative in practice.

Abstract

from arXiv · show

In recent years, a white-box neural network called ReduNet has been proposed, which employs the maximal coding rate reduction (MCR$^2$) principle to transform raw data into low-dimensional discriminative features via a forward layer-wise construction process. Unlike traditional deep networks that rely on backpropagation, ReduNet explicitly derives the parameters of each layer from the features of its preceding layer, offering a mathematically interpretable paradigm. However, this layer-wise construction often requires a large number of layers for the MCR$^2$ objective to reach a stable value, which increases the parameter storage of the unfolded module. To address this issue, we propose LA-ReduNet, a lightweight adaptive architecture that refines the layer-wise update rule and enables discriminative feature representations to be obtained with substantially fewer unfolded layers. Specifically, LA-ReduNet employs hyperspherical manifold learning and adaptive step sizes, thereby reducing by an order of magnitude the number of layers required for the MCR$^2$ objective to reach a stable value. Simulation results demonstrate that, while maintaining comparable classification accuracy, LA-ReduNet requires significantly fewer layers for the MCR$^2$ objective to reach a stable value. Remarkably, under the considered experimental settings, LA-ReduNet requires only approximately $1/29$ of the parameter storage of the unfolded ReduNet module for the MCR$^2$ objective to reach a stable value.

1. Introduction

ReduNet offers an interpretable, MCR2-derived alternative to black-box networks, but its Euclidean layer-wise updates can require many unfolded layers. LA-ReduNet redesigns these updates on the hypersphere and reaches stable objectives with fewer layers and lower storage.

  • ReduNet and MCR2: ReduNet derives its architecture from the MCR2 principle rather than relying primarily on label fitting.The principle maximizes overall feature coding rate while minimizing within-class coding rate.
  • Motivation: A fixed Euclidean step followed by normalization does not directly control angular displacement on the feature sphere.This can produce small or highly variable angular updates across samples.
  • LA-ReduNet: LA-ReduNet introduces a normalized, truncated, sample-adaptive Riemannian update tailored to forward layer-wise MCR2 construction.The update direction is determined in tangent space, while angular step sizes depend on sample-wise cosine similarity and thresholding.
  • Theory: LA-ReduNet theoretically establishes Lipschitz continuity, an MCR2 increment lower bound, and finite termination under specified conditions.These properties are derived on products of unit spheres and depend on threshold and step-size conditions.
  • Results: Classification accuracy converges within approximately 5–10 layers, while simulations cover CIFAR-10, CIFAR-100, and CINIC-10.The reported experiments indicate substantially fewer layers for both objective and accuracy stabilization.
  • Results: Approximately 35 layers versus approximately 1000 layers are required for MCR2 objective convergence, corresponding to approximately 1/29 of ReduNet’s parameter storage.The comparison uses the considered experimental settings and original ReduNet step-size parameters.

2. Preliminaries

The paper reviews MCR2, ReduNet, and AR-ReduNet as foundations for its approach. It also describes the improved rate-distortion approximation used by AR-ReduNet and adopted by LA-ReduNet.

  • MCR2 Principle: MCR2 optimizes feature representations by maximizing overall coding rate while minimizing within-class coding rate.Coding rates are based on a multivariate Gaussian rate-distortion function.
  • ReduNet Framework: ReduNet updates its feature matrix through gradient ascent and initializes samples by projecting them onto the unit sphere.Its architecture is directly derived from the MCR2 objective.
  • ReduNet Framework: During testing, ReduNet estimates class-membership matrices from the data because test labels are unavailable.Training membership matrices are known, whereas testing uses estimated memberships.
  • AR-ReduNet: AR-ReduNet uses an adaptive multivariate Gaussian rate-distortion approximation that more closely approaches the exact function than ReduNet’s approximation.The approximation introduces adaptive regularization and requires re-evaluating parameters during updates.
  • AR-ReduNet: LA-ReduNet adopts AR-ReduNet’s rate-distortion approximation and uses binary search to obtain its parameters efficiently.The parameters are solved under a specified error threshold.

3. Lightweight Adaptive ReduNet (LA-ReduNet)

LA-ReduNet constructs feature updates directly in the tangent space and evolves features along geodesics on the unit sphere. Its adaptive thresholded updates support finite termination through a theoretically analyzed Riemannian mapping.

  • LA-ReduNet constructs updates directly in the tangent space and evolves features along corresponding geodesics on the unit sphere.
  • The Euclidean update decomposes into radial and tangential components, but only the tangential component provides a feasible direction on the sphere.
  • The geodesic update rotates each feature by angle θ toward the normalized tangential direction, with tan(θ/2) reparameterized by t0.
  • When ||gT||2 ≤ τ, the sample is no longer updated; when ||gT||2 > τ, it remains active and uses the normalized tangential direction.
  • The adaptive parameter t controls angular displacement, while β controls its maximum expansion factor and τ determines whether samples remain active.
  • The Riemannian update mapping is Lipschitz continuous, supporting an objective-variation inequality and finite termination under suitable threshold and step-size conditions.

on M = (Sn−1)m with Lipschitz constant

The proposed Riemannian update scheme establishes finite termination under prescribed threshold and step-size conditions, while the LA-ReduNet architecture reduces storage mainly through substantially smaller network depth. The shared front-end produces normalized lower-dimensional features for the subsequent ReduNet-type modules.

  • Riemannian update and termination: The objective variation is characterized under the Riemannian update, with each update direction formed while current implicit parameters are fixed.The update direction is defined column-wise with adaptive step sizes and geodesic movement on the unit sphere.
  • Riemannian update and termination: The algorithm terminates no later than a finite iteration bound involving Fmax, F(Z(0)), δ, and τ.Fmax is a finite upper bound on the objective over the feasible manifold, and the bound uses the ceiling function.
  • Riemannian update and termination: Theorem 1 establishes finite termination for the Riemannian update scheme under prescribed threshold and step-size conditions.The guarantee concerns sequences generated from an initial sample matrix on the manifold.
  • Conditions and limitations: The sufficient termination conditions may be conservative because Cα and Ls use global worst-case bounds over the entire feasible manifold.The conditions are intended primarily as theoretical guarantees rather than practical parameter-selection rules.
  • LA-ReduNet architecture: LA-ReduNet’s lower parameter storage relative to ReduNet and AR-ReduNet mainly results from substantially smaller network depth.A shared lightweight convolutional front-end reduces feature dimension from n to d, where d ≪ n, and produces normalized inputs for the subsequent modules.

4. Simulation Results

Across the evaluated datasets, LA-ReduNet reaches stable classification accuracy and MCR^2 objective values with substantially fewer unfolded layers than ReduNet and AR-ReduNet, while using less parameter storage.

  • Overall comparison: LA-ReduNet requires substantially fewer unfolded layers than ReduNet and AR-ReduNet, with correspondingly smaller parameter storage.The comparison uses ReduNet, AR-ReduNet, and LA-ReduNet as feature extractors across CIFAR-10, CIFAR-100, and CINIC-10.
  • Parameter storage: Under the considered settings, LA-ReduNet requires approximately 35/1000 ≈ 1/29 of ReduNet's parameter storage for MCR^2 objective convergence.The storage comparison follows the approximate layer counts required for objective convergence.
  • Ablation findings: The normalized truncated Riemannian update and adaptive step size improve convergence, with the thresholding mechanism providing the primary objective-convergence improvement.The adaptive step-size strategy further improves progress in early layers.

5. Conclusions and Discussions

The paper concludes that LA-ReduNet combines a lightweight adaptive architecture with a theoretically analyzed Riemannian update scheme. Experiments report faster stabilization and more separated features under the considered settings.

  • Conclusions: LA-ReduNet is proposed as a lightweight ReduNet architecture with an adaptive step-size mechanism.The architecture is presented as a refinement of the ReduNet update process.
  • Theoretical results: The proposed Riemannian update scheme has a finite-termination guarantee under explicit sufficient threshold and step-size conditions.The paper establishes this result theoretically.
  • Experimental results: Compared with gradient ascent-based ReduNet and AR-ReduNet, LA-ReduNet requires substantially fewer layers for the MCR^2 objective to reach a stable value.This conclusion is reported for the considered experimental settings.
  • Experimental results: LA-ReduNet's classification accuracy generally stabilizes within approximately 5–10 layers, while its MCR^2 objective stabilizes in approximately 35 layers.These values summarize the reported experimental convergence behavior.
  • Feature representations: In conjunction with the convolutional module, LA-ReduNet yields features with superior classification accuracy and better-separated class boundaries.The paper presents this as an experimental result.

Appendix A. Positive Definiteness and Uniform Eigenvalue Bounds

Appendix A establishes that the matrix defined from α, the identity matrix, and the normalized Gram matrix is positive definite, with bounded eigenvalues.

  • Positive definiteness: For unit-norm-column Z and α ∈ [0, 1] satisfying the stated condition, αI + n/tr(ZZT)ZZT is positive definite.The appendix states this as Lemma 1.

eigenvalues satisfy

The appendix bounds the eigenvalues of the matrix αI + n/tr(ZZT)ZZT, supporting its positive definiteness and uniform spectral control.

  • Eigenvalue bounds: The eigenvalues satisfy 1/(n+1) ≤ λ_i ≤ n+1 for i = 1, …, n.The bound is used in the appendix's argument for positive definiteness and uniform eigenvalue control.
  • Proof strategy: Writing λ_i = α + q_i separates the eigenvalues into the α contribution and the normalized Gram-matrix eigenvalues q_i.The appendix uses this decomposition to derive the bounds.
  • Proof strategy: The proof handles n = 1 separately, where the stated condition gives α = 0 and λ_1 = 1.For n ≥ 2, the appendix proceeds through trace and inequality arguments.
  • Positive definiteness: The appendix concludes that all eigenvalues are positive, so αI + n/tr(ZZT)ZZT is positive definite.This conclusion follows from the derived lower bound and nonnegative eigenvalue components.

Appendix B. Lipschitz Continuity of the Implicit Parameter Map-

Appendix B establishes that the implicitly defined parameters α(Z) and αj(Zj) vary Lipschitz-continuously on the product of unit spheres. The proof bounds their derivatives along geodesics and converts geodesic-distance bounds to Euclidean-distance bounds.

  • Parameter mappings: α(Z) and αj(Zj) are unique solutions of the defining equations and are treated as parameter mappings over the product manifold.The manifold consists of n × m matrices with unit-norm columns, M = (S^n−1)^m.
  • Geodesic analysis: The proof establishes differentiability of α(s) along a minimizing geodesic by showing that the defining scalar equation has a strictly positive partial derivative with respect to α(s).Positive definiteness of A(s) and the identity log det A(s) = 0 support the derivative argument.
  • Lipschitz bound: An upper bound on |α′(s)| is integrated along the geodesic to obtain a Lipschitz bound for α(Z).The resulting bound is expressed using the geodesic distance on M.
  • Distance comparison: Because M is a product of unit spheres, its geodesic distance can be bounded by Euclidean distance, giving the desired continuity relation in an ordinary matrix norm.The distance comparison completes the proof after integration and the triangle inequality.
  • Class-wise parameters: The same geodesic argument applies to each class-wise parameter αj(Zj), yielding analogous continuity results.The class-wise submatrices contain the samples belonging to the j-th class.

Appendix C. Lipschitz Continuity of the Inverse Matrices in the MCR2 Objective

Appendix C establishes Lipschitz continuity for the inverse matrices used in the MCR^2 update. The argument combines continuity of the implicit parameters with positive spectral bounds and inverse-matrix identities.

  • Main continuity result: The inverse matrices appearing in the MCR^2 update are shown to vary Lipschitz-continuously with Z.This result is stated as Lemma 3 on M = (S^n−1)^m.
  • Proof mechanism: The difference-of-inverses identity reduces the comparison of two inverse matrices to bounds on the underlying matrices and their inverses.Frobenius-norm inequalities and symmetry are used in the resulting estimates.
  • Spectral control: A positive lower bound on the eigenvalues of Qi ensures that the relevant inverse matrices remain uniformly controlled.The proof invokes the spectral lower bound together with Lemma 1.
  • Class-wise extension: The continuity bounds extend to class-wise submatrices by applying the same argument to each class.The class projections preserve the relevant class columns and satisfy orthogonality relations.

Appendix D. Lipschitz Continuity of the Riemannian Update Map-

Appendix D proves Lipschitz continuity of the Riemannian update map on the product of spheres. It first controls the Euclidean update direction and then transfers the bound through tangent-space projection.

  • Euclidean update: The Euclidean update direction is bounded by decomposing its difference into terms involving implicit parameters, inverse matrices, and feature-dependent factors.The estimates use the earlier Lipschitz lemmas, Frobenius-norm inequalities, and spectral bounds.
  • Parameter handling: The proof treats α(Z) and αj(Zj) as fixed quantities when differentiating the update after recomputing them from the current feature.This separates parameter recomputation from the update-direction calculation.
  • Update construction: The Riemannian update direction GT(Z) is obtained by projecting the Euclidean update direction onto the tangent space of the product manifold.This projection is the geometric step connecting the Euclidean MCR^2 derivative to the constrained update.
  • Riemannian continuity: Lemma 4 states that the Riemannian update directions at Z1 and Z2 satisfy a Lipschitz bound with constant Lgrad.The result applies to any two points on M = (S^n−1)^m.
  • Final estimate: Combining the component bounds yields an explicit upper bound for the update difference and completes the Lipschitz proof.The derivation uses projector identities, norm bounds, and the assembled constants in the final inequality.

Appendix E. Proof of Proposition 1

Appendix E proves Proposition 1 by integrating the change of the objective along a geodesic on the product manifold. Lipschitz control of the Riemannian update and parameter-dependent terms yields the stated descent inequality.

  • Geodesic setup: The proof connects an arbitrary tangent vector ξ at Z to Geo(Z, ξ) through a geodesic on the product of spheres.The geodesic starts at Z and ends at Geo(Z, ξ), with each component respecting the sphere constraint.
  • Derivative decomposition: Along the geodesic, Z(s), α(s), and αj(s) all vary with s, so the objective derivative includes both feature and implicit-parameter contributions.The proof separately bounds the term associated with α(s) and the class-wise terms.
  • Parameter bound: The parameter-dependent contribution is bounded using Lemma 1, triangle inequalities, trace bounds, and Cauchy-Schwarz inequalities.The resulting estimate introduces the constant Cα for the implicit-parameter terms.
  • Update bound: The Riemannian update contribution is controlled using Lemma 4 and the orthogonal projection of G(Z) onto the tangent space.The proof combines norm inequalities with the previously established update-map continuity.
  • Conclusion: Integrating the derivative inequality along s and combining the bounds yields Proposition 1 with the constant Ls.The conclusion follows after combining the estimates and setting Ls as specified.
Loading 2608.20668v1…