PaperlayerView exact revision on arXiv

Source-linked AI summary

Lightweight Adaptive ReduNet via Hyperspherical Manifold Learning

Zhenglin Huang, Qifa Yan, Bin Dai, Xiaohu Tang

arXiv:2608.20668v1cs.LGcs.AI

TL;DR

ReduNet’s Euclidean layer-wise updates can require many layers for the MCR^2 objective to reach a stable value, increasing parameter storage. LA-ReduNet refines ReduNet’s updates through hyperspherical manifold learning, normalized adaptive Riemannian updates, and sample-adaptive step sizes. Approximately 1/29 of the unfolded ReduNet module’s parameter storage is required for MCR^2 stabilization, with substantially fewer layers reaching a stable MCR^2 objective value.

  • Problem

    ReduNet’s Euclidean layer-wise updates can require many layers for the MCR^2 objective to reach a stable value, increasing parameter storage.

    Source: §1 · pp. 3–4

  • Method

    LA-ReduNet refines ReduNet’s updates through hyperspherical manifold learning, normalized adaptive Riemannian updates, and sample-adaptive step sizes.

    Source: §1 · p. 4, §5 · p. 38

  • Results

    Approximately 1/29 of the unfolded ReduNet module’s parameter storage is required for MCR^2 stabilization, with substantially fewer layers reaching a stable MCR^2 objective value.

    Source: §1 · p. 5, §5 · p. 38

  • Takeaways & Limitations

    LA-ReduNet achieves stable MCR^2 representations with substantially fewer unfolded layers while retaining comparable classification accuracy.

    Source: §5 · p. 38

    The reported LA-ReduNet setting does not satisfy the sufficient conditions derived from global analysis.

    Source: §4.1.1 · p. 25

Abstract

from arXiv · show

In recent years, a white-box neural network called ReduNet has been proposed, which employs the maximal coding rate reduction (MCR$^2$) principle to transform raw data into low-dimensional discriminative features via a forward layer-wise construction process. Unlike traditional deep networks that rely on backpropagation, ReduNet explicitly derives the parameters of each layer from the features of its preceding layer, offering a mathematically interpretable paradigm. However, this layer-wise construction often requires a large number of layers for the MCR$^2$ objective to reach a stable value, which increases the parameter storage of the unfolded module. To address this issue, we propose LA-ReduNet, a lightweight adaptive architecture that refines the layer-wise update rule and enables discriminative feature representations to be obtained with substantially fewer unfolded layers. Specifically, LA-ReduNet employs hyperspherical manifold learning and adaptive step sizes, thereby reducing by an order of magnitude the number of layers required for the MCR$^2$ objective to reach a stable value. Simulation results demonstrate that, while maintaining comparable classification accuracy, LA-ReduNet requires significantly fewer layers for the MCR$^2$ objective to reach a stable value. Remarkably, under the considered experimental settings, LA-ReduNet requires only approximately $1/29$ of the parameter storage of the unfolded ReduNet module for the MCR$^2$ objective to reach a stable value.

1. Introduction

The paper proposes LA-ReduNet, a lightweight ReduNet that redesigns layer-wise MCR2 updates using hyperspherical manifold learning and adaptive step sizes. It reduces the layers and parameter storage needed for stable objective convergence while preserving comparable classification accuracy.

Background

  • ReduNet derives its architecture from the MCR2 principle, maximizing overall coding rate while minimizing within-class coding rate to capture low-dimensional data structure.Unlike cross-entropy, MCR2 explicitly targets the low-dimensional structures underlying high-dimensional data.Source: §1 · pp. 2–3

Motivation

  • The method addresses Euclidean gradient updates followed by normalization, which do not directly control feature angular displacement on the sphere.Fixed Euclidean step sizes can therefore produce small or highly variable angular updates across samples.Source: §1 · p. 3

Method

  • LA-ReduNet redesigns ReduNet’s layer-wise update under a unit-sphere constraint using a normalized, truncated, sample-adaptive Riemannian update.The update direction is determined in tangent space, while angular step sizes adapt to each sample and small update norms trigger thresholding.Source: §1 · p. 4

Theoretical properties

  • The proposed Riemannian scheme has Lipschitz continuity, a lower-bound MCR2 increment on products of unit spheres, and finite termination under specified conditions.Finite termination depends on the specified threshold and step-size conditions.Source: §1 · pp. 4–5

Simulation results

  • 35 layers versus approximately 1000 layers enable LA-ReduNet to reach stable MCR2 convergence, requiring approximately 1/29 of ReduNet’s parameter storage.These results are reported under the considered experimental settings.Source: §1 · p. 5

2. Preliminaries

The preliminaries introduce MCR2, ReduNet, and AR-ReduNet, then describe ReduNet’s coding-rate optimization and AR-ReduNet’s improved rate-distortion approximation. ReduNet updates features through a gradient-ascent construction, while AR-ReduNet adds adaptive regularization to improve approximation accuracy.

2. Preliminaries

  • The section introduces MCR2, ReduNet, and its improved version AR-ReduNet.Source: §2 · p. 5

2.1. MCR2 Principle and ReduNet Framework

  • ReduNet formulates feature transformation through the MCR2 optimization problem and updates the feature matrix by gradient ascent.The network initializes samples by projecting them onto the unit sphere, with class-membership matrices available during training and estimated from data during testing.Source: §2.1 · pp. 6–8

2.2. AR-ReduNet Based on Improved Approximation of Rate-Distortion Function

  • AR-ReduNet adds an adaptive regularization term to the multivariate Gaussian rate-distortion approximation, bringing it closer to the exact rate-distortion function.The regularization parameter α is selected so the approximation reaches zero at D = tr(Σ), and α and αj are re-evaluated as the relevant matrices update.Source: §2.2 · p. 9

3. Lightweight Adaptive ReduNet (LA-ReduNet)

LA-ReduNet replaces Euclidean updates with tangent-space geodesic updates on the unit sphere and adaptively controls their magnitude. The analysis establishes Lipschitz continuity and finite termination under suitable threshold and step-size conditions.

3.1. MCR 2 Optimization Objective Based on Hyperspherical Manifold Learning

  • Tangent-space updates evolve features along unit-sphere geodesics, discarding ineffective radial gradients while retaining the feasible tangential component.The update rotates each feature through an angle determined by the normalized tangential direction.Source: §3.1 · pp. 11–12
  • The adaptive rule stops samples with ||gT||2 ≤ τ and updates active samples with ||gT||2 > τ using an adaptive step-size parameter t.The threshold τ determines whether a sample remains active, while t controls the update magnitude; Figure 3 specifies t ∈ [t0, t0(1 + β)].Source: Figure 3 · p. 12, §3.1 · p. 13

3.2. Riemannian Update Analysis and Finite-Termination of MCR2

  • The Riemannian update mapping is Lipschitz continuous, providing the basis for analyzing geodesic updates and their MCR2 objective variation.The analysis derives eigenvalue bounds, Lipschitz continuity of implicit parameter and inverse-matrix mappings, and then the update mapping.Source: §3.2 · p. 14
  • Under suitable threshold and step-size conditions, the proposed scheme satisfies its stopping criterion within a finite number of iterations.The objective-variation inequality supplies the theoretical foundation for proving finite termination.Source: §3.2 · p. 14

on M = (Sn−1)m with Lipschitz constant

The section establishes a finite-termination guarantee for the proposed Riemannian update on the product of spheres under prescribed threshold and step-size conditions, while introducing LA-ReduNet’s lightweight architecture. The architecture reduces storage primarily through substantially smaller network depth and uses a shared normalized-feature front-end.

on M = (Sn−1)m with Lipschitz constant

no later than iteration

  • Theorem 1 guarantees that the truncated Riemannian update terminates in finite iterations under the prescribed threshold and step-size conditions.These sufficient conditions are theoretical, may be conservative because they use global worst-case bounds, and guarantee termination even in the worst case when satisfied.Source: no later than iteration · p. 19

3.3. LA-ReduNet Architecture

  • LA-ReduNet’s unfolded-module storage scales as O(L(k + 1)d^2) after reducing feature dimension from n to d, with d ≪ n.All ReduNet variants use the same lightweight dimension-reduction front-end, so LA-ReduNet’s storage reduction mainly comes from its substantially smaller depth.Source: §3.3 · p. 20
  • The shared convolutional front-end is trained separately by backpropagation, whereas the subsequent LA-ReduNet module retains ReduNet’s white-box construction.It produces normalized low-dimensional features for the downstream modules; the illustrated module uses four convolutional blocks, global average pooling, and flattening to produce a 512-dimensional vector.Source: §3.3 · p. 20
  • The front-end combines cross-entropy and MCR2 losses to make learned low-dimensional features compatible with the subsequent MCR2-based modules.Because outputs are unit-normalized, classification uses scaled cosine similarities as logits, making decisions depend on angular similarity.Source: §3.3 · pp. 21–22

4. Simulation Results

This section evaluates LA-ReduNet and competing methods across CIFAR-10, CIFAR-100, and CINIC-10, comparing classification accuracy, MCR² convergence, feature separability, and unfolded-module storage. It also examines the effects of normalization-thresholding, adaptive step sizes, and hyperparameters on convergence and classification.

4. Simulation Results

  • The experiments evaluate ReduNet, AR-ReduNet, and LA-ReduNet as feature extractors on CIFAR-10, CIFAR-100, and CINIC-10 using a Nearest Subspace classifier.For fairness, the baseline and proposed models use the same dimension-reduction module and receive identical input features during forward propagation.Source: §4 · p. 23, §4.1.1 · p. 24

4.1. Effect of Network Layers on Feature Visualization and Classification Accuracy

  • Classification-accuracy convergence can occur earlier than objective convergence, so the experiments distinguish stable classification accuracy from a stable MCR2 objective.Objective convergence denotes a stable MCR2 objective as layers increase, whereas classification-accuracy convergence denotes accuracy remaining near a stable level.Source: §4.1 · p. 24

4.1.1. CIFAR-10 and CIFAR-100; 4.1.2. CINIC-10

  • On CIFAR-10 and CIFAR-100, LA-ReduNet outperforms ReduNet and AR-ReduNet, achieving substantially faster classification-accuracy convergence and higher classification accuracy.Source: §4.1.1 · p. 25, Figure 6 · p. 26, §4.1.2 · p. 27
  • On CIFAR-10 and CIFAR-100, LA-ReduNet achieves objective convergence at approximately 35 layers, whereas ReduNet and AR-ReduNet require between 900 and 1000 layers.Source: §4.1.1 · p. 25, Figure 7 · p. 26
  • With fewer layers, LA-ReduNet extracts more discriminative features than ReduNet and AR-ReduNet, while greatly reducing parameter storage relative to ReduNet; at L = 35, its features exhibit much clearer boundaries.Source: §4.1.1 · pp. 26–27

4.1.2. CINIC-10

  • LA-ReduNet’s parameter storage is approximately 35/1000 ≈ 1/29 of the baseline, while classification-accuracy convergence is generally achieved earlier, typically within approximately 5–10 layers.Source: §4.1.2 · p. 32

4.2.1. Component-wise Ablation Study of LA-ReduNet

  • The normalization-threshold mechanism provides the primary convergence improvement, while adaptive step sizes further accelerate early-layer progress; with β = 1, their curves remain relatively close.The threshold reduces dependence of spherical angular progress on tangent-update magnitude while retaining stopping information.Source: Figure 13 · p. 33, §4.2.1 · pp. 33–34

4.2.2. Impact of Hyperparameter t0 on the Objective Convergence of LA-ReduNet

  • A moderate t0 from 0.05 to 0.1 best balances objective convergence speed and attained objective value; excessively small t0 increases required layers, whereas excessively large t0 lowers the converged value.A relatively small t0 yields a higher MCR2 objective value, but t0 = 0.01 requires more layers for convergence.Source: Figure 14 · p. 34, §4.2.2 · p. 35

4.2.3. Impact of Hyperparameter β on the Objective Convergence of LA-ReduNet

  • Larger β accelerates MCR2 growth, especially for depths L ≤15, while barely changing its peak value; adaptive steps are more stable than large fixed steps and faster than small ones.The recommended combination is larger β with relatively small t0 to preserve the peak MCR2 value.Source: Figure 15 · p. 35, §4.2.3 · p. 36

4.2.4. Ablation Study on the Influence of µ in the Convolutional Module

  • For the convolutional-module weight µ, lower CE loss generally corresponds to higher accuracy, while µ = 10^-3 raises MCR2 without noticeably increasing CE loss and yields the best overall accuracy.At µ = 10^-2, CE loss is substantially higher and classification accuracy is markedly lower than with smaller µ.Source: Table 3 · p. 36, Figure 16 · p. 37, §4.2.4 · p. 37

5. Conclusions and Discussions

LA-ReduNet combines adaptive step sizes with a Riemannian update scheme that has a finite-termination guarantee under explicit conditions. Experiments show substantially fewer layers for stable MCR2 optimization, stable classification within approximately 5–10 layers, and improved convolutional features.

5. Conclusions and Discussions

  • LA-ReduNet introduces an adaptive step-size mechanism and establishes finite termination for its Riemannian updates under explicit threshold and step-size conditions.The architecture is presented as a lightweight ReduNet design.Source: §5 · p. 38
  • Approximately 35 layers suffice for the MCR2 objective to reach a stable value, while LA-ReduNet’s classification accuracy generally stabilizes within approximately 5–10 layers.These results hold under the considered experimental settings and compare favorably in layer efficiency with gradient ascent-based ReduNet and AR-ReduNet.Source: §5 · p. 38
  • The authors expect LA-ReduNet to extend to broader downstream tasks and integrate as a modular component in different neural network architectures.This is presented as a future application direction rather than an established experimental result.Source: §5 · p. 38
  • With a convolutional module, LA-ReduNet yields features with superior classification accuracy and better-separated class boundaries.The supplied passages report this as an experimental finding.Source: §5 · p. 38

Appendices

  • Proposition 1 is supported by four lemmas proved in Appendices A–D, followed by the proposition’s detailed proof in Appendix E.The appendices provide the supporting proof structure for Proposition 1.Source: Appendices · p. 38

Appendix A. Positive Definiteness and Uniform Eigenvalue Bounds

The appendix establishes positive definiteness for a matrix constructed from any matrix with unit-norm columns and α ∈ [0, 1].

  • For any Z ∈ R^n×m with unit-norm columns, the lemma considers α ∈ [0, 1].Source: Appendix A · p. 39
  • Under these conditions, αI + n/tr(ZZT)ZZT is positive definite.Source: Appendix A · p. 39

eigenvalues satisfy

The eigenvalues λ_i satisfy the uniform bounds 1/(n+1) ≤ λ_i ≤ n+1, which implies positive definiteness of the associated matrix.

  • 1/(n+1) ≤ λ_i ≤ n+1 for i = 1, . . . , n.The bound is stated in (A.2) and reiterated in (A.22).Source: eigenvalues satisfy · pp. 39, 41
  • The eigenvalues are strictly positive, including the n = 1 case where λ_1 = 1.Nonnegativity of α and q_i gives positivity, while the n = 1 reduction yields α = 0 and λ_1 = 1.Source: eigenvalues satisfy · p. 39
  • Therefore, αI + n/tr(ZZT)ZZT is positive definite.Its eigenvalues are positive and obey the stated upper and lower bounds.Source: eigenvalues satisfy · p. 41

Appendix B. Lipschitz Continuity of the Implicit Parameter Map-

Appendix B establishes that the implicitly recomputed parameters α(Z) and α_j(Z_j) are unique solution mappings with Lipschitz continuity on hyperspherical product manifolds. The proof uses geodesics, positive definiteness, differentiability, and analogous class-wise arguments.

Lemma 2

  • α(Z) is the unique solution of (14a), and its mapping is Lipschitz continuous for feature matrices on M = (S^(n−1))^m.The manifold consists of n × m matrices with unit-norm columns.Source: Appendix B · p. 42

Proof of Lemma 2

  • The proof bounds variation in α along a minimizing geodesic by differentiating the implicit log-determinant condition and integrating the derivative bound.Positive definiteness ensures differentiability of the unique solution along the geodesic.Source: Appendix B · pp. 42, 43, 45

Class-specific parameters

  • Analogous Lipschitz results apply to the class-specific implicit parameters α_j(Z_j), with Z_j containing the samples from class j.The argument is repeated for the corresponding class-wise parameter along a geodesic.Source: Appendix B · pp. 46–47

Appendix C. Lipschitz Continuity of the Inverse Matrices in the MCR2 Objective

Appendix C establishes that the inverse matrices in the MCR2 update vary Lipschitz-continuously with Z on the hyperspherical manifold. The proof uses inverse-matrix identities, product-norm bounds, eigenvalue lower bounds, and unit-norm column constraints.

Appendix C. Lipschitz Continuity of the Inverse Matrices in the MCR2 Objective

  • For any Z1, Z2 ∈ M = (S^(n−1))^m, the inverse matrices in the MCR2 update satisfy a Lipschitz bound with constant Linv > 0.The result is derived for the full hyperspherical-manifold domain and extends similarly to class-wise submatrices.Source: Appendix C · p. 47

Proof. Let

  • The proof bounds the difference between inverse matrices using the identity for inverse differences, matrix-product norm inequalities, and positive lower eigenvalue bounds for Qi.Lemma 1 and Lemma 2 control the resulting terms, while the unit-norm column constraint bounds the difference between the associated covariance-like matrices.Source: Proof. Let · pp. 47–49
  • The combined estimates establish Lipschitz continuity of the inverse-matrix mapping and complete the proof with Linv defined as in (C.2).The same argument is applied to class-wise submatrices.Source: Proof. Let · pp. 49–50

Appendix D. Lipschitz Continuity of the Riemannian Update Map-

Appendix D establishes Lipschitz continuity of the Riemannian update mapping on the product manifold of spheres. The proof derives this by bounding the Euclidean update direction and projecting it onto the tangent space.

Update-map construction

  • The update map recomputes α(Z) and αj(Zj), treats them as fixed when differentiating the MCR2 objective, and then forms the Euclidean update direction.The Riemannian direction is obtained by projecting this Euclidean direction onto the tangent space of the product of spheres.Source: Appendix D · pp. 50–51

Lipschitz-continuity result

  • Lemma 4 states that the Riemannian update directions at any two points on M = (S^(n−1))^m satisfy a Lipschitz bound with constant Lgrad > 0.The directions are denoted GT(Z1) and GT(Z2).Source: Appendix D · pp. 50–51

Proof strategy

  • The proof establishes separate Lipschitz bounds for the Euclidean update direction’s terms and combines them using the Frobenius-norm triangle inequality and prior lemmas.The final bound uses Lemmas 1 and 3, bounds involving the sphere-product dimensions, and the constraint ∥Z2∥F = √m.Source: Appendix D · pp. 51, 52, 55

Appendix E. Proof of Proposition 1

Appendix E proves Proposition 1 by analyzing the relevant quantities along a geodesic on the manifold and bounding their variation. Integrating the resulting differential inequality and selecting Ls as in (28) establishes (25).

  • The proof fixes a point Z on M and a tangent vector ξ satisfying the tangent-space constraint, then differentiates the geodesic-dependent quantities.The argument uses Z(s), α(s), and αj(s) as functions of s and applies the chain rule.Source: Appendix E · pp. 55, 56, 58
  • The proof bounds the differential terms using orthogonal projection, class-wise estimates, the triangle inequality, and Cauchy–Schwarz inequality.The bounds combine Lemma 1 and the definition Cα = 2n√n + 1 m−1/2.Source: Appendix E · pp. 58–59
  • Integrating the differential inequality over s ∈ [0, 1], with Z(0) = Z and Z(1) = Geo(Z, ξ), yields the required geodesic comparison.The proof separately bounds the two terms in the integrated expression before combining them.Source: Appendix E · pp. 59–61
  • Setting Ls as specified in (28) completes the proof of Proposition 1 and establishes (25).Source: Appendix E · p. 61

Related papers

Loading 2608.20668v1…