PaperlayerView exact revision on arXiv

Source-linked AI summary

Symmetry-quotient Flatness and Generalization

Taiki Miyagawa

arXiv:2609.31634v1cs.LGstat.ML

TL;DR

Despite the influential view that SGD favors flat minima that generalize, the pipeline from optimization-side stability and geometry to function-side regularity and generalization had not been formulated in symmetry-quotient geometry. This paper develops quotient-flatness results linking stability to input smoothness and, under local assumptions, generalization.

  • Problem

    The optimization-side stability-and-geometry to function-side regularity-and-generalization pipeline had not been formulated in symmetry-quotient geometry, despite the influential view associating SGD with flat minima that generalize.

    Source: §1 · pp. 1–2

  • Method

    For square-loss interpolation models with smooth function-preserving group actions, the paper defines quotient flatness on the regular quotient manifold and develops stability, smoothness, and generalization results, including higher-order tensor moments.

    Source: §1 · p. 2

  • Results

    One-step mean-square quotient linear stability bounds quotient flatness; quotient flatness controls input smoothness, and local covering and boundedness assumptions yield generalization bounds from both quantities.

    Source: §1 · p. 2

  • Takeaways & Limitations

    For square-loss interpolation solutions, quotient flatness equals the average squared norm of quotient gradients evaluated on the training data.

    Source: §1 · p. 2

    Extending the theory beyond the regular stratum of the global permutation quotient may require orbifold or stratified geometry; empirical evidence for deep architectures remains limited.

    Source: §6 · p. 12

Abstract

from arXiv · show

This paper develops a theorem-level pipeline in symmetry-quotient settings: quotient linear stability implies quotient flatness, quotient flatness implies input smoothness, and input smoothness yields generalization under local covering assumptions. Flatness is often associated with generalization, and Stochastic Gradient Descent (SGD) is frequently viewed as implicitly biased toward flat solutions. However, standard flatness measures are typically defined in the raw parameter space and are therefore not invariant under function-preserving symmetries such as positive rescaling. We develop a symmetry-aware theory of quotient flatness, quotient linear stability, input smoothness, and generalization on quotient spaces of neural-network parameters. For square loss and models equipped with function-preserving group actions, we define quotient flatness as the trace of the Hessian of the empirical loss on the regular quotient manifold. We show that quotient flatness controls input smoothness through a quotient-space analogue of the flatness-to-smoothness argument. We also prove that one-step mean-square quotient linear stability of the linearized SGD dynamics implies an explicit quotient-flatness bound in terms of the batch size and learning rate, and extend this analysis to higher-order tensor moments. Finally, under local covering and boundedness assumptions, we derive population generalization bounds in terms of quotient flatness and, consequently, in terms of quotient linear stability.

1 Introduction

The paper addresses the lack of an explicit flatness-to-generalization theory in symmetry-quotient geometry and the representation dependence of raw parameter-space flatness. It develops a quotient-space pipeline from linear stability through flatness and input smoothness to generalization.

1 Introduction

  • The paper formulates an explicit symmetry-quotient route from optimization-side stability and geometry to function-side smoothness and generalization, under suitable distributional assumptions.Earlier work established this route in parameter space, but it had not been formulated in quotient geometry.Source: §1 · pp. 1–2
  • Raw parameter-space flatness is generally not invariant to function-preserving symmetries, including hidden-unit positive rescaling and permutation.The paper illustrates the distinction with raw and quotient loss slices for a two-layer ReLU network trained on MNIST.Source: §1 · p. 2, Figure 1 · p. 3
  • For interpolation solutions under square loss, quotient flatness equals the average squared norm of quotient gradients evaluated on training data.This identity provides the geometric bridge for the subsequent analysis.Source: §1 · p. 2
  • Quotient flatness controls input smoothness through a multiplicative first-layer representation and a local gauge-fixing section.These ingredients let derivatives with respect to first-layer parameters be compared with quotient derivatives.Source: §1 · p. 2
  • One-step mean-square quotient linear stability of linearized SGD implies an explicit upper bound on quotient flatness, and higher-order quotient moments yield higher-order input-smoothness estimates.The stability implication runs from quotient linear stability to quotient flatness, not the reverse.Source: §1 · p. 2
  • Under local covering and boundedness conditions around training data, the paper derives generalization bounds from quotient flatness and, by combining results, quotient linear stability.The introduction presents the resulting pipeline as quotient linear stability ⇒ quotient flatness ⇒ input smoothness ⇒ generalization.Source: §1 · p. 2

2 Related Work

Prior work links flatness and SGD stability to input smoothness and generalization, while symmetry studies show that raw parameter geometry can depend on representation. This paper reformulates that theoretical route on quotient loss geometry.

2.1 Flatness, Generalization, and Implicit Regularization

  • Empirical studies report that SGD often selects flatter solutions that generalize better under suitable local measures, motivating flatness-aware optimization and related views.The passage also situates this literature in minimum-description-length and PAC-Bayes perspectives.Source: §2.1 · p. 3
  • Prior work showed that parameter-space flatness and higher-order SGD stability control input-side Sobolev seminorms, yielding generalization bounds under distributional covering assumptions [9].The present work follows this route but reformulates it on symmetry-reduced quotient loss geometry.Source: §2.1 · pp. 3–4

2.2 Symmetry, Quotient Geometry, and Function-space Viewpoints

  • Function-preserving symmetries make raw parameter distance, connectivity, and flatness difficult to interpret; raw loss slices can contain redundant scale directions.The quotient slice removes those directions.Source: §2.2 · p. 4
  • Earlier symmetry-reduced work normalized hidden units and aligned permutation-equivalent networks to study flat regions and geodesic paths; this paper defines quotient flatness intrinsically and proves theorem-level links.The stated links connect quotient flatness to quotient stability, input smoothness, and generalization.Source: §2.2 · p. 4

3 Preliminaries

The preliminaries define a regular quotient manifold for a function-preserving group action and equip it with a Riemannian metric. Under square loss and interpolation, quotient flatness has an exact gradient-norm characterization.

3 Preliminaries

  • A smooth function-preserving group action leaves model outputs unchanged, and its free, proper regular stratum defines the quotient manifold M = Θreg/G.The quotient map is denoted π.Source: §3 · p. 4
  • The empirical square loss on the quotient averages squared prediction residuals, and an interpolation solution matches every training label.The loss is defined using the quotient model at each training input.Source: §3 · p. 4
  • The quotient manifold is equipped with a fixed Riemannian metric gQ, which determines the gradients, Hessian, traces, and norms used to define quotient flatness.The definition takes the metric trace of the empirical-loss Hessian at an interpolation solution.Source: §3 · p. 4
  • At interpolation under square loss, quotient flatness equals the empirical average of squared quotient-gradient norms; without interpolation, a residual-dependent second-derivative term remains.The identity makes the quantity a first-order gradient-norm measure only under the stated interpolation condition.Source: §3 · p. 5

4 Quotient Flatness Implies Input Smoothness

The section links quotient flatness to input smoothness and shows that quotient linear stability can bound flatness, with higher-order stability controlling stronger derivative moments. These local estimates support population generalization only under additional covering and boundedness assumptions.

4 Quotient Flatness Implies Input Smoothness

  • With a local gauge-fixing section, first-layer factorization, and nonzero training inputs, quotient flatness controls average input sensitivity through the first-layer chain rule.The section makes the first-layer block well defined on the quotient; the input-norm condition prevents the bound from degenerating.Source: §4 · p. 5

4.1 Quotient Linearized SGD and Quotient Flatness Bound

  • One-step mean-square quotient linear stability bounds quotient flatness by 2B/η; in the two-layer ReLU illustration, all 11 stable pairs satisfy the bound.The illustration compares stable and unstable learning-rate and batch-size pairs.Source: §4.1 · pp. 6–7, Figure 2 · p. 7

4.2 Higher-order Quotient Moment Control and Bound for Input Smoothness

  • A closed tensor recursion separates second-order flatness control from higher empirical gradient moments needed for stronger input-smoothness bounds.The higher moments are not needed for the basic flatness bound, but are used to control higher moments of input gradients.Source: §4.2 · p. 8
  • One-step 2k-mean quotient stability yields empirical higher moments of quotient gradients, which transfer to input-derivative moments through the gauge-section comparison.The resulting derivative estimates are local to the training inputs and the linearized quotient dynamics near interpolation.Source: §4.2 · pp. 8–9
  • Population generalization bounds additionally require local covering and boundedness conditions around the training data.Under these conditions, the input-smoothness estimates control prediction error on covered regions, while boundedness controls uncovered mass.Source: §4.2 · p. 9

5 Quotient Flatness Implies Generalization under Local Covering

Under local covering and boundedness assumptions, quotient flatness yields a population generalization bound, and quotient stability can reach that bound through the preceding results. The paper illustrates the bound in a two-layer ReLU teacher–student task.

5 Quotient Flatness Implies Generalization under Local Covering

  • Under the stated local covering, Lipschitz, boundedness, and local-affinity assumptions, Theorem 4 derives a population generalization bound from quotient flatness.The balanced covering condition requires disjoint neighborhoods near training points and a mass-balance constraint.Source: §5 · p. 10
  • The bound separates quotient flatness as a geometric term from local data coverage as a separate contribution.The authors note that balanced local covering may be restrictive when high-dimensional data are diffuse.Source: §5 · p. 10

Corollary 2 (From quotient stability to generalization). Under the assumptions of Thms. 2 and 4,

  • In the teacher–student two-layer ReLU illustration, empirical generalization error remains below the theoretical bound, and quotient flatness tracks the intermediate input-smoothness relation.The reported result is an empirical illustration of Theorem 4.Source: Corollary 2 (From quotient stability… · p. 10

6 Limitations

The paper identifies open extensions involving cross-entropy, quotient geometry beyond the regular stratum, and limited empirical evidence in deep architectures.

6 Limitations

  • Cross-entropy introduces a curvature-residual term that breaks the exact interpolation identity used in the analysis.The identity for square loss relies on interpolation removing the residual-dependent second-derivative term.Source: §6 · p. 10
  • Extending the theory beyond the regular stratum may require orbifold or stratified geometry because the global permutation quotient is naturally stratified.The stated results concern the regular quotient manifold.Source: §6 · p. 12
  • Empirical evidence remains limited, and reproducing flatness–generalization phenomena in deep architectures is difficult.The illustrated experiment uses a two-layer ReLU teacher–student task.Source: Figure 3 · p. 11, §6 · p. 12

A.1 Proof of Prop. 1

At an interpolation solution, the proof uses the square-loss Hessian chain rule to eliminate the residual term and express quotient flatness through squared quotient gradients.

A.1 Proof of Prop. 1

  • The square-loss chain rule decomposes each sample Hessian into a squared differential term and a residual-weighted Hessian term.For ϕ_i = h ◦ r_i with h(t) = 1/2 t^2, the proof uses h′(t) = t and h′′(t) = 1.Source: §A.1 · p. 13
  • At an interpolation solution, each residual is zero, so the sample Hessian reduces to the outer product of the sample-loss differential with itself.This follows because ¯f_i([θ⋆]) = y_i for every sample.Source: §A.1 · p. 13
  • Taking the quotient-metric trace and averaging the sample contributions yields the quotient-flatness identity.The proof takes the trace in a quotient-metric orthonormal basis and uses linearity of the Hessian and trace.Source: §A.1 · p. 13

A.2 Proof of Thm. 1

The proof uses a gauge-fixed first-layer factorization to compare input derivatives with first-layer parameter derivatives, then bounds the input gradient by quotient-controlled derivatives.

A.2 Proof of Thm. 1

  • The gauge-fixed representation writes the model through W1x, allowing the chain rule to express the input gradient using first-layer parameters.The proof sets z = W1,⋆x and differentiates the represented function with respect to z.Source: §A.2 · p. 13
  • Differentiating the first-layer representation with respect to W1 produces a rank-one matrix whose Frobenius norm factors into the norms of the z-gradient and x.The proof uses ∇W1f = gz(x)x⊤.Source: §A.2 · p. 13
  • Combining the derivative relations bounds the input-gradient norm by the first-layer gradient norm scaled by ∥W1,⋆∥op/∥x∥2.The proof applies this inequality at each training input to obtain the claimed estimate.Source: §A.2 · p. 13

A.3 Proof of Thm. 2

The proof uses the linearized update and a Cauchy–Schwarz bound to derive a quotient-flatness bound. It handles the zero-flatness case separately.

A.3 Proof of Thm. 2

  • The linearized update is written as M_I = I − ηA_I.The accompanying inequality relates the expected trace of A_I to the update.Source: §A.3 · p. 13
  • Cauchy–Schwarz is used to bound the expected trace of A_I squared.Source: §A.3 · p. 14
  • When quotient flatness is nonzero, the proof divides by it to conclude FlatQ([θ⋆]) ≤ 2B.The zero-flatness case is stated to be trivial.Source: §A.3 · p. 14

A.4 Proof of Prop. 2

The proof establishes higher-order quotient-moment bounds, applies them to an input-gradient bound, and uses local covering arguments to obtain generalization bounds and a corollary by substitution.

A.5 Proof of Thm. 3

  • The mini-batch moment argument bounds the higher-order quotient stability coordinate-wise and then in the Riemannian norm.It uses a mini-batch sampled with replacement, the one-step 2k-mean stability assumption, and upper and lower bounds on the k-th moment.Source: §A.5 · p. 14

A.6 Proof of Cor. 1

  • The corollary combines a pointwise training-input gradient estimate with the Riemannian-norm higher-order quotient-moment bound.Source: §A.6 · p. 15

A.7 Proof of Thm. 4

  • For interpolation solutions, local affinity and Lipschitz continuity bound prediction error near training points, while a uniform bound controls the complement.The proof integrates the complement bound and combines the inequalities with Theorem 1.Source: §A.7 · p. 15

A.8 Proof of Cor. 2

  • The final corollary follows by substituting the quotient-flatness bound from Theorem 2 into Theorem 4.Source: §A.8 · p. 15
Loading 2609.31634v1…