PaperlayerView exact revision on arXiv

Source-linked AI summary

Across the Loss Landscape with Progressive Growth

Paul Caillon, Christophe Cerisara, Alexandre Allauzen

arXiv:2608.24568v1cs.LGcs.AI

TL;DR

The paper asks how progressive growth changes which minima remain accessible, amid interest in loss-landscape geometry and generalization. It models growth as gradual relaxation of frozen-coordinate constraints and finds an accessibility bias toward flatter minima, while showing that geometric effects do not ensure improved test accuracy.

  • Problem

    How progressive growth changes which minima remain accessible is the question, against a backdrop of research connecting flat minima with robustness and generalization.

    Source: §1 · p. 1, §5 · p. 16

  • Method

    The paper models progressive growth as nested affine-slice constraint relaxation and analyzes local accessibility through frozen-direction curvature and initialization compatibility.

    Source: §2.1 · p. 4, §2.5 · p. 7, §5 · p. 16

  • Results

    Progressive growth biases accessibility toward flatter minima, while toy selection reverses when sharper basins are closer to initialization and deep-network curvature effects do not ensure accuracy gains.

    Source: §4 · p. 15, §5 · p. 16

  • Takeaways & Limitations

    Growth-induced flatness is an accessibility bias under constraint relaxation, not a universal predictor of generalization.

    Source: §5 · p. 16

    The study probes the mechanism rather than training efficiency, and its implementation provides no clear final-predictive-performance advantage over full-model training.

    Source: §4 · p. 16

Abstract

from arXiv · show

Deep neural networks generalize well despite their highly nonconvex, overparameterized loss landscapes, a phenomenon often associated with the geometry of the minima found by stochastic optimization. We study how incremental grow-and-optimize strategies bias training toward flatter regions by viewing growth as progressive constraint relaxation. Starting from a low-dimensional submodel, we iteratively expand the trainable parameters by unlocking nested random subspaces while freezing the orthogonal complement at the network initialization, re-optimizing after each expansion until the full architecture is reached. Under standard local regularity conditions around non-degenerate minima, we prove that local sublevel sets are well approximated by ellipsoids and that basin accessibility under frozen constraints can be characterized by an explicit effective curvature in the frozen directions. This leads to an explanation of the bias: progressive growth increases the relative weight of wide basins and suppresses sharp ones through a volume effect induced by the frozen constraints. We empirically validate these predictions in controlled toy landscapes and in a realistic ResNet/CIFAR-100 setting and confirm that although progressive subspace growth reliably produces flatter solutions, curvature reductions do not universally translate into improved test performance, highlighting subtleties in the flatness-generalization connection. The code is available at https://github.com/p0lcAi/Across-the-Loss-Landscape.

1 Introduction and Related Work

The paper models progressive network growth as optimization under nested frozen-coordinate constraints and characterizes how relaxing those constraints biases basin accessibility. It uses this framework to study growth-induced geometry in toy landscapes and ResNet/CIFAR-100, while treating flatness as a training bias rather than a universal predictor of generalization.

Background

  • Flatness is a setting-dependent geometric bias, not a universal explanation of generalization, because it depends on parameterization and measurement choices and proposed predictors have mixed results,, [18], [14].The classical account links SGD’s stochasticity to flat solutions and diffusion-like dynamics,.Source: §1 · pp. 1–2

Related work

  • Low-loss connectivity and interpolation barriers probe broad basins beyond local curvature, while optimizer dynamics and costly, parameterization-sensitive Hessian measures complicate geometric comparisons, [16],, [19], [33],, [25].Top eigenvalue and trace estimates, including stochastic Hessian-vector-product estimators, remain useful diagnostics.Source: §1 · p. 2
  • Prior evolving-capacity methods include function-preserving expansion, grow-and-prune, architecture search, and sparse training [7],, [20], [28], [12] [34],; earlier growth studies found flatter optima.Those theoretical arguments relied on a stylized basin-volume view in which lower-dimensional constraints more often intersect wider basins.Source: §1 · pp. 2–3

Proposed framework

  • Progressive growth starts from a low-dimensional submodel and unlocks nested parameter subspaces, with inactive coordinates frozen at initialization, to characterize which minima remain accessible as constraints relax.Each stage is optimization on an affine slice of the full parameter space.Source: §1 · p. 3
  • Accessibility reflects an energy–entropy tradeoff: random frozen constraints favor basins compatible with more constraints, with location and effective frozen-direction curvature determining the probability.The curvature term follows from a Schur-complement reduction; the mechanism arises from the feasible set rather than an explicit objective modification, unlike Entropy-SGD [5].Source: §1 · p. 3

Empirical protocol

  • The empirical protocol tracks basin transitions with interpolation barriers, retention, and leakage metrics, tests the mechanism in toy problems and ResNet/CIFAR-100, and separates growth from delayed capacity release.The evaluation is explanatory, not a recipe for improving test accuracy, and considers transition geometry alongside final predictive performance.Source: §1 · p. 3

2 Theoretical Analysis: Progressive Growth Biases Optimization Toward Flat Regions

The theory models progressive growth as optimization on increasingly relaxed affine slices and explains its flat-minimum bias locally: broad minima remain accessible under a larger set of frozen constraints. This is a local accessibility mechanism, not a claim that optimization finds the globally flattest minimum or that flatness universally explains generalization.

2.1 Setup: progressive growth as optimization on affine slices

  • At each stage, only a random active subspace is trainable, so optimization follows affine slices whose dimensions grow as constraints are relaxed.The orthogonal complement remains frozen at initialization during that stage.Source: §2.1 · p. 4

2.2 Local minima and accessibility

  • A minimum is stage-accessible only if its basin lies sufficiently near the current affine slice; frozen-coordinate mismatch can make it unreachable despite optimization within the slice.Accessibility means the stage permits optimization to come sufficiently close to the minimum.Source: §2.2 · p. 5

2.3 Local regularity and quadratic approximation

  • Under local positive-definite, Lipschitz-Hessian regularity, minima's basins are approximated by Hessian-governed ellipsoids.For networks with exact or approximate symmetries, the analysis is interpreted on normal directions to a local minimum manifold; a full treatment of degenerate manifolds is out of scope.Source: §2.3 · p. 5

2.4 Frozen coordinates induce an effective curvature

  • Re-optimizing active coordinates yields a locally quadratic reduced loss whose Schur-complement curvature Σ_i,t measures the cost of frozen offsets and thus effective flatness.Smaller Σ_i,t means loss rises more slowly under frozen-coordinate perturbations, leaving more frozen configurations compatible with the minimum.Source: §2.4 · p. 6

2.5 Compatibility volume and local accessibility law

  • Locally, accessibility combines initialization density near a minimum with a det(Σ_i,t)^-1/2 compatibility-volume factor, favoring lower frozen-direction curvature and, under isotropic Gaussian initialization, proximity to initialization.The density term reflects how likely the frozen coordinates place the slice near the minimum; the determinant term reflects the size of compatible frozen configurations.Source: §2.5 · p. 7

2.6 Interpretation of the Theoretical Results

  • Progressive growth biases optimization toward broad minima because they tolerate more frozen-coordinate mismatches and persist as constraints relax, without guaranteeing the globally flattest minimum.The theory predicts earlier appearance of flatter regions, stage-to-stage retention with limited immediate degradation after expansion, and low-barrier interpolation between successive solutions that remain in one basin.Source: §2.6 · p. 7

3 Empirical Investigation

Toy landscapes show that progressive growth biases selection toward flatter basins when their locations are comparable, but nearby sharp basins can reverse that bias. In ResNet/CIFAR-100, growth leaves a geometric signature without improving final accuracy; controls indicate that the curvature effect is tied to gradual relaxation.

3.1 Toy validation of the growth-induced flatness bias

  • In the multiplicity regime, more growth stages monotonically increase selection of the top-5 flattest minima and decrease selected-basin curvature.This matches the accessibility-law prediction that repeated freezing and release disproportionately preserves broad minima.Source: Figure 1 · p. 8, §3.1 · p. 9
  • When sharp basins are closer to initialization, progressive growth increasingly selects them over distant flat basins, reversing the flatness trend.The reversal appears in both the top-5 flat-basin selection probability and selected-basin curvature, consistent with competition between distance and compatibility volume.Source: Figure 1 · p. 8, §3.1 · p. 9

3.2 Does the bias translate to deep learning models?

  • Validation-accuracy growth stays geometrically mild, while training-loss growth becomes less stable as stages increase.Validation-accuracy retention rises from 0.83 (S = 3) to 0.92 (S = 5) and 0.96 (S = 10); training-loss mean barriers rise from 0.00 to 0.125 and 0.144, while retention shifts from 1.0 to 0.75 and then 0.89. For training-loss growth at S = 10, patience changes barriers more than final accuracy; patience 2 yields the lowest mean barrier, with retention essentially unchanged.Source: §3.2 · pp. 10–12, Figure 2 · p. 10
  • The full-model baseline achieves the best test accuracy; progressive-growth schedules perform slightly worse.Validation-accuracy growth yields 74.99% (S = 3), 74.94% (S = 5), and 73.94% (S = 10) test accuracy; training-loss growth yields 75.54%, 74.40%, and 73.37%, respectively.Source: §3.2 · p. 10, Table 1 · p. 11
  • Transitions with larger interpolation barriers or leakage tend to have larger absolute endpoint gaps, with Spearman correlations of approximately 0.31 and 0.38, respectively.These transition metrics predict how strongly a growth step perturbs the current solution.Source: §3.2 · p. 11, Figure 3 · p. 12
  • Across schedules and triggers, newly released directions are flatter than active ones, with λnew_max/λact_max below 1 in every configuration.The ratio is about 0.31 at S = 3, roughly 0.20 for validation-accuracy growth and 0.17 for training-loss growth at S = 5, and about 0.09 at S = 10.Source: §3.2 · pp. 11–13, Table 1 · p. 11, Figure 4 · p. 13

3.3 Delayed release versus progressive relaxation

  • Fixed-subspace training reaches only 3.13% test accuracy, while one-shot unfreezing has curvature ratio 0.7913 ± 0.0076, above progressive-growth examples.Validation-accuracy growth has ratios 0.201 ± 0.014 at S = 5 and 0.0896 ± 0.0058 at S = 10; one-shot unfreezing recovers most full-model performance with almost zero interpolation barrier, indicating that the low-curvature signal is tied to gradual relaxation rather than delayed capacity release alone. The same qualitative contrast appears in an overparameterized MLP on a hard two-moons task at comparable test accuracy.Source: §3.3 · pp. 13–14, Table 2 · p. 14

4 Discussion and Limitations

Progressive growth biases optimization toward minima that remain accessible as frozen constraints are relaxed, shaping training geometry without guaranteeing better predictive performance. In the studied ResNet setting, this geometric effect was clear while its effect on final accuracy was subtle and often negative.

Discussion

  • Progressive growth shapes optimization geometry by favoring minima accessible under relaxed frozen constraints, but it does not guarantee improved prediction and can reduce final accuracy.In ResNet/CIFAR-100, moderate schedules preserve consecutive stage solutions, keep interpolation barriers small, and make newly released directions flatter; fragmented schedules can degrade performance.Source: §4 · pp. 14–16
  • Toy results show flatter-minimum selection when basin locations are balanced, but the bias can reverse when sharper basins lie sufficiently closer to initialization.In ResNet/CIFAR-100, one-shot unfreezing recovers most full-model accuracy without producing the same low-curvature newly released directions; more progressive stages strengthen the curvature bias, while barrier and accuracy effects remain schedule-dependent.Source: §4 · pp. 14–15
  • Transition-level diagnostics such as barrier and leakage reveal stage stability more directly than final curvature scalars and correlate with endpoint disruption after expansion.The authors propose using these measures as possible online signals for adapting or stopping growth, trading off transition stability against predictive performance.Source: §4 · p. 15

Limitations

  • The theory assumes isolated nondegenerate minima with positive-definite Hessians, while deep-network evidence is limited to one architecture family and dataset and uses local curvature proxies.Extending the theory to degenerate minima or minimum manifolds and broadening empirical coverage are identified as next steps; local curvature estimates do not by themselves define basin membership.Source: §4 · pp. 15–16
  • The study probes growth’s theoretical mechanism rather than optimizing training efficiency, and its schedules offer no clear final-performance advantage over full-model training.The paper frames expansion more broadly as a way to filter minima fragile under partial freezing, with possible connections to sparse or masked optimization, pruning-and-regrowth, parameter-subset curricula, and multi-stage warm starts.Source: §4 · p. 16

5 Conclusion

Progressive growth steers optimization by relaxing nested constraints, favoring minima compatible with initialization and broad in frozen directions. Experiments support this geometric bias while showing it is not a universal predictor of generalization.

Conclusion

  • Progressive growth constrains training to nested affine slices, favoring minima compatible with initialization and broad in frozen directions, rather than universally predicting generalization.The authors characterize this as an energy–entropy trade-off in accessibility as constraints are relaxed.Source: §5 · p. 16
  • Toy landscapes and ResNet/CIFAR-100 support a geometric bias: staged growth changes accessible regions even when its effect on final accuracy is limited.Toy results reverse when sharper basins are sufficiently closer to initialization; on CIFAR-100, moderate schedules preserve consecutive stage solutions, keep interpolation barriers small, and release systematically flatter directions.Source: §5 · pp. 16–17
  • Controls indicate that gradual constraint relaxation, not merely starting from a smaller submodel, underlies the geometric bias.One-shot unfreezing recovers most full-model accuracy but lacks the same low-curvature released directions, while fixed-subspace training fails on CIFAR-100.Source: §5 · pp. 16–17
  • Progressive growth is therefore a way to steer optimization and motivate training strategies that exploit accessibility under structured constraint relaxation.The conclusion frames staged expansion as changing the regions explored by training, beyond simply allocating capacity.Source: §5 · p. 17

A Proofs for Section 2 · A.1 A local Taylor estimate

The appendix recalls the assumptions and results for the theoretical analysis, then proves a local Taylor estimate with a cubic remainder. Its proof section states local ellipsoidal, effective-curvature, and accessibility results under Assumption 2.

A Proofs for Section 2

  • The proof appendix fixes a basin index and training stage, then recalls the analysis assumptions and results.These conventions apply throughout the appendix proof.Source: A Proofs for Section 2 · p. 19
  • The local Hessian is assumed Lipschitz within a neighborhood of the minimizer, with positive radius and Lipschitz constants.Source: A Proofs for Section 2 · p. 19
  • Under Assumption 2, Proposition 2 states a local ellipsoidal approximation with positive constants.Source: A Proofs for Section 2 · p. 19
  • Under Assumption 2 and for frozen offsets near zero, Theorem 3 characterizes effective curvature using a Hessian-block Schur complement.The Schur complement is expressed in coordinates (a, b).Source: A Proofs for Section 2 · p. 19
  • Theorem 4 gives a local accessibility law when the frozen offset has a continuous conditional density near zero; its stated limit is as ε approaches zero.The theorem’s expression includes the volume of the unit ball in R^p_t.Source: A Proofs for Section 2 · p. 19

A.1 A local Taylor estimate

  • Under Assumption 2, Lemma 1 gives a second-order Taylor expansion with cubic remainder for perturbations of norm at most r_i.Source: §A.1 · p. 20
  • The proof derives the remainder using Taylor’s theorem with an integral remainder.Source: §A.1 · p. 20

A.2 Proof of Proposition 2 · A.3 Preliminaries for the reduced problem

The proof establishes local comparability between the loss and its quadratic model, then develops the reduced-coordinate Hessian and Taylor expansion needed for the reduced problem.

A.2 Proof of Proposition 2

  • For sufficiently small radii, the loss is comparable to its quadratic model around the local minimum.The proof uses a radius choice to control the cubic remainder relative to the quadratic term.Source: §A.2 · p. 20
  • A positive loss gap on an annulus ensures the relevant sublevel point lies within the radius where the local comparison applies.The argument then applies the local bound from (2).Source: §A.2 · p. 21

A.3 Preliminaries for the reduced problem

  • In reduced coordinates, the Hessian is expressed in blocks after an orthogonal change of coordinates, which preserves positive definiteness.The block representation is formed at (a, b) = (0, 0).Source: §A.3 · p. 22
  • When the original Hessian is positive definite, the reduced Hessian’s principal block and Schur complement are positive definite.Source: §A.3 · p. 22
  • The reduced-coordinate function admits a local Taylor expansion near (0, 0), with a positive constant bounding its remainder.Source: §A.3 · p. 22
  • The Taylor bound follows by applying Lemma 1 to the coordinate composition, since the orthogonal map preserves Euclidean norms and transfers the Hessian’s local Lipschitz bound.Source: §A.3 · p. 22

A.4 Proof of Theorem 3 · A.5 A local comparison for the compatibility set

Theorem 3’s proof locally eliminates the constrained variables through an implicit function, yielding a quadratic expansion. The compatibility set is then compared with its associated ellipsoid using bounds that hold for sufficiently small ε.

A.4 Proof of Theorem 3

  • The implicit function theorem gives a unique local C1 map a(b) for solving the constrained variables as a function of b.The map is defined in a neighborhood of 0.Source: §A.4 · p. 23
  • After estimating a(b), substituting it into the quadratic part yields the claimed expansion in b.The displayed quadratic form involves Hbb and the correction term HbaH−1…; the passage fragment truncates the latter.Source: §A.4 · p. 23
  • Continuity of a, together with a(0) = 0, allows the neighborhood to be shrunk for the local estimate.Source: §A.4 · p. 23

A.5 A local comparison for the compatibility set

  • The local comparison bounds the compatibility set Bi,t(ε) against the ellipsoid associated with Σi,t, with slack factors 1 ± ηε.The passages establish inclusions using quadratic-form bounds with thresholds (1 − ηε)ε and (1 + ηε)ε.Source: §A.5 · pp. 24–25
  • Because φi,t has a strict local minimum at 0, sufficiently small sublevel sets Bi,t(ε) remain inside a chosen radius δ.Source: §A.5 · p. 24

A.6 Proof of Theorem 4 · A.7 Remark on the Gaussian case · B Additional Experimental Details

Theorem 4’s proof conditions on the chosen subspace and uses local density and continuity to obtain its asymptotic law. Under isotropic Gaussian initialization, the accessibility law makes the energy–entropy tradeoff explicit: minima are favored when they are close to initialization and broad in frozen directions.

A.6 Proof of Theorem 4

  • At a fixed stage and conditioned on the chosen subspace, the proof uses the initialization’s density near zero and continuity as the relevant sets shrink to establish Theorem 4’s asymptotic law.The supplied passages do not include the displayed formula for that law.Source: §A.6 · p. 25

A.7 Remark on the Gaussian case

  • With isotropic Gaussian initialization conditional on the chosen subspace, Theorem 4’s accessibility law takes a Gaussian-specific form.The supplied passage omits the displayed form of the law.Source: §A.7 · p. 26
  • The Gaussian-case interpretation favors minima close to initialization in frozen coordinates and broad along those directions.This makes explicit the energy–entropy tradeoff discussed in the main text.Source: §A.7 · p. 26

B.1 Transition metrics … B.8 Hessian estimation details

The appendix specifies how progressive-growth experiments are implemented and evaluated, from stage expansion and checkpoint selection to transition and restricted-curvature diagnostics. It also documents the CIFAR-100/ResNet-18 setup, internal controls, and Hessian-estimation settings.

B.1 Transition metrics

  • Transition diagnostics measure the maximum excess validation loss along the linear interpolation between stage solutions and classify transitions as retained or leaked using a fixed barrier threshold.Retention and leakage are complementary binary indicators of whether the transition remains in the low-barrier regime.Source: §B.1 · pp. 26–27

B.2 Restricted curvature metrics

  • Restricted curvature is estimated on masked parameter subspaces using top Hessian eigenvalues, trace estimates, and, when needed, stochastic log-determinants.The reported masks are the active and newly released subspaces; trace may also be normalized by the number of active parameters.Source: §B.2 · p. 27

B.3 CIFAR-100 / ResNet-18 setup

  • Deep-learning experiments use CIFAR-100 with a CIFAR-style ResNet-18 and compare full-model training against 3-, 5-, and 10-stage growth triggered by validation accuracy or training loss, averaged over 3 seeds.The training set is split 90/10 into train and validation subsets; standard CIFAR-style augmentation is applied to training data.Source: §B.3 · pp. 27–28

B.4 Progressive growth mechanism

  • Growth groups convolutional and linear parameters by output channel or unit, activating roughly (t + 1)/S of coordinates at stage t while frozen parameters remain clamped to initialization.The classifier is active from the first stage, and matching optimizer states and frozen buffers are also kept at their initial values.Source: §B.4 · p. 28

B.5 Internal control protocols

  • Internal controls compare full-model training, permanently fixed-subspace training, one-shot unfreezing, and progressive growth to distinguish restricted-submodel training, delayed capacity release, and gradual constraint relaxation.Source: §B.5 · p. 28

B.6 Algorithmic summary

  • Progressive growth activates nested parameter groups, freezes the remaining parameters and buffers at initialization, and repeats training and transition diagnostics until the full architecture is released.The final model uses the full architecture; transition diagnostics include interpolation barrier, retention, leakage, endpoint gap, and restricted curvature on active and newly released subspaces.Source: §B.6 · p. 29

B.7 Stage selection and transition evaluation

  • Stage transitions follow patience-based non-improvement in validation accuracy or negative training loss, and transition metrics compare the pre-expansion checkpoint with the best post-expansion checkpoint.Interpolation diagnostics use 11 equally spaced points and at most 2 validation batches; the best validation checkpoint over the run is reported.Source: §B.7 · p. 29

B.8 Hessian estimation details

  • Restricted Hessian estimates use one non-augmented training batch of size 128, with 10 power iterations, 4 Hutchinson probes, and 2 stochastic Lanczos probes of 8 steps with damping 10^-3.The main paper relies primarily on top-eigenvalue and trace-based quantities.Source: §B.8 · p. 30

B.9 Patience ablation

The patience ablation tests stage stability in the aggressive S = 10 regime by varying patience in training-loss-triggered experiments.

B.9 Patience ablation

  • For aggressive S = 10, the training-loss-triggered experiments compare patience p ∈{1, 2, 5} while keeping other optimization and data settings identical to the main ResNet experiments.The passage describes the ablation setup but reports no stability results.Source: §B.9 · p. 30

B.10 MLP control experiment · B.11 Schedule sensitivity. · Disclosure of Generative AI Use

In the MLP control, progressive growth released flatter directions than one-shot unfreezing despite comparable predictive performance on the simple task. Forced schedules retained the qualitative curvature contrast, but their effects on accuracy and transition stability depended on schedule timing; the disclosure states that generative AI was used only for readability edits.

B.10 MLP control experiment

  • The controlled two-moons experiment used an overparameterized four-hidden-layer MLP of width 256, with 1200 training examples, 3000 test examples, and noise level 0.30.It compared full-model training, fixed-subspace training, one-shot unfreezing, and progressive growth.Source: §B.10 · p. 30
  • All methods reached comparable test accuracy of around 91.5%–91.7%, and interpolation barriers were nearly zero across growth variants.The results indicate that growth did not materially affect predictive performance in this simple setting, which did not induce strongly separated stage-wise solutions.Source: §B.10 · p. 30, Table 3 · p. 31

B.11 Schedule sensitivity.

  • Forced expansion schedules preserved the qualitative curvature contrast between one-shot and progressive growth, but could degrade final accuracy and increase interpolation barriers.The timing of constraint relaxation therefore mattered: transition stability and predictive performance were schedule-dependent.Source: §B.11 · p. 30

B.10 MLP control experiment

  • Restricted curvature was lower under progressive growth than one-shot unfreezing: λnew_max was 0.1691 ± 0.0938 for S = 5 and 0.1142 ± 0.0329 for S = 10, versus 0.5733 ± 0.1552.This supports the interpretation that the curvature bias is not specific to residual convolutional networks.Source: §B.10 · p. 30, Table 3 · p. 31

Disclosure of Generative AI Use

  • Generative AI was used only for grammar correction and stylistic editing, not to generate scientific claims, results, proofs, experiments, figures, or references.The authors produced and verified the technical content, interpretations, and conclusions, and take responsibility for the final manuscript.Source: Disclosure of Generative AI Use · p. 31

Related papers

Loading 2608.24568v1…