Across the Loss Landscape with Progressive Growth

Paul Caillon; Christophe Cerisara; Alexandre Allauzen

Source-linked AI summary

Across the Loss Landscape with Progressive Growth

Paul Caillon, Christophe Cerisara, Alexandre Allauzen

arXiv:2608.24568v1Aug 25, 2026cs.LGcs.AI

TL;DR

The paper asks how growth strategies bias optimization in complex loss landscapes and studies growth as progressive constraint relaxation. Its theory and experiments show that gradual subspace release favors flatter, more accessible solutions, while this geometric effect does not reliably improve test performance.

Problem
Flatness has a subtle, setting-dependent relationship with generalization, motivating a more precise account of how training dynamics bias which minima remain accessible.
Method
The paper models growth as nested affine-slice optimization, derives local ellipsoidal geometry and effective frozen-direction curvature, and tests the mechanism in toy landscapes and ResNet/CIFAR-100 experiments.
Results
Progressive growth biases accessibility toward flatter minima through compatibility volume, with toy and deep-network experiments supporting the geometric effect.
Takeaways & Limitations
Progressive growth can steer optimization geometry and alter explored regions even when its effect on final accuracy is limited.
Takeaways & Limitations
The study does not establish universality across architectures, optimizers, normalization schemes, or larger-scale settings, and its implementation offers no clear final-performance advantage over full-model training.

Abstract

from arXiv · show

Deep neural networks generalize well despite their highly nonconvex, overparameterized loss landscapes, a phenomenon often associated with the geometry of the minima found by stochastic optimization. We study how incremental grow-and-optimize strategies bias training toward flatter regions by viewing growth as progressive constraint relaxation. Starting from a low-dimensional submodel, we iteratively expand the trainable parameters by unlocking nested random subspaces while freezing the orthogonal complement at the network initialization, re-optimizing after each expansion until the full architecture is reached. Under standard local regularity conditions around non-degenerate minima, we prove that local sublevel sets are well approximated by ellipsoids and that basin accessibility under frozen constraints can be characterized by an explicit effective curvature in the frozen directions. This leads to an explanation of the bias: progressive growth increases the relative weight of wide basins and suppresses sharp ones through a volume effect induced by the frozen constraints. We empirically validate these predictions in controlled toy landscapes and in a realistic ResNet/CIFAR-100 setting and confirm that although progressive subspace growth reliably produces flatter solutions, curvature reductions do not universally translate into improved test performance, highlighting subtleties in the flatness-generalization connection. The code is available at https://github.com/p0lcAi/Across-the-Loss-Landscape.

1 Introduction and Related Work

The paper situates progressive growth within debates about why SGD finds generalizing solutions, emphasizing that flatness is a geometric training bias rather than a universal explanation of generalization. It builds on work showing that evolving parameter constraints and network growth can favor flatter optima.

SGD often reaches strong-generalizing solutions in high-dimensional, nonconvex networks, motivating geometric accounts based on flat minima and stochastic training dynamics.
Flatness is parameterization-dependent and only partially predicts generalization across metrics, architectures, and training regimes.
Recent evaluations therefore support treating flatness as a setting-dependent geometric bias of training dynamics rather than a universal generalization explanation.
Prior growth studies provided theoretical and empirical evidence that growing neural networks can converge to flatter optima than fixed-architecture counterparts.

2 Theoretical Analysis: Progressive Growth Biases Optimization Toward Flat Regions

The theory models progressive growth as optimization on nested affine slices whose frozen-coordinate compatibility depends on local curvature and initialization. Under local regularity, flatter directions yield larger accessibility volumes, producing an energy–entropy bias toward broad minima without claiming globally flattest selection.

2.1 Setup: progressive growth as optimization on affine slices: Progressive growth enlarges a trainable subspace through nested affine slices while keeping the orthogonal complement fixed, gradually relaxing optimization constraints.
2.2 Basin accessibility: A basin is accessible at a stage when the constrained slice comes sufficiently close to its minimum; frozen-coordinate mismatch can make it unavailable.
2.3 Local regularity and quadratic approximation: Under local quadratic regularity, basin sublevel sets are approximated by ellipsoids governed, up to constant factors, by the Hessian at the minimum.
2.4 Frozen coordinates induce an effective curvature: After re-optimizing active coordinates, the effective frozen-direction curvature measures the loss cost of frozen offsets and therefore the compatible constraint range.
2.5 Compatibility volume and local accessibility law: Compatible frozen-coordinate volume scales as det(Σ_i,t)^-1/2, so smaller effective curvature makes a basin accessible under more frozen configurations.
2.5 Compatibility volume and local accessibility law: With Gaussian initialization, accessibility combines proximity in frozen coordinates with breadth, creating an energy–entropy trade-off rather than a curvature-only rule.
2.6 Interpretation of the Theoretical Results: The mechanism predicts earlier selection and stage-to-stage retention of broad minima, along with low interpolation barriers between consecutive stage solutions.

3 Empirical Investigation

Experiments in toy landscapes and ResNet-18/CIFAR-100 show that progressive growth biases accessibility toward flatter regions, but this geometric effect does not improve final test accuracy.

Toy validation: Toy experiments isolate progressive growth's accessibility bias using synthetic quadratic basins under nested frozen-coordinate constraints.The study tests multiplicity and energy–entropy trade-off regimes with known basin geometry.
Toy validation: In the multiplicity regime, more growth stages monotonically increase selection of the top-5 flattest minima while decreasing selected-basin log det(H).Repeated freezing and release acts as a geometric filter favoring broad minima.
Toy validation: When sharp basins are closer to initialization, progressive growth instead selects nearby sharp basins, revealing an energy–entropy trade-off rather than unconditional flatness preference.The reversal appears in both flat-basin selection probability and selected-basin curvature.
Deep-learning experiments: On ResNet-18/CIFAR-100, progressive growth does not improve final test accuracy over the full-model baseline and generally reduces it as stages increase.Validation-accuracy growth falls from 74.99% at S = 3 to 73.94% at S = 10; training-loss growth falls from 75.54% to 73.37%.
Deep-learning experiments: Moderate schedules often preserve the current broad low-loss region, with validation-accuracy growth retention increasing from 0.83 at S = 3 to 0.96 at S = 10.The mild training-loss schedule with S = 3 has an essentially zero mean interpolation barrier and retention of 1.0.

4 Discussion and Limitations

Progressive growth reshapes optimization geometry by favoring minima that remain accessible under successive frozen-coordinate constraints. This geometric bias can produce flatter solutions without reliably improving accuracy, and its practical effects depend on the growth schedule.

Progressive growth acts as a sequence of constraint relaxations that favors minima compatible with many frozen-coordinate configurations.The mechanism is framed as an accessibility bias induced by the geometry of frozen constraints.
The local mechanism combines effective frozen-direction curvature with an initialization-dependent location term.The effective curvature is obtained through a Schur-complement reduction.
In toy landscapes, flatter minima gain selection weight when basin locations are balanced, but the bias can reverse when sharper basins are sufficiently closer to initialization.This reflects an energy–entropy trade-off between basin breadth and proximity to initialization.
In ResNet/CIFAR-100, moderate schedules preserve consecutive stage solutions, maintain small interpolation barriers, and make newly released directions flatter than already active ones.These geometric effects do not appear as a gain in final accuracy.
Growth should therefore be understood as shaping the training trajectory rather than as an unconditional recipe for improving accuracy.The distinction separates accessibility bias from predictive performance.
The study does not establish the effect's universality across architectures, optimizers, normalization schemes, or larger-scale settings.Its curvature estimates are local proxies, and fragmented schedules can degrade final performance in the reported ResNet experiments.
Model growth can be viewed as a tool for filtering minima that are fragile under partial freezing and favoring regions robust to successive constraint relaxations.This reframes expansion from capacity management toward optimization-geometry control.

5 Conclusion

The paper treats progressive growth as a geometric intervention that changes which minima remain accessible during nested constraint relaxation. Theory and experiments support a flatter-minimum accessibility bias, while deep-network accuracy effects remain limited and schedule-dependent.

Progressive growth constrains training to nested affine slices, changing which minima remain accessible as constraints are relaxed.The analysis describes growth as more than a change in model capacity.
Local analysis identifies an energy–entropy trade-off: minima are favored when they are both initialization-compatible in frozen coordinates and broad along those directions.This yields a precise accessibility-based interpretation of flatness.
Toy landscapes show stronger selection of flat minima with balanced basin locations and a predicted reversal when sharper basins lie sufficiently closer to initialization.The experiments test both sides of the theoretical trade-off.
ResNet/CIFAR-100 exhibits the mechanism through transition geometry: moderate schedules preserve stage solutions, barriers remain small, and newly released directions are flatter.These effects are observed through geometry rather than final accuracy.
One-shot unfreezing recovers most of the full-model accuracy but lacks the same low-curvature released directions, while fixed-subspace training fails on CIFAR-100.Together, these controls support gradual constraint relaxation as the source of the geometric bias.

A Proofs for Section 2

The appendix proves the local geometric results under regularity assumptions, including ellipsoidal sublevel-set approximations and effective curvature under frozen constraints. The accessibility law additionally requires a local density condition for frozen offsets.

The proof framework fixes a basin and growth stage, using local Hessian regularity assumptions around a non-degenerate minimum.The assumptions include a locally Lipschitz Hessian.
Proposition 2 establishes a local ellipsoidal approximation for sublevel sets under the stated assumptions.The result introduces positive constants controlling the approximation.
Theorem 3 characterizes frozen-constraint curvature through the Schur complement of the Hessian block matrix.The Schur complement is taken in the active and frozen coordinate blocks.
Theorem 4 gives a local accessibility law under an additional continuous-density assumption for the frozen offset near zero.The density is conditional on the stage subspaces.
The local volume calculations use κ_pt, defined as the volume of the unit ball in R^pt.
The derivation begins with a second-order Taylor expansion with cubic remainder and applies it to reduced coordinates.The appendix uses the composition with the orthogonal active/frozen coordinate map.

A.2 Proof of Proposition 2

The proof of Proposition 2 bounds the Taylor remainder so the loss remains comparable to its quadratic model near a strict local minimum. An annulus argument then confines relevant points to the local neighborhood where the approximation applies.

The proof first establishes that, for sufficiently small radii, the loss is comparable to its quadratic model.This is the central local approximation step.
The radius is chosen using the Hessian's smallest curvature and the local Lipschitz constant to control the cubic remainder.The bound converts the cubic term into a fraction of the quadratic form.
The strict local minimum yields a strictly positive loss gap on an annulus surrounding the local neighborhood.Continuity provides the positive minimum used in the confinement argument.
Any point relevant to the local sublevel set must lie inside the chosen radius, because points in the annulus have loss at least the positive gap.This permits application of the local Taylor bound.
Orthogonal active and frozen coordinates preserve Euclidean norms and transfer the local Hessian-Lipschitz bound to the reduced function.The Hessian is written in block form in the reduced coordinates.
Positive definiteness of the original Hessian implies positive definiteness of the active block and its Schur complement.This guarantees the local quadratic structure needed for the reduced analysis.
The reduced-coordinate Taylor lemma supplies the local expansion used to complete the proposition's ellipsoidal bounds.

A.4 Proof of Theorem 3

The proof locally solves the active coordinates as a function of frozen coordinates, then compares the resulting compatibility set with an ellipsoid defined by effective curvature. Under strict local minimality and sufficiently small sublevel thresholds, the two sets coincide up to controlled multiplicative slack.

Local expansion: The implicit function theorem expresses active coordinates a as a unique C1 function of frozen coordinates b near the minimum.Continuity of a allows the neighborhood to be shrunk while preserving the local representation.
Local expansion: Substituting a(b) into the local quadratic expansion yields an effective frozen-direction curvature involving Hbb − HbaHaa^-1Hab.The resulting quadratic form is the Schur-complement curvature after optimizing locally over active coordinates.
Conclusion: The proof concludes that the local compatibility geometry is approximated by the effective-curvature ellipsoid with ε-independent constants.The final statement follows from the preceding inclusions and remainder control.
Ellipsoidal approximation: The compatibility set is compared against the ellipsoid associated with Σi,t through upper and lower quadratic-form inclusions.The proof establishes both inclusions using remainder bounds and sufficiently small ε.
Ellipsoidal approximation: Strict local minimality ensures that sufficiently small compatibility sets remain inside the neighborhood where the local estimates apply.A positive minimum on a compact sphere provides the localization step.

A.6 Proof of Theorem 4

The proof derives an asymptotic accessibility law by combining the local ellipsoidal approximation with the frozen-coordinate initialization density. In the isotropic Gaussian case, accessibility favors minima that are both close to initialization in frozen directions and broad there.

Accessibility asymptotics: Conditioning on the selected subspace, the frozen-coordinate initialization has a local density near the target minimum.The density is continuous at the origin, enabling a small-set asymptotic approximation.
Accessibility asymptotics: Combining the local geometry and density estimates yields the theorem’s asymptotic accessibility law.The proof explicitly identifies the limiting scaling of the compatibility probability.
Gaussian specialization: Under isotropic Gaussian initialization, the accessibility expression specializes to an explicit dependence on frozen-direction displacement and effective curvature.The Gaussian assumption supplies the closed-form density factor used in the specialization.
Gaussian specialization: The resulting energy–entropy tradeoff favors minima close to initialization in frozen coordinates and broad along those directions.Distance contributes through initialization proximity, while breadth contributes through the curvature-dependent volume term.

B.1 Transition metrics

The experiments quantify stage-to-stage behavior using interpolation, endpoint, retention, leakage, and restricted-curvature diagnostics. Progressive growth is implemented by freezing initialization values, releasing nested parameter groups, and re-optimizing under validation- or training-loss-based triggers.

Interpolation diagnostics: The interpolation barrier is the maximal excess validation loss along the linear path between consecutive stage solutions, relative to the worse endpoint.The loss is evaluated at equally spaced interpolation points using a fixed validation-batch budget.
Transition classification: Retention records whether a transition remains below a fixed barrier threshold, while leakage is its binary complement.These metrics classify whether the transition stays within the retained low-barrier regime.
Curvature diagnostics: Restricted Hessian diagnostics estimate top eigenvalue, trace, and optionally log-determinant on active and newly released parameter subspaces.Masked Hessian-vector products project both input and output directions onto the selected coordinates.
Experimental design: Deep-learning evaluations use CIFAR-100 ResNet experiments with transition metrics computed between pre-expansion and post-expansion checkpoints.The transition protocol includes interpolation barrier, retention, leakage, endpoint gap, and restricted curvature.
Experimental design: The experimental controls compare full-model, fixed-subspace, one-shot-unfreezing, and progressive-growth training to separate constraint and release schedules.All methods begin from the same initial constrained submodel where applicable, while progressive growth distributes release across multiple transitions.
Growth procedure: Progressive growth activates nested tensor groups while clamping frozen parameters, buffers, gradients, and matching optimizer states at initialization.The procedure releases groups after patience-based non-improvement of validation accuracy or negative training loss, then computes transition diagnostics.

B.9 Patience ablation

For the aggressive S = 10 regime, the study examines how patience affects stage stability in training-loss-triggered experiments.

B.9 Patience ablation: Patience values p ∈{1, 2, 5} are evaluated while all other optimization and data settings remain identical to the main ResNet experiments.The ablation focuses on stage stability under the aggressive S = 10 regime.

B.10 MLP control experiment

A controlled two-moons MLP experiment finds comparable predictive performance across training strategies but substantially lower restricted curvature under progressive growth. The curvature contrast persists beyond convolutional residual architectures, while schedule timing affects accuracy and interpolation barriers.

Experimental setup: The experiment compares full-model training, fixed-subspace training, one-shot unfreezing, and progressive growth in an overparameterized four-layer MLP.The task uses 1200 training examples, 3000 test examples, noise level 0.30, and hidden-layer width 256.
Predictive performance: All methods reach comparable test accuracy, around 91.5%–91.7%, while growth variants exhibit nearly zero interpolation barriers.These results indicate little predictive-performance separation in this simple task.
Curvature effect: Restricted curvature is substantially lower after progressive growth than after one-shot unfreezing: 0.1691 ± 0.0938 for S = 5 and 0.1142 ± 0.0329 for S = 10 versus 0.5733 ± 0.1552.The reported λnew max values show flatter directions released by progressive growth.
Schedule sensitivity: Forced expansion schedules preserve the qualitative curvature contrast but can degrade final accuracy and increase interpolation barriers.Thus, the qualitative curvature bias is robust, whereas transition stability and predictive performance depend on relaxation timing.