PaperlayerView exact revision on arXiv

Source-linked AI summary

When the Preconditioning Exponent Turns Negative: Learning-Rate Coupling and Cross-Environment Generalization

Gongyue Zhang, Honghai Liu

arXiv:2609.30271v1cs.LG

TL;DR

The paper studies how a preconditioning exponent interacts with learning rate to allocate a classifier across stable, spurious, and noise features under environmental change. In a paired four-environment experiment, the cross-environment-optimal exponent declines as learning rate rises, while source validation favors a systematically different regime.

  • Problem

    How the preconditioning exponent interacts with learning rate to allocate a classifier across stable, spurious, and noise features under distribution shift is less understood.

    Source: §1 · pp. 1–2

  • Method

    The study scans optimizer settings in a paired four-environment classification problem, selects checkpoints by source validation, and evaluates them across environments.

    Source: §1 · p. 2

  • Results

    The cross-environment-optimal exponent decreases nearly linearly with log10 η, while source validation remains systematically higher than the cross-environment optimum.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Negative p is a high-learning-rate allocation regime associated with reduced relative reliance on spurious and noise features, not a universally better setting.

    Source: §1 · p. 2, §8 · p. 11

    The evidence is limited to an interpretable linear model and synthetic features, with a finite training horizon that leaves transient allocation versus asymptotic bias unresolved.

    Source: §7 · p. 10

Abstract

from arXiv · show

Adaptive optimizers are commonly parameterized by a fixed power of the second-moment estimate. Existing partially adaptive methods study exponents between momentum-like updates and the standard Adam square root, while the interaction between this exponent and the global learning rate is less understood. We perform a controlled cross-environment study using a paired four-environment classification problem with stable sparse features, environment-dependent spurious sparse features, dense features, and high-dimensional noise. Across \NumRuns{} source-training runs covering 21 preconditioning exponents $p\in[-0.5,0.5]$ and five learning rates $\eta\in[10^{-4},10^{-2}]$, we find that the exponent maximizing cross-environment accuracy decreases almost linearly with $\log_{10}\eta$. The fitted slopes range from $-0.270$ to $-0.300$, with $R^2$ between $0.972$ and $0.996$. At $\eta=10^{-2}$, source-validation selection still prefers positive exponents in all four environments, whereas cross-environment and worst-environment criteria prefer negative exponents. Checkpoint decomposition shows that lower $p$ reduces the learned spurious-to-stable and noise-to-stable weight ratios; under reversed correlation, it also reduces the magnitude of the harmful spurious margin. Negative $p$ is therefore not a universally optimal setting. It is a high-step-size allocation regime produced by the joint action of learning rate and preconditioning. The study also exposes a model-selection conflict: source-domain validation systematically selects a different preconditioning regime from the one that maximizes robustness to environmental change. The results are a single-seed, finite-budget mechanism study rather than a broad benchmark claim.

1 Introduction

The paper asks how the preconditioning exponent and learning rate jointly shape feature allocation and cross-environment generalization. In a paired four-environment study, it finds that the robust exponent falls as learning rate rises, while source-validation selection favors a different regime.

1 Introduction

  • The question arises because learning rate changes trajectories and future moment estimates, so the same exponent can yield different effective coordinate gains across learning rates.The paper asks where the cross-environment optimum lies as the two controls interact.Source: §1 · pp. 1–2
  • The controlled design pairs four environments that differ only in the spurious block’s label correlation, isolating optimizer-dependent feature allocation from unrelated dataset variation.Checkpoints are selected by source validation and evaluated on all four test environments.Source: §1 · p. 2
  • The cross-environment-optimal exponent decreases approximately linearly with log10 η, and the negative-p regime emerges at large learning rates rather than being universally better.The study describes this as a learning-rate-dependent reversal of the preferred exponent.Source: §1 · p. 2
  • Source-validation-optimal and cross-environment-optimal exponents are systematically separated, creating an optimizer model-selection conflict under distribution shift.The study compares source validation with cross-environment criteria after selecting checkpoints using source validation.Source: §1 · p. 2
  • Lower and negative p reduce relative reliance on spurious and noise coordinates in the high-learning-rate regime.The paper examines this allocation through learned linear weights and signed margins.Source: §1 · p. 2
  • The study uses one random seed, one affine classifier, and a fixed training budget; it establishes a controlled phenomenon rather than a universal real-world benchmark improvement.Source: §1 · p. 2

2 Related Work

Prior work studies adaptive optimization, learning-rate effects on feature learning, and domain generalization. This paper distinguishes itself by scanning negative exponents and treating exponent and learning rate as coupled controls.

2 Related Work

  • Prior work links adaptive optimizer choices to different generalization properties and studies partial adaptivity, while this study examines exponent–learning-rate coupling under distribution shift.Source: §2 · pp. 2–3
  • Unlike earlier partial-adaptivity work, this study scans exponents into the negative region and evaluates the exponent jointly with the global learning rate.Source: §2 · p. 3
  • The study complements work on large-learning-rate feature learning by finding that the robust exponent shifts with learning rate and can become negative.Source: §2 · p. 3
  • Rather than proposing a domain-generalization objective, the experiment measures how a source-only optimizer reallocates a fixed linear model across stable, spurious, and noise feature blocks.Source: §2 · p. 3
  • The paper applies an information-allocation perspective to feature blocks, relating preconditioning-dependent coordinate update mass to environment-dependent consequences.Source: §2 · p. 3

3 Learning-Rate–Exponent Coupling

The paper formulates adaptive updates with a continuous preconditioning exponent and gives a local compensation argument for its interaction with learning rate. It distinguishes source-validation selection from oracle cross-environment criteria.

3.1 Generalized preconditioning

  • Adam-style moving averages and bias corrections define the generalized update, while p directly changes coordinate allocation and learning rate changes later gradients, moments, and ratios across steps.The learning rate cancels from the same-step coordinate update ratio.Source: §3.1 · p. 3
  • For p < 0, low-second-moment coordinates are suppressed rather than amplified, reversing the usual Adam ordering without implying that every high-moment coordinate is useful.Source: §3.1 · p. 3

3.2 A local compensation relation

  • Under the local assumption that effective gain stays near a constant, 0 < s < 1 predicts that increasing η reduces the exponent needed for comparable updates.The paper says this is a sign prediction, not a full dynamical theory, because s changes along the trajectory and across coordinates.Source: §3.2 · p. 4
  • The local compensation relation explains why the preferred exponent may pass below zero as learning rate rises, while remaining a testable directional prediction rather than a full trajectory model.Source: §3.2 · p. 4

3.3 Source and cross-environment optima

  • Source-validation, worst-environment, and cross-environment optima are distinguished; the latter two are oracle analysis criteria, not deployable source-only selection rules.Source: §3.3 · p. 4

4 Controlled Cross-Environment Experiment

The experiment pairs four environments to isolate a spurious-correlation shift, scans a linear classifier over optimizer settings, and diagnoses feature allocation through block weights and signed margins.

4.1 Paired environments

  • Four paired environments share labels, dense and stable sparse features, masks, amplitude noise, and pure-noise coordinates; only the spurious block’s label correlation changes.The correlations are 0.9, 0.5, 0, and −0.5, making cross-environment differences attributable to the intended shift rather than resampling noise.Source: §4.1 · pp. 4–5

4.2 Model and optimization grid

  • The study trains a single affine binary classifier over 21 exponents and five learning rates, selecting each checkpoint only by source validation loss before evaluating it on all four test environments.This yields 420 runs and 1680 cross-evaluations; every run uses seed 42 and 50 epochs.Source: §4.2 · p. 5

4.3 Feature-allocation diagnostics

  • Feature-allocation diagnostics decompose learned block weights and mean signed test margins, with “spur” denoting a spurious component that opposes the correct label.Source: §4.3 · p. 5

5 Results

Across the four source environments, the cross-environment-optimal exponent shifts lower as learning rate increases, while source validation selects higher exponents than robust criteria. At high learning rates, lower exponents reduce relative spurious and noise allocation, but the findings characterize a finite-budget setting rather than a universal optimum.

5.1 The cross-environment-optimal exponent decreases with learning rate

  • The cross-environment-optimal exponent shifts from positive toward negative as learning rate rises, with fitted slopes of −0.270 to −0.300 and R2 of 0.972 to 0.996.The approximately linear pattern is consistent across the four shifts, but the paper does not claim an exact universal law.Source: §5.1 · pp. 5–6, Figure 1 · p. 6

5.2 Source validation and robust selection disagree

  • At η = 10−2, source validation selects positive p in every environment, while mean cross-environment and worst-environment criteria select negative p.The ID–cross gap ranges from 0.20 to 0.40.Source: §5.2 · p. 6, Table 2 · p. 7, Figure 2 · p. 8

5.4 Lower p reduces relative spurious and noise allocation

  • At the weak-positive cross optimum p = −0.20, the spurious-to-stable ratio is 0.381 versus 0.435 at p = 0, and the noise-to-stable ratio is 0.054 versus 0.066.For the reversed source, the spurious-to-stable ratio falls from 0.447 to 0.368; the strong-positive source shows only a small cross-environment improvement, so the effect is not generic shrinkage of all spurious coordinates.Source: §5.4 · p. 7

5.5 Margin decomposition under correlation reversal

  • Under correlation reversal, the stable sparse block contributes positive margin and the spurious block negative margin; increasing p grows both contributions, placing total-margin maximization at lower p than source fitting alone would select.Lower p yields a smaller harmful spurious component relative to the stable component, although both feature blocks are sparse and the optimizer is not given their semantic distinction.Source: §5.5 · pp. 7–8

5.6 Finite-budget character of the result

  • In 92.38% of the 420 runs, the best source-validation checkpoint occurs at epoch 50, so the experiment primarily characterizes finite-budget feature allocation rather than a unique asymptotic solution.A longer training horizon may narrow some differences or move the optimum.Source: §5.6 · p. 8

5.3 Negative p is a high-learning-rate regime, not a universal optimum

  • At η = 10−2, negative p improves robustness over the best nonnegative setting, but excessively negative p underfits useful structure.Cross-environment gains are small for strong-positive and neutral sources, but reach 0.35 percentage points for the weak-positive source and 0.87 points for the reversed source; the reversed-source worst-environment gain is 2.48 points.Source: §5.3 · pp. 6–7, Figure 3 · p. 9

6 Discussion

In the controlled sparse-feature setting, larger learning rates shift the robust allocation toward lower p, sometimes across zero. But p and learning rate remain distinct controls, and source validation can favor a different regime from environmental robustness.

6.1 Why can the robust optimum become negative?

  • In this controlled sparse-feature regime, increasing the learning rate shifts the robust allocation toward lower p and crosses zero at sufficiently large η.Lower p counteracts amplification of weakly activated or inconsistent coordinates, but the mapping depends on feature and training conditions.Source: §6.1 · p. 9

6.2 Learning rate and p are not interchangeable

  • Although p and η can produce similar aggregate performance, p immediately changes coordinate ratios while η scales the update and changes the next parameter state.Table 3 compares the best negative and nonnegative exponents for mean cross-environment accuracy at η = 10−2.Source: §6.2 · pp. 9–10, Table 3 · p. 10

6.3 Optimizer model selection under distribution shift

  • Source-validation selection can prefer a positive exponent even when the robust optimum is negative, and no deployable source-only selection rule is proposed.Target environments are used only as retrospective diagnostic oracles in this study.Source: §6.3 · p. 10

6.4 Relation to partially adaptive and root-free methods

  • The useful exponent is task-, learning-rate-, and criterion-dependent; the negative regime appears here under cross-environment robustness, a large global step, and sparse high-dimensional feature competition.This result does not contradict partially adaptive or root-free methods, which examine different exponent regimes and settings.Source: §6.4 · p. 10

7 Limitations

The study’s conclusions are bounded by its single-seed, synthetic linear-model design, finite training horizon, oracle robust-selection criteria, and single optimizer family.

Single seed

  • The paired construction reduces sampling variation but does not replace a multi-seed estimate of optimizer variability.The paper proposes repeating a reduced grid across several seeds as a validation step.Source: §7 · pp. 10–11

Linear model and synthetic data

  • The interpretable linear model and synthetic data do not establish that the result applies to deep representations or natural datasets.The feature blocks are directly interpretable, enabling the paper’s mechanism analysis.Source: §7 · p. 10

Finite training horizon

  • Most checkpoints are selected at the final epoch, so longer-horizon and matched-source-loss experiments are needed to distinguish transient allocation from asymptotic implicit bias.Additional long-horizon experiments are also proposed as validation.Source: §7 · pp. 10–11

Oracle robust selection

  • Cross- and worst-environment selection use target test environments retrospectively, and the study proposes no source-only rule for selecting negative p.A source-observable selection criterion remains outside the study’s scope.Source: §7 · pp. 10–11

One optimizer family

  • The experiment varies the exponent within one Adam-style optimizer family; changes to momentum, moment decay, clipping, weight decay, or schedules may alter the coupling.The proposed validation path includes testing whether source-observable allocation statistics predict the robust exponent.Source: §7 · p. 11

8 Conclusion

In a controlled four-environment sparse-feature problem, the cross-environment-optimal exponent decreases nearly linearly as learning rate rises, becoming negative at high rates even as source validation prefers positive exponents. The results link lower p to reduced spurious and noise allocation and a smaller harmful spurious margin under correlation reversal, but do not establish negative p as universally superior.

Conclusion

  • In the four-environment study, the cross-environment-optimal exponent decreases nearly linearly with log learning rate, while source validation continues to prefer positive exponents at high rates.The study frames negative p as a high-step-size allocation regime, not a universally better setting.Source: §8 · p. 11
  • The useful exponent depends on step-size regime and whether optimization is judged by source fit or environmental robustness.The paper characterizes its evidence as a controlled mechanism study rather than a broad benchmark claim.Source: §8 · p. 11
  • On the reversed test environment, lower p reduces the harmful spurious margin while the stable sparse contribution remains positive.The margin decomposition concerns models trained on strong- and weak-positive sources at η = 10−2.Source: §8 · p. 11, Figure 5 · p. 12

B Source-Selected Cross-Environment Matrix

Table 5 reports test accuracy for the source-validation selection protocol, not the oracle robust result.

B Source-Selected Cross-Environment Matrix

C Reproducibility and Data Files

The accompanying source package provides training and analysis scripts, run and evaluation records, derived metrics, and figure and table files. Its configuration and analysis CSVs document the experiment defaults and main analyses.

C Reproducibility and Data Files

  • The package includes the original training script, data-preparation and checkpoint-decomposition script, a 420-run manifest, and all 1680 cross-environment evaluations.Source: C Reproducibility and Data Files · p. 12
  • Enriched run-level metrics include feature-block weight statistics and margin decompositions, alongside compact CSVs used for each figure and table and vector PDF figures generated from those CSVs.Source: C Reproducibility and Data Files · p. 12
  • Experiment defaults are stored in data/fixed config.json, with four named CSV files covering optimal p by learning rate, linear fits, high-learning-rate p curves, and run metrics with feature allocation.Source: C Reproducibility and Data Files · p. 12
Loading 2609.30271v1…