Source-linked AI summary
The Conflict Between Logic and Memory: Training Conditions for Optimizer-Dependent Rule Acquisition
Gongyue Zhang, Honghai Liu
TL;DR
The paper asks which training conditions determine whether optimizers learn reusable relations rather than merely fitting familiar examples. It studies shallow ReLU networks with structured tasks and parameter-level interventions. Nuisance-connected trainability changes shared failures and relative optimizer advantages, while strict parity reveals optimizer separation followed by a shared finite-budget loss of generalization.
Problem
The paper asks which reusable target relations different optimizers learn and which training conditions determine whether their observed advantages persist.
Source: §1 · pp. 1–2Method
Results
Nuisance-connected trainability reshapes shared failures and optimizer advantages; fixing nuisance-connected weights raises fourth-stage accuracy from approximately 50% to 70.6%, 68.5%, and 69.0% for SGDM, Adam, and Muon.
Source: §1 · p. 2, §5 · p. 19Takeaways & Limitations
Permitted adaptation is a training variable that changes which predictive functions a fixed architecture learns.
Source: §3 · p. 5, §5 · p. 19Muon selects its lowest tested rate, 0.005, at orders seven through twelve, so its deficit in the transition region is not a comparison after exhaustive tuning.
Source: §4.4 · p. 17
Abstract
from arXiv · showhide
Optimizers can fit the same task while acquiring different generalizing relations. We study the training conditions governing these differences in single-hidden-layer ReLU networks, combining composite evidence tasks, parameter-level interventions, and a three-seed strict-parity scan. Our central finding is that nuisance-connected trainability reshapes both shared failures and relative optimizer advantages. In a nuisance-heavy task, all twenty tested optimizer configurations remain near chance on the hardest stage. Retaining every input but fixing nuisance-connected first-layer weights at initialization raises that stage's accuracy from approximately 50\% to 70.56\%, 68.47\%, and 69.00\% for momentum SGD, Adam, and Muon. Masking the same inputs only after full training does not recover this performance. On a separate pairwise-mode task, background freezing reduces Muon's rare-mode advantage over momentum SGD by 12.48 percentage points, while the target and mode frequencies remain fixed. Each intervention is evaluated under a common validation-selection protocol with condition-specific learning rates and checkpoints. A strict-parity sweep over orders 1--20 provides a complementary reference without spurious cues or extra nuisance coordinates: the optimizers separate at orders 9--11, then approach chance despite substantial remaining Bayes predictability. A mixed task establishes a recovery boundary, and CIFAR-10 supplies an external architecture comparison. Together, these findings connect optimizer comparison to the acquisition and use of specified relations, identifying permitted adaptation as a concrete training variable that changes what a fixed architecture learns.
1 Introduction
The paper asks which training conditions let shallow networks learn reusable target relations, comparing optimizers across structured tasks and controlled parameter-update interventions. It finds that nuisance-connected trainability changes shared failures and optimizer advantages, while strict parity reveals a separate finite-budget boundary.
1 Introduction
- The motivating question is which reusable target relations different optimizers learn and which training conditions determine whether their advantages persist.The paper uses one-hidden-layer ReLU networks with known data-generating relations, distinguishing sample-specific fitting from generalization of reusable relations.Source: §1 · p. 2
- Fourth-stage accuracy rises from approximately 50% to 70.6%, 68.5%, and 69.0% for SGDM, Adam, and Muon when nuisance-connected first-layer weights are fixed.Masking the same nuisance inputs only after full training leaves accuracy near chance; the rare-mode advantage of Muon over SGDM also disappears after background connections are frozen.Source: §1 · p. 2
- At orders thirteen to twenty, all three optimizers approach chance while the Bayes reference remains approximately 99.1%–90.5%.They separate at orders nine to eleven, with no lower-order predictive entry point in strict parity.Source: §1 · p. 2
- Across 20 optimizer configurations, the hardest stage of the nuisance-heavy task remains near chance, showing that aggregate optimizer comparisons can conceal unresolved stage-level failure.The paper also notes that aggregate scores can understate rare-mode improvement.Source: §1 · pp. 2–3
- The paper connects optimizer comparisons to acquisition and use of specified relations, using experimentally testable training conditions to examine outcomes within the same shallow architecture.Its contributions cover nuisance-connected trainability, divergence between overall and relation- or mode-level performance, and finite-budget higher-order learning profiles.Source: §1 · pp. 2–3
2 Related Work
Related work connects optimizer choice to learned predictors, feature competition, training interventions, and higher-order learning. This paper extends those comparisons to specified target relations and the conditions under which optimizer-dependent outcomes persist.
2.1 Optimizer bias and the acquisition of specified relations
- Prior work links optimizer choice to different predictors and feature properties; this study asks whether advantages persist when background connections cannot adapt.It distinguishes strict parity, where every proper subset of independent latent factors is label-uninformative, from staircase tasks with a low-order entry point.Source: §2.1 · p. 3
2.2 Feature competition, spurious cues, and nuisance fitting
- The experiments compare update rules through cue reversal, stage- and mode-specific evaluation, and training interventions, finding both recoverable shared failure and optimizer-dependent responses.The mixed task provides a contrasting case in which designated nuisance removal is insufficient.Source: §2.2 · pp. 3–4
2.3 Training interventions, representations, and readout
- The intervention separates nuisance exposure, adaptation of first-layer connections, and retaining nuisance contributions at inference.Freezing preserves initial nuisance projections while preventing selected weights from changing; post-training masking does not reproduce the training-restriction effects.Source: §2.3 · p. 4
2.4 Higher-order structure, expressivity, and finite-budget learning
- Strict parity tests higher-order learning without a lower-order entry point, but its results are specific to the task family, architecture, search, and stopping protocol.At fixed total signal norm, increasing interaction order also reduces signal allocated to each factor; a high Bayes reference indicates available information, not that the fixed-width network can attain it.Source: §2.4 · pp. 4–5
3 Optimizer-Dependent Rule Acquisition
The section frames optimizer comparison around the functions and target relations acquired through training, evaluated under specified tasks and adaptation constraints. It distinguishes fitting, generalization, and cue dependence rather than treating accuracy as a unique readout of internal representation.
3 Optimizer-Dependent Rule Acquisition
- The study holds the forward architecture fixed within comparisons and varies the update rule or which connections may adapt to test how training conditions shape acquired functions.Task structure specifies available evidence, functional diagnostics locate successes and failures, and interventions test which conditions outcomes depend on.Source: §3 · p. 5
3.1 From parameter updates to acquired behavior
- A validation-selected predictor depends on the optimizer, training constraint, hyperparameters, initialization and minibatch schedule, and checkpoint and learning-rate selection.The section distinguishes fitting at a fixed checkpoint from generalization to a specified evaluation distribution; rule acquisition is operationally defined as resulting generalizing behavior.Source: §3.1 · p. 5
3.2 Target evidence, spurious cues, and independent nuisance
- Inputs are divided into target observations, spurious cues, and independent nuisance; spurious cues correlate with labels in training but approach chance agreement at test.Nuisance is independent of target factors, labels, and spurious cues, while sample-specific associations with background remain possible.Source: §3.2 · p. 6
3.3 Composite evidence and strict interaction order
- Composite tasks vary evidence organization and cue competition, while strict parity requires all factors jointly and includes no spurious cue or extra nuisance coordinate.Composite-task stages are diagnostic information categories, not necessarily the computation performed by a trained network; strict-parity order denotes a joint information requirement.Source: §3.3 · pp. 6–7
3.4 Update rules and exposure–adaptation interventions
- The compared optimizers use distinct update rules, while freezing selected first-layer connections preserves their initial projections but prevents their weights from adapting.The intervention retains all inputs and constrains the trajectory; test-time masking instead changes the input to an already trained predictor.Source: §3.4 · p. 8, Table 2 · p. 8
3.5 Functional measurements and optimizer–constraint comparisons
- Group accuracies expose stage- or mode-specific outcomes, and paired contrasts distinguish whether a constraint changes one optimizer’s result or the gap between optimizers.Low reflection sensitivity is meaningful alongside high target accuracy, since a constant predictor would also have zero sensitivity.Source: §3.5 · pp. 8–9
3.6 Architectural capacity and access to useful updates
- Interaction order and matrix rank measure different properties, while nuisance may affect activation gates and prediction-dependent errors that shape target-coordinate updates.These are candidate pathways, not established mediators; architectural references do not establish noisy-task Bayes attainability or a successful optimization path.Source: §3.6 · p. 9
3.7 Selection, pairing, and reporting
- The protocols use validation-selected learning rates and checkpoints, paired configurations within seeds, and test scores only for reporting, not selection.CIFAR-10 is a one-seed, fixed-learning-rate comparison; synthetic summaries report three-seed means and sample standard deviations, not statistical significance.Source: §3.7 · pp. 9–10
4 Experiments and Discussion
The experiments compare optimizer-dependent relation learning across composite tasks, strict parity, and CIFAR-10, then intervene on which connections can adapt. Restricting nuisance-connected adaptation recovers a shared fourth-stage failure and changes optimizer comparisons, while other tasks reveal persistent limits and architecture-dependent outcomes.
4.1 Experimental scope and reporting protocol
- The experiments use composite tasks, strict parity, an expanded optimizer comparison, and CIFAR-10; condition-specific rates and checkpoints are selected using validation accuracy, with loss breaking ties.A prior fixed-condition contrast found AdamW rose from 44.73% to 95.07% overall when independent-noise connections were frozen, while test-only zeroing gave 48.38%.Source: §4 · p. 10, §4.1 · p. 10, §4.2 · p. 12
4.2 Nuisance-connected trainability changes rule generalization
- Freezing nuisance-connected weights raises fourth-stage accuracy from 50.24%, 50.40%, and 49.86% to 70.56%, 68.47%, and 69.00% for SGDM, Adam, and Muon, respectively.All inputs remain present; test-time masking alone leaves accuracy near 50%.Source: §4.2 · p. 11, Table 4 · p. 11, Figure 1 · p. 12
4.3 Optimizer advantages depend on the interference conditions
- Cue reliance varies by task: Muon’s cue-conflict accuracy is 24.72% in the vote task versus 83.44% in the second-order mode task, while the mixed task remains near chance on stage four after non-target inputs are removed.The vote and mode tasks also differ in cue agreement, signal organization, and nuisance structure; the mixed-task comparison does not identify the separate effects of its lower stage frequency and weaker target signal.Source: §4.3 · pp. 13–14, Table 9 · p. 15
- Background freezing reduces Muon’s rare-mode gap over SGDM from +11.99 to −0.49 pp, a 12.48 pp reduction, without changing the target relation or mode frequency.The frozen gaps show no consistent seed-wise reversal in favor of SGDM.Source: §4.3 · p. 14, Table 8 · p. 14
4.4 Strict parity separates available information from achieved learning
- At orders nine through eleven, SGDM leads Adam in each reporting seed; at order ten, SGDM, Adam, and Muon score 84.78%, 78.14%, and 74.96%.All three methods exceed 99.9% through order six; the transition occurs under a fixed shallow architecture and budget.Source: §4.4 · p. 16, Table 10 · p. 16
- At orders thirteen through twenty, all methods remain within 0.57 pp of chance while the Bayes reference falls from approximately 99.09% to 90.46%.The gap does not distinguish insufficient representation, training time, or unfavorable update dynamics; Muon’s lower-rate regime also remains untested.Source: §4.4 · pp. 16–17
4.5 Expanding the optimizer set leaves the fourth-stage bottleneck
- Across twenty optimizer configurations, higher-order-task fourth-stage means remain 49.86–50.53%, while nuisance-adaptation intervention reaches 68.47–70.56% with full inputs retained.The expanded optimizer set improves some overall scores without recovering the hardest stage.Source: §4.5 · p. 17
4.6 CIFAR-10 provides an external architecture comparison
- On CIFAR-10, VGG13-BN reaches 93.47% with convolutional Muon and an SGDM or PAdam head, versus 93.19% for SGDM; the head intervention’s direction differs by architecture.The comparison does not isolate which architectural change produces that dependence, and the image results use one seed without a learning-rate sweep.Source: §4.6 · p. 18, Table 11 · p. 18
5 Conclusion
The conclusion frames optimizer-dependent rule acquisition as shaped jointly by target evidence, competing inputs, and permitted adaptation. The parity appendix supplies a population reference while distinguishing it from empirical test-observation averages.
5 Conclusion
- Optimizer-dependent rule acquisition varies with update rules and adaptation permissions, with the same optimizer advantage changing across modes and background-trainability conditions.Twenty unrestricted configurations can share a difficult-stage failure that nuisance-connected freezing substantially recovers.Source: §5 · p. 19
- A useful account must explain conditional recovery, changing optimizer gaps, and persistent failure, while distinguishing exposure from adaptation.The proposed next steps include matched-parameter controls, necessity tests on recovered target inputs, and a lower-rate Muon extension.Source: §5 · p. 19
A.1 Parity posterior and Bayes accuracy
- The parity population reference assumes independent balanced factors, while held-out observations are fresh noisy draws that may repeat latent combinations across splits.The empirical reference averages posterior confidence on actual test observations, whose sampling fluctuations can differ from oracle hard-label accuracy.Source: §A.1 · p. 22
A.2 Clean parity with rank-one first-layer weights
A rank-one first-layer construction represents parity on clean Boolean vertices across all scanned orders within width 64. The construction does not establish Bayes-optimal prediction under observation noise.
A.2 Clean parity with rank-one first-layer weights
- At t = t_l, the number of positive factors is l, so parity equals p_l.The slope changes telescope, yielding the stated parity values at the vertices.Source: §A.2 · p. 22
- For clean observed blocks, a rank-one first-layer construction represents parity on clean vertices within width 64 for every scanned order.All hidden units share one projection direction and use different biases; the result does not identify a successful training trajectory.Source: §A.2 · p. 22
- With observation noise, the construction does not show Bayes-optimal prediction on noisy observations.The projected noise standard deviation is σ_k/α_k.Source: §A.2 · p. 22
A.3 The odd-parity restriction without hidden bias
The appendix gives an architectural restriction for odd parity without hidden bias and separately characterizes Gaussian-background effects on displacement and activation gates. These arguments do not identify the mediator of the reported recoveries.
A.3 The odd-parity restriction without hidden bias
- For odd k ≥ 3, correct classification of every Boolean vertex would imply strict positivity of y(s)a⊤s everywhere, contradicting the established restriction.Even parity is not excluded, and the argument does not explain optimizer differences with trainable hidden biases.Source: §A.3 · p. 22
A.4 Independent background, displacement, and gates
- The independent zero-mean background argument proves Eq. (4) for Gaussian background, not arbitrary structured distractors.The scope is explicitly tied to the Gaussian-background setting.Source: §A.4 · p. 23
- The displacement statistic measures allocation of squared displacement, not information, memory capacity, or functional importance.It is retained as a descriptive adaptation statistic.Source: §A.4 · p. 23
- For a fresh test observation, Gaussian background flips a unit’s activation gate with probability Φ(−|m|/s_B), but this does not establish a recovery mediator.A gate may remain open while transmitting background variation, and the fresh-input expression does not directly apply to training noise used to fit weights.Source: §A.4 · p. 23
B Composite-stage information audit
The staged score weights evidence across stages, and an exact latent-prefix audit evaluates what prefixes can optimally predict. A stage label does not establish the minimum degree a learned predictor requires.
B Composite-stage information audit
- The staged score H weights terms from stages 1–4 by 4, 2, 1, and 1.5, respectively.For K = 4, the score is defined by the displayed weighted sum.Source: B Composite-stage information audit · p. 23
- Enumerating all 210 configurations yields conditional optimal accuracies, and reweighting stages leaves within-stage quantities unchanged.The ten signs q are independent and balanced before stage reweighting, and the atom mapping is invertible along each chain.Source: B Composite-stage information audit · p. 23
- The exact latent-prefix audit reports best possible conditional accuracy by atom prefix, with stage J known to the diagnostic, but no fitted model or observation noise.Stage is not an input supplied to the trained network.Source: Table 12 · p. 23
- A stage certificate tests whether a prefix is sufficient before later evidence is revealed; it does not prove a learned predictor’s minimum degree.Conditioning on a stage being first certifiable supplies information about later terms, so this differs from the prefix-sufficiency question.Source: B Composite-stage information audit · p. 23
C.1 Composite configurations
The composite configurations vary distractor and spurious relations across nuisance, mixed, and mode tasks. Search grids and checkpoints are selected separately for each intervention.
C.1 Composite configurations
- The task identifiers are preset labels, not conclusions about optimizer superiority; all tasks use view dimension eight, input dimension 3,124, and code seed 20261001.The identifiers are adam_low_order, sgd_high_order, muon_rare_modes, and mixed_review.Source: §C.1 · p. 24
- Table 13 organizes input composition as target, spurious, relational nuisance, and background dimensions, with factor channels specified by (a, σ).A dash denotes an absent block; the passage does not provide the table’s individual cell values.Source: Table 13 · p. 24
- The nuisance task has four independent distractor relations at each order 1–4 and stage proportions (.10, .15, .25, .50).The mixed task has two distractors and one spurious relation per order with equal stage proportions; the mode task has two distractors at orders 1–2.Source: §C.1 · p. 24
C.2 Search grids and selected checkpoints
- Each intervention selects anew using seeds 11 and 12, then runs seeds 21–23; FWSTEP02 retains the selected rate and neighboring rates, extrapolating at endpoints.The archive retains each task–optimizer–intervention grid and validation-selected reporting epochs.Source: §C.2 · p. 24
- Muon selects the lower endpoint at strict-parity orders 7–12, leaving the untested lower-rate regime as a specific coverage gap.Evaluation is recorded at epochs 1, 2, 5, and each multiple of ten through 200.Source: §C.2 · p. 24
C.3 All twenty strict orders
The strict-order table reports three-seed test accuracy alongside a posterior-based Bayes reference. The expanded comparison covers all twenty configurations but reuses the original three configurations.
C.3 All twenty strict orders
- Table 14 reports complete FWSTEP03 selected test accuracies as mean ± sample SD across three seeds, with a posterior-based Bayes reference and validation-based learning-rate selection.The passage supplies the table’s scope, not its individual accuracy values.Source: Table 14 · p. 25
C.4 The expanded optimizer comparison
- Table 15 reports three-seed test means for all twenty configurations, including nuisance and mixed overall accuracy, nuisance Stage 4, and the rarest pairwise mode.The original three configurations are reused, so this is not an independent rerun; full standard deviations and selections are retained in the archive.Source: Table 15 · p. 25
C.5 Source records and reproducibility scope
The manuscript bundle preserves records and derived data sufficient to reproduce numerical tables without rerunning training. Its reproducibility scope excludes some original trajectories and checkpoints.
C.5 Source records and reproducibility scope
- The bundle preserves experiment reports, generator and runner snapshots, configurations, selected seed records, diagnostics, and derived figure and table data.Derivation scripts reproduce the numerical tables without running training.Source: §C.5 · p. 26
- The archive does not claim to include every original trajectory and model checkpoint, and the twenty-configuration extension lacks all stage-specific trajectories..Source: §C.5 · p. 26
- The English source archive contains files needed to compile the paper; the separate full archive also contains Chinese comparison text and evidence records..Source: §C.5 · p. 26