Planning to Learn

1 Introduction

• On ImageNet with ResNet-50 at a shared learning rate, exact policy gradient reaches 4.2% expected accuracy, versus 62% for cross-entropy.The comparison removes exploration, credit assignment and action-sampling noise; the expected-accuracy gradient is computed exactly by summing over labels.Source: §1 · p. 1
• The exact gradient values an update by its immediate gain, but the best allocation depends on how much training remains.A short run favors an already-likely correct example, while a longer run can favor a harder example with more potential gain.Source: §1 · p. 1
• Figure 1 shows exact policy gradient falling behind cross-entropy on expected test accuracy, while the horizon loss pulls ahead late in training.The figure compares expected test error over training at a shared flat learning rate on MNIST with an MLP and ImageNet with ResNet-50.Source: Figure 1 · p. 2
• The horizon loss weights examples by the accuracy they could gain from the learning that remains, moving from cross-entropy toward exact policy gradient as training ends.The paper interprets cross-entropy as total error under unlimited unit-speed log-odds learning and exact policy gradient as the zero-horizon endpoint.Source: §1 · p. 2
• 1.0–2.4 points on ImageNet: the horizon loss improves top-1 accuracy over cross-entropy at a flat learning rate, with gains arriving late.The reported evaluation spans MNIST and ImageNet with ResNet-50, ResNet-101 and ViT-S/16; gains grow with label noise.Source: §1 · p. 2

2 Learning as Allocation

• Training is modeled as a fixed optimizer and learning-rate schedule on a fixed labeled dataset for T steps, with the loss as the choice and final expected accuracy as the objective.The allocation model strips away the network and lets each example's logit gap move independently under a shared budget.Source: §2 · p. 3
• With orthonormal example gradients, exact policy gradient moves each logit gap in proportion to σ′(z), the example's current rate of accuracy improvement.After normalizing total progress, this yields the allocation flow used in the analysis.Source: §2 · p. 3
• Figure 2 contrasts immediate slope, finite-horizon attainable gain and unlimited remaining gain, and shows that the preferred example switches at H = a.At H = 4, the horizon weight targets examples still within reach, unlike the endpoint weighting rules.Source: Figure 2 · p. 3
• For gaps (a, −2a), the optimal allocation spends all budget on the easy example when H < a, but on the hard example when a < H ≤ 3a.Thus which example should receive learning depends on the amount of budget remaining.Source: §2 · p. 4
• Exact policy gradient cannot make that reversal: because σ′(a) > σ′(−2a) independently of H, it initially favors the easy example for every budget.At H = 2a, it captures only a fraction 4σ′(a) of the available gain, which vanishes exponentially with a.Source: §2 · p. 4

3 Cross-Entropy Is Patient Accuracy

• Cross-entropy is the total error an example would pay along a unit-speed log-odds path if learning never stopped.The future error over the next H units is the integral of 1 − σ(z + s), equal to CE(z) − CE(z + H).Source: §3 · p. 4

3.1 The Learning Horizon

• The finite-horizon loss weights an example by the accuracy it could gain over the remaining H units, becoming exact policy gradient at H = 0 and cross-entropy as H → ∞.The horizon in a neural network is approximated by the learning rate still to come, converted to logit progress by a dataset- and schedule-calibrated κ.Source: §3.1 · pp. 4–5
• The planning operator sums a per-example loss along the path, but each resulting horizon loss prices the full remaining budget as if it were available to that example alone.This separable plan is tractable but is not a solution to the shared-budget problem.Source: §3.1 · p. 5

3.2 Two Wrong Horizons

• Exact policy gradient wastes budget polishing already-likely examples, while cross-entropy can spend it on lost causes beyond the remaining horizon.The exact-gradient weight is symmetric around even odds; the cross-entropy weight is largest for the examples least likely to succeed.Source: §3.2 · p. 5
• On the polishing trap, exact policy gradient ends before the hard example reaches even odds; on the lost-cause trap, cross-entropy gives the finishable example at most a 1/(M + 1) share of the budget.The lost causes cannot reach even odds within the available budget, so the optimum spends everything on the finishable example.Source: §3.2 · p. 5
• The horizon loss escapes both traps, with regret shrinking exponentially as distractions move outside the horizon and growing only linearly with their number.The bound applies to the polishing and lost-cause traps.Source: §3.2 · p. 5

4 Experiments

• The horizon loss improves expected accuracy over cross-entropy on every ImageNet model and schedule tested, but top-1 accuracy does not improve for ViT-S/16 at a flat learning rate.The experiments report both expected and top-1 accuracy because sharper probabilities can improve expected accuracy without changing a prediction.Source: §4 · p. 6

4.1 Planning Beats Both

• At a shared learning rate of 10^-3, ResNet-50 expected accuracy rises from 4.2% with exact policy gradient to 67.5% with the horizon loss.On MNIST, the horizon loss cuts exact policy gradient's expected error by 24% and top-1 error by 29%.Source: §4.1 · p. 6
• At a flat learning rate, horizon loss lowers top-1 error versus cross-entropy by 1.0–2.4 points on ImageNet and 0.16 ± 0.04 on MNIST (Figure 3).The horizon loss has lower mean error than cross-entropy for every model and schedule shown, though cosine decay shrinks the gain.Source: §4.1 · p. 6, Figure 3 · p. 6
• The ImageNet gain arrives in the last few percent of training, when the shrinking remaining horizon falls to the scale of the logit gaps.On MNIST, where the initial horizon is already on the scale of the gaps, the curves separate much earlier.Source: §4.1 · p. 7

4.2 The Gain Grows with Label Noise

• Top-1 gains rise with label noise, from 3.3 to 7.3 points on ResNet-50 and from 0.1 to 2.7 points on MNIST as noise increases from 10% to 50%.On MNIST, exact policy gradient has higher top-1 error than cross-entropy without noise, but lower mean top-1 error from 20% noise onward; the horizon loss beats both at every noise level.Source: §4.2 · p. 7

4.3 Long Early, Short Late

• Only shrinking horizons give a consistent gain across tested κ values on MNIST; no tested fixed horizon matches the shrinking horizon on ResNet-50.Constant and growing horizons give at most marginal gains and lose to cross-entropy at small κ; fixed horizons H ≥10 tie cross-entropy, while H ≤3 perform substantially worse.Source: §4.3 · p. 7

4.4 Planning Cross-Entropy

• Planned cross-entropy lowers test NLL from 0.103 to 0.089 on MNIST and from 1.467 to 1.364 on ResNet-50, while the accuracy-oriented horizon loss raises NLL.The horizon loss raises test NLL to 0.122 on MNIST and 1.77 on ResNet-50; the two planned losses target different objectives.Source: §4.4 · p. 8

• Surrogate-loss theory asks which loss has the right minimizer, while this work asks how to spend a finite learning budget on the way there.Source: §5 · p. 8
• Maximum-likelihood RL interpolates between expected reward and log-likelihood as rollout counts rise; the horizon loss instead indexes that interpolation by remaining training budget.Source: §5 · p. 8
• DAPO and Dr. GRPO alter how gradients spread across prompts but do not use remaining training time; Delightful Policy Gradient matches a fixed horizon H = log 2 in a specified setting.Source: §5 · pp. 8–9
• At fixed H, the loss relates to skew divergence, categorical MAE, and generalized cross-entropy; the contribution is a budget-based reason for scheduling H.Source: §5 · p. 9
• Focal loss, curricula, hard-example mining, and importance sampling reweight examples by difficulty or schedule, while RHO-LOSS prioritizes learnable, not-yet-learned examples using a holdout model.Source: §5 · p. 9

6 Discussion

• The allocation analysis assumes independent fixed-speed example paths, a scalar horizon, and a test-error-calibrated κ; its separable rule can miss better coupled allocations.Source: §6 · p. 9
• Unlike classification, sampled-action RL must estimate success probability and future learning value from partial feedback, leaving planning to the remaining budget open.Source: §6 · p. 9
• Training updates trade accuracy now against accuracy later, a finite-budget allocation question also arising in choices about examples, compute, and data.Source: §6 · p. 9

A Proofs

• The proofs use Section 2's allocation model and proceed through the loss family, endpoint traps, and finite-horizon allocation bounds.Source: A Proofs · p. 12

A.1 Notation

• The notation distinguishes toy-flow budget spent τ and remaining hτ from training step t and estimated remaining horizon Ht.Source: §A.1 · p. 12, Table 1 · p. 12

• For H > 0, CH(z) = CE(z) − CE(z + H), and LH(z) normalizes CH(z) by 1 − e^-H.Source: §A.2 · p. 12
• As H grows, LH approaches cross-entropy; as H shrinks to zero, it approaches 1 − σ(z), whose negative gradient is the policy-gradient weight.Source: §A.2 · pp. 12–13
• At fixed H, LH is a scaled skew divergence, generalized cross-entropy mixes horizons, and Delightful Policy Gradient gives another fixed-horizon member; the new element is scheduling H by remaining budget.Source: §A.2 · p. 13
• A finite-horizon population minimizer preserves the Bayes decision and widens probability gaps on its support, though low-posterior classes can tie at zero.Source: §A.2 · p. 13
• Finite-horizon sharpening can raise expected accuracy without changing the Bayes classifier; cross-entropy's population optimum need not maximize expected accuracy.Source: §A.2 · p. 13

A.3 Greed

• With gaps (a, −2a), the optimal allocation reverses with budget: H < a goes to the easy example, while a < H ≤ 3a goes to the hard one; exact policy gradient can finish at the same polishing-trap endpoint.Source: §A.3 · pp. 13–14
• On the lost-cause trap, allocating budget to unreachable examples is suboptimal; cross-entropy's relative gain over the optimum can vanish as the number and difficulty of lost causes grow.Source: §A.3 · p. 14

A.4 Planning

• The horizon rule is near-optimal on the polishing and lost-cause traps, but separable deterministic rules preserve symmetry between identical examples even when concentrating budget yields more value.Source: §A.4 · p. 15

B Implementation and Experimental Details

• Within each implementation recipe, the model, optimizer, and schedule stay fixed; only the per-example loss changes.Source: B Implementation and Experimental… · p. 15

B.1 Implementation

• The horizon loss uses CH(z)/(1 − e^-H), with H set from remaining learning rate in logit units using dataset- and schedule-calibrated κ; horizon cross-entropy uses a related gradient factor.Source: §B.1 · pp. 15–16

B.2 Selection and Reporting

• There is no separate validation split; selections use test metrics, and three-seed ±2 SEM intervals are only a rough guide.The authors report no significance tests.Source: §B.2 · p. 16
• For MNIST’s flat-learning-rate comparison, every method uses 10^-3; horizon loss at κ=1 is reported from the same 30-seed sweep containing the κ grid and switch fractions.The separate robustness-check learning-rate grid is {1, 3, 10, 30, 100} × 10^-4 and is not used for selection.Source: §B.2 · p. 16
• ImageNet’s flat-rate κ = 3 was selected on ResNet-50 over three seeds, then reused unchanged across architectures and noise levels.The headline comparison was rerun over 10 seeds, including the three selection seed indices.Source: §B.2 · p. 16

C Robustness, Baselines and Extensions

• Every tested κ from 0.3 to 100 improves expected error over cross-entropy on MNIST and ResNet-50 at a flat learning rate.On ImageNet, κ = 100 retains more than 40% of the gain.Source: §B.2 · p. 16

B.3 MNIST

• The MNIST experiments use a two-hidden-layer width-50 MLP, with label flips shared across methods and seeds for each noise setting.The training labels are flipped with probability from 0 to 50%; test labels remain clean.Source: §B.3 · pp. 16–17

B.4 ImageNet

• ImageNet uses ResNet-50, ResNet-101, and ViT-S/16; κ = 3 for flat schedules and κ = 0.3 for cosine are reused across architectures.The protocols include random-resized crops and horizontal flips for training, and center crops for evaluation.Source: §B.4 · p. 17

C.1 Full Results

• Tables 2–5 report final expected and top-1 accuracy, with seed count n.The supplied captions identify MNIST flat-rate and cosine results and ImageNet flat-rate and cosine results.Source: §C.1 · p. 17, Table 2 · p. 17, Table 3 · p. 17

C.2 Learning Rate

• On MNIST, horizon loss beats cross-entropy at every tested learning rate; on ResNet-50, exact PG stays below 5% expected accuracy across 10^-4 to 10^-2.The ResNet-50 cross-entropy sweep is best at the grid edge, so it is not claimed to be fully tuned.Source: §C.2 · p. 17, Figure 7 · p. 18

C.5 Learning-Rate Decay and Weight Decay

• About 0.6 points of ImageNet top-1 gain over cross-entropy appears on all three models under cosine decay, and 0.2 points with weight decay.On cosine-decay MNIST, expected error falls 7% and top-1 error 3%, with a marginal top-1 interval.Source: Table 3 · p. 17, Table 5 · p. 18, §C.5 · p. 19

C.3 Horizon Calibration and Switches

• At κ = 1, ImageNet top-1 gain over cross-entropy is 1.5 points on both ResNets and −0.3 on ViT-S/16.The κ basin is broad on both datasets.Source: §C.3 · p. 19, Figure 8 · p. 19
• On ResNet-50, a 0.9 cosine-schedule switch slightly exceeds horizon loss on both metrics, while a flat-rate switch trades top-1 for expected accuracy.On MNIST, a tuned late switch roughly matches horizon loss.Source: §C.3 · p. 19, Figure 9 · p. 19

C.4 Focal Loss

• Horizon cross-entropy beats focal loss on all three metrics; its NLL is 1.364 against at least 1.39 for focal loss.Every tested focal-loss γ has higher expected and top-1 error than cross-entropy on ResNet-50 at a flat learning rate.Source: §C.4 · p. 19, Figure 10 · p. 20

C.6 Width and Training Budget

• Across MNIST widths, horizon loss improves expected error, while its top-1 gain grows from 0 to 10%.The expected-error gain holds at every tested width.Source: §C.6 · p. 20, Figure 12 · p. 20
• At κ = 1, horizon loss wins at every tested MNIST budget from 1k to 100k steps.The comparison tracks final error against training budget.Source: §C.6 · p. 20, Figure 12 · p. 20
• Exact PG catches cross-entropy at 100k MNIST steps, so its deficit is speed, not reachability.This result is reported for the training-budget sweep.Source: §C.6 · p. 20

C.7 Horizon Cross-Entropy

• The planning operator transforms cross-entropy into a horizon loss, implemented by multiplying its gradient by a capped factor; normalization recovers cross-entropy as H approaches zero.The factor is capped at 10.Source: §C.7 · p. 20
• On flat-rate ResNet-50, NLL falls from 1.467 to 1.364 while expected error rises from 0.378 to 0.393.The paper attributes the difference to planning bounded accuracy versus unbounded NLL.Source: §C.7 · p. 21
• Under cosine decay, horizon cross-entropy has NLL 1.270 ± 0.003 versus 1.289 ± 0.001 for cross-entropy.The horizon cross-entropy result uses 3 seeds; cross-entropy uses 10 headline runs without weight decay.Source: §C.7 · p. 21
• On MNIST, horizon cross-entropy has lower NLL and top-1 error than cross-entropy at every κ, but higher expected error for κ ≥1.Figure 13 compares these metrics against κ.Source: §C.7 · p. 20, Figure 13 · p. 21

D Further Related Work

• Unlike budgeted schedules or example-selection methods, the horizon loss puts remaining budget into the objective in closed form, without an unroll.The related-work discussion notes that self-paced learning and meta-reweighting do not derive weights from remaining optimization budget.Source: D Further Related Work · pp. 21–22