PaperlayerView exact revision on arXiv

Source-linked AI summary

Short-to-Long: Detached Features for Long-Horizon Value Learning

Tong Che, Boris Ivanovic, Marco Pavone

arXiv:2610.30461v1cs.LGcs.AI

TL;DR

Non-stationary value targets complicate representation learning, while reducing the behavior discount would change the control objective. S2L transfers features from smaller-discount source networks through detached connections; it reports gains across Atari and online and offline continuous control, with fixed-trajectory measurements showing reduced source-target drift.

  • Problem

    Changing experience distributions and bootstrap predictions make value-learning supervision non-stationary, complicating useful representation learning in deep reinforcement learning.

    Source: §1 · p. 1

  • Method

    S2L transfers features from separately trained smaller-discount value sources to a long-horizon behavior learner through detached connections, leaving its objective unchanged.

    Source: §1 · p. 1, §2.2 · p. 3

  • Results

    S2L reports aggregate gains across Atari, online continuous control, and offline continuous control, improving terminal performance on 40 of 57 Atari games and return in 465/720 matched continuous-control pairs.

    Source: §1 · p. 2, §3.1 · p. 4, §3.2 · p. 4

  • Takeaways & Limitations

    Matched Atari ablations support feature transfer, smaller source discounts, and gradient isolation, while fixed-trajectory measurements show reduced drift in source value targets with mixed discounts.

    Source: §4.3 · p. 7, §7 · p. 9

Abstract

from arXiv · show

Learning useful representations from continually changing value targets remains a challenge in deep reinforcement learning. We study the temporal horizon of this supervision: smaller discounts make these targets less sensitive to distant bootstrap changes, but using them for behavior changes the control objective. We introduce Short-to-Long (S2L), which supplies a long-horizon behavior learner with features trained by smaller-discount source networks. Detached connections prevent the behavior value loss from updating the sources, allowing each source to follow its own value objective while the behavior discount remains unchanged. Across Atari, online continuous control, and offline continuous control, the approach yields aggregate performance gains, improving the paired terminal median on 40 of 57 Atari games. Matched Atari ablations support the roles of feature transfer, smaller source discounts, and gradient isolation. Fixed-trajectory measurements further show reduced drift in source value targets with mixed discounts.

1 INTRODUCTION

Changing policies and bootstrap predictions make value-learning supervision non-stationary, tying representation learning to deep RL optimization difficulties. S2L separates feature-source objectives from the behavior learner’s unchanged long-horizon objective through detached feature transfer.

1 INTRODUCTION

  • Policy updates change experience distributions, while value updates change bootstrap targets that shape the hidden features used for future predictions.This feedback connects representation learning to optimization difficulties in deep RL.Source: §1 · p. 1
  • Prior architectural, optimizer, and auxiliary-target responses leave the temporal structure of value supervision untouched.The discount couples future rewards in the control objective with the targets used to train hidden features.Source: §1 · p. 1
  • At k = 32, a bootstrap change is transmitted 21.1× less with γ = .9 than with γ = .99, while the longest-horizon loss can dominate shared-feature updates.Reducing the behavior learner’s discount would change how it values future rewards, motivating shorter-horizon sources instead.Source: §1 · p. 1
  • S2L trains smaller-discount value sources on shared experience and transfers their activations through stop-gradient connections, so the receiving loss cannot redirect source learning.Only the final network supplies action values or the actor objective, preserving the behavior learner’s original discounted objective.Source: §1 · pp. 1–2
  • S2L reports gains across Atari, online continuous control, and offline continuous control, with matched ablations, target-drift measurements, and a parameter-matched shared-trunk control.The Atari median paired human-normalized score gain is +.049 over 57 games relative to scaled PQN (Castanyer et al., 2025).Source: §1 · p. 2

2 SHORT-TO-LONG

S2L-4 uses four separately trained value learners that share transitions, passing features from shorter to longer horizons through detached additions. The final learner retains the base algorithm’s control objective; training details vary across Atari and continuous control.

2 SHORT-TO-LONG

  • Four learners use separate parameters, normalization statistics, optimizer states, and targets while sharing training transitions rather than parameters.At each residual depth, a learner receives its predecessor’s activation through a detached addition.Source: §2 · p. 2, Figure 1 · p. 2

2.1 FOUR HORIZONS, ONE BEHAVIOR OBJECTIVE

  • The final learner retains the base algorithm’s control objective, while the first three provide features rather than averaged values or mixed targets.Atari actions use the final Q-values; in continuous control the actor is optimized using only the final learner’s value.Source: §2.1 · p. 2

2.2 DETACHED ADDITIONS

  • Detached additions pass each predecessor’s same-depth activation forward, with sg acting as the identity in the forward pass and stopping gradients backward.No learned gates or projection parameters are used, and transferred information can pass through the short-to-long chain.Source: §2.2 · p. 3
  • For j < k, ∇θjLk = 0, while each network continues learning from its own value loss.In online Atari, each learner forms a Q(λ) return; losses are summed without averaging, gradients are clipped per learner, and λ = .65 with 32-step rollouts.Source: §2.2 · p. 3, §2.3 · p. 3

2.3 TRAINING AND DOMAIN INSTANTIATIONS

  • Online continuous control uses four disjoint critics with one-step targets and unchanged actor and interaction protocols.Critic updates detach predecessor features; actor updates hold critic parameters fixed while retaining action derivatives through all four critics, then update the actor from the final .99 value.Source: §2.3 · p. 3

3 ONLINE CONTROL RESULTS

S2L-4 improves Atari terminal performance and online continuous-control returns against their respective baselines. Continuous-control gains emerge late in training, while late-training return-dynamics measures also favor S2L-4.

3.1 ATARI: GAINS ACROSS THE FULL 57-GAME SUITE

  • S2L-4 improves terminal performance on 40 of 57 Atari games, with a median paired game improvement of +.049 human-normalized score (HNS).Thirty-six games improve by more than +.01, 11 fall within the descriptive ±.01 band, and 10 fall below −.01.Source: §3.1 · p. 4, Figure 2 · p. 4
  • The Atari comparison uses scaled PQN, four paired seeds, and 2.50M environment steps per game; five games were used for development and 52 were not.Terminal HNS averages the last ten logged values, each averaging the 20 most recently completed training episodes; paired differences are formed within seed.Source: §3.1 · p. 4

3.2 CONTINUOUS CONTROL: LARGER GAINS LATE IN TRAINING

  • At one million environment steps, S2L-4 improves return in 465/720 matched pairs, with a mean difference of +39.09.The mean paired difference is −12.71 in the first half, +27.03 in the second half, and +34.99 in the final quarter; every subgroup has a negative first-half difference but a positive endpoint mean.Source: §3.2 · p. 4, Figure 3 · p. 5
  • The continuous-control evaluation combines 720 matched pairs across two algorithms, two optimizers, four tasks, five seeds, and nine scales.Runs train for one million environment steps and are evaluated for ten episodes every 50,000 steps.Source: §3.2 · p. 4
  • Over 500k–1M steps, S2L-4 has lower aggregate local fluctuation, maximum drawdown, cumulative negative change, and worst single-evaluation drop.These measures describe retention of acquired return and are distinct from target drift.Source: §3.2 · p. 5, Table 2 · p. 5

4 UNDERSTANDING THE DESIGN

This section tests how target horizon, gradient isolation, feature transfer, and separate learners contribute to S2L. It examines target drift, controlled ablations, and a shared-trunk comparison.

4.1 TARGET SENSITIVITY AND MEASURED DRIFT

  • At k=32, changing γ=.99 to .9 reduces transmitted perturbations 21.1×; mixed discounts also lower measured source-target drift in all 64 paired runs and 16 game averages.Source: §4.1 · p. 6, Table 3 · p. 6

4.2 GRADIENT ISOLATION CONTROLS FEATURE-UPDATE GAIN

  • In the local shared-feature model, the linearized angular update is contractive exactly when 0 < ηK_W < 2; detaching the longest-horizon loss removes its update contribution.Source: §4.2 · p. 7

4.3 FOUR-LEARNER ABLATIONS AND CAPACITY CONTROL

  • Removing feature transfer lowers full-training AUC by .0797; all-.99 discounts lower it by .0652, receiving-loss backpropagation lowers full-training and second-half AUC by .0731 and .1530, and parameter-matched PQN trails by .1260 at the endpoint.Source: §4.3 · p. 7

4.4 SEPARATE LEARNERS VERSUS A SHARED TRUNK

  • S2L-4 leads shared-trunk MDH-4 on all four metrics, and a widened parameter-matched MDH-4 still trails in mean endpoint HNS (1.0286 against 1.0648).Source: §4.4 · p. 8, Figure 4 · p. 8

5 OFFLINE CONTINUOUS CONTROL

S2L is evaluated on nine Minari MuJoCo datasets against matched controls without feature transfer. It improves macro AUC, though gains are not positive for every dataset mean.

5 OFFLINE CONTINUOUS CONTROL

  • The evaluation covers HalfCheetah, Hopper, and Walker2d, each with simple, medium, and expert data.Experiments use TD3+BC with SimBa critics, five paired seeds, and 200k updates; the matched control omits feature transfer.Source: §5 · p. 8
  • Table 6 reports S2L-minus-control normalized-score AUC differences across all nine conditions, with positive paired seeds in parentheses.Each condition uses five paired seeds and 200k updates.Source: Table 6 · p. 9
  • +3.39 macro AUC gain, with six of nine dataset means positive.The reported hierarchical-bootstrap 95% interval is [+.17, +7.65].Source: §5 · p. 9

6 RELATION TO PRIOR WORK

Prior work has used multiple discounts, auxiliary representation objectives, and architectures for deep value optimization. S2L differs by transferring activations between separately trained networks while changing which value losses update those features.

Multiple horizons

  • S2L transfers hidden activations between separately trained networks without combining their Q-values, unlike shared-trunk multiple-discount heads (Fedus et al., 2019) and neighboring-discount decompositions (Romoff et al., 2019; Tang et al., 2021).In the shared-trunk approach, all head losses update shared features.Source: §6 · p. 9

Representation learning under non-stationarity

Deep value optimization

  • PQN and SimBa improve deep value optimization through parallel replay-free TD learning and residual architectures, respectively; S2L builds on these architectures while changing which value losses update transferred features.The passage also names skip connections, LayerNorm, and structured preconditioning as PQN improvements.Source: §6 · p. 9

7 DISCUSSION AND CONCLUSION

S2L preserves a single long-horizon behavior objective while learning features at multiple horizons, and the paper reports online-return gains across Atari and continuous control. The accompanying statements describe AI assistance, research scope and resource costs, and reproducibility practices.

7 DISCUSSION AND CONCLUSION

  • The fixed four-learner chain improves online return across Atari and continuous control, with ablations supporting its component choices.Fixed-trajectory probes show lower source-target drift during training; adaptive horizon selection and more efficient feature sharing are identified as extensions.Source: §7 · p. 9

AI USE STATEMENT

  • AI tools assisted with literature discovery, experiments, analysis, and manuscript drafting, while authors conducted experimental design and mathematical verification.Figures and tables were produced by analysis code.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

  • The work uses simulated Atari and continuous-control environments, involves no human subjects or personal data, and has computation and energy as its main direct resource costs.Matched experiment grids and reuse of valid checkpoints reduce repeated computation where available.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • Versioned artifacts and analysis scripts support the empirical summaries, and training and analysis code will be released.Continuous-control summaries include only matched pairs reaching the stated interaction budget and completing required evaluations; exclusions are reported in the appendix.Source: REPRODUCIBILITY STATEMENT · p. 10

A PROOFS

The proofs establish a sharp bound on how bootstrap-value changes affect targets, analyze conditions for contraction of a linearized joint update, and derive a polynomial identity. The section also identifies a base configuration for two-learner Atari experiments.

A PROOFS

  • Repeated substitution cancels common rewards and expresses target differences through value and return differences; absolute values and masks bounded by one establish Eq. (7), with equality attainable for same-sign, equal-magnitude value changes.The proof defines ∆Vj, ∆Gj, and Mk, and states the coefficient is sharp.Source: A PROOFS · p. 12
  • The semi-gradient update analysis gives head multipliers 1−η(1−co) and a feature multiplier, with joint-update contraction requiring 0 < η(1 −co) < 2 for every head in addition to the stated angular condition.The prose says targets are recomputed at each pre-update point when differentiating the numerical update map.Source: A PROOFS · p. 12
  • The proof extends cn to γ = 1, where cn(1, λ) = 1, and uses a finite sum that equals n when λ = 1 to prove Eq. (11).Source: A PROOFS · p. 12
  • Table 7 gives the base configuration for the two-learner Atari experiments.Source: Table 7 · p. 13

B EXACT TRAINING DETAILS

The training details define paired Atari development studies and compare detached feature interfaces while holding learners and updates fixed. The transfer comparisons isolate the feature choice without adding parameters or changing target construction or action selection.

B EXACT TRAINING DETAILS

  • The horizon study tests seven effective source horizons across five Atari games using four paired seeds.The tested horizons are H ∈ {4, 8, 10, 16, 32, 64, 100}, with γs = 1 − 1/H.Source: B EXACT TRAINING DETAILS · p. 13
  • Development comparisons use training-return AUC, and source discounts are selected by median SNI across games and seeds.Per-game discounts maximize four-seed median SNI; the global discount maximizes median SNI over all twenty game–seed pairs.Source: B EXACT TRAINING DETAILS · p. 13

C FEATURE-TRANSFER RULES

  • The Atari interface study changes only the detached feature supplied to the long learner while keeping both learners and their updates fixed.No interface adds parameters, optimizer state, target mixing, or an additional behavior policy.Source: C FEATURE-TRANSFER RULES · p. 14

D ATARI FEATURE-TRANSFER RESULTS

  • Table 8 compares feature-transfer methods against the scaled PQN base using paired training-return AUC, with every method retaining γ = .99 targets and the same action-selection rule.The table reports geometric means across the listed paired seeds.Source: Table 8 · p. 14

F FULL ALE-57 EVALUATION

The full ALE-57 evaluation includes matched Atari controls, a shared-trunk comparison, and fixed-trajectory measurements of target drift. The reported ablation and drift results favor detachment and mixed discounts in their respective comparisons.

F FULL ALE-57 EVALUATION

  • The full-suite control matrix matches the ALE protocol, optimizer, interaction budget, and four paired seeds across detached, no-transfer, all-.99, and writable-connection comparisons.The four-discount comparison spans 57 games; the two-learner comparison spans 52 games, excluding development games.Source: F FULL ALE-57 EVALUATION · p. 15
  • Detachment improves endpoint, full-training AUC, and second-half AUC, with full-training game medians favoring detachment on 42/52 games.Mean and median full-training AUC gains are +.0817 and +.0082.Source: F FULL ALE-57 EVALUATION · p. 15, Table 11 · p. 15
  • The shared-trunk control uses four heads at S2L-4's discounts, with behavior controlled by the .99 head and matched games, seeds, and training budgets.S2L-4 and the shared-trunk model differ in gradient clipping, preconditioning, loss aggregation, and detached feature transfer.Source: F FULL ALE-57 EVALUATION · p. 16

G FIXED-TRAJECTORY TARGET-DRIFT PROTOCOL

  • The drift protocol recomputes Q(λ) targets on fixed random and base-policy trajectories at 51 training landmarks without advancing environments or changing training state.The 16 selected games use four fresh paired seeds; the probe strata contain 256 trajectories each, with 32 transitions per trajectory.Source: G FIXED-TRAJECTORY TARGET-DRIFT PROTOCOL · p. 17
  • Mixed discounts lower both normalized target drift and raw RMS drift in every paired run and game.The pattern also holds for every primary source position and both probe strata.Source: G FIXED-TRAJECTORY TARGET-DRIFT PROTOCOL · p. 17

H CONTINUOUS-CONTROL TRAINING DETAILS

The continuous-control details specify matched S2L-4 and SimBa evaluations and report cross-seed variability. Additional passages describe offline activation capping, a training-AUC table, and a two-learner Atari update procedure.

H CONTINUOUS-CONTROL TRAINING DETAILS

  • The continuous-control design contains 720 matched pairs across two algorithms, two optimizers, four tasks, five seeds, and nine scales.Runs train for one million environment steps and are evaluated every 50,000 steps; Table 1 combines the paired evaluations.Source: Table 13 · p. 17, H CONTINUOUS-CONTROL TRAINING DETAILS · p. 18
  • Across 144 configurations, S2L-4 has lower mean cross-seed endpoint MAD than SimBa, 31.79 versus 35.21, and lower 500k–1M AUC MAD, 35.97 versus 46.21.Endpoint MAD is lower for S2L-4 in 88 configurations, for SimBa in 50, with six ties; AUC MAD is lower for S2L-4 in 87 configurations.Source: H CONTINUOUS-CONTROL TRAINING DETAILS · p. 18

I OFFLINE-CONTROL IMPLEMENTATION DETAIL

J SEED-LEVEL TERMINAL AUC

K TWO-LEARNER ATARI UPDATE

  • The two-learner Atari update uses shared transitions for long and short targets, transfers a detached source feature, then backpropagates the combined losses with separate clipping.The rollout actions come from the long learner; both losses use the same minibatch permutation, followed by one Kron step.Source: K TWO-LEARNER ATARI UPDATE · p. 18
Loading 2610.30461v1…