Interference and Generalization in Temporal Difference Learning

• Interference is characterized by the inner product of gradients for two objectives sharing parameters, with positive and negative values indicating constructive and destructive influence, respectively.Source: §1 · p. 1
• Positive interference means that an update for one process can improve the other, including when the processes use training and test losses or separate i.i.d. samples.Source: §1 · p. 1
• In TD, lower interference correlates with a higher generalization gap, whereas in supervised learning the relationship appears reversed with early stopping.Source: §1 · p. 1
• The paper notes that TD evaluation often omits generalization measures, while other work has documented brittleness and trained across environment distributions.Source: §1 · p. 1
• TD optimization is non-stationary, and bootstrapping in value-based methods creates an asymmetric flow of information from newly explored to known states.Source: §1 · p. 2
• Prior work links interference to improved reinforcement-learning performance and notes that popular meta-learning methods implicitly maximize it.Source: §1 · p. 2
• The authors investigate this contrast empirically, reinterpret results from popular environments in terms of memorization and temporal coherence, and offer mathematical insights.Source: §1 · p. 2

2. Preliminaries

• An MDP specifies states, actions, rewards, and transition probabilities; value functions express expected returns recursively, providing the basis for TD updates that bootstrap from current estimates.Source: §2 · p. 2
• Q-Learning and fitted-Q use bootstrapping, while TD(λ) averages future targets through a λ-return to trade off the unbiased return against the biased TD(0) target.Source: §2 · p. 2
• Agents may act greedily from a value function or directly parameterize a policy, which can be updated with policy-gradient methods such as REINFORCE.Source: §2 · p. 2

2.1. Computing interference quantities

• Because RL loss landscapes evolve, the paper also measures function interference: output-gradient alignment that is experimentally more stable and reflects representational rather than direct loss-level interference.Source: §2.1 · pp. 2–3
• For multi-output functions, the paper uses the maximum output, such as the confidence of the predicted class, when computing function interference.Source: §2.1 · p. 3

• Image-classification experiments use mutually exclusive training and test seeds, with each seed indexing an example from SVHN or CIFAR10 for both supervised classifiers and agents exploring masked images.Source: §3 · p. 3
• Image-exploration agents begin with a central 8 × 8 window, move it in four directions or choose among ten classification actions, and stop upon a correct classification or after 20 steps.Source: §3 · p. 3
• RL and supervised models use the same architectures; RL uses Double DQN, with REINFORCE included to remove dependence on value estimation.Source: §3 · p. 3
• Regression experiments use SARCOS and California Housing because value-function estimation is considered closer to regression than classification.Source: §3 · p. 3
• Atari experiments train DQN agents and evaluate policies with Q-Learning or TD(λ), using expert trajectories from an agent pretrained with Rainbow.Source: §3 · p. 3

• After sufficient data, TD methods show a strong negative correlation between interference and generalization gap, whereas classification methods show a positive correlation.Source: §4 · p. 3, Figure 1 · p. 3
• Regression trends are less consistent: SARCOS loses its correlation at 44k examples, California Housing shows positive correlation mainly with most or all data, and PG correlations are generally weak except PG CIFAR at 100 seeds (r = −.60).Source: §4 · pp. 3–4
• Across runs, classification interference is strongly correlated with the generalization gap, while regression and value-based correlations are negative but too weak for a clear trend; training-set size drives the gap most.Source: §4 · p. 4, Figure 2 · p. 4
• The combined findings suggest that low-interference solutions tend to generalize in classification but often memorize in TD, while the regression pattern appears only at a fixed data quantity.Source: §4 · p. 4

5. Interference in Atari domains

• Across three Atari games, the study compares supervised objectives, Q-learning in control and policy evaluation, and TD(λ) using loss changes on neighboring replay-buffer states.Experiments average ten runs per environment; control runs use 10M steps and other settings use 500k.Source: §5 · p. 4
• Regression loss-difference curves are positive around the update sample, and regression and TD* curves mostly overlap because their targets are similar.The curve around offset 0 indicates constructive interference; TD* approximates the regression target when Qθ* is accurate.Source: §5 · p. 4, Figure 3 · p. 5
• The control setting may develop a natural curriculum: newly seen states map onto initially learned factors, while its initially narrower feature spread diminishes as training progresses.The authors report a consistently larger spread of last-hidden-layer singular values for policy evaluation than control after 100k steps.Source: §5 · p. 5
• Control models' low interference is not explained by their longer training: measurements over the first 500k steps have similar magnitudes, and interference remains constant after initial epochs.The comparison addresses whether 10M-step control training simply forces more generalization than shorter policy-evaluation runs.Source: §5 · p. 5
• TD's pointwise loss gains are more than an order of magnitude smaller than regression's and decay faster with offset; Q-learning control gains exceed policy evaluation's.The comparison is reported for Atari experiments; the figure distinguishes control from policy evaluation.Source: §5 · pp. 4–5, Figure 4 · p. 5

5.1. TD(λ) and bootstrapping

• As λ approaches 1, neighboring-state gradients become more coherent; loss-difference gains exceed Q-learning's, suggesting sufficiently high λ yields targets that support faster learning.TD(λ) reduces reliance on bootstrapping by trading short bootstrapped targets for longer targets approaching Monte-Carlo returns.Source: §5.1 · pp. 5–6, Figure 5 · p. 6

5.2. The high variance of bias in TD(0)

• TD(0) targets inherit next-state prediction error, and bootstrap bias may compound across updates—especially with frozen targets—unlike the problem in regression, which may fade at convergence.High-variance approximators can make bootstrap bias inconsistent across nearby states, limiting the benefit of constructive gradient interference.Source: §5.2 · pp. 6–7
• On MsPacman, sign variance and interference correlated at r = 0.71; high values coincided with worse performance, while much smaller α required more updates.Sign variance is measured over trajectory windows of length five as a proxy for local target incoherence.Source: Figure 6 · p. 6, §5.2 · p. 6

6. Understanding the evolution of interference

• The analysis treats ρ′ as interference’s change with learning rate after an update; its sign depends on parameters, curvature, and loss changes.The derivation assumes θ≈θ′, and the authors note that interpreting the sign requires further assumptions.Source: §6 · p. 7
• For different-class examples, opposite signs in softmax-loss derivatives can mismatch gradients with curvature and reduce interference.The passage identifies target distribution and loss, alongside input-space structure, as important to gradient alignment.Source: §6 · p. 7
• At random initialization, ρ′ has a median near zero but consistently negative, with a heavy-tailed distribution and a larger negative tail.Source: §6 · p. 7
• Higher-capacity networks show smaller interference in TD but the reverse pattern in classification, consistent with nearby states having opposite-sign TD errors.The authors suggest that changing predictions and targets can flip only one of two nearby states’ TD-error signs, misaligning their gradients.Source: §6 · p. 7
• In TD, r2 has a larger share of negative samples than regression, while r3 is smaller in magnitude and negative on average.The decomposition writes ρ′ as −r1 −r2 −r3; Figure 7 reports the relative magnitudes of the terms.Source: §6 · p. 8, Figure 7 · p. 8

6.1. Interference and momentum

• Momentum may reduce ρ′ variance by making its evolution depend more on gradient alignment with the slowly changing moving average than on high-variance curvature alignment.The authors suggest this could dampen high-variance interference evolution, possibly including TD’s negative effects.Source: §6.1 · p. 8

7. Discussion

• Q-Learning generalizes poorly and produces networks that memorize training data, while TD(λ) improves generalization.Source: §7 · p. 8
• The authors find that interference evolves differently in supervised and temporal-difference objectives, but a full account of the mechanisms remains elusive.Source: §7 · p. 8
• The authors argue that treating TD bootstrapping naively as supervised learning may fail to leverage DNN generalization potential.They identify optimizer–TD coupling as an important future research direction.Source: §7 · p. 8

A.1. Interference

• An SGD update at sample B affects the objective at sample A, and the derivative of that loss change with respect to learning rate defines gradient interference.The derivation also relates the quantity to a Taylor expansion of the loss difference.Source: §A.1 · p. 12

A.2. Second order quantities

• The appendix establishes notation for gradients, Hessians, and sample-indexed objectives before deriving second-order quantities.Source: §A.1 · p. 12, §A.2 · p. 12
• The appendix rewrites the derivative of interference and gives a form that automatic differentiation can compute without an explicit Hessian.The authors report that it holds empirically for commonly used small step sizes.Source: §A.2 · pp. 12–13
• For regression-like objectives, the residual δ is prediction minus target, with its sign indicating whether the prediction needs to decrease or increase.Source: §A.2 · p. 13
• The TD derivation considers an update at B affecting interference at A, with successor states and θ≈θ′ simplifying the expression.The treatment allows the bootstrap target to change after a weight update.Source: §A.2 · pp. 12–13

B. Architectures, hyperparameter ranges, and other experimental details

• Natural-image experiments compare RL and supervised learning, while Atari experiments examine information propagation, TD(λ), and local target coherence.Atari is chosen for long-term decision making and as a standard benchmark; PyTorch and BackPACK support implementation and gradient computation.Source: B. Architectures, hyperparameter… · p. 14

B.1. Figure 1, 2, 8 and 9

• SVHN and CIFAR10 experiments use shared convolutional architectures and hyperparameter ranges across classification, DDQN, and REINFORCE, with at least three seeds per setting.REINFORCE results are analyzed mainly to highlight trend differences from TD; policy-gradient methods are left for independent analysis.Source: §B.1 · pp. 14–15

B.2. Figure 3, 4, 10, 12, and 13

• Atari Figure 3 uses DQN agents on MsPacman, Asterix, and Seaquest for 500k steps and ten runs per game.For Figure 4, policy evaluation runs for 500k steps and control for 10M; frozen targets update every 10k updates.Source: §B.2 · p. 15

B.3. Figure 5, 11, and 13

• Figure 5 evaluates λ-targets on MsPacman for 500k steps, computing them with the forward view and a frozen target network updated every 10k steps.Each setting uses five random seeds.Source: §B.3 · p. 15

B.4. Figure 6

• Figure 6 reuses the policy-evaluation experiments from Figure 4 that used Adam.Source: §B.4 · p. 15

B.5. Figure 7

• Figure 7 measures individual ρ′ terms during the first 100k steps of MsPacman regression and policy-evaluation experiments, using minibatches of 32.Source: §B.5 · p. 15

C. Reproducibility checklist

• The checklist reports unmodified algorithms, measures with constant instrumentation overhead, and supplementary code with documented dependencies.It also points to the mathematical setting and algorithm descriptions in the technical background.Source: C. Reproducibility checklist · p. 16
• The empirical-results checklist reports repeated runs, standard train/validation/test splits, and bootstrapped 90% or 95% confidence intervals for figures with error bars.Figure 8 uses 90% intervals because of many outliers.Source: C. Reproducibility checklist · p. 16

D. Extra figures

• Figure 8 plots generalization gap against training seeds, with circle size proportional to hidden units.Source: Figure 8 · p. 17
• Figure 9 shows average function interference during training as a function of capacity, with TD methods and classifiers having different trends.Source: Figure 9 · p. 17
• Figure 10 reports a surprisingly large expected gain at x = 0 but a negative gain around x = 0, suggesting RMSprop enables more memorization than Adam.Source: Figure 10 · p. 18
• Figure 11 shows that the TD pointwise loss difference varies asymmetrically around λ = 0.Source: Figure 11 · p. 18

D.1. Singular values, control vs policy evaluation

• All runs show a dramatic singular-value cutoff after about the 200th value, but whether fewer large singular values indicate generalization for value functions is unclear.The cited classifier finding is not established for regression or value functions.Source: §D.1 · pp. 18–19
• After 100k minibatch updates, policy evaluation initially captures more factors of variation than control, although control has seen fewer unique states despite the same data amount.The authors say it remains unclear whether this difference leads to a feature curriculum.Source: §D.1 · p. 18, Figure 12 · p. 19
• After 500k minibatch updates, larger λ values in TD(λ) yield larger singular values and a wider distribution.The authors suggest the less biased objective may capture factors of variation faster than relying on bootstrapping to learn them gradually.Source: §D.1 · p. 18, Figure 13 · p. 19

D.2. Evolution of TD gain with training

• Figure 14 maps temporal offset from the update sample to one axis, training time to the other, and TD gain magnitude to color.The figure caption identifies the control experiment as Adam on MsPacman, averaged over 10 runs.Source: §D.2 · p. 20, Figure 14 · p. 20
• The plotted temporal offset refers to the replay-buffer position relative to the sample used for the update.Source: §D.2 · p. 20
• Index 0 is excluded because its magnitude would be large enough to dim the other values.Source: §D.2 · p. 20