Source-linked AI summary
Representation over Routing: Diagnosing Temporal Routing Pathologies in Multi-Timescale PPO
Jing Sun
TL;DR
The paper asks whether dynamically routing multi-timescale value predictions into the actor update yields reliable temporal abstraction or a numerical shortcut. In controlled PPO experiments on LunarLander-v2, it diagnoses gradient-based and error-based routing pathologies and evaluates Target Decoupling, which removes actor-side routing while retaining auxiliary critic heads. Target Decoupling removes the identified exploitable routing pathway and improves observed worst-seed return in the tested run set.
Problem
Actor-side routing among multi-timescale advantage heads may exploit numerical differences across unnormalized signals instead of representing temporally relevant control.
Method
The paper studies differentiable softmax routing, gradient-free error-based routing, and Target Decoupling in a controlled PPO setting on LunarLander-v2.
Results
Target Decoupling removes the actor-side routing pathway and improves observed worst-seed return across the tested seeds, while auxiliary heads do not reduce within-batch long-advantage variance in this run set.
Takeaways & Limitations
Multi-horizon critics can be retained as auxiliary regularization while temporal routing is removed from the actor objective.
Takeaways & Limitations
The evidence is limited to a controlled PPO study on LunarLander-v2 and does not establish broad benchmark superiority or persistence beyond this setting.
Abstract
from arXiv · showhide
Temporal credit assignment in reinforcement learning is often approached by introducing value estimates at multiple discount factors. A natural next step is to let the actor dynamically route among these temporal heads, using either differentiable attention or heuristic uncertainty weights. This paper argues that such routing can create a numerical shortcut rather than a reliable temporal abstraction. We study this issue in a controlled PPO setting on LunarLander-v2, using the environment as a visual sandbox for diagnosing failure modes. First, we formalize Surrogate Objective Hacking: a differentiable softmax router exposed to the PPO surrogate receives a direct gradient toward advantage heads that are numerically favorable for the current update, even when this routing change does not correspond to improved physical control. Because unnormalized advantages at different discount factors have different effective scales, this creates a scale-discrepancy vulnerability. Second, we identify the Paradox of Temporal Uncertainty in gradient-free error-based routing: short-horizon heads can receive the largest routing share because their prediction targets are easier, even when they are less aligned with delayed task success. As a structural response, we study Target Decoupling: the critic may retain multi-timescale auxiliary heads, but the actor is updated only with the long-horizon advantage. Target Decoupling is not presented as a broad performance booster; in this run set it removes the exploitable actor-side routing pathway and improves the observed worst-seed return. Code is available at https://github.com/ben-dlwlrma/Representation-Over-Routing.
1 Introduction
The paper diagnoses how actor-side routing among multi-timescale advantage heads can optimize PPO’s surrogate numerically rather than represent temporal context reliably. In controlled LunarLander-v2 experiments, it identifies two routing pathologies and evaluates Target Decoupling as a structural response.
- Motivation: Multi-timescale critics offer short-horizon, low-variance signals and long-horizon targets that better match delayed objectives but are harder to estimate.A single discount factor imposes one temporal scale on value prediction and policy improvement.
- Problem: Differentiable routing can change its weights to improve the PPO surrogate without producing a stable physical abstraction of temporal context.The paper calls this failure mode Surrogate Objective Hacking.
- Problem: Unnormalized advantages across discount factors have different horizons, magnitudes, and variances, creating a scale-discrepancy vulnerability for softmax routing.The router may select numerically favorable heads rather than semantically meaningful horizons.
- Experimental design: LunarLander-v2 serves as a controlled diagnostic environment for differentiable routing, error-based routing, and Target Decoupling rather than as a broad benchmark.The study examines episodic return, HackRate, attention entropy, myopic degeneration, and actor–critic separation.
- Contributions: The paper identifies Surrogate Objective Hacking and the Paradox of Temporal Uncertainty as distinct routing failures.The former concerns gradient exploitation inside the PPO surrogate; the latter concerns prediction reliability becoming misaligned with temporal relevance.
- Contributions: Target Decoupling improves observed worst-seed return and reduces observed across-seed dispersion, while auxiliary heads do not reduce within-batch long-advantage variance in this run set.The actor uses the long-horizon advantage while the critic retains auxiliary heads as regularizers.
2 Related Work
Related work motivates multi-horizon value prediction, differentiable routing, and uncertainty weighting, while highlighting distinct risks when these mechanisms enter temporal actor updates. The paper focuses on whether such predictions should be routed into the actor objective.
- Temporal credit assignment: GAE reduces variance through exponentially decaying traces but still depends on a chosen discount factor.Multi-horizon value representations extend this motivation with distributed temporal coding across futures.
- Routing and proxy optimization: A differentiable router inside a policy-gradient objective can alter the scalar advantage used by the surrogate without improving physical control.This creates a proxy-optimization channel within temporal aggregation rather than external reward design.
- Uncertainty weighting: Short-horizon heads can have smaller prediction errors because their targets are easier, not because they better align with long-term objectives.Using error directly for routing can therefore produce myopic degeneration.
3 Methodology
The methodology diagnoses how differentiable routing among multi-timescale value heads can exploit PPO through cross-horizon scale mismatch, then removes that actor-side pathway with Target Decoupling.
- Multi-timescale critic: The multi-timescale critic predicts value and GAE estimates across discount factors, while auxiliary heads can support representation learning without entering the actor update.The standard critic objective averages losses across heads; the studied issue arises when the actor routes among them.
- Differentiable routing: A differentiable router converts head outputs into a routed advantage and remains an optimization variable inside the PPO surrogate objective.Its logits receive gradients through the routed advantage and the ratio-weighted policy-loss term.
- Cross-horizon scale mismatch: Different discount factors produce advantages with different effective scales and variances, so raw cross-head comparisons are not intrinsically calibrated.Batch-normalizing the final routed scalar does not calibrate the raw advantage values used to compute softmax routing weights.
- Entropy-collapse signature: Repeated preference for one head can drive its logit upward until the softmax collapses toward a hard selector, rather than representing a graded temporal mixture.The paper identifies this entropy collapse as the predicted signature of scale mismatch.
- Target Decoupling: Target Decoupling keeps auxiliary critic heads but updates the actor only with the long-horizon advantage, removing the actor-side shortcut by construction.The design does not train a better router; it removes routing from the policy objective while allowing auxiliary short-horizon critic losses.
4 Experiments and Diagnostic Evidence
In a controlled PPO study on LunarLander-v2, diagnostic experiments expose failure modes in actor-side temporal routing and test Target Decoupling as structural separation. Differentiable routing collapses toward numerically favorable heads without reliable return improvement, error-based routing favors easier short-horizon predictions, and Target Decoupling improves observed worst-seed reliability in this run set.
- Experimental setting: The experiments use PPO on LunarLander-v2 as a controlled diagnostic setting rather than a broad benchmark.The environment combines short-term shaping, control penalties, and delayed landing outcomes.
- Differentiable attention routing: HackRate initially rises above the random-routing baseline of 1/4, then attention entropy collapses toward zero while return remains unstable and poor.This pattern indicates early surrogate chasing followed by hard routing collapse rather than sustained temporal abstraction.
- Error-based routing: Gradient-free error routing can assign dominant weight to the γ = 0.5 short-horizon head because its prediction errors are easier to reduce, while episodic return remains poor.The result is an observed myopic-degeneration failure mode in this PPO setup, not a universal impossibility result for uncertainty weighting.
- Target Decoupling: Target Decoupling removes the actor-side router, retains auxiliary critic prediction tasks, and keeps the actor aligned with the long-horizon advantage.The tested hypothesis that auxiliary heads reduce within-batch long-advantage variance was not supported by the ablation.
- Target Decoupling: Target Decoupling raises the observed worst-seed return and reduces across-seed dispersion relative to the strict long-horizon baseline.Figure 3 is a reliability diagnostic, not evidence of mean-performance significance.
5 Conclusion
The paper frames multi-timescale PPO as an optimization-geometry problem: actor-side routing over unnormalized advantage heads can exploit surrogate updates through scale mismatch. Target Decoupling removes this routing channel while retaining auxiliary multi-horizon critic prediction, improving the observed worst-seed return in the tested setting.
- Surrogate Objective Hacking occurs when a differentiable router inside the PPO surrogate selects numerically favorable advantage heads rather than semantically meaningful temporal horizons.The mechanism is a direct gradient through routing weights combined with uncalibrated advantage scales.
- Error-based routing can prefer easier, low-error short-horizon heads even when they are less aligned with delayed task success.The paper names this gradient-free reliability-versus-relevance failure the Paradox of Temporal Uncertainty.
- Target Decoupling preserves multi-horizon critic prediction as auxiliary regularization while removing temporal routing from the actor objective.The actor is thereby kept from using the identified routing pathway during policy improvement.
- Target Decoupling improved the observed worst-seed return across the tested seeds, but the paper does not present it as a general performance booster.Its reported value is specific to removing the actor-side routing channel in this study.
- Whether these routing pathologies persist beyond PPO and LunarLander-v2 remains an open question for other algorithms and control settings.Suggested tests include SAC, TD3, offline RL, and robotic or safety-critical tasks.