PaperlayerView exact revision on arXiv

Source-linked AI summary

Let Credit Follow Computation: Architecture-Aware Credit Transport for Large Language Model Reinforcement Learning

Qifan Shi, Zhaolu Kang, Chenghua Zhu

arXiv:2608.21501v1cs.AI

TL;DR

LLM credit estimators usually ignore the trajectory-specific computation performed by Transformer policies. This paper introduces computation-conditioned credit transport and reports 61.4% final held-out accuracy versus 53.8% for GRPO.

  • Problem

    LLM credit estimators are usually architecture-agnostic despite trajectory-specific, history-structured Transformer computation.

  • Method

    Computation-conditioned credit uses detached policy-internal statistics to control a trajectory-specific transport kernel, instantiated in CompPO with attention gates, path-dependent GAE, and an aligned critic.

  • Results

    61.4% final held-out development accuracy versus 53.8% for GRPO, with full CompPO exceeding dynamic-transport and fixed-gate critic ablations.

  • Takeaways & Limitations

    Internal computation can serve as a first-class variable in reinforcement-learning credit transport, with evidence extending across controls, stability tests, and two-backbone frozen evaluation.

  • Takeaways & Limitations

    The method does not establish that attention is causal or optimal, and gate staleness across multiple update epochs remains an approximation.

Abstract

from arXiv · show

Credit assignment in large-language-model reinforcement learning (LLM RL) can be separated into three objects: evidence about success, a transport operator that converts this evidence into token-level advantages, and an update geometry that turns advantages into policy changes. Recent work has greatly improved evidence, sampling, and update geometry, but the transport operator is usually architecture-agnostic. Fixed-discount GAE applies a stationary geometric kernel along token time; group-relative methods broadcast an outcome statistic across an entire response. Neither operator represents the trajectory-specific computation used by the Transformer policy itself. We introduce computation-conditioned credit transport (CCT), a general framework in which a detached statistic of the behavior policy's internal computation parameterizes the causal kernel that transports downstream value through a rollout. Our concrete algorithm, CompPO, maps native attention concentration to a bounded per-token retention gate, uses the gate in both the one-step bootstrap and a path-dependent generalized-advantage trace (Comp-GAE), and co-designs a transport-aligned critic (TAC) that reuses the actor's hidden states and routing information without a second same-scale Transformer. The task reward and clipped PPO policy objective remain unchanged; a constant gate recovers fixed-coefficient GAE. Across five Qwen3-4B seeds, CompPO reaches 61.4% final held-out accuracy (95% CI [60.8,62.0]) versus 53.8% [52.9,54.7] for tuned GRPO. Neither Comp-GAE with a standard critic (55.2%) nor TAC with a fixed gate (56.4%) matches the full model (interaction +2.4 [1.9,2.9]). Shuffle and position controls confirm trajectory-specific alignment; CompPO is stable in 10/12 PPO-grid runs versus 3/12. Frozen evaluation improves over GRPO by 4.3 and 3.9 greedy pass@1 macro points on Qwen3-4B and Llama-3.1-8B-Instruct.

1 Introduction

The paper identifies an architecture–credit mismatch in LLM reinforcement learning: standard transport operators ignore the Transformer’s trajectory-specific computation. It introduces computation-conditioned credit transport and CompPO, which use detached internal computation to shape advantage transport while preserving the external reward and clipped PPO objective.

  • Motivation: The Transformer’s content-dependent dependencies motivate architecture-aware transport because downstream decisions may rely on distant, trajectory-specific events.The paper treats attention concentration as a structural proxy rather than causal token attribution and tests its usefulness with alignment controls.
  • Problem formulation: The paper separates credit evidence, credit transport, and update geometry, showing that fixed-discount GAE and group-relative estimates discard realized internal computation.GAE depends only on temporal separation, while group-relative estimation broadcasts one response-level statistic across positions.
  • Contribution: CCT makes credit transport trajectory-specific by converting detached behavior-policy computation into a bounded, path-dependent causal kernel.The gate is computed by the old policy, detached, and stored with the rollout; constant gating recovers fixed GAE.
  • Method: CompPO uses native attention concentration to gate one-step bootstrapping and Comp-GAE traces, while TAC reuses actor hidden states and routing information for value estimation.The outer clipped PPO objective, verifier, rollout format, and reward remain unchanged.
  • Empirical evidence: 61.4% final held-out development accuracy versus 53.8% for GRPO, with every CompPO seed exceeding every GRPO seed at best and final endpoints.The comparison uses the same 3×3 actor-learning-rate/KL grid and five seeds.

2 Credit Assignment as Evidence, Transport, and Update

Credit assignment is factorized into evidence, a causal transport operator, and update geometry, clarifying which component an algorithm changes. Fixed temporal and group-broadcast transport omit realized Transformer computation, motivating architecture-aware transport as an explicit, optimizable coordinate.

  • Factorization: Credit assignment separates evidence, transport, and update geometry, with the framework identifying which object a contribution changes.Evidence may include rewards, TD residuals, process scores, or counterfactual estimates; transport produces token coefficients, and update geometry applies them through mechanisms such as PPO clipping and KL.
  • Temporal transport: Fixed GAE uses a stationary distance-only kernel, so its continuation geometry cannot represent how the policy computed the realized action.The limitation concerns transport geometry rather than the absence of local TD information.
  • Group-relative transport: Group-relative methods broadcast one outcome-derived coefficient across valid tokens, failing to distinguish pivotal tokens from routine ones and sometimes vanishing when group rewards are identical.DAPO, Dr. GRPO, and GSPO improve clipping, normalization, sampling, or sequence-ratio behavior but do not insert a trajectory-specific computation variable into Bellman/GAE transport.
  • Architecture-aware transport: CompPO makes the transport coordinate explicit and optimizable by allowing behavior-policy computation to parameterize the causal path, while co-designing the value representation used to construct it.Internal computation is treated as an unused information channel rather than a guaranteed causal explanation, requiring controls for critic capacity, position, marginal gate distribution, and training engineering.

3 Computation-Conditioned Credit Transport

Computation-conditioned credit transport uses detached behavior-policy computation to construct rollout-conditioned retention gates and causal kernels, while leaving the task reward and PPO objective unchanged. The framework generalizes beyond attention to other computation-flow signals and transport designs.

  • 3 Computation-Conditioned Credit Transport: CCT extracts a detached statistic from the behavior policy’s computation and maps it to a bounded retention gate stored with the rollout.The same realized gate is reused across optimization epochs, without differentiating through a policy-dependent task discount or redefining environmental utility.
  • 3 Computation-Conditioned Credit Transport: Comp-GAE uses each gate both in the one-step bootstrap and to retain later residuals, producing a policy-computation-conditioned causal kernel whose paths vary across positions and trajectories.This contrasts with standard GAE’s single stationary kernel across rollouts.
  • 3 Computation-Conditioned Credit Transport: The gate is an estimator-side retention factor, not an environment termination probability or normative task discount; the external task remains fixed and the same coordinate flows through the trace and value representation.CCT therefore differs from transition- or state-dependent discounting in role and coupling.
  • 3 Computation-Conditioned Credit Transport: CCT leaves reward computation, rollout sampling, KL regularization, and the clipped PPO surrogate unchanged, but implements a detached rollout-conditioned surrogate rather than claiming an unbiased gradient theorem.Gate staleness across multiple update epochs is identified as a limitation.
  • 3 Computation-Conditioned Credit Transport: Attention concentration is one deployable scalar-gate instantiation within a broader family that can use graph-valued kernels, alternative evidence, aligned critics, and computation-conditioned update geometry.Future signals include attention-flow graphs, retrieval provenance, tool-observation edges, MoE assignments, recurrent memory gates, and modality-specific routing.

4 CompPO: An Attention-Routed Instantiation

CompPO derives a detached, position-normalized retention gate from native attention concentration and uses it to route downstream value through both advantage estimation and a transport-aligned critic. The design reuses actor features, couples local and routed-history value predictions to the gate, and adds controls for stability and efficient long-context execution.

  • Transport-Aligned Critic: The transport-aligned critic avoids a second Transformer by reusing actor activations and adding fewer than 0.5% new parameters.It fuses upper-layer actor features, applies learned token-relevance routing, and combines local and routed-history values according to the same transition regime as the bootstrap target.
  • Credit Transport: The same gate controls one-step bootstrapping and path-dependent Comp-GAE, while remaining detached from critic updates within each rollout batch.All variants share actor features from the same forward pass, isolating controlled contrasts to the gate and critic architecture.
  • Transport-Aligned Critic: Under concentrated routing, the critic relies more on explicitly pooled history, whereas diffuse routing shifts weight toward the local actor state.This gate-aligned local/global value mixture is presented as a co-design hypothesis tested by the factorial experiment, not as a universal critic property.
  • Stability and Efficiency: CompPO stabilizes history-conditioned bootstrapping with bounded value targets, critic warm-up, zero-output initialization, and a closed-loop controller, while chunked extraction and top-K pooling reduce critic work to O(TK) with K = 64.The implementation avoids materializing full T × T attention and confines added computation to attention extraction and the lightweight critic.

5 Properties, Identifiability, and Claim Boundary

Under detached fixed-batch gates, CompPO exactly recovers standard fixed-γ GAE/PPO when gates are constant, while otherwise implementing a bounded, generally path-dependent transport kernel. The analysis establishes fixed-policy well-posedness but limits claims about PPO convergence, causality, universal dominance, and arbitrarily long-range reward transmission.

  • Properties: Constant gates recover standard fixed-γ GAE/PPO exactly, whereas varying gates produce generally path-dependent and nonstationary transport.With κ_t = γ and the standard critic restored, the bootstrap, Comp-GAE, and clipped policy update reduce exactly to the standard estimator path; otherwise the kernel is not generally stationary.
  • Properties: The method remains a scalar sequential transport mechanism: computation changes each transition’s local retention and the sensitivity of earlier advantages to later residuals, not nonlocal attribution.The transport kernel formalizes computation-following credit without claiming undiminished transmission across arbitrarily long chains; TD residuals can still provide local value-change signals when r_t = 0.
  • Properties: For fixed-policy evaluation with 0 ≤ κ_t ≤ κ̄ < 1, the detached backup is a sup-norm contraction with a unique bounded fixed point.The contraction modulus is at most κ̄, but this result does not establish global PPO convergence while the policy, critic, and future gates change.
  • Identifiability: The identifiability design separates average horizon, position schedule, trajectory-specific alignment, and critic capacity using mean-matched fixed-gate, position-only, shuffled, and factorial controls.The fixed gate uses empirical mean 0.61, while position-only controls freeze that mean across 20 equal-width position bins; global shuffling preserves values but destroys token correspondence.
  • Claim Boundary: The claims are bounded by detached-gate staleness, finite-temperature gate envelopes, and the absence of proofs for causal explanation, optimal concentration, universal critic failure, or dominance across regimes.Gates are computed from θ− and reused for two actor epochs; the actual range is [g(0), g(1)] within the configured envelope, and the theory concerns fixed-batch properties rather than universal performance.

6 Experimental Design

The experiments test performance, mechanism, signal specificity, robustness, and transfer through matched-budget comparisons, factorial ablations, alignment controls, stress grids, and frozen evaluation. The design fixes endpoints, training conditions, and baseline sensitivity checks to distinguish trajectory-specific effects from tuning artifacts.

  • Experimental questions: Five falsifiable questions assess performance against GRPO, joint necessity of the gate and critic, trajectory-specific alignment, PPO stability, and transfer across evaluation settings.The questions explicitly cover matched search budgets, component ablations, position and marginal controls, stress-grid stability, frozen evaluation, a second backbone, and larger GRPO groups.
  • Training and evaluation setup: Training uses hard DAPO-Math-17K prompts with 16K responses, Qwen3-4B as the primary backbone, Llama-3.1-8B-Instruct as a transfer control, and deterministic terminal rewards.Malformed reasoning tags receive negative terminal reward, while the length penalty applies only to correct, format-valid responses.
  • Mechanism and ablations: A 2×2 transport-by-critic factorial crosses fixed or dynamic transport with standard or aligned critics while matching checkpoints, prompts, seeds, rewards, optimization, and budgets.The fixed gate is κ = 0.61, equal to CompPO’s empirical mean gate, and removes trajectory dependence in both GAE and critic mixing; the design reports complementarity beyond additive main effects.
  • Robustness and controls: Robustness uses a 150-step PPO stress grid with two seeds per cell, while shuffle, position-only, and 8×8 GRPO controls test alignment and baseline sensitivity without replacing the matched five-seed comparison.Stability means final accuracy at least 42.4%, peak-to-final drop below 15 points, and no five-checkpoint postpeak run below base; the permitted inference is only a wider stable region in the tested grid.

7 Results

CompPO outperforms selected GRPO on best and final accuracy, with gains that persist across learning-rate/KL settings and reflect complementary contributions from its trajectory-specific gate and aligned critic. Alignment controls, stability tests, and frozen benchmarks support broader robustness and transfer, while bounding claims against tuned PPO and stronger GRPO baselines.

  • Main results: 61.4% final accuracy [60.8, 62.0] and 61.7% best [61.3, 62.1] for CompPO exceed GRPO’s 53.8% [52.9, 54.7] and 56.2% [55.6, 56.9].Every CompPO seed exceeds every GRPO seed at both endpoints; the best-to-final drop is 0.3 point versus 2.4 points for GRPO.
  • Factorial attribution: 61.4% final accuracy for full CompPO exceeds Comp-GAE with a standard critic at 55.2% and fixed-gate plus aligned critic at 56.4%, with a +2.4 [1.9, 2.9] interaction.The factorial table reports Full 61.4% [60.8, 62.0] final accuracy, while Fixed+Std, Comp-GAE +Std, and Fixed+Aligned reach 52.6%, 55.2%, and 56.4%, respectively.
  • Robustness: 10/12 CompPO PPO-grid runs are stable versus 3/12 for PPO, with mean best/final/drop of 52.9/45.6/7.3 versus 47.7/34.4/13.3.The evidence supports a wider tested stable region and lower average degradation, not universal dominance over method-specifically tuned PPO.
  • Frozen and transfer evaluation: 4.3 greedy macro points on Qwen3-4B and 3.9 on Llama-3.1-8B-Instruct separate CompPO from GRPO on frozen benchmarks.On Qwen3-4B, AIME25 greedy is the exception, while all seven Llama greedy differences favor CompPO; stronger 8×8 GRPO closes the frozen gap to 1.5 points in a single-run control.

8 Related Work

CompPO builds on classical credit estimators, LLM policy optimization, fine-grained credit, critic design, and internal-signal methods, while distinguishing itself by coupling a detached policy-internal computation statistic across bootstrap, trace, and critic representation. It treats attention as an observable routing signal rather than causal proof and frames computation as a coordinate of Bellman-style credit transport.

  • Fixed and transition-dependent credit in RL: Classical TD learning, eligibility traces, GAE, and PPO provide the estimator basis, while prior work studies transition-dependent or per-decision discounts as temporal-abstraction and bias–variance controls.AdaGamma learns state-dependent discounts with a return-consistency objective and analyzes the resulting operator (Wang et al., 2026).
  • Novelty boundary: CompPO differs from state-dependent discounting by using an unchanged task reward and a detached statistic of the LLM policy’s own computation across bootstrap, trace, and critic representation.This coupled construction instantiates policy-internal computation as an explicit coordinate of Bellman-style credit transport.
  • LLM policy optimization: LLM policy optimization includes PPO-based RLHF with actor and value models, critic-free RLOO and GRPO baselines, and scaling methods such as DAPO, Dr. GRPO, and GSPO.These methods use clipped updates and GAE or alter sampling, clipping, bias correction, normalization, or importance-ratio design.
  • Fine-grained LLM credit: Fine-grained LLM credit uses process labels, learned or implicit rewards, additional continuations, selective traces, or external statistics rather than one policy-internal statistic coupled across transport components.These works reinforce the importance of nonuniform token credit but rely on external rollouts, branch statistics, outcome influence, entropy, oracle scores, or critic-free trace surrogates.
  • Attention and internal signals as optimization primitives: Attention-based optimization work motivates internal signals, but attention’s causal faithfulness is debated; CompPO uses attention as an observable routing signal, not proof of causal attribution.The paper therefore does not claim the first use of attention or the first token-level credit method, but emphasizes its coupled construction and broader abstraction.

9 Discussion: Toward Architecture-Aware RL

The discussion frames CompPO as both a credit estimator and evidence that a Transformer’s trajectory-specific computation can parameterize reinforcement-learning credit transport. It argues that architecture-aware critics and transport may recover value-learning benefits while emphasizing composability, broader computational coordinates, and substantial limits on current evidence.

  • Architecture-aware perspective: CompPO makes trajectory-specific Transformer routing an explicit design variable for credit transport rather than treating the policy as a black box.Attention concentration is used because it is native, bounded after normalization, inexpensive to extract, and testable through shuffling, though it is unlikely to be definitive.
  • Critic diagnosis: The standard critic’s negative EV, low TD-target correlation, and stalled policy gains indicate representation–target mismatch, while the aligned critic tracks the computation-conditioned target and enables the full algorithm.This supports a specific diagnosis of critic failure rather than proving that one architecture is universally necessary.
  • Factorization and composability: CompPO primarily changes credit transport and co-designs its critic representation, leaving the outer objective unchanged so the intervention remains composable with alternative clipping, ratios, rewards, and baselines.The paper positions the method within an evidence–transport–update factorization and isolates the new transport axis experimentally.
  • Future directions: Architecture-aware RL could use distinct internal coordinates for value transport, exploration, KL control, and critic construction across MoE routing, retrieval, memory, multimodal, and recurrent mechanisms.Each exposed mechanism is presented as a candidate computational coordinate beyond attention.
  • Future directions: A richer extension would replace scalar κ_t transport with graph Bellman propagation over historical tokens or latent states and graph-based node or subgraph critics.The current scalar gate intentionally compresses upstream computation into one transition coefficient.
  • Scope and limitations: The evidence is limited to mathematical reasoning with two Transformer backbones up to 8B, terminal verifiers, and 16K responses, leaving other domains and larger models untested.Attention concentration can be corrupted by attention sinks, positional bias, induction heads, and output-layer calibration; the gate is stale across actor epochs and lacks a global policy-improvement bound.

10 Conclusion … B.6 Frozen Qwen3-4B evaluation

The paper frames CompPO as architecture-aware credit transport: detached internal-computation statistics control value propagation while preserving external rewards and PPO updates. Across controlled experiments, the method’s gains are attributed to trajectory-specific alignment, critic co-design, broader stability, and transfer across models.

  • 10 Conclusion: CompPO addresses the architecture–credit mismatch by using detached attention concentration to control a path-dependent GAE trace and computation-aligned critic while preserving reward and clipped PPO updates.The framework transports downstream value according to the policy’s internal computation rather than an architecture-agnostic kernel.
  • 10 Conclusion: The empirical evidence supports CompPO through matched tuning, factorial interaction, trajectory controls, stress-grid stability, and frozen evaluation across Qwen3-4B and Llama-3.1-8B-Instruct.The conclusion identifies consistent support for the proposed mechanism at multiple experimental levels.
  • A Full Algorithm: CompPO computes detached attention-derived gates from valid-history attention and actor hidden states, then reuses them in transport and critic construction.The algorithm extracts final-layer GQA-group attention, summarizes concentration, maps it to a bounded gate, and stores the gate detached for the rollout batch.
  • A Full Algorithm: The algorithm applies each gate in both the one-step TD residual and generalized-advantage trace before PPO actor updates and detached-target critic regression.The fixed-gate control replaces every gate occurrence in the TD residual, trace, and value mixture with one constant.
  • B Complete Experimental Tables: The complete experimental tables cover tuning selection, five-seed confirmation, seed-level factorial endpoints, gate controls, realized gate statistics, stress-grid stability, and frozen evaluation.These tables collectively document the paper’s selection procedure, ablations, descriptive statistics, robustness checks, and final-checkpoint evaluation.
  • B.1 Common tuning screen; B.2 Five-seed confirmation; B.3 Per-seed factorial results: The common tuning screen selects configurations without using frozen benchmarks, followed by five-seed confirmation and a gate-by-critic factorial analysis.The confirmation and factorial tables provide the primary multi-seed and interaction-focused experimental checks.
  • B.4 Gate controls and descriptive statistics: Gate-control experiments compare fixed-gate variants and report five-seed intervals, while realized-gate summaries document checkpoint statistics with some archived summaries excluded pending implementation reconciliation.The fixed-gate factorial uses κ = 0.61, the full method’s empirical mean, to separate trajectory dependence from mean scale.
  • B.5 Matched PPO stress grid; B.6 Frozen Qwen3-4B evaluation: Across 12 matched PPO stress-grid runs, PPO is stable in 3 whereas CompPO is stable in 10; frozen Qwen3-4B results are reported as final-checkpoint point estimates.The stress grid evaluates robustness across learning-rate and KL settings, and the frozen-evaluation table reports the corresponding endpoint estimates.

B.7 Frozen Llama-3.1-8B-Instruct evaluation … D.5 Why the coefficient is not a post-hoc loss weight

The merged sections establish reproducibility controls, evaluation conventions, theoretical properties of the detached transport operator, and why its coefficient changes credit propagation rather than merely reweighting policy loss gradients.

  • C.2 Phase controller; C.1 Primary configuration; B.8 Stronger GRPO group control: The same closed-loop controller is used across factorial cells, with hysteretic escalation and de-escalation, preventing it from inducing the gate-by-critic interaction.Escalation requires two consecutive trigger events and de-escalation requires five clear steps.
  • C.2 Phase controller; B.7 Frozen Llama-3.1-8B-Instruct evaluation: The evaluation controller excludes frozen benchmarks from control, checkpoint selection, and hyperparameter search, while treating the 500-problem validation set as development data.Controller signals include policy KL, gradient norm, clip fraction, entropy, and ID validation accuracy.
  • C.3 Value clamp control: The clamp affects only comparable early critic transients and then becomes the identity in the bounded grid, so it is not a method-specific adaptive intervention.PPO and CompPO were each repeated three times without the clamp in the most collapse-prone cell.
  • C.4 Attention extraction and memory; C.5 Statistical conventions; C.6 Data and result availability: The critic uses final-layer GQA attention with response-only storage and top-K pooling, adds fewer than 0.5% trainable parameters, and has no measured wall-clock or peak-memory superiority claim.The study reports five-run two-sided 95% Student-t training intervals, but frozen results are final-checkpoint point estimates with unquantified checkpoint variance.
  • D.1 Strict reduction: The fixed-gate path recovers standard GAE/PPO when κ_t is constant and the critic is restored to the standard control.The outer objective remains unchanged by construction.
  • D.2 Contraction of the detached operator: A fixed policy and detached gate induce a contraction with a unique bounded fixed point, establishing well-posedness of the operator.The contraction result applies to bounded value functions for any history.
  • D.3 Bounded trace; D.4 Length normalization of attention concentration: Attention concentration is length-normalized so uniform attention maps to c_t = 0, point-mass attention to c_t = 1, and ordering is preserved across positions.The normalized quantity n_tH_t ranges from 1 to n_t.
  • D.5 Why the coefficient is not a post-hoc loss weight: κ_t changes both one-step bootstrapping and later residual-retention weights, while also entering the critic representation, so it is not a post-hoc policy-loss weight.The policy loss consumes the resulting advantage without directly multiplying the loss by attention.

E Limitations, Open Questions, and Broader Impact

The paper identifies limitations in attribution validity, policy staleness, reward quality, evaluation scope, and systems evidence, while noting potential efficiency benefits and risks of estimator manipulation. Future work must test stronger internal signals, consistency strategies, broader settings, and safeguards.

  • Limitations: Attention concentration is a proxy rather than causal attribution, potentially distorted by sinks, positional biases, delimiters, induction heads, and calibration.Future work should compare layers and heads, mask known sinks, test interventions, and consider graph-valued signals.
  • Limitations: Behavior-policy gates become stale across actor epochs, motivating analysis of gate drift, PPO ratios, critic error, clipping, recomputation, and consistency regularization.Holding gates fixed avoids differentiating through attention but creates policy-dependence concerns as the actor changes.
  • Open Questions: Herfindahl concentration is useful but not established as optimal, leaving entropy, top-k, specialist-head, multilayer, learned, and attention-flow statistics open for comparison.The paper establishes the usefulness of one bounded, monotone, inexpensive, length-normalizable trajectory-specific statistic—not its superiority over alternatives.
  • Limitations: Bad rewards can be amplified alongside good ones, as verifier errors, formatting artifacts, answer conventions, or attention-dominating reward artifacts may reinforce spurious evidence.Robust verifiers and audit sets remain essential.
  • Scope and Evaluation: Evidence is limited to mathematical reasoning on Qwen3-4B and Llama-3.1-8B-Instruct with sparse terminal rewards and responses up to 16K, while frozen evaluations and the 8×8 GRPO control are point estimates.Dialogue, tools, code, retrieval, multimodal generation, agents, dense rewards, larger models, and measured systems efficiency remain experimentally unestablished; exact hardware, wall-clock, throughput, and peak memory are unavailable.
  • Broader Impact: More accurate transport may improve sample efficiency and reduce training requirements, but capable-model optimization could enable harmful uses or reward hacking through manipulated attention or routing.Detached gates and current controls do not rule out long-run co-adaptation, so deployments should monitor gate distributions.
Loading 2608.21501v1…