PaperlayerView exact revision on arXiv

Source-linked AI summary

LS-AR: Future-Predictive Latent Steering in Autoregressive LLMs

Anubha Gupta, Eduardo Pignatelli

arXiv:2610.03093v1cs.LG

TL;DR

Autoregressive models keep high-level goals and local generation in one token stream, where context noise can undermine goal retention. LS-AR adds FiLM-conditioned latent steering, and reports stronger retention on long-horizon retrieval and recovery under perturbed planning. Its evaluation is limited to synthetic symbolic environments, with single-vector capacity bounds on out-of-distribution structural generalisation.

  • Problem

    AR models process macro-objectives in the same token stream as local generation, exposing goals to attention dilution and context eviction over long sequences.

    Source: §1 · p. 1

  • Method

    LS-AR adds a continuous latent channel, using static goal encoding or recurrent state tracking to steer autoregressive decoding via FiLM.

    Source: §1 · pp. 1–2

  • Results

    Across tasks, LS-AR reports 100% semantic goal recall at H = 1024 and W = 500 (Static) and 89.0% recovery under off-trajectory perturbations (Dynamic).

    Source: §6 · p. 8

  • Takeaways & Limitations

    Textual goal dropout can give the latent steering channel dominant authority, allowing latent vectors to override explicit text instructions.

    Source: §1 · p. 2

    Evaluation is restricted to synthetic symbolic environments, and single-vector conditioning yields 0% success at N = 6 block scaling under out-of-distribution shift.

    Source: §5.4 · p. 8, §5.2 · p. 8

Abstract

from arXiv · show

Standard autoregressive (AR) models process high-level task instructions, state history, and transient tokens within a single shared sequence of tokens. Consequently, they lack the architectural mechanisms needed to isolate macro-objectives from context noise. To overcome this single-channel limitation, we introduce Latent-Steered Autoregressive (LS-AR), a dual-channel architecture that decouples continuous goal steering from discrete token decoding via FiLM conditioning. We evaluate a Static Goal Encoder (P_0) for persistent macro-objective retention across long rollouts and a Dynamic State Tracker (P_t) for recurrent latent updates during generation. On long-horizon retrieval past context limits (H=1024, W=500), LS-AR (Static) achieves 100% target recall where parameter-matched baselines collapse (0%), while increasing throughput by ~35% and cutting peak VRAM by 52.8%. In Blocksworld planning under forced perturbations (k=1), LS-AR (Dynamic) sustains an 89.0% completion rate vs. 71.0% for the baseline, though zero-shot entity scaling (N -> N+1) exposes single-vector capacity limits (0%). Finally, dual-channel authority analysis shows that text goal dropout establishes latent-dominant control, offering structural defence against text prompt injection while introducing a latent vector attack surface.

1 Introduction

Autoregressive models keep macro-objectives in the same token stream as local generation, leaving goals vulnerable to context dilution and eviction. LS-AR adds a continuous steering channel, evaluated in synthetic domain-tokenised environments.

Motivation and scope

  • AR models process macro-objectives in the same token stream as local generation, where attention dilution and context eviction can make them lose sight of primary goals.Evaluation is bounded to synthetic, domain-tokenised Treasure Hunt and Blocksworld environments, with parameter-matched baselines at 32M and 85M scales.Source: §1 · pp. 1–2

Architecture

  • LS-AR routes latent goal vectors into autoregressive decoders via FiLM, separating continuous macro-objective steering from discrete token decoding.Static Goal Encoder P0 targets retention across extended horizons; Dynamic State Tracker Pt recurrently updates steering to absorb trajectory perturbations.Source: §1 · pp. 1–2

Results

  • At H = 1024 and W = 500, LS-AR (Static) maintains 100% target recall while parameter-matched standard AR baselines fall to 0%.The reported contributions also include a ∼35% throughput increase and up to 52.8% lower peak VRAM.Source: §1 · p. 2

2 Related work

The related work contrasts context or memory extensions and token-space planning with LS-AR’s approach of integrating predicted latent representations directly into decoding.

2.1 Context extensions and recurrent architectures

  • Window scaling and external memory address context limits but incur quadratic compute or retrieval costs and do not structurally isolate macro-goals; recurrent SSMs can still dilute signals in one state.LS-AR instead uses a dedicated steering channel to keep target intent separate from local token generation.Source: §2.1 · p. 2

2.2 Explicit planning and agentic reasoning

  • Chain-of-Thought, Tree-of-Thoughts, and ReAct verbalise plans in token space, where scratchpads consume context and remain susceptible to eviction and task amnesia.LS-AR projects target states into continuous latents, leaving textual generation to focus on local execution while preserving global goal alignment.Source: §2.2 · p. 2

2.3 Predictive architectures (JEPA) and latent conditioning

  • Unlike LLM-JEPA’s auxiliary latent prediction and Agentic-JEPA’s offline state simulation, LS-AR injects latent predictions into the decoder via FiLM for autoregressive steering.The discussion situates the approach in JEPA-style prediction of abstract representations rather than exact future tokens.Source: §2.3 · pp. 2–3

3 Methodology

LS-AR addresses context eviction by maintaining a latent steering vector outside the token window. Its methodology predicts static or dynamically updated goal representations, injects them through FiLM, and trains the components in phases.

3.1 Problem formulation: asymmetric modalities and context limits

  • When rollout length exceeds context window W, prompt tokens leave active attention and can induce plan drift; LS-AR maintains a persistent steering vector without consuming context slots.The formulation maps a discrete goal prompt and a satisfying execution trajectory into a joint latent space.Source: §3.1 · p. 3

3.2 Static goal encoding (P0)

  • The Static Goal Encoder mean-pools frozen base-encoder prompt representations, then trains Epred to match the ground-truth rollout’s target representation during Phase 2 alignment.The target is the mean-pooled representation of the future trajectory, aligned using VICReg.Source: §3.2 · p. 3, Figure 1 · p. 3

3.3 Dynamic state tracker (Pt)

  • At predefined trigger tokens, the Dynamic State Tracker encodes the execution history’s trigger-position state and uses attention plus an MLP to conditionally update the active prediction vector.With α initialized at zero, the tracker begins as an identity function and learns corrective updates when observed states diverge from expectations; state encoding adds sparse O(i) work at trigger boundaries.Source: §3.3 · pp. 3–4, Figure 2 · p. 4

3.4 Latent steering via FiLM

  • LS-AR projects the steering vector into layer-wise FiLM scale and shift parameters that modulate decoder-layer normalisations and condition next-token prediction.Zero-initialising FiLM parameters makes the modulation an identity transform at initialisation.Source: §3.4 · p. 4, Figure 3 · p. 4

3.5 Phased training curriculum

  • Training proceeds through base domain adaptation, frozen-encoder predictor alignment, and end-to-end conditioned generation; textual goal dropout replaces the prompt with padding in 30% of Phase 3 steps.The dropout step trains the decoder to rely entirely on the latent vector, transferring primary steering authority to that channel.Source: §3.5 · pp. 4–5

4 Experiments

Across controlled, parameter-matched autoregressive comparisons, LS-AR is evaluated for long-horizon retrieval, decoding efficiency, perturbed planning, and channel authority. Static conditioning addresses context expiration, while dynamic tracking targets recovery from intermediate perturbations.

4.1 Mitigating context expiration (Treasure Hunt)

  • At H = 1024, LS-AR (Static) maintained 100.0% semantic recall while the AR baseline achieved 0.0%.At H = 1024, passkey retrieval was 84.4%; Figure 4 also shows target-identity clusters for LS-AR and distractor-noise collapse for the baseline.Source: §4.1 · p. 5, Table 1 · p. 5, Figure 4 · p. 6

4.2 Computational efficiency of static conditioning

  • +34.23% to +40.82% throughput was achieved by LS-AR over Baseline-Large; at W = 500, peak VRAM was reduced by 52.81% (1127.38 MB vs. 2389.06 MB).LS-AR offloads prompt extraction to a run-once encoder pass and decodes with a 4-layer backbone.Source: §4.2 · pp. 5–6, Table 2 · p. 6

4.3 Blocksworld: logical planning and dynamic resilience

  • 89.0% recovery under K = 1 was achieved by LS-AR (Dynamic), versus 71.0% for the AR baseline and 66.0% for LS-AR Static.At K = 2, Goal success was 5.0%–7.0%; zero-shot scaling to N = 6 yielded 0.0% success.Source: §4.3 · p. 6, Table 3 · p. 7

4.4 Dynamic state-tracking overhead

  • +33.56% throughput over the 32M baseline was achieved by LS-AR (Dynamic), with a +0.41% VRAM increase.Sparse execution of Epred keeps per-step generation on a lightweight 4-layer backbone.Source: §4.4 · p. 7, Table 4 · p. 7

4.5 Dual-channel control and security trade-offs

  • Under the Standard Curriculum, LS-AR executed Goal A (56.2%) while suppressing Goal B (0.4%); latent-forced training instead made latent steering the primary control channel.Under the Latent-Forced Curriculum, the model executed Goal B (67.2%) while suppressing Goal A (1.4%); the authors describe this as a trade-off between text authority and vulnerability to covert steering-vector injection.Source: §4.5 · p. 7, Table 5 · p. 7

5 Discussion

The discussion argues that latent steering preserves global goal control beyond active context boundaries, while exposing limits in out-of-distribution structural generalization and security trade-offs shaped by training. It also bounds the findings to synthetic symbolic evaluations and preliminary comparisons with unaugmented autoregressive models.

5.1 Latent steering vs. temporal context scaling

  • Latent steering maintains global control past active context boundaries and delivers ∼35% throughput gains by offloading compute to a shallow 4-layer decoder backbone.Standard context-window expansion remains vulnerable to attention dilution and prompt eviction.Source: §5.1 · p. 8

5.2 Limits of out-of-distribution structural generalisation

  • N = 6 block scaling yields 0% success across all variants, indicating that a single fixed-dimensional vector lacks relational capacity for unseen object configurations.This is a structural capacity bound under out-of-distribution shift.Source: §5.2 · p. 8

5.3 Modality authority and security dynamics

  • Goal dropout yields 67.2% injected goal execution, while standard training yields 56.2% primary goal execution and suppresses unsanctioned vector swaps to 0.4%.Latent-dominant control provides structural defense against text prompt injections but creates vulnerability to gradient-based vector perturbations.Source: §5.3 · p. 8

5.4 Limitations

  • Evaluation is restricted to synthetic symbolic environments; single-vector compression causes minor BPE digit scrambling in numeric passkeys, and unaugmented-AR comparisons leave memory-augmented baselines untested.The paper identifies these comparisons as a preliminary bounds check.Source: §5.4 · p. 8

6 Conclusion and future work

LS-AR modulates autoregressive token generation with latent goal vectors via FiLM conditioning. The conclusion reports long-horizon recall and perturbation-recovery results, while future work proposes extensions for multi-slot memory, adaptive updates, and open-vocabulary scaling.

6 Conclusion and future work

  • LS-AR modulates autoregressive token generation with latent goal vectors via FiLM conditioning.Source: §6 · p. 8
  • 100% semantic goal recall at H = 1024 and W = 500 with a +35% speedup over parameter-matched baselines; LS-AR (Dynamic) yields 89.0% recovery under off-trajectory perturbations.Source: §6 · p. 8
  • Future directions include k-slot latent memory for N → N + 1 scaling, entropy-guided state updates, and evaluation on multi-billion-parameter backbones and complex agentic benchmarks.Source: §6 · p. 8

A.1 Context aggregation: trigger-position vs. explicit pooling

The appendix constructs a historical context summary from the generated sequence by extracting the hidden state at the trigger boundary, rather than pooling across sub-action tokens.

A.1 Context aggregation: trigger-position vs. explicit pooling

  • The historical context summary s_i is constructed by extracting the hidden state at trigger boundary i from generated sequence x_1:i.Source: §A.1 · p. 10
  • Because the base encoder uses causal self-attention, the terminal-position representation integrates contextual information from preceding tokens in the prefix.Source: §A.1 · p. 11
  • Using s_i directly avoids storing intermediate activation matrices across token spans for auxiliary pooling, reducing memory overhead during long-trajectory generation.Source: §A.1 · p. 11

A.2 Event-driven vs. fixed-interval triggers

The Dynamic State Tracker updates at domain-specific event-boundary tokens rather than at fixed token intervals, accommodating variable-length actions and avoiding mid-action updates.

A.2 Event-driven vs. fixed-interval triggers

  • Dynamic State Tracker updates are triggered by domain-specific event-boundary tokens rather than fixed token intervals.Source: §A.2 · p. 11
  • Event-driven triggers align updates with variable-length sub-actions, avoiding fixed-interval updates that could re-steer the latent prior before a sub-goal is complete.Source: §A.2 · p. 11
  • Event-driven triggers adapt to actions of varying token lengths without artificial padding or manual segment alignment.Source: §A.2 · p. 11

B Appendix B: dataset formalisation and generation algorithms

Appendix B formalizes procedurally generated Treasure Hunt and Blocksworld evaluations targeting sequence length, search traps, and dynamic error recovery. It also documents domain tokenization, latent-predictor alignment, and training procedures.

B.1 Procedural Treasure Hunt (semantic & numeric memory)

  • Treasure Hunt tests exact retrieval of a semantic target and numeric passkey after a distractor sequence of length H, prompted by a fixed local trigger.The terminal action must retrieve both the passkey and target item.Source: §B.1 · p. 11
  • Trajectories sample one of 12 semantic items and a passkey from [1000, 9999], then construct a distractor filler trajectory of token length H.The generated goal prompt carries the target and code; the terminal action supplies them after the local trigger.Source: §B.1 · p. 11

B.2 Deterministic planning and perturbations (Blocksworld)

  • Blocksworld is modeled as a deterministic transition system, with initial states and goals generated from a ground state and optimal paths solved by BFS.Perturbation trajectories inject up to K legal sub-optimal actions and recalculate optimal recovery paths during execution.Source: §B.2 · p. 12

B.3.1 Domain-specific tokenisation

  • Domain-specific word-level tokenization maps entities, actions, and predicates directly to vocabulary IDs to isolate architectural capacity from BPE fragmentation.Structural delimiters and the update trigger are registered as immutable special tokens.Source: §B.3.1 · pp. 12–13

B.3.2 VICReg formulation and hyperparameters

  • The 3-layer latent predictor is aligned between P0 and Zfuture using VICReg invariance, variance, and covariance terms.The terms respectively match batch predictions to targets, maintain feature standard deviation above γ = 1.0, and decorrelate latent channels.Source: §B.3.2 · p. 13
  • VICReg uses weights λs = 25.0, λv = 25.0, and λc = 1.0, with numerical stability constant ϵ = 10−4.The predictor alignment uses batch size B = 32.Source: §B.3.2 · p. 13

B.3.3 Training hyperparameters

  • The experiments used NVIDIA A100 GPUs with 80GB VRAM, and the total project budget was approximately 350 A100 GPU hours.Individual training runs took approximately 3–5 hours for 32M-parameter models and 5–8 hours for 85M-parameter models.Source: §B.3.3 · p. 13
  • Training uses three 30-epoch phases: base domain adaptation, predictor alignment with the base encoder frozen, and end-to-end conditioned generation.The final phase applies textual goal dropout with probability pdrop = 0.30 and initializes the Tracker gate scalar to α = 0.0.Source: §B.3.3 · p. 13

Related papers

Loading 2610.03093v1…