PaperlayerView exact revision on arXiv

Source-linked AI summary

Where Predictive Supervision Goes Shapes What VLA Policies Learn

Hanseul Kim, Jewon Yeom, Youngjoon Jeong, Minsoo Jo, Taesup Kim

arXiv:2609.36645v2cs.ROcs.AI

TL;DR

VLA policies can remain brittle under shifts in objects and scene geometry, and forecast quality alone does not show whether prediction improves their control-relevant representations. This paper compares matched future-prediction interfaces and traces how their errors shape the visual stream. Vision-token supervision preserves more transferable spatial, dynamics, and action information, and more direct scene-matched supervision is associated with stronger robustness under simulated and physical shifts.

  • Problem

    Future prediction may improve VLA representations, but forecast accuracy alone does not establish whether the visual representation used for control preserves actionable structure under distribution shift.

    Source: §1 · p. 1

  • Method

    The study holds predictive targets and training conditions matched while varying prediction interfaces and examining how forecast errors reach the policy’s visual stream.

    Source: §1 · p. 2, §3.2 · p. 3, §6 · p. 8

  • Results

    Vision-token features retain spatial, dynamics, and action information on spatial-tail evaluation, while policies with more direct, scene-matched supervision show stronger robustness under simulated and physical shifts.

    Source: §1 · p. 2, §4 · p. 5, §7 · p. 10

  • Takeaways & Limitations

    The value of future prediction for control depends on whether supervision reaches and shapes the visual representations through which the policy acts.

    Source: §7 · pp. 9–10

    The study fixes prediction targets to fixed-horizon residuals of momentum-encoded visual features, so it does not establish that one route is optimal for every predictive objective.

    Source: G LIMITATIONS AND FUTURE WORK · p. 38

Abstract

from arXiv · show

Future prediction is increasingly used to improve vision-language-action (VLA) policies, based on the premise that anticipating scene evolution encourages representations useful for control. However, forecast quality alone does not establish that a policy has learned a better representation for action. This distinction matters under distribution shift, where successful control depends on preserving spatial state and likely scene change beyond familiar configurations. We study what determines whether predictive supervision improves the visual representation used by a VLA policy. Through controlled comparisons with matched target constructions, prediction horizons, and training conditions, we find that different prediction interfaces produce markedly different forecasts and visual representations, including in the spatial, dynamics, and action information that transfers beyond familiar scenes. We trace these differences to how predictive errors shape the policy's visual stream. Consistent with this controlled finding, VLA policies trained with more direct, scene-matched future supervision show stronger robustness under simulated and physical distribution shifts. Together, our results frame future prediction as a representation-learning design problem whose value for control depends on whether its supervision reaches the representations through which the policy acts.

1 INTRODUCTION

VLA policies can remain brittle when objects or scene geometry shift, making representations that preserve actionable spatial structure and likely scene change important for robust control. This paper asks whether future prediction improves those representations and argues that its effect depends on how predictive errors reach the visual stream used for action.

1 INTRODUCTION

  • Policies that succeed on standard tasks can remain brittle when object identity, spatial configuration, viewpoint, language, or robot state changes.Such changes are unavoidable in deployment, motivating representations that preserve actionable spatial structure and likely scene change.Source: §1 · p. 1
  • Future prediction uses trajectory observations of scene evolution to encourage representations of the state and dynamics needed for control.Prior work predicts future frames, latent states, visual features, or motion alongside action objectives.Source: §1 · p. 1
  • Predictive accuracy alone does not establish that a policy learned a better representation for control, leaving the relevant design question unresolved.The auxiliary objective may shape the visual stream used for action only weakly.Source: §1 · pp. 1–2
  • Matched future objectives can produce substantially different visual representations depending on how their errors reach the policy’s visual stream.The most transferable features emerge when scene-matched supervision directly shapes the spatial visual stream used for action.Source: §1 · p. 2
  • The representation-level distinction remains consequential for VLA robustness under simulated and physical distribution shifts.The paper separates forecast output from policy representation and examines whether the learned information remains accessible for action.Source: §1 · p. 2

2 RELATED WORK

Prior robot-policy work predicts varied future targets through different modules and interfaces, while visual representation research studies where objectives train hidden states. This paper isolates how the route of a matched future target into the policy’s visual stream changes what that stream learns.

2.1 PREDICTIVE SUPERVISION IN ROBOT POLICIES

  • Prior robot-policy approaches vary prediction targets and delivery mechanisms, including outputs, learned carriers, and separate world models.Examples include CoT-VLA’s future image codes, attention-interacting carriers in FLARE, DreamVLA, World Guidance, and HiF-VLA, and state prediction or alignment in VLA-JEPA and FutureVLA [Xu et al., 2026].Source: §2.1 · p. 2
  • Prior methods establish future prediction and spatial supervision but vary targets, modules, or training stages together, leaving the effect of changing only the supervision route unisolated.This study holds target construction, horizon, readout count, backbone, and optimization fixed while varying spatial address and direct coupling.Source: §2.1 · p. 2

2.2 SUPERVISION PLACEMENT IN VISUAL REPRESENTATION LEARNING

  • Representation-learning work distinguishes flexible learned readouts from patch correspondence and notes that attention or positional structure can give readouts spatial address.Address alone does not specify how supervision reaches patch representations; head placement can also change where transferable features emerge.Source: §2.2 · p. 2
  • Patch-level and future-representation objectives provide complementary precedents, while this paper asks whether attaching the same temporal target to different positions changes a trainable policy’s visual stream.Examples include iBOT and GLaD for patch or geometry-aware supervision, and Dense Predictive Coding, SPR, DINO-WM, DINO-Foresight, and V-JEPA 2 for future representations.Source: §2.2 · p. 3

3 PROBLEM FORMULATION

The formulation separates the future target from the policy position receiving its supervision, enabling matched comparisons of prediction interfaces. It frames questions about what the visual stream learns, why interfaces differ, and whether those differences affect VLA robustness.

3.1 FUTURE PREDICTION AS AUXILIARY SUPERVISION

  • The patchwise objective predicts residual future features from current interface-selected tokens against momentum-encoder targets.The online encoder supplies current visual tokens and the EMA target encoder supplies features at the current and future observations; RQ1 and RQ2 optimize future loss alone, while RQ3 adds it to action loss.Source: §3.1 · p. 3

3.2 PREDICTION INTERFACES AS CREDIT ROUTES

  • The special-token interface reads contextualized special tokens, whereas the vision-token interface reads contextualized tokens at corresponding patches.Special-token forecast credit reaches patches through attention-mediated mixing; vision tokens additionally have an explicit same-position identity route.Source: §3.2 · p. 3
  • Special tokens may gain spatial correspondence through attention or positional structure, but their readout does not guarantee it.The comparison concerns how spatially matched forecast credit reaches the visual stream, not whether either token type can contain visual information.Source: §3.2 · p. 3
  • Fixing target construction and prediction horizon makes the prediction interface an independently examined factor in policy learning.This controlled comparison is the basis for the research questions.Source: §3.2 · p. 3

3.3 RESEARCH QUESTIONS

  • RQ1 asks whether prediction interfaces shape what the visual stream learns, since those differences could change information available to the policy.Source: §3.3 · p. 3
  • RQ2 asks why interfaces lead to different representations, to identify a source that may generalize beyond the compared architectures.Source: §3.3 · p. 3
  • Figure 1 depicts the downstream VLA comparison with shared pretrained VLM and action expert, momentum-encoder targets, and two forecast interfaces.Special tokens predict from special tokens; vision tokens predict at main-camera patch positions with same-position credit.Source: Figure 1 · p. 4
  • RQ3 tests whether representational differences influence policy behavior by evaluating robustness under distribution shift.Source: §3.3 · p. 4

4 THE PREDICTION INTERFACE SHAPES WHAT THE VISUAL STREAM LEARNS

With matched prediction targets, the interface changes both forecast content and the information accessible in the visual stream. Vision-token features retain spatial, dynamics, and action information on spatial-tail evaluation, whereas special-token visual features are weaker on these measures.

4 THE PREDICTION INTERFACE SHAPES WHAT THE VISUAL STREAM LEARNS

  • The forecast-content comparison uses a frozen pixel decoder trained on real-frame features from disjoint episodes, with copy and future-feature oracle references.The probe comparison freezes encoders and evaluates state, displacement, action, and geometry readouts on near-center and spatial-tail clips.Source: §4 · p. 4
  • Vision-token forecasts track scene-change magnitude and location, closing 45.0% of the full-image gap and 56.4% on moving pixels, while special-token forecasts close almost none.The ordering holds for near-center and spatial-tail samples.Source: §4 · p. 4
  • Figure 2 contrasts decoded forecasts and scene-change measures with frozen probes of state, dynamics, action, and geometry across near-center and spatial-tail splits.Effective rank is shown below each probe radar.Source: Figure 2 · p. 5
  • On spatial-tail evaluation, vision-token features retain linear decoding of state, future dynamics, and action after probes are fitted only on near-center clips.Upcoming action has R2 = 0.612 for vision-token features, while the special-token visual stream does not support a reliable readout; position, displacement, and geometry show the same ordering.Source: §4 · p. 5
  • Special-token visual features do not support reliable spatial-tail decoding of state and dynamics, and both special-token readout sites remain lower rank than the broader vision-token representation.The contrast indicates the interface changes what the visual stream learns, not merely where forecasts are decoded.Source: §4 · p. 5
  • Matched future targets can produce different visual representations, and vision-token forecasting keeps scene change, spatial state, and future action accessible in the stream used for control.Better forecasting alone does not close the gap in transferable visual features, as special tokens can forecast well under a frozen pretrained target or stable spatial indexing.Source: §4 · pp. 5–6
  • Figure 3 examines aligned-path strength and spatially routed gradients as interventions on the forecast credit reaching the visual stream.Its caption identifies probe outcomes under preserved, shuffled, or blocked special-token-to-patch gradients, alongside vision-token and unindexed special-token references.Source: Figure 3 · p. 6

5 DIRECT FORECAST COUPLING SUPPORTS TRANSFERABLE VISUAL FEATURES

The route by which forecast errors reach the visual stream shapes the features the policy learns. Direct same-position coupling most reliably yields transferable, action-readable features; correctly routed attention gradients alone do not.

5 DIRECT FORECAST COUPLING SUPPORTS TRANSFERABLE VISUAL FEATURES

  • Vision-token forecasting gives each forecast error a direct same-position gradient path to its matching patch, unlike an unindexed special-token forecast at initialization.The aligned-path intervention tests this route directly.Source: §5 · p. 6
  • A moderate aligned path rescues special-token forecasting, representation breadth, and spatial transfer toward vision-token performance, while a very weak path is seed-dependent.Removing the path at evaluation preserves part of the forecast recovery, indicating a persistent training change; the intervention changes both the head's readout and gradient route.Source: §5 · p. 7
  • Correctly routed attention gradients improve forecast decoding, feature rank, and position decoding, but not action decoding; mismatched routing collapses forecast and visual readouts.With shared Transformer weights, route-enabled and blocked conditions remain comparable in action transfer, indicating an additional learning route through shared updates.Source: §5 · p. 7
  • Direct same-position coupling provides the most reliable route from future prediction to transferable visual features, while correctly addressed attention gradients alone do not reproduce its action transfer.The broader conclusion is that the prediction interface determines which representations future supervision trains.Source: §5 · p. 7

6 FORECAST ROUTING AND TARGET CONTENT SHAPE VLA ROBUSTNESS

The policy-scale comparisons test how routing and future-target content affect robustness in simulation and on a physical robot. Direct vision-token forecasting with scene-matched targets performs strongest overall, while anchoring helps and shuffled future content removes the direct-route gain.

6 FORECAST ROUTING AND TARGET CONTENT SHAPE VLA ROBUSTNESS

  • Anchored spatial indices improve simulated position-shift and mean PRO success, but remain below vision-token forecasting; the patchwise controls stay near baseline on unseen pot layouts.The comparisons match token count and patchwise target granularity; anchoring fixes special-token-to-patch offsets.Source: §6 · p. 8
  • Vision-token forecasting achieves the strongest simulated object-position and overall PRO performance, and the highest pooled real-robot OOD success with the largest unseen-pot improvement.Figure 4 reports the frozen-policy probes and real-robot success rates; the unseen-pot condition combines a large layout shift with a rare, unstaged lid-knob grasp.Source: §6 · p. 8, Figure 4 · p. 9
  • A direct vision-token route does not improve simulated robustness when its future target is shuffled away from the scene.The result indicates that an effective route must carry relevant future change, not spatial coupling alone.Source: §6 · pp. 8–9
  • Across 12 checkpoints, future-displacement readout is associated with position-shift success (r = 0.82), a pattern consistent with but not isolating a link to robustness.Anchored indices improve displacement readout without similarly improving current-position decoding.Source: §6 · p. 9
  • On physical-policy probes, vision-token forecasting gives the strongest upcoming-action readout, while controls that improve container-position decoding do not match its unseen-pot performance.This comparison shows current-position encoding alone does not explain robustness.Source: §6 · p. 9

7 CONCLUSION

The conclusion frames future prediction as a representation-learning design problem: supervision must reach the visual stream used for action and carry scene-matched future content. The policy-scale evidence agrees with controlled findings across simulated and physical shifts, though it remains correlational.

7 CONCLUSION

  • Direct same-position coupling preserves spatial state, future dynamics, and action information that attention-mediated routing alone does not recover.At VLA scale, stable addressing partly closes the robustness gap, while target shuffling removes direct-route gains.Source: §7 · pp. 9–10
  • Effective predictive supervision depends on both a route that shapes the visual representation used for action and scene-matched future content.Policy-scale evidence remains correlational, though it agrees with the controlled findings across simulated and physical shifts.Source: §7 · p. 10
  • The authors report that generative AI assisted manuscript and methodology work, while empirical results came from their pipelines; they will provide training, intervention, probe, and evaluation code.Appendices document protocols, variants, and evaluation procedures.Source: AI USE STATEMENT · p. 10, REPRODUCIBILITY STATEMENT · p. 10

SUPPLEMENTARY MATERIAL

The supplementary material classifies future-prediction interfaces by readout and by how prediction losses reach the policy’s visual stream, then describes controlled comparisons that isolate supervision placement. It also situates robustness evaluation and specifies matched special-token and vision-token forecasting setups.

A.1 INTERFACE TAXONOMY

  • The interface taxonomy separates prediction readout from how supervision reaches the visual stream; a direct coupling requires an explicit same-position path from prediction i to visual token i.A one-to-one target assignment or attention pattern alone does not constitute direct coupling.Source: §A.1 · p. 15

A.2 EVALUATING ROBUSTNESS IN VLA POLICIES

  • Robustness benchmarks test behavioral outcomes under changes to objects, spatial arrangements, language, and environments, while leaving the representations underlying robust behavior less understood.The cited benchmarks include LIBERO-PRO, LIBERO-Plus, and Colosseum V2.Source: §A.2 · p. 15

A.3 RELATION TO OUR CONTROLLED COMPARISON

  • The controlled comparison isolates supervision placement by matching target construction, prediction horizon, readout count, backbone, and optimization.The broader study separates supervision placement, spatial address versus direct coupling, and VLA robustness under shifts.Source: §A.3 · p. 15

B CONTROLLED FORECAST AND REPRESENTATION ANALYSIS (RQ1)

  • RQ1 holds target construction, prediction horizon, forecast readouts, backbone, and optimization fixed while changing the token stream used to read each patch forecast.This setup compares special-token and vision-token interfaces.Source: B CONTROLLED FORECAST AND… · p. 16, §B.1 · p. 16

B.1 MATCHED INTERFACES AND TRAINING

  • Both interfaces predict residuals for the same 64 future patches, but special-token forecasts use contextualized added tokens whereas vision-token forecasts use corresponding contextualized patch tokens.The vision-token interface supplies a direct route to the visual position described by each forecast; the special-token interface does not impose that route.Source: §B.1 · p. 16, Figure B.1 · p. 16
  • The controlled prediction-only experiment uses 500 LIBERO-Object episodes, a 24-frame momentum-encoder target, five seeds, and no action or language objective.Every tenth episode is held out, yielding 450 training and 50 validation episodes.Source: §B.1 · p. 17

B.2 DECODED FORECAST EVALUATION

The evaluation compares decoded forecasts and frozen-feature probes, then tests whether target construction and positional addressing alter the interface comparison. Forecast quality and transferable representation quality do not always move together, and their relative outcomes depend on the target and positional structure.

B.2 DECODED FORECAST EVALUATION

  • Vision-token forecasts capture the magnitude and spatial location of future change, whereas special-token forecasts stay close to the current frame.Comparable oracle slopes rule out a weaker pixel decoder as the explanation; spatial-tail examples contain larger changes, so their higher gap closure is not evidence of stronger tail forecasting.Source: §B.2 · pp. 17–18

B.3 FROZEN-FEATURE PROBES

  • Vision-token features retain the strongest spatial-tail position, future-displacement, action, and geometry readouts, while neither special-token readout site recovers that profile.Effective rank is 36.0±0.5 for vision-token features, versus 6.3±5.1 for special-token visual tokens and 7.4±5.7 for their special-token stream; identity also degrades across models.Source: §B.3 · p. 19

B.4 FROZEN-TEACHER AND POSITIONAL-STRUCTURE CONTROLS

  • With a frozen forecast-trained teacher and learned absolute positions, special-token forecasts remain near the copy baseline and their representations transfer less strongly on position and action than vision-token representations.Gap closure is 0.3% versus 17.1%; position R2 is 0.13 versus 0.59, and action-chunk readout is 0.14 versus 0.57.Source: §B.4 · p. 19
  • With frozen SigLIP targets and learned absolute positions, special-token forecasts improve substantially, but their representations remain lower-rank and weaker on spatial-tail position and action readouts.The future-displacement readout is not reliably separated, and two of five special-token seeds have effective rank below five; forecast quality alone does not determine transfer.Source: §B.4 · pp. 19, 21
  • Stable 1-D rotary addressing narrows the interface gap: with frozen SigLIP, interfaces are nearly matched in forecast gap closure, effective rank, and action transfer, while vision-token features retain higher position readout.With the momentum target, special-token gap closure exceeds the vision-token reference, but its action-chunk readout remains lower.Source: §B.4 · p. 21

C CREDIT-ROUTING INTERVENTIONS (RQ2)

  • RQ2 separates whether forecast gradients reach the matching visual region from whether forecasts have a direct same-position path to that region.The appendix provides measurement definitions, intervention protocols, and complete results for this comparison.Source: C CREDIT-ROUTING INTERVENTIONS (RQ2) · p. 21

C.1 MEASURING SPATIAL ALIGNMENT OF FORECAST CREDIT

The interventions examine how forecast credit is routed, whether aligned paths can rescue special-token learning, and whether spatially coarsening visual-token updates weakens learned structure. The VLA-scale section then specifies simulation variants and controls for prediction routing and positional addressing.

C.1 MEASURING SPATIAL ALIGNMENT OF FORECAST CREDIT

  • At initialization, vision-token forecast credit concentrates in the corresponding image region, while special-token and shuffled routes remain at chance, before either interface has learned spatial correspondence.The same routing distinction persists in trained π0.5 policies: vision-token credit remains diagonally concentrated even with shuffled targets, whereas special-token credit returns to similar object regions across forecast locations.Source: §C.1 · pp. 21–23

C.2 ALIGNED-PATH CONSTRUCTION AND SENSITIVITY

  • Moderate aligned-path coupling moves the special-token model away from collapse and recovers forecast quality, representation breadth, and spatial transfer; stronger coupling does not improve transfer uniformly.After training, setting the path strength to zero leaves partial recovery, indicating that learning changes what the special-token pathway retains rather than relying entirely on direct forward access.Source: §C.2 · pp. 23–24

C.3 BACKWARD-ONLY ATTENTION-ROUTING INTERVENTION

  • In the shared-weight control, shuffling returned attention credit reduces spatial and action readouts, while blocking and native conditions remain closer than in the untied setting.The shared-weight results are a robustness check, not a second clean isolation, because the visual tokens can learn through weights updated by the special-token stream.Source: §C.3 · p. 25

C.4 DEGRADING THE SPATIAL RESOLUTION OF VISION-TOKEN CREDIT

  • Coarsening vision-token forecast credit to four regions or the whole image approaches the collapsed special-token profile, despite norm-matched updates.Sixteen-region updates retain part of the forecast and transfer benefit; shuffling is only a location-sensitivity diagnostic because its update is not a descent direction for the original loss.Source: §C.4 · p. 27

D VLA-SCALE SIMULATION EVALUATION (RQ3)

  • The VLA-scale evaluation compares policy variants under distribution shift and examines whether routing and scene-matched future content shape the visual representation used for control.The variants share a backbone and training protocol.Source: D VLA-SCALE SIMULATION EVALUATION (RQ3) · p. 27

D.1 VLA VARIANTS AND SHARED TRAINING PROTOCOL

  • The π0.5 variants share a pretrained checkpoint, LIBERO demonstrations, and a common forecasting target, horizon, loss, and auxiliary weight; the action-only baseline retains flow matching.Forecast variants use an EMA SigLIP target, a 16-step horizon, motion-weighted loss, and isolated prediction-token placement.Source: §D.1 · p. 27
  • The VLA controls distinguish patchwise vision-token prediction from region-pooled special-token prediction, shuffled future targets, and special tokens with matching patch count.An anchored-index control fixes each special token 512 rotary positions after its corresponding main-camera patch regardless of instruction length.Source: §D.1 · pp. 27–28

D.2 EVALUATION PROTOCOL AND ADDITIONAL RESULTS

Across simulation and physical evaluation, prediction interfaces differ in how well they support robust control and preserve information about future change. Anchoring stabilizes spatial credit delivery, while scene-matched vision-token forecasting yields the strongest future-change readouts.

D.2 EVALUATION PROTOCOL AND ADDITIONAL RESULTS

  • Vision-token forecasting improves mean LIBERO-PRO success over the baseline and M = 16 special-token interface, while anchored indices improve object-position shift and mean success across all three seeds.Matching the number of special tokens to visual patches alone produces no consistent gain.Source: §D.2 · p. 29
  • Vision-token forecasting beats the baseline in all four LIBERO suites, and anchored controls beat variable-index controls in all four.The evaluation covers 24 suite-condition cells, with ten trials per task; retention is perturbed success divided by unperturbed success, averaged across suites.Source: §D.2 · p. 29

D.3 ANCHORING STABILIZES SPATIAL CREDIT DELIVERY

  • Anchoring keeps forecast-gradient delivery spatially aligned from the visual prefix through later layers, unlike variable indices, whose alignment weakens at the prefix and final measured layer.The matched M = N = 256 comparison isolates anchoring while holding token count and patchwise targets fixed; patchwise targets alone do not ensure a stable route.Source: §D.3 · p. 30

D.4 INFORMATION ABOUT FUTURE CHANGE IN THE VLA VISUAL STREAM

  • Vision-token forecasting gives the strongest held-out readouts of future object and end-effector displacement, while shuffled-future forecasting is nearly tied on current position and object-to-gripper geometry.The comparison tests probes on a shared frame bank; LIBERO-10 is excluded from held-out-task scores because changing its task also changes the scene and object set.Source: §D.4 · p. 31
  • Anchoring improves future-displacement decoding across matched seeds without improving current object-position readout; layer-12 displacement decoding is associated with object-position-shift success (r = 0.82).The association is consistent with, but does not establish, a role for features that retain information about future scene change in robustness.Source: §D.4 · p. 31, Table D.5 · p. 31

E REAL-ROBOT EVALUATION (RQ3)

  • The real-robot evaluation tests whether simulated interface ordering persists under changed visual evidence and object-container layouts, but does not independently isolate the prediction interface.The study compares five π0.5 policies on matched blocks and analyzes held-out-episode representations.Source: E REAL-ROBOT EVALUATION (RQ3) · pp. 31–32

E.1 POLICIES AND TRAINING DATA

  • Five policies share a pretrained checkpoint, data split, random seed, optimization schedule, action horizon, and 8,000-step evaluation checkpoint; forecasting variants predict main-camera features 0.8 s ahead, while the wrist camera receives no prediction target.The dataset contains 442 teleoperated episodes across three tasks; 44 episodes were reserved at the episode level for representation probes.Source: §E.1 · p. 32

E.2 BLIND EVALUATION PROTOCOL AND CONDITIONS

  • Blind evaluation uses matched instruction-layout blocks and three trials per policy; the unseen-pot condition is a composite shift, not an isolated position intervention.The pot layouts are roughly twice as far from training as training layouts are from one another, and the condition also involves an uncommon lid grasp and longer manipulation sequence.Source: §E.2 · pp. 32–33

E.3 FULL POLICY RESULTS AND FAILURE DIAGNOSTICS

On shifted real-robot blocks, vision-token forecasting resolves comparisons against the baseline and M = 16 special-token policy, but not against both larger-token controls. Representation probes indicate its advantage is concentrated in future behavior information in the visual stream.

E.3 FULL POLICY RESULTS AND FAILURE DIAGNOSTICS

  • The M = N = 256 controls were competitive under camera blur but returned to the baseline range on unseen-pot layouts, with little difference between native and anchored indices.This condition dependence leaves an anchored-index effect on the robot unresolved.Source: §E.3 · p. 34
  • 11 shifted blocks favored vision-token forecasting over the baseline without a loss (p = 0.001); against M = 16 special tokens, it won 11 blocks with two losses (p = 0.022).Comparisons with the two M = N = 256 controls were unresolved; the pooled analysis resolves only the two primary comparisons.Source: §E.3 · p. 34, Table E.4 · p. 35
  • On the eight original pot blocks, vision-token forecasting removed the lid in 22 of 24 trials, versus 17 for the baseline and 15 for M = 16 special tokens.Stage annotations place the performance gap at the unfamiliar lid interaction, before object placement is completed.Source: §E.3 · p. 35

E.4 HELD-OUT REPRESENTATION DIAGNOSTICS

  • Vision-token forecasting is highest on both future gripper-width and future joint-motion readouts, while the larger-token controls reach its level on current container position but remain weaker on future readouts.The advantage over M = 16 is resolved for future gripper width, but not future joint motion; no forecasting policy improves present joint-angle decoding.Source: §E.4 · p. 35
  • At the trained 0.8 s horizon, the vision-minus-baseline future-gripper-width gap is 0.252 in main-camera tokens versus 0.021 in language tokens; the advantage is unresolved at 2.4 s.The additional future information is concentrated in the visual stream receiving the forecasting objective.Source: §E.4 · p. 35

F FORMAL ANALYSIS OF PREDICTION CREDIT ROUTES

The formal analysis separates prediction error from the route by which it reaches visual tokens, distinguishing explicit same-position residual credit from attention-mediated credit. Anchored rotary indices stabilize positional correspondence without guaranteeing gradient alignment, and structural decompositions alone do not determine gradient magnitude or performance.

F FORMAL ANALYSIS OF PREDICTION CREDIT ROUTES

  • Table E.5 measures retained information in real-robot main-camera tokens with episode-grouped held-out readouts; R2 is higher-is-better and position error lower-is-better.It compares native special-token indices with anchored indices.Source: Table E.5 · p. 36

F.1 GENERAL TOKEN-LEVEL DECOMPOSITION

  • The token-level decomposition separates upstream prediction error u_i from interface Jacobian B_ij, which determines how that error reaches visual token j.The per-token prediction contributions combine into the total auxiliary gradient at that visual token.Source: §F.1 · p. 36

F.2 DIRECT AND ATTENTION-MEDIATED CREDIT ROUTES

  • Special-token prediction reaches visual tokens through attention-mediated mixing, whereas the vision-token stream also has an explicit same-position identity route.Special tokens can still deliver visual gradients; their route lacks the vision-token interface’s same-position identity term.Source: §F.2 · p. 36

F.3 WHAT ROPE ANCHORING PROVIDES

  • Anchored rotary indices assign the same relative offset to corresponding spatial pairs across locations and instructions, giving each special token a prompt-invariant address to its matching visual position.Anchoring does not add a same-position identity term or guarantee that attention or gradients concentrate on that visual token.Source: §F.3 · p. 37

F.4 PROPAGATION THROUGH DEPTH AND ENCODER PARAMETERS

  • Depth adds contextual routes to both interfaces, but only the position-preserving vision stream retains an explicit same-position residual path.Post-normalization and dimensionality-changing projections modify that path without creating an identity route from a visual position to a distinct special-token position.Source: §F.4 · p. 37
  • The structural decomposition does not rank gradient magnitudes or downstream performance; contextual contributions may reinforce, redistribute, or cancel aligned components.The encoder Jacobian further transforms token-wise contributions, so these effects require empirical evaluation.Source: §F.4 · p. 38

G LIMITATIONS AND FUTURE WORK

  • The study fixes the predictive target to a fixed-horizon residual of momentum-encoded visual features, so it does not establish that one route is optimal for every predictive objective.Targets can differ in spatial granularity, temporal abstraction, and relevance to action.Source: G LIMITATIONS AND FUTURE WORK · p. 38
  • Predicting from every vision token is one direct-coupling design; sparse region-level or object-centric carriers and routing interventions remain candidates for testing transfer and extending causal evidence to jointly trained VLA policies.The proposed alternatives could preserve spatial addresses or residual links while allocating prediction capacity to changing scene regions.Source: G LIMITATIONS AND FUTURE WORK · p. 38

Related papers

Loading 2609.36645v2…