PaperlayerView exact revision on arXiv

Source-linked AI summary

RecastVLA: From Past Interaction to Future Control with Adaptive Policy States

Wenbo Li, Jun Yang, Yiteng Chen, Wei Zhang, Qingyao Wu

arXiv:2609.32155v1cs.RO

TL;DR

Sequential manipulation requires tracking earlier events that the current scene may not reveal, and RecastVLA forms a shared adaptive policy state from action-generation features. It improves mean success over a matched policy across simulation benchmarks and real-robot tasks, including a 10.68-point gain on RoboTwin Clean→Clean.

  • Problem

    Sequential manipulation can depend on earlier events, such as whether a visually similar button has already been pressed, that current observations may not reveal.

    Source: §1 · p. 1

  • Method

    RecastVLA uses depth-specific interfaces to read shared fast weights, keeps the state fixed during action generation, and updates it once for the next call using policy-generated features.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    RecastVLA improves success over Matched Base in all six reported settings, including 10.68 percentage points on RoboTwin Clean→Clean.

    Source: §4.1 · p. 6

  • Takeaways & Limitations

    Retaining state improves within-episode control on RoboTwin, and the shared design achieves higher success than its layer-local counterpart under the tested per-state architecture.

    Source: §5 · p. 9

    Training uses four adjacent demonstration frames, while deployment uses successive action chunks and policy-generated write features; shared-state interference, restricted temporal credit assignment, and real-robot errors remain.

    Source: §5 · p. 9

Abstract

from arXiv · show

Sequential manipulation requires a robot to track what has already happened, even when the current scene no longer reveals it. Policies with explicit history representations make past interactions available as context for current decisions. We ask how action generation itself can form a persistent state for subsequent control. Building on action-side test-time training, RecastVLA maintains an adaptive policy state within a flow-matching vision-language-action policy. The state is represented by shared fast weights and remains fixed throughout action generation. Depth-specific interfaces read the same state, while features across depths and flow evaluations jointly define one update for the next policy call. Subsequent action losses train the initialization, interfaces, and update rule by differentiating through earlier state transitions. At deployment, updates use the policy's own action-generation features without expert action labels. Across LIBERO, RoboTwin, RoboDojo, and twelve real-robot tasks, RecastVLA improves mean success over a matched policy trained without test-time training, including 10.68 percentage points on RoboTwin Clean-to-Clean. In controlled RoboTwin comparisons, retaining state improves success, and the shared design exceeds independently trained layer-local TTT by 2.58 points.

1 INTRODUCTION

Sequential manipulation requires tracking earlier events that may no longer be visible, motivating execution memory beyond reusable manipulation skills. RecastVLA makes action generation form a shared adaptive policy state that guides subsequent decisions.

1 INTRODUCTION

  • Sequential manipulation may require remembering earlier events, such as whether a visually similar button has already been pressed.Occlusion and partially completed instructions create similar needs to connect current observations with earlier events.Source: §1 · p. 1
  • History-conditioned policies provide retrieved observations, compressed context, or recurrent representations, while RecastVLA asks how action generation can form persistent state for subsequent decisions.MemoryVLA and MEM use perceptual and semantic memories to support control dependent on earlier events (Torne et al., 2026).Source: §1 · p. 1
  • Action-side TTT places history in small-model weights, and RoboTTT applies this principle in an action expert using trajectory supervision (Jiang et al., 2026).The introduction contrasts layer-local fast weights with jointly forming and accessing a common state.Source: §1 · p. 1
  • Figure 1 contrasts context tokens from retrieved or compressed history, independent layer-local fast weights, and RecastVLA’s shared state with depth-specific interfaces.RecastVLA combines features into one update per policy call.Source: Figure 1 · p. 2
  • RecastVLA uses shared fast weights, read through depth-specific interfaces, to form a persistent policy state from action-generation features.The incoming state remains fixed throughout a call; features jointly define one update for the next call, and later action losses train the initialization, interfaces, and update rule.Source: §1 · p. 2
  • The evaluation covers LIBERO, RoboTwin, and RoboDojo, with state-continuity and imported-history controls, a layer-local comparison, and twelve real-robot tasks.The real-robot tasks examine progress tracking and ordering constraints while revealing remaining geometric failures.Source: §1 · p. 2

2 RELATED WORK

Prior work studies history through retrieval, compressed representations, temporal prediction, and test-time adaptation. RecastVLA focuses on shared state updated by action-generation features across depths and flow evaluations.

Long-Context Robot Policies

  • Long-context policies retain earlier experience through retrieval or compression, while Past-Token Prediction trains diffusion policies to model dependencies between earlier and later actions.MemoryVLA and Zeva retrieve memory tokens and interaction context (Chen et al., 2026a); MEM combines compressed recent video with longer-term language memory (Torne et al., 2026).Source: §2 · p. 2

Test-Time Training

  • RoboTTT places fast-weight layers in a diffusion action expert and learns their updates through sequential flow-matching losses (Jiang et al., 2026).It is identified as the closest precedent for RecastVLA’s action-side recurrence and supervision; WAM-TTT adapts video-side memory for a world-action model, while VITA targets value estimation (Feng et al., 2026; Ziakas & Russo, 2026).Source: §2 · p. 2

Test-Time Training; Shared Recurrent State Across Network Depth

  • RecastVLA combines depth-specific access with one shared state, keeps it fixed during action generation, and updates it once for the next policy call.Related work connects this design to cross-depth sharing and aggregate writes, with subsequent action losses training a joint update from multiple depths and flow evaluations.Source: §2 · p. 3

3 LEARNING EXECUTION MEMORY THROUGH ADAPTIVE POLICY STATES

RecastVLA represents execution memory as shared fast weights: each policy call reads the state to generate an action chunk, then uses internal features to update it for the next call.

3.1 ADAPTIVE POLICY STATE FOR SEQUENTIAL CONTROL

  • All computations within a policy call read the incoming state; the learned initialization starts each episode, and deployment updates the state using action-generation features.At deployment, the update occurs before physical execution and does not require feedback on the chunk’s outcome.Source: §3.1 · p. 3

3.2 SPECIALIZED ACCESS TO SHARED EXECUTION MEMORY

  • Depth-specific interfaces let different layers query the shared state in their own feature coordinates, allowing information written at one depth to affect later reads at another.The interfaces use learned latent slots as a compact state interface.Source: §3.2 · p. 4

3.3 FORMING MEMORY FROM ACTION-GENERATION FEATURES

  • Features across access depths and flow evaluations jointly define one update for the next call, separating the fixed within-call state from across-call adaptation.Subsequent action losses train the write projections and update rule through their effects on later predictions.Source: Figure 2 · p. 3, §3.3 · p. 4

3.4 LEARNING FROM SUBSEQUENT CONTROL

  • Ordered training windows let later action losses backpropagate through earlier state transitions to train the initialization, interfaces, and update rule.At deployment, iterative action-generation features define the update; no expert action labels are required for that update.Source: §3.4 · pp. 4–5

4 EXPERIMENTS

RecastVLA is evaluated on three simulation benchmarks, controlled RoboTwin state comparisons, and twelve real-robot tasks. It improves over Matched Base across all six reported simulation settings, while controls show benefits from state continuity and shared state organization.

4.1 MAIN RESULTS

  • 10.68 percentage points on RoboTwin Clean→Clean; RecastVLA improves success over Matched Base in all six reported simulation settings.The results also include gains on LIBERO and RoboDojo, while Clean→Random remains substantially below Clean→Clean.Source: Table 1 · p. 6, §4.1 · p. 6

4.2 DOES ACCUMULATED HISTORY IMPROVE CONTROL?

  • 4.68 percentage points higher SR for Full than Reset; Similar-task initialization raises SR by 2.22 points, while Other-task initialization lowers it by 11.98 points versus Full.Other also underperforms Reset by 7.3 points (60.4% versus 67.7%); these imported states continue updating during the target episode.Source: §4.2 · pp. 6–7, Table 2 · p. 7

4.3 HOW DOES STATE SHARING AFFECT CONTROL?

  • 2.58 percentage points higher success for RecastVLA than layer-local TTT (72.38% versus 69.8%), despite layer-local TTT having eight times the persistent-state capacity.The 10.68-point gain over Matched Base measures the complete TTT architecture.Source: §4.3 · p. 7, Table 3 · p. 7

4.4 REAL-ROBOT EVALUATION

  • Across 240 trials per policy, RecastVLA succeeds in 131 (54.58%), versus 102 (42.50%) for π0.5 and 74 (30.83%) for Matched Base.Policies were fine-tuned on the same 550 demonstrations and evaluated on twelve held-out task variants.Source: §4.4 · p. 8, Table 4 · p. 8
  • Against π0.5, the advantage concentrates in button sequencing and repeated pressing: T1–T3 achieve 66.67% success versus 23.33%, accounting for 26 of 29 net additional successes.The remaining nine tasks average 50.56% versus 48.89%; some placement tasks still favor π0.5.Source: §4.4 · p. 8
  • Supplementary executions preserve task order after perturbations, but observed failures include extra button presses, mixed-color stacking, cup-rim misalignment, and unstable stacks.These illustrative executions are separate from the scored trials; ordered handling did not ensure accurate placement in T7.Source: §4.4 · pp. 8–9

5 CONCLUSION

RecastVLA uses shared fast weights updated from action-generation features as execution memory, improving mean success across simulation and real-robot evaluations. The paper also identifies training, state-interference, and execution-error limitations, and proposes longer-horizon training as future work.

5 CONCLUSION

  • RecastVLA improves mean success over Matched Base across three simulation benchmarks and twelve real-robot tasks.On RoboTwin, retaining state improves within-episode control; prior-state effects depend on the source, and the shared design outperforms its layer-local counterpart under the tested architecture.Source: §5 · p. 9
  • Shared fast weights updated from action-generation features form execution memory, while subsequent action losses train the initialization, interfaces, and update rule.Source: §5 · p. 9
  • Training uses four adjacent demonstration frames, unlike deployment’s successive action chunks and policy-generated write features; shared-state interference, restricted temporal credit assignment, and real-robot errors remain.Reported errors include extra button presses and cup-rim misalignment.Source: §5 · p. 9
  • Future work asks how to train shared states over longer interaction horizons and shape which information continues to affect control as tasks progress.Source: §5 · p. 9

AI DISCLOSURE

Selected sentences were language-polished with generative AI tools; the authors reviewed and verified all scientific content, analyses, and reported results.

AI DISCLOSURE

  • Generative AI tools polished selected sentences, while authors reviewed and verified the scientific content, analyses, and reported results.Source: AI DISCLOSURE · p. 9

A.1 TRAINING DATA, CONFIGURATION, AND EVALUATION SCOPE

The appendix specifies the simulation training data, evaluation scope, and shared-state implementation, then describes a layer-local control matched in architecture and update timing. The control uses eight independent fast-weight states rather than one shared state.

A.1 TRAINING DATA, CONFIGURATION, AND EVALUATION SCOPE

  • LIBERO training combines four suites with 432, 454, 428, and 379 demonstrations; RoboTwin uses 50 clean demonstrations per task, and RoboDojo uses 3,500 demonstrations from 35 tasks.RoboTwin randomized scenes are used only for evaluation.Source: §A.1 · p. 13
  • Training jointly updates the backbone, action expert, and TTT slow parameters, while Matched Base freezes and disables TTT modules under the same four-frame action objective.The matched policies use 100,000 optimizer updates, four windows per update, window length T = 4, and frame stride one; flow-matching replicas reuse observations.Source: §A.1 · p. 13
  • Matched Base and RecastVLA use one evaluation run per setting; LIBERO Avg includes 2,000 trials, RoboTwin uses 5,000 trials per setting, and RoboDojo reports 400 Long-Horizon and 300 Memory trials.Task success rates are averaged within each category; equal task trial counts make these averages equal pooled episode success rates.Source: §A.1 · p. 13, Table 5 · p. 13

A.2 LAYER-LOCAL TTT CONTROL

  • The layer-local control gives each of eight access depths an independent fast-weight module, using the shared design’s per-module architecture and call-boundary update timing.It uses eight states totaling 2,103,296 values, compared with RecastVLA’s single 262,912-value state; Figure 5 contrasts independent states with depth-specific interfaces reading one shared state.Source: §A.2 · pp. 13–14, Figure 5 · p. 14

A.3 TASK GROUPS AND AVERAGING

The state-control experiment groups tasks by their defining interaction to select eligible source histories, while scoring target tasks equally. For target episodes, Full, Similar, and Other begin from specified states and continue ordinary updates; Reset restores the learned initial state before every call.

A.3 TASK GROUPS AND AVERAGING

  • The ten task groups assign each of the 50 tasks by its defining interaction, with the distinguishing requirement taking precedence for compound tasks.Examples include drawer opening, shoe orientation, and arm-to-arm transfer.Source: §A.3 · p. 14
  • Similar sources are other tasks in the target’s group, whereas Other sources are tasks outside that group; another episode of the target task is excluded.Group membership selects eligible histories; target success measures whether imported experience helps.Source: §A.3 · p. 14
  • Each target task receives equal weight, groups determine history-source eligibility only, and source episodes do not contribute to target scores.NCS is averaged over complete target episodes within each task and then equally across tasks.Source: §A.3 · p. 14

A.4 CLOSED-LOOP STATE CONTROLS

  • Similar and Other initialize the target from one complete source rollout, then continue ordinary read–generate–update steps; Reset restores W0 before each call.All conditions use the same checkpoint, observation interface, action chunking, and slow parameters; source histories are retained regardless of source-episode outcome.Source: §A.4 · pp. 14–15
  • Conditions share the target task, initial environment seed, and policy sampling randomness, and each rollout ends at the benchmark’s success criterion or episode limit.Source: §A.4 · p. 15

A.5 STATE-CONTROL METRICS

The section defines episode-level success and normalized completion steps to assess target-task execution. NCS penalizes failures and reflects completion-budget use among successful trials.

A.5 STATE-CONTROL METRICS

  • Success rate uses the benchmark’s binary criterion over the complete target episode.The episode-level success indicator is the first successful environment action step under the condition.Source: §A.5 · p. 15
  • NCS assigns failures a score of one and measures target execution without counting the cost of generating source histories.Lower NCS reflects higher success, a smaller completion-budget fraction among successful trials, or both.Source: §A.5 · p. 15
  • NCS combines the failure fraction with the mean completion-step fraction among successful trials under the equal-count protocol.The reported derived fractions are approximately 30.16%, 30.18%, 29.21%, and 47.53% for Full, Reset, Similar, and Other, respectively.Source: §A.5 · p. 15
  • The listed task groups cover pouring and shaking, articulated opening, direct pressing, and tool use, with their benchmark task identifiers.The supplied group entries include G7–G10 and examples such as shake_bottle, open_microwave, press_stapler, and scan_object.Source: §A.5 · p. 16

B REAL-ROBOT SETUP AND EXECUTION SEQUENCES

The real-robot evaluation uses three policies trained on the same demonstrations and tests held-out variants across twelve tasks. The first three tasks probe whether the policy tracks button identity, sequence, and press count as visible scenes change little.

B REAL-ROBOT SETUP AND EXECUTION SEQUENCES

  • The twelve-task evaluation compares π0.5, Matched Base, and RecastVLA on held-out variants using the same 550 demonstrations from 55 training tasks.The variants involve new operation sequences or combinations and new scene or object configurations; Table 8 reports 20 trials per policy and task.Source: B REAL-ROBOT SETUP AND EXECUTION… · p. 16, Table 8 · p. 17
  • Button-task success requires the exact instructed press sequence and count, followed by 10 seconds without another press while the policy remains active.The real-robot sequences are illustrative; task results are reported separately in the quantitative tables.Source: B REAL-ROBOT SETUP AND EXECUTION… · p. 16
  • Button releases can leave a scene resembling an earlier step, whereas packing and stacking often leave visible progress cues.These tasks therefore place different demands on history and current perception.Source: B REAL-ROBOT SETUP AND EXECUTION… · p. 17

B.1 T1: ORANGE ONCE, THEN BLUE TWICE

  • T1 requires orange–blue–blue; the illustrated execution shows the two blue contacts separated by withdrawal, while another execution returned to orange.The return to orange violates the instructed sequence.Source: §B.1 · p. 17, Figure 6 · p. 17

B.2 T2: BLUE ONCE, THEN ORANGE TWICE

  • T2 reverses the required sequence to blue–orange–orange, distinguishing instruction progress from button identity because the same objects demand a different continuation.A reported failure returns to blue after the initial press.Source: §B.2 · p. 18, Figure 7 · p. 18

B.3 T3: PRESS BLUE THREE TIMES

  • T3 fixes the target as blue but requires stopping after three presses; reported failures include losing track of the completed count.Completion requires returning to the same button and ceasing presses at the requested count.Source: §B.3 · p. 18, Figure 8 · p. 18

B.4 T4: PACK FRUIT, BANANA LAST

The execution sequences illustrate varied multi-step manipulation demands, from preserving order through scene changes to grasp adjustment, alignment, and tool use. They also show that following a required order does not guarantee accurate placement.

B.4 T4: PACK FRUIT, BANANA LAST

  • The robot preserves the banana-last constraint through fruit and box movements, including repositioning and partial concealment, in illustrative executions separate from scored trials.It places the watermelon before the banana despite the scene changes.Source: §B.4 · p. 18, Figure 9 · p. 19

B.5 T5: STACK BLOCKS BY COLOR / B.6 T6: STACK BLOCKS

  • T5 requires separate same-color blue and yellow stacks; the illustrated execution builds them in sequence, while color mixing is a characteristic failure.T6 shows renewed grasping and placement after a stack collapses, but alignment, clearance, and physical stability remain limiting factors.Source: §B.5 · p. 19, Figure 10 · p. 19, §B.6 · p. 20

B.7 T7: PLACE TOOTHBRUSH, THEN TOOTHPASTE

  • In T7, the robot handles the toothbrush before the toothpaste but leaves the toothbrush across the cup rim, separating correct order from secure placement.The toothpaste is then lowered into the opening.Source: §B.7 · p. 20, Figure 12 · p. 21

B.8 T8: PLACE BALLS INTO A PAPER CUP

  • T8 successively places three balls into one cup, combining small-object grasping with repeated alignment to a receptacle.The changing tabletop objects can provide a visible progress cue for later placements.Source: §B.8 · p. 21, Figure 13 · p. 21

B.9 T9: STACK PAPER CUPS IN ORDER

  • T9 nests the red cup into the pink group, then transports and aligns the larger combined assembly in the yellow group.The first nesting changes the grasped object for the second operation.Source: §B.9 · p. 22, Figure 14 · p. 22

B.10 T10: FOLD A TOWEL INTO THIRDS

  • T10 uses grasp adjustments and two opposite-end folds to form a towel folded into thirds, with local retries between manipulation phases.The sequence includes a transition between phases on a deformable object.Source: §B.10 · p. 22, Figure 15 · p. 22

B.11 T11: ORIENT A CUP AND REMOVE ITS LID

  • T11 switches contact targets on one cup: the handle reorients it before the knob is grasped to remove and set down the lid.Source: §B.11 · p. 23, Figure 16 · p. 23

B.12 T12: WIPE THE TABLE WITH A SPONGE

  • T12 carries a grasped sponge through multiple tabletop wiping positions before release.The sequence depicts an extended manipulation phase with a tool.Source: §B.12 · p. 23, Figure 17 · p. 23

Related papers

Loading 2609.32155v1…