Fast-VLA: You Only Think Once for Robot Manipulation

1 Introduction

• VLA policies commonly rerun a large backbone for each action chunk, although task understanding changes more slowly than the scene and robot state.The section asks whether task context can be computed once while control continues to adapt to changing observations.Source: §1 · p. 2
• A frozen 4B VLM builds fixed, multi-layer task memory from the instruction and initial views once per episode, while an online actor uses live visual and proprioceptive feedback to retrieve context and predict action chunks.The actor forms action queries from learned queries and current feedback, then retrieves task context across VLM depths in one forward pass.Source: §1 · p. 2, Figure 2 · p. 3
• Fast-VLA reaches 92.46% success on RoboTwin 2.0 Clean, 92.40% under domain randomization, 95.95% on LIBERO, and 79.11% on LIBERO-Plus, with 10.9–17.1× lower RoboTwin amortized inference time than three same-hardware baselines.The inference-time comparison includes the one-time initial call; post-prefill online resident memory is 1.22 GiB.Source: §1 · p. 2

2 Related Work

• Fast-VLA shifts the efficiency focus from reducing work per backbone call to invoking the frozen VLM only once per episode.Unlike HyperVLA, it keeps the online actor shared across episodes and retains task context as explicit layer-wise key–value memory.Source: §2 · p. 3
• Other approaches reuse stable visual tokens, omit future generation, sparsify video–action interaction, or attach a shallow action module, but retain a large visual or video backbone in the repeated control path.Fast-VLA instead fixes task context at episode start and delegates changing-state estimation to a lightweight online actor.Source: §2 · p. 3

3.1 Overview

• At each action-chunk boundary, the actor combines current visual and proprioceptive observations with fixed episode memory to predict the next chunk.For single-arm LIBERO, the design uses agent-view and wrist images, one action-query set, and a 10 × 7 action chunk without visual history.Source: §3.1 · p. 4, §3.3 · p. 5

3.2 Episode-level task memory

• A frozen 4B VLM contextualizes the instruction and initial views once per episode, retaining key-value tensors from all 34 layers as fixed task memory.The VLM leaves the control loop after prefill.Source: §3.2 · p. 4

3.3 Online actor

• The online actor fuses learned time- and arm-indexed queries with current arm state and wrist-view features to form action queries.Each query reads visual tokens and task memory before the action head predicts a chunk in one forward pass.Source: §3.3 · pp. 4–5

3.4 Training and inference

• The online actor, including its visual encoders, is trained with an ℓ1 loss on normalized actions; inference uses one VLM prefill followed by one actor call per action chunk.Amortized latency is computed from prefill and actor time over observed episode lengths.Source: §3.4 · p. 5

4.1 RoboTwin 2.0: performance and inference cost

• 92.46% Clean and 92.40% Randomized success on RoboTwin 2.0; Fast-VLA is at least 10.9× faster than measured baselines with at least 90% mean success.Against DM0.5, mean success is 1.02 percentage points lower with 11.2× faster amortized inference at the same 50-step replanning interval; against Fast-WAM, it scores 0.60 points higher with a 17.1× speedup.Source: Table 2 · p. 6, §4.1 · p. 6

4.2 LIBERO and LIBERO-Plus

• 95.95% standard LIBERO and 79.11% LIBERO-Plus success; on LIBERO-Plus, Fast-VLA exceeds Faster-WAM (HUST) by 5.54 points with 21.2× faster amortized inference.It also exceeds Faster-WAM (Noah) by 4.11 points with 3.1× faster amortized inference, although standard LIBERO success is lower; it trails π0.5 and DM0.5 on LIBERO-Plus by 6.59 and 6.79 points.Source: §4.2 · p. 6, Table 3 · p. 7

4.3 Qualitative rollouts and live visual grounding

• In the illustrated rollouts, task memory is built from initial observations and reused, including a successful LIBERO-Plus drawer task despite corrupted third-person vision.Across the stack-blocks rollout, head-camera attention shifts from the red to green block as control passes between arms, while each wrist focuses on its arm’s object.Source: §4.3 · p. 6, Figure 3 · p. 7, Figure 4 · p. 8

4.4 Ablations and Analysis

• Swapping task memory while keeping actor weights fixed sharply reduces success: other-episode memory for the same task reaches only 6.26%/8.36%.Swapping the language segment yields 35.18%/29.66%, while replacing the initial-view segment causes a larger collapse to 2.64%/2.62%.Source: §4.4 · pp. 7–8, Table 4 · p. 8
• Full instruction-and-visual memory across multiple VLM layers gives the highest success on both LIBERO benchmarks, at 95.95/79.11.Removing task memory yields 69.90/53.69; text-token-only memory reaches 93.20/73.84, and last-layer memory reaches 94.70/73.70.Source: §4.4 · p. 8, Table 5 · p. 8
• Wrist vision is a key closed-loop input: wrist-only success is 87.10/68.72, versus 70.80/30.80 for head-only input.Without wrist vision, success is 65.60/33.49; without proprioception, it is 94.30/72.41.Source: §4.4 · p. 8, Table 6 · p. 9
• Refreshing memory every action chunk changes success by +1.80 points on LIBERO and −1.23 points on LIBERO-Plus, while increasing foundation-model calls 17.37×.Amortized inference time rises 3.12×, from 3.975 to 12.391 ms per environment step, relative to once-per-episode construction.Source: §4.4 · p. 8, Table 8 · p. 9
• Fast-VLA achieves 94.00%, 90.27%, and 90.25% success on short-, medium-, and long-horizon tasks; it reaches 87.00% on a 1,700-step task using up to 34 actor calls with the same memory.Memory-encoder choice also affects results: PaliGemma’s 9.8-MB cache yields 73.15% LIBERO-Plus success, versus DM0.5’s 79.11% with a 79.7-MB cache.Source: §4.4 · p. 9, Table 9 · p. 9

• Fast-VLA uses a frozen 4B VLM to build episode-level task memory, while an online actor uses visual and proprioceptive feedback to produce action chunks.Source: §5 · p. 10
• Fast-VLA reaches 92.46%/92.40% success on RoboTwin Clean/Randomized and 95.95%/79.11% on standard LIBERO/LIBERO-Plus.Source: §5 · p. 10
• Fast-VLA has 10.9–17.1× lower RoboTwin amortized inference time than Faster-WAM (HUST), DM0.5, and Fast-WAM; task-memory swaps show reliance on episode-matched memory.Source: §5 · p. 10

AI Use Statement

• The authors used AI tools for related-work exploration, research ideation, and English polishing, and take full responsibility for the paper’s content.Source: AI Use Statement · p. 10

A Architecture and training configuration

• Table 10 covers architecture and training configuration for the RoboTwin 2.0 and LIBERO models; each camera has a separate trainable DINOv2 encoder, and batch sizes are effective global batch sizes.Source: A Architecture and training… · p. 13, Table 10 · p. 13

B Latency and memory measurement protocol

• Table 11 specifies the execution settings and measurement scope for inference profiling.Source: B Latency and memory measurement… · p. 13, Table 11 · p. 13

Inference profiling

• Profiling covered 10,000 RoboTwin episodes and 2,372,728 executed environment steps, accounting for each method’s call schedule.Fast-VLA’s reusable task memory occupies 174.6 MiB of its 1.22 GiB post-prefill online resident memory in the RoboTwin configuration.Source: C Additional evaluation details · p. 14

Sources of baseline success rates

• Benchmark baseline success rates are drawn from cited papers, benchmark compilations, official leaderboards, or released-checkpoint evaluations, with latency and memory measured separately.The passage identifies exceptions and variants, including RGB-only LingBot-VLA and select entries evaluated from released checkpoints.Source: C Additional evaluation details · p. 14

Task-horizon evaluation

• Task-horizon results pool Clean and Randomized outcomes within official step-budget groups, using equal episode counts across tasks and settings.Fast-WAM was evaluated on 50 tasks with 100 episodes per task in each setting; its results are sourced separately from aggregate Table 2 results.Source: C Additional evaluation details · p. 14

D Qualitative rollouts

• Figure 5 shows three successful RoboTwin 2.0 and three LIBERO-10 rollouts; Figure 6 shows the same LIBERO-Plus drawer task under five perturbations.The perturbations are camera viewpoint, background texture, lighting, object layout, and robot initial state; memory is built from perturbed initial observations and stays fixed during each episode.Source: D Qualitative rollouts · p. 14

E Limitations

• Task memory is constructed once at episode initialization and remains fixed throughout execution.The successful-rollout examples likewise use a single memory without refresh, including perturbed LIBERO-Plus initial observations.Source: E Limitations · p. 14, Figure 5 · p. 15, Figure 6 · p. 16
• Handling changed task goals or newly introduced instructions during an episode remains unexplored.The limitation concerns dynamic changes to the task context, not the actor’s continued response to observations.Source: E Limitations · p. 14
• Event-triggered memory updates and high-level replanning are proposed as possible extensions for dynamic settings.The passage frames these as prospective extensions that could retain memory reuse efficiency between updates.Source: E Limitations · p. 14