ContextVLA: Vision-Language-Action Model with Amortized Multi-Frame Context

• Many robotic tasks are non-Markovian because occlusion, prior robot motions, and moving-object trajectories require sequential observations.Source: §1 · p. 1
• Prior behavior-cloning studies report mixed effects from multi-frame observations, with some gains and some performance degradation.Source: §1 · p. 1
• VLAs mitigate the degradation seen in standard policies because their VLM component extracts more meaningful temporal context, but video inputs increase computational cost.Source: §1 · p. 2, Figure 2 · p. 2
• ContextVLA compresses past observations into one context token before using VLM features for autoregressive or diffusion-based action generation.Source: §1 · p. 2, Figure 3 · p. 3
• ContextVLA consistently improves VLA performance across manipulation benchmarks, including a 14.4% average success-rate gain for π0 on Simpler-WidowX.The average success rate increases from 41.8% to 56.2%.Source: §1 · p. 2

2.1 PRELIMINARIES

• The policy predicts l + 1 future actions from k + 1 observations and a language instruction, while longer histories increase computation and memory costs.Source: §2.1 · p. 3
• The VLA encodes multi-frame observations and instructions with a pre-trained VLM, then conditions either autoregressive or diffusion-based action generation on the extracted features.Source: §2.1 · p. 3, §2.2 · p. 4

2.2 CONTEXTVLA

• ContextVLA compresses past observations into a single context token at an intermediate VLM layer to reduce video-token computation and memory overhead.Average pooling produces the token, which is then processed with the current observation and language instruction.Source: §2.2 · p. 4
• The method processes all tokens through the first n VLM blocks, averages past-observation states into m, and processes m with the current observation through the remaining blocks.Source: §2.2 · p. 4
• The training objective uses next-token prediction for autoregressive actions or flow matching for diffusion-based actions.Source: §2.2 · p. 4
• KV-caching avoids reprocessing past observations at each timestep by reusing cached features and the precomputed context token.Source: §2.2 · p. 4

3.2 SIMULATED ROBOTIC TASKS

• ContextVLA improves baseline performance across Libero, Simpler-WidowX, and Robocasa, including 14.6% on Simpler-WidowX and 3.0% for π0-FAST on Robocasa PnP.The reported comparisons are 41.8% →56.2% for π0 on Simpler-WidowX and 45.5% →48.5% for π0-FAST on Robocasa PnP.Source: §3.2 · p. 6, Table 1 · p. 6, Table 2 · p. 6

3.3 REAL-WORLD ROBOTIC TASKS

• On real-world repetitive and sequential tasks, ContextVLA outperforms single-frame baselines and achieves the highest success rates on tasks requiring temporal context.These tasks include Clench/Unclench, PnP Twice, and CoverNStack, where single-frame policies often fail or succeed only partially.Source: §3.3 · p. 7, Table 4 · p. 7
• ContextVLA reaches 60% success on CoverNStack versus 45% for π0 trained with uncompressed 8-frame observations.The paper attributes this advantage to faster inference, noting that deployment latency can degrade real-world robot performance.Source: Table 4 · p. 7, §3.3 · p. 8

3.4 ANALYSIS AND ABLATION STUDY

• The compression scheme reaches π0’s best performance 5.5× faster on Libero and delivers 2.4× faster inference than uncompressed 8-frame observations.KV-caching further reduces inference latency by 33.6 ms.Source: §3.4 · p. 8, Figure 5 · p. 8, Table 5 · p. 8
• Performance increases from 52.0% with 4 frames to 56.2% with 8 frames on Simpler-WidowX, while compression at shallow depth n = 2 is optimal.Compression at shallow blocks also enhances training and inference efficiency.Source: §3.4 · p. 8
• Using the global context token improves Simpler-WidowX performance from 49.0% to 56.2%, supporting its role in retaining temporal information while reducing frame redundancy.The context-token variant also outperforms π0 trained on uncompressed 8-frame observations.Source: §3.4 · p. 9

• Existing VLAs commonly use single-frame observations, limiting performance on tasks that require temporal context.Source: §4 · p. 9
• Full multi-frame inputs preserve context but increase VLA computation and memory overhead, slowing training and inference.Source: §4 · p. 9
• TraceVLA summarizes observations through robot-trace tracking but still requires an external point-tracking model, making inference slow.Source: §4 · p. 9
• Prior efficiency strategies compress past observations, select key frames, summarize sequences, or freeze a visual encoder before multi-frame training.Source: §4 · p. 9

• ContextVLA compresses past observations into a single context token so VLAs can capture temporal context efficiently for action generation.Source: §5 · p. 9
• Experiments show that ContextVLA improves the performance of existing VLAs using multi-frame observations.Source: §5 · p. 9
• The method retains the benefits of multi-frame training while reducing training and inference time.Source: §5 · p. 9

• The implementation fine-tunes π0, GR00T N1.5, and π0-FAST using eight consecutive frames as multi-frame observations.Source: A IMPLEMENTATION DETAILS · p. 14
• Past observations are compressed into context tokens at the output of the second VLM block, using n=2.Source: A IMPLEMENTATION DETAILS · p. 14
• Simulated tasks use 60K, 30K, and 60K iterations for π0, π0-FAST, and GR00T N1.5, respectively.Source: A IMPLEMENTATION DETAILS · p. 14
• Real-world tasks train π0 + ContextVLA and GR00T N1.5 + ContextVLA for 30K iterations using 50 demonstrations for a single task.Source: A IMPLEMENTATION DETAILS · p. 14

Libero

• Libero uses a Franka robot across Spatial, Object, Goal, and Long sub-benchmarks, each containing 10 tasks.Source: §B.1 · p. 14

• All three benchmarks use end-effector position actions and report average success rates over repeated evaluations with varying random seeds or object positions.Source: §B.1 · p. 14

Simpler-WidowX

• Simpler-WidowX uses a WidowX robot and evaluates four tasks under a more challenging simulated setup.Source: §B.1 · p. 14

Robocasa

• Robocasa evaluates 24 tasks in simulated kitchen environments using demonstrations sampled from its machine-generated training dataset.Source: §B.1 · p. 14

B.2 REAL-WORLD ROBOTIC TASKS

• 50 demonstrations are collected per real-world task, and each model is evaluated 20 times per task.The task suite includes clenching and unclenching a humanoid hand, moving a cube between plates twice, and covering and stacking cups.Source: §B.2 · p. 15

C BASELINES

• The evaluated baselines include Octo, OpenVLA, RoboVLMs, TraceVLA, SpatialVLA, NORA, π0, π0-FAST, and GR00T N1.5, using varied VLA designs.Their differences include diffusion or autoregressive action generation, multi-frame processing, spatial encoding, and distinct action tokenizers.Source: C BASELINES · p. 16

D.1 AUTOREGRESSIVE MODELING.

• Autoregressive VLAs prompt a VLM with visual observations and instructions, generate discrete action tokens, and decode them into continuous actions.π0-FAST uses frequency-domain DCT transformation followed by quantization and BPE compression through the FAST tokenizer.Source: §D.1 · p. 16

D.2 DIFFUSION-BASED MODELING.

• Diffusion-based VLAs extract features from visual observations and instructions, condition a diffusion transformer, and generate actions by iterative denoising.Training adds Gaussian noise at a sampled timestep and learns the denoising direction; inference starts from random noise and applies an ODE or SDE sampler.Source: §D.2 · p. 17

E WEIGHT-INITIALIZATION ANALYSIS

• Vision-encoder-only initialization degrades with multi-frame inputs, whereas VLM or VLA initialization alleviates or overcomes this degradation.The analysis compares π0 and GR00T N1.5 under ViT-init, VLM-init, and VLA-init schemes.Source: E WEIGHT-INITIALIZATION ANALYSIS · p. 17, Figure 7 · p. 17

F QUALITATIVE RESULTS

• Single-frame policies often fail to determine the correct next action, while ContextVLA uses multi-frame observations to complete the task successfully.The comparison specifically attributes the improved action choice to information about previous robot movement.Source: F QUALITATIVE RESULTS · p. 18, Figure 8 · p. 18

G ADDITIONAL QUANTITATIVE RESULTS

• ContextVLA achieves performance comparable to, or surpassing, VLAs that use eight-frame observations without compression.The result indicates that the amortized context token captures temporal context effectively.Source: G ADDITIONAL QUANTITATIVE RESULTS · p. 18