What Matters in Building Vision-Language-Action Models for Generalist Robots

Motivation

• VLAs are a model-free policy family that uses pretrained VLMs as state encoders to map natural-language instructions to generalist policies, with promising semantic generality in simulation and real-world tasks [7] [20] [22].Their anticipated advantage is inheriting VLM representations learned from web-scale multimodal data, which may support adaptation to diverse open-world scenes with limited robot data.Source: §I · p. 1

Backbone choice

• The paper asks which VLM backbones best support robot control, since models differ in capacity, visual encoders, fusion, and training data, and their downstream manipulation effects remain understudied.Effective backbones must encode physical object properties, spatial relationships, and task-relevant semantics.Source: §I · pp. 1, 3

Data use

• The study also asks when extra robot data, including cross-embodiment data, should be used to improve VLA performance in tested scenarios.The considered data sources include in-domain robot data from other tasks and cross-embodiment data.Source: §I · p. 1, Figure 2 · p. 3

VLA formulation

• The paper compares VLA formulations by how they incorporate history and actions and whether actions are continuous or discrete, including one-step versus history modeling and interleaved versus policy-head aggregation.These formulations may differ in robustness, generalization, and data efficiency across environments and tasks.Source: §I · p. 3

3. When should we leverage the public diverse robot data for optimal training?

• Prior approaches either pre-train with cross-embodiment data before in-domain training [4] [54] or co-train with it [7] [20] [33] [38]; such pretraining is expected to support new skills with few demonstrations [12].Source: §3 · p. 3
• The study compares co-training with Open X-Embodiment, target-dataset finetuning, and co-training followed by target-dataset finetuning.The final configuration is referred to as post-training.Source: §3 · p. 4
• Post-training a cross-embodiment-pretrained model on the target dataset can improve performance, whereas cross-embodiment pretraining alone does not consistently improve final results.The study also finds that manipulation data from the same robots or tasks provides a clear performance boost.Source: §3 · p. 4

• The work aims to establish VLAs as robust generalist robotic policies by analyzing contemporary architectures and identifying the factors that drive performance.Source: §II · p. 4
• The study addresses four questions—why, which, how, and when—through six executable research problems and successive experiments.Extended Table 1 presents the questions and research problems.Source: §II · p. 4

A. Benchmarks and Evaluation Metrics

• The study benchmarks VLAs on CALVIN [30] and SimplerEnv [39] in simulation and on a real-world robot manipulation benchmark with more than 100 tasks.The benchmarks span simulation and real-world evaluation.Source: A. Benchmarks and Evaluation Metrics · p. 4
• CALVIN [30] evaluates multitask tabletop manipulation across four scene-setting splits, with success rates for 1–5 consecutive tasks and average achieved-task length.Its dataset provides 34 basic tasks and 24K language-instructed human teleoperation demonstrations.Source: A. Benchmarks and Evaluation Metrics · p. 4
• SimplerEnv [23] benchmarks simulated robot policies against private real-world settings, reporting each task’s success rate over valid trials.The suite includes settings such as Google Robot [6] [7] and Bridge V2.Source: A. Benchmarks and Evaluation Metrics · p. 4
• The Real Robot Benchmark [8] covers 105 manipulation tasks and evaluates models across 20 tasks, five settings per task, and three rollouts per setting.It uses over 70K teleoperated trajectories for fine-tuning; evaluation includes one Simple and four challenging Unseen settings, with average success rate reported for each setting, following [21].Source: A. Benchmarks and Evaluation Metrics · p. 4

• On CALVIN ABC-to-D transfer, it improves single-task execution by 12.6% and five-task execution by 30.3%; zero-shot rollouts average 4.25 of 5 tasks, 1.19 more than GR-1.Source: B. Why do we prefer VLAs? · pp. 5, 7
• RoboVLMs’ strongest VLA sets new state-of-the-art results on CALVIN and SimplerEnv, outperforming other robot policies by a clear margin.The results are reported for simulation benchmarks and real-world experiments.Source: B. Why do we prefer VLAs? · p. 5, Figure 4 · p. 6
• On SimplerEnv, RoboVLMs achieves the highest average performance on WidowX + Bridge and Google Robot; vision-language pre-training is essential for generalization and data efficiency.The study evaluates model scales from 3B to 9B and data scales from 10% to 500% of ABCD.Source: B. Why do we prefer VLAs? · p. 7
• In real-robot evaluations, KosMos P.H. achieves the best performance among Octo-Base, OpenVLA, and KosMos P.H. across all settings, especially Simple and Unseen Background.The benchmark covers 20 tasks and five settings, including novel skill descriptions, unseen distractors, target objects, and backgrounds.Source: B. Why do we prefer VLAs? · p. 7
• KosMos P.H. shows self-correction absent from the tested baselines, correcting end-effector positioning and future trajectories despite this behavior being absent from training data.Source: B. Why do we prefer VLAs? · p. 7

C. Which kind of VLM backbones are suitable for VLAs?

• The study compares off-the-shelf pretrained Flamingo, LLaVA, Qwen-VL, MoonDream, UForm, Paligemma, and KosMos backbones, training all models with static images.The backbones vary in architecture, training-data scale, model size, and latent embeddings; Flamingo is encoder-decoder, while the others are decoder-only.Source: C. Which kind of VLM backbones are… · p. 7
• KosMos [34] and Paligemma [3] perform distinctly better than the other tested VLM backbones.The paper attributes their advantage to sufficient pretraining on large vision-language datasets, which strengthens visual-linguistic alignment important for language-conditioned manipulation.Source: C. Which kind of VLM backbones are… · pp. 7–8

Question 3.1: How to select the best-performing VLA structure?

• Multi-step history with continuous actions performs best, and a policy head is the stronger, more efficient way to fuse history.History improves success rates over one-step inputs across encoder-decoder and decoder-only models; longer histories can help but add computation. Interleaved fusion performs worse and costs more memory and FLOPs than policy-head fusion.Source: D. How should VLAs be formulated to… · pp. 8–9

Question 3.2: How do different formulations affect the generalization and data efficiency for VLAs?

• The KosMos model with policy-head history fusion shows only a slight zero-shot performance drop and leads in data efficiency among the tested formulations.Its performance declines more slowly as training data is reduced; larger encoder-decoder models also tend to be more data efficient.Source: D. How should VLAs be formulated to… · p. 9

Question 3.3: How do training objectives and inference-time aggregation strategies impact the performance?

• Flow Matching slightly outperforms MSE+BCE, but the gap is not significant; full-chunk execution performs best in novel scenarios, while first-action execution underperforms.In seen scenarios, chunk execution is not significantly better than ensembling, though chunking can enable real-time deployment above 30 Hz. First-action execution is especially weak as sequential tasks progress.Source: Table II · p. 10, D. How should VLAs be formulated to… · pp. 10–11

Question 3.4: Does Mix-of-Expert structure benefit VLA learning?

• MoE improves visually zero-shot generalization under both MSE+BCE and Flow Matching, but does not improve performance on seen scenarios.The authors attribute this trade-off to MoE preserving pretrained vision-language representations, while omitting MoE allows more specialization to familiar tasks.Source: Table II · p. 10, D. How should VLAs be formulated to… · p. 11

E. When should we leverage the public diverse robot data for optimal training?

• The experiments compare three strategies for using external cross-embodiment datasets: co-training, post-training, and in-domain-only fine-tuning.The study frames these strategies around which data are most beneficial and when and how to use them effectively.Source: E. When should we leverage the public… · p. 11
• Co-training combines in-domain manipulation data with cross-embodiment data; RT-2 [7], OpenVLA [20], and OCTO [38] use this one-stage approach.The section notes that prior work has considered co-training alone.Source: E. When should we leverage the public… · p. 11
• Post-training first co-trains on in-domain and cross-embodiment data, then fine-tunes on in-domain tasks; π0 [4] adopts this approach.Source: E. When should we leverage the public… · p. 11
• The in-domain-only fine-tuning setting excludes cross-embodiment data and splits RT1 data into target tasks and extra in-domain data when RT1 contains the exact target tasks.Experiments use the best-performing KosMos backbone with a history-fusion policy head and Open X-Embodiment (OXE) [33] as the cross-embodiment dataset.Source: E. When should we leverage the public… · p. 11

D.1 In Domain Evaluation

• For Google Robot and Bridge, cross-embodiment co-training without post-training brought no substantial gains, while task-agnostic in-domain data improved Google Robot performance.RT Finetune, which adds same-robot data from different tasks, achieved higher success rates than RT-Partial Finetune on Google Robot; the authors conclude that in-domain data is essential for robust, consistent performance.Source: §D.1 · p. 12
• Post-training reached 50% average performance on Bridge versus 44% for in-domain-only fine-tuning, but on Google Robot it improved only pick-coke-can tasks and reduced performance on others.The authors attribute this trade-off to the high proportion of pick-and-place data in Open X-Embodiment: cross-embodiment co-training transfers high-frequency skills but can sacrifice lower-frequency skills such as moving near objects or opening and closing drawers.Source: §D.1 · p. 12, §D.2 · p. 12

D.2 Out-of-Distribution Evaluation

• Pretraining improved CALVIN few-shot performance by 17.2% in single-task execution and 0.25 more executed tasks per rollout.The evaluation sampled 10 trajectories per task from ABCD for training, tested on D, and used a single on-head camera view; the reported gains suggest faster adaptation to novel tasks, objects, and environments.Source: §D.2 · p. 12

VLM FFN

• The Mix-of-Expert design gives action and vision-language tokens separate self-attention projections and feed-forward networks, while action tokens interact with vision-language tokens through self-attention.The passage says this structure preserves the original VLM parameters and forward process and is claimed to benefit VLA generalization.Source: VLM FFN · p. 18
• The considered VLA formulations include policy-head-continuous, one-step-discrete-action, and interleaved-continuous-action designs; RoboFlamingo uses the first, RT-2 and OpenVLA the second, and Octo and GR the third with a fixed window size.Source: VLM FFN · p. 18
• Bridge Post Train uses cross-embodiment co-training with Bridge V2 and RT1 data, followed by refinement using only Bridge V2.The co-training stage includes cross-embodiment data, Bridge V2, RT1 target data, and RT1 extra data.Source: VLM FFN · p. 19
• The cross-embodiment ablation tests Bridge finetuning, OXE co-training, and post-training on WidowX+Bridge, and four corresponding recipes on Google Robot.The Google Robot recipes are RT-Partial Finetune, RT Finetune, OXE Co-Train, and Post-Train.Source: VLM FFN · p. 19
• LLaVA and Qwen behave badly without an additional resampler to downsample image tokens.Extended Table 5 reports models trained for 5 epochs on ABCD splits, using a single side-view image for fair comparison.Source: VLM FFN · p. 22

Discussion

• Across three simulators and more than 240 real-world rollouts in 20 tasks, the experiments position VLAs as promising generalist policies and identify beneficial design choices.VLMs with sufficient large-scale vision-language pre-training were suitable backbones; integrating history observations was essential, and a policy head was more effective and efficient than interleaving for history aggregation. Extra in-domain data was beneficial, while large-scale cross-embodiment pre-training further improved overall and few-shot performance.Source: Appendix A · p. 26
• Adding a perceiver resampler after the vision encoder substantially improved Qwen-VL- and LLaVA-based VLAs, which otherwise underperformed relative to their vision-language task performance.The authors hypothesize that the gain relates to image resolution and the number of vision tokens in the input sequence.Source: Appendix A · p. 26
• The study leaves action tokenization, policy heads, and training objectives underexplored, considers a limited set of VLM backbones, and faces the challenge of real-time deployment of large models.VQ-VAE, VQGAN, and FAST action tokenizers are specifically noted as under-explored for VLAs.Source: Appendix A · p. 26
• Future work aims to develop policies that handle long-horizon complex instructions, reason through executable actions step by step, and generate meaningful physical interactions.The discussion presents these capabilities as goals beyond semantic generalization.Source: Appendix A · p. 26
• The authors open-sourced the codebase, detailed guidelines, weights for RoboVLMs’ strongest VLAs, and the real-world dataset used in the experiments.They anticipate this release will support progress in vision-language learning and foundational models for robotics.Source: Appendix A · p. 26

Hyper-parameters and Training Details

• Hyperparameter settings can vary by VLA formulation, and a varied setting can improve performance over reusing VLM pretraining settings.The authors grid-search weight decay from [0, 1e −1] and learning rate from [1e −4, 5e −5, 2e −5, 1e −5], using global batch size 128 and a warmup ratio of 0.25 epoch; models train on 4 x 8 A100-80G GPUs.Source: Appendix B · p. 26

Checkpoint selection

• Because offline metrics may not predict long-horizon rollout performance, checkpoint selection is challenging; all VLAs train for fixed epochs or timesteps for fair comparisons.CALVIN reports the final model after 5 epochs; SimplerEnv reports the best model at 10K-iteration intervals during 100K iterations; real-world experiments report the last model after 5 epochs.Source: Appendix B · p. 26

APPENDIX C BENCHMARK DETAILS

• CALVIN evaluation uses 24k language-annotated demonstrations, tests on split D with 1,000 rollouts of five consecutive tasks, and reports sequential success rates and completed-task length.The benchmark contains 34 predefined skills and four scene splits; trajectories are shorter than 64 time steps.Source: Appendix C · p. 27
• Google Robot tests cover Coke-can picking, moving objects near one another, drawer operation, and a sequential open-drawer-then-place-apple task.The setups specify 75, 60, 54, and 27 trials, respectively.Source: Appendix C · p. 27
• WidowX + Bridge tasks include spoon-on-towel, carrot-on-plate, block stacking, and placing an eggplant into a yellow basket.The spoon, block-stacking, and eggplant tasks each specify 24 trials; the carrot setup is described as similar to spoon-on-towel.Source: Appendix C · pp. 27–28

APPENDIX D DETAILED PERFORMANCE ON CALVIN

• Supplementary Table 2 covers CALVIN generalization and data efficiency across settings, including models with or without vision-language pretraining and 5x regenerated training data.The caption says 5x results are at one epoch, while other results use the best-behaved checkpoints within five epochs.Source: Table 2 · p. 29

APPENDIX E DIVERSE BACKBONE

• Supplementary Table 3 reports CALVIN sub-task success rates across VLM backbones, with all models trained for five epochs on ABCD splits.Source: Table 3 · p. 30

APPENDIX F DIVERSE PH

• Supplementary Table 4 reports CALVIN sub-task success rates across training splits and VLM backbones, with models trained for up to five epochs.Source: Table 4 · p. 31

APPENDIX G ROLLOUT EXAMPLES IN SIMPLERENV

• Supplementary Figure 1 presents qualitative rollout results for SimplerEnv tasks.Source: Figure 1 · p. 32

APPENDIX H ROLLOUT EXAMPLES IN REAL-WORLD EXPERIMENTS

• Models are evaluated across 20 real-world tasks, each with 5 rollouts, involving unseen distractors, backgrounds, target objects, and novel skill descriptions; some tasks exclude unseen-target-object settings when suitable alternatives are unavailable.Source: Figure 2 · p. 33
• Typical failures include premature gripper closure, inaccurate grasps, repeated oscillations, and picking up an object without placing it correctly.Supplementary Figure 6 illustrates these failure cases.Source: Figure 6 · p. 37