Benchmarking Vision, Language, & Action Models on Robotic Learning Tasks

• Robotic systems often generalize poorly to unfamiliar instructions, object configurations, lighting, occlusion, novel objects, and clutter, hindering unconstrained deployment.The cited examples span task descriptions [6], spatial variation [5], lighting and occlusion, and novel objects or clutter [39].Source: §1 · p. 1
• Unified sequence architectures can use diverse training data without custom domain-specific policy architectures, and have shown gains with scale.The paper situates this approach within a broader trend toward unified neural sequence architectures [19] [15] and general-purpose models.Source: §1 · pp. 1–2
• Foundation models’ semantic and visual capabilities do not resolve the challenge of generating precise, physically grounded robotic actions.Many integrations instead use language or vision-language models for high-level planning, with separate low-level controllers that cannot access their semantic understanding.Source: §1 · pp. 1–2
• The paper identifies systematic evaluation across intended multimodal domains and out-of-distribution scenarios as a critical need.Source: §1 · p. 2
• The contributions include profiling VLM, VLA, and emerging generalist models, evaluation splits and metrics for OpenX, action-space mapping, and open-source infrastructure.Figure 1 describes comparisons of predicted and ground-truth actions at every timestep across 20 real-world robotics datasets; the paper also introduces MultiNet v0.1.Source: §1 · pp. 2–3, Figure 1 · p. 2

• General multimodal benchmarks span domains and capabilities, but the cited examples do not specifically address robotics control or offer limited robotics scope.MultiBench includes robotics but has limited robotics evaluation; MMMU focuses on college-level multimodal understanding.Source: §2 · p. 3
• Multimodal language-model evaluations range from single-task benchmarks to broad capability suites and specialized tests for reasoning or multimodal handling.The passage names VQA, POPE [24], LAMM, MathVista [28], and GAIA [32], among others.Source: §2 · p. 3
• Robotics benchmarks include simulated task suites, real-world manipulation, environmental perturbations, lifelong transfer, and specialized embodied tasks.Examples include THE COLOSSEUM’s 14 environmental perturbations [35], FactorWorld’s 11 factors across 19 tasks, and KitchenShift’s seven kitchen variation factors.Source: §2 · p. 3
• The paper identifies a gap in systematically evaluating behavior-cloning models at scale in both simulated and real-world settings.Source: §2 · p. 3
• This work evaluates actions from real-world robotic trajectories and zero-shot generalization across action spaces and robot morphologies using OpenX datasets.It distinguishes its scope from simulated environments and static vision-language tasks.Source: §2 · p. 3

3.1 Data

• 20 OpenX datasets are reported for all three models, while JAT results cover the full 53-dataset benchmark subset selected from 72 available datasets.OpenX contains over 1 million real-robot trajectories from 22 embodiments; new evaluation splits were created where datasets lacked predefined ones.Source: §3.1 · pp. 3–4

3.1.1 Dataset Curation

• Curation excluded low-quality or inaccessible datasets, removed redundant platform-matched datasets, and left some datasets out of version 0.1 because of TFDS access limitations.When datasets matched across robot, gripper, action, sensor, language, calibration, and proprioception features, the one with more episodes was retained.Source: §3.1.1 · p. 4

3.2 Models

• JAT, GPT-4o, and OpenVLA represent contrasting approaches: broad multimodal generalism, a powerful VLM mapped to action, and an open-source VLA, respectively.The evaluated models differ in architecture and training strategy, and span model sizes and computational requirements.Source: §3.2 · pp. 4–5

3.3 Evaluation Metrics

• Offline performance is assessed by comparing predicted and ground-truth action tensors with MSE variants and a completion-rate measure.AMSE averages timestep MSEs, NAMSE averages min-max-normalized MSEs, and completion rate compares final predicted and ground-truth actions; together, the metrics cover prediction accuracy and approximate task completion.Source: §3.3 · pp. 5–6

4.1 Profiling Configuration

• JAT was evaluated zero-shot using the current timestep, with preprocessing for image channels, dictionary observations and actions, and multiple images.The model receives the observation state, image, and language instruction; when multiple images are available, the primary image is selected.Source: §4.1 · p. 6
• GPT was evaluated zero-shot with prompts combining available observations, an image, language instructions, action-dimension descriptions, and action-space statistics.The setup also included task or environmental descriptions when available, selected a primary image, and used a random action tensor as fallback for incompatible outputs.Source: §4.1 · p. 6
• OpenVLA configuration focused on action-space handling and gripper-command conversions, including binary, ternary, and continuous mappings.The continuous mapping converts values from [0, 1] to [-1, 1].Source: §4.1 · pp. 6–7
• Non-standard image-key names affected access to image observations for other models, while OpenVLA used them; this occurred in conq_hose_manipulation and viola.Source: §4.1 · p. 7

4.2 Inference Infrastructure

• JAT and GPT inference used a GCP e2-standard-8 instance, while OpenVLA used a GCP g2-standard-8 instance with an NVIDIA L4 GPU.The L4's 24 GB memory was sufficient for full-model OpenVLA inference without optimization; the JAT/GPT instance had 8 vCPU and 32 GB memory.Source: §4.2 · p. 7

5.1 Average Model Performance Analysis

• Across 20 OpenX datasets, JAT has higher AMSE on most datasets, while GPT-4o and OpenVLA show comparable performance; GPT is comparable or slightly better on many datasets.OpenVLA also performs especially well on tasks within its training distribution, with complex manipulation as a notable exception.Source: §5.1 · pp. 7–8, Figure 2 · p. 8

5.1.1 Model-Specific Analysis

• OpenVLA’s robotics training is associated with steadier performance across task types, while GPT-4o’s detailed prompting appears to aid precise predictions; both struggle with complex, multi-step tasks.JAT’s higher AMSE suggests its architecture may be less optimized for precise robotics control.Source: §5.1.1 · p. 9

5.1.2 Implications for Future Development

• The authors point to platform-agnostic control and combining sophisticated prompting with robotics-specific training as directions for improving performance across platforms and tasks.GPT-4o’s consistent NAMSE is presented as a possible indication that detailed action-space, task, and environment information helps.Source: §5.1.2 · p. 9, Figure 3 · p. 10

5.2 Normalized Performance Analysis

• NAMSE compares each model’s performance across tasks relative to its own capabilities, helping distinguish task difficulty from differences in action-space scale.The paper contrasts this with AMSE, which can be affected by action-space scales.Source: §5.1 · p. 9, §5.2 · p. 10

5.2.1 Model-Specific Performance Patterns

• GPT-4o’s NAMSE generally remains below 0.2, while JAT’s typically ranges from 0.2 to 0.4; JAT is more stable than OpenVLA but has a higher baseline than GPT-4o.The paper suggests GPT-4o’s structured prompting may contribute to its consistency.Source: §5.2.1 · pp. 10–11
• OpenVLA shows the greatest NAMSE variation, including NAMSE 0.003 on the pretrained toto task and 0.82 on columbia_cairlab_pusht_real.This pattern is described as task-specific performance variation.Source: §5.2.1 · p. 11

5.2.2 Cross-Model Insights

• Complex, multi-step, and precise-control tasks show higher NAMSE across models, while simple pick-and-place tasks tend to show lower error; model patterns range from GPT-4o’s consistency to OpenVLA’s specialization.The authors suggest structured task and action-space information may contribute to consistency across diverse tasks.Source: §5.2.2 · pp. 11–12

5.3 Key Takeaways

• The key takeaways are a specialization–generalization trade-off, shared difficulty with complex multi-step or precise-control tasks, and sensitivity to action-space characteristics.Structured prompting with explicit action-space and environmental context may be as important as architectural choices.Source: §5.3 · p. 12

6 Future Work

• Future versions will evaluate pure vision-language and language tasks to assess whether control-task fine-tuning or co-training affects those modalities.The authors say the analysis could inform architectural and training strategies for maintaining performance across modalities.Source: §6 · p. 12
• Future evaluations will expand beyond OpenX to out-of-distribution data and few-shot learning and fine-tuning, beyond the current zero-shot focus.The authors say this would provide a more complete picture of model adaptability.Source: §6 · p. 12
• The authors propose exploring VLA transfer to non-robotic domains by fine-tuning models for software environments.They suggest this could provide insight into whether embodied learning principles generalize to virtual environments.Source: §6 · p. 12
• Proposed evaluations target compositional generalization, long-sequence reliability, cross-embodiment transfer, memory and long-term planning, and multi-agent interaction.The examples include combining learned primitives for novel tasks and assessing goal maintenance over extended sequences.Source: §6 · p. 13
• The authors plan online evaluation with simulation environments for 2D and 3D control tasks.These environments are intended to support more dynamic assessment of real-time decision-making.Source: §6 · p. 13

7 Conclusion

• GPT-4o shows the most consistent normalized performance across datasets, while OpenVLA varies more by scenario and JAT generally has higher error rates.The authors attribute GPT-4o's consistency likely to prompt engineering that structures action-space and environmental context.Source: §7 · p. 13
• All models struggle with complex manipulation, and performance varies across robot platforms and action spaces.The authors also identify prompt engineering's impact, as demonstrated by GPT-4o, as a potential direction for improvement.Source: §7 · p. 13
• Structured task representations and better handling of temporal dependencies could help address limitations in complex manipulation.The authors identify consistent performance across diverse tasks and multi-step manipulation as continuing challenges.Source: §7 · p. 13
• The open-source evaluation framework provides a foundation for systematic assessment of future VLA models.The study presents a comprehensive framework and systematic analysis across diverse robotics tasks.Source: §7 · p. 13

8.1 Dataset Coverage, Completion Rate, and Additional AMSE Recordings

• Figure 6 reports AMSE across datasets with action-space units in millimeters.The caption identifies the quantity and unit, but supplies no dataset-specific values.Source: Figure 6 · p. 17
• Table 3 covers dataset coverage and action-space types.Source: Table 3 · p. 18
• The listed action abbreviations are pos for position, orient for orientation, grip for gripper, term for terminate, vel for velocity, ang for angular, and quat for quaternion.Source: §8.1 · p. 18
• Some datasets were excluded because of space constraints or incomplete information.Source: §8.1 · p. 18
• Table 4 reports task completion rates across models and datasets; success is the percentage of episodes whose final action matched ground truth.Source: Table 4 · p. 19, §8.1 · p. 19