Source-linked AI summary
RoboSemanticBench: Diagnosing Semantic Grounding in Action Prediction for VLA Models
Bin Yu, Yao Zhang, Haishan Liu, Shijie Lian, Yuliang Wei, Xiaopeng Lin, Zhaolong Shen, Changti Wu, Ruina Hu, Bailing Wang, Cong Huang, Kai Chen
TL;DR
VLA research assumes pretrained semantic competence should guide robot actions, but imitation fine-tuning and shortcut-prone evaluations leave semantic grounding insufficiently diagnosed. RoboSemanticBench turns math and general-knowledge questions into embodied answer-selection tasks, and finds many policies grasp candidates while selecting the correct target near or below random after controlling for grasp success. The results motivate VLA training interfaces that preserve semantic targets between backbone reasoning and action generation.
Problem
Existing VLA training and evaluations provide limited evidence that pretrained semantic competence is grounded in action prediction rather than visual or instruction-action shortcuts.
Method
RoboSemanticBench presents multiple-choice math or general-knowledge questions with visible answer blocks and requires a VLA to select and grasp the block corresponding to the correct answer.
Results
Many representative VLA policies learn to grasp candidate blocks but select semantically correct targets near or below random after controlling for grasp success.
Takeaways & Limitations
RSB motivates future VLA architectures and objectives that preserve selected semantic targets in a stable, scene-grounded form for the action module.
Takeaways & Limitations
RSB’s semantic sources are intended not to exceed modern language backbones, including Qwen3-4B’s reported over 85% GSM8K and over 70% MMLU accuracy.
Abstract
from arXiv · showhide
Vision-language-action (VLA) models are built on the premise that semantic understanding from pretrained language or vision-language backbones should guide robot action prediction. Yet robot fine-tuning is optimized as imitation over task-specific action distributions, and many evaluations can be solved through visual or instruction-action shortcuts. We introduce RoboSemanticBench (RSB), an embodied benchmark for diagnosing semantic grounding in action prediction: whether post-trained VLA models can use complex instruction semantics to select and manipulate the correct physical target. In each episode, a robot receives a multiple-choice math or general-knowledge question, observes candidate answer blocks, and must grasp the block corresponding to the correct answer. RSB covers controlled arithmetic, grade-school mathematical understanding, and commonsense or factual understanding under four-choice and ten-choice suites. Across representative VLA models, we find that many policies learn to grasp candidate blocks but select the semantically correct block at near-random or below-random rates after controlling for grasp success, revealing a persistent gap between backbone-level semantic competence and action prediction.
1 Introduction
VLA post-training may disconnect pretrained semantic competence from action prediction, allowing visual or instruction-action shortcuts. RoboSemanticBench isolates this issue by requiring semantic answer selection followed by grasping, and finds many models near or below random semantic selection despite learning the grasping primitive.
- Motivation: Robot imitation data can reward conditional action fitting without forcing models to expose the semantic decision that makes a trajectory correct.This creates room for instruction-action or visual shortcuts instead of semantic target selection.
- Motivation: Existing task-success evaluations can conflate motor execution, recognition, and language grounding, making high success ambiguous.Short commands such as picking a named object may be solvable through visual regularities or dataset shortcuts.
- RoboSemanticBench: RoboSemanticBench converts multiple-choice math and general-knowledge questions into embodied answer-selection tasks requiring instruction understanding, target binding, and grasp execution.Its six evaluation suites cover math, hard-math, and general-semantic understanding across four-choice and ten-choice settings.
- RoboSemanticBench: Fixed manipulation with varied semantic content and option sets tests whether instruction semantics guide target selection rather than instruction-action correlations.The benchmark keeps the motor primitive simple while changing the semantic problem and candidate options.
- Diagnosis: GSR, TSR, and nSG separate grasping ability from semantic target selection, exposing whether semantic decisions participate in action prediction.A large GSR–TSR gap indicates successful manipulation but failed semantic grounding.
- Findings: Many representative VLA models perform near or below random semantic target selection after grasp success is controlled for.The benchmark therefore reveals a persistent semantic grounding gap rather than merely measuring grasping skill.
2 Related Work
Prior benchmarks assess language-conditioned manipulation, robustness, and generalization, but often entangle motor execution with recognition and language grounding. RoboSemanticBench complements these suites by making the instruction itself a math or general-knowledge semantic problem tied to a visible target.
- Manipulation Benchmarks: CALVIN and LIBERO study multi-task and lifelong manipulation, while SimplerEnv evaluates simulation-based reproduction.Other benchmark suites address household tasks, robot data generation, world-knowledge manipulation, or VLA comparison.
- Limitations of Prior Benchmarks: Existing language-conditioned manipulation benchmarks often entangle motor execution, object recognition, and language grounding in task-success metrics.This makes it difficult to determine whether success reflects semantic understanding or other capabilities.
- Language-Use Diagnostics: Related diagnostics test instruction perturbations, counterfactual commands, linguistic diversity, and distribution shift.They complement RSB, which makes the instruction itself a math or general-knowledge problem requiring target binding.
3 RoboSemanticBench
RoboSemanticBench evaluates whether VLA policies solve semantic questions, map correct answers to visible targets, and execute a fixed pick-and-place action. Its suites vary semantic difficulty and choice-set size while randomizing mappings and layouts to prevent position or appearance shortcuts.
- Each episode requires solving a question, mapping the correct option to a visible block, and placing that block in the answer zone.The target is determined by the question and episode-specific option mapping, not by fixed color, letter, position, or trajectory.
- RSB-Math uses arithmetic with nearby distractors, while RSB-HardMath requires extracting quantities and composing relations in grade-school word problems.
- RSB-General tests commonsense and factual understanding, complementing arithmetic and word-problem subsets across three semantic demands.
- Four-choice and ten-choice suites vary the semantic action space, with randomized layouts and option mappings to reduce guessing and visual shortcuts.The ten-choice suite uses same-color letter blocks with procedural strokes, while unseen templates test dependence on narrow surface forms.
- Evaluation logs task success, grasp success, and semantic metadata to distinguish low-level control failures from incorrect semantic target selection.Expert demonstrations use scripted answer-to-block mapping and motion-planned pick-and-place trajectories.
4 Experiments
Experiments fine-tune representative VLA models on disjoint expert-demonstration questions and evaluate them with grasp-aware metrics. Most models grasp candidate blocks but select the semantically correct target near randomly, with the gap widening in ten-choice suites.
- Training and evaluation questions are disjoint within each semantic subset, and models use comparable fine-tuning budgets where supported.
- nSG measures semantic target selection conditioned on successfully grasping a candidate block, with nSG = 0 representing random selection and nSG = 1 perfect selection.TSR measures correct-block grasps, while GSR measures grasps of any candidate block.
- 25% for four-choice suites and 10% for ten-choice suites: most models remain near random target selection after grasp success is factored out.Most average nSG scores are near or below zero, while some models have high GSR but low TSR.
- The semantic grounding gap is especially visible in ten-choice suites, where more plausible targets make shortcut-based selection less reliable.
- π0.5 achieves the highest average TSR and the only clearly positive average nSG, although its nSG remains modest.
- Replacing lettered blocks with everyday objects leaves grasp success high while TSR remains low, indicating that failures are not specific to uniform block appearance.
5 Failed Exploration
The paper tests whether explicit reasoning or language-centric supervision can close the semantic grounding gap. ReasoningVLA improves TSR but remains unreliable, while VQA cotraining consistently worsens TSR.
- ReasoningVLA generates a textual chain-of-thought identifying the target option before conditioning action prediction on that intermediate solution.Action Query Tokens connect the VLM’s textual reasoning to the Action Expert’s action-chunk generation.
- ReasoningVLA jointly supervises action-chunk generation and next-token prediction for the distilled chain-of-thought.
- ReasoningVLA improves average TSR over QwenGR00T, but absolute TSR remains low and GSR decreases, especially in ten-choice suites.
- Explicit reasoning traces alone are insufficient to reliably ground the selected answer into robot actions.
- VQA cotraining drops TSR on all six RSB suites, reducing average TSR from 10.7% to 8.2%.The paper suggests language-centric auxiliary supervision may conflict with the action-learning objective.
6 Error Analysis
Error analysis shows that grasping a candidate block is usually not the main failure: models often select the wrong target, and language reasoning may fail to control action selection.
- Most grasp-success/task-failure episodes are target-selection errors rather than placement failures.
- 95.92% of QwenGR00T’s grasp-success/task-failure cases involve choosing the wrong target.
- Only 6.70% of ReasoningVLA’s grasp-success/task-failure episodes involve an incorrect chain-of-thought answer.
- Most ReasoningVLA failures occur when chain-of-thought identifies the correct answer but the action grasps the wrong block.
7 Discussion
RSB evaluates whether semantic understanding is grounded in physical target selection, not merely whether a model can answer questions or execute grasps. Its results motivate interfaces and objectives that preserve semantic decisions for the action module.
- RSB measures whether instruction semantics select the correct physical target during action prediction after accounting for candidate-block grasping.
- A strong pretrained backbone alone does not ensure that semantic decisions reach the action pathway.
- Future training may need objectives and interfaces that preserve the selected semantic target in a stable, scene-grounded form.
- Successful future systems should maintain high GSR while raising TSR and nSG above random selection.
8 Conclusion
The conclusion presents RoboSemanticBench as an embodied test of whether instruction understanding determines action selection. Across representative VLA models, grasping is often learned, but semantically correct target choices remain near or below random after controlling for grasp success.
- RoboSemanticBench converts math, hard-math, and general-knowledge questions into embodied answer-selection tasks across six evaluation suites.
- Many policies learn the grasping primitive, but target choices remain near or below random once grasp success is controlled for.
- The results indicate that semantic decisions from pretrained backbones often fail to route into the action pathway.
- Table 6 summarizes mean GSR and TSR across the six evaluation suites when all six results are available.
B Beyond Blocks Results
Beyond Blocks results preserve the central RSB pattern with everyday objects: models generally grasp candidates successfully but select the semantically correct target much less often. The metrics distinguish generic grasping from semantic target selection.
- π0.5 and GR00T N1.7 retain high GSR but much lower TSR when lettered blocks are replaced with everyday objects.
- The Beyond Blocks results indicate that semantic target selection, rather than object-specific grasping difficulty, remains the main bottleneck.
- GSR counts successful grasps of any candidate object, whereas TSR requires grasping the object associated with the correct semantic answer.
- nSG measures semantic target selection conditioned on successfully grasping a candidate, with values near zero indicating random selection and negative values indicating worse-than-random selection.
D ReasoningVLA Data Construction
The data construction combines robot demonstrations with semantic reasoning annotations and, in a separate cotraining setup, VQA supervision. These additions are designed to connect language-space answering with continuous action generation while preserving a direct semantic-grounding evaluation.
- ReasoningVLA Data Construction: Gemini 3 Flash produces concise rationales that solve each multiple-choice question, identify the correct option, and map it to the corresponding color block.Responses lacking a unique final option or disagreeing with ground-truth metadata are discarded.
- ReasoningVLA Data Construction: The rationale is wrapped in <think> tags and concatenated with the original observation–instruction demonstration, while the expert action chunk remains unchanged.The augmented sample therefore supervises both semantic reasoning in language space and continuous action generation.
- VLA Cotraining: Cotraining mixes standard robot-demonstration samples with RoboVQA image–question–answer samples that share the VLM backbone but supervise different pathways.Robot data supervises expert action chunks, whereas VQA data supervises next-token prediction over answer text.
- VLA Cotraining: The cotraining objective combines the original action-generation loss with a VQA language-modeling loss weighted by 0.1.The VQA branch encourages language-space semantic answering ability while the action branch continues imitation learning on expert trajectories.
- VLA Cotraining: After cotraining, evaluation uses only simulator observations and generated RSB instructions, without the correct answer label or VQA supervision.This protocol directly tests whether language-oriented supervision improves semantic grounding during action prediction.
F Training Details for Evaluated Models
The evaluation covers multiple VLA architectures, including generalist, autoregressive, diffusion-based, compact, action-chunking, and humanoid-robot models. Each model is fine-tuned and evaluated on RSB using its corresponding implementation and a shared benchmark protocol where specified.
- Common Training Setup: Training budgets are aligned by matching training steps multiplied by global batch size, while other model-specific settings follow official codebases whenever possible.All evaluated models use 8 NVIDIA H100 GPUs.
- Evaluated Models: GO1 is reproduced with its official codebase and fine-tuned on RSB demonstrations using the released observation, instruction, and action interface.
- Evaluated Models: OpenVLA adapts a pretrained vision-language backbone into an autoregressive robot policy with tokenized actions and is fine-tuned from its pretrained checkpoint on the RSB training split.
- Evaluated Models: DexVLA uses a VLM for semantic perception and instruction processing alongside a plug-in diffusion expert for continuous robot control.It is fine-tuned on the same semantic answer-selection demonstrations as the other evaluated models.
- Evaluated Models: TinyVLA provides a compact, data-efficient baseline, while PD-VLA preserves action chunking and accelerates autoregressive decoding through parallel fixed-point decoding.Both are trained or evaluated under the RSB protocol.
- Evaluated Models: π0 and π0.5 connect pretrained VLM backbones to continuous action generation through action experts and flow matching, with π0.5 adding open-world-oriented robot-data pretraining signals.
- Evaluated Models: GR00T N1.7 is fine-tuned for RSB and evaluated using the same 500-episode simulator protocol, while QwenGR00T is fine-tuned within the StarVLA framework.