PaperlayerView exact revision on arXiv

Source-linked AI summary

Does VLA Even Know the Basics? Measuring Commonsense and World Knowledge Retention in Vision-Language-Action Models

Nikita Kachaev, Andrey Moskalenko, Matvey Skripkin, Nikita Kurlaev, Daria Pugacheva, Albina Burlova, Mikhail Kolosov, Denis Shepelev, Andrey Kuznetsov, Elena Tutubalina, Aleksandr I. Panov, Alexey K. Kovalev, Vlad Shakhuro

arXiv:2606.19297v1cs.LGcs.RO

TL;DR

It is unclear how much commonsense and factual knowledge VLA models retain after robotics adaptation, because standard embodied-task success conflates knowledge with control. Act2Answer evaluates knowledge through simple object-placement choices and finds strong performance on simple perceptual categories but larger gaps on richer semantic categories than source VLMs.

  • Problem

    Existing VLA benchmarks provide limited systematic evidence about retained commonsense and factual knowledge, while end-to-end success conflates knowledge with perception, control, and environment.

  • Method

    Act2Answer adapts VLM knowledge questions into tabletop episodes requiring a minimal object-placement action, supplemented by layerwise probing of answer-relevant representations.

  • Results

    VLAs perform strongly on simple perceptual categories but show substantially larger gaps on richer semantic categories than source VLMs, while VQA co-training is associated with stronger knowledge-sensitive performance.

  • Takeaways & Limitations

    VLA adaptation tends to preserve shallow perceptual knowledge while weakening performance on richer task-relevant semantic distinctions.

Abstract

from arXiv · show

Embodied Vision-Language-Action (VLA) models are typically obtained by fine-tuning powerful pretrained VLMs on robotics data, yet it is unclear how much commonsense and factual knowledge they retain after adaptation. Failures on knowledge-sensitive tasks are ambiguous, conflating missing knowledge with poor generalization of low-level control. We introduce Act2Answer, a lightweight protocol that adapts VLM knowledge benchmarks to VLA evaluation by requiring agents to answer through action. Each question becomes a short tabletop episode where the agent performs a single object-placement action to select among candidate answers, yielding an action-grounded success rate with reduced control confounds. We curate a test suite of such environments across diverse commonsense and world-knowledge categories and introduce layerwise intent probing to localize answer-relevant information across the VLM backbone and action head. In a large-scale study of 7 VLA models and 9 VLM baselines, we systematically rank models across categories, finding that VLAs show solid performance on simple concepts while exhibiting larger gaps on richer semantic categories relative to their source VLMs, that VQA co-training is associated with better knowledge retention, and that answer-relevant signals peak in middle VLA layers but attenuate in upper layers. Act2Answer is available at https://tttonyalpha.github.io/act2answer/.

1 Introduction

The paper addresses whether VLA models retain commonsense and world knowledge after robotics training, beyond manipulation-centric success. It introduces ACT2ANSWER, a controlled action-based benchmark, and uses it to compare VLAs with VLMs and probe where answer-relevant information is represented.

  • Existing VLA benchmarks emphasize manipulation success under perturbations, domain shifts, and new layouts while rarely testing retained commonsense distinctions about objects, scenes, and goals.
  • ACT2ANSWER adapts VLM knowledge benchmarks into short simulated episodes requiring action-based answer selection, reducing confounds from long-horizon planning and low-level control.
  • The benchmark contains 1,720 unique binary questions across 12 categories spanning commonsense and world knowledge.
  • A study of 7 modern VLA systems and 9 VLM baselines finds strong VLA performance on simple perceptual categories but larger gaps on richer semantic categories relative to source VLMs.VQA co-training is associated with stronger performance on knowledge-sensitive tasks.
  • Layerwise intent probing uses linear classifiers over per-layer representations to measure answer-relevant information across model depth, including information that remains represented despite incorrect actions.

2 Related Work

Existing VLA benchmarks mainly measure language-conditioned manipulation and control generalization, leaving explicit commonsense and world-knowledge assessment largely unaddressed. Prior knowledge-transfer evaluations test VLM components with textual VQA answers, while broad VLM benchmarks assess multimodal understanding across reasoning, text reading, diagrams, science, and multidisciplinary questions.

  • VLA Benchmarks: Current VLA benchmarks emphasize manipulation success and control generalization across tasks, scenes, embodiments, and language variations rather than explicit knowledge assessment.Examples include LIBERO, CALVIN, VLABench, RoboBench-Mart, and BEHAVIOR-1K.
  • Knowledge Transfer: Recent knowledge-transfer studies evaluate VLM components on VQA-style benchmarks by decoding textual answers, indirectly measuring retained answering ability rather than action-grounded knowledge use.This strategy is associated with works such as Cai et al. (2026a) and Chen et al. (2025).
  • VLM Benchmarks: VLM benchmarks cover multimodal understanding through compositional reasoning, image-text reading, diagram understanding, multimodal science, and large-scale multidisciplinary evaluation.Representative benchmarks include GQA, TextVQA, DocVQA, AI2D, ScienceQA, and MMMU.

3 Methodology

The methodology separates embodied task success into perception, knowledge, control, and environment, then structures commonsense evaluation through practical knowledge categories. Act2Answer converts VLM benchmark questions into tabletop episodes requiring an agent to place a cube on the correct answer plate.

  • Conceptual decomposition: Embodied task success is decomposed into perception, knowledge, control, and environment because identical success rates can reflect different underlying weaknesses.The decomposition distinguishes strong knowledge with weak control from weak knowledge with strong motor execution.
  • Knowledge taxonomy: Commonsense knowledge is defined as knowledge that can affect which action is appropriate, using practical categories to guide task selection, coverage, and error analysis.The taxonomy is intended to structure benchmark design rather than provide a universal classification.
  • Act2Answer episodes: Act2Answer adapts VLM benchmark questions into embodied episodes where the agent interprets an instruction and moves a cube onto the correct answer plate.This action-based format tests knowledge through a short tabletop placement task.
  • Knowledge taxonomy: The proposed categories span physical, temporal, quantitative, biological, social, normative, and cultural knowledge relevant to embodied decisions.They cover object properties and affordances, event structure, quantities, biological constraints, agent interactions, norms, and shared cultural references.
  • Data curation: The Act2Answer task suite is constructed from VLM benchmarks through selection, filtering and normalization, and conversion.Figure 3 summarizes the data curation pipeline used to build the suite.

4 Act2Answer: Embodied Evaluation of Knowledge

Act2Answer evaluates whether VLA models can use commonsense and world knowledge through a minimal object-placement action, reducing confounds from perception, control, and environment complexity. The study combines benchmark adaptation, success-rate evaluation, and layerwise probing to characterize knowledge retention and accessibility after VLA adaptation.

  • Protocol: Act2Answer converts VLM-style questions into tabletop episodes where agents select visual answer options by placing a cube on the chosen image.The protocol uses known object positions and scores an episode correct when the selected image matches the ground-truth answer.
  • Protocol: The task suite adapts established VLM benchmarks across representative knowledge categories, filtering items for instruction length and visual reliability.Human annotators remove examples whose relevant objects are too small or visually ambiguous for the relatively low resolution of many VLA models.
  • Analysis: Answer-relevant information is often above chance in intermediate VLM-backbone layers but attenuates in upper layers, suggesting retained knowledge may become less accessible for action.Layerwise linear probing examines VLA representations across categories including ATTRIBUTE, STATE, EMOTION, and COUNTING.
  • Results: Nearly all evaluated models perform strongly on primitive COLOR and SHAPE tasks, establishing a lower bound for action-based use of simple visual concepts.These categories test whether models can use basic perceptual distinctions in action.
  • Results: Across non-primitive categories, current VLAs mostly remain near the random threshold, with unstable performance on richer semantic, quantitative, temporal, normative, cultural, and biological distinctions.Magma is a notable exception, while category-specific gains include SpatialVLA on TRAFFIC (57%) and CELEBRITY (55%), and InternVLA-M1 on LIVING WORLD (58%).

5 Conclusion

ACT2ANSWER evaluates knowledge-sensitive VLA behavior through minimal embodied actions that closely match VLM multiple-choice selection. The conclusion reports that VLAs preserve low-level perceptual discrimination but perform substantially worse on richer semantic categories.

  • Protocol: ACT2ANSWER evaluates VLA knowledge-sensitive behavior by requiring a tabletop placement action to select among candidate images.The protocol preserves embodied interaction while keeping the choice close to VLM multiple-choice formats.
  • Findings: Current VLA systems show strong performance on simple perceptual categories but substantially weaker performance on richer semantic categories.The reported gap distinguishes low-level perceptual abilities from higher-level semantic performance.
  • Findings: The transition from VLM to VLA tends to preserve low-level visual discrimination, including color, shape, and coarse object identity.Performance drops markedly on higher-level categories despite preserved shallow perceptual abilities.

A Evaluation and Setup Ablations · A.1 Effect of Image Resolution · A.2 Influence of Prompt Formulation

The appendix tests whether the main ACT2ANSWER conclusions depend on evaluation choices, finding that image resolution and prompt formulation alter some results but do not change the qualitative VLM–VLA comparison. The standardized and action-style prompts both preserve VLM advantages over most VLA results in knowledge-sensitive categories.

  • A Evaluation and Setup Ablations: The ablations examine image resolution, prompt formulation, texture rendering, answer-tile size, and lighting intensity to test sensitivity to evaluation and embodied-setup choices.These experiments assess whether conclusions are driven by a single prompt, preprocessing choice, or simulation setting.
  • A.1 Effect of Image Resolution: The VLM–VLA comparison estimates how much task-relevant information remains accessible to the VLM backbone without action requirements.The comparison is intended as an action-free estimate rather than a test of low-level control.
  • A.1 Effect of Image Resolution: The VLM baselines are evaluated at 224 × 224 and 560 × 480 image resolutions.Resolution changes slightly affect some categories and models, particularly fine-grained categories such as ATTRIBUTE, STATE, and SYMMETRY.
  • A.1 Effect of Image Resolution: Changing image resolution does not alter the qualitative interpretation: strong VLM baselines outperform many VLA policies that remain near chance.This indicates that the VLM–VLA gap is not solely an artifact of the chosen image resolution.
  • A.2 Influence of Prompt Formulation: The standardized VLM prompt combines an image placeholder, a natural-language question, answer options, and an instruction to output only the corresponding option letter.This constrained multiple-choice format yields consistent and comparable predictions.
  • A.2 Influence of Prompt Formulation: An alternative action-style prompt, closer to the VLA instruction format, slightly reduces VLM performance for some models and categories.The effect is most visible for Qwen3-8B and Qwen2.5-7B.
  • A.2 Influence of Prompt Formulation: Despite the action-style prompt effect, the overall comparison remains qualitatively similar, with VLM performance substantially above most VLA results in many knowledge-sensitive categories.Thus, prompt formulation does not eliminate the observed VLM–VLA difference.

A.3 Robustness to Texture Rendering · A.4 Effect of Answer-Tile Size · A.5 Effect of Lighting Intensity

The ablations show that the main evaluation conclusions are broadly robust to texture rendering, answer-tile size, and lighting intensity, although rendering choices can cause localized category- and model-specific changes. Simple perceptual categories remain easier than richer semantic categories, while EMOTION and ATTRIBUTE remain difficult for OpenVLA and π0.

  • A.3 Robustness to Texture Rendering: A.3 tests whether simulated visual appearance changes the study’s main conclusions.The evaluation explicitly examines robustness to environment rendering choices.
  • A.3 Robustness to Texture Rendering: A.3 compares default simulator rendering with Visual Matching, which reduces the visual gap using real-background compositing and tuned foreground textures for salient assets.The Raw Sim setting retains original simulated textures, materials, and backgrounds.
  • A.4 Effect of Answer-Tile Size: A.4 finds that moderate answer-tile size changes preserve the main qualitative trends, although larger tiles reduce performance for some models and categories.Tile size is treated as a possible out-of-distribution shift for VLA models.
  • A.4 Effect of Answer-Tile Size: A.4 reports that simple perceptual categories remain easier than more semantic categories and that model rankings are largely unchanged across tile sizes.OpenVLA and π0 can decline on COLOR when tiles increase, but the broader pattern remains stable.
  • A.5 Effect of Lighting Intensity: A.5 shows that lighting perturbations cause localized changes, especially under darker lighting and in harder categories.The lighting ablation tests whether results depend on a narrow rendering condition.
  • A.5 Effect of Lighting Intensity: A.5 preserves the main conclusions: EMOTION and ATTRIBUTE remain difficult for OpenVLA and π0, COLOR remains comparatively accessible, and Magma is strongest among the tested models.The results are not tied to a single lighting condition, although perceptual rendering choices can affect some categories.

B Benchmark Sources and Data Construction · B.1 Near Format-Preserving Adaptations

Act2Answer combines five complementary knowledge benchmarks into a unified action-compatible evaluation setup. Its adaptations prioritize preserving source-task intent, with MLLM-CompBench and VL-Think requiring especially limited restructuring.

  • B Benchmark Sources and Data Construction: Act2Answer draws on five source benchmarks: MLLM-CompBench, IconQA, MMBench, OK-VQA, and VL-Think.The benchmarks were selected for complementary coverage of the targeted knowledge domains.
  • B Benchmark Sources and Data Construction: The unified setup maps heterogeneous benchmarks into short action-compatible instructions and embodied answer-selection episodes.The supplied passage describes this mapping but truncates the remaining format details.
  • B Benchmark Sources and Data Construction: Adaptation varies by benchmark because of source-format mismatch, not arbitrary redesign, while preserving the original task’s intended test.Examples were restricted to those with visually perceivable evidence at typical VLA resolution and expressible answers.
  • B Benchmark Sources and Data Construction: The resulting tasks span social, biological, culturally grounded, temporal, public-convention, physical, and quantitative categories.The categories are organized into three broad panels in the accompanying examples.
  • B.1 Near Format-Preserving Adaptations: MLLM-CompBench supplies EMOTION, ATTRIBUTE, and STATE because its two-image comparative format closely matches embodied answer selection.The original image pair was retained and the question converted into a short action-compatible instruction.
  • B.1 Near Format-Preserving Adaptations: MLLM-CompBench adaptation was near format-preserving, requiring almost no additional restructuring.These categories therefore support direct VLM–VLA behavioral comparisons in the study.
  • B.1 Near Format-Preserving Adaptations: VL-Think supplies PUBLIC INFO, TRAFFIC, and COLOR because its compact symbols and public conventions fit embodied semantic selection.The adaptation unified instruction style and episode format while remaining near format-preserving.
  • B.1 Near Format-Preserving Adaptations: VL-Think uses small closed concept vocabularies rather than large candidate-image pools.The passage introduces this distinction but truncates its further implications.

B.2 Moderately Adapted Benchmarks · B.3 Open-Ended to Binary Adaptation · B.4 Instruction Rewriting and Binary Conversion

Act2Answer adapts diverse knowledge benchmarks to binary embodied selection while preserving their original semantic targets. The protocol combines benchmark-specific filtering and conversion with short, visually grounded instructions rewritten through LLM assistance and human review.

  • B.2 Moderately Adapted Benchmarks: IconQA items were converted from multi-option visual choice into binary selection by pairing each correct answer with one distractor while preserving the semantic target.Its schematic icon diagrams are relatively robust under the lower visual resolution typical of embodied evaluation.
  • B.2 Moderately Adapted Benchmarks: A curated MMBench derivative retained identity recognition while selecting more broadly recognizable public figures to target shared world knowledge.The underlying recognition problem was unchanged, but the subset was not a verbatim extraction.
  • B.3 Open-Ended to Binary Adaptation: OK-VQA supplied the LIVING WORLD category, covering animal identity, flora identity, and living-versus-nonliving distinctions.The benchmark required substantial adaptation because its native format is single-image open-ended VQA rather than binary embodied selection.
  • B.3 Open-Ended to Binary Adaptation: OK-VQA examples were filtered to short, visually grounded answers with stable annotator agreement and biologically relevant question-answer patterns.Selection used both question-pattern and answer-level filtering.
  • B.4 Instruction Rewriting and Binary Conversion: The conversion changed responses from text to action while preserving the original knowledge requirement through short imperative instructions.An LLM performed first-pass rewriting, followed by human review and manual editing; changes were limited to template normalization rather than semantic reformulation.
  • B.4 Instruction Rewriting and Binary Conversion: Instructions remained short, visually grounded, and close to the original semantic target across categories such as EMOTION, STATE, and LIVING WORLD.Examples include selecting the more smiling person, dryer grass, or the picture of sheep.

C Discussion · D Details of Score Averaging

The discussion identifies world-knowledge retention during VLA training as an open challenge and finds lightweight mitigation probes helpful but insufficient, especially for richer semantic categories. Score averaging separately normalizes each model’s mean Act2Answer accuracy relative to 50% random guessing.

  • C Discussion: Preventing world-knowledge erosion while acquiring strong control policies remains an open challenge for VLA training.Suggested directions include multi-task or continual learning, representation-preserving regularization or distillation, and architectural decoupling of knowledge from control.
  • C Discussion: Preliminary mitigation experiments test language-rephrasing augmentation and latent distillation on representative Act2Answer categories.Both variants fine-tune π0 on BridgeDataV2 pick-and-place data without using Act2Answer evaluation examples for training.
  • C Discussion: Language rephrasing slightly improves COLOR and SHAPE but does not improve EMOTION or ATTRIBUTE.The augmentation varies action verbs, spatial expressions, sentence structures, descriptive nouns, and robot-directed commands.
  • C Discussion: Latent distillation improves SHAPE, preserves strong COLOR performance, and leaves EMOTION and ATTRIBUTE near chance.The method adds a representation-preservation loss between mid-layer VLA hidden states and final patch embeddings from a frozen vision foundation teacher.
  • C Discussion: The mitigation pattern indicates that simple perceptual distinctions are easier to preserve or recover than richer semantic distinctions.The results motivate stronger mechanisms, including representation-preservation losses during VLA pretraining and improved teacher representations.
  • C Discussion: Large-scale multi-task or continual training may be prohibitively compute-intensive when combined with high-resolution perception and long-horizon control.The discussion proposes more efficient options such as parameter-efficient fine-tuning, targeted regularization, selective rehearsal, and sparsely updated knowledge modules.
  • D Details of Score Averaging: For each model, LIBERO and Act2Answer scores are averaged across environments, then normalized relative to 50% random guessing by linear rescaling.LIBERO values come from original papers, while Act2Answer values come from Table 2.

E Chance Margin ∆and Significance Thresholds

Act2Answer interprets binary success rates relative to a chance baseline of 0.5 using a category-specific significance margin ∆. With α = 0.05, results above, below, or within this margin indicate usable knowledge, failure, or no reliable knowledge, respectively.

  • Significance thresholds: α = 0.05 and z1−α/2 = 1.96 define the two-sided Wald confidence margin ∆ around 0.5.The band |SR − 0.5| ≤ ∆ contains scores not significantly different from chance.
  • Interpretation regimes: Scores above 0.5 + ∆ indicate usable knowledge, scores below 0.5 − ∆ indicate instruction or perceptual failure, and intermediate scores indicate no reliable usable knowledge.The thresholds operationalize the three interpretation regimes of the Soft Success Rate relative to chance.
  • Category-specific margins: ∆≈0.057 for most categories evaluated over N = 300 episodes, while the CELEBRITY set has ∆≈0.083 with N = 140 episodes.The wider CELEBRITY margin reflects its smaller number of evaluated episodes.
  • Dependence and robustness: ∆≈0.08 under a conservative effective sample size of N = 150, and qualitative conclusions remain unchanged because reported category gaps substantially exceed ∆.The two swapped views of each item are not fully independent, making the standard episode-level treatment mildly anticonservative.

F Left/Right Swapped-Layout Analysis

Act2Answer evaluates each item in original and left/right-swapped layouts, averaging both configurations to reduce fixed side-preference effects. The breakdown exposes positional asymmetries and helps distinguish knowledge-sensitive action selection from layout-driven behavior, though it does not remove all spatial or perceptual confounds.

  • Swapped-layout protocol: Each question is evaluated with candidates in both the original and swapped left/right configurations, and the Act2Answer score averages the two.This controls for misleading scores caused by fixed spatial preferences when an item is tested in only one arrangement.
  • Positional effects: Left/right breakdowns make positional effects visible, with substantial layout variation indicating side preference, layout-specific failure, or near-random action selection.Table 8 reports this breakdown for selected VLA models, including large asymmetries on harder categories such as FIC and LIVING WORLD.
  • Interpretation: High performance in both configurations, as observed for COLOR in most evaluated models, more strongly indicates responses to visual-semantic content rather than position.This contrast provides evidence that consistent success across layouts is less likely to reflect positional exploitation.
  • Limitations: Swapped-layout evaluation reduces fixed side-preference effects and diagnoses position-driven outcomes, but it does not eliminate spatial or perceptual confounds.Layout may still interact with model-specific perception and control behavior.
Loading 2606.19297v1…