PaperlayerView exact revision on arXiv

Source-linked AI summary

Do VLAs Understand and Adapt to the Objects They Handle, or Simply Replay Learned Behaviors?

Xinnuo Xu

arXiv:2610.06078v1cs.ROcs.AI

TL;DR

The paper asks whether VLA generalization reflects understanding of object physical properties or replay of learned motions. It probes seven VLAs and tests mass changes in LIBERO, finding that physical properties are weakly represented and do not reliably guide motion adaptation.

  • Problem

    It remains unclear whether VLA generalization reflects understanding of object physical properties that adapts motion to new setups or replay of learned motions that happen to work.

    Source: §I · p. 1

  • Method

    The paper uses linear probing and RSA to examine seven VLAs’ activations, then tests action adaptation by increasing object mass in LIBERO and signalling the change through language or vision.

    Source: §I · pp. 1–2

  • Results

    Physical properties are less decodable than non-physical properties in almost every modality, while most VLAs show little lifting-action change for heavier objects and task success declines.

    Source: §I · pp. 1–2

  • Takeaways & Limitations

    The results suggest that VLAs encode physical properties weakly and do not reliably use them to adapt their motion.

    Source: §I · p. 2

Abstract

from arXiv · show

This paper asks whether VLA generalization is grounded in a global understanding of objects' physical properties that enables policies to adapt their motion to unseen setups, or if policies simply replay the motions they've learnt that happen to succeed in new setups. The former reflects genuine generalization; the latter reflects incidental robustness. We first examine awareness of physical properties in seven VLAs by applying linear probing and representational similarity analysis (RSA) to their activations. We find that physical properties, including mass, fragility, deformability, friction and size are less decodable than non-physical properties such as semantic category, material, sound and price in nearly every modality stream. Compared with their base VLMs, robot pre-training weakens the linear encoding of physical properties in the language stream. Neither pre-training nor downstream fine-tuning strengthens the alignment between physical-property differences and activation distances. We then ask whether the weak physical information present in these activations shapes the actions a VLA generates. In a controlled LIBERO case study, we increase the mass of an in-domain object and signal the change through language or vision. Most VLAs use similar lifting behaviour for the heavier and original-mass objects, leading to task success declines. The few exceptions change their behaviour in response to lexical or visual cues rather than to mass itself. These results suggest that VLAs encode physical properties weakly and do not reliably use them to adapt their motion.

I. INTRODUCTION

The paper tests whether VLAs adapt to objects’ physical properties or merely replay motions that happen to work in new setups. Across representation analyses and a controlled mass-change study, it finds physical properties weakly represented and not reliably used to adapt motion.

I. INTRODUCTION

  • The awareness analysis uses linear probes to test whether properties can be read from activations and RSA to test whether property differences track activation distances.It examines seven VLAs across OXE and DROID.Source: §I · p. 1
  • Physical properties were less decodable than category, material, sound, and price in almost every modality, and training did not strengthen their alignment with activation distances.Robot pre-training and downstream fine-tuning weakened physical-property decodability in the language stream compared with base VLMs.Source: §I · p. 1
  • When object mass increased across 23 LIBERO tasks, most of six VLAs lifted much as before and success fell; three exceptions responded to wording or visual changes, not mass itself.None of the exceptions matched the human pattern of lifting heavier objects more slowly.Source: §I · p. 2

II. RELATED WORK

  • The study uses linear probes as a uniform readout across architectures, complementing prior work on how robot training can degrade pretrained representations and multimodal capabilities.Earlier VLA probing studies recovered object and action states, success structure, and end-effector variables.Source: §II · p. 2
  • Unlike prior work using object properties for high-level object selection, this paper asks whether they are reflected in low-level motion execution.The cited planning work grounds object selection in physical-property annotations.Source: §II · p. 2

III. DATASET AND ANNOTATION

The paper augments OXE, DROID, and LIBERO with object-property annotations for one-hand-manipulable objects. It validates many annotations against human references, while noting that several properties lack such validation.

III. DATASET AND ANNOTATION

  • The study annotates 200 OXE, 88 DROID, and 13 LIBERO objects with twelve physical, semantic, and everyday-context properties.The five designated physical properties are mass, fragility, deformability, friction, and size.Source: §III · p. 2
  • GPT-4.1 annotations meet or exceed the stated 70% inter-human agreement threshold for the evaluated category, material, size, sound, mass, fragility, and deformability labels.Friction, price, location, colour, and disposability have no established human dataset and are not validated against one.Source: §III · p. 2
  • For LIBERO, object mass is labeled using simulator density times collision-mesh volume, a size proxy that ignores material and contents.The authors discard GPT-4.1 real-world mass labels because LIBERO fine-tuning used this simulator physics.Source: §III · p. 2
  • Across autoregressive and flow-matching policies, activations are mean-pooled by modality; action-token embeddings use generated tokens or the flow expert’s final-step tokens.The figure also includes a JEPA-style policy family in the study overview.Source: Figure 2 · p. 3

IV. MODELS AND METHODOLOGY

The study examines seven VLA policies using frozen activations to assess how object properties are represented. Linear probes test whether properties are readable, while RSA tests whether they organize representation geometry.

IV. MODELS AND METHODOLOGY

  • Seven VLAs spanning three action-decoding paradigms are analyzed with linear probing and RSA to locate object-property information in frozen activations.The paradigms are autoregressive discrete tokens, flow-matching action experts, and JEPA-style latent prediction.Source: §IV · p. 3
  • For each modality and layer, activations are pooled into a representation; object-text, camera, robot-state, and action tokens form separate streams.The recorded activations come from a single forward pass on a robot-manipulation frame.Source: §IV · p. 3
  • Linear probes test whether properties can be read from activations, using balanced accuracy above chance as evidence of decodability.Categorical and ordinal chance levels are 1/K and 0.5, respectively.Source: §IV · p. 3
  • RSA compares neural and property dissimilarity matrices, with their Spearman rank correlation measuring whether embedding distances track property differences.Unlike probing, RSA fits no parameters; a property may be decodable even when other properties dominate embedding distances.Source: §IV · pp. 3–4
  • Figure 3 summarizes decodability and RSA by modality stream and property, averaging results across the tested VLAs and datasets.Its physical-property axes run from mass to size.Source: Figure 3 · p. 4

V. EXPERIMENT RESULTS

This section reports results aggregated across models, while Section VI examines individual models in detail.

V. EXPERIMENT RESULTS

  • Results are aggregated across models here; Section VI examines individual models in detail.Source: §V · p. 4

A. Property Decodability in Pre-trained VLAs

The section evaluates property decodability in pre-trained VLAs using object-naming instructions and early-interaction frames from OXE and DROID. Physical properties are decodable, but trail several non-physical properties, especially in the object-text stream.

A. Property Decodability in Pre-trained VLAs

  • Seven pre-trained VLA checkpoints are evaluated on OXE and DROID using early-interaction frames and pick-and-place instructions naming each object.OXE supplies one episode per annotated object; DROID supplies 410 episodes covering the annotated objects.Source: A. Property Decodability in… · p. 4
  • Probes use held-out objects, with ordinal test pairs formed only among held-out objects and the best-performing layer selected by cross-validation.Source: A. Property Decodability in… · p. 4
  • Object-text mass (+0.19) and fragility (+0.19) rank 11th and 10th of 12 properties, below category (+0.41), material (+0.37), sound (+0.31), and price (+0.30).Every property is decodable above chance in each stream on average; the physical-property trend also holds in the other streams.Source: A. Property Decodability in… · p. 4

B. Representation Geometry of Pre-trained VLAs

RSA shows that non-physical properties most strongly organize representation geometry across all modality streams. The analysis uses peak layer scores on the same models, datasets, and early-frame representations as the decodability study.

B. Representation Geometry of Pre-trained VLAs

  • RSA uses the same early-frame representations, models, and datasets as the decodability analysis, reporting each stream’s peak across layers.Because RSA fits no parameters, it uses all annotated objects without a train/test split; peak scores are an upper bound.Source: B. Representation Geometry of… · p. 4
  • Non-physical properties are the strongest organizers of every stream’s geometry; fragility leaves no trace in vision or action geometry despite being decodable from every stream.Object-text is organized most by location (0.29) and material (0.28); price leads in third-person vision (0.28), wrist vision (0.31), and action (0.17).Source: B. Representation Geometry of… · p. 4
  • Vision and action streams share a geometry profile in which price leads and deformability ranks next; mass, friction, and size contribute moderately.Deformability scores are 0.18 for third-person vision, 0.26 for wrist vision, and 0.14 for action.Source: B. Representation Geometry of… · p. 4

C. Base VLMs vs. Pre-trained VLAs

The section compares base VLMs with their pre-trained VLAs to test how robot pre-training changes physical-property decodability and representation geometry. Pre-training weakens physical-property encoding in object-text while not making physical properties more prominent in RSA.

C. Base VLMs vs. Pre-trained VLAs

  • The comparison uses each VLA and its corresponding base VLM on the same pick-and-place instruction and third-person frame, without generation.The analysis asks how further training on physical-interaction data changes decodability and representation geometry.Source: C. Base VLMs vs. Pre-trained VLAs · p. 4
  • Robot pre-training lowers object-text decodability for 11 of 12 properties; three of the four largest drops are fragility (−0.10), mass (−0.08), and deformability (−0.07).Vision decodability changes little overall (+0.02 average), with mass (+0.08) and fragility (+0.03) among its largest gains.Source: C. Base VLMs vs. Pre-trained VLAs · p. 4
  • Pre-training does not increase physical-property RSA in either stream, while non-physical properties remain the strongest organizers of representation geometry.Object-text RSA falls for all properties; vision RSA changes by at most 0.024.Source: C. Base VLMs vs. Pre-trained VLAs · p. 5

D. Pre-trained VLAs vs. Downstream Dataset FT

This section examines whether downstream LIBERO fine-tuning makes physical properties more prominent in VLA representations, using RSA because the dataset has only 13 objects. It reports that this fine-tuning does not strengthen physical-property representation geometry.

D. Pre-trained VLAs vs. Downstream Dataset FT

  • The study compares five pre-trained VLAs with their released LIBERO fine-tunes, using 104 demonstrations across 13 objects and reporting RSA only because probe estimates are too noisy.Embeddings are averaged across eight episodes per object, using frames from both cameras and the episode’s own LIBERO instruction.Source: D. Pre-trained VLAs vs. Downstream… · p. 5
  • LIBERO fine-tuning fails to make physical properties more prominent in representation geometry: vision-stream physical-property RSA falls while non-physical-property RSA rises.Object-text RSA falls less for physical than non-physical properties (mean −0.01 vs. −0.07); action-stream gains are led by non-physical sound (+0.13).Source: D. Pre-trained VLAs vs. Downstream… · p. 5

VI. LIBERO CASE STUDY: Mass

  • The neighboring LIBERO case study asks whether language or vision cues indicating changed object mass alter VLA actions.It tests this behavioral question separately from the representation analysis.Source: §VI · pp. 5–6

A. Controlled Experimental Setup

The controlled LIBERO setup changes object mass while holding geometry and initial pose fixed, signaling the change through language for sealed containers or vision for open containers. Success declines under Full conditions, although language rewording alone also reduces success.

A. Controlled Experimental Setup

  • The setup isolates mass by fixing object geometry and initial pose, then varying mass and its cue; sealed containers use language and open containers use vision.The language conditions include Original, Rephrase, Empty, and Full; the vision conditions compare Original with Full.Source: A. Controlled Experimental Setup · p. 6, Table I · p. 6

B. Findings: Task Success

  • Each model runs 20 episodes per task and setup from the same initial states, and success is averaged over 13 language-cue and 10 vision-cue tasks.Figure 6 reports standard error across tasks.Source: B. Findings: Task Success · p. 6
  • Full lowers task success for every model under both cue types, with the largest reported drops for SmolVLA and OpenVLA under language and π0-FAST and π0.5 under vision.Language: SmolVLA 0.71 → 0.12 and OpenVLA 0.81 → 0.49; vision: π0-FAST 0.93 → 0.74 and π0.5 0.90 → 0.77.Source: B. Findings: Task Success · pp. 6–7
  • Changing only the object name at default mass in Rephrase or Empty costs nearly as much success as Full, whose added mass lowers success by at most a further 0.08 versus Empty.π0.5 and VLA-JEPA remain within 0.05 of Original in every language setup.Source: B. Findings: Task Success · p. 7

C. Episode Analysis

The episode analysis tests whether actions in the 10 steps after grasp change beyond run-to-run variation when mass cues change. Most models’ lifts remain similar, while the exceptions do not adjust in a way that tracks mass.

C. Episode Analysis

  • The analysis compares 10 commanded actions beginning at the grasp point, identified when gripper opening first falls below 90% of its initial width.It focuses on the post-grasp window because the object first loads the arm then.Source: C. Episode Analysis · p. 7
  • RMS action distance is used because action magnitude matters: lifting twice as fast doubles RMS distance, unlike cosine similarity or correlation.For chunk-planning models, only windows contained in the chunk including the grasp are retained to avoid differences driven by changed post-grasp inputs.Source: C. Episode Analysis · p. 7
  • Most models’ actions under changed setups differ no more than run-to-run variation, indicating that they do not adapt their lift to the cue.The same pattern appears in full action space and in vertical end-effector displacement alone.Source: C. Episode Analysis · p. 7
  • The three exceptions do not track mass: SmolVLA slows its lift even under Rephrase and Empty, while π0-FAST and VLA-JEPA lift faster with visible contents.The latter responses are opposite to the cited human pattern of lifting heavier objects more slowly.Source: C. Episode Analysis · p. 7

VII. CONCLUSION AND DISCUSSION

Across seven policies, physical properties are represented but remain weakly prominent and are not reliably reflected in motion. The authors suggest that success-focused teleoperation data may favor a single motion strategy across objects, and call for data or objectives that make physical properties change the required motion.

VII. CONCLUSION AND DISCUSSION

  • Mass, fragility, deformability, friction, and size are decodable from every modality stream but rank among the least prominent represented properties, and robot pre-training and fine-tuning do not promote them.The conclusion describes awareness as a qualified yes: the properties are present in representations, but weakly represented.Source: §VII · p. 7
  • When the same object is made heavier, lifting actions generally remain within original-mass run-to-run variation; cue-surface changes are exceptions but do not track mass.The exceptions involve a reworded name or visible contents and move actions in a direction unrelated to mass.Source: §VII · p. 7
  • The authors suggest that success-focused teleoperated demonstrations may favor one motion strategy across objects, and call for data or objectives where physical properties change the required motion.They frame the training-signal account as a likely reason, not a demonstrated cause.Source: §VII · p. 7
Loading 2610.06078v1…