PaperlayerView exact revision on arXiv

Source-linked AI summary

RoboTwin-Phys: Do WAMs and VLAs Understand the Physical World?

Jiaqi Zhang, Feng Ye, Mingjia Yang, Zhihong Chen, Mingkang Xiang, Xinglin Yao, Yanbin Li, Siwei Ma, Chuanmin Jia

arXiv:2609.26292v1cs.RO

TL;DR

Robot-manipulation benchmarks often vary visual and scene conditions while keeping physical parameters fixed, leaving robustness to physical-condition diversity insufficiently measured. RoboTwin-Phys adds continuous variation across 13 physical attributes and releases annotated expert demonstrations. Evaluations find that representative WAMs and VLAs can degrade substantially under physical variation despite effectiveness under visual and layout randomization.

  • Problem

    Existing manipulation benchmarks commonly keep physical parameters fixed, and lack systematic evaluation, broad physical-condition training data, and ground-truth physical labels.

    Source: §1 · p. 2

  • Method

    RoboTwin-Phys adds 13 continuously varied physical attributes within plausible, task-aware ranges and releases more than 5,000 expert demonstrations annotated with ground-truth physical parameters.

    Source: §1 · pp. 2–3

  • Results

    Representative WAMs and VLAs that perform effectively under visual and layout randomization can substantially degrade under physical-condition diversity.

    Source: §1 · p. 3, §5 · p. 7

  • Takeaways & Limitations

    RoboTwin-Phys provides a testbed for evaluating physical-condition robustness and studying physical-condition perception, condition-aware modeling, and adaptive robot control.

    Source: §4.4 · p. 7, §5 · p. 7

Abstract

from arXiv · show

Physical-condition diversity is largely missing from current benchmarks for robot manipulation. While large-scale simulation benchmarks increasingly incorporate variations in object appearance, scene layout, and visual observations, they typically keep the underlying physical parameters fixed. As a result, important sources of real-world variability, such as changes in mass, friction, and joint dynamics, remain largely untested. We introduce RoboTwin-Phys, a physics-diverse benchmark that treats physical-condition diversity as an explicit dimension of robot manipulation evaluation. The benchmark continuously varies 13 physical attributes within physically plausible ranges, providing a unified setting for evaluating policies across diverse physical operating conditions. We further release more than 5,000 expert demonstrations with ground-truth physical parameters, enabling physical-attribute estimation, condition-aware modeling, and physics-conditioned policy training. Evaluations of representative WAMs and VLAs reveal a substantial robustness gap: models that remain effective under existing visual and layout randomization can degrade markedly under changes in physical conditions. RoboTwin-Phys provides the benchmark, data, and evaluation protocol needed to systematically measure and improve robustness to physical-condition diversity in robot manipulation.

1 Introduction

Existing manipulation benchmarks vary visual and scene conditions but commonly hold physical parameters fixed, leaving physical-condition robustness without systematic evaluation. RoboTwin-Phys adds continuous physical variation, annotated demonstrations, and evaluation of representative WAMs and VLAs, which degrade under physical changes.

1 Introduction

  • Current benchmarks often vary appearance and layout while fixing physical parameters, and lack systematic protocols, diverse training data, and ground-truth physical labels.Prior studies also tend to examine isolated factors or small-scale ablations rather than joint variation.Source: §1 · p. 2
  • RoboTwin-Phys adds 13 continuously sampled physical attributes within plausible ranges to RoboTwin-2.0’s 50-task bimanual platform.It augments the existing tasks rather than introducing new manipulation tasks.Source: §1 · pp. 2–3
  • Representative WAMs and VLAs that remain effective under visual and layout randomization can substantially degrade when physical conditions vary.The benchmark identifies a robustness dimension largely unmeasured by existing protocols.Source: §1 · p. 3
  • The released dataset contains more than 5,000 expert demonstrations with 13-dimensional ground-truth physical parameters for each episode.The data support physical-condition estimation, condition-aware representation learning, and physics-conditioned policy training.Source: §1 · pp. 2–3
  • The benchmark provides a systematic protocol for comparing representative WAMs and VLAs under diverse physical conditions.Its evaluation characterizes the resulting robustness gap across models.Source: §1 · p. 3

2 RoboTwin-Phys Benchmark

RoboTwin-Phys treats physical conditions as a composable evaluation dimension, sampling attributes continuously while preserving physically plausible, task-feasible environments. Its design uses episode-level sampling, task-specific ranges where needed, and expert-planner checks before including instances.

2.1 Benchmark Design

  • The benchmark varies physical conditions alongside or separately from visual and layout variation, with each episode receiving a complete condition fixed during execution.Its design emphasizes physical validity, continuous diversity, and composability.Source: §2.1 · p. 3

2.2 Physical-Condition Sampling

  • The 13 physical attributes are sampled independently from continuous ranges at environment initialization and remain fixed through the rollout.Episode seeds make the sampled conditions reproducible; the attributes cover object, contact, damping, environment, and camera properties.Source: §2.2 · p. 4, Figure 2 · p. 4

2.3 Task-Aware Physical Validity

  • Nine physics-sensitive tasks use dedicated configurations calibrated to their sensitive factors, while other tasks use the global configuration.Task-aware ranges avoid configurations that are numerically valid but implausible for a task.Source: §2.3 · p. 4

2.4 Feasibility and Benchmark Instances

  • An instance is retained only if an expert planner can complete the task under its sampled condition, and the same feasibility principle guides demonstration generation.This check focuses evaluation on meaningful, executable operating conditions rather than settings without a valid solution.Source: §2.4 · pp. 4–5

3 Dataset

RoboTwin-Phys Training Set v1 makes physical conditions explicit in a reusable collection of expert demonstrations. Each episode pairs observations and actions with ground-truth physical attributes, supporting learning tasks that use those conditions directly.

3 Dataset

  • RoboTwin-Phys Training Set v1 samples expert demonstrations from the benchmark’s physical-condition space, making conditions observable learning targets rather than hidden simulator states.The collection contains more than 5,000 executable expert episodes spanning the physical-condition distribution.Source: §3 · p. 5, §3.1 · p. 5

3.2 Physical Ground Truth

  • Each demonstration includes the 13 ground-truth physical attributes instantiated during its rollout.The explicit correspondence between observations, actions, and physical conditions supports physical-attribute estimation, condition-aware representation learning, and policy training conditioned on physical attributes.Source: §3.2 · p. 5
  • The dataset turns physical-condition diversity into a reusable learning resource that complements the benchmark’s evaluation protocol.Its demonstrations come from valid physical operating environments rather than arbitrary simulator configurations.Source: §3.2 · p. 5, §3.1 · p. 5

4 Experiments

Across representative WAMs and VLAs, physical randomization exposes substantial degradation beyond the comparatively small changes under conventional visual and layout randomization. Task-wise outcomes show repeatable variation, with some manipulation behaviors more vulnerable than others.

4.2 Overall Results

  • All five evaluated models show pronounced degradation under Physical Random, and the challenge appears across WAMs and VLAs rather than one model design.The overall results report success rates of 44.24% for Fast-WAM, 39.60% for Motus, 39.14% for FACT, 31.60% for π0.5, and 37.82% for Galaxea-VLA under Physical Random.Source: §4.2 · p. 6, Table 1 · p. 6
  • Clean-to-Official-Random differences are comparatively small for the three WAMs, while physical variation reveals a distinct robustness dimension.The reported decreases are 0.10 points for Fast-WAM, 1.64 points for Motus, and 1.80 points for FACT; π0.5 decreases by 5.98 points and Galaxea-VLA by 0.90 points.Source: §4.2 · p. 6

4.3 Task-Level Analysis

  • Task-level outcomes vary systematically: hammering, bottle-picking, stapler-moving, and bottle-placement tasks are challenging, while roller-grabbing and bottle-shaking tasks remain comparatively robust.The pattern is shared across models and particularly stresses behaviors involving contact dynamics, object stability, mass distribution, or articulated motion.Source: §4.3 · pp. 6–7

4.4 Discussion

  • The experiments characterize physical-condition diversity as a broad stressor with structured, reproducible task-level effects across WAMs.Ground-truth conditions provide a basis for investigating whether failures relate to factors such as mass, friction, sensing configuration, or articulated dynamics.Source: §4.4 · p. 7

4 Experiments

  • The study supports physical-condition diversity as an independent evaluation dimension and RoboTwin-Phys as a testbed for physical-condition perception, condition-aware modeling, and adaptive control.The setup evaluates representative WAMs and VLAs across three environment protocols.Source: §4 · p. 5, §4.1 · pp. 5–6, §4.4 · p. 7

5 Conclusion

RoboTwin-Phys adds physical-condition diversity to manipulation evaluation and provides demonstrations with per-episode physical ground truth. Evaluations show that robustness to visual and layout variation does not necessarily transfer to physical variation.

5 Conclusion

  • RoboTwin-Phys varies 13 physical attributes within physically plausible, task-aware domains to evaluate policies across diverse operating conditions.The accompanying training set contains more than 5,000 expert demonstrations with per-episode ground-truth physical parameters.Source: §5 · p. 7
  • The benchmark and demonstrations provide a reusable resource for physical-condition estimation and condition-aware policy learning.The authors hope the benchmark encourages models to perceive, reason about, and adapt to changing physical conditions.Source: §5 · p. 7
  • Robustness to visual and layout randomization does not necessarily extend to physical-condition diversity, where representative WAMs and VLAs show substantial performance degradation.The authors frame physical variation as a distinct robustness dimension that existing benchmarks largely leave unmeasured.Source: §5 · p. 7

A Physical Attribute Definitions and Global Ranges

RoboTwin-Phys characterizes episode-level physical conditions with 13 attributes spanning several physical domains. Its global configuration is sampled at episode initialization and remains fixed throughout each rollout.

A Physical Attribute Definitions and Global Ranges

B Task-Specific Physical Domains

Task-specific physical domains address cases where global ranges would produce configurations inconsistent with a task's geometry or operation. Nine sensitive tasks receive dedicated configurations calibrated to preserve diversity while maintaining plausible conditions.

B Task-Specific Physical Domains

  • Global ranges can yield simulator configurations that are numerically valid but physically implausible for tasks with tighter geometric or operational constraints.For example, the global center-of-mass range can place the slender hammer in beat_block_hammer outside its physical body.Source: B Task-Specific Physical Domains · p. 8
  • Nine physics-sensitive tasks receive dedicated configurations, while attributes not listed retain their global settings.Table 3 presents the task-specific configurations.Source: B Task-Specific Physical Domains · p. 8, Table 3 · p. 9
  • Task-specific ranges are calibrated to preserve non-trivial physical diversity while avoiding combinations inconsistent with task geometry or operation.Admissible ranges can vary substantially for center-of-mass displacement, joint damping, camera displacement, and object scaling.Source: B Task-Specific Physical Domains · p. 8

C Data Format and Organization

The released demonstrations remain compatible with the official RoboTwin data format and include trajectory, video, and physical-condition metadata. A manifest maps tasks to benchmark configurations and episode counts.

C Data Format and Organization

  • The demonstrations are compatible with the official RoboTwin format, allowing existing WAM and VLA pipelines to consume them without trajectory-interface modifications.Source: C Data Format and Organization · p. 9
  • Each task directory includes HDF5 trajectories with actions, states, and camera streams, alongside corresponding MP4 recordings.Source: C Data Format and Organization · p. 9
  • Each episode's 13 instantiated physical attributes are recorded in JSON metadata, and a manifest maps tasks to configurations and episode counts.Source: C Data Format and Organization · p. 9

D Per-Task Physical Random Results

Table 4 reports per-task success rates for five evaluated models under RoboTwin-Phys’s Physical Random setting, with 100 rollout episodes per task.

D Per-Task Physical Random Results

Loading 2609.26292v1…