PaperlayerView exact revision on arXiv

Source-linked AI summary

CometVLA: Co-Training on an Embodied Data Pyramid towards Physical Understanding

Hanwen Wan, Dafeng Chi, Linbo Zhai, Tianao Shen, Yuzheng Zhuang, Tianle Zhang, Peidong Liu, Liang Lin, Xiaoqiang Ji

arXiv:2608.30289v1cs.RO

TL;DR

VLA models remain brittle because physical-VQA data is often disembodied and misaligned with robot action domains, while the downstream value of improved VLM physical understanding remains unclear. CometVLA introduces aligned embodied data and GAP-token co-training across the embodied data pyramid, achieving stronger manipulation results and a positive VLM–VLA performance relationship. Its evidence is limited by a moderate correlation range and a narrow set of real-robot scenarios and tasks.

  • Problem

    Physical-VQA data is often disembodied and misaligned with robot action domains, and the benefit of improved VLM physical understanding for downstream action generation remains unclear.

  • Method

    CometVLA builds aligned embodied physical-VQA data and uses GAP tokens with co-training across the embodied data pyramid.

  • Results

    CometVLA achieves superior performance on simulation benchmarks and real-world manipulation tasks, while stronger CometBench VLM performance consistently correlates positively with VLA performance.

  • Takeaways & Limitations

    The results support using embodied physical understanding as part of VLA pre-training for downstream manipulation.

  • Takeaways & Limitations

    The correlation analysis uses success rates concentrated around 85%, and real-robot evaluation covers only a subset of manipulation behaviors.

Abstract

from arXiv · show

Vision-language-action (VLA) models remain brittle in manipulation tasks that require physical commonsense. Current physical VQA data is typically disembodied and misaligned with robot action domains. Egocentric videos are used only as auxiliary pre-training. It remains unclear whether improved VLM physical understanding actually benefits downstream action generation. Therefore, we present CometVLA to close this gap. We construct CometData and CometBench, an embodied physical VQA corpus and benchmark strictly aligned with the robot's action data and embodiment. We introduce Global Action Prior (GAP) tokens, a compact learnable bottleneck that isolates task-agnostic motion regularities and lets the action head consume physical commonsense without corrupting the pre-trained VLM backbone. We co-train CometVLA across the embodied data pyramid, spanning teleoperation, simulation, egocentric trajectories, and VQA layers. On real-world manipulation tasks and RoboTwin simulation, CometVLA consistently outperforms strong VLA baselines. Correlation analysis shows that stronger VLM performance on CometBench indicates higher VLA success rates. Results demonstrate that physical understanding pre-training genuinely benefits downstream manipulation.

1 Introduction

The paper identifies a mismatch between VLM representations and the contact-rich, evolving dynamics required by VLAs, alongside physical-VQA data that is largely static and disembodied. CometVLA addresses these gaps through aligned embodied data, GAP tokens, and co-training across the embodied data pyramid.

  • VLM representations diverge from the contact-rich, temporally evolving physical dynamics required for reliable VLA manipulation.
  • Existing physical-knowledge efforts mainly use static or disembodied VQA settings rather than robot-aligned embodied data.
  • CometData and CometBench align embodied physical-commonsense understanding with the recipe and embodiment of robot action data.
  • Global Action Prior tokens let the action head consume physical commonsense and aggregate motion regularities across samples.
  • CometVLA co-trains across the embodied data pyramid and reports robust action generation on simulation benchmarks and real-world manipulation tasks.

2 Related Work

Related work organizes embodied learning around a data pyramid and examines how VLM capabilities transfer to action generation. CometVLA contributes an embodied physical-VQA pipeline and an analytical framing that preserves a full action expert while studying VLM–VLA relationships.

  • 2.1 Action Generation with Embodied Data Pyramid: The embodied data pyramid ranges from abundant VQA and egocentric video to scarcer simulation and real-robot teleoperation data.
  • 2.1 Action Generation with Embodied Data Pyramid: Earlier VLA systems primarily learned action distributions from hundreds of hours of real-robot demonstrations, while later work incorporated simulation and scripted data.
  • 2.2 Physical Understanding for VLAs: Curated embodied or physical VQA datasets target capabilities including grounding, affordance reasoning, task planning, and error detection.
  • 2.3 Correlation Analysis from VLMs to VLAs: Prior correlation studies either couple multiple improvements or simplify the action expert, limiting isolation or practical applicability of VLM representation effects.
  • 2.2 Physical Understanding for VLAs: CometData’s pipeline combines heterogeneous trajectories, physics-aware processing, and category-specific generators spanning 5 domains and 16 categories.

3 Method

CometVLA combines embodied physical VQA data, unified vision-language and action modeling, and a GAP-token interface to connect physical understanding with continuous control.

  • CometData: CometData contains 1 million physics-aware QA pairs and more than 1.8 million images generated from heterogeneous embodied trajectories.Its pipeline extracts structured physical states, selects manipulation-critical keyframes, and synthesizes category-specific grounded questions and answers.
  • CometBench: CometBench reserves 2,000 held-out questions across five domains to evaluate embodied physical reasoning in vision-language models.A large language model judges candidate answers against reference answers for correctness.
  • Unified modeling: The unified architecture processes multi-view observations, language, and FAST actions with a pre-trained Qwen3-VL backbone while a diffusion expert predicts continuous actions.The backbone uses autoregressive language and discretized-action objectives, and the action expert uses flow matching for continuous control.
  • Training objective: The total training objective combines autoregressive language modeling, FAST action-token supervision, and flow-matching action generation as Ltotal = LAR + Lfast + Lfm.A stop-gradient barrier prevents the flow-matching loss from updating the backbone while updating the action head and GAP parameters.
  • Global Action Prior: GAP tokens provide the sole communication interface between the autoregressive VLM and bidirectional diffusion policy, aggregating semantic and temporal context for action generation.The tokens traverse the stop-gradient boundary through a detached latent pathway and condition the DiT-based action expert via cross-attention.

4 Experiments

Experiments evaluate CometVLA on real-robot manipulation, RoboTwin simulation, and analyses of physical-understanding transfer and GAP-token representations.

  • Real-robot experiments: Real-robot evaluation uses a G1 dual-arm robot in a dressing-table organization scenario, with each task tested for 16 trials.Success rates vary across objects because of differences in object size and texture.
  • RoboTwin evaluation: CometVLA achieves 89.24% success on RoboTwin easy and 88.38% on hard, outperforming prior methods including π0.5.The comparison spans discrete-token, diffusion, and flow-matching action-head baselines.
  • Ablations: Replacing embodied physical VQA with an equal amount of generic VQA produces the largest ablation drop, while removing GAP also consistently degrades performance.These ablations use the same training configuration and support contributions from embodied physical VQA and GAP.
  • VLM–VLA correlation: Across semantic domains, CometBench VLM performance consistently positively correlates with RoboTwin VLA success, with Spatial Reasoning showing the strongest correlation.Physics & Dynamics has the second strongest correlation, while Task Understanding shows a moderate correlation.
  • GAP semantics: The GAP representation forms localized interaction-semantic clusters and preserves broad embodiment concepts such as “arm” and “left” in t-SNE visualizations.The analysis uses TF-IDF-clustered task annotations to interpret token-level features.
  • GAP analysis: GAP outperforms VLM-mean in action probing by 7.0% for gripper state and 3.9% for motion magnitude.The probing evaluates action-relevant labels using five-fold cross-validation; motion direction is also reported.

5 Limitations

The study identifies two main limitations: limited statistical support for the correlation analysis and limited coverage of real-robot manipulation scenarios.

  • The correlation analysis uses model checkpoints whose VLA success rates cluster around 85%, limiting the diversity of evidence.More checkpoints with greater performance disparity could strengthen the statistical basis of the observed correlation.
  • The real-robot evaluation covers only a subset of manipulation behaviors, scenarios, and task difficulties.Broader environments and tasks would provide additional evidence about generalizability.

6 Conclusion

CometVLA combines embodied physical VQA data, GAP tokens, and co-training across the embodied data pyramid for physical-commonsense-aware action generation. The authors report superior simulation and real-world manipulation performance, with correlation and action-probing analyses supporting the approach.

  • CometData and CometBench provide an embodied physical VQA corpus and benchmark aligned with the robot action data pyramid.The benchmark shows positive correlation with downstream action performance.
  • GAP tokens let the action head consume physical commonsense and aggregate cross-sample motion regularities through a lightweight information bottleneck.The bottleneck is designed to support physical understanding without requiring the backbone to directly absorb the action pathway.
  • CometVLA co-trains across the full embodied data pyramid for end-to-end action generation with physical understanding.The reported training paradigm jointly leverages the pyramid’s embodied data sources.
  • CometVLA achieves superior performance on simulation benchmarks and real-world dual-arm manipulation tasks.Correlation analysis and action probing further validate the proposed methods.

A Comparisons of Embodied VQA Datasets and Benchmarks

The appendix compares embodied VQA datasets by their data sources and benchmarks by the physical reasoning capabilities and scale they cover.

  • Data sources: Table A1 categorizes datasets by robot-centric and non-robot data sources, including teleoperation, robot simulation, robot ego-centric views, web images, and non-robot simulation.The comparison distinguishes data composition according to whether sources are tied to robot embodiment.
  • Comparison framework: The comparison framework separates dataset composition from the reasoning capabilities evaluated by each benchmark.Together, the tables characterize both what data benchmarks use and what competencies they test.
  • Reasoning capabilities: Table A2 summarizes benchmark coverage across spatial, planning, dynamics, error detection and recovery, affordance reasoning, and tool use.It also reports benchmark scale where available.

B Data Generators

The data generators turn synchronized robot trajectories, telemetry, and task annotations into grounded VQA supervision spanning spatial, temporal, planning, progress, physics, object, motion, error, recovery, and constraint reasoning.

  • Generator pipeline: Each generator consumes synchronized multi-view frames, telemetry, and task annotations, then selects evidence frames and instantiates grounded VQA questions.A vision-language model server is queried to obtain an answer grounded in the selected trajectory evidence.
  • Spatial and temporal reasoning: Spatial and multi-view generators supervise 3D spatial relations, viewpoint-invariant object grounding, and cross-view temporal tracking.The timeline generator combines object matching across views with tracking between two moments.
  • Task understanding: Task-understanding generators cover next-subtask prediction and qualitative or quantitative progress estimation from observable scene evidence.Quantitative progress combines structural subtask completion with the frame’s relative temporal position in the episode.
  • Physics and motion: Physical and object generators supervise causal outcome prediction, dense task-relevant object properties, and contact-centered manipulation recognition.The motion generator links action recognition with contact, causality, stability, and risk cues.
  • Error and safety reasoning: Error, recovery, and constraint generators train detection of incorrect task descriptions, safe correction plans, and direction, force, or space constraints.Recovery planning is conditioned on hypothetical failure while respecting torque and workspace limits.

D Model Training configurations

CometVLA uses a two-stage training setup combining vision-language and vision-language-action data, then fine-tuning on task-specific robot data. The implementation supports bi-manual action prediction, remote inference, and end-effector control.

  • Action representation: Actions represent each arm with position, rotation, and gripper state, producing 14 bi-manual control dimensions.The action space is padded to 32 dimensions for future adaptation.
  • Pre-training: Pre-training mixes general VQA, CometData, teleoperation, simulation, and egocentric trajectory datasets for 100,000 steps.A cosine learning-rate schedule uses a 3e-5 base learning rate and 5,000 warmup steps.
  • Post-training: Post-training initializes from the pretrained checkpoint and fine-tunes on RoboTwin- or real-robot-specific data without freezing modules.The VLA loss scale is 1.0 and the VLM loss scale is 0.1.
  • Deployment: Remote H200 servers perform policy inference while a workstation runs robot middleware and sends end-effector pose commands through WebSocket connections.This architecture separates cloud-based inference from edge control.

E.2 Visualization of the Real-Robot Experiments

The real-robot evaluation covers five object-organization tasks involving coordinated placement by either the left or right arm. A representative failure shows that premature gripper opening can knock over the target and terminate execution for safety.

  • Task coverage: The evaluation includes five real-robot tasks: lotion, serum, makeup sponge, BB cream, and liquid foundation organization.Each task places an object into a specified compartment of a storage box.
  • Task execution: The lotion, serum, and makeup-sponge tasks use the right arm, while BB cream and liquid foundation use the left arm.The target compartments differ across the five tasks.
  • Failure analysis: In a lotion failure, the right gripper opened before reaching the bottle, knocking it over while the left arm remained static as instructed.Testers terminated the execution to avoid potential collisions and unsafe motions.
  • Supplementary evaluation: The appendix reports per-task success counts for the 50-task RoboTwin 2.0 evaluation and provides sampled simulation rollout frames.The supplied passages identify these supplementary evaluations but do not provide the counts.

F.3 Action Analysis

Action analysis examines generated trajectories through raw time series, end-effector paths, frequency spectra, and gripper events. The reported visualizations indicate smooth motion, limited high-frequency jitter, and coordinated bi-manual timing, while per-domain correlations are positive.

  • Action trajectories: The policy’s 14-dimensional action sequence is recorded to diagnose the temporal structure and quality of generated motions.A representative episode contains 481 timesteps and step-like transitions associated with gripper events and target-reaching phases.
  • Bi-manual coordination: The visualizations show smooth, physically plausible trajectories with minimal high-frequency jitter and coordinated timing between the two grippers.The appendix combines action time series, 3D end-effector paths, frequency analysis, and gripper-event timelines.
  • Per-domain correlation: Across domains, CometBench VLM performance and RoboTwin 2.0 VLA success rate have consistently positive linear slopes.Each scatter plot includes a linear fit and Pearson correlation coefficient, with domain-dependent strength.

G Ablation on the GAP Token

The GAP analysis probes whether a compressed representation captures action-relevant physical attributes independently of the action head. It evaluates gripper state, motion direction, and motion magnitude using action chunks and complementary representation analyses.

  • Representation setup: The GAP token is extracted before cross-attention into the DiT action expert and compared with a VLM mean-pooling baseline.Features are collected from a held-out subset of AgiBot-World-Beta using the original training-time pipeline.
  • Linear probing: Linear probing tests whether gripper state, motion direction, and motion magnitude remain separable in the compressed GAP representation.The analysis isolates representation quality from the action head and optimizer.
  • Gripper state: Gripper state is a four-class target encoding whether both grippers are open or closed at the start of each 30-command action chunk.This probes discrete bi-manual control intent.
  • Motion attributes: Motion direction uses six classes derived from left end-effector displacement, while motion magnitude uses static, slow, and fast bins.Direction is assigned among ±x, ±y, and ±z; magnitude thresholds are empirical one-third and two-thirds quantiles.
  • Complementary analysis: The three targets jointly probe discrete control intent, geometric direction, and motion scale, complementing qualitative t-SNE visualizations.Task annotations are clustered with TF-IDF and visualized alongside GAP features.
Loading 2608.30289v1…