PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Data Scaling: Representation-Centric Continued Pre-training for Vision-Language-Action Models

Senqiao Yang, Chengyao Wang, Yuxin Chen, Zixuan Wang, Longxiang Tang, Haokun Gui, Jinhui Ye, Changsheng Lu, Xiaoyang Wu, Mingkang Zhu, Pengguang Chen, Shu Liu, Zhuotao Tian, Hengshuang Zhao, Bei Yu, Jiaya Jia

arXiv:2608.27550v1cs.ROcs.CV

TL;DR

Robot trajectories are costly and sparsely cover the physical interaction space, so VLA continued pre-training must learn transferable representations under limited data. VLAct preserves VLM priors, diversifies action supervision, and aligns action semantics across embodiments, consistently improving downstream performance across simulation, real-world, and unseen-embodiment settings. Its results show that continued pre-training design can improve fixed resource budgets beyond scaling robot data or compute alone.

  • Problem

    Robot trajectories are costly to collect and provide sparse coverage of the physical interaction space, making transferable visual-action representation learning important under fixed data budgets.

  • Method

    VLAct uses VLM-prior preservation, multi-head continuous action co-supervision, and a partially unified cross-embodiment action layout before task-specific fine-tuning.

  • Results

    VLAct consistently improves downstream performance across simulation, real-world, and unseen-embodiment transfer, including 82.6% on LIBERO-Plus and 92.5% on RoboTwin 2.0.

  • Takeaways & Limitations

    Representation-centric continued pre-training can substantially improve what fixed robot-data and compute budgets deliver.

Abstract

from arXiv · show

Scaling robot data is crucial for building generalist Vision-Language-Action (VLA) models, yet robot trajectories are harder to scale than web-scale image-text data because embodied collection is costly and sparsely covers the physical world. This makes representation quality a central bottleneck: under a fixed robot-data budget, continued pre-training must turn limited trajectories into transferable visual-action knowledge rather than merely fit actions. We propose VLAct, a VLA-oriented VLM backbone trained on broad, heterogeneous, multi-embodiment robot data before task-specific fine-tuning. VLAct preserves the broad VLM prior and encourages shared action semantics across embodiments through VLM-prior preservation, multi-head continuous action co-supervision, and a partially unified cross-embodiment action layout, while allowing task-specific action heads during fine-tuning. Across simulation, real-world, and unseen-embodiment transfer, VLAct consistently improves downstream performance under fixed fine-tuning protocols. On LIBERO-Plus and RoboTwin 2.0, VLAct surpasses industrial VLA systems including ABot-M0 and LingBot-VLA, achieving success rates of 82.6% and 92.5%. On RoboDojo, VLAct ranks sixth among all policies by success rate and outperforms all explicitly designated world-action model (WAM) entries on both metrics. Most notably, on RoboCasa-GR1, an unseen humanoid embodiment, VLAct using only 20% of downstream trajectories outperforms the full-data GR00T-N1.6 baseline. These results are obtained using fully open-source data and only a 16-GPU training setup, showing that representation-centric continued pre-training can deliver highly competitive performance under a modest compute budget and is an important independent axis of VLA progress beyond data scaling.

1 Introduction

Robot trajectories are costly to collect and sparsely cover the combinatorial physical interaction space, making transferable representations—not data volume alone—a central VLA bottleneck. VLAct addresses this through representation-centric continued pre-training and reports consistent gains across benchmarks and transfer settings.

  • Motivation: Robot trajectories require embodied collection and remain sparse, uneven samples of scenes, objects, tasks, embodiments, and contact-rich dynamics.
  • Motivation: Under a fixed robot-data budget, continued pre-training must convert limited trajectories into transferable visual-action representations rather than merely fit observed actions.
  • VLAct: VLAct preserves VLM priors, diversifies action-head supervision, and shares action semantics across embodiments through representation-centric continued pre-training.
  • Evaluation: VLAct evaluates multi-embodiment simulation, real-world experiments, and unseen-embodiment transfer under fixed downstream protocols.
  • Results: 82.6% on LIBERO-Plus, 54.8% on VLA-Arena, and 92.5% on RoboTwin 2.0 are reported alongside a 10.66 RoboDojo score and 7.60% success rate.
  • Results: Across comparisons, only the VLM backbone weights change, while action heads, downstream data, optimizers, and fine-tuning budgets remain identical; gains are 7.6–21.4 points.

2 Pilot Study: How Action Supervision Shapes VLA Backbones

Pilot experiments show that action supervision directly shapes VLA backbone representations. Discrete supervision can transfer but loses fine-grained information, whereas single-head continuous supervision can collapse representations toward one decoder geometry.

  • Experimental setup: The pilot fixes the Qwen3-VL-4B backbone while varying pre-training and fine-tuning action heads on LIBERO-Plus and RoboTwin-Clean.
  • Discrete supervision: FAST-token pre-training transfers structure to a continuous GR00T head, but retaining the FAST head suffers from discretization-induced information loss.
  • Continuous supervision: OFT pre-training substantially improves downstream performance with an OFT fine-tuning head, confirming that continuous supervision injects fine-grained action information.
  • Continuous supervision: The OFT-pretrained backbone performs poorly with PI or GR00T, indicating that single-head continuous supervision can induce head-specific representation collapse.
  • Design implication: A transferable backbone should expose action-relevant information for multiple downstream heads instead of encoding it around one head’s decoding geometry.

3 VLAct: Representation-Centric VLA Continued Pre-training

VLAct shapes a reusable VLA backbone by preserving the VLM prior, applying multi-head continuous supervision, and partially unifying action semantics across embodiments. Pre-training heads are discarded during fine-tuning, leaving downstream tasks to use freshly initialized task-specific heads.

  • Preserving the VLM’s Vision-Language Representation: VLAct freezes the vision encoder and shallow LLM layers, mixes caption data with robot trajectories, and updates upper layers to limit representation drift.
  • Fine-tuning: During fine-tuning, VLAct discards pre-training heads and caption data, attaches a freshly initialized task-specific action head, and trains under each baseline’s downstream setting.
  • Preserving the VLM’s Vision-Language Representation: Shallow-layer protection improves downstream performance by 3.7% on LIBERO-Plus and 3.4% on Agilex.
  • Preserving the VLM’s Vision-Language Representation: Captioning data provides the strongest anchor among tested VLM mixtures by supervising objects, attributes, spatial relations, and scene context.
  • Aligning and Diversifying the Action Representation: OFT, PI, and GR00T heads share one backbone latent and action targets, making head diversity a representation-learning signal rather than introducing a new alignment module.
  • Aligning and Diversifying the Action Representation: Multi-head co-supervision improves transfer across downstream heads and can outperform single-head pre-training even when the downstream head matches a pre-training head.
  • Unifying the Action Representation across Embodiments: VLAct shares physically aligned action dimensions across embodiments while masking unavailable or incompatible dimensions.

4 Experiments

VLAct is evaluated across open-data simulation, real-world manipulation, and unseen-embodiment transfer under fixed downstream protocols. It consistently performs strongly across these settings, including competitive results with limited data and compute.

  • Experimental Setup: VLAct uses fully open-source robot datasets, captioning data, Qwen3-VL-4B, and 16 GPUs for continued pre-training.The released setup includes DROID, InternA1, RoboCoin, and MolmoAct.
  • Single-arm Franka Manipulation: 82.6% on LIBERO-Plus, improving over Qwen3VL-OFT’s 75.0% and surpassing Abot-M0 by 2.1%.Both VLAct and Qwen3VL-OFT use the same Qwen3-VL backbone family and OFT downstream head.
  • Bimanual AgileX Manipulation: 92.5% Clean and 90.8% Random success on RoboTwin 2.0 make VLAct-OFT the strongest compared method in the Base setting.The Base setting uses 50 clean trajectories per task, while Data Scaling adds 500 randomized trajectories per task.
  • Real-world Evaluation: 92.5% average success versus 77.5% for the baseline in real-world single-arm short-horizon in-domain tasks.VLAct also achieves 90.0% on each of two short-horizon novel-object tasks, versus 73.3% and 65.0% for the baseline.
  • Unseen-embodiment Transfer: 49.5% RoboCasa-GR1 success with 20% of downstream trajectories exceeds full-data GR00T-N1.6 at 47.6%.GR-1 is held out during VLAct continued pre-training.
  • RoboDojo: VLAct ranks sixth by success rate among 35 RoboDojo policies and exceeds every designated WAM entry on both aggregate metrics.It exceeds X-WAM by 2.97 score points and 3.77 percentage points in success.

5 Related Work

Related work frames generalist robot policies around VLA models, human-video representation learning, Internet-scale foundation models, and cross-embodiment transfer. It also notes reproducibility concerns because frontier systems often rely on closed-source data.

  • Generalist Robot Policies: VLA research fine-tunes web-scale vision-language models on diverse robotic datasets to generalize across instructions and environments.The section contrasts this direction with earlier hierarchical systems using semantic planners and robot-specific skills.
  • Data Pretraining: Frontier VLA systems often rely on closed-source data, which hinders reproducibility.The paper positions its recipe study as using public data.

6 Conclusion

The conclusion presents VLAct as a representation-centric continued-pretraining recipe that improves transfer across tasks, embodiments, and evaluation settings. Its results indicate that better pretraining design can improve performance under fixed data and compute resources.

  • Conclusion: VLAct builds more transferable VLM backbones from limited robot data by preserving VLM priors, avoiding single-head specialization, and sharing action semantics.These design goals define the paper’s representation-centric approach.
  • Conclusion: VLAct improves downstream performance across multi-embodiment simulation, real-world experiments, and unseen-embodiment transfer.The conclusion cites strong results on LIBERO-Plus, RoboTwin 2.0, RoboDojo, and RoboCasa-GR1.
  • Conclusion: Open-source data and a 16-GPU setup yield competitiveness against predominantly industry-backed systems.The conclusion frames continued-pretraining design as an additional contributor beyond robot-trajectory or compute scaling.

Authors

The paper situates VLAct within the development of generalist robotic policies, VLA models, VLMs, and cross-task or cross-embodiment generalization.

  • Background: Foundation models have shifted robotics from task-specific controllers toward generalist policies and VLA-based end-to-end optimization.Earlier approaches commonly used pretrained models as high-level planners connected to robot-specific skills.
  • Generalist Robot Policies: Robot-policy scaling is constrained by the high cost of collecting high-fidelity teleoperation trajectories.Instrumented human trajectories and Internet-scale human videos are being explored as alternative sources.
  • Cross-task and cross-embodiment generalization: Cross-task and cross-embodiment generalization is a central objective in robot learning.Research combines human-video bootstrapping, foundation-model knowledge, and heterogeneous robot datasets.
  • Vision Language Models: Modern VLMs unify multimodal tasks through a vision encoder, projector, and LLM backbone, often producing textual outputs.Visual instruction tuning provides flexibility for open-ended multimodal reasoning.

B.1 Additional Results on Franka

VLAct improves performance across VLA-Arena and DOMINO while ablations show that protecting shallow VLM layers strengthens downstream transfer.

  • VLA-Arena: 54.8% overall success is achieved on VLA-Arena, with VLAct best on every behavioral generalization axis.The result uses official suite weighting over 11 task suites.
  • VLA-Arena: 10.5 points separate VLAct from π0.5 overall, including gains of 21.0 points on Long-Horizon and 11.7 points on Safety.All methods use the same downstream task data.
  • DOMINO: 18.50 SR and 34.20 MS are achieved by VLAct-OFT on DOMINO, leading both reported metrics.DOMINO evaluates one policy on 35 dynamic manipulation tasks.
  • DOMINO: 7.64 SR points and 3.71 MS points are gained over Qwen3VL-OFT on DOMINO under the same downstream setting.The comparison highlights the backbone benefit for dynamic bimanual manipulation.

C.2 Ablation Results

The ablations test representation-preserving auxiliary supervision and show that diverse non-action data can strengthen VLA backbones under fixed robot-data and training budgets.

  • Training design: Caption, grounding, spatial-QA, and language supervision are combined with robot trajectories through an auxiliary VLM cross-entropy objective.Each minibatch contains both robot-trajectory and auxiliary samples, with Ltotal = Laction + 0.5 LVLM-CE.
  • Auxiliary supervision: All tested auxiliary sources improve over robot-only training when robot data, backbone, recipe, and total pre-training budget remain fixed.The sources include captioning, grounding, spatial reasoning, and pure language instruction supervision.
  • Auxiliary supervision: Pure language instruction data improves the robot-only baseline despite being text-only and unrelated directly to robot perception or control.This supports auxiliary co-training as representation preservation and feature diversification beyond direct robotics transfer.

D.3 Effect of Different Auxiliary Co-training Data

Under fixed robot data and pre-training budgets, every auxiliary source improves downstream performance; caption data gives the strongest individual gain, while mixing slightly lowers its sampling frequency.

  • Overall comparison: All auxiliary sources improve LIBERO-Plus performance over robot-only training under fixed robot data, backbone, recipe, and total pre-training budget.The comparison covers captioning, grounding, spatial-QA, and pure language instruction data.
  • Single-source results: 82.6 is achieved with image-caption data, improving the robot-only baseline from 75.0 and giving the strongest single-source gain.The result aligns with captions serving as the primary representation-preservation signal.
  • Single-source results: Pure language instruction data improves over robot-only training despite lacking direct ties to robot perception, visual grounding, or control.Its gain indicates that auxiliary co-training is not explained only by direct transfer from VLM- or robotics-related tasks.
  • Mixed-data results: 82.5 is reached by the mixed-data setting, slightly below caption-only co-training because fixed steps reduce image-caption sampling frequency.The passage characterizes this as a likely sampling effect rather than evidence that mixing is harmful.
  • Conclusion: Caption data provides the strongest anchor, while grounding, spatial, and pure-language instruction data provide additional gains for preserving and enriching the backbone.The ablation supports auxiliary data as targeted representation-preserving supervision rather than generic extra data.

E.1 Head-Diverse Pre-training Prevents Decoder Lock-in

Head-diverse continuous pre-training reduces decoder lock-in and improves compatibility across downstream action heads, while wrap-aware objectives align periodic joint representations.

  • Decoder lock-in: Decoder lock-in describes action features organized for one pre-training decoder and less accessible to alternative downstream decoders.The concern matters because downstream users may choose heads based on embodiment, frequency, horizon, or deployment constraints.
  • Decoder lock-in: 60.5 falls to 55.1 after OFT-only pre-training for PI fine-tuning, whereas OFT+GR00T pre-training raises PI fine-tuning to 63.1.PI is excluded during OFT+GR00T pre-training, making this the key unseen-head comparison.
  • Decoder lock-in: 77.0 is reached when PI is included during pre-training, showing the separate benefit of direct downstream-head exposure.This result is distinct from the unseen-head comparison because PI receives pre-training supervision.
  • Same-head adaptation: Head-diverse pre-training improves matched-head adaptation for OFT, PI, and GR00T by 1.7, 1.6, and 4.3 points, respectively.The gains are moderate relative to pre-training itself but indicate broader compatibility.
  • Unified joint space: Wrapping absolute revolute-joint angles into [−π, π] and wrapping residuals before L1 regression makes physically equivalent states numerically consistent near the boundary.The wrap-aware term applies only to absolute joint-angle dimensions, excluding gripper commands and delta translations.

G Adding Heterogeneous UMI Data to Continued Pre-training

The study tests whether VLAct can absorb heterogeneous trajectories collected with a different embodiment and interface while retaining a controlled continued-pretraining recipe. Adding 20K UMI trajectories improves LIBERO-Plus performance, alongside action-space alignment and action-head design choices used throughout the study.

  • Experimental Control: The heterogeneous-data experiment keeps model architecture, the remaining pre-training mixture, optimization budget, and downstream fine-tuning protocol unchanged.Actions are converted to delta end-effector representations compatible with the partially unified action layout.
  • Adding Heterogeneous UMI Data: 1.1 points: Adding 20K UMI trajectories improves LIBERO-Plus from 82.6 to 83.7 under otherwise unchanged settings.The trajectories come from a different embodiment and UMI-style collection paradigm and receive lightweight filtering without task-specific curation.
  • Data and Action Heterogeneity: The training mixture combines multiple robot platforms and datasets, including Franka single-arm and AgileX dual-arm data with distinct action spaces.The AgileX space has 14 dimensions, while the Franka setting draws on DROID and MolmoAct data.
  • Data Cleaning: The cleaning pipeline removes invalid task annotations and masks unreliable action steps rather than discarding useful trajectories wholesale.A whole action chunk is discarded only when its invalid-step ratio exceeds 0.5.
  • Action Alignment: Action alignment normalizes gripper values to [0, 1], rescales delta end-effector actions per second, and wraps valid joint angles into [−π, π].The procedure creates consistent representations across datasets despite differences in hardware, frame rates, and angular conventions.
  • Action-Head Families: The study compares discrete FAST token prediction with continuous OFT regression and PI/GR00T flow-matching action generation.OFT predicts an entire action chunk in one forward pass, whereas FAST retains an autoregressive token interface and PI/GR00T use separate DiT-style motor modules.

J.3 Detailed Real-World Results and Failure Analysis

Real-world evaluations cover short-horizon, long-horizon, object-shift, sequence-extension, and dual-arm tasks using success-rate protocols tailored to task type. VLAct outperforms the Qwen3VL-4B-OFT baseline across these settings, with gains especially visible under distribution shift and multi-stage manipulation.

  • Evaluation protocol: Real-machine success rates use 10 rollouts per task, with binary scoring for short-range and collaborative tasks and step-based scoring for long-range tasks.For table cleaning, each of three completed steps contributes 0.33 points.
  • Single-arm short-horizon tasks: 92.5% average success rate: VLAct outperforms the baseline’s 77.5% on in-domain single-arm short-horizon tasks.The largest cited in-domain gap is on cube stacking, where VLAct reaches 90% versus 60% for the baseline.
  • Short-horizon OOD generalization: 90.0% average success rate: VLAct exceeds the baseline’s 73.3% and 65.0% on novel-object from-pot and in-cup tasks, respectively.These object-level shifts replace familiar objects with unseen categories such as egg, pepper, garlic, cube, and egg.
  • Single-arm long-horizon tasks: 86.6% weighted success rate: VLAct beats the baseline’s 73.3% on table cleaning and reaches 80.0% versus 33.3% on scooping beans.The reported behaviors emphasize preserving the intended sequence across multiple manipulation stages.
  • Long-horizon OOD generalization: 82.5% weighted success rate: VLAct nearly doubles the baseline’s 47.5% under sequence extension, while scoring 83.3% versus 46.6% under full object substitution.The baseline often misses objects or fails to recover from grasping errors in these harder OOD settings.
  • Dual-arm coordination: 72.0% average success rate: VLAct outperforms the baseline’s 44.0% across dual-arm coordination tasks.The cited tasks include breakfast preparation, unplugging, banana handover-place, and deformable-object folding.
Loading 2608.27550v1…