PaperlayerView exact revision on arXiv

Source-linked AI summary

A Systematic Study of Data Modalities and Strategies for Co-training Large Behavior Models for Robot Manipulation

Fanqi Lin, Kushal Arora, Jean Mercat, Haruki Nishimura, Paarth Shah, Chen Xu, Mengchao Zhang, Mark Zolotas, Maya Angeles, Owen Pfannenstiehl, Andrew Beaulieu, Jose Barreiros

arXiv:2602.01067v1cs.RO

TL;DR

Robot-data scarcity leaves large behavior models with limited generalization, and the effectiveness of heterogeneous co-training signals is not well established. This study systematically compares five modalities and training strategies across large simulation and real-world evaluations, finding that vision-language and cross-embodiment co-training improve generalization while discrete action tokens do not; combining effective modalities further improves adaptation and preserves backbone capabilities.

Abstract

from arXiv · show

Large behavior models have shown strong dexterous manipulation capabilities by extending imitation learning to large-scale training on multi-task robot data, yet their generalization remains limited by the insufficient robot data coverage. To expand this coverage without costly additional data collection, recent work relies on co-training: jointly learning from target robot data and heterogeneous data modalities. However, how different co-training data modalities and strategies affect policy performance remains poorly understood. We present a large-scale empirical study examining five co-training data modalities: standard vision-language data, dense language annotations for robot trajectories, cross-embodiment robot data, human videos, and discrete robot action tokens across single- and multi-phase training strategies. Our study leverages 4,000 hours of robot and human manipulation data and 50M vision-language samples to train vision-language-action policies. We evaluate 89 policies over 58,000 simulation rollouts and 2,835 real-world rollouts. Our results show that co-training with forms of vision-language and cross-embodiment robot data substantially improves generalization to distribution shifts, unseen tasks, and language following, while discrete action token variants yield no significant benefits. Combining effective modalities produces cumulative gains and enables rapid adaptation to unseen long-horizon dexterous tasks via fine-tuning. Training exclusively on robot data degrades the visiolinguistic understanding of the vision-language model backbone, while co-training with effective modalities restores these capabilities. Explicitly conditioning action generation on chain-of-thought traces learned from co-training data does not improve performance in our simulation benchmark. Together, these results provide practical guidance for building scalable generalist robot policies.

I. INTRODUCTION

Large behavior models remain limited by robot-data scarcity, motivating systematic co-training with heterogeneous modalities. This study evaluates a VLA policy, five co-training modalities, training strategies, and broad simulation and real-world benchmarks.

  • Robot datasets are orders of magnitude smaller than internet-scale corpora, limiting LBMs’ semantic, spatial, and open-world generalization.Source: §I · p. 1
  • The policy combines a pretrained VLM backbone with an Action Flow Transformer and supports continuous-action flow matching alongside discrete-token cross-entropy objectives.Source: §I · p. 1, 2) Model Architecture · p. 3, 1) Problem Formulation · p. 3
  • The study compares standard vision-language data, dense trajectory language annotations, cross-embodiment robot data, human videos, and discrete robot action tokens.Source: §I · pp. 1–2
  • Co-training is evaluated across seen and unseen tasks, nominal and distribution-shift settings, language following, and long-horizon dexterous manipulation.Source: Figure 1 · p. 2, §I · p. 2
  • The study trains and compares 89 policies using about 4,000 hours of manipulation data and 50M vision-language samples across 58,000 simulation and 2,835 real-world rollouts.Source: §I · p. 2

3) Co-training and Inference Strategies:

The study organizes co-training around multiple data phases and a unified dataset spanning target-robot demonstrations, vision-language resources, annotations, and cross-embodiment data. These choices support controlled comparisons of when and how heterogeneous supervision is introduced.

Co-training and Inference Strategies

Target Robot Data

  • The unified dataset contains approximately 4,000 hours of manipulation data and 50M vision-language samples spanning robot, human, standard VL, and dense annotation sources.Source: B. Data Curation · p. 3, Figure 2 · p. 4
  • TRI-Ramen contains 523 hours across 403 tasks and 53,411 demonstrations collected through teleoperation on dual Franka Panda arms.Source: 1) Target Robot Data · p. 4
  • Robot observations contain four RGB images and a natural-language instruction, while actions are relative trajectories with a 16-step horizon.Source: 1) Target Robot Data · p. 4

Dense Language Annotations

Cross-embodiment Robot Data

5) Human Videos:

Human videos are converted into motion-relevant supervision through either learned latent action tokens or VLM-generated language annotations. The latent-action pipeline learns codes from visual dynamics and robot-action reconstruction where ground-truth actions exist.

Latent Actions

  • The latent-action approach uses 2,271 hours of filtered egocentric human videos from multiple public datasets.Source: 5) Human Videos · p. 5
  • A latent action model combines an inverse dynamics model, visual forward dynamics model, and action forward dynamics model to learn a quantized codebook.Source: 5) Human Videos · p. 5
  • Robot supervision reconstructs ground-truth action chunks, whereas human-video training omits action reconstruction because ground-truth actions are unavailable.Source: 5) Human Videos · p. 5
  • The pipeline concatenates latent segments and quantizes each representation into 8 discrete tokens from a codebook of size C = 32.Source: 5) Human Videos · p. 5

VLM-generated Annotations

  • The alternative human-video approach uses a VLM to generate per-frame descriptions of motions, goals, and objects from visual context and task instructions.Source: 5) Human Videos · p. 5

6) Discrete Robot Action Tokens:

The study examines discrete robot action tokens as co-training modalities alongside broader evaluation settings for robot policies. It compares tokenization approaches and training configurations across simulation, real-world language following, and long-horizon adaptation.

Discrete action-token modalities

  • FAST tokens compress continuous action chunks into near-lossless discrete sequences, while VQ-VAE provides an alternative discrete action representation.The study evaluates both token types as co-training modalities.Source: 6) Discrete Robot Action Tokens · p. 5

Real-world evaluation

  • Real-world evaluation measures language following with seen and unseen objects, instruction generalization through paraphrasing, and adaptation to unseen long-horizon dexterous tasks.The language-following suite includes seen objects, varied instruction phrasing, and objects absent from target-robot training data.Source: Figure 3 · p. 6, 2) Real-world Evaluation · pp. 6–7

Evaluation settings

  • The research evaluates policies on in-distribution performance, distribution-shift robustness, unseen tasks, and language following, with additional fine-tuning on unseen long-horizon dexterous tasks.Simulation includes 13 seen and 8 unseen tasks under nominal and distribution-shift conditions.Source: A. Evaluation · p. 6, 1) Simulation Benchmark · p. 6, Figure 3 · p. 6

Unseen-task evaluation

  • The benchmark’s unseen tasks test semantic understanding, multi-step manipulation, and compositional generalization beyond the training data.Underlying objects and environments may remain familiar even though the skills themselves are absent from training.Source: 1) Simulation Benchmark · p. 6

3) Statistical Analysis Framework:

The paper uses statistical comparisons to evaluate co-training modalities and phases against a no-co-training baseline. Results favor diverse vision-language and cross-embodiment data, whereas discrete action-token variants provide limited or no benefit.

Statistical analysis

  • Pairwise hypothesis tests and Compact Letter Displays identify significant differences at a 5% family-wise error rate, with Bayesian uncertainty shown for individual strategies.The analysis also visualizes empirical and posterior uncertainty over policy performance.Source: 3) Statistical Analysis Framework · p. 7

Training-strategy comparisons

  • Each modality is tested with single-phase, two-phase first-phase-only, and two-phase full co-training against a continuous-action no-co-training baseline.Effective modalities are subsequently evaluated in real-world language-following experiments.Source: different training phases, influence… · p. 7

Effective modalities

  • Standard vision-language data improves distribution-shift robustness, unseen-task generalization, and language following, with continued second-phase co-training further helping unseen-task and language-following performance.These gains do not significantly change in-distribution performance.Source: different training phases, influence… · p. 7
  • VLM-generated annotations for robot trajectories improve robustness, unseen-task generalization, and language following, but adding them during the second phase provides no additional benefit.VLM-based annotations yield larger improvements than scripted annotations.Source: different training phases, influence… · p. 7
  • Cross-embodiment robot data is most effective when confined to first-phase training, particularly for unseen-task generalization and distribution-shift robustness.Continuing it during the second phase yields negligible additional language-following benefit.Source: different training phases, influence… · p. 7

Discrete-token ablations

  • Discrete action-token co-training yields no statistically significant improvement overall; FAST tokens can reduce unseen-task generalization, while VQ-VAE tokens show no improvement in data- and compute-rich comparisons.Latent-action benefits are limited to low target-robot-data regimes and diminish as fine-tuning data increases.Source: different training phases, influence… · p. 9

C. Does combining effective co-training modalities yield cumulative performance gains?

Combining effective co-training modalities produces cumulative gains across simulation and real-world evaluations, while fine-tuning the combined model enables rapid adaptation to unseen dexterous tasks. Explicit CoT conditioning does not improve the simulation benchmark over implicit co-training.

IV. RELATED WORK

Related work frames co-training as a response to the severe robot-data scarcity limiting embodied model generalization. This paper systematically studies heterogeneous modalities and finds benefits from vision-language and cross-embodiment data, cumulative gains from combining them, and scope-dependent CoT effects.

  • Robot datasets remain orders of magnitude smaller than internet-scale text and image corpora, contributing to limited embodied-model generalization.Source: A. Large Behavior Models · pp. 12–13
  • Co-training jointly uses target-robot and heterogeneous data to bridge the robot-data gap.Source: B. Co-training for Robot Learning · p. 13
  • Prior work explores public vision-language data, robot-trajectory annotations, cross-embodiment robot data, human videos, and discrete action tokens.Source: B. Co-training for Robot Learning · p. 13
  • Discrete action tokens can limit low-level control precision and slow inference, motivating alternative co-training strategies.Source: B. Co-training for Robot Learning · p. 13
  • This study reports that vision-language and cross-embodiment co-training improve distribution-shift, unseen-task, and language-following generalization, while discrete action tokens provide no statistically significant benefits.Source: §V · p. 13
  • The study’s limitations include no task-taxonomy analysis of vision-language data, coarse human-video representations, restricted CoT forms, and an imitation-learning-only scope.Source: §V · p. 14

A. Model Architecture Ablation.

The appendix documents architecture, training, data, and hyperparameter choices for the policy ablations. It shows that balancing losses and robot-to-co-training ratios trades off in-distribution performance against generalization benefits.

  • The architecture ablation compares the proposed VLM-plus-flow-transformer policy with πFAST-equivalent and π0-equivalent architectures.Source: A. Model Architecture Ablation · p. 19
  • The study treats flow-matching versus cross-entropy weighting and the target-robot-to-co-training ratio as empirically critical hyperparameters.Source: B. Hyperparameter Ablation · p. 19
  • Increasing either the loss weight or co-training data ratio degrades in-distribution performance, while setting them too low reduces co-training generalization benefits.Source: B. Hyperparameter Ablation · p. 19
  • The selected configuration uses w = 0.02 and a 9:1 robot-to-co-training ratio for unpaired modalities.This configuration preserves in-distribution performance while maximizing generalization improvements.Source: B. Hyperparameter Ablation · p. 19
  • Cross-embodiment robot data is mixed during first-phase training with a 6:4 target-robot-to-cross-embodiment batch ratio.Source: B. Hyperparameter Ablation · p. 19
  • Each standard multi-task training phase runs for 200k steps and requires approximately 64 hours on 16 H100 GPUs.Source: §2 · p. 19
  • The VQ-VAE compresses horizon-16, dimension-20 action chunks into 8 discrete tokens using a codebook of size 32.Source: C. Discrete Robot Action Tokens · pp. 20–21

4. POLICY DEPLOYMENT

Deployment uses chunked action prediction on a dual-arm Franka platform, with temporal ensembling for smoother real-world execution. Evaluation compares policies statistically across controlled simulation conditions and rollouts.

  • Real-world deployment uses continuous inference and temporal ensembling to address discontinuities between predicted action chunks.Average inference latency is 0.146 seconds.Source: §4 · p. 21
  • Average task-completion comparisons use Welch’s t-test, while success-rate comparisons use the STEP test.Source: §5 · p. 21
  • Pairwise tests control family-wise error at 5% with Bonferroni correction when comparing k strategies.Source: §5 · p. 21
  • Bayesian uncertainty is estimated with uniform Beta priors for success rates and uniform Dirichlet priors for task-progress values.Source: §5 · pp. 21–22
  • The simulation benchmark contains 13 seen and 8 unseen tasks under nominal and distribution-shift conditions.Source: §6 · p. 22

A. Language Following.

Language-following evaluation measures task completion through four ordered milestones, from reaching and grasping the target to placing it according to the instruction.

B. Long-horizon Dexterous Manipulation.

The long-horizon benchmark decomposes three dexterous tasks into ordered milestones covering object manipulation, bimanual coordination, and multi-step placement outcomes.

Sunglasses

Drawstring bag

Soup preparation

C. Experimental Procedure

Real-world evaluation controls environmental variation by testing all policy checkpoints on the same initial condition before changing the scene.

D. Rubric QA

The rubric quality-assurance procedure uses blinded secondary review to estimate human evaluation discrepancies in real-world rollouts.

  • 895 rollouts, representing 31.6% of all real-world evaluations, were independently reviewed by a separate reviewer pool.Reviewers were not told which checkpoint each video represented.Source: D. Rubric QA · p. 23
  • 2.01% was the overall task-completion discrepancy across episodes and milestones.Source: D. Rubric QA · p. 23

Related papers

Loading 2602.01067v1…