Source-linked AI summary
A Systematic Study of Data Modalities and Strategies for Co-training Large Behavior Models for Robot Manipulation
Fanqi Lin, Kushal Arora, Jean Mercat, Haruki Nishimura, Paarth Shah, Chen Xu, Mengchao Zhang, Mark Zolotas, Maya Angeles, Owen Pfannenstiehl, Andrew Beaulieu, Jose Barreiros
TL;DR
Robot-data scarcity leaves large behavior models with limited generalization, and the effectiveness of heterogeneous co-training signals is not well established. This study systematically compares five modalities and training strategies across large simulation and real-world evaluations, finding that vision-language and cross-embodiment co-training improve generalization while discrete action tokens do not; combining effective modalities further improves adaptation and preserves backbone capabilities.
Problem
Robot datasets are orders of magnitude smaller than internet-scale corpora, while existing co-training studies provide inconsistent evidence about which modalities are effective.
Method
The study trains and evaluates VLA policies using five co-training modalities and multiple training strategies across robot and human manipulation data.
Results
Vision-language and cross-embodiment co-training improve distribution-shift, unseen-task, and language-following performance, while discrete action-token variants provide no significant benefit; combined modalities yield cumulative gains.
Takeaways & Limitations
Effective co-training enables rapid fine-tuning adaptation to unseen long-horizon dexterous tasks and helps preserve the VLM backbone’s visiolinguistic understanding.
Takeaways & Limitations
The reported conclusion about explicit chain-of-thought conditioning is limited to the manipulation tasks and simulation benchmark studied.
Abstract
from arXiv · showhide
Large behavior models have shown strong dexterous manipulation capabilities by extending imitation learning to large-scale training on multi-task robot data, yet their generalization remains limited by the insufficient robot data coverage. To expand this coverage without costly additional data collection, recent work relies on co-training: jointly learning from target robot data and heterogeneous data modalities. However, how different co-training data modalities and strategies affect policy performance remains poorly understood. We present a large-scale empirical study examining five co-training data modalities: standard vision-language data, dense language annotations for robot trajectories, cross-embodiment robot data, human videos, and discrete robot action tokens across single- and multi-phase training strategies. Our study leverages 4,000 hours of robot and human manipulation data and 50M vision-language samples to train vision-language-action policies. We evaluate 89 policies over 58,000 simulation rollouts and 2,835 real-world rollouts. Our results show that co-training with forms of vision-language and cross-embodiment robot data substantially improves generalization to distribution shifts, unseen tasks, and language following, while discrete action token variants yield no significant benefits. Combining effective modalities produces cumulative gains and enables rapid adaptation to unseen long-horizon dexterous tasks via fine-tuning. Training exclusively on robot data degrades the visiolinguistic understanding of the vision-language model backbone, while co-training with effective modalities restores these capabilities. Explicitly conditioning action generation on chain-of-thought traces learned from co-training data does not improve performance in our simulation benchmark. Together, these results provide practical guidance for building scalable generalist robot policies.
I. INTRODUCTION
Large behavior models remain limited by robot-data scarcity, motivating systematic co-training with heterogeneous modalities. This study evaluates a VLA policy, five co-training modalities, training strategies, and broad simulation and real-world benchmarks.
- Robot datasets are orders of magnitude smaller than internet-scale corpora, limiting LBMs’ semantic, spatial, and open-world generalization.
- The policy combines a pretrained VLM backbone with an Action Flow Transformer and supports continuous-action flow matching alongside discrete-token cross-entropy objectives.
- The study compares standard vision-language data, dense trajectory language annotations, cross-embodiment robot data, human videos, and discrete robot action tokens.
- Co-training is evaluated across seen and unseen tasks, nominal and distribution-shift settings, language following, and long-horizon dexterous manipulation.
- The study trains and compares 89 policies using about 4,000 hours of manipulation data and 50M vision-language samples across 58,000 simulation and 2,835 real-world rollouts.
3) Co-training and Inference Strategies:
The study organizes co-training around multiple data phases and a unified dataset spanning target-robot demonstrations, vision-language resources, annotations, and cross-embodiment data. These choices support controlled comparisons of when and how heterogeneous supervision is introduced.
- Co-training and Inference Strategies: The study compares single-phase co-training, two-phase co-training only in phase one, and two-phase full co-training.
- Co-training and Inference Strategies: Co-training is treated primarily as auxiliary supervision, while selected experiments condition action prediction on language or latent-action chain-of-thought traces.
- Target Robot Data: The unified dataset contains approximately 4,000 hours of manipulation data and 50M vision-language samples spanning robot, human, standard VL, and dense annotation sources.
- Target Robot Data: TRI-Ramen contains 523 hours across 403 tasks and 53,411 demonstrations collected through teleoperation on dual Franka Panda arms.
- Target Robot Data: Robot observations contain four RGB images and a natural-language instruction, while actions are relative trajectories with a 16-step horizon.
- Dense Language Annotations: Dense robot annotations combine heuristic action primitives with VLM-generated frame-level descriptions of actions, goals, objects, and spatial relations.
- Cross-embodiment Robot Data: OXE-Ramen provides cross-embodiment data spanning 1,150 hours, 12 robot setups, 924 tasks, and 466,415 demonstrations.
5) Human Videos:
Human videos are converted into motion-relevant supervision through either learned latent action tokens or VLM-generated language annotations. The latent-action pipeline learns codes from visual dynamics and robot-action reconstruction where ground-truth actions exist.
- Latent Actions: The latent-action approach uses 2,271 hours of filtered egocentric human videos from multiple public datasets.
- Latent Actions: A latent action model combines an inverse dynamics model, visual forward dynamics model, and action forward dynamics model to learn a quantized codebook.
- Latent Actions: Robot supervision reconstructs ground-truth action chunks, whereas human-video training omits action reconstruction because ground-truth actions are unavailable.
- Latent Actions: The pipeline concatenates latent segments and quantizes each representation into 8 discrete tokens from a codebook of size C = 32.
- VLM-generated Annotations: The alternative human-video approach uses a VLM to generate per-frame descriptions of motions, goals, and objects from visual context and task instructions.
6) Discrete Robot Action Tokens:
The study examines discrete robot action tokens as co-training modalities alongside broader evaluation settings for robot policies. It compares tokenization approaches and training configurations across simulation, real-world language following, and long-horizon adaptation.
- Discrete action-token modalities: FAST tokens compress continuous action chunks into near-lossless discrete sequences, while VQ-VAE provides an alternative discrete action representation.The study evaluates both token types as co-training modalities.
- Real-world evaluation: Real-world evaluation measures language following with seen and unseen objects, instruction generalization through paraphrasing, and adaptation to unseen long-horizon dexterous tasks.The language-following suite includes seen objects, varied instruction phrasing, and objects absent from target-robot training data.
- Evaluation settings: The research evaluates policies on in-distribution performance, distribution-shift robustness, unseen tasks, and language following, with additional fine-tuning on unseen long-horizon dexterous tasks.Simulation includes 13 seen and 8 unseen tasks under nominal and distribution-shift conditions.
- Unseen-task evaluation: The benchmark’s unseen tasks test semantic understanding, multi-step manipulation, and compositional generalization beyond the training data.Underlying objects and environments may remain familiar even though the skills themselves are absent from training.
3) Statistical Analysis Framework:
The paper uses statistical comparisons to evaluate co-training modalities and phases against a no-co-training baseline. Results favor diverse vision-language and cross-embodiment data, whereas discrete action-token variants provide limited or no benefit.
- Statistical analysis: Pairwise hypothesis tests and Compact Letter Displays identify significant differences at a 5% family-wise error rate, with Bayesian uncertainty shown for individual strategies.The analysis also visualizes empirical and posterior uncertainty over policy performance.
- Training-strategy comparisons: Each modality is tested with single-phase, two-phase first-phase-only, and two-phase full co-training against a continuous-action no-co-training baseline.Effective modalities are subsequently evaluated in real-world language-following experiments.
- Effective modalities: Standard vision-language data improves distribution-shift robustness, unseen-task generalization, and language following, with continued second-phase co-training further helping unseen-task and language-following performance.These gains do not significantly change in-distribution performance.
- Effective modalities: VLM-generated annotations for robot trajectories improve robustness, unseen-task generalization, and language following, but adding them during the second phase provides no additional benefit.VLM-based annotations yield larger improvements than scripted annotations.
- Effective modalities: Cross-embodiment robot data is most effective when confined to first-phase training, particularly for unseen-task generalization and distribution-shift robustness.Continuing it during the second phase yields negligible additional language-following benefit.
- Discrete-token ablations: Discrete action-token co-training yields no statistically significant improvement overall; FAST tokens can reduce unseen-task generalization, while VQ-VAE tokens show no improvement in data- and compute-rich comparisons.Latent-action benefits are limited to low target-robot-data regimes and diminish as fine-tuning data increases.
C. Does combining effective co-training modalities yield cumulative performance gains?
Combining effective co-training modalities produces cumulative gains across simulation and real-world evaluations, while fine-tuning the combined model enables rapid adaptation to unseen dexterous tasks. Explicit CoT conditioning does not improve the simulation benchmark over implicit co-training.
- Combining effective modalities yields consistent cumulative gains across evaluation dimensions.
- 72.6% empirical success on simulation unseen tasks represents a 36.4% improvement over baseline for the Final Model.
- 69.4% empirical average task completion on real-world language following represents a 45.3% improvement over baseline.
- 90.2% average task completion after 200 demonstrations is 22.8% above the FT Baseline and 42.9% above the Single Task policy.The FT Final Model consistently performs fine-grained manipulations that challenge the comparison policies.
- Explicit CoT conditioning does not improve upon implicit two-phase co-training across settings in the simulation benchmark.Performance degradation is discernible when VLM-based annotations and latent actions serve as CoT sources.
- Errors or imprecision in generated CoT can propagate into action prediction and produce less precise manipulation behavior.
IV. RELATED WORK
Related work frames co-training as a response to the severe robot-data scarcity limiting embodied model generalization. This paper systematically studies heterogeneous modalities and finds benefits from vision-language and cross-embodiment data, cumulative gains from combining them, and scope-dependent CoT effects.
- Robot datasets remain orders of magnitude smaller than internet-scale text and image corpora, contributing to limited embodied-model generalization.
- Co-training jointly uses target-robot and heterogeneous data to bridge the robot-data gap.
- Prior work explores public vision-language data, robot-trajectory annotations, cross-embodiment robot data, human videos, and discrete action tokens.
- Discrete action tokens can limit low-level control precision and slow inference, motivating alternative co-training strategies.
- This study reports that vision-language and cross-embodiment co-training improve distribution-shift, unseen-task, and language-following generalization, while discrete action tokens provide no statistically significant benefits.
- The study’s limitations include no task-taxonomy analysis of vision-language data, coarse human-video representations, restricted CoT forms, and an imitation-learning-only scope.
A. Model Architecture Ablation.
The appendix documents architecture, training, data, and hyperparameter choices for the policy ablations. It shows that balancing losses and robot-to-co-training ratios trades off in-distribution performance against generalization benefits.
- The architecture ablation compares the proposed VLM-plus-flow-transformer policy with πFAST-equivalent and π0-equivalent architectures.
- The study treats flow-matching versus cross-entropy weighting and the target-robot-to-co-training ratio as empirically critical hyperparameters.
- Increasing either the loss weight or co-training data ratio degrades in-distribution performance, while setting them too low reduces co-training generalization benefits.
- The selected configuration uses w = 0.02 and a 9:1 robot-to-co-training ratio for unpaired modalities.This configuration preserves in-distribution performance while maximizing generalization improvements.
- Cross-embodiment robot data is mixed during first-phase training with a 6:4 target-robot-to-cross-embodiment batch ratio.
- Each standard multi-task training phase runs for 200k steps and requires approximately 64 hours on 16 H100 GPUs.
- The VQ-VAE compresses horizon-16, dimension-20 action chunks into 8 discrete tokens using a codebook of size 32.
4. POLICY DEPLOYMENT
Deployment uses chunked action prediction on a dual-arm Franka platform, with temporal ensembling for smoother real-world execution. Evaluation compares policies statistically across controlled simulation conditions and rollouts.
- Real-world deployment uses continuous inference and temporal ensembling to address discontinuities between predicted action chunks.Average inference latency is 0.146 seconds.
- Average task-completion comparisons use Welch’s t-test, while success-rate comparisons use the STEP test.
- Pairwise tests control family-wise error at 5% with Bonferroni correction when comparing k strategies.
- Bayesian uncertainty is estimated with uniform Beta priors for success rates and uniform Dirichlet priors for task-progress values.
- The simulation benchmark contains 13 seen and 8 unseen tasks under nominal and distribution-shift conditions.
A. Language Following.
Language-following evaluation measures task completion through four ordered milestones, from reaching and grasping the target to placing it according to the instruction.
- Four milestones assess reaching, grasping, transporting, and correctly positioning the picked object.The final placement criterion depends on the instruction’s spatial relation.
- “In/On” requires the object to rest inside or on top of the destination.
- “Next to” requires positioning the object within 10 cm of the destination.
B. Long-horizon Dexterous Manipulation.
The long-horizon benchmark decomposes three dexterous tasks into ordered milestones covering object manipulation, bimanual coordination, and multi-step placement outcomes.
- Three tasks are evaluated through milestone sequences defined from sample initial conditions.
- Sunglasses: The sunglasses task requires pickup, folding both temples, and placing the folded sunglasses inside a bag.
- Drawstring bag: The drawstring-bag task requires grasping both drawstrings and closing the bag by pulling them.
- Soup preparation: The soup task includes handling the pot, lid, bag, bowls, spatula, and three ingredient transfers into the pot.
C. Experimental Procedure
Real-world evaluation controls environmental variation by testing all policy checkpoints on the same initial condition before changing the scene.
- Each session samples an initial condition specifying the objects and their poses, then manually configures the workspace to match it.
- Every selected policy checkpoint is executed once on that initial condition, with outcomes recorded using the task rubrics.
- Evaluating checkpoints under nearly identical physical conditions makes policy behavior the intended source of performance differences.
D. Rubric QA
The rubric quality-assurance procedure uses blinded secondary review to estimate human evaluation discrepancies in real-world rollouts.
- 895 rollouts, representing 31.6% of all real-world evaluations, were independently reviewed by a separate reviewer pool.Reviewers were not told which checkpoint each video represented.
- 2.01% was the overall task-completion discrepancy across episodes and milestones.