Source-linked AI summary
JoyAI-RA 0.1: A Foundation Model for Robotic Autonomy
Tianle Zhang, Zhihao Yuan, Dafeng Chi, Peidong Liu, Dongwei Li, Kejun Hu, Likui Zhang, Junnan Nie, Ziming Wei, Zengjue Chen, Yili Tang, Jiayi Li, Zhiyuan Xiang, Mingyang Li, Tianci Luo, Hanwen Wan, Ao Li, Linbo Zhai, Zhihao Zhan, Xiaodong Bai, Jiakun Cai, Peng Cao, Kangliang Chen, Siang Chen, Yixiang Dai, Shuai Di, Yicheng Gong, Chenguang Gui, Yucheng Guo, Peng Hao, Qingrong He, Haoyang Huang, Kunrui Huang, Zhixuan Huang, Shibo Jin, Yixiang Jin, Anson Li, Dongjiang Li, Jiawei Li, Ruodai Li, Yihang Li, Yuzhen Li, Jiaming Liang, Fangsheng Liu, Jing Long, Mingxi Luo, Xing Pan, Hui Shen, Xiaomeng Tian, Daming Wang, Song Wang, Junwu Xiong, Hang Xu, Wanting Xu, Zhengcheng Yu, He Zhang, Jiyao Zhang, Lin Zhao, Chen Zhou, Nan Duan, Yuzheng Zhuang, Liang Lin
TL;DR
Open-world robotic autonomy is limited by insufficient data diversity and cross-embodiment transfer challenges. JoyAI-RA addresses these gaps with heterogeneous multi-source pretraining and explicit action-space unification, and reports strong results across simulation and real-world benchmarks, including a 63.2% average success rate on RoboCasa GR1 Tabletop.
Problem
Open-world robot learning lacks sufficient task and scene diversity, while differing embodiments impede effective behavior knowledge transfer.
Method
JoyAI-RA combines web data, egocentric human videos, simulation trajectories, and real-robot demonstrations in multi-level pretraining with a unified action space.
Results
JoyAI-RA outperforms prior methods across simulation and real-world benchmarks, achieving a 63.2% average success rate on RoboCasa GR1 Tabletop.
Takeaways & Limitations
Structured heterogeneous pretraining with action-space unification supports cross-embodiment behavior transfer and generalizable robotic manipulation.
Takeaways & Limitations
Precise visual reasoning and graded sequential manipulation remain challenging, with Food Scraps difficult for all methods in absolute terms.
Abstract
from arXiv · showhide
Robotic autonomy in open-world environments is fundamentally limited by insufficient data diversity and poor cross-embodiment generalization. Existing robotic datasets are often limited in scale and task coverage, while relatively large differences across robot embodiments impede effective behavior knowledge transfer. To address these challenges, we propose JoyAI-RA, a vision-language-action (VLA) embodied foundation model tailored for generalizable robotic manipulation. JoyAI-RA presents a multi-source multi-level pretraining framework that integrates web data, large-scale egocentric human manipulation videos, simulation-generated trajectories, and real-robot data. Through training on heterogeneous multi-source data with explicit action-space unification, JoyAI-RA effectively bridges embodiment gaps, particularly between human manipulation and robotic control, thereby enhancing cross-embodiment behavior learning. JoyAI-RA outperforms state-of-the-art methods in both simulation and real-world benchmarks, especially on diverse tasks with generalization demands.
1 Introduction
JoyAI-RA addresses limited task coverage and cross-embodiment transfer in open-world robotic autonomy through heterogeneous pretraining and explicit action-space unification. It reports strong performance across simulation and real-world benchmarks.
- Open-world robotic autonomy is constrained by insufficient data diversity and embodiment gaps that hinder multi-source behavior sharing.
- Robot datasets often underrepresent long-tail interactions, rare failures, and diverse scenes because high-quality interaction data is expensive to collect.
- JoyAI-RA is a VLA foundation model for generalizable robotic manipulation in open-world settings.
- Its multi-source pretraining combines web data, egocentric human videos, simulation trajectories, and real-robot demonstrations, while action-space unification supports knowledge transfer across embodiments.
- The paper reports strong downstream performance in both simulated and real-world evaluations, attributing gains to heterogeneous pretraining and action-space unification.
2 Related Work
Related work increasingly combines heterogeneous data and unified vision-language-action architectures to improve semantic coverage, behavioral diversity, and embodiment-grounded supervision. The supplied passages position JoyAI-RA within this broader movement without detailing a separate comparison.
- Embodied foundation models increasingly combine heterogeneous sources to expand semantic coverage, behavioral diversity, and embodiment-grounded supervision.
- Real-world robot demonstrations remain important because they expose policies to sensing noise, contact uncertainty, and hardware constraints.
- VLA systems unify perception, language understanding, and action generation for universal manipulation policy learning.
- The overview frames JoyAI-RA as combining web, human, simulation, and real-robot data through multi-level pretraining and action-space unification.
3 Approach
JoyAI-RA organizes web, human, simulation, and real-robot data within a multi-level training framework. A unified action representation reconciles heterogeneous embodiments and supports transfer from semantic pretraining to executable robot control.
- The training pipeline integrates multimodal inputs and diverse data sources to predict temporally consistent action sequences for simulation and real-world control.
- A unified action space projects states and actions into a fixed-dimensional shared representation, with masking accommodating heterogeneous robot morphologies.
- The framework combines complementary supervision from semantic web data, human manipulation videos, simulation, and real-robot trajectories.
- Web data supplies perceptual and linguistic priors but does not provide executable manipulation trajectories.
- EgoLive spans 1,969 object categories, 1,796 action categories, and 3,779 household, 3,686 retail, and 2,518 logistics tasks.
- Fine-grained per-frame annotations and cross-embodiment action alignment provide supervision for temporally grounded and transferable manipulation policies.
- Simulation bridges semantic pretraining and embodiment-grounded robot action learning, while real-robot data captures physical deployment conditions.
- Camera-frame end-effector representations align actions with visual observations using decomposed 6-DoF pose components.
3.4 Model Architecture
JoyAI-RA separates vision-language understanding from perception-action control while jointly modeling textual outputs and continuous action chunks. Its flow-matching expert predicts temporally consistent actions from multimodal context and noisy action latents.
- The architecture uses a VLM for spatially grounded semantic and geometric representations and a Perceiver-based perception-action expert for action generation.
- JoyAI-RA predicts both a high-level textual subtask description and a sequence of low-level actions from observations and language instructions.
- Flow matching trains the perception-action expert to estimate velocities of noisy action latents for stable continuous control.
- The latent sequence concatenates proprioceptive state, learnable future tokens, and timestep-conditioned noisy action features.
- State and action encoders process the state and noisy action trajectory, while the expert predicts an action-chunk velocity field conditioned on visual-language context and timestep.
- Time-aware Perceiver blocks and timestep-adaptive normalization condition predictions across denoising stages, supporting temporal consistency and stable action generation.
3.5 Training Recipe
JoyAI-RA trains progressively from broad visual-language and embodied priors to continuous manipulation behavior, using heterogeneous action data in a unified action space. Target-robot post-training then specializes the pretrained policy for downstream deployment.
- Training stages: Training proceeds through VLM co-pretraining, VLA co-pretraining, and post-training, progressively specializing the model while preserving cross-source transfer.The stages move from broad vision-language and embodied priors to continuous action learning and target-embodiment specialization.
- VLM Co-Pretraining: VLM co-pretraining combines General VQA, Embodied VQA, cross-embodiment action data, and human videos to build visual, spatial, planning, and trajectory-generation capabilities.Human videos contribute broader visual inputs and richer action distributions, while embodied data strengthens spatial reasoning and long-horizon planning.
- VLA Co-Pretraining: VLA co-pretraining progressively introduces General VQA and Embodied VQA while supervising actions from simulated trajectories, real-robot demonstrations, and retargeted human videos.All action sources are represented in the unified action space so heterogeneous motions remain aligned.
- VLA Co-Pretraining: The VLA objective adds a flow-matching loss to the autoregressive loss, using perturbed action chunks and predicting the flow from sampled noise toward the action sequence.The action-flow term is balanced against the standard vision-language modeling loss by a loss multiplier.
- Post-Training on Target Robots: Post-training uses only target-robot data and optimizes the flow-matching objective, allowing the policy to adapt jointly to target tasks and the deployment embodiment.The auxiliary autoregressive objective is discarded, and end-to-end updates support both real-world and corresponding simulation evaluations.
4 Experiments
JoyAI-RA is evaluated across simulation and real-world manipulation benchmarks, where heterogeneous human, robot, and other data sources support broad task generalization. Ablations further show that EgoLive data and two-stage co-pretraining improve performance, while gains vary by task and data-domain alignment.
- RoboTwin 2.0: JoyAI-RA achieves 90.48% and 89.28% average success rates on RoboTwin 2.0 Easy and Hard settings, respectively.It reaches 100% on Adjust Bottle, Grab Roller, and Place Empty Cup.
- RoboCasa GR1 Tabletop: JoyAI-RA reaches a 63.2% average success rate on RoboCasa GR1 Tabletop, exceeding prior methods on long-horizon tasks.Gains include +16.0 on CanToDrawerClose, +24.0 on MilkToMicrowaveClose, and +18.0 on TrayToPot.
- Real-World AgiBot Benchmark: On the real-world AgiBot benchmark, JoyAI-RA improves the cross-task average success rate from 0.62 with π0.5 to 0.74.The benchmark averages six household tasks.
- Real-World AgiBot Benchmark: JoyAI-RA performs best relative to π0.5 on Headphones and Remedy, while π0.5 remains higher on Cup and Croissant.Food Scraps remains difficult for all methods in absolute terms.
- EgoLive Data: Full EgoLive and JDAgibot training reaches an 87.4% average success rate on RoboTwin 2.0, outperforming training from scratch and robot-only pretraining.Using the full EgoLive dataset yields an approximately 6% improvement over its 10% subset.
- In-Domain Data: In-domain human videos improve average success and win on 4 of 6 tasks, but distributional mismatch reverses the benefit on Mouse and Food Scraps.The largest improvements occur on Remedy and Headphones.
- Dataset Analysis: EgoLive exhibits broader vocabulary coverage and a larger, more continuous semantic-space region than comparison datasets, while combining datasets reaches 89.30 performance.The analysis links temporal structure and semantic diversity with generalizable manipulation policies.
5 Conclusions
JoyAI-RA is a vision-language-action embodied foundation model for generalizable robotic manipulation. Its multi-source, multi-level pretraining and unified action space support manipulation-knowledge transfer across diverse embodiments and data sources.
- JoyAI-RA is a vision-language-action embodied foundation model for generalizable robotic manipulation.
- The framework jointly leverages web data, egocentric human videos, simulation trajectories, and real-robot demonstrations.
- Action-space unification bridges embodiment gaps and enables transfer of manipulation knowledge across heterogeneous data sources.
- Comprehensive simulation and real-world testing, together with ablation studies, validate the approach.
A Contributions •
This passage set lists contributors to the paper in core and alphabetical groups.
- The paper identifies a core contributor group led by Tianle Zhang and Zhihao Yuan.
- The core contributor list includes researchers spanning the JoyAI-RA project.
- Additional contributors are listed alphabetically by last name.
B Real-World Experiment Details
The real-world experiment details define household manipulation tasks across office, tea-table, kitchen, dining-table, and pharmacy settings. Tasks require sequential, often bimanual actions such as picking, placing, pouring, passing, and packaging.
- Office: The office benchmark includes hanging headphones and picking a mouse from a desk onto a mouse pad.
- Tea Table: Cup discarding requires picking up a paper cup, pouring out its water, and placing it in a trash can.
- Kitchen: Croissant toasting requires opening the oven, pulling out a tray, placing the croissant on it, pushing the tray back, and closing the door.
- Dining Table: Food-scraps cleaning requires emptying a plate, passing it between arms, and stacking it on another plate.
- Pharmacy: Remedy packaging requires placing a package bag, selecting a designated drug, and putting it into the bag.
C Experiment Results
The experiment-results materials comprise full evaluation tables for RoboTwin 2.0, RoboCasa GR1 Tabletop tasks, and an egocentric-human-data ablation.
- The full versions of Tables 1, 2, and 3 are shown as Tables A, B, and C.
- Table A contains full evaluation results on the RoboTwin 2.0 benchmark.
- Table B reports evaluation results on RoboCasa GR1 Tabletop tasks.
- Table C presents an egocentric human-data ablation on the RoboTwin 2.0 benchmark.