Source-linked AI summary
RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunking
Homanga Bharadhwaj, Jay Vakil, Mohit Sharma, Abhinav Gupta, Shubham Tulsiani, Vikash Kumar
TL;DR
RoboAgent addresses the challenge of building generalizable multi-task manipulation agents from costly, limited robotics data. It combines semantic dataset augmentation, multimodal action representations, and language conditioning; with 7,500 trajectories, it performs 12 skills and outperforms alternatives by 40% in generalization studies.
Problem
Real-world multi-skill robot learning requires diverse data, but collecting such datasets is costly, effortful, and safety-constrained, motivating generalization within a fixed 7,500-trajectory budget.
Method
MT-ACT multiplies data through semantic scene augmentations and uses a multimodal, language-conditioned Transformer policy with action representations for multi-task manipulation.
Results
RoboAgent exhibits 12 manipulation skills and is 40 % more performant than alternatives, with semantic augmentations especially improving L2 and L3 generalization.
Takeaways & Limitations
The framework supports a single language-conditioned policy for diverse manipulation tasks under a relatively small data budget and can be extended through fine-tuning.
Takeaways & Limitations
The tasks consist of individual skills rather than automatically composed long-horizon tasks, and language generalization is not explored.
Abstract
from arXiv · showhide
The grand aim of having a single robot that can manipulate arbitrary objects in diverse settings is at odds with the paucity of robotics datasets. Acquiring and growing such datasets is strenuous due to manual efforts, operational costs, and safety challenges. A path toward such an universal agent would require a structured framework capable of wide generalization but trained within a reasonable data budget. In this paper, we develop an efficient system (RoboAgent) for training universal agents capable of multi-task manipulation skills using (a) semantic augmentations that can rapidly multiply existing datasets and (b) action representations that can extract performant policies with small yet diverse multi-modal datasets without overfitting. In addition, reliable task conditioning and an expressive policy architecture enable our agent to exhibit a diverse repertoire of skills in novel situations specified using language commands. Using merely 7500 demonstrations, we are able to train a single agent capable of 12 unique skills, and demonstrate its generalization over 38 tasks spread across common daily activities in diverse kitchen scenes. On average, RoboAgent outperforms prior methods by over 40% in unseen situations while being more sample efficient and being amenable to capability improvements and extensions through fine-tuning. Videos at https://robopen.github.io/
1 Introduction
RoboAgent targets generalizable multi-task manipulation under a limited data budget, where collecting diverse real-world robot data is costly and safety-constrained. It combines semantic data augmentation, action representations, and language-conditioned policy learning to produce a single agent spanning many skills and tasks.
- Real-world multi-skill robot learning requires diverse experiences, but data collection imposes substantial effort, operational costs, and safety challenges.
- The work restricts training to 7,500 trajectories while emphasizing generalization to novel deployment scenarios.
- MT-ACT combines semantic data multiplication with action representations to improve efficiency in low-data settings.
- RoboAgent is a single language-instructed policy capable of exhibiting 12 manipulation skills from diverse multi-modal trajectory data.
- 40 % more performance than alternatives is reported alongside superior generalization to novel scenarios and fine-tuning-based extensions.
- RoboSet contains teleoperation trajectories covering 12 skills across 38 tasks in diverse kitchen scenes.
2 Related Work
Related work spans self-supervised learning, simulation-to-real transfer, and alternate data sources for robot learning. RoboAgent instead emphasizes efficient use of limited real-world data and semantic augmentation without additional fine-tuning for open-vocabulary segmentation and language grounding.
- Self-supervised learning and large-scale simulation are prominent approaches for scaling robot learning toward general-purpose agents.
- Many alternate-data approaches use pretrained visual representations, but commonly focus on single-task settings or simulated environments.
- RoboSet is presented as a large open-source real-world dataset with 12 skills across 38 tasks and diverse kitchen scenes.
- The approach avoids further fine-tuning of a separate open-vocabulary segmentation and language-grounding module while examining augmentation effects on unseen-scenario generalization.
3 MT-ACT: Multi-Task Action Chunking Transformer
MT-ACT is built around two stages: offline semantic augmentation diversifies a small robot dataset, while policy learning uses a language-conditioned architecture for multi-modal, multi-task behavior. The system combines a diverse RoboSet corpus with CVAE-based action modeling, action chunking, temporal aggregation, and multi-view inputs.
- Semantic Augmentation: Semantic augmentation multiplies existing demonstrations by creating varied objects, textures, and backgrounds without additional human or robot cost.
- Policy Learning: Policy learning trains MT-ACT to recover multiple skills from augmented multi-modal datasets using language conditioning and CVAE-modeled action distributions.
- Dataset: RoboSet(MT-ACT) contains 7,500 human-teleoperated trajectories covering 12 skills across multiple tasks and scenes.
- Dataset: RoboSet includes contact-rich and articulated-object skills, while varying everyday objects and kitchen setups across physical environments.
- Data Augmentation: The augmentation process applies fully automatic, text-prompted inpainting per frame while preserving the robot behavior within each trajectory.
- MT-ACT Architecture: MT-ACT encodes action sequences into latent style embeddings and uses a Transformer conditioned on task language and visual observations to model multi-modal behavior.
- MT-ACT Architecture: Predicting H future actions and temporally aggregating overlapping predictions supports smooth motions, while four camera views improve robustness to clutter and occlusion.
4 Experimental Design
The experiments evaluate MT-ACT and its generalization under diverse real-world kitchen conditions. They use multi-camera robot data, language-specified tasks, and four defined generalization levels.
- The study investigates MT-ACT’s performance, task and environment generalization, augmentation benefits, policy efficiency, and action chunking.
- The RoboPen setup uses a Franka arm, adaptive gripper, three fixed cameras, and a wrist camera providing complementary workspace views.
- The dataset contains 7,500 human-teleoperated trajectories collected across four physical kitchen-like setups with changing objects and backgrounds.
- Tasks combine a language command, interacted object, executed skill, and contextual information such as location.
- Evaluation spans L1 effectiveness, L2 robustness, L3 unseen-task generalization, and L4 strong generalization to a new kitchen.
5 Experiments
Real-world experiments show that MT-ACT’s action chunking, CVAE-based policy, and semantic augmentations improve multi-task performance and generalization. The policy also demonstrates robustness, fine-tuning plasticity, and strong performance relative to baselines.
- Performance: MT-ACT uses action chunking and CVAE modeling of multi-modal sub-trajectories, significantly outperforming the evaluated imitation-learning baselines.
- Generalization and Robustness: Semantic augmentations improve MT-ACT most strongly at harder generalization levels: approximately 30% relative for L1, 100% for L2, and 400% for L3.
- Generalization and Robustness: Semantic augmentations positively affect every reported activity, with larger relative improvements for Making-Tea, Stowing-Bowl, and Heating Soup.
- Generalization and Robustness: In the new-kitchen L4 evaluation, MT-ACT achieves 25% average success while all other baselines achieve 0%.
- Ablation: Removing FiLM conditioning causes a 5-10% performance drop averaged across activities.
- Ablation: A chunk size of 20 performs best; chunk size 10 causes a 0-5% drop, while chunk size 40 causes more than a 20% drop.
- Ablation: More augmentations per frame correlate with larger overall gains, especially for L2 and L3 out-of-domain generalization.
- Ablation: Under scene perturbations and one-to-three blocked cameras, the policy solves approximately 70% of 20 evaluations on average.
6 Discussion and Limitations
The paper presents MT-ACT as a sample-efficient framework for generalizable multi-task robot manipulation. It also identifies limitations in long-horizon skill composition and language generalization.
- Semantic scene augmentations multiply a small dataset, while a language-conditioned policy learns from diverse multi-modal trajectories.
- Action chunking and temporal aggregation adapted from single-task policies provide significant performance boosts in the evaluated multi-task settings.
- All tasks are individual skills; automatic composition for long-horizon tasks remains future work.
- The study does not explore language generalization and uses pretrained language embeddings without modification.
A Dataset details
The MT-ACT dataset comprises multi-modal robot trajectories collected through human teleoperation. Each trajectory includes synchronized visual, robot-state, control, and timing information.
- MT-ACT uses 7,500 human teleoperated demonstrations from the RoboSet dataset.
- The dataset contains RGB and depth frames from four camera views: right, left, top, and wrist.
- Recorded signals include Franka joint positions and velocities, end-effector and gripper states, applied controls, and 40 time-steps.
- Data was collected at 5Hz with an Oculus Quest 2 controller in a kitchen tabletop setup and stored in HDF5 format.
A.1 Dataset Terminology
RoboAgent distinguishes skills by recurring motion patterns and tasks by skill–object instances, while activities are sequences of tasks and policies differ by training scope.
- Skill: A skill is a similar robot motion performed across different object instances, with articulated-object differences creating separate skills.Examples include Pick, Place, Open, and Close; opening a drawer and opening an oven are separate because their joint motions differ.
- Skill: RoboSet contains 12 skills, including Slide-Open, Flap-Close, Pick, Place, Wipe, Plunge, Slide-in, and Slide-out.
- Task: A task is each instantiation of a skill with a particular object class, so Pick Mug and Pick Butter are distinct tasks.
- Activity: An activity is a sequence of tasks, such as making tea, and the work aims for one robot agent that performs all activities.
- Policies: Policies are categorized as single-task, multi-task single-activity, or multi-task universal according to their training task and activity scope.RoboAgent is trained as a multi-task universal policy over all tasks and activities.
A.2 Details on Semantic Augmentations
RoboAgent multiplies demonstrations through automatic semantic scene edits that vary interaction objects and backgrounds without manual mask or mesh specification.
- Overview: Semantic augmentations create varied scenes for novel distractors and interaction objects while multiplying existing data automatically.
- Augmenting interaction objects: Interaction-object augmentation recovers the robot mask and end-effector position from joint angles, then segments, text-inpaints, and tracks the interacted object.The edited object remains consistent across trajectory frames through tracking.
- Dataset context: The full RoboSet includes 9,500 teleoperated, 20,500 kinesthetic, and about 70,000 heuristic bin-setting trajectories.
- Augmenting backgrounds: Background augmentation segments non-overlapping background objects and inpaints the combined masked region to alter the scene.
- Automation: Unlike prior approaches, the augmentations require no manual masks, object meshes, texture templates, or separately trained open-vocabulary detection modules.
B Train and Evaluation Details
Training and evaluation use multi-view tabletop kitchen environments, specified policy hyperparameters, and baseline implementations adapted with comparable language conditioning.
- Robot environment: Evaluation uses four complementary cameras, a Franka Emika Panda with 8-dimensional joint-and-gripper actions, and 5Hz joint-position control.The setup includes a two-finger gripper and wrist camera.
- Robot environment: The evaluation scenes are tabletop kitchen setups containing diverse real objects.
- Training: All policies are trained for 2000 epochs, while the augmented MT-ACT agent requires about 48 hours on one 2080Ti GPU with batch size 8.
- Generalization evaluation: The rollout examples show L2 generalization with distractor and background changes, and L3 additionally changes task objects.Examples replace butter with watermelon or banana while retaining the preceding scene changes.
- Baselines: Baseline RT-1 and BET implementations are adapted from released code and receive language conditioning through a similar FiLM implementation.
- Hyperparameters: MT-ACT hyperparameter searches vary discrete action sizes among 64, 256, and 512, learning rates from 1e−3 to 1e−4, and AdamW weight decay values.
C Additional Results
Additional results compare single-task and multi-task policies for Heat Soup and report performance of the universal policy across all activities.
- Heat Soup comparison: Multi-task single-activity policies outperform single-task policies on most Heat Soup tasks.
- Heat Soup comparison: Single-task policies perform well on about 80% of tasks but only about 20% on the challenging slide-in-bowl task.
- Heat Soup comparison: Multi-task single-activity policies outperform the universal policy on most Heat Soup tasks.The passage attributes this pattern to the universal policy covering a much larger variety of tasks and skills.
- Universal policy: The universal policy performs well on most tasks across all activities, except challenging small-deformable-object tasks such as Pick Tea at 40% and Pick Lid at 50%.