PaperlayerView exact revision on arXiv

Source-linked AI summary

Semantically Controllable Augmentations for Generalizable Robot Learning

Zoey Chen, Zhao Mandi, Homanga Bharadhwaj, Mohit Sharma, Shuran Song, Abhishek Gupta, Vikash Kumar

arXiv:2409.00951v1cs.ROcs.AIcs.CVcs.LG

TL;DR

Robot learning needs diverse data to generalize, but collecting large, varied demonstrations is costly. The paper uses pre-trained generative models to create semantic augmentations, reporting improved generalization with minimal additional human data collection.

  • Problem

    Robot demonstrations are costly to collect, and limited data diversity constrains exposure to unseen real-world scenarios.

    Source: Introduction · p. 1

  • Method

    The framework uses pre-trained generative models to semantically augment robot data, with controllable low-data and automatic large-data regimes.

    Source: Introduction · pp. 1–2, Semantic Data Augmentation · p. 4

  • Results

    Semantically augmented data can lead to widely improved generalization with minimal additional burden on human data collection.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    Generative augmentation provides broader visual experiences than on-robot demonstrations alone while relying on only a small number of demonstrations.

    Source: Introduction · p. 2

    The approach does not augment action labels or reason about physical parameters, and assumes the original action remains valid in augmented scenes.

    Source: Limitations · p. 12

Abstract

from arXiv · show

Generalization to unseen real-world scenarios for robot manipulation requires exposure to diverse datasets during training. However, collecting large real-world datasets is intractable due to high operational costs. For robot learning to generalize despite these challenges, it is essential to leverage sources of data or priors beyond the robot's direct experience. In this work, we posit that image-text generative models, which are pre-trained on large corpora of web-scraped data, can serve as such a data source. These generative models encompass a broad range of real-world scenarios beyond a robot's direct experience and can synthesize novel synthetic experiences that expose robotic agents to additional world priors aiding real-world generalization at no extra cost. In particular, our approach leverages pre-trained generative models as an effective tool for data augmentation. We propose a generative augmentation framework for semantically controllable augmentations and rapidly multiplying robot datasets while inducing rich variations that enable real-world generalization. Based on diverse augmentations of robot data, we show how scalable robot manipulation policies can be trained and deployed both in simulation and in unseen real-world environments such as kitchens and table-tops. By demonstrating the effectiveness of image-text generative models in diverse real-world robotic applications, our generative augmentation framework provides a scalable and efficient path for boosting generalization in robot learning at no extra human cost.

Introduction

Robot demonstrations are costly to collect and often come from few setups, while conventional augmentations miss structured semantic scene changes. The paper uses pre-trained generative models to create semantic variations of robot data and reports improved generalization with little additional human data collection.

Introduction

  • Robot demonstrations require active scene interaction, making large and diverse datasets expensive and difficult to collect.Many robotics datasets therefore involve a single setup with only a few hours of robot data.Source: Introduction · p. 1
  • Conventional augmentations handle minor appearance variation but fall short on structured changes such as distractors, backgrounds, and object appearance.The paper identifies realistic, semantically meaningful scenarios as important for generalization when robotic data is constrained by operational and safety challenges.Source: Introduction · p. 1
  • The proposed framework leverages pre-trained generative models to augment limited on-robot data and expose policies to broader visual experiences.The authors describe this as imposing invariances to semantic variation.Source: Introduction · pp. 1–2
  • Semantically augmented data can lead to widely improved generalization with minimal additional burden on human data collection.The framework cheaply creates a large quantity of augmented data from a small number of demonstrations, providing more diverse scenes than on-robot demonstrations alone.Source: Introduction · p. 2
  • Language prompts and generative models change object textures and shapes and add distractors or backgrounds while aiming to preserve physical consistency with the original scene.The augmented data and corresponding language descriptions are used to train policies evaluated in single-task and multi-task tabletop settings, simulation, and real-world experiments.Source: Introduction · p. 2

Related Work

Prior work expands robot learning with visual data, external datasets, and augmentation, but the paper distinguishes semantic scene variation from local perceptual robustness. It frames learning from demonstrations and positions its framework across low-data and large-scale settings.

Related Work

  • Prior augmentation and geometric methods offer local robustness to perceptual noise but do not provide generalization to novel object shapes or scenes.The paper contrasts its semantic observation changes with approaches that use text-image models for task specification.Source: Related Work · p. 2
  • Related work uses pretrained representations, language, videos, and generated goal images, while many approaches focus on single tasks or simulation.The cited passage also describes robot-generated datasets as limited in the number of distinct environments.Source: Related Work · pp. 2–3
  • The framework uses semantic visual diversity to address generalization beyond the constrained environments of robot-generated data and specific external datasets.The authors present a unified approach spanning low-data single-task learning and large-scale language-conditioned multi-task policies.Source: Related Work · p. 3

Problem Formulation

  • The paper formulates robot manipulation as supervised policy learning from expert demonstrations, with policies mapping observations to action distributions.The stated setup uses maximum-likelihood training on demonstration data.Source: Problem Formulation · p. 3
  • Conventional augmentation functions sample modified observation-action pairs, but their hand-defined variations add no new semantic meaning and may not reflect real-world data.The formulation describes limited demonstration data and augmentation as a way to enlarge the training set.Source: Problem Formulation · pp. 3–4

Leveraging Generative Models for Data Augmentation

The method uses pre-trained text-to-image models to modify observations with semantic scene variation while retaining demonstrated actions. It describes augmentation and policy-learning stages, with different trade-offs between controllability and scale.

Leveraging Generative Models for Data Augmentation

  • Pre-trained text-to-image models modify robot observations toward real-image distributions rather than relying on hand-defined variations that may not match real-world data.The paper formalizes the generator as taking an image, text description, and noise vector to produce a modified image.Source: Leveraging Generative Models for Data… · p. 4
  • The generative models create varied observations but not new actions, so augmented observations are paired with the demonstrated action when semantic equivalence is maintained.The authors identify this as a limitation of the approach.Source: Leveraging Generative Models for Data… · p. 4
  • Generative augmentations create semantically meaningful observations with high likelihood under real-image distributions, supporting generalization across novel scenes and visual changes.The stated changes include objects, distractors, and backgrounds; the paper also notes that generated observations can be physically inaccurate and cannot supply novel actions.Source: Leveraging Generative Models for Data… · p. 4

Framework Overview

Semantic Data Augmentation

The framework offers a controllable, structure-aware augmentation regime for low-data settings and an automatic regime for scaling augmentation to large datasets. Both generate semantic scene variation while retaining task-relevant consistency.

Semantic Data Augmentation

  • The low-data regime offers controllable, structure-aware augmentation, while the large-data regime automates augmentation at scale but gives up some manual control.The low-data approach uses 3D meshes and segmentation masks; the scalable approach can handle thousands of trajectories without manually specified masks or meshes or model training or fine-tuning.Source: Semantic Data Augmentation · p. 4, Scalable Augmentation for Multi-Task… · p. 6

Structure-Aware Augmentation for Low-Data Regime

  • For low-data augmentation, object masks and text prompts change in-category appearance while preserving object position and underlying scene geometry.Prompts vary properties such as color and material; the same object masks are reused across prompts.Source: Structure-Aware Augmentation for… · p. 5
  • Cross-category changes use rendered 3D object meshes and depth-aware diffusion to maintain geometric consistency and physical plausibility.Distractors are rendered from object assets and removed when their image-space bounding boxes overlap masks for the manipulated object or target receptacle.Source: Structure-Aware Augmentation for… · p. 6, Figure 4 · p. 6
  • The structure-aware regime’s reliance on 3D assets and manually labeled masks limits its scalability for larger video-trajectory datasets.The paper introduces fully automatic augmentation for those larger datasets as an alternative.Source: Structure-Aware Augmentation for… · p. 6

Scalable Augmentation for Multi-Task Data

  • The scalable video-trajectory pipeline automatically segments and tracks interaction objects, then inpaints selected background regions across frames.The end-effector location prompts SAM for an interaction-object mask, while TrackAnything maintains mask consistency across frames.Source: Figure 5 · p. 6, Scalable Augmentation for Multi-Task… · p. 6

Policy Learning

The paper uses distinct policy designs for single-task and multi-task learning. The multi-task policy combines language conditioning, a CVAE-based Transformer, and temporally extended action prediction.

Single-Task Policy Learning

  • The single-task policy combines CLIP language features with a Transporter Network to predict pick-and-place locations from language and RGBD observations.The Transporter Network provides spatial correspondences, while CLIP connects visual features with language descriptions.Source: Single-Task Policy Learning · pp. 6–7

Multi-Task Policy Learning

  • The multi-task policy is a Transformer-based CVAE decoder conditioned on latent style embeddings that encode action sequences.The CVAE is used to address multimodal action data, and the policy is designed for multimodal, multi-task datasets.Source: Multi-Task Policy Learning · p. 7
  • The multi-task policy conditions on task language and four camera views, predicts H future actions, and temporally aggregates overlapping predictions during execution.The paper reports that extended action prediction helps reduce compounding error and temporal aggregation produces smoother actions for high-frequency control.Source: Multi-Task Policy Learning · p. 7

System Setup

The evaluation spans single-step pick-and-place and multi-step kitchen tasks across four levels of generalization. The setup compares structure-aware and scalable generative augmentation for expanding robot training data.

System Setup

Task Overview

  • The four evaluation levels progress from unseen object poses and lighting (L1), to new backgrounds and distractors (L2), new tasks and objects (L3), and different rooms (L4).Source: Task Overview · p. 8, Figure 8 · p. 8
  • Single-step pick-and-place tests generalization across cluttered, different environments and objects, while multi-step kitchen tasks test complex skill learning from video trajectories.The task groups target L3–L4 and L1–L4 generalization, respectively.Source: Task Overview · p. 8

Generative Augmentation

  • Structure-aware augmentation uses depth-aware RGBD generation and object meshes to vary scene components, applying 100 augmentations per demonstration to create 1,000 augmented environments per task.The method uses 40 meshes, including 11 for the object of interest and 12 for the receptacle; remaining meshes can serve as distractors.Source: Generative Augmentation · p. 8
  • Scalable augmentation automates image-only augmentation without manually specified masks or object meshes, and therefore uses a transformer-based policy that takes RGB observations.Source: Generative Augmentation · p. 8

Real-world Setup

The real-world experiments use separate pick-and-place and multi-task kitchen setups. The section also defines the augmented and unaugmented conditions and the multi-task comparison baselines.

Real-world Setup

  • Pick-and-place demonstrations use a 6 DoF xArm5, a frontal camera projected into a top-down image and height map, and user-annotated pick-and-place locations.Ten tasks each have 10 demonstrations collected in one environment.Source: Real-world Setup · p. 8
  • The kitchen setup uses a Franka Panda with a two-finger gripper, three static cameras, and a wrist camera for multi-task policy learning.Source: Real-world Setup · p. 8, Figure 10 · p. 8
  • The multi-task video dataset contains 7,500 human-teleoperated trajectories collected over two months across kitchen-like settings and everyday-object manipulation skills.Source: Real-world Setup · p. 9

Baseline Definition

  • ST-AUG and MT-AUG denote augmented training for single-task pick-and-place and multi-step kitchen tasks, respectively; ST-NoAUG and MT-NoAUG are their unaugmented counterparts.Source: Baseline Definition · p. 9
  • Multi-task comparisons include task-specific ACT policies, language-conditioned visual imitation learning, CACTI, RT1-like agents, and language-conditioned BeT.Task-specific agents serve as a rough oracle for task-specific performance.Source: Baseline Definition · p. 9

Experiments

In real-world pick-and-place, generative augmentation improves performance on unseen environments and objects. In multi-task kitchen tasks, gains grow with generalization difficulty, with nonzero performance also reported in a new kitchen.

Experiments

Pick-and-Place (L3 and L4 Generalization)

Real-World Experiments

  • The single-task evaluation trains from demonstrations in one environment and tests in rearranged scenes with unseen environments and objects.The evaluation includes test environments styled as a playground, study desk, kitchen island, garage, and bathroom.Source: Pick-and-Place (L3 and L4… · p. 10

Pick-and-Place (L3 and L4 Generalization)

  • Across evaluations on 5 unseen environments, 5 unseen objects to pick, and 5 unseen objects to place per task, generative augmentation achieved 80% vs 38%, 46% vs 10%, and 54% vs 8% success, respectively.Source: Pick-and-Place (L3 and L4… · p. 10
  • The multi-task kitchen results average activity performance across 4–5 tasks and cover L1–L3, with L2 testing background and distractor variation and L3 testing new skill and object combinations.Source: Multi-Task Kitchen Tasks · pp. 10–11

Multi-Task Kitchen Tasks

  • Semantic augmentations improved average results by about 30% relatively at L1, around 100% at L2, and approximately 400% at L3; in a new kitchen across three tasks, MT-AUG averaged 25% success versus 0% for all other baselines.L2 includes varied table backgrounds and distractors; L3 includes new combinations of skills and objects.Source: Multi-Task Kitchen Tasks · p. 11

Ablations

The ablations examine augmentation quantity, robustness to disturbances, policy fine-tuning, and the trade-off between structural consistency and scalability. Reported findings include stronger generalization with more augmentation and reuse of a trained multi-task agent for a new task.

Ablations

Impact of the number of augmentations

  • Increasing augmentation quantity improves performance, with especially notable gains at L2 and L3; multi-task results also correlate more augmentations per frame with overall improvement.The analyses include varying scene augmentations and per-frame video augmentations.Source: Impact of the number of augmentations · p. 11, Figure 16 · p. 11, Figure 17 · p. 11
  • The universal MT-AUG agent completed the task in approximately 70% of 20 evaluations involving manual scene changes or obstruction of one, two, or three cameras.The authors note that camera-view resilience may also reflect the MT-ACT multi-view transformer.Source: Robustness Analysis · p. 11

Plasticity

  • Fine-tuning the agent trained on 38 tasks with 50 new-task trajectories expanded to 250 augmented trajectories learned the new task without notable decline on the original six activities.The fine-tuned agent showed marginally better L2 and L3 generalization than a single-task policy trained only on augmented data for the new task.Source: Plasticity · pp. 11–12, Figure 18 · p. 12

Trade-off between structure consistency and scalability

Failure Cases

Generative augmentation has prompt- and mask-dependent failure modes, and visual changes cannot cover unseen behaviors. The approach also assumes augmented scenes preserve action validity and focuses on isolated skills.

Failure Cases

Failures in Generative Augmentation

  • Failures often arise when the background resembles the pick or place object, or distractors have bright or similar colors.The authors expect increasing training augmentations to improve scene-combination coverage.Source: Failures in Robot Experiments · p. 12

Failures in Robot Experiments

  • MT-AUG fails when a test-time skill is absent from the teleoperation data because visual augmentation cannot add different behaviors.The authors expect more augmentations to improve scene coverage, but not to supply new actions.Source: Failures in Robot Experiments · p. 12

Limitations

  • The method does not augment action labels or physical parameters, assumes actions remain valid in augmented scenes, and focuses on isolated skills.For cluttered scenes, it assumes the original trajectory will not collide with augmented objects; augmentation takes about 30 seconds per scene and may not suit some on-policy RL approaches.Source: Limitations · p. 12

Conclusion and Future Work

  • The conclusion presents Generative Augmentation as a way to grow small demonstration datasets and generalize to unseen environments and objects.Future work includes faster, physically plausible augmentation, language- and vision-language-model scene generation, and action-level augmentation using generated video and inferred inverse dynamics.Source: Conclusion and Future Work · pp. 12–13

Appendix A: Augmented Dataset in Simulation

Simulation experiments compare generative augmentation with other augmentation approaches and out-of-domain visual priors. Results report improved generalization to held-out layouts as layout variation increases, and notable gains over baselines on pick-and-place tasks.

Appendix A: Augmented Dataset in Simulation

Simulation

  • Simulation baselines include spatial augmentation, copy-paste, random backgrounds, and random distractors; the study also fine-tunes pretrained R3M weights on the dataset.The baselines are evaluated with TransportNet and CLIPort using RGB observations because they cannot update augmented-image depth.Source: Simulation · p. 17

Pick-and-place environment

  • GenAug notably outperforms other approaches on most simulated pick-and-place tasks, while copy-paste and background replacement show robustness in unseen environments but not comparable performance on unseen objects.The authors identify semantically meaningful, physically plausible scene generation as important.Source: Simulation · p. 17
  • The multi-task kitchen simulation uses 18 tasks, eight primary objects, and randomized layouts; augmentation changes layout, texture, and lighting.The study generates 100 layout variations for each task and trains an expert policy for each layout.Source: Simulation · p. 17, Figure 24 · p. 18
  • Evaluation introduces 10 new layouts per task and varies visual attributes; success requires the object to remain within the goal-error threshold for over five time steps.The training setup uses a 43-dimensional context vector encoding target-object pose and layout configuration.Source: Simulation · pp. 17–18
  • Generalization to new layouts improves from Sim-10 to Sim-100 as training includes more layout variation.Table 2 evaluates success rates on familiar training layouts and held-out layouts.Source: Simulation · p. 18, Table 2 · p. 18

Appendix C: More Real-World Experiments

In real-world pick-and-place experiments, GenAug improves zero-shot generalization over CLIPort trained without GenAug on unseen environments and objects.

Appendix C: More Real-World Experiments

  • GenAug improves zero-shot generalization over CLIPort without GenAug across unseen environments and unseen pick and place objects.Success rates were 80% versus 38% on unseen environments, 54% versus 8% on unseen objects to place, and 46% versus 10% on unseen objects to pick.Source: Appendix C · p. 18
  • Evaluation compares predicted pick-and-place affordances with ground-truth locations across five unseen environments, five unseen pick objects, and five unseen place objects per task.The paper visualizes affordance predictions and additional augmented real-world examples.Source: Appendix C · p. 18, Figure 26 · p. 18

Appendix D: Computational Cost

The appendix reports augmentation and training costs and illustrates potential applications beyond manipulation, including locomotion, navigation, and articulated-object tasks.

Appendix D: Computational Cost

  • Augmenting one mask takes about 4 seconds on a 2080Ti GPU, and per-image augmentation takes about 10–30 seconds depending on masks and distractors.The appendix also reports training times of about 48 hours for MT-ACT and one day for CLIPort on a single 2080Ti GPU.Source: Appendix D · pp. 18–19

Appendix E: Potential Applications

  • The framework is presented as potentially applicable to locomotion, indoor navigation, and articulated-object manipulation.Figure 29 illustrates examples of introducing visual variation to these tasks.Source: Appendix E · pp. 19–20, Figure 29 · p. 20
Loading 2409.00951v1…