PaperlayerView exact revision on arXiv

Source-linked AI summary

GenAug: Retargeting behaviors to unseen situations via Generative Augmentation

Zoey Chen, Sho Kiami, Abhishek Gupta, Vikash Kumar

arXiv:2302.06671v2cs.RO

TL;DR

Robot learning needs diverse demonstrations for generalization, but collecting such real-world data is expensive and constrained. GenAug uses pretrained image-text generative models for semantically meaningful augmentation of tabletop manipulation demonstrations, achieving improved generalization to unseen scenes and objects with few demonstrations.

  • Problem

    Real-world robot demonstrations are expensive to collect and often lack the diversity needed for generalization across novel environments and objects.

  • Method

    GenAug uses pretrained text-to-image generative models to augment RGBD demonstrations with realistic, semantically equivalent objects, distractors, and backgrounds while retaining actions.

  • Results

    GenAug improves generalization to unseen real-world scenes and objects, including 85% success on unseen environments and 45% and 52% on unseen objects to pick and place.

  • Takeaways & Limitations

    A small set of demonstrations can be bootstrapped into a larger, visually diverse dataset for tabletop policies that generalize to unseen environments and objects.

  • Takeaways & Limitations

    GenAug assumes the original action remains valid because it does not model physics parameters or augment action labels, and it assumes augmented trajectories avoid collisions.

Abstract

from arXiv · show

Robot learning methods have the potential for widespread generalization across tasks, environments, and objects. However, these methods require large diverse datasets that are expensive to collect in real-world robotics settings. For robot learning to generalize, we must be able to leverage sources of data or priors beyond the robot's own experience. In this work, we posit that image-text generative models, which are pre-trained on large corpora of web-scraped data, can serve as such a data source. We show that despite these generative models being trained on largely non-robotics data, they can serve as effective ways to impart priors into the process of robot learning in a way that enables widespread generalization. In particular, we show how pre-trained generative models can serve as effective tools for semantically meaningful data augmentation. By leveraging these pre-trained models for generating appropriate "semantic" data augmentations, we propose a system GenAug that is able to significantly improve policy generalization. We apply GenAug to tabletop manipulation tasks, showing the ability to re-target behavior to novel scenarios, while only requiring marginal amounts of real-world data. We demonstrate the efficacy of this system on a number of object manipulation problems in the real world, showing a 40% improvement in generalization to novel scenes and objects.

I. INTRODUCTION

GenAug addresses the scarcity and limited diversity of real-world robot demonstrations by using pretrained generative models for semantically meaningful augmentation. It applies this framework to tabletop imitation learning, improving generalization from only 10 demonstrations in one simple environment.

  • I. INTRODUCTION: Real-world robot demonstrations are expensive and difficult to collect diversely, while conventional augmentation mainly handles low-level visual changes rather than semantic differences.The motivating semantic differences include distractors, background changes, and object-appearance changes.
  • I. INTRODUCTION: GenAug uses pretrained text-to-image generative models to create semantically augmented RGBD observations for real-world robot imitation learning.The framework changes object textures and shapes, adds distractors, and varies backgrounds while preserving physical consistency with the original scene.
  • I. INTRODUCTION: Training on semantically augmented data significantly improves imitation-learning generalization to entirely unseen real-world environments using only 10 demonstrations collected in one simple environment.This result targets the broader generalization potential of learning methods beyond highly controlled settings.
  • I. INTRODUCTION: GenAug replaces the limited imitation-learning dataset with a larger augmented dataset while building on CLIPort for tabletop manipulation.The framework is instantiated for tabletop tasks and includes ablations and visualizations to examine design choices.

B. Leveraging Generative Models for Data Augmentation

GenAug uses pretrained generative models to produce realistic, semantically equivalent observation-action pairs rather than relying only on hand-specified visual disturbances. The approach improves realism and diversity but cannot create novel actions and may invalidate actions through physical inconsistencies or collisions.

  • B. Leveraging Generative Models for Data Augmentation: Pretrained generative models can modify training observations toward realistic scene distributions, addressing uncertainty about whether heuristic augmentations match evaluation environments.GenAug uses text, images, and noise to generate modified observations with models such as text-to-image inpainting systems.
  • B. Leveraging Generative Models for Data Augmentation: GenAug generates multiple semantically equivalent observations for each demonstration while retaining the original action label across the augmented set.The intended invariance is that all generated observations correspond to the same ground-truth action as the original observation.
  • B. Leveraging Generative Models for Data Augmentation: Unlike hand-specified augmentations, GenAug targets realistic observational disturbances and produces a large dataset intended to improve robustness across realistic scenes.The method treats generative models as a source of semantic data augmentation rather than merely applying color, crop, or noise transformations.
  • B. Leveraging Generative Models for Data Augmentation: GenAug does not generate novel actions, and careless augmentation can invalidate the original action through physical inconsistencies or collisions.The limitation follows because the same action label is reused for generated observations.

C. Instantiating GenAug for Tabletop Robotic Manipulation

GenAug instantiates semantic augmentation for top-down tabletop pick-and-place by controlling masked object regions and using depth-guided generation. It varies object appearance and category while preserving interaction-relevant geometry and semantic invariance.

  • C. Instantiating GenAug for Tabletop Robotic Manipulation: GenAug targets tabletop rearrangement with a robot arm, using top-down visual observations and spatial pick-and-place action maps for a suction gripper.The setup builds on transporter-network visual imitation learning.
  • C. Instantiating GenAug for Tabletop Robotic Manipulation: The framework augments the grasped object or target receptacle, distractors, and background to maximize visual diversity while preserving the task-relevant action.These augmentation targets are defined for tabletop pick-and-place tasks.
  • C. Instantiating GenAug for Tabletop Robotic Manipulation: GenAug uses object and receptacle masks with depth-guided text-to-image generation to produce controlled, semantically invariant scene variations.Masks are required only for the small number of collected demonstrations, not at inference time.
  • C. Instantiating GenAug for Tabletop Robotic Manipulation: Cross-category generation replaces an object using rendered meshes and depth-aware generation to improve physical plausibility and 3-D consistency.For top-down suction grasping, the interaction point remains largely unchanged even when the object category changes.

2) Generating Distractors with Diverse Visual Appearances:

GenAug extends semantic augmentation to clutter and backgrounds by adding physically plausible distractors and varying scene styles while preserving relevant object positions. It then trains the manipulation policy with standard maximum-likelihood learning on the combined augmented data.

  • 2) Generating Distractors with Diverse Visual Appearances:: GenAug adds diverse distractors by rendering object meshes on the table, checking collisions, and applying text-to-image generation for realistic visual appearance.The resulting scenes preserve semantic invariance while becoming more cluttered and visually diverse.
  • 2) Generating Distractors with Diverse Visual Appearances:: Background augmentation holds object, receptacle, and distractor masks fixed while generating varied kitchen, living-room, or restaurant scenes.Because the masks remain fixed, object positions stay invariant while the background and table appearance change.
  • 2) Generating Distractors with Diverse Visual Appearances:: GenAug combines visual object generation, distractor generation, and background generation into visually diverse data with semantic invariance.The augmented data is intended to overlap with environments encountered in the real world.
  • 2) Generating Distractors with Diverse Visual Appearances:: The augmented dataset is used with standard maximum-likelihood training for tabletop manipulation, based on RGBD observations, masks, and a calibrated camera pose.The real-world implementation uses manually labeled demonstration masks and a RealSense RGBD camera.

B. Demonstration Collection

Demonstrations are collected in one tabletop environment through user-labeled pick-and-place actions, then used to construct varied test environments for generalization evaluation.

  • B. Demonstration Collection: Users click pick and place locations on top-down scene images, which are converted into Cartesian end-effector actions.The robot saves demonstrations only after successfully completing the task.
  • B. Demonstration Collection: The study collects 10 demonstrations for each of 10 tasks in a single demonstration environment.
  • B. Demonstration Collection: Evaluation uses five test-environment styles with randomly rearranged and added objects to create unseen scenes.The styles include Playground, Study Desk, Kitchen Island, Garage, and Bathroom.

2) Result:

GenAug substantially improves zero-shot generalization in real-world tabletop manipulation, especially to unseen environments and objects, and outperforms alternative augmentation and representation baselines in simulation.

  • 2) Result:: The real-world experiments demonstrate generalization across unseen environments, pick objects, and place objects.
  • 2) Result:: 80% versus 38% success on unseen environments, 54% versus 8% on unseen objects to place, and 46% versus 10% on unseen objects to pick favor GenAug over CLIPort without augmentation.These averages compare methods across 10 real-world tasks using unseen environments and objects.
  • 2) Result:: GenAug reaches 85% success on unseen environments, 52% on unseen objects to place, and 45% on unseen objects to pick.The evaluation uses augmented RGBD images and text prompts from real-world task demonstrations.
  • 2) Result:: In simulation, GenAug significantly outperforms competing approaches on most tasks and is particularly stronger on unseen objects than copy-paste or background replacement.The comparison uses TransportNet and CLIPort with several augmentation and out-of-domain representation baselines.

C. Ablations

Ablations show that increasing the number of GenAug augmentations improves generalization across unseen environments and objects in simulation.

  • 1) Impact of the number of augmentations:: Performance improves as GenAug augmentations increase from 0 to 10, 50, and 100 samples across three unseen-scene categories.The experiment evaluates 100 scenes each for unseen environments, unseen pick objects, and unseen place objects.
  • 1) Impact of the number of augmentations:: The augmentation-count trend supports using more generated examples to robustify generalization capability.

2) Failure cases:

GenAug’s scope is constrained by visual and physical assumptions: it can fail in cluttered or low-contrast scenes, does not model action-label or physics changes, and is costly for some video-based methods.

  • 2) Failure cases:: Failures usually occur when the object and background have similar colors or when the table contains too many objects.
  • 2) Failure cases:: GenAug assumes unchanged action labels remain valid despite changes in material, friction, deformation, and added-object collisions.The method does not augment action labels or reason about these physics parameters.
  • 2) Failure cases:: GenAug cannot guarantee visual consistency across video-frame augmentation and typically takes about 30 seconds to augment one scene.This may be impractical for some on-policy reinforcement-learning approaches.
  • 2) Failure cases:: The demonstrated system is scoped to tabletop manipulation with a robot arm, while future work targets other learning settings and harder manipulation tasks.

APPENDIX A REAL-WORLD EXPERIMENTS

The real-world experiments evaluate GenAug on tabletop manipulation using sparse demonstrations, compare affordance predictions, and examine augmentation choices and failure cases.

  • Real-world experimental setup: 10 tasks used 10 demonstrations each in one environment, with GenAug producing 1,000 augmented demonstrations per task.Each demonstration was augmented 100 times.
  • Generalization evaluation: GenAug significantly improves affordance generalization over training without GenAug on unseen environments and objects.The comparison evaluates predicted pick and place affordances against ground-truth locations.
  • Failure cases: Failures usually occur when background and object colors are similar or distractors are unusually bright or similarly colored.The authors expect broader augmentation coverage to improve performance on such scene combinations.
  • Augmentation model choice: Depth-guided diffusion with predefined 3D meshes produces realistic new objects and scenes, whereas direct inpainting often does not.This comparison motivates the augmentation model used by GenAug.

E. Real-World Unseen Environments

The real-world evaluation visualizes training and unseen environments, generated datasets, and affordance predictions, while simulation evaluates performance across tasks and demonstration counts.

  • Real-world unseen environments: The real-world experiments evaluate CLIPort on unseen environments and objects across 10 tabletop manipulation tasks.The paper visualizes both the unseen test scenes and the resulting pick-and-place affordance predictions.
  • Simulation evaluation: Simulation evaluation covers five packing tasks with 1, 10, and 100 demonstrations and reports average success rate across tasks.Success rate is defined by the fraction of the pick object’s volume inside the place object.
  • Augmented simulation data: GenAug is applied 100 times per simulation demonstration, with augmented examples visualized in Figure 16.The supplied passages identify the augmentation count and visualization, but not comparative success-rate values.
  • Generated appearance diversity: Different text prompts generate diverse, realistic textures for the same object template.This demonstrates appearance diversity in the generated dataset.

APPENDIX C VISUALIZATION OF BASELINE DATA AUGMENTATION

The appendix visualizes GenAug outputs alongside a random copy-and-paste baseline and affordance predictions on varied tasks and unseen settings.

  • Baseline augmentation: Randomly copying and pasting LVIS segmented images often produces unrealistic, low-quality generations that do not match test-time observations.The baseline directly pastes queried segmented images onto original demonstration images.
  • Unseen test settings: The real-world evaluation uses unseen environments and objects for 10 tasks and includes corresponding affordance-prediction visualizations.The supplied passages reference the evaluation scenes and figures but do not provide numerical outcomes.
  • GenAug visualizations: The appendix shows augmented demonstrations collected in simple environments and visualizes GenAug affordance predictions across various tasks.These visualizations include predictions from CLIPort trained with GenAug.
Loading 2302.06671v2…