PaperlayerView exact revision on arXiv

Source-linked AI summary

Data Augmentation for Manipulation

Peter Mitrano, Dmitry Berenson

arXiv:2205.02886v4cs.RO

TL;DR

Robotic manipulation often lacks cheap datasets, and existing computer-vision augmentation does not transfer easily to its data and tasks. The paper optimizes physically grounded trajectory augmentations for validity, relevance, and diversity, reporting better downstream performance and increased real-robot online success. The method is not compatible with image data.

  • Problem

    Robotic manipulation often lacks cheap data, while computer-vision augmentation methods do not readily extend to manipulation data and tasks.

    Source: §I · p. 1

  • Method

    The method optimizes rigid-body transformations of robot trajectories using physics-derived objectives for validity, relevance, and diversity.

    Source: §I · p. 2, §III · p. 3, A. Algorithm Overview · p. 5

  • Results

    27% to 50% success in 30 real-robot trials; augmentation also improved downstream performance in simulated planar pushing and rope manipulation.

    Source: §I · p. 2

  • Takeaways & Limitations

    The results support using augmentation for more data-efficient online learning on real-robot manipulation tasks.

    Source: §I · p. 2

    The method is not compatible with image data.

    Source: §VI · pp. 8–9

Abstract

from arXiv · show

The success of deep learning depends heavily on the availability of large datasets, but in robotic manipulation there are many learning problems for which such datasets do not exist. Collecting these datasets is time-consuming and expensive, and therefore learning from small datasets is an important open problem. Within computer vision, a common approach to a lack of data is data augmentation. Data augmentation is the process of creating additional training examples by modifying existing ones. However, because the types of tasks and data differ, the methods used in computer vision cannot be easily adapted to manipulation. Therefore, we propose a data augmentation method for robotic manipulation. We argue that augmentations should be valid, relevant, and diverse. We use these principles to formalize augmentation as an optimization problem, with the objective function derived from physics and knowledge of the manipulation domain. This method applies rigid body transformations to trajectories of geometric state and action data. We test our method in two scenarios: 1) learning the dynamics of planar pushing of rigid cylinders, and 2) learning a constraint checker for rope manipulation. These two scenarios have different data and label types, yet in both scenarios, training on our augmented data significantly improves performance on downstream tasks. We also show how our augmentation method can be used on real-robot data to enable more data-efficient online learning.

I. INTRODUCTION

The paper addresses scarce robotic-manipulation data with an optimization-based augmentation method grounded in physical validity, task relevance, and diversity. It reports improved downstream performance in simulation and higher real-robot success during online learning.

I. INTRODUCTION

  • Robotic data are costly and scarce, while vision augmentations do not transfer straightforwardly; random transforms must work across examples, and generative models performed poorly on small datasets.Source: §I · p. 1
  • The framework formalizes useful augmentation as an optimization problem based on physical validity, task relevance, and diversity.Source: §I · p. 1
  • The method transforms trajectories of object poses and velocities, preserving contacts and gravity, starting near examples, and steering augmentations toward random targets.Source: §I · p. 2
  • 27% to 50% success in 30 real-robot trials; augmentation also improved downstream performance in simulated planar pushing and rope manipulation.Source: §I · p. 2

II. RELATED WORK

  • Generative augmentations [20] handle varied data types but lack physical priors and add small noise; the proposed method yields larger meaningful changes at the cost of generality.Source: §II · p. 2

III. PROBLEM STATEMENT

The section defines stochastic augmentation as generating transformed examples from existing data, with labels assumed unchanged. It frames selecting augmentations as an optimization over validity, relevance, and diversity, then motivates approximations for manipulation tasks.

  • Augmentation is a stochastic function that maps an example to a transformed example, with parameters sampled to produce a set of augmentations; labels are assumed unchanged.Source: §III · p. 3
  • Useful augmentations are physically possible, likely in the target task domain, and diverse, with diversity favored when transformation parameters are uniformly distributed over their range.Source: §III · p. 3
  • Because the validity, relevance, and label information needed for direct optimization is rarely available in manipulation, the paper approximates these criteria using physics-derived measures.Source: §III · p. 4

A. Assumptions

The method relies on known scene geometry, identifiable moving and stationary objects, time-series examples, and contact information. It assumes rigid transformations preserve internal forces and that motion arises from contacts or gravity; performance may suffer when assumptions fail.

  • The scene must have known robot and object geometry, separable moving and stationary objects, time-series examples, and detectable robot contacts.Source: A. Assumptions · p. 4
  • The method assumes rigid transformations preserve internal forces and objects move through contact or gravity; it excludes forces such as magnetism and wind.Source: A. Assumptions · p. 4
  • Moving-versus-stationary contacts share a friction coefficient, but friction between two moving objects is unrestricted; violated assumptions may reduce performance.Source: A. Assumptions · p. 4

IV. METHODS

The method augments manipulation trajectories by transforming moved objects while accounting separately for robot state, actions, and stationary surroundings. It formulates augmentation as an optimization over rigid-body transforms that balances validity, relevance, and diversity.

A. Algorithm Overview

  • The method augments trajectories of moved-object states, robot states, and actions, while leaving stationary objects unchanged.It distinguishes moved objects, robots, and stationary objects as separate scene categories.Source: A. Algorithm Overview · p. 4
  • Rigid-body transforms in SE(2) or SE(3) are applied to moved objects, with robot states and actions recomputed to match.The paper motivates rigid transforms as likely to preserve internal forces, including for articulated or deformable objects.Source: A. Algorithm Overview · p. 4
  • Randomly sampling transforms independently of the data is unlikely to produce valid and relevant augmentations.The method instead approximates the augmentation problem using transform parameters as decision variables.Source: A. Algorithm Overview · p. 4
  • The approximation moves validity into the objective and balances relevance and validity terms against a diversity term that targets uniformly sampled transforms.The objective weights are manually defined.Source: A. Algorithm Overview · p. 5
  • The bounding-box objective keeps augmented states within workspace or scene bounds, encouraging relevance to the task.States outside those bounds are described as unlikely to be relevant.Source: 1) Bounding Box Objective · p. 5

1) Bounding Box Objective:

The section describes objectives that constrain candidate transforms and preserve properties of the original manipulation data. These include task-specific transform validity, occupancy, near-contact structure, and robot contacts.

2) Transformation Validity Objective:

  • The transformation-validity objective penalizes transforms that are consistently invalid or irrelevant for a task or domain.It can be defined manually per task or learned from data; in rope manipulation, sideways-floating rotations are nearly always invalid, while the term has no effect in cluttered pushing.Source: 2) Transformation Validity Objective · p. 5

3) Occupancy Objective:

  • The occupancy objective prevents originally separate objects from penetrating and preserves existing penetrations by matching augmented-point occupancy to the original.It assumes uniform environmental friction so contact or penetration can move between regions.Source: 3) Occupancy Objective · p. 5

4) Delta Minimum Distance Objective:

  • The delta minimum-distance objective preserves near-contact events, which may mark task-relevant states such as proximity to a goal object or obstacle avoidance.It identifies a moved-object point with minimum distance to the environment; the illustration shows transforms alternating between target steps and projections into the valid-state set.Source: 4) Delta Minimum Distance Objective · p. 5, Figure 5 · p. 5

5) Robot Contact Objective:

  • The robot-contact objective requires contacts involving moved objects to persist in the augmented example, supporting validity of robot state and action.The text identifies robot contact points and moved-object contact points as part of this objective.Source: 5) Robot Contact Objective · p. 5

5) Robot Contact Objective:

The solver separates augmentation of object states from augmentation of robot states and actions. It iteratively seeks diverse state transforms while preserving validity and relevance, then uses inverse kinematics to maintain robot contacts.

5) Robot Contact Objective:

  • Additional task-specific objectives can preserve labels, but the experiments did not require them.The stated condition is that such a term enforces ℓ(x) = ℓ(˜x).Source: 5) Robot Contact Objective · p. 5

B. Solving the Augmentation Optimization Problem

  • The procedure splits the optimization into aug_state and aug_robot, with aug_robot computing robot states and actions for the augmented object states and environment.The robot-contact objective is minimized using inverse kinematics to preserve scene contacts.Source: B. Solving the Augmentation… · pp. 5–6
  • aug_state steps iteratively toward a uniformly sampled target transform, alternating with optimization for validity and relevance.The procedure is illustrated as movement toward the target followed by projection into the valid-state set.Source: B. Solving the Augmentation… · p. 5
  • The inner validity-and-relevance optimization is solved with gradient descent until a step limit or gradient threshold is reached.The stopping conditions are Mp steps or a gradient smaller than ϵp.Source: B. Solving the Augmentation… · p. 5
  • Starting aug_state at the identity yields the original relevant example and avoids beginning from a uniformly sampled transform that may be invalid or irrelevant.The text says this makes solving the inner optimization less difficult than starting from a random transform.Source: B. Solving the Augmentation… · pp. 5–6

C. Learning the Valid Transforms Objective

The paper learns a transform-validity function for settings where validity is difficult to specify manually, such as flexible-rope manipulation. It collects tested transforms and trains a small network to predict transformation error, relying on a small set of system examples and low-dimensional transforms.

C. Learning the Valid Transforms Objective

  • For flexible-rope manipulation, the validity of a transform may be hard to define manually because a sideways-floating arc can be invalid for a flexible rope but valid for a stiff cable.The learned function provides a way to represent transformation validity in this setting.Source: C. Learning the Valid Transforms… · p. 6
  • The data-collection procedure tests increasingly large sampled augmentations on the system, producing a dataset of transforms and validity labels for training a small neural network to predict error.The trained predictor is used as the transformation-validity objective.Source: C. Learning the Valid Transforms… · p. 6
  • The approach assumes fewer than 1,000 system examples can be collected and transformations can be tested; its efficiency also relies on low-dimensional transform parameters.The experiments use transformation spaces of dimensions 3 and 6, for which the model is said to generalize with relatively few examples.Source: C. Learning the Valid Transforms… · p. 6

D. Application to Cluttered Planar Pushing

For cluttered planar pushing, the method augments trajectories by transforming moved cylinders and the robot while leaving stationary cylinders unchanged. Inverse kinematics matches the transformed robot state while preserving contacts with moved cylinders.

D. Application to Cluttered Planar Pushing

E. Application to Bimanual Rope Manipulation

For bimanual rope manipulation, the method transforms the rope’s 3D state and preserves its binary validity label. It maintains the robot’s grasps by matching its state and actions to the transformed rope points.

E. Application to Bimanual Rope Manipulation

V. RESULTS

The experiments evaluate whether augmentation improves downstream learning in simulated planar pushing and bimanual rope manipulation. Across both tasks, augmentation improves the reported performance, using small original datasets compared with prior work.

A. Cluttered Planar Pushing

  • The pushing dataset had 60 trajectories (3,000 time steps), compared with over 100,000 time steps in prior work on the same problem.After augmentation, the dataset contained 75,000 time steps.Source: A. Cluttered Planar Pushing · p. 7

V. RESULTS

A. Cluttered Planar Pushing

  • Augmentation significantly outperformed no augmentation, Gaussian noise, and VAE baselines for planar-pushing position error.The reported significance tests give p < 0.0362, p < 0.0001, and p < 0.0002, respectively; the Gaussian-noise and VAE baselines performed worse than no augmentation. Fig. 7 contrasts predictions with and without augmentation and reports reduced drift for objects that should remain stationary.Source: A. Cluttered Planar Pushing · p. 7, Figure 7 · p. 7

B. Bimanual Rope Manipulation

  • Across 100 simulated rope-manipulation iterations, augmentation exceeded both no augmentation and Gaussian noise; final success rates were 70%, 48%, and 31%, respectively.The differences from both baselines were statistically significant (p < 0.001). The VAE baseline was omitted because it performed poorly in planar pushing and was computationally prohibitive to retrain at each iteration.Source: Figure 8 · p. 8, B. Bimanual Rope Manipulation · p. 8
  • The rope experiment gathered an average of 3,038 examples before augmentation, compared with over 100,000 examples in prior work on a similar classifier.The augmented dataset averaged 75,950 examples across the 10 repetitions.Source: B. Bimanual Rope Manipulation · p. 8

C. Real Robot Results

On real robot hardware, augmentation improved rope-placement success after 30 learning iterations. The paper also identifies limitations in image compatibility and in whether its objectives ensure valid, relevant, and diverse augmentations for every application.

C. Real Robot Results

  • After 30 learning iterations, rope placement succeeded 13/26 times with augmentation and 7/26 times without.The evaluation used one start configuration and one goal region; the experiment used noisy sensor data.Source: C. Real Robot Results · p. 8

VI. LIMITATIONS

  • The method is not compatible with image data; image-applicable manipulation augmentation remains an open research area.Source: §VI · pp. 8–9
  • The proposed objectives do not ensure validity, relevance, and diversity for every problem or application.Source: §VI · p. 8

VII. CONCLUSION

  • The paper concludes that augmentation improves downstream task performance on small datasets, including real-robot rope manipulation with noisy sensor data.The conclusion reports a real-world success-rate increase from 27% to 50%.Source: §VII · p. 9

I. APPENDIX

The appendix ablates augmentation objectives, examines how many augmentations to generate for rope constraint checking, and reports optimization hyperparameters.

A. Ablation of Objective Terms

  • Without the occupancy objective, contacts and penetrations are not preserved, and both tasks perform worse than training without augmentation.Each ablation omitted one objective term and was run with 10 random seeds.Source: A. Ablation of Objective Terms · p. 11, Figure 1 · p. 11, Table I · p. 11
  • Omitting delta minimum distance slightly hurts simulated rope performance and moves hook-adjacent examples into free space, but has no notable planar-pushing effect.The authors relate the planar-pushing result partly to clutter, which keeps most transformations small.Source: A. Ablation of Objective Terms · p. 11, Figure 2 · p. 11
  • The valid-transformation objective can be omitted without changing performance in these tasks, despite occasional physically impossible sideways-floating rope augmentations.The authors suggest the objective may still benefit other applications.Source: A. Ablation of Objective Terms · p. 11

B. Choosing the Number of Augmentations

  • The rope classifier's test-example output improved with added augmentations and plateaued around 20–25; the authors chose 25 per example.The prose reports improvement beginning around 10 augmentations and plateauing by 20; the Figure 4 caption says the example plateaus around 25.Source: B. Choosing the Number of Augmentations · p. 12, Figure 4 · p. 12

C. Hyperparameters

  • Optimization used N_p=5 stepping and projection iterations, M_p=25, δ_p=0.001 transform-change tolerance, and ε_p=0.0003 gradient threshold.Objective weights were β1=0.05, β2=1, β3=1, and β4=0.1.Source: C. Hyperparameters · p. 12
Loading 2205.02886v4…