PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving Imitation Learning Efficiency for Manipulation through Geometric Prior Pretraining

Shogo Iwakata, Tomohiro Motoda, Ryosuke Yamada, Koshi Makihara, Ryoichi Nakajo, Keitaro Tanaka, Masaki Murooka, Roman Mykhailyshyn, Hirokatsu Kataoka, Shigeo Morishima, Yukiyasu Domae

arXiv:2609.12721v1cs.RO

TL;DR

Collecting new demonstrations for each manipulation task makes imitation learning costly, while large-scale pretraining and task-specific augmentation have resource or task-dependence tradeoffs. The paper pretrains ACT on automatically generated scenes containing only a plane, object, and hand; across many robot-task combinations, this geometric prior improves early-stage learning over training from scratch.

  • Problem

    Adapting imitation policies to new tasks requires costly task-specific demonstrations, while large-scale pretraining demands substantial resources and augmentation often remains task-specific.

    Source: §I · p. 1

  • Method

    The method pretrains ACT on automatically generated geometric scenes and hand-object motions, then fine-tunes on task-specific data.

    Source: §I · p. 2

  • Results

    Across many robot-task combinations, pretraining improved early-stage learning, with mean success-rate gains over scratch of 10.3% for UR, 8.5% for xArm, and 8.3% for Kinova.

    Source: 2) Results and Discussion · pp. 5–6

  • Takeaways & Limitations

    Simple geometric synthetic scenes can serve as an inexpensive pretraining signal that improves early learning across many robot-task combinations.

    Source: §I · p. 2

    All experiments use ACT, so whether the benefit transfers to other visuomotor policies remains an open question.

    Source: §V · p. 8

Abstract

from arXiv · show

Applying an imitation learning policy to a new manipulation task usually requires collecting new demonstrations and retraining the model, which makes sample efficiency a practical concern. Pretraining on large-scale robot datasets is effective in this respect, but such datasets are costly to collect and train on, while data augmentation techniques typically require a new round of data generation and retraining for each task. A complementary question is what useful prior can be provided to a policy at negligible cost before any task-specific data are collected. In this study, we construct a geometric visual pretraining dataset in which each scene contains only a plane, an object, and a hand, and trajectories are generated automatically. The scenes contain neither textures nor backgrounds; pretraining primarily exposes the policy to the geometric relationship between the hand and the object. Furthermore, representing the hand as a cube avoids tailoring the dataset to a specific robot morphology. We evaluate this geometric prior using ACT on three simulated robots across five manipulation tasks each, as well as on three real-world robot tasks. Across many of these robot--task combinations, fine-tuning from the geometric prior achieves higher success rates in the early stages of training than training from scratch while using only a small number of task demonstrations. These results suggest that even highly simplified geometric scenes can provide a useful initialization that transfers across robots and to real-world tasks when task data are limited.

I. INTRODUCTION

Task-specific demonstrations make adapting imitation policies costly, while common pretraining and augmentation approaches carry their own resource or task-dependence burdens. The paper proposes a task-agnostic geometric pretraining signal built from simple scenes and automatically generated hand-object motions.

I. INTRODUCTION

  • Task-specific demonstrations remain a major bottleneck when adapting imitation policies, often making real-world data collection prohibitively expensive and time-consuming.The stated goal is accurate models trained efficiently with little real data.Source: §I · p. 1
  • Large-scale pretraining requires substantial computation, while augmentation typically enriches task-specific distributions and may require new generation and retraining for each task.The paper motivates pretraining signals independent of downstream tasks and obtainable without additional robot data.Source: §I · pp. 1–2
  • The proposed geometric prior uses a scene containing only an object, a hand, and a plane to pretrain policies on their relative geometry.The hand is represented as a cube rather than a robot-specific morphology.Source: §I · p. 2
  • Automatically generated trajectories expose hand-object relationships while excluding backgrounds and lighting from the pretraining signal.The paper evaluates the approach in simulation and the real world, reporting improved early-stage learning across many robot-task combinations.Source: §I · p. 2

II. RELATED WORKS

The related work covers large-scale robot datasets and task-specific data augmentation. The paper positions geometric prior pretraining as a lower-cost, task-agnostic alternative to those approaches.

A. Large-scale Datasets and Generalist Robot Policies

  • Large-scale robot datasets and generalist policies support multi-task, multi-embodiment learning, but typically require extensive demonstrations and substantial computation.The paper investigates geometric pretraining as a lower-cost inductive bias without relying on massive real-world datasets.Source: A. Large-scale Datasets and Generalist… · p. 2

B. Data Augmentation for Robotic Manipulation

  • Data-augmentation methods generate demonstrations through perturbations, recombination, interventions, image or video generation, and task-constrained planning.Examples include Domain Randomization, MimicGen, IntervenGen, DemoGen [18], DreamGen [19], and CP-Gen [20].Source: B. Data Augmentation for Robotic… · p. 2
  • Unlike augmentation tied to task demonstrations, interventions, or manually defined structures, the proposed framework offers task-agnostic pretraining without task-specific demonstrations.It aims to improve learning efficiency across downstream tasks through a single pretraining stage.Source: B. Data Augmentation for Robotic… · p. 2

III. CREATION OF GEOMETRIC PRIOR DATA FOR PRETRAINING

The pretraining data are generated automatically in simple, geometry-only scenes. Each demonstration records pose and visual observations over time, with the hand represented in a robot-agnostic way.

III. CREATION OF GEOMETRIC PRIOR DATA FOR PRETRAINING

  • The framework automatically generates geometry-only pretraining data without physics, reproducing pick, place, and push actions in scenes with a plane, object, and hand.The recorded observations and hand trajectories are generated separately for each motion.Source: §III · p. 2

A. Scene Construction

  • Scenes use point clouds and a floor plane, with object variation from geometric primitives, randomized sizes, and shape modifications.Objects include prisms, pyramids, cylinders, cones, spheres, and truncated variants; the floor dimensions and grid resolution also vary slightly.Source: A. Scene Construction · pp. 2–3

B. Automatic Generation of Training Labels

  • Place and push scenes represent target poses with floor markers formed by projecting object shapes onto the plane.Each time-series sample records hand pose, hand-camera imagery, and third-person imagery; the hand is rendered as a cube so its pose is visible without assuming a robot morphology.Source: Figure 2 · p. 3, B. Automatic Generation of Training… · p. 3

1) Pick:

Pick trajectories move the hand from its initial pose to a geometrically selected grasp pose. Position and orientation are interpolated, and the gripper closes at the final pose.

1) Pick:

  • The grasp pose is selected from a slender object side, using the face nearest the hand and an outward-offset grasp point oriented toward the object center.This geometric heuristic defines the final pose of the pick trajectory.Source: 1) Pick · p. 3
  • Hand position moves linearly from the initial pose to the grasp pose, while the forward direction is interpolated using spherical linear interpolation.The right and up directions are updated to remain orthogonal to the forward direction.Source: 1) Pick · p. 3
  • The gripper value remains 0 throughout the pick motion except at the final pose, where it is set to 1.Source: 1) Pick · pp. 3–4

2) Place:

The place trajectory uses a point-cloud marker to specify the object’s target pose while preserving the hand–object relationship. The supplied push description defines a pre-contact approach followed by translation toward a marker.

2) Place:

  • A randomly rotated projection of the object point cloud forms the placement marker, specifying the object’s final position and orientation.The grasped object is moved to align with this marker.Source: 2) Place · p. 4
  • Placement preserves the hand–object relative position and ends with the object overlapping the marker in the top-down view.The target position is computed from the marker center, with the hand’s relative offset to the object maintained.Source: 2) Place · p. 4
  • The placement path uses a Bézier curve for lifting and lowering, SLERP for orientation, and gripper values that open, close for transport, then reopen.The final object rotation matches the random rotation used to create the marker.Source: 2) Place · p. 4
  • The push direction is the horizontal component of the vector from the object center to the marker center.The push motion begins with an approach phase; the hand cube’s offset prevents overlap with the object point cloud.Source: 2) Place · p. 4
  • During pushing, the hand and object translate together along the push direction while preserving their relative pose and orientation.The gripper remains open throughout the push.Source: 2) Place · p. 4

IV. EXPERIMENTS

The experiments use ACT to evaluate synthetic pretraining in simulation and real-world environments. They cover three simulated robots and five manipulation tasks, with randomized object positions in evaluation trials.

IV. EXPERIMENTS

  • ACT receives hand-camera and third-person images plus the current hand pose, then predicts the hand pose at the next time step.The hand pose uses an 8-dimensional representation: position, quaternion, and gripper opening.Source: §IV · p. 4
  • Simulation experiments cover pick, place, push, pile, and door tasks across three robot types: UR5e, xArm7, and KinovaGen3.The tasks involve lifting a cube, placing it inside an enclosure, pushing it to an area without lifting, stacking a bottle on a can, and opening a door.Source: Figure 3 · p. 5
  • Object positions are randomized in every evaluation trial in both simulated and real-world environments to reduce the possibility of solving tasks without visual information.Source: §IV · p. 5

1) Experimental Setup:

The setup compares scratch training and two pretraining conditions using 20 task demonstrations per task. The supplied table captions describe peak-success comparisons and a five-task ablation averaged across three robots.

1) Experimental Setup:

  • Each robot–task combination uses 20 task-specific training episodes, with scratch training compared against synthetic and same-environment pretraining.Both pretraining conditions use 2,000 epochs before fine-tuning on those 20 demonstrations.Source: 1) Experimental Setup · p. 5
  • Synthetic pretraining uses 600 episodes: 200 each for pick, place, and push, followed by fine-tuning on task-specific demonstrations.The dataset is trained for 2,000 epochs and converted to the same 8-dimensional hand-pose representation used for robot execution.Source: 1) Experimental Setup · p. 5
  • Same-environment pretraining uses 600 subsampled episodes from nine tasks collected in the MuJoCo setting, with background and table color matching downstream tasks.The dataset is collected using the UR robot and pretrained for 2,000 epochs.Source: 1) Experimental Setup · p. 5
  • Table I compares peak success rates across scratch, synthetic-pretraining, and same-environment-pretraining models; Table II averages peak success across three robots for five tasks and five data-generation conditions.Table II’s conditions include original synthetic data, black images, random motion, no hand cube, and sphere-only objects.Source: Table I · p. 5, Table II · p. 5

2) Results and Discussion:

Synthetic pretraining improved average success over scratch training across the three simulated robots, with faster early learning across many tasks, although individual task outcomes varied. Ablations indicate that visual information and structured motion contribute to performance, while contact-rich pushing remains a challenge.

a) Q1: How does the proposed synthetic dataset affect the convergence speed of task learning and task success rate?

  • Within the first 1,000 epochs, synthetic pretraining increased success rates faster than scratch training across many tasks.The trend toward faster convergence was also observed with same-environment data pretraining.Source: 2) Results and Discussion · p. 5
  • Figure 4 compares scratch training, synthetic pretraining, and same-environment pretraining using mean success rates across repeated evaluations.Models were evaluated every 250 epochs over 50 test runs and three random seeds.Source: 2) Results and Discussion · p. 5, Figure 4 · p. 6
  • 10.3%, 8.5%, and 8.3%: synthetic pretraining improved mean success over scratch for UR, xArm, and Kinova, respectively, across five tasks per robot.Same-environment pretraining had lower average success than the proposed method for all robots; gains were not uniform across tasks.Source: 2) Results and Discussion · p. 6
  • Pretraining gains were not universal: it did not improve over scratch on Push for xArm and Kinova, or on Door for Kinova.The passage attributes limited Push improvement under both pretraining settings to contact dynamics, including friction and slippage, that geometric priors do not model.Source: 2) Results and Discussion · p. 6

b) Q2: Which design elements of the generated data contribute to the pretraining effect?

  • Across five ablation conditions, average performance rose from black images through random motion, no-hand cube, and sphere-only data to original synthetic data.The ordering describes averages and is not monotone for every task; on Push, original data was weakest and random motion strongest, consistent with the importance of contact dynamics.Source: 2) Results and Discussion · p. 7
  • 52.8 vs. 51.6 on average: original synthetic data outperformed no-hand cube data, suggesting third-person views add a small gain beyond hand-camera cues.The hand camera provides visual cues about the hand–object relationship, while third-person images visualize it explicitly.Source: 2) Results and Discussion · p. 7
  • 52.8 vs. 51.7 overall: sphere-only data came closest to original data, but Door favored original data, 56.2 vs. 52.2.The passage suggests object-shape diversity matters more when task objects differ from pretraining shapes.Source: 2) Results and Discussion · p. 7

B. Real Robot Experiments

The real-world experiments compare scratch training with synthetic pretraining on three UR robot tasks, using 30 task-specific episodes per task for scratch training and 600 synthetic episodes for pretraining.

Experimental setup

Training conditions

2) Results and Discussion:

In the real-robot evaluation, geometric pretraining outperformed scratch at most measured stages, with the 1,000-epoch model achieving the highest successful-trial count across tasks and conditions. The conclusion reports improved mean success across the evaluated simulation and real-world task sets, while noting mixed per-task results and limits to the demonstrated scope.

2) Results and Discussion:

  • At most evaluated real-robot stages, pretraining outperformed scratch; the 1,000-epoch model had the highest successful-trial count across tasks and conditions.Figure 6 compares mean success counts for scratch and synthetic-pretrained models across the real-robot tasks.Source: Figure 6 · p. 7, 2) Results and Discussion · p. 7

V. CONCLUSIONS

  • Mean success rate improved over the evaluated task sets in simulation and the real world, although per-task results were mixed.The conclusion summarizes performance across the task sets rather than claiming gains on every individual task.Source: §V · p. 7
  • The framework generates purely geometric data for pretraining followed by fine-tuning, aiming to provide a relative hand–object pose prior for learning from few demonstrations.The stated goal is efficient policy training with a small number of task demonstrations.Source: §V · p. 7
  • The study claims no robustness to arbitrary clutter, and transfer beyond ACT remains an open question.All experiments use ACT as the policy architecture.Source: §V · p. 8
Loading 2609.12721v1…