PaperlayerView exact revision on arXiv

Source-linked AI summary

You Only Demonstrate Once: Category-Level Manipulation from Single Visual Demonstration

Bowen Wen, Wenzhao Lian, Kostas Bekris, Stefan Schaal

arXiv:2201.12716v2cs.ROcs.AIcs.CVcs.SYeess.SY

TL;DR

Instance-specific transfer and complex action sequences remain challenging for category-level manipulation, which can also require costly data collection and keypoint annotation. The paper presents a simulation-trained, visual framework that transfers a single demonstrated trajectory to novel objects and uses tracking for closed-loop behavior cloning. Experiments report robustness to manipulation dynamics and generalization across object instances and scenes in high-precision, long-horizon assembly tasks.

  • Problem

    Category-level manipulation still faces costly real-world data collection and task-specific keypoint annotation, while coarse predictions and ignored intermediate actions limit complex tasks beyond pick-and-place.

    Source: §I · p. 2

  • Method

    The framework learns an object-centric category representation in simulation, parses a single visual demonstration, reprojects its trajectory to novel objects, and uses model-free 6 DoF tracking for closed-loop last-inch behavior cloning.

    Source: §I · p. 2, §IV · p. 3, A. Offline Learning of a… · p. 4

  • Results

    Experiments demonstrate efficacy on high-precision, long-horizon assembly tasks, with robustness to manipulation-dynamics uncertainty and generalization across object instances and scenes.

    Source: §VI · p. 10

  • Takeaways & Limitations

    A robot can be taught different manipulation strategies from a single demonstration without manual programming, within the evaluated high-precision assembly setting.

    Source: §I · p. 2, §VI · p. 10

    Battery assembly can push the battery out of the gripper, while gear insertion can lose collision-free IK solutions after contact-driven in-hand rotation; regrasping or in-hand adjustment may recover the latter.

    Source: B. Limitations and failure modes · p. 13

Abstract

from arXiv · show

Promising results have been achieved recently in category-level manipulation that generalizes across object instances. Nevertheless, it often requires expensive real-world data collection and manual specification of semantic keypoints for each object category and task. Additionally, coarse keypoint predictions and ignoring intermediate action sequences hinder adoption in complex manipulation tasks beyond pick-and-place. This work proposes a novel, category-level manipulation framework that leverages an object-centric, category-level representation and model-free 6 DoF motion tracking. The canonical object representation is learned solely in simulation and then used to parse a category-level, task trajectory from a single demonstration video. The demonstration is reprojected to a target trajectory tailored to a novel object via the canonical representation. During execution, the manipulation horizon is decomposed into longrange, collision-free motion and last-inch manipulation. For the latter part, a category-level behavior cloning (CatBC) method leverages motion tracking to perform closed-loop control. CatBC follows the target trajectory, projected from the demonstration and anchored to a dynamically selected category-level coordinate frame. The frame is automatically selected along the manipulation horizon by a local attention mechanism. This framework allows to teach different manipulation strategies by solely providing a single demonstration, without complicated manual programming. Extensive experiments demonstrate its efficacy in a range of challenging industrial tasks in highprecision assembly, which involve learning complex, long-horizon policies. The process exhibits robustness against uncertainty due to dynamics as well as generalization across object instances and scene configurations. The supplementary video is available at https://www.youtube.com/watch?v=WAr8ZY3mYyw

I. INTRODUCTION

Instance-specific manipulation skills can be costly to transfer, while category-level approaches face challenges with complex action sequences, visual-feedback robustness, and data collection. This work proposes visual, single-demonstration category-level manipulation and frames its goal as transfer to unseen objects without additional fine-tuning or robot-object interaction.

I. INTRODUCTION

  • Complex, long-horizon tasks make manually specifying goals and constraints for each action segment tedious, while open-loop execution and extensive instance collection or annotation create further challenges.The battery-assembly example requires pressing at an angle, aligning both ends, and pressing down.Source: §I · p. 2
  • The framework combines simulation-trained object-centric representations, model-free 6 DoF tracking, single-video imitation, and attention-based local coordinate-frame selection for category-level behavior cloning.The authors report real-world experiments on high-precision, long-horizon manipulation and claim generalization across environments and task configurations.Source: §I · p. 2

II. RELATED WORK

  • Prior video-based behavior-cloning methods often use robot action labels, while earlier single-video work addressed simple tasks in a constrained 4D space rather than full SE(3) manipulation.The proposed approach is described as object-centric and does not require robot-object interaction for training.Source: §II · p. 3

III. PROBLEM SETUP

  • The stated objective is to learn a skill from one demonstration for unseen objects in the same category, without additional fine-tuning or robot-object interaction.The demonstration may be a third-person view of a teleoperated robot or a human performing the task.Source: §III · p. 3

IV. APPROACH

The approach parses a demonstrated object trajectory, reprojects it to a novel instance, and divides execution between collision-free travel and last-inch behavior cloning. Tracking feedback guides the latter toward the target trajectory.

IV. APPROACH

  • The demonstration is parsed as an object-pose trajectory in the receptacle’s coordinate frame, allowing the representation to generalize across different absolute scene configurations.Each pose is represented in SE(3).Source: §IV · p. 3
  • Category-level behavior cloning reprojects the demonstrated trajectory to a novel object using dense correspondence, then follows the tailored target by comparing tracked and reference poses.The figure describes nearest-neighbor matching between category-level NUNOCS representations to transfer the trajectory.Source: §IV · p. 3, Figure 1 · p. 4
  • Execution first plans a collision-free path to the keypose marking the start of the last-inch demonstration, then uses behavior cloning through task completion.The approach assumes the grasp does not obstruct the downstream manipulation task.Source: §IV · pp. 3–4

A. Offline Learning of a Category-Level Representation

The framework establishes cross-instance correspondence with a category-level representation and uses model-free 6 DoF tracking to parse demonstrations and provide execution feedback. Its representation is learned from synthetic data and applied to real objects without retraining.

A. Offline Learning of a Category-Level Representation

  • NUNOCS extends category-level object representation to 9 dimensions—6D pose and 3D scaling—to establish dense correspondence between a demonstrated object and a novel instance.The work adopts this representation for longer-horizon tasks and category-level behavior cloning.Source: A. Offline Learning of a… · p. 4
  • NUNOCS Net maps a scanned partial point cloud to canonical-space coordinates and non-uniform scales, then solves the transformation in closed form using least squares.The network predicts coordinates and scales in separate branches; the training objective combines NUNOCS and scale losses.Source: A. Offline Learning of a… · pp. 4–5
  • The NUNOCS representation is learned from synthetic data and applied in the real world without retraining or fine-tuning.The synthetic pipeline uses domain randomization and depth alignment to address the sim-to-real gap.Source: A. Offline Learning of a… · p. 5

B. Model-free 6 DoF Object Motion Tracking

  • Model-free 6 DoF tracking extracts a demonstration trajectory relative to the receptacle and supplies online visual feedback for closed-loop execution.The tracker does not require object CAD models and can operate at the RGBD camera’s 10 Hz acquisition rate after initialization.Source: B. Model-free 6 DoF Object Motion… · p. 5

C. Category-Level Behavior Cloning as Last-Inch Policy

Category-level behavior cloning follows a trajectory reprojected for the novel object, using online motion tracking during last-inch manipulation. A local attention mechanism transfers and dynamically anchors the coordinate frame to task-relevant object regions.

C. Category-Level Behavior Cloning as Last-Inch Policy

  • NUNOCS correspondence transfers the attention heatmap to novel objects, dynamically anchoring the coordinate system at a local attended point to capture scaling and local-typology variation.The coordinate system is translated to the attended region while retaining the original canonical-frame orientation.Source: Figure 2 · p. 6
  • CatBC reprojects the demonstration into a target trajectory for a novel object and uses updated visual tracking to keep the object on that path despite manipulation-dynamics uncertainty.The trajectory is discretized densely, with neighboring poses about 2 mm or 2° apart in the implementation.Source: C. Category-Level Behavior Cloning as… · p. 6

D. Dynamic Category-Level Frame via Local Attention

A fixed canonical frame may not fit different tasks or remain task-relevant throughout a complex trajectory. Local attention dynamically selects an anchor from the demonstration and transfers it to novel objects through dense correspondence.

D. Dynamic Category-Level Frame via Local Attention

  • A single canonical origin can misalign objects with a receptacle: battery standing favors the terminal surface center, while assembly favors the negative pole’s lowest edge center.The examples illustrate why a manually chosen task frame may not remain suitable throughout a complex task.Source: D. Dynamic Category-Level Frame via… · p. 6
  • Local attention dynamically selects a category-level frame origin along the manipulation horizon, addressing tasks where the relevant local frame changes.The frame is translated to the attended region while retaining the canonical orientation.Source: D. Dynamic Category-Level Frame via… · pp. 6–7
  • The attention map is computed over the demonstrated object’s 3D points using its pose relative to the receptacle along the trajectory.The method assigns higher attention to object parts closer to the receptacle, then transfers the heatmap to novel objects through NUNOCS dense correspondence.Source: D. Dynamic Category-Level Frame via… · pp. 6–7
  • Compared with a few pre-specified keypoints,, dynamic attention generates implicit keypoints with orientation information that adjusts along the horizon.The authors state this reduces human effort and enables high-precision category-level behavior cloning.Source: D. Dynamic Category-Level Frame via… · p. 7

E. Grasping the Object and Transferring it to the Keypose

The framework allows task-relevant grasp planning as long as the grasp supports downstream manipulation. After grasping, online pose updates feed collision-free transport to a keypose before last-inch manipulation.

E. Grasping the Object and Transferring it to the Keypose

  • The framework is not tied to one grasp planner; task-relevant grasp planning can use CAD-model-free methods when the grasp supports downstream manipulation.Grasp planning is not the focus of this work.Source: E. Grasping the Object and… · p. 7
  • After grasping, the 6 DoF tracker updates the in-hand pose, which initializes a planner for collision-free transport to the keypose.The implementation uses RRT*; the keypose is empirically set 5 cm from the receptacle along the demonstrated trajectory.Source: E. Grasping the Object and… · p. 7
  • The long-range, collision-aware path is followed by last-inch manipulation, focusing on actions critical to task completion and supporting safety in scenes with novel obstacles.The authors describe this decomposition as an implicit attention mechanism that ignores unimportant background information in the demonstration.Source: E. Grasping the Object and… · p. 7

V. EXPERIMENTS

The evaluation uses real-world experiments across two object categories and three manipulation tasks. A single demonstration per task is used to test transfer to novel objects and varying scenes, alongside seven baselines.

V. EXPERIMENTS

  • Real-world evaluation uses a Kuka IIWA14 arm, Robotiq Hand-E gripper, 10 Hz Photoneo grayscale-and-depth camera, and a spring-damper device for passive compliance.The robot is position-controlled using joint commands from the manipulation policies.Source: A. Experimental Setup · p. 7
  • Tests cover seven gear and nine battery instances chosen to vary in shape and appearance and differ from training objects.The proposed approach learns in simulation from virtual 3D models, while real-world training objects supply data for baseline training.Source: A. Experimental Setup · p. 7, Figure 3 · p. 7
  • The evaluation comprises 1,560 task trials across three tasks, seven baselines, and varied initial object and receptacle configurations.Each object-task-tolerance setting uses five trials unless otherwise specified.Source: A. Experimental Setup · p. 7
  • For each task, one video of a human manipulating a randomly selected training object is recorded offline and used to manipulate novel objects without additional data collection.The evaluation asks about single-demonstration learning, generalization, robustness to scenes and dynamics, and contact-rich interaction.Source: A. Experimental Setup · p. 7

B. Baseline Methods

The baselines include vision-based category-level methods, including behavior-cloning variants, and an open-loop ablation. In the battery tasks, the results distinguish simple placement from contact-rich assembly, where intermediate actions and visual feedback matter.

B. Baseline Methods

  • The baseline set includes DON, KPAM, and KPAM 2.0 with manually specified task keypoints, their behavior-cloning variants, and Ours-no-tracking.The behavior-cloning variants replace manual goal specification with visual demonstration and integrate video parsing and CatBC; Ours-no-tracking disables the visual tracker.Source: B. Baseline Methods · pp. 7–8

C. Battery Standing Task

  • For battery standing, Ours succeeds across battery instances, while disabling motion tracking reduces success to 80.0% after gripper closing perturbs battery orientation.The gain from behavior cloning is insignificant for this pick-and-place task; the authors attribute failures of keypoint and correspondence baselines to unreliable goal constraints.Source: Table I · p. 8, C. Battery Standing Task · p. 8

D. Battery Assembly Task

  • Battery assembly requires pressing a spring at least halfway, 16 mm, before inserting the battery; releasing it lets the spring hold the battery against the receptacle wall.The task is sensitive to contact forces and the last-inch action sequence.Source: D. Battery Assembly Task · p. 8
  • For battery assembly, goal-only DON, KPAM, and KPAM 2.0 fail; behavior-cloning variants improve performance, while closed-loop Ours outperforms Ours-no-tracking under uncertain contact dynamics.The spring can be squeezed off-axis, and friction and elastic interaction can move the battery in-hand; tracking updates the state to guide the target trajectory.Source: Table II · p. 8, D. Battery Assembly Task · p. 8

E. Gear Insertion Task

The gear-insertion experiments test precision across shaft tolerances, while the framework analysis examines generalization to new scenes, disturbance robustness, and alternative manipulation strategies.

E. Gear Insertion Task

  • Gear insertion tests use 0.1, 0.5, and 5 mm shaft tolerances (or 3 mm when limited by the gear hole); success means the hole passes through the shaft.The task probes precision and robustness under contact-rich uncertainty.Source: E. Gear Insertion Task · pp. 8–9
  • 100% versus 42.9%: adding motion tracking raises performance over Ours-no-tracking; Ours remains feasible at 0.1 mm with 28.6% success.KPAM, KPAM 2.0, and DON struggle even at 5 mm, and the performance gap grows at 0.5 mm tolerance.Source: E. Gear Insertion Task · p. 9

F. Framework Analysis

  • The framework generalizes to unseen gear poses and obstructed battery-assembly scenes, planning collision-free paths around novel obstacles.The authors attribute scene generalization to separating long-range motion from last-inch manipulation and representing the task trajectory independently of the scene.Source: F. Framework Analysis · p. 9, Figure 4 · p. 10
  • After human dragging changes a gear’s pose, continuous visual tracking lets CatBC follow the desired trajectory.The tracker feeds the latest object state to the controller.Source: F. Framework Analysis · p. 9
  • The “human pivot” demonstration, which anchors the gear against the shaft before pivoting, improves success over simple top-down insertion.Fig. 4(f) shows the demonstration and the robot executing the learned policy on a test object.Source: F. Framework Analysis · p. 9, Figure 4 · p. 10
  • Teleoperation demonstrations yield slightly improved performance, but may be cumbersome for complex long-horizon actions.Motion tracking runs at 10 Hz in parallel; last-inch runtime varies mainly with target-trajectory length, while long-range runtime depends on distance and obstacle complexity.Source: F. Framework Analysis · pp. 9–10

VI. DISCUSSIONS AND FUTURE WORK

The framework supports single-demonstration, closed-loop category-level manipulation across objects and scenes. Its current scope is limited to vision and rigid objects.

VI. DISCUSSIONS AND FUTURE WORK

  • The framework uses visual feedback to learn category-level manipulation from one demonstration, with experiments covering high-precision assembly and long-horizon policies.The authors report robustness to manipulation-dynamics uncertainty and generalization across object instances and scenes.Source: §VI · p. 10
  • Vision is the only sensing modality; force or tactile sensing could further improve behavioral-cloning accuracy in high-precision manipulation.This is a proposed direction, not an evaluated result in the stated scope.Source: §VI · pp. 10–11
  • Experiments consider rigid objects only; articulated and deformable objects require suitable category-level representations.Cables are given as an example of deformable objects.Source: §VI · p. 11

Appendix

The appendix details synthetic-data training and trajectory handling, documents two failure modes, and describes the comparison baselines and their evaluation scope.

A. Implementation details

  • NUNOCS training uses randomized simulated scenes, including object scale, pose, and physical parameters, with depth-missing corruption and point-cloud augmentation.The network is trained on synthetic data generated with Blender.Source: A. Implementation details · p. 13
  • Demonstration trajectories are discretized with neighboring goals at least 2 mm or 2° apart, and symmetric poses are checked in distance order for collision-free IK.Source: A. Implementation details · p. 13

B. Limitations and failure modes

  • Battery spring compression can eject the grasped battery, while gear contact can rotate it until collision-free IK fails; tactile feedback or dexterous regrasping are proposed responses.Source: B. Limitations and failure modes · p. 13

C. Baseline methods

  • DON, KPAM, and KPAM 2.0 provide dense-correspondence or semantic-keypoint baselines; KPAM 2.0 adds orientation information to keypoints,,.KPAM 2.0 disables force sensing so vision remains the primary sensing modality.Source: C. Baseline methods · p. 13
  • DON BC, KPAM BC, and KPAM 2.0 BC replace manual goals with a single video, reuse the extracted trajectory, and rely on their respective baseline correspondences,,.Direct DON correspondence parsing produced poor trajectories under significant demonstration occlusion, so these variants use the approach's extracted trajectory.Source: C. Baseline methods · pp. 13–14
  • A concurrent approach requiring four tabletop RGB-D cameras was excluded because the evaluated methods use a single-camera setting [62].Source: C. Baseline methods · p. 14

D. Training data collection for baseline methods

The baseline datasets follow prior collection pipelines, with synthetic NUNOCS labels and real-world keypoint or correspondence data collected for comparison methods.

Loading 2201.12716v2…