Source-linked AI summary
POIL: Point-based One-Shot Imitation Learning with Stable Dynamical Systems
Sang Min Kim, Jinwoo Seo, Hyeongjun Heo, Junho Lee, Yonghyeon Lee, Young Min Kim
TL;DR
One-shot manipulation must transfer a demonstrated trajectory to novel objects while handling changing execution conditions, but trajectory transfer and stable closed-loop control have largely remained separate. POIL shares functional-part points across transfer and execution, and experiments show transfer across object categories, grasp poses, and goal geometries, with recovery from disturbances.
Problem
One-shot manipulation must transfer a demonstrated trajectory to novel objects and execute robustly under changing scene conditions, grasp configurations, and external disturbances.
Source: §I · p. 1Method
POIL grounds functional-part points with an MLLM, transfers corresponding point trajectories, and tracks them online with Point-set BCSDM for closed-loop execution.
Source: §I · pp. 1–2Results
In simulation and on a real robot, POIL transfers a single demonstration across object category, grasp pose, and goal geometry, while recovering from external disturbances.
Source: §V · p. 7Takeaways & Limitations
Functional-part points provide a shared representation for one-shot trajectory transfer and stable closed-loop execution without requiring a known object pose or full 3D geometry.
Source: §I · p. 1, §V · p. 7Experiments use a parallel gripper and single grasp, which can leave no collision-free, IK-feasible grasp under large pose changes or no graspable configuration for flat objects.
Source: §V · p. 8
Abstract
from arXiv · showhide
We present POIL, a point-based one-shot imitation learning framework with stable dynamical systems. While one-shot imitation avoids collecting extensive demonstrations, successful one-shot manipulation requires not only transferring a demonstrated trajectory to a novel object but also executing it robustly under changing scene conditions, grasp configurations, and external disturbances. POIL addresses both problems through a shared representation: a set of 3D points on the object's functional part, used jointly for trajectory transfer and closed-loop execution. The one-shot transfer from the demonstrated trajectory is enabled with point correspondences. POIL grounds the shared functional part with a multi-modal large language model, and transfers the trajectory across viewpoint, pose, and object category changes. During execution, multi-view tracking observes the same points online, and Point-set BCSDM drives them in closed loop by projecting per-point velocities onto a single rigid-body twist computed from the tracked points alone. This extends stable dynamical models from an SE(3) pose to a point set without requiring a known 3D model or pose estimator. We show that at the goal the controller becomes a gradient flow on the classical SO(3) potential, so its terminal phase inherits the almost-global convergence of that potential under a rigid-object assumption. Across simulation and real-robot experiments, POIL transfers a single demonstration across object category, grasp pose, and goal geometry, while recovering from external disturbances during execution. Project page: https://sangminkim-99.github.io/poil
I. INTRODUCTION
POIL addresses the gap between one-shot trajectory transfer and stable closed-loop execution by using functional-part points as a shared representation. It transfers a demonstrated motion to novel scenes and tracks those points during execution.
I. INTRODUCTION
- Prior trajectory-transfer methods commonly replay actions open loop, while stable dynamical systems assume an appropriate state representation and reference trajectory.The introduction identifies integrating transfer and stable execution as largely unexplored.Source: §I · p. 1
- POIL bridges trajectory transfer and stable execution using 3D points anchored to an object's functional part; stable-controller tracking closes the loop and provides a convergence guarantee at the goal without dense 3D object geometry.The representation can generalize across different embodiments or object categories.Source: §I · p. 1
- POIL grounds the functional part with an MLLM, establishes point correspondences, and transfers the demonstrated point motion across viewpoints, poses, and object categories.For a changed goal geometry, the trajectory is adapted using a few user-clicked correspondences.Source: §I · pp. 1–2
- Point-set BCSDM tracks the transferred points online and projects their per-point velocities onto one rigid-body twist for the gripper.At the goal, the projection is a gradient descent on a point-set alignment cost, yielding almost-global convergence under the stated rigid-object assumption.Source: §I · p. 2
- Simulation and real-robot experiments show transfer across object category, grasp pose, and goal geometry, with recovery from external disturbances.The introduction also frames Table I as comparing approaches on action-labeled pretraining, object generalization, closed-loop execution, and convergence guarantees.Source: §I · pp. 1–2, Table I · p. 2
A. One-Shot Imitation Learning
One-shot imitation aims to generalize manipulation from a single demonstration, through either learned policies or direct trajectory transfer. Related keypoint methods vary in their supervision, generalization scope, and execution guarantees.
A. One-Shot Imitation Learning
- One-shot imitation seeks to transfer a manipulation skill from one demonstration to unseen scenarios, avoiding the need for many demonstrations.Learning-based approaches rely on action-labeled pretraining and remain limited by its data coverage; correspondence-based transfer can reach novel geometry or object categories but often replays open loop.Source: A. One-Shot Imitation Learning · p. 2
B. Keypoint Representation for Robotic Manipulation
- Keypoint methods offer an object-centric interface, but related approaches may require category- or task-specific supervision, fixed instances, or many demonstrations.RoboTAP follows demonstrated trajectories on a fixed instance, while Point Policy learns control from many demonstrations; neither provides a convergence guarantee.Source: B. Keypoint Representation for Robotic… · p. 2
C. Stable Dynamical Systems
Stable dynamical systems encode demonstrations as time-independent velocity fields that correct deviations and converge to a goal. POIL extends this execution approach through object-centric points, while its convergence analysis assumes a rigid object.
C. Stable Dynamical Systems
- A stable dynamical system uses a time-independent velocity field whose solutions converge to the goal from any initial state while correcting deviations.Prior manipulation formulations generally assume a suitable state representation and reference trajectory, often an object or gripper SE(3) pose.Source: C. Stable Dynamical Systems · p. 2
- POIL combines stable execution with one-shot transfer using object-centric points, rather than requiring an estimated pose or full 3D geometry.Its semantic transfer stage extends the formulation to novel objects; the analysis assumes rigidity, although experiments observe tolerance to mild non-rigidity.Source: C. Stable Dynamical Systems · pp. 2–3
- The overview depicts transfer grounding the functional part and establishing point correspondences, followed by online tracking and Point-set BCSDM execution.The same points connect the transferred trajectory to closed-loop control.Source: Figure 1 · p. 3
III. METHOD
POIL takes one demonstration and transfers its manipulation skill to a novel target object sharing a functional part. A shared set of object points supports closed-loop execution with convergence guarantees.
III. METHOD
- POIL transfers a single demonstration to a novel target object that shares a functional part, then executes with a closed-loop controller.The transfer and execution stages share 3D points on the object's functional part.Source: §III · p. 3
- The demonstration is a multi-view RGB-D video, with camera index n and timestamp τ defining the observation frames.At test time, the robot observes a target scene containing a novel object.Source: §III · p. 3
- At test time, the robot grasps the novel object and drives its functional part along the transferred trajectory.The target-scene observation is denoted Itgt_n in the passage.Source: §III · p. 3
A. Point Trajectory Transfer
POIL transfers object-centric functional-part point trajectories by grounding the part semantically, matching points across objects, and tracking their demonstrated motion. When the target fixture differs, user-clicked correspondences support adapting the trajectory to its geometry.
A. Point Trajectory Transfer
- POIL localizes the functional part by semantic label and uses multi-view prompting to reduce spurious detections caused by self-occlusion.Rex-Omni detects the part from object crops; masks and depth produce 3D part point clouds.Source: A. Point Trajectory Transfer · p. 3
- Assuming similar functional-part structure across objects, POIL aligns their point clouds with a coarse transform followed by occlusion-aware non-rigid registration.The coarse transform combines rigid motion with anisotropic scaling, and is refined using Chamfer distance before OAReg.Source: A. Point Trajectory Transfer · p. 3
- MVTracker follows the demonstrated functional-part points through large out-of-plane rotations, and Kabsch alignment transfers their motion to the target points.Source: A. Point Trajectory Transfer · p. 4
- For changed fixture geometry or pose, user-clicked point pairs guide fixture registration; a fitted TPS warp adapts every transferred trajectory point.The figure illustrates adaptation from a straight rack to an L-shaped rack.Source: A. Point Trajectory Transfer · p. 4, Figure 3 · p. 4
B. Closed-Loop Execution via Point-set BCSDM
POIL executes transferred motions by tracking functional-part points and using Point-set BCSDM to produce closed-loop object motion. Under the rigid-object and general-position assumptions, its terminal rotational dynamics are almost-globally asymptotically stable.
Grasp selection
- POIL selects a feasible grasp by evaluating grasp–matching pairs for joint-limit and collision violations, including alternative symmetric point matchings.Candidate paths are checked for inverse-kinematics feasibility and approximate robot-scene collision.Source: B. Closed-Loop Execution via Point-set… · p. 4
Online point tracking
- Online multi-view tracking updates functional-part points, while whole-object support points stabilize their estimates under occlusion and large rotations.Between tracker updates, point positions are propagated using end-effector motion from robot forward kinematics.Source: B. Closed-Loop Execution via Point-set… · p. 4
Point-set BCSDM controller
- Point-set BCSDM converts per-point reference velocities into one least-squares rigid-body twist, then maps that twist to the wrist through the fixed grasp.The fit uses RANSAC to reject outlier point velocities before the final least-squares solve.Source: B. Closed-Loop Execution via Point-set… · p. 4
Rotational stability
- At the target, the rotational controller is gradient descent on D(R)=tr(W(I−R)) in metric A, with W derived from the tracked point set.The proof assumes general position, with W having distinct positive eigenvalues; the other critical rotations are nondegenerate and not local minima.Source: B. Closed-Loop Execution via Point-set… · p. 5
- The rotational dynamics are almost-globally asymptotically stable at the target, with convergence outside a measure-zero set of initial rotations.The proof treats the controller as gradient flow on D in metric A; the three other critical rotations are not local minima.Source: B. Closed-Loop Execution via Point-set… · p. 5
IV. EXPERIMENTS
The evaluation separates point transfer and control studies from full-pipeline tests across scene and execution variations.
Evaluation overview
- The evaluation comprises four studies: functional-part point transfer, control with ground-truth inputs, full-pipeline variation tests, and external-disturbance recovery.The full-pipeline study varies pose, grasp, object, and goal.Source: §IV · p. 5
Evaluation setup
- For all experiments, POIL uses 10 functional-part points, η = 3, and 40 support points.Source: §IV · p. 5
Evaluation overview
- The transfer and control studies isolate those components, while the full-pipeline study evaluates them across pose, grasp, object, and goal variations.Source: §IV · p. 5
A. Functional-part Point Transfer
POIL evaluates functional-part point transfer under viewpoint and pose changes, comparing name-grounded prompting with dense visual-feature matching. It reports stronger transfer accuracy across the tested angular offsets.
A. Functional-part Point Transfer
- Baselines use DINOv2 or DIFT visual features for single-view or multi-view matching, while POIL receives object and functional-part names for prompting.The comparison isolates whether name-based part grounding improves correspondence under pose change; a per-view Rex-Omni prompting ablation is also included.Source: A. Functional-part Point Transfer · pp. 5–6
- The controlled evaluation uses mugs, pans, and scissors across 12 viewpoints, testing target views at azimuth offsets of 30°–90° from each demonstration view.The evaluation examines whether functional-part transfer remains accurate under large viewpoint and pose changes.Source: A. Functional-part Point Transfer · pp. 5–6
- POIL achieves the lowest normalized point error and highest correct-point fraction at every tested angular offset and threshold.The results indicate that name-based functional-part grounding is more robust to pose change than dense visual-feature matching alone.Source: A. Functional-part Point Transfer · p. 6
B. Pose-Robust Control with Point-set BCSDM
Point-set BCSDM is evaluated for reaching demonstrated goals from poses outside the demonstration distribution, using ground-truth point trajectories to isolate control. It maintains 100% success across tested distances from a single demonstration.
B. Pose-Robust Control with Point-set BCSDM
- The control comparison uses ground-truth point trajectories and direct object actuation, comparing Point Policy and Instant Policy with Point-set BCSDM.Point Policy memorizes a world-frame trajectory from one demonstration; Instant Policy conditions on demonstrations through a motion prior learned from synthetic data.Source: B. Pose-Robust Control with Point-set… · p. 6
- The four tasks range from rigid cube translation to reshelving, mug-handle insertion, and hanging a slightly deformable bag.Mug Insertion requires following the demonstration trajectory rather than taking a shortcut to the goal pose.Source: B. Pose-Robust Control with Point-set… · p. 6
- Point-set BCSDM maintains 100% success across all tested initial-pose distances from a single demonstration.The comparison includes four tasks and evaluates near and far initial poses across the full yaw range.Source: B. Pose-Robust Control with Point-set… · p. 6
- With one demonstration, Point Policy often drives toward the final pose rather than the nearest trajectory configuration under out-of-distribution poses, causing collisions with task geometry.Its 50-demonstration success remains high near the training distribution but drops with distance.Source: B. Pose-Robust Control with Point-set… · p. 6
C. End-to-End evaluation
End-to-end evaluation tests POIL across object displacement, grasp changes, goal-geometry changes, and cross-category transfer in simulation and on hardware. The reported executions include disturbance recovery and successful transfers, alongside detection, grasp, and feasibility failures.
C. End-to-End evaluation
- OAReg and TPS adapt the demonstrated functional-part trajectory to changed fixtures, including a taller box, tilted rack, and L-shaped rod.In the plate-rack task, fixture alignment also accommodates a change in rack position.Source: Figure 8 · p. 7, C. End-to-End evaluation · p. 7
- POIL executes a transferred mug-hanging demonstration with a side-lying mug and a different feasible grasp, driving the handle to the rack from that grasp.The demonstrated rim grasp is blocked by collision in this setting.Source: C. End-to-End evaluation · p. 7
- POIL completes 39 of 50 randomized side-lying mug trials; seven failures stem from part detection, three from no valid grasp, and one from mug-floor collision.The collision is not captured by the current gripper-trajectory feasibility check; the authors identify detection and grasp selection as the main pipeline bottlenecks.Source: C. End-to-End evaluation · p. 7, Figure 9 · p. 7
- In real-world transfer, POIL succeeds in 6 of 10 sim-to-real mug trials, with 3 tracking failures and 1 infeasible IK solution; other transfers are shown qualitatively.Source: C. End-to-End evaluation · p. 7
- After a human pulls the gripper and rotates the mug within the grasp, the tracked points still guide its handle to the demonstrated goal.The disturbance-recovery example is a sim-to-real mug-hanging task.Source: C. End-to-End evaluation · p. 7
V. CONCLUSION AND LIMITATIONS
POIL uses functional-part points for both one-shot trajectory transfer and stable closed-loop execution. The authors report transfers across object category, grasp pose, and goal geometry, while identifying assumptions and execution constraints.
V. CONCLUSION AND LIMITATIONS
- POIL uses functional-part points as a shared representation for one-shot trajectory transfer and stable closed-loop execution.The conclusion contrasts this with prior trajectory-transfer methods that execute open loop and pose-based dynamical systems that assume known object pose or a fixed grasp.Source: §V · p. 7
- The method assumes object and functional-part queries, a few user-clicked correspondences for changed fixtures, and geometrically similar source and target parts.Execution also depends on continued point tracking; large out-of-view motions, severe occlusion, or unreliable depth on transparent or specular surfaces can break tracking.Source: §V · pp. 7–8
- In the sim-to-real mug-hanging example, the handle reaches the demonstrated goal after the mug is rotated within the grasp.The figure depicts disturbance recovery during execution.Source: Figure 11 · p. 8
- Experiments use a parallel gripper and a single grasp, which can leave no collision-free, IK-feasible grasp under large pose changes or no graspable configuration for flat objects.The authors suggest sequential transfers with regrasping or dexterous hands as future directions.Source: §V · p. 8