PaperlayerView exact revision on arXiv

Source-linked AI summary

One-Shot Learning from Demonstration of Contact-Rich Robotic Manipulation by Identifying Physical Interactions

A. H. G. Overbeek, H. van der Kooij, M. Vlutters

arXiv:2608.24741v1cs.RO

TL;DR

Contact-rich manipulation demands physical data and interaction modeling that many LfD methods do not explicitly provide. The paper identifies what, where, and when interactions occur, derives contact-aware transitions, and tracks demonstrations with hybrid position-force control. Across real robot tasks, the approach supports single-demonstration reproduction, geometric robustness and generalization, and online adaptation, while relying on discrete threshold-based modeling and constrained control assumptions.

  • Problem

    Contact-rich, sequential manipulation requires substantial physical data, while existing LfD methods often lack explicit modeling of contact interactions and available datasets may lack haptic information.

  • Method

    The method identifies interaction classes, frames, and segments, derives contact-aware transition conditions, and tracks demonstrated poses and wrenches with hybrid position-force control.

  • Results

    Explicitly modeling physical interactions facilitated single-demonstration reproduction, robustness to unknown geometric variation, generalization with known variation, and online adaptation across evaluated manipulation tasks.

  • Takeaways & Limitations

    Interaction-aware modeling provides an interpretable basis for implementing robustness, generalization, and adaptivity in few-shot robotic LfD.

  • Takeaways & Limitations

    The approach uses discrete interaction classes and prespecified thresholds, whose sensitivity may limit robustness when thresholds are difficult to choose.

Abstract

from arXiv · show

Learning from Demonstration (LfD) allows robots to learn manipulation tasks directly from humans, thereby supporting the versatile application of robots. Most LfD methods do not explicitly model the physical interactions between a robot and its environment, such as the making and breaking of contact, while these are crucial during manipulation tasks. Because the same basic physical interactions recur often, they can be a basis for robust, generalizable, and adaptive task reproduction. We propose an LfD method that explicitly uses what physical interactions take place where and when. Using that information, a hybrid position-force controller tracks demonstrated trajectories until contact-based transition conditions from the demonstrations are met. We evaluate our method in real robot experiments consisting of opening doors and locks, bolt picking and screwing, dislodging, and surface contouring. We show that explicitly modeling physical interactions benefits LfD in four ways. First, by allowing reproduction of complex, sequential, and contact-rich manipulation tasks using only a single demonstration and no prior knowledge of the task. Second, by facilitating robustness to unknown geometric variations in the environment. Third, by facilitating generalization when geometric variations are known. Fourth, by facilitating online adaptation using geometric information explored during task reproduction. We discuss how robustness, generalization, and adaptivity can be explicitly implemented, which is generally lacking in the LfD literature. Thereby, our work aims to close a gap in interpretable few-shot LfD of robotic manipulation.

I. INTRODUCTION

The paper addresses data and modeling challenges in contact-rich robotic manipulation by explicitly representing physical interactions in few-shot LfD. It extends interaction-aware task modeling to support single-demonstration reproduction, robustness, generalization, online adaptation, and interpretability.

  • Motivation: Contact-rich manipulation requires multidimensional physical data, while available datasets often lack essential haptic information such as interaction forces.These tasks involve long horizons, nonlinear dynamics, constraints, and geometric variation.
  • Motivation: Robotics-specific LfD methods reduce data requirements by using domain knowledge and few demonstrations with little task-specific prior knowledge.The paper focuses on the imitation-learning stage of robot learning.
  • Related Work: Explicit physical-interaction modeling represents what, where, and when, enabling interaction-specific control, recurring task segments, and geometry-aware task representations.These representations can support control selection, recognition, and reproduction in similar geometric situations.
  • Research Gap: Prior work did not simultaneously model all three interaction aspects for complex, sequential, contact-rich tasks.This gap motivates the paper’s extension of contact-based task modeling.
  • Contribution: The proposed method derives contact-aware transition conditions and tracks demonstrated poses and wrenches after identifying interaction classes, frames, and segments.The approach is evaluated with conventional reproduction control across manipulation tasks.
  • Contribution: Explicit interaction modeling facilitates single-demonstration learning, robustness to unknown geometry, generalization with known geometry, and online adaptation after exploration.The method is designed to remain human-interpretable while making these properties explicit.

III. EXTENDED TASK MODELING

The extended task model represents demonstrations as segmented, interaction-aware trajectories in task-relevant frames. It adds transition conditions and unilateral interaction classification to support contact-aware reproduction.

  • Transition Conditions: The extended model derives transition conditions between segments so reproduction can switch when demonstrated interaction changes are detected.The model explicitly separates how trajectories are tracked from how transitions are triggered.
  • Task Modeling Summary: The method records poses, twists, and wrenches, then identifies interaction classes, frames, and time segments from one demonstration.These elements define what interactions occur, where they occur, and when they occur.
  • Task Modeling Summary: Demonstrations are segmented when twist-wrench signal norms cross prespecified thresholds, after which extraneous segments are filtered.The resulting N segments use normalized progress τ ∈ [0, 1].
  • Interaction Frame Identification: Each segment receives a task-relevant interaction frame that minimizes and decouples Cartesian mechanical-power components.Segment twist-wrench pairs are expressed in their corresponding interaction frames.
  • Interaction Classification: Interaction classes distinguish idle, free motion, impeded motion, and constrained motion across rotational and translational degrees of freedom.Classes are assigned by applying thresholds to RMS twist-wrench signals.
  • Interaction Classification: The original six-class formulation assumes bilateral interactions, motivating the extension to directional interaction classes.This assumption limits representation of unilateral contacts.

B. Deriving Unilateral Interaction Classes

The method extends interaction classification from six bilateral classes to twelve directional classes by separating positive and negative directions along each frame axis. This represents unilateral contacts such as one-sided constraints.

  • Directional Classification: Unilateral interactions are modeled by allowing positive and negative directions of frame axes to have different interaction classes.The extension produces twelve interaction classes instead of six.
  • Directional Classification: The twist-wrench pair is split into components corresponding to positive and negative directions of each frame axis.Each directional component is classified separately.
  • Classification Rule: Directional classification uses whether each twist-wrench pair lies in the top-right positive or bottom-left negative half-region of a force-velocity plot.The regions are separated by the dashed black line in Fig. 2b.
  • Static-Environment Constraint: For static environments, positive and negative directions of an axis must share a class or have one direction idle.A ratchet is a counterexample because it permits motion in one direction while constraining the other.

C. Inferring Transition Conditions

The method infers transition conditions by detecting changes in interaction classes between consecutive demonstration segments. These conditions are expressed in interaction frames and monitored during reproduction to trigger segment switching.

  • Interaction classes change between consecutive segments, revealing which physical transitions occurred during the demonstration.The classes distinguish interaction states such as inertia, free motion, zero motion, and constraint.
  • The approach assumes that all information needed to control segment transitions is present in the measured demonstration data.
  • Each three-dimensional segment yields twelve transition conditions for positive and negative translations and rotations along its interaction-frame axes.The conditions cover both making and breaking constraints.
  • Transition conditions encode events such as encountering a linear constraint along a specified interaction-frame direction at a segment endpoint.During reproduction, these conditions are monitored to control switching to the next segment.
  • Transitions are compared in a common interaction frame, choosing the frame with the fewest constraint-change conditions for a sparser representation.Either adjacent segment’s frame can be used after expressing both segments in that frame.

IV. POSE TRAJECTORIES AND GEOMETRIC GENERALIZATION

Pose trajectories are represented through task-relevant interaction frames and adapted to geometric variation using constant offsets or trajectory blending. These transformations preserve suitable trajectory behavior while enabling reproduction in changed situations.

  • The method tracks demonstrated poses rather than twists to avoid drift from the desired final pose during trajectory reproduction.
  • Interaction-frame trajectories express task motion using frame parameters that can be changed to represent new geometric situations.The frames use time-dependent orientation and displacement components, with four frame types determined by their reference bodies.
  • Known geometric variations can be incorporated from vision or haptic measurements obtained before or during reproduction.
  • Applying Constant Offsets: A constant pose offset is applied to all segments when every segment shares the same displacement from the demonstration.Different segment offsets require modifying the intermediate trajectory that links them.
  • Geometric Generalization: The implementation assumes that only the environment varies geometrically, so adaptations modify only ground-respective interaction-frame components while grasped objects remain unchanged.
  • Blending Trajectories: Trajectory blending links segments with different offsets by interpolating between poses transformed to the desired start and end configurations.A sigmoid blending function keeps the trajectory near the original curvatures at its start and end, including contact-making and contact-breaking directions.

V. TASK REPRODUCTION

Task reproduction uses established hybrid position/force control extended with contact-aware monitoring and switching. Control is selected per interaction-frame axis from the interaction classes identified in each demonstration segment.

  • The reproduction controller combines hybrid position/force control with contact-aware monitoring and switching between task segments.
  • The controller operates in identified interaction frames and tracks demonstrated poses and wrenches for each segment.
  • Pose control is selected for free-motion axes, while wrench control is selected for constrained axes identified during the demonstration.
  • For idle axes, pose control is chosen to avoid drift; when both twists and wrenches are substantial, control selection is less clear.

B. Hybrid Position/Force Control

The controller combines pose and wrench feedback in interaction frames and switches segments only after progress, tracking, and expected contact-transition conditions are satisfied. Relaxed tolerances and exploratory motions address contact changes that occur earlier or later than demonstrated.

  • Hybrid Position/Force Control: The hybrid controller computes a six-dimensional control wrench by combining pose and wrench control terms with a binary per-axis selector.The selector chooses pose or wrench control element-wise in the interaction frame.
  • Hybrid Position/Force Control: Constrained axes use proportional-integral wrench control with feedforward based on the error between demonstrated and measured wrenches.
  • Hybrid Position/Force Control: Non-constrained axes use proportional-integral-derivative pose control based on pose error expressed in the interaction frame.
  • Conditions for Tracking Completion: Tracking-only segments complete when progress reaches the endpoint and pose-tracking errors satisfy prespecified tolerances.Pose errors are checked on pose-tracked axes, while wrench-tracked axes prioritize maintaining contact.
  • Conditions for Contact-Aware Transitions: Expected contact transitions are monitored online and can guide segment completion when completion tolerances are relaxed by α > 1.
  • Monitoring and Switching Control: Earlier contact causes switching at the detected transition within relaxed tolerances, whereas later contact triggers pose or wrench exploration.
  • Monitoring and Switching Control: Exploration continues only until the expected constraint transition occurs or the relaxed tracking tolerances are reached.

D. Online Trajectory Adaptation using Haptic Information

Online trajectory adaptation uses the pose reached at a segment transition to adjust subsequent trajectories for changed geometry. Demonstrations are collected once and reproduced across geometric variations.

  • Online adaptation: Subsequent segments can be adapted when a segment ends in a pose different from the demonstration because of unknown geometric variation.The measured pose offset is used during segment transitions to generalize the next segment online.
  • Online adaptation: Whether to use a constant offset or trajectory blending depends on whether the measured offset also applies to the complete next segment.Shared contact with the same locally offset geometry supports a constant offset; otherwise, trajectory blending may be needed.
  • Experimental setup: Each complex task was demonstrated once by a human guiding the robot in low impedance mode before autonomous modeling and reproduction with geometric variations.The variations included changes such as object poses.

A. Reproduction Evaluation

The evaluation progressively enabled pose control, interaction-frame hybrid control, contact-aware switching, online adaptation, and offline generalization as geometric variations increased. Real-robot experiments covered cabinet subtasks and used repeated trials to assess robustness, adaptivity, and prior-information-based generalization.

  • Evaluation procedure: Each task variation was reproduced 10 times while controller steps were enabled progressively after failures, and success rates were reported over increasingly large geometric variations.The first three steps evaluated robustness to unknown variations; later steps evaluated online adaptivity and offline generalization.
  • Experimental setup: The experiments used a Franka Research 3 robot, Franka Hand gripper, force-torque sensor, household objects, and a custom end effector for the final task.The monitor/control system ran at 100 Hz over ROS2 Humble on Ubuntu 22.04.
  • Experimental setup: Transition tolerances were selected to produce 100% success in the demonstrated geometric situations, so tracking accuracy did not limit sensitive segments such as peg insertion or small-object grasping.The tolerances were chosen using parameters from prior work.
  • Tasks: The evaluated tasks were cabinet opening, bolt screwing, and surface following, divided into sustained-contact subtasks separated by free-space reaching and grasping.Static waiting segments were excluded because they did not limit reproduction success.
  • Cabinet opening: The barrel-latch task remained limited by a 5.6 deg latch offset because expected and actual bolt axes were misaligned, blocking the handle from arching upward.The same failure occurred with pose and hybrid controllers.
  • Cabinet opening: Online adaptation completed the hinge-latch task at an 11.3 deg offset, while 14.1 deg caused a grasping collision and required offline generalization for larger offsets.Without adaptation, later trajectories reproduced the demonstrated upward handle arch and became blocked.

D. Bolt Screwing

Bolt picking and screwing, dislodging, and surface contouring were evaluated as contact-rich subtasks under geometric variation. Hybrid control, contact-aware transitions, and trajectory adaptation progressively addressed failures caused by lost contact, transition tolerances, friction, and changed contact geometry.

  • Bolt picking: Bolt picking with pose control failed under a 5 deg plane offset because the earlier rotation contact forced mechanical interference and violated transition tolerances.Hybrid control met translation tolerances but still failed rotation tolerances until contact-aware transitions relaxed them.
  • Bolt picking: Online trajectory adaptation completed bolt picking at 10 deg plane offset by using the measured pose to avoid mechanical interference; online or offline adaptation succeeded from 10–15 deg.Adaptation was applied at the failing transition from constrained grasped-bolt motion to free space.
  • Hole search and screwing: Pose control could not maintain contact for hole catching and screwing, whereas hybrid control succeeded without geometric variation by maintaining contact with force control.When the plane was lowered by 16 mm, the wrench controller established contact after pose-controlled approach halted above the demonstrated contact point.
  • Hole search and screwing: With a hole displaced on a 20 mm-radius circle, linear extrapolation found 2 of 10 holes, and online adaptation completed those cases for 20% success.Finding the remaining holes required more elaborate exploration than linear extrapolation.
  • Hole search and screwing: If all ten hole positions were known beforehand, offline generalization adapted trajectories to find every hole, producing 100% success.The prior geometric information could come from a vision system.
  • Dislodging: Contact-aware switching let dislodging succeed under 10–20 mm clamp offsets by automatically increasing pulling wrenches to overcome increased maximum static friction.Without contact-aware switching, 5 mm offsets caused transition failures despite successful dislodging.
  • Surface contouring: During surface contouring, hybrid control maintained contact but achieved 50% success because edge loss did not always meet perpendicular tracking tolerances.Contact-aware switching reached 66% success at 10 mm offset, and online adaptation maintained all prong contacts.

VII. DISCUSSION

The discussion argues that explicit physical-interaction modeling enables interpretable one-shot reproduction of complex contact-rich tasks while supporting robustness, generalization, and online adaptation. It also identifies threshold-based control, demonstration count, dynamics assumptions, and controller speed as important limitations or trade-offs.

  • One-Shot LfD of Complex Manipulation Tasks: Explicit interaction modeling enabled complex, contact-rich, sequential tasks to be reproduced from one demonstration without task-specific prior knowledge.The approach uses conventional hybrid pose/wrench control and physically meaningful parameters.
  • Robustness to Unknown Geometric Variations: Hybrid control in task-relevant frames improved robustness by keeping interaction wrenches low and guaranteeing contact where needed.Task-relevant frames decouple signals and control goals, simplifying classification and monitoring.
  • Robustness to Unknown Geometric Variations: Monitoring tracking and contact transitions together increased robustness by combining prescribed accuracy with haptic checkpoints grounded in the environment.Contact monitoring complements tracking, including when pose estimates such as vision are noisy.
  • Robustness to Unknown Geometric Variations: Explicit monitoring of unilateral contact transitions supports exploration when expected transitions occur early or fail to occur.Pose extrapolation was sufficient for many cases, while hole search and some wrench exploration required more specialized methods.
  • Adaptivity to Online Discovered Geometric Variations: Contact changes discovered during exploration reveal unknown pose offsets that can adapt task models for future reproduction.Online adaptation uses newly measured pose and contact information, although a single pose sample provides limited accuracy.
  • Generalization using a Priori Known Geometric Variations: Known geometric variations can be generalized using constant segment offsets and blended trajectories, while interaction frames identify task-relevant visual features.Examples include hinge, latch, door, screw-tip, and screwing-hole axes.
  • Limitations and Future Work: The method maintains interpretability through discrete interaction classes, prespecified thresholds, and switching between position and force control.Continuous probabilistic monitoring and smoothly varying controller impedances are proposed as future extensions because thresholds can be sensitive.
  • Limitations and Future Work: Multiple demonstrations could reduce user-error effects, reveal alternative robust execution paths, and help tune tracking tolerances or controller stiffness.The paper therefore treats one-shot learning as feasible but not universally preferable.

APPENDIX

The appendix filters threshold crossings to segment demonstrations into meaningful interaction phases. It combines integral thresholds, directional-crossing cleanup, short-segment removal, and temporal pooling, illustrated by a four-phase example.

  • Segmentation Filter: The segmentation filter uses minimum integral criteria to reject threshold crossings caused by small variations or noise.The appendix retains crossings only when signal magnitude between crossings is sufficiently relevant.
  • Segmentation Filter: Same-direction crossing pairs are reduced by removing one crossing, combining ambiguous segments with an adjacent segment.The removal rule depends on crossing direction and the sign of the intervening integral.
  • Segmentation Filter: Up-down and down-up crossing pairs are removed when their intervening absolute integral is below the minimum, eliminating short spike-induced segments.This suppresses momentary excursions that would otherwise create extra segments.
  • Segmentation Filter: Separate threshold crossings within the dead-time window are pooled and averaged in time.Pooling is the final step after crossing filtering and ambiguous-segment removal.
  • Segmentation Example: A simple task is segmented into idle waiting, free-space motion until contact, contact, and contact-force relief.These four phases are illustrated in Fig. 9.
Loading 2608.24741v1…