PaperlayerView exact revision on arXiv

Source-linked AI summary

Kinematic Nonlinear Spatio-Temporal Trajectory Warping for Contact-Rich Dexterous Manipulation Demonstrations

Hyojae Park, Arjun S. Lakshmipathy, Nancy S. Pollard

arXiv:2609.36676v1cs.RO

TL;DR

Reconstructing dexterous manipulation demonstrations is labor-intensive, and existing trajectory warping must handle spatial and temporal constraints without breaking hand–object coupling. This paper warps object trajectories and recovers hand motion through contact-based inverse kinematics; the reported warped trajectories preserve the hand–object relationship to within 1 mm on average and generalize across end-effectors.

  • Problem

    High-quality dexterous demonstrations are time-consuming to reconstruct, while nonlinear warping must satisfy spatial and temporal constraints without disrupting hand–object coupling.

    Source: §I · p. 1

  • Method

    The method warps object trajectories to waypoint, barrier, and timing constraints, then uses per-frame contact correspondences in inverse kinematics to recover hand trajectories.

    Source: §III · p. 2, D. Hand Warping · p. 4

  • Results

    Warped trajectories preserve the hand–object relationship to within 1 mm on average and generalize across different end-effectors, including a parallel-jaw gripper and the Allegro hand.

    Source: 1) Hand-barrier collisions · p. 7

  • Takeaways & Limitations

    The method is positioned as a practical upstream component for generating spatially and temporally diverse training data in demonstration-augmentation pipelines.

    Source: 1) Hand-barrier collisions · p. 7

    Evaluation is purely in simulation and in isolation, and the effectiveness of these warps in sim2real deployment pipelines remains unclear.

    Source: B. Limitations · p. 6

Abstract

from arXiv · show

We present a straightforward but effective method for repurposing existing contact-rich dexterous manipulation demonstrations. Starting from inputs of hand and object trajectories, our method outputs high-quality nonlinear trajectory warps that account for intermediate waypoints, environmental barriers, temporal shifts, and varied start/end configurations. Foundational to our method is the utilization of contact distributions, which we show allows us to reliably compute complex and high-dimensional dexterous hand trajectories following a simple object-centric warp specification pipeline. We evaluate our method across 12 variations sourced from 4 demonstrations in a publicly available dataset of human hand motion data, perform baseline comparisons, and demonstrate generalization of our approach to different manipulators. Results and code will be made available on publication.

I. INTRODUCTION

Dexterous manipulation demonstrations are costly to reconstruct, while adapting them requires coordinated nonlinear changes to hand and object motion. The paper proposes object-centric trajectory warping and evaluates it across varied constraints and manipulators.

I. INTRODUCTION

  • Reconstructing high-quality dexterous demonstrations is labor-intensive, and no simple off-the-shelf solver handles nonlinear spatial and temporal changes while preserving hand–object coupling and avoiding artifacts.Breaking that coupling can cause motion misalignment, penetrations, or loss of contact.Source: §I · p. 1
  • The method generates coupled hand and object trajectory warps for barriers, spatial and temporal waypoints, and changed start/end configurations.Its object-centric pipeline warps the object first, then recovers hand poses using per-frame contact distributions.Source: §I · p. 1, Figure 1 · p. 1
  • Across 12 variations from four public-dataset demonstrations, the authors report baseline comparisons and generalization to different manipulators.They describe CPU runtimes of minutes and a strategy with few parameters beyond input specifications.Source: §I · p. 1

A. Data Augmentation

  • Trajectory-altering augmentation methods such as MimicGen use rigid linear SE(3) transformations, limiting intermediate-path, timing, and obstacle changes; this method targets nonlinear warps instead.Other augmentation approaches described here replay trajectories or leave them to be learned rather than explicitly altering them.Source: A. Data Augmentation · p. 2

B. Contact-Driven Grasp and Manipulation Synthesis

The paper situates contact distributions within manipulation synthesis and distinguishes trajectory warping from motion planning. Its method refines existing demonstrations through an object-centric sequence of spatial, barrier, temporal, and hand-trajectory transformations.

B. Contact-Driven Grasp and Manipulation Synthesis

  • Contact information serves as an optimization loss, generative-model constraint, retargeting prior, or planning primitive; this method uses it to maintain coupling during non-trivial spatio-temporal warps.Human motion and interaction datasets increasingly include detailed contact information.Source: B. Contact-Driven Grasp and… · p. 2

C. Motion Planning

  • Motion planning constructs trajectories from scratch or sparse initial conditions, whereas trajectory warping refines existing data while attempting to preserve its movement characteristics.Warping adapts trajectories to new spatio-temporal constraints; planning faces difficulty producing nuanced movements such as smooth, natural motion.Source: C. Motion Planning · p. 2

III. METHOD

  • The pipeline spatially warps the object through position and orientation waypoints, avoids barrier meshes, retimes the path, then recovers the hand trajectory by contact-based IK with barrier constraints.A waypoint specifies position, rotation, and original/output timesteps; start and end positions are implicit first and last waypoints when altered.Source: §III · pp. 2–3

A. Spatial Waypoint-Constrained Object Warping

Object warping first fits waypoint positions and rotations, then resolves barrier intersections and retimes the resulting trajectory. The three stages use blended spatial shifts, conservative obstacle inflation, and spline-based temporal mapping.

A. Spatial Waypoint-Constrained Object Warping

  • Waypoints partition the in-contact trajectory into segments that are shifted between endpoint displacements; adaptive quadratic/linear blending balances preserving the original shape with a more direct path.A cumulative rotation is imposed at each waypoint using SLERP; temporal retiming is deferred until barrier constraints are resolved.Source: A. Spatial Waypoint-Constrained Object… · p. 3

B. Barrier-Constrained Object Warping

  • For object-barrier avoidance, the method inflates each barrier’s convex hull using a bounding-ball approximation, then projects trajectory points outside the inflated hull.The conservative overapproximation may exclude some valid configurations near barrier surfaces, but is described as computationally inexpensive and sufficient for resolving intersections.Source: B. Barrier-Constrained Object Warping · p. 3

C. Temporal Waypoint-Constrained Object Warping

  • Temporal retiming composes per-segment chord-length B-spline timewarps, then fits a global cubic B-spline to satisfy positional and temporal constraints with C2-continuous motion.The retimed contact segment is uniformly sampled and combined with unchanged pre- and post-contact segments.Source: C. Temporal Waypoint-Constrained… · p. 3

D. Hand Warping

After object constraints are resolved, hand recovery uses per-frame inverse kinematics driven by contact correspondences. The optimization also penalizes hand–barrier intersections, followed by light smoothing.

D. Hand Warping

  • Under non-uniform timewarps, rigid-body transforms do not reliably recover hand motion, so the method retrieves the contact distribution from the last input timestep mapped before each output timestep.The paper says this retrieval is needed because proven interpolation methods for arbitrary contact distributions are lacking.Source: D. Hand Warping · p. 3
  • Per output timestep, hand configuration is optimized to reduce contact-correspondence distances while penalizing hand–barrier intersections.The contact term measures L2 distance for each corresponding hand/object point pair; the barrier term penalizes intersections.Source: D. Hand Warping · p. 4
  • The barrier penalty encourages clearance margin m and rises sharply with penetration; light translation and rotation smoothing reduces adjacent-frame jitter without substantial drift from the IK solution.Experiments set n = 2.0 and m = 0.05 m; complex barrier meshes are approximated by collision geometries for efficient SDF evaluation.Source: D. Hand Warping · p. 4

IV. RESULTS

The evaluation uses 12 warping configurations derived from four GRAB demonstrations and optimizes hand trajectories after satisfying object-trajectory waypoint and barrier constraints.

IV. RESULTS

  • 12 warping configurations of varying difficulty were generated from four GRAB demonstrations, with kitchen-like actions such as pouring into two cups.The source demonstrations came from the GRAB dataset, and the background kitchen scene came from RoboCasa [45].Source: §IV · p. 4
  • For each configuration, the object trajectory was first warped to meet waypoint and barrier constraints, then the hand trajectory was optimized at 10 mm, 5 mm, and 3.5 mm thresholds.Per-timestep optimization used Adam at learning rate 0.01 until mean contact distance fell below the threshold; each timestep started from the prior timestep’s solution.Source: §IV · p. 4
  • All experiments ran exclusively on an Apple M2 SoC with 16 GB of memory, using the CPU.Source: §IV · p. 4

A. Qualitative evaluation

The qualitative evaluation specifies its trial configurations and reports successful adaptation across varied constraint setups in the supplementary animations.

A. Qualitative evaluation

B. Quantitative evaluation

The quantitative evaluation measures object-space preservation, contact fidelity, runtime, path-length ratio, timewarp discrepancy, and contact-correspondence robustness.

B. Quantitative evaluation

  • DW O measures mean wrist-position distance from the object center, while DC measures mean distance between corresponding hand-object contacts.Lower values indicate better performance in the tables.Source: B. Quantitative evaluation · p. 4
  • At the 10 mm, 5 mm, and 3.5 mm thresholds, DW O differs from its initial value by 5.5 mm, 2.2 mm, and 1 mm, respectively.DW O measures mean wrist-to-object-center distance; the results indicate that hand-object relative positioning is maintained under substantial deformations.Source: B. Quantitative evaluation · p. 4, Table II · p. 4
  • At the 3.5 mm threshold, DC deviates from its initial value by 1.4 mm on average and is 1.0 mm lower than the initial value on average.Post-hoc Savitzky–Golay smoothing shifts DC by only −0.62 mm on average.Source: B. Quantitative evaluation · pp. 4–5, Table III · p. 5
  • The most difficult 3.5 mm threshold takes about 4.6 minutes on average on a CPU-only architecture.Table IV reports runtime across optimization thresholds, with non-timestep values in seconds.Source: B. Quantitative evaluation · p. 5, Table IV · p. 5
  • PLR and timewarp discrepancy are reported across all trajectories and optimization thresholds.PLR is the warped-to-original path-length ratio; timewarp discrepancy measures deviation from uniform timing.Source: B. Quantitative evaluation · p. 4, Table V · p. 5
  • Both contact-corruption curves maintain correspondence quality up to 50%, with DC near 5.0 mm.The figure tests contact-correspondence robustness in the teapot-pour-cups wall scene.Source: Figure 4 · p. 5

C. Generalization

The method is presented as applicable beyond human hands, including parallel-jaw grippers, and a Franka example demonstrates waypoint-constrained box movement.

C. Generalization

  • The method is not specific to human hands and can be used with parallel-jaw grippers and multi-fingered robot platforms.Source: C. Generalization · p. 5

1) Franka arm with parallel grippers:

2) Allegro hand:

Contact-based warping better preserves grasps than interpolation-based baseline warping, particularly during temporal stretching. On the Allegro hand, substantial warping produces a non-uniform time redistribution while changing DW O only marginally.

2) Allegro hand:

  • On Allegro, DW O changes only marginally, from 0.133 m to 0.137 m, despite substantial warping.The warped apple trajectory passes through waypoints and terminates on an elevated plate.Source: 2) Allegro hand · pp. 5–6, Figure 6 · p. 6

D. Baseline comparison

  • The baseline extends rigid transforms at mapped input timesteps, then uses a timewarp-parameterized B-spline to interpolate hand poses at other timesteps.The ground-truth hand–object relationship is available only at input timesteps mapped directly to output timesteps.Source: D. Baseline comparison · p. 6
  • Between timesteps 529 and 750, temporal stretching forces the baseline to interpolate, and its hand trajectory diverges from the object’s path.The figure marks the in-contact region and maps input timesteps to output timesteps.Source: D. Baseline comparison · p. 6
  • At interpolated timesteps, the baseline can intersect the object or lose contact, while contact-based warping preserves the hand–object relationship throughout the trajectory.The baseline maintains a proper grasp at timesteps with ground-truth relationships, but fails at interpolated points.Source: D. Baseline comparison · p. 6

V. DISCUSSION

Contact-based warping is particularly useful for substantial spatio-temporal deformations, precise-contact tasks, and cluttered scenes. Its current scope is constrained by object and barrier assumptions, contact correspondence reuse, and simulation-only evaluation.

A. Advantages and Disadvantages

  • Contact-based warping is especially suited to substantial spatio-temporal deformations, precise-contact tasks, and cluttered scenes with obstacles.Mild warps without temporal shifts or barriers may be handled by faster techniques.Source: A. Advantages and Disadvantages · p. 6

B. Limitations

  • The method assumes rigid objects and convex barriers; concave barriers may be approximated too conservatively.Articulated or deformable objects are unsupported, and tracking their components during both warps would require significant computation.Source: B. Limitations · p. 6
  • Evaluation is limited to simulation and isolated settings, leaving the method’s effectiveness in sim-to-real deployment pipelines unclear.The authors identify large-scale domain randomization as future work.Source: B. Limitations · p. 6
  • Because contact correspondences are reused rather than interpolated between timesteps, rapid finger movements or fine-grained contact changes may not be captured.The authors identify contact interpolation as a possible direction for addressing this limitation.Source: B. Limitations · p. 6
  • The comparison figure shows the contact-based method maintaining a proper grasp across all visualized timesteps, while the baseline degrades at interpolated timesteps.The figure marks the in-contact segment and maps input to output timesteps.Source: Figure 7 · p. 7

C. Failure cases

The method can fail when an object’s path leaves insufficient room for the hand or when barriers form narrow gaps. The conclusion reports contact-preserving warps with average hand–object relationship error within 1 mm and generalization across end-effectors.

C. Failure cases

  • An object may clear a barrier while leaving too little room for the hand to maintain a grasp without collision.Increasing the object’s bounding ball can mitigate these cases by providing additional clearance.Source: C. Failure cases · p. 7

2) Narrow gaps:

  • When barriers form narrow gaps, iterative collision resolution can oscillate trajectory points without converging to a valid trajectory.Source: 2) Narrow gaps · p. 7

1) Hand-barrier collisions:

  • The conclusion frames the object-centric pipeline and contact-preserving hand warping as a modular, practical upstream component for demonstration augmentation.The pipeline uses per-timestep inverse-kinematics optimization for contact-preserving hand warping.Source: §VI · p. 7, 1) Hand-barrier collisions · p. 7

VI. CONCLUSION

  • The evaluation reports warped trajectories preserving the hand–object relationship to within 1 mm on average, with generalization to a parallel-jaw gripper and the Allegro hand.The method is described as computationally tractable on a CPU and a potential upstream component for generating spatially and temporally diverse training data.Source: 1) Hand-barrier collisions · p. 7

Related papers

Loading 2609.36676v1…