PaperlayerView exact revision on arXiv

Source-linked AI summary

SAKI: Skill Assembly and Kinematic Imitation from Human Videos for Long-Horizon Mobile Manipulation

Yijie Lu, James Zhao, Weiming Zhi

arXiv:2609.36031v1cs.RO

TL;DR

Human-video manipulation skills must transfer across changing scenes and robot configurations, then connect into long-horizon mobile tasks. SAKI prepares and assembles object-centric skills and uses closed-loop whole-body execution; real-robot trials demonstrate local-task gains and continuous task completion.

Abstract

from arXiv · show

Learning from human videos offers a promising route to acquiring diverse manipulation skills. Extending this capability beyond tabletop settings to long-horizon mobile manipulation requires adapting and composing demonstrated interactions across changing scenes and robot configurations. We present Skill Assembly and Kinematic Imitation (SAKI), a framework connecting human-video skill acquisition, cross-demonstration assembly and closed-loop whole-body execution. SAKI prepares reusable object-centric skills that preserve task-critical interactions while allowing transfer paths to adapt. Given a goal and supplied task dependencies, it selects and orders skills, binds their object roles to the current scene, and carries scene estimates and robot configuration between successive skills. Whole-body kinematic imitation generates coordinated base, arm and gripper motion. During execution, persistent object estimates maintain task references across viewpoint changes, while visual feedback updates remaining trajectories. Real-robot experiments demonstrate skill reuse across layouts and the composition of independently demonstrated interactions into continuous mobile tasks, including tidying and wiping. Ablation results show that task-conditioned reference preparation substantially improves long-horizon task completion with whole-body optimisation and visual feedback held fixed. Check https://aus.bot/research/saki/ for video demos!

I. INTRODUCTION

SAKI addresses the challenge of reusing human-demonstrated interactions across changing layouts, viewpoints, and robot configurations. It separates task-critical interaction requirements from adaptable robot motion, then grounds and connects skills during execution.

I. INTRODUCTION

  • Human-video trajectories mix essential interaction geometry with demonstrator- and scene-specific motion, while mobile actions change the scene and robot state for subsequent interactions.A destination may need to remain available after turning takes it out of view.Source: §I · p. 1
  • SAKI preserves demonstrated interaction requirements while regenerating robot motion for the current scene through object-role binding and whole-body optimisation.Assets retain contact, clearance, orientation, and terminal relations while allowing transfer motion to change.Source: §I · p. 1
  • Cross-view execution uses retained object estimates and accepted visual updates to revise remaining targets while replacement plans preserve committed motion.The preceding terminal robot configuration and retained scene estimates initialise each successor approach.Source: §I · pp. 1–2
  • Given supplied goals and dependencies, SAKI selects, orders, repeats, and binds prepared skills, carrying scene estimates and robot configuration into successor motion.This connects independently demonstrated interactions into continuous mobile tasks.Source: §I · p. 2

II. RELATED WORK

Prior work addresses object-centric imitation, mobile manipulation, and demonstration reuse through varied representations and planning strategies. SAKI connects object motion to phase-specific interaction requirements and composes independently demonstrated interactions using scene bindings and successor state.

II. RELATED WORK

  • Object-centric methods transfer object-relative motion, recover object graphs, or represent task geometry with trajectories, alignments, and spatial constraints.SAKI couples recovered object motion to phase-specific requirements that determine which segments can change while retaining the interaction.Source: §II · p. 2
  • Mobile-manipulation work includes whole-body learning, base-and-arm trajectory optimisation, continuous planning, and systems that maintain scene information.SAKI retains object estimates across views and uses accepted visual updates to revise targets and replan to the execution boundary.Source: §II · p. 2
  • Demonstration reuse and skill-composition methods adapt trajectories, retime segments, connect contact-rich motions, or compose learned skills.SAKI connects independently demonstrated interactions through object-role bindings, with retained scene estimates and the preceding terminal robot configuration initialising each successor.Source: §II · pp. 2–3

III. METHOD

SAKI prepares object-centric interaction references from human videos, binds their roles to scene entities, and regenerates targets and robot motion for execution. Skill assets preserve phase-specific interaction requirements while marking motion that can adapt.

III. METHOD

  • SAKI prepares reusable skills from human demonstrations, binds them to scene objects, and generates whole-body motion; visual updates revise remaining motion and the preceding terminal configuration initialises each successor.Scene estimates, measured robot state, and execution context track object geometry, the robot configuration, and plan progress and committed motion.Source: §III · p. 3

A. Video-Derived Skill Assets

  • An object-task asset packages an accepted object-motion reference, object roles, ordered phases, contact events, and adaptation rules.An invocation binds the roles to scene entities while retaining the specification and regenerating targets and robot realisation.Source: A. Video-Derived Skill Assets · p. 3
  • Stereo depth, tracked masks, calibration, and camera poses recover timestamped object clouds; the reference retains observed positions and supported orientation components.Unobservable rotations remain unspecified, and inferred connections across missing observations are distinguished from measured motion.Source: A. Video-Derived Skill Assets · p. 3
  • A preparation agent proposes roles, phases, contact events, and permitted reference changes; deterministic tools generate and check candidate references for agent review.Protected intervals retain requirements such as lift clearance, pouring orientation, or wiping contact, while adjustable segments use control points and interpolation.Source: A. Video-Derived Skill Assets · p. 3
  • Accepted candidates preserve protected source samples and timing, use linear or PCHIP interpolation on adjustable positions, and require an assessment for every declared segment.The phase record links source intervals, motion rules, and contact events to reference samples, preserved or interpolated positions, and attachment or release.Source: A. Video-Derived Skill Assets · p. 3
  • For can disposal, a 24-frame reference separates lift, adaptable transfer, and final alignment, with gripper opening scheduled after alignment.The lift endpoint and final-interaction boundary define the transfer adaptation interval.Source: A. Video-Derived Skill Assets · p. 3

B. Skill Assembly and Scene Grounding

SAKI assembles skills by binding their roles to scene entities and generating targets and connecting approaches from retained references. Transfer correction adapts the path between contact and release anchors while preserving protected interaction segments.

B. Skill Assembly and Scene Grounding

  • Given a goal and supplied dependencies, the agent selects and orders existing skills, binds their roles to scene entities, and specifies terminal gripper intent.Repeated invocations can use new bindings; compatibility preserves attachment, support, and release events.Source: B. Skill Assembly and Scene Grounding · pp. 3–4
  • Calibrated observations and capture-time odometry register entities in a common frame, while retained estimates anchor targets even when an entity leaves the current image.A base scan can acquire entities across viewpoints when needed.Source: B. Skill Assembly and Scene Grounding · p. 4
  • The compiler regenerates object targets, robot motion, and timing for the current scene while keeping references and phase requirements fixed across invocations.Approaches use the source object and prepared grasp, final alignment uses the destination, and transfer connects them.Source: B. Skill Assembly and Scene Grounding · p. 4
  • Three independently prepared assets—can disposal, tape placement, and board wiping—are assembled in order with scene bindings and terminal events.Container openings determine release targets, while the board frame determines tool orientation and sweeping motion.Source: B. Skill Assembly and Scene Grounding · p. 4, Table I · p. 4
  • Transfer-path correction aligns the accepted object-motion curve to current contact and release anchors, with correction zero through lift and constant during final interaction.The mapping assumes nondegenerate horizontal separation and an admissible transfer interval; initial binding can absorb changed destination height before terminal lowering.Source: B. Skill Assembly and Scene Grounding · p. 4

C. Whole-Body Kinematic Imitation

Whole-body optimisation converts grounded object targets into coordinated base and arm motion while accounting for interaction feasibility and execution boundaries. Phase-specific pose targets and contact schedules cover articulated, bimanual, pouring and wiping interactions.

C. Whole-Body Kinematic Imitation

  • Whole-body optimisation evaluates candidate approaches against the complete attached trajectory, starting from measured robot state and joining replacement plans at the execution boundary.Stored solutions provide numerical initialisation.Source: C. Whole-Body Kinematic Imitation · p. 4
  • The objective matches grounded object targets while regularising motion, with rigid-transport residuals built from end-effector kinematics, attachment transforms and object-frame points.Smoothness, base motion, posture and clearance costs are combined with joint bounds, base-motion constraints and fixed execution boundaries.Source: C. Whole-Body Kinematic Imitation · p. 4
  • Bimanual packing places both arms on a common time base so the supporting arm retains lid contact throughout the other arm’s insertion and release.The arm target combines scene registration, object motion relative to a reference entity and the prepared grasp.Source: C. Whole-Body Kinematic Imitation · p. 4
  • Pouring targets change cup orientation while retaining its grasp, whereas wiping targets preserve tool contact and alignment as strokes trace the board.The gripper remains closed through wiping, and engagement and withdrawal adjust separation from the board.Source: C. Whole-Body Kinematic Imitation · p. 5
  • Adjustable transfer and optional surface-view costs can change approach and camera heading, creating opportunities for closer or less-occluded observations.Accepted observations update persistent object state and can revise future targets.Source: C. Whole-Body Kinematic Imitation · p. 5

D. Stateful Closed-Loop Execution

Within an invocation, accepted observations revise unexecuted targets while preserving the committed motion prefix. At skill switches, the sequence retargeter carries scene estimates and robot configuration forward to initialise the next approach.

D. Stateful Closed-Loop Execution

  • Consistent observations update object pose, while occlusion or ambiguous identity preserves retained estimates and attached objects propagate through robot kinematics.Accepted revisions change unexecuted targets while participant identities and the prepared reference remain fixed.Source: D. Stateful Closed-Loop Execution · p. 5
  • A permitted solve consumes the accepted scene snapshot and continuation boundary, then regenerates and retimes remaining base, arm and gripper motion while retaining committed commands.Candidate promotion records the accepted executable, and subsequent planning carries it as the source plan.Source: D. Stateful Closed-Loop Execution · p. 5
  • Each sequence approach starts from the preceding terminal base pose, arm joints and gripper state, with a revised endpoint rebuilding the immediately following approach.L1’s tape approach starts from the state left by can disposal, and persistent estimates retain scene context across viewpoint changes.Source: D. Stateful Closed-Loop Execution · p. 5

IV. EXPERIMENTS

The evaluation tests interaction reuse through continuous-task composition, reference-preparation ablation and changed-layout trials with frozen skills. It also examines a recorded visual update and compares local manipulation reference-generation pipelines under shared downstream conditions.

IV. EXPERIMENTS

  • Continuous tasks test composition of interaction specifications, while changed-layout trials test reuse with frozen skills.Reference ablation tests preparation’s benefit.Source: §IV · p. 5
  • A recorded visual update traces the runtime connection from object estimates to activated motion.Source: §IV · p. 5
  • Local manipulation comparisons evaluate reference-generation pipelines under shared downstream conditions.Source: §IV · p. 5

A. Setup and Shared Protocol

Across the physical suite, SAKI completes most single-task trials and over two-thirds of continuous-sequence trials, with extended interactions and later sequence stages remaining challenging. The protocol evaluates full outcomes across single tasks and continuous sequences, alongside reference preparation and interaction demands.

A. Setup and Shared Protocol

  • The platform uses a Ranger Mini V3 with two Nero arms, and skills are prepared from calibrated stereo human demonstrations and task context.Seven single tasks and three continuous sequences each have 15 initiated trials; a separate local suite tests fixed-base-reach operations.Source: A. Setup and Shared Protocol · p. 5
  • Sequence success requires every evaluated stage to complete without intermediate resets or human assistance; full-task rates include all 15 initiated attempts.Stage rates use only trials reaching that stage, and wiping is judged visually by a human reviewer rather than by pixel-area measurement.Source: A. Setup and Shared Protocol · p. 5

B. Complete-Pipeline Capability

  • 85/105 single-task trials (81.0%) and 31/45 continuous-sequence trials (68.9%) complete; door opening, long wiping and box opening remain challenges.Table II reports complete physical outcomes across seven single tasks and three continuous sequences, with 15 initiated trials per condition.Source: B. Complete-Pipeline Capability · p. 6, Table II · p. 6
  • All 15 L1 attempts complete can disposal and tape placement, while nine also complete wiping, concentrating its six full-sequence losses at the final stage.L1 tests assembly of independently prepared assets as one mobile sequence.Source: B. Complete-Pipeline Capability · p. 6
  • L2 stage rates are 14/15 and 13/14; L3 achieves 13/15, 11/13 and 9/11 at locations A, B and C.L3’s first stage includes basket acquisition, can acquisition and placement.Source: B. Complete-Pipeline Capability · p. 6
  • The interaction cases require sustained door-handle contact, overlapping lid support and placement, changing cup orientation, and aligned wiping contact.These demands are represented through pose targets and contact schedules beyond object-centre displacement.Source: B. Complete-Pipeline Capability · p. 6, Figure 5 · p. 6, Figure 6 · p. 6
  • Reference preparation improves continuous-task completion with scene grounding, whole-body solving and online updates retained.Source: Table III · p. 6

C. Reference Optimisation on Continuous Sequences

Task-conditioned reference preparation improves long-horizon completion while whole-body solving and visual feedback remain active. Direct interpolation can constrain motion adjustments useful for reach and target observation.

C. Reference Optimisation on Continuous Sequences

  • 31/45 (68.9%) versus 7/45 (15.6%) completion; reference preparation improves every sequence by 53.3 percentage points.The comparison replaces task-conditioned reference refinement with direct interpolation while retaining scene binding, whole-body optimisation, visual updates and replanning.Source: C. Reference Optimisation on… · p. 6
  • For L1, completion was 9/15 with preparation versus 2/15 with direct interpolation, with the same downstream mechanisms active.The interpolated object path can restrict base rotation and posture changes that improve reach or target observation; failed trials included poor target observation and collisions with task objects.Source: C. Reference Optimisation on… · p. 6
  • Allowing transfer motion to change gives the downstream solve freedom to accommodate observation and motion feasibility while preserving the interaction.The completion comparison measures the resulting benefit across the full sequence.Source: C. Reference Optimisation on… · p. 6

D. Visual Updates in Recorded Execution

SAKI updates remaining motion from visual feedback while preserving committed execution, and its local comparison reports higher completion than DITTO-adapted across three operations. The results also distinguish local interaction tests from surrounding mobile tasks.

D. Visual Updates in Recorded Execution

  • Pouring requires time-varying cup orientation, while wiping requires sustained tool alignment.These phase-specific pose requirements govern physical interaction.Source: Figure 7 · p. 7
  • A 7.12 cm bin-estimate shift revised remaining targets and changed subsequent base and arm motion and timing, while retaining executable points through the continuation boundary.Robot reports confirm that the replacement plan was activated and used as the source plan for the next planning request.Source: D. Visual Updates in Recorded Execution · pp. 6–7, Figure 8 · p. 7

E. Comparison on Shared Local Tasks

  • 45/45 (100.0%) completions versus 31/45 (68.9%) for DITTO-adapted across placement, pouring and wiping.The methods share perception, grasping and execution modules, with method-specific reference generation and identical local outcome definitions.Source: Table IV · p. 7, E. Comparison on Shared Local Tasks · p. 7
  • The larger pouring and wiping gaps are associated with orientation change or sustained contact; complete tasks additionally test long wiping and basket handling around placement.The two suites assess local interaction quality and completion of the surrounding mobile task.Source: E. Comparison on Shared Local Tasks · p. 7

F. Position Generalisation and Skill Reuse

Frozen prepared skills are reused across changed object positions and robot starting configurations, with trials completing multiple interactions. The conclusion frames scene estimates and execution boundaries as inputs to motion and points to richer task reasoning as future work.

F. Position Generalisation and Skill Reuse

  • Reuse trials changed source and destination positions and robot starts; pouring varied cup and tray positions and robot starting configuration while keeping the table fixed.Bindings, targets and plans were regenerated for the changed conditions.Source: F. Position Generalisation and Skill… · p. 7
  • Across twenty trials, can disposal completed 18/20 (90.0%), and lid opening with placement completed 15/20 (75.0%); with pouring, results establish position reuse across three interactions while prepared specifications stayed fixed.Source: F. Position Generalisation and Skill… · p. 8

V. CONCLUSION AND FUTURE WORK

  • The conclusion identifies more complex task dependencies, richer scene information and execution feedback as future directions for broader task-level reasoning and adaptation.It describes the demonstrated framework as separating reusable interaction specifications from trajectories that ground, update and connect them.Source: §V · p. 8

Related papers

Loading 2609.36031v1…