PaperlayerView exact revision on arXiv

Source-linked AI summary

Embodiment-Aware Planning for Heterogeneous Bimanual Manipulation

Sukhvansh Jain, Anurag Maurya, Shivam Vats, Narendra Kumar Dhar, Ravi Prakash

arXiv:2610.23551v1cs.RO

TL;DR

Complementary simple grippers can have phase-specific affordances, making some demonstrated manipulation tasks infeasible for either gripper alone. The paper plans grasp-conditioned arm assignments and handovers over the task horizon; exact dynamic programming achieves higher task success than a greedy baseline in simulation.

  • Problem

    Different task phases can require different grasp modalities, yet coordinating complementary grippers to execute tasks beyond either gripper’s capability requires systematic planning.

    Source: §I · pp. 1–2

  • Method

    The planner optimizes arm–trajectory assignments over a demonstrated object motion, jointly accounting for grasp feasibility, handover compatibility, and switching penalties.

    Source: §I · p. 2

  • Results

    Exact DP achieves 72.5% task success versus 20.0% for the greedy baseline across 520 simulated runs.

    Source: D. Quantitative Results · p. 7

  • Takeaways & Limitations

    The framework demonstrates globally planned heterogeneous-gripper execution on a dual Franka FR3 platform with suction and parallel-jaw grippers.

    Source: §I · p. 2, §VI · p. 8

    The framework relies on pre-computed CAD meshes for tracking and grasp generation, limiting zero-shot deployment on novel objects.

    Source: §VI · p. 8

Abstract

from arXiv · show

Many manipulation tasks that are effortless for dexterous human hands remain challenging for robotic systems equipped with simple, task-specialized grippers. Industrial platforms commonly rely on end-effectors such as suction cups or parallel-jaw grippers, yet no single mode can reliably satisfy the diverse constraints encountered across different phases of complex manipulation tasks, including pickup, reorientation, insertion, and constrained placement. In this work, we investigate whether complementary simple grippers within a heterogeneous bimanual setup can collectively realize tasks that exceed the capability of any individual gripper alone. Given a single human demonstration specifying the desired object motion, we formulate embodiment-aware execution as a finite-horizon optimization over arm-trajectory assignments. At each segment, the planner determines whether single-arm manipulation or a coordinated dual arm handover is required, jointly reasoning over heterogeneous grasp feasibility, motion constraints, manipulability, and cross-gripper compatibility. Exploiting this formulation, we compute globally consistent assignments with explicit switching regularization. Experiments on a dual Franka FR3 platform equipped with suction and parallel-jaw grippers demonstrate improved feasibility and reliability in tasks that are infeasible under single-gripper execution.

I. INTRODUCTION

Simple industrial grippers offer limited, different grasp affordances, so complex tasks may exceed any single gripper’s capability. The paper proposes coordinating complementary grippers to execute a demonstrated task through globally planned arm assignments and handovers.

I. INTRODUCTION

  • Suction cups and parallel-jaw grippers have complementary limits: suction struggles with confined insertions, while jaws can fail on thin or flush objects.The paper frames this as a capability gap relative to dexterous human hands, without requiring anthropomorphic hardware.Source: §I · p. 1
  • A thin plate cannot be picked and placed by either gripper alone, but a coordinated handover allows suction to complete placement.The parallel-jaw gripper performs the initial pick and transport; table collision constraints prevent it from completing placement.Source: §I · p. 2
  • The planner uses a single human demonstration as the desired object motion and selects an arm at each segment, including when handovers are needed.Its decisions account for grasp, kinematic, and collision constraints.Source: §I · p. 2
  • The paper’s contributions are heterogeneous gripper coordination, globally consistent embodiment-aware assignment, and a demonstration on dual Franka FR3 arms with suction and parallel-jaw grippers.The formulation addresses tasks infeasible for a single modality, including beyond simple pick-only tasks.Source: §I · p. 2

B. Bimanual Manipulation and Grasp Planning

Prior bimanual planning largely assumes identical end-effectors, while heterogeneous-gripper work often focuses on individual picks or high-level assignments. This paper jointly plans grasp-conditioned arm trajectories and transitions across a demonstrated task.

B. Bimanual Manipulation and Grasp Planning

  • Prior bimanual work covers coordinated transport, reorientation, grasp generation, and handovers, but much of it assumes identical end-effectors,,, [12],.More recent learning-based work includes coordinated dual-arm behaviors and manipulability-aware grasp planning [15],,,,.Source: B. Bimanual Manipulation and Grasp… · p. 2

C. Heterogeneous Gripper Systems

  • Existing heterogeneous-gripper work focuses on gripper selection for individual pick actions, while holistic task planning and feasibility across heterogeneous embodiments remain understudied.The paper contrasts its approach with methods that separate grasp selection from task planning or rely on heuristic or manual switching between arms.Source: C. Heterogeneous Gripper Systems · p. 3

III. PROBLEM FORMULATION

  • The task starts from a single human-provided object trajectory discretized into segments, with different segments potentially executable only by specific arms.The system has two arms with possibly different end-effectors, producing asymmetric grasp affordances.Source: §III · p. 3

IV. EMBODIMENT-AWARE PLANNING FRAMEWORK

  • For each segment, the framework evaluates feasible grasp-conditioned, collision-free arm trajectories and uses dynamic programming to select a minimum-cost sequence over the task horizon.Transitions encode switching penalties and grasp compatibility, yielding a feasible and smooth bimanual execution plan.Source: §IV · p. 3, §III · p. 3

A. Trajectory Extraction from Human Demonstration

The method extracts a smooth object trajectory from a human demonstration and constructs diverse grasp-conditioned end-effector trajectories. Its grasp generation is not restricted to a single trajectory source.

A. Trajectory Extraction from Human Demonstration

  • FoundationPose tracks object poses from RGB-D observations using a known CAD mesh, then filters the poses to produce smooth motion segments.Pose alignment uses observed depth and the rendered mesh; temporal filtering suppresses jitter while preserving rigid-body consistency under partial occlusion.Source: A. Trajectory Extraction from Human… · p. 3
  • The framework can use any method that supplies an object trajectory; the authors also test RRT* for trajectory generation.The CAD mesh is required for the described planning pipeline, while mesh reconstruction from multiple RGB-D images is noted as an alternative.Source: A. Trajectory Extraction from Human… · p. 3

B. Grasp Generation, Ranking and End-Effector Trajectory Formation

  • Grasp feasibility requires at least one robot to have a valid grasp at the object pose, with gripper-dependent affordances modeled jointly across both grippers and the object.The method uses geometry-based grasp generation and avoids scene-conditioned generation because a grasp infeasible in one segment may become feasible later.Source: B. Grasp Generation, Ranking and… · p. 3
  • DPP subset selection balances grasp quality and spatial-rotational diversity, and each selected grasp generates an end-effector trajectory sharing the object trajectory’s time parameterization.The subset is chosen greedily by maximizing log det(LS).Source: B. Grasp Generation, Ranking and… · p. 3

C. Segment-wise Feasibility and Cost Formulation

The segment formulation filters arm–trajectory actions by grasp and motion feasibility, then assigns costs for motion quality and feasibility. These segment costs support global planning alongside transition penalties and grasp compatibility.

C. Segment-wise Feasibility and Cost Formulation

  • Each trajectory segment is evaluated for feasible robot–trajectory pairs, and dynamic programming selects a globally optimal assignment using segment and transition costs.The transition costs encode switching penalties and grasp compatibility.Source: Figure 2 · p. 4
  • An action transports the object between consecutive poses using one arm, and is feasible only when its grasp and motion checks pass.Infeasible actions receive infinite cost in the combined task-cost formulation.Source: C. Segment-wise Feasibility and Cost… · p. 4, 3) Segment Cost · p. 4

1) Grasp Feasibility:

  • Grasp feasibility requires a candidate grasp at the pose and collision-free grasp pairs across robots for handovers.The candidate grasp set must be nonempty for the relevant gripper.Source: 1) Grasp Feasibility · p. 4

2) Motion Planning Feasibility:

  • Motion feasibility requires collision-free transport and feasible pick, release, and handover primitives; any infeasible primitive rejects the action.An analytical inverse-kinematics solver first filters endpoint reachability before required motion primitives are evaluated.Source: 2) Motion Planning Feasibility · p. 4

3) Segment Cost:

  • The task cost combines joint-motion smoothness with a Yoshikawa manipulability cost to discourage awkward or near-singular configurations.The smoothness term penalizes large joint variations between consecutive configurations.Source: 3) Segment Cost · p. 4
  • Manipulability cost evaluates both endpoint configurations of a transport segment, with a small positive constant used for numerical stability.The measure is the Yoshikawa manipulability index [28].Source: 3) Segment Cost · p. 4
  • The combined cost weights smoothness and manipulability by α and β, set to 40 and 0.5 after grid search.For redundant manipulators, manipulability is also included directly in inverse-kinematics optimization to favor well-conditioned configurations.Source: 3) Segment Cost · p. 4

D. Transition Design for Heterogeneous Handover

The transition design combines a three-stage handover primitive with feasibility constraints and a penalty for changing active robots.

D. Transition Design for Heterogeneous Handover

  • Handover uses pre-alignment, receiver approach, and transfer retreat to coordinate the two arms around predefined grasp transforms.The receiver aligns to an offset pose, approaches along the grasp normal, and takes the object before the original arm retreats.Source: D. Transition Design for Heterogeneous… · p. 4
  • Identical robot states cannot transition directly, and handover costs discourage unnecessary frequent switching when a grasp change is needed.A grasp change can instead be achieved through two handovers.Source: D. Transition Design for Heterogeneous… · pp. 4–5
  • Infeasible paired grasps are excluded, while feasible handover costs use the combined smoothness–manipulability cost.The total handover cost is defined analogously to the segment cost.Source: D. Transition Design for Heterogeneous… · p. 5
  • A positive switching penalty regularizes transitions between states with different active robots.The transition cost also accounts for the associated trajectory index and handover feasibility.Source: D. Transition Design for Heterogeneous… · p. 5

E. Dynamic Programming and Optimal Sequence Recovery

Dynamic programming finds a minimum-cost state sequence across task segments, then recovers the robot, trajectory, and switching sequences.

E. Dynamic Programming and Optimal Sequence Recovery

  • The value function represents the minimum cumulative cost of reaching segment i and terminating in state s.The method uses optimal substructure and a Bellman recursion over segments.Source: E. Dynamic Programming and Optimal… · p. 5
  • Infeasible transitions are excluded by assigning them infinite cost, yielding a shortest-path computation on a layered directed acyclic graph.Predecessor mappings are stored for policy recovery.Source: E. Dynamic Programming and Optimal… · p. 5
  • Each optimal state specifies a robot and its associated trajectory, from which the full robot and trajectory sequences are extracted.Robot changes between consecutive optimal states define the switching events.Source: E. Dynamic Programming and Optimal… · p. 5

F. Complexity Reduction via Beam Search and Stride

Beam search with stride reduces the dynamic-programming search burden by limiting handover evaluation and retaining only low-cost states.

F. Complexity Reduction via Beam Search and Stride

  • The dynamic-programming pass has worst-case time complexity O(N|S|2), motivating a beam-search approximation for longer horizons.The stated goal is to mitigate complexity and improve scalability.Source: F. Complexity Reduction via Beam… · p. 5
  • With stride k, potential dual-arm handovers are evaluated only every k discretized trajectory waypoints.Source: F. Complexity Reduction via Beam… · p. 5
  • At each forward-pass step, beam width B retains only the B lowest-cost states, pruning the state space and branching factor.The passage reports that this preserves performance while drastically reducing planning times.Source: F. Complexity Reduction via Beam… · p. 5

V. EXPERIMENTS AND RESULTS

The experiments examine the framework’s performance and components on a heterogeneous dual-arm platform, with setup choices designed to test grasp and reachability constraints.

V. EXPERIMENTS AND RESULTS

  • The evaluation asks whether dynamic programming improves task completion with heterogeneous grippers and how diversity-aware grasp selection and manipulability costs affect performance.It also examines crucial components and observed failure modes.Source: §V · p. 5

A. Experimental Setup

  • Experiments use two 7-DoF Franka FR3 arms, one with a parallel-jaw gripper and one with suction, with overlapping workspaces and varied relative configurations.The varied configurations induce reachability and feasibility constraints.Source: A. Experimental Setup · p. 5
  • The setup combines EAIK-based inverse kinematics, MuJoCo collision checks, and reactive mink QP control for the follower arm during simultaneous motion.Transport segments use constrained joint tracking to follow the object trajectory.Source: A. Experimental Setup · pp. 5–6
  • Grasp candidates are sampled for both gripper types and collision-filtered; Task A uses 60 grasps per type, while the others use 40.Source: A. Experimental Setup · p. 5
  • The four evaluated tasks are ordered left to right and designed to highlight heterogeneous bimanual advantages and coordination challenges.Source: Figure 3 · p. 6

B. Task Design

The tasks are designed to require heterogeneous gripper use through phase-specific feasibility constraints, including both single-switch and two-switch cases. The evaluation compares methods overall and reports task-specific performance for the proposed configuration.

B. Task Design

  • The task suite enforces at least one segment infeasible for a single arm and includes reachability-based constraints beyond collision constraints.It covers both single-switch and two-switch tasks.Source: B. Task Design · p. 6
  • Task A pairs a flat plate’s suction-required pickup with a shelf clearance that forces a handover for final placement..Source: B. Task Design · p. 6
  • Task B requires parallel-jaw pickup of a recessed plate, then a handover to suction for top-down placement around tight target boundaries.Source: B. Task Design · p. 6
  • Task C uses suction to pick up an inverted mug, then a parallel jaw to flip and place it upright in a container.Source: B. Task Design · p. 6
  • Task D requires two handovers because suction can access the containers but not the intermediate region behind a wall.The plate is constrained from passing over the wall.Source: B. Task Design · p. 6

C. Baselines

  • Table I compares overall method performance across five metrics, while Table II breaks down the proposed Beam 10, Stride 3 configuration by task.Source: Table I · p. 7, Table II · p. 7

1) Greedy Reachability:

The results show that global dynamic programming improves task success over greedy reachability, while grasp diversity and switching regularization matter to performance. Beam search reduces planning time, with outcomes varying across the four tasks.

1) Greedy Reachability:

  • Greedy reachability selects the feasible robot with higher instantaneous manipulability at each segment, equivalent to beam width 1.Source: 1) Greedy Reachability · p. 7

2) No-Diversity Grasp Selection:

D. Quantitative Results

  • 72.5% task success for exact DP compares with 20.0% for the greedy baseline, while the no-diversity variant reached 67.5% and took 1015.87 s to plan.Source: D. Quantitative Results · p. 7
  • 91.05 s average planning time for Beam 10, Stride 3 compares with 532.96 s for Exact DP, with 65.0% task success and 100.0% handover success.Source: D. Quantitative Results · p. 7

E. Per-Experiment Details and Evaluation

F. Ablation Studies

Ablations show that switching regularization and manipulability matter for reliable execution, and real-world validation reproduces digital-twin behavior on two tasks. The framework remains bounded by mesh and trajectory assumptions.

1) Switching Penalty and Frequency (Cswitch = 0):

  • Without the switching penalty, average switches rose from 1.3 to 10.7, task success fell to 25.0%, handover success to 81.6%, and execution time increased by 211.4%.Valid plans with two switches achieved 87.0% task success, compared with 90.3% for one-switch plans.Source: 1) Switching Penalty and Frequency… · p. 7

2) Manipulability Cost (β = 0):

  • Without the manipulability term, task success was 52.5% and handover success was 89.1%, with poorly conditioned states producing jerky trajectories and unstable grasps.The passage identifies prioritizing manipulability as vital for smooth, safe physical handovers.Source: 2) Manipulability Cost (β = 0) · p. 7

G. Real-World Validation via Sim and Real Digital Twin

  • For Tasks B and C, dynamically verified digital-twin trajectories demonstrated 60% and 80.0% task success; physical deployment replicated the twin’s behavior.Impedance control safely absorbed minor calibration offsets during contact-rich handovers.Source: G. Real-World Validation via Sim and… · pp. 7–8

VI. LIMITATIONS AND CONCLUSION

  • Pre-computed CAD meshes for tracking and grasp generation limit zero-shot deployment on novel objects, and the method assumes a fixed human-provided object trajectory.Trajectory refinement is proposed as future work.Source: §VI · p. 8
  • The framework jointly optimizes segment reachability, asymmetric grasp compatibility, and transition penalties to turn one human demonstration into a globally optimal execution plan.The conclusion describes support from simulation sweeps and real-world digital-twin validation.Source: §VI · p. 8
Loading 2610.23551v1…