PaperlayerView exact revision on arXiv

Source-linked AI summary

DualManip: Agentic Dynamic Manipulation via Dual-Path Semantic Reasoning and Geometric Adaptation

Chengxi Li, Yan Di, Yingyue Li, Ruida Zhang, Mingyang Li, Xiangyang Ji

arXiv:2609.31112v1cs.RO

TL;DR

Dynamic manipulation remains challenging because scene geometry can change while task intent stays valid, yet current systems struggle to adapt efficiently to continuous change. DualManip separates semantic reasoning from live geometric grasp adaptation, using transferred contacts and feasibility checks. Across real-world tasks it demonstrates robust manipulation, particularly under continuous scene changes, with lower adaptation latency than agentic verification and replanning.

  • Problem

    Existing systems struggle to adapt to continuously changing geometry without unnecessary high-latency semantic inference or acting on stale geometry.

    Source: §I · pp. 1–2

  • Method

    DualManip separates semantic planning and task grounding from live geometric contact transfer, validating grasp updates and returning to semantic planning when updates fail.

    Source: §I · p. 2

  • Results

    Across six real-world tasks, DualManip shows superior manipulation robustness, particularly under continuous changes, maintaining a 53.3% mean success rate versus 28.9% for ReKep and 17.8% for OmniManip.

    Source: §I · p. 2, D. Dynamic Manipulation Results · p. 6

  • Takeaways & Limitations

    Real-world experiments indicate that validated contact transfers let DualManip preserve task intent while adapting to rigid motion, articulated reconfiguration, and non-rigid deformation.

    Source: §V · p. 7

    DualManip relies on pre-constructed templates and object-specific dense-correspondence training, constraining direct zero-shot generalization to unseen instances.

    Source: §V · p. 7

Abstract

from arXiv · show

Vision-language models (VLMs) enable open-vocabulary reasoning for robot manipulation, but their high inference latency limits responsiveness in dynamic scenes. Many scene changes, however, alter object geometry without invalidating task intent. We present DualManip, a dual-path framework that decouples infrequent semantic reasoning from responsive geometric adaptation. The semantic path decomposes the task and grounds task-relevant interactions, followed by a constraint-solving module for pose optimization. During execution, the geometric path continuously updates template-to-observation correspondences from live RGB-D observations via a shape-adaptive network. These correspondences transfer task-relevant grasp contacts across observations, enabling online grasp reconstruction under object motion and non-rigid deformation. The Information Interaction Module bridges the two paths by initializing task-relevant grasps from semantic grounding, validating geometric updates, and triggering semantic replanning upon update failures. Real-world evaluation spans six manipulation tasks covering non-rigid deformation, articulated reconfiguration, rigid motion, and high-precision assembly across three settings: static, single-change, and continuous dynamic. DualManip demonstrates superior manipulation robustness, particularly under continuous scene changes, while achieving geometric adaptation approximately 46$\times$ faster than agentic verification and semantic replanning. Our project page: https://lichengxi1.github.io/Dualmanip.

I. INTRODUCTION

Dynamic manipulation must accommodate changing geometry while preserving task intent, but current approaches struggle to combine rapid updates with non-rigid adaptation. DualManip separates semantic planning from geometric adaptation and evaluates this strategy across varied real-world tasks and scene-change settings.

I. INTRODUCTION

  • Existing modular VLM systems remain limited in dynamic scenes: geometric tracking can degrade under non-rigid deformation, while verification and replanning incur excessive latency.End-to-end VLA and WAM approaches also typically require substantial task- or embodiment-specific training data and incur large-model inference costs.Source: §I · p. 1
  • Geometric changes can leave task intent valid, but existing agentic systems may trigger unnecessary high-latency VLM inference or operate on stale geometry.Semantic changes instead alter task interpretation and require renewed reasoning.Source: §I · p. 2
  • DualManip separates semantic planning and task-relevant grounding from live geometric updates that transfer template contacts, reconstruct grasps, and validate feasibility before execution.Unreliable correspondences or infeasible geometry trigger renewed semantic grounding and planning.Source: §I · p. 2
  • DualManip achieves superior manipulation robustness, particularly under scene changes, with markedly lower adaptation latency than agentic verification and replanning across six real-world tasks.The tasks span non-rigid deformation, articulated reconfiguration, rigid motion, and precision assembly; evaluation includes static, single-change, and continuous-dynamic settings.Source: §I · p. 2
  • A shape-adaptive correspondence network jointly models rigid motion and non-rigid deformation to match points across object states.This supports task-relevant grasp reconstruction across changing geometry.Source: §I · p. 2
  • Task-conditioned contact transfer, feasibility validation, and selective fallback connect grasp reconstruction to semantic replanning.The framework combines agentic semantic planning with responsive geometric adaptation to update actions while preserving intent.Source: §I · p. 2

II. RELATED WORK

Prior work spans foundation-model manipulation, agentic replanning, and geometric tracking, but these approaches leave challenges in continuous adaptation to changing geometry. DualManip combines semantic task planning with template-based grasp updates and constraint-optimized poses.

A. Foundation Models for Robot Manipulation

  • Foundation-model approaches include end-to-end VLA and WAM policies [2], [4], [19]–[24] and modular VLM systems with explicit geometric or planning representations [9]–, –[29].DualManip follows the modular paradigm and targets continuous adaptation to object motion and non-rigid deformation.Source: A. Foundation Models for Robot… · p. 2

B. Agentic and Dynamic Manipulation

  • Agentic systems use observation, verification, memory, orchestration, and recovery [14],, [30]–[34], but typically adapt at the task or skill level through discrete replanning.DualManip instead updates planned grasps continuously through a geometric path while preserving valid task semantics.Source: B. Agentic and Dynamic Manipulation · p. 2

C. Geometric Tracking and Deformable Correspondence

  • Geometric tracking uses rigid SE(3) transforms,, point tracking –, and deformable correspondence methods –, while existing deformable methods are often category- or interaction-specific.DualManip uses a unified template-to-observation correspondence mechanism for execution-time action transfer.Source: C. Geometric Tracking and Deformable… · p. 2

A. Overview

  • DualManip separates semantic reasoning from geometric adaptation: semantic grounding initializes the grasp, live RGB-D updates transfer grasp contacts, and constraints yield executable poses.The information interaction module validates updates and invokes semantic replanning when adaptation is unreliable or infeasible.Source: A. Overview · pp. 2–3, Figure 2 · p. 3

B. Semantic Path and Constraint Solving

The Semantic Path parses a manipulation task into stages, grounds task-relevant interactions, and converts them into spatial constraints for end-effector pose optimization.

B. Semantic Path and Constraint Solving

  • The VLM decomposes the task into stages with active and passive objects, which SAM 3 localizes and segments to recover point clouds from depth.Source: B. Semantic Path and Constraint Solving · p. 3
  • For each stage, the VLM identifies functional regions and directions, while SAM 3 produces local masks and 3D interaction points.Sofar infers task-conditioned semantic orientations.Source: B. Semantic Path and Constraint Solving · p. 3
  • Grounded locations and directions are combined with predefined geometric relations, including point distances and directional alignments, to generate stage-specific constraints.The VLM receives an annotated scene image, stage intent, and the predefined relation set.Source: B. Semantic Path and Constraint Solving · pp. 3–4
  • Each constraint becomes a geometric violation cost over the end-effector pose, and pose optimization minimizes the total violation.A weight controls each constraint’s contribution.Source: B. Semantic Path and Constraint Solving · p. 4

C. Geometric Path for Dynamic Adaptation

DualManip adapts grasps by transferring task-relevant contacts from a fixed template to live RGB-D observations through point correspondences, avoiding repeated semantic reasoning during geometric updates. Geometry-aware matching supports motion and deformation, while reconstructed grasps are routed for execution or fallback.

C. Geometric Path for Dynamic Adaptation

  • Template-to-observation correspondences transfer task-relevant grasp contacts to current geometry, supporting adaptation to object motion and non-rigid deformation.A persistent 3D template is reconstructed from multi-view images; the template and observed point clouds are jointly processed for point-level matching.Source: C. Geometric Path for Dynamic Adaptation · p. 4
  • The shape-adaptive network couples Sinkhorn-normalized point similarities with predicted deformation and rigid transformation to constrain matches by geometry.Each assignment row represents a correspondence distribution over the reference template, and τ controls matching sharpness.Source: C. Geometric Path for Dynamic Adaptation · p. 4
  • Training combines canonical-match, geometric, projection, and regularization objectives to supervise correspondence consistency and template coverage.The losses include alignment to canonical template regions, geometric consistency, projected keypoints and masks, and regularization against excessive deformation and assignment collapse.Source: C. Geometric Path for Dynamic Adaptation · p. 4
  • Shared-template affinity transfers each source grasp contact to its highest-scoring target point across object states.Soft assignments form the source-to-target affinity; Sinkhorn normalization encourages balanced marginals and reduces many-to-one assignment collapse.Source: C. Geometric Path for Dynamic Adaptation · pp. 4–5
  • The transferred contacts determine the grasp center and closing direction, with a height correction and initial-grasp orientation alignment completing pose reconstruction.The reconstructed grasp pose and its confidence are passed to feasibility checking and subsequent execution or fallback routing.Source: C. Geometric Path for Dynamic Adaptation · p. 5
  • Unreliable or infeasible updates trigger fallback; collision-prone and unintended grasps are rejected before execution.The illustrated rejection cases trigger semantic replanning.Source: C. Geometric Path for Dynamic Adaptation · p. 5, Figure 3 · p. 5

D. Information Interaction Module

The Information Interaction Module turns semantic grounding into an initial grasp, then checks adapted grasps and routes unreliable or infeasible updates back to semantic replanning.

D. Information Interaction Module

  • The module connects semantic grounding, geometric adaptation, and execution by initializing grasps, validating updates, and routing failures for replanning.Source: D. Information Interaction Module · p. 5
  • Candidate grasps are generated from the object point cloud and interaction-grounded mask, and contacts outside the mask are discarded.The top-scoring remaining pose becomes the initial grasp, with its two surface contacts retained as task-relevant anchors.Source: D. Information Interaction Module · p. 5
  • Each reconstructed grasp is checked for correspondence confidence, reachability, and collision safety before execution or fallback.Collision checking considers the current scene point cloud during gripper approach and closing, including interference with surroundings or unintended target parts.Source: D. Information Interaction Module · p. 5

A. Experimental Setup

The evaluation covers six real-world manipulation tasks across static, single-change, and continuous-dynamic settings, with assembly tasks limited to static trials. Experiments use a KUKA arm, two RGB-D cameras, and object-specific correspondence training data.

A. Experimental Setup

  • Real-world experiments use a KUKA LBR iiwa 14 R820 arm, a Robotiq 2F-140 gripper, two ZED 2i cameras, and an RTX 3090 workstation.One camera is wrist-mounted and the other fixed at an elevated viewpoint opposite the robot.Source: A. Experimental Setup · p. 5
  • Six tasks span non-rigid deformation, articulated reconfiguration, rigid motion, and precision assembly.The tasks include snake placement, toy positioning, block placement, RAM insertion, GPU insertion, and fan placement.Source: A. Experimental Setup · p. 5
  • The three scene settings are static, single-change, and continuous dynamic, with 15 trials per object-condition-method combination.General tasks are tested in all three settings, while the three PC chassis assembly tasks are tested only in the static setting.Source: A. Experimental Setup · p. 5
  • Each object uses a fixed reference template, and the correspondence network is trained from 100 RGB-D observations per camera viewpoint.Template and observed point clouds are downsampled to 1,024 points; GPT-5.6 Terra is used for semantic reasoning, with local Qwen3.5-2B additionally used for agentic latency evaluation.Source: A. Experimental Setup · p. 5

B. Baselines

The baselines represent sparse keypoint tracking, rigid object-pose tracking, agentic VLM verification and replanning, and open-loop execution. Where applicable, methods share an execution backend to isolate scene-adaptation mechanisms.

B. Baselines

  • ReKep uses sparse keypoint tracking, OmniManip uses rigid 6D pose tracking, and CLEA handles scene changes through visual validation and high-level replanning.Source: B. Baselines · p. 5
  • Table I reports successful trials and average success rate (%) under the static setting.Source: Table I · p. 6
  • Open-loop execution uses the initial grasp without geometric adaptation and is evaluated only in dynamic settings.All methods share the same execution backend where applicable, isolating their scene-adaptation mechanisms.Source: B. Baselines · p. 6

C. Static Manipulation Results

Under static conditions, DualManip matches the best baseline on general tasks and leads on assembly tasks, supporting competitive baseline manipulation capability.

C. Static Manipulation Results

D. Dynamic Manipulation Results

DualManip performs best under continuous scene changes and ties CLEA under single changes, adapting task-relevant grasps across rigid, articulated, and deformable geometry.

D. Dynamic Manipulation Results

  • Rigid pose tracking degrades on articulated and deformable objects, while sparse keypoint tracking remains sensitive to unreliable tracks.DualManip transfers task-relevant contacts through a shared template space; Fig. 5 depicts toy reconfiguration and block displacement.Source: D. Dynamic Manipulation Results · p. 6, Figure 5 · p. 6
  • 53.3% mean success under continuous changes surpasses ReKep (28.9%) and OmniManip (17.8%), while CLEA and Open-loop fail completely.Figure 6 shows grasp poses updated as the snake moves and deforms while task intent remains valid.Source: D. Dynamic Manipulation Results · p. 6, Figure 6 · p. 6, Table II · p. 7
  • 64.4% mean success under single changes ties DualManip with CLEA, ahead of ReKep (42.2%) and OmniManip (26.7%).CLEA can recover from an isolated change through high-level replanning, but at considerably higher computational cost.Source: D. Dynamic Manipulation Results · p. 6, Table II · p. 7

E. Latency Analysis

DualManip's geometric adaptation is much faster than agentic verification and replanning, while the conclusion identifies template dependence as a boundary on zero-shot generalization.

E. Latency Analysis

  • 138.7 ms per adaptation for DualManip compares with CLEA's 6.376 s using GPT and 4.329 s with local Qwen3.5-2B, approximately 31× DualManip.Latency measures scene-update arrival to internal update completion; DualManip includes correspondence estimation and grasp reconstruction, averaged over 20 independent updates.Source: E. Latency Analysis · p. 7

V. CONCLUSION

  • DualManip decouples semantic reasoning from geometric adaptation and validates shape-adaptive contact transfers before execution.The conclusion reports robust adaptation to rigid motion, articulated reconfiguration, and non-rigid deformation, with lower latency than agentic verification and replanning.Source: §V · p. 7
  • Pre-constructed templates and object-specific dense-correspondence training constrain direct zero-shot generalization to unseen instances.Future work will explore category-level correspondence learning to relax template dependencies while preserving real-time responsiveness.Source: §V · p. 7

Related papers

Loading 2609.31112v1…