PaperlayerView exact revision on arXiv

Source-linked AI summary

AnchorDP3: 3D Affordance Guided Sparse Diffusion Policy for Robotic Manipulation

Ziyan Zhao, Ke Fan, He-Yang Xu, Ning Qiao, Bo Peng, Wenlong Gao, Dongjiang Li, Hui Shen

arXiv:2506.19269v2cs.ROcs.AI

TL;DR

Dual-arm visuomotor policies must handle diverse, randomized scenes, while clutter, shared representations, and long-horizon action modeling pose challenges. AnchorDP3 uses simulator-rendered affordance cues, task-conditioned features, and sparse affordance-anchored keyposes in a shared diffusion policy. It achieved the highest success rate in the RoboTwin Challenge simulation tracks, with 98.7% reported.

  • Problem

    Dual-arm policies face perceptual ambiguity in clutter, task interference under shared representations, and inefficient action modeling for long-horizon tasks.

    Source: §1 · p. 2

  • Method

    AnchorDP3 combines simulator-supervised affordance segmentation, task-conditioned feature encoders, and sparse affordance-anchored keypose prediction in a diffusion policy.

    Source: §1 · p. 2, §3.1 · p. 3, §3.3 · p. 7

  • Results

    98.7% was the highest reported success rate in the RoboTwin Challenge simulation tracks, where AnchorDP3 also achieved top performance in both tracks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The challenge results support the potential of simulation-trained policies as a scalable approach to dual-arm visuomotor learning.

    Source: §1 · p. 2

Abstract

from arXiv · show

We present AnchorDP3, a diffusion policy framework for dual-arm robotic manipulation that achieves state-of-the-art performance in highly randomized environments. AnchorDP3 integrates three key innovations: (1) Simulator-Supervised Semantic Segmentation, using rendered ground truth to explicitly segment task-critical objects within the point cloud, which provides strong affordance priors; (2) Task-Conditioned Feature Encoders, lightweight modules processing augmented point clouds per task, enabling efficient multi-task learning through a shared diffusion-based action expert; (3) Affordance-Anchored Keypose Diffusion with Full State Supervision, replacing dense trajectory prediction with sparse, geometrically meaningful action anchors, i.e., keyposes such as pre-grasp pose, grasp pose directly anchored to affordances, drastically simplifying the prediction space; the action expert is forced to predict both robot joint angles and end-effector poses simultaneously, which exploits geometric consistency to accelerate convergence and boost accuracy. Trained on large-scale, procedurally generated simulation data, AnchorDP3 achieves a 98.7% average success rate in the RoboTwin benchmark across diverse tasks under extreme randomization of objects, clutter, table height, lighting, and backgrounds. This framework, when integrated with the RoboTwin real-to-sim pipeline, has the potential to enable fully autonomous generation of deployable visuomotor policies from only scene and instruction, totally eliminating human demonstrations from learning manipulation skills.

1 Introduction

Dual-arm manipulation policies must generalize across cluttered, varied scenes, and simulation benchmarks offer a way to train them with less reliance on costly real-world demonstrations. AnchorDP3 addresses three identified challenges and achieved the highest success rate in the RoboTwin Challenge simulation tracks.

1 Introduction

  • RoboTwin and Mimicgen provide procedurally generated environments and automated task definitions and expert-data synthesis for simulation-based visuomotor policy training.The platforms support fast iteration and reduce reliance on costly real-world demonstrations.Source: §1 · p. 1
  • The RoboTwin challenge asks participants to learn a single dual-arm visuomotor model from simulation that follows language instructions under extreme scene randomization.Its pipeline generates 3D assets from RGB images, assigns affordance coordinates, and synthesizes expert trajectories.Source: §1 · pp. 1–2
  • 98.7% success rate was the highest reported for AnchorDP3 in the RoboTwin Challenge simulation tracks.The framework also achieved top performance in both competition simulation tracks.Source: §1 · p. 2
  • AnchorDP3 combines simulator-rendered semantic segmentation, task-conditioned feature encoding, and affordance-aligned keypose supervision in a modular diffusion policy.These contributions target perceptual ambiguity, task interference, and inefficient long-horizon action modeling, respectively.Source: §1 · p. 2

2 Related Work

Prior work develops diffusion-based visuomotor policies, affordance-guided motion generation, and multi-task robotic learning. AnchorDP3 draws on these areas while extracting affordance cues from simulator-rendered layers to constrain keypose prediction.

2.1 Visuomotor policy learning based on diffusion models

  • Diffusion Policy frames vision-to-action control as conditional denoising, while 3D diffusion policy uses compact point-cloud representations for data-efficient, generalizable imitation.Other approaches include policy-gradient fine-tuning in DPPO and SO(2)-equivariant behavior cloning in Equivariant DP.Source: §2.1 · pp. 2–3

2.2 Affordance-guided motion generation

  • Affordance work progressed from object-level detection to part-level localization and 3D understanding, including point-cloud datasets and image-query-driven detection.AnchorDP3 extracts affordance cues from simulator-rendered layers, appends them to point clouds, and anchors keypose prediction to affordance-aligned constraints.Source: §2.2 · p. 3

2.3 Multi-task learning for robotic manipulation

  • Multi-task approaches include instruction-conditioned RT-1, video-pretrained GR-2, vision-language-conditioned π0, RoboVLMs design analyses, and FAST action tokenization.FAST converts continuous robot actions into discrete tokens for Transformer-based VLA models.Source: §2.3 · p. 3

3 AnchorDP3 Framework

AnchorDP3 maps RGB-D observations and language instructions to sparse keypose sequences grounded in 3D affordances. Its framework combines simulator-supervised perception, task-specific observation encoders, keypose diffusion, and sparse trajectory data construction.

3.1 Overview

  • AnchorDP3 predicts sparse keypose sequences anchored to 3D affordances from RGB-D observations and language instructions.The overview frames the policy as a diffusion-based approach for manipulation in randomized environments.Source: §3.1 · p. 3

3.2 Observation processing

  • Rendered scene differences generate task-critical point labels, while task-specific encoders feed a shared action expert across eight manipulation tasks.The task list includes three separate placement tasks for mouse, stapler, and bell; each trajectory updates only its corresponding encoder.Source: §3.2 · pp. 5–6

3.3 Affordance-Anchored Keypose Diffusion

  • Affordance-anchored keyposes compress prediction to 10–30 meaningful action anchors per task, including pre-grasp and grasp states.The anchors mark transitions between motion primitives and are geometrically positioned relative to object affordances.Source: §3.3 · p. 7
  • A shared diffusion U-Net predicts eight future 32-dimensional anchors representing joint angles and end-effector pose, with full-state L2 supervision.Only the first predicted anchor is executed during inference; the authors report that supervising all output elements benefits convergence.Source: §3.3 · p. 7

3.4 Large-scale dataset construction based on action anchors

  • Sparse dataset construction stores action anchors and samples two observations between consecutive key times, enabling 14.5 times more trajectories within the same budget and size.The authors attribute improved accuracy to greater exposure to diverse environmental settings.Source: §3.4 · p. 8
  • During rendering, 10% of observations come from deliberately incorrect poses, training the policy to recover toward correct action anchors.These examples target states that deviate from expert trajectories or approach incorrect affordances.Source: §3.4 · p. 8

4 Conclusion

AnchorDP3 addresses dual-arm manipulation in unstructured environments with three components targeting perceptual ambiguity, multi-task interference, and inefficient long-horizon action modeling. It reaches a 98.7% average success rate under extreme scene randomization.

4 Conclusion

  • 98.7% average success rate under extreme scene randomization was the highest result in the RoboTwin Challenge.The authors identify this as the outcome of the combined AnchorDP3 innovations.Source: §4 · p. 8
  • Simulator-supervised segmentation uses rendered ground truth to identify task-relevant objects and add affordance awareness to point-cloud representations.The authors present this as addressing perceptual ambiguity in cluttered scenes.Source: §4 · p. 8
  • Task-conditioned feature encoders specialize feature extraction while retaining a unified action expert, mitigating multi-task interference.Source: §4 · p. 8
  • Affordance-anchored keypose diffusion predicts sparse, geometrically meaningful keyposes supervised with full kinematic state to address inefficient long-horizon action modeling.Source: §4 · p. 8
  • Integration with procedurally generated simulation workflows is presented as a pathway for complex manipulation; future work will explore sim-to-real transfer and dynamic environments.Source: §4 · p. 8

Related papers

Loading 2506.19269v2…