Source-linked AI summary
A0: An Affordance-Aware Hierarchical Model for General Robotic Manipulation
Rongtao Xu, Jian Zhang, Minghao Guo, Youpeng Wen, Haoting Yang, Min Lin, Jianzheng Huang, Zhe Li, Kaidong Zhang, Liqiong Wang, Yuxuan Kuang, Meng Cao, Feng Zheng, Xiaodan Liang
TL;DR
Robotic manipulation needs better understanding of the spatial affordances that specify where and how objects should be contacted, because existing approaches remain limited in spatial reasoning. A0 addresses this gap with a hierarchical affordance-aware diffusion model that predicts object-centric contact points and trajectories, then executes them through a low-level module. Across multiple robotic platforms, A0 outperforms state-of-the-art methods on tasks including wiping and stacking, while long-horizon planning and orientation-sensitive manipulation remain scope boundaries.
Problem
Existing robotic-manipulation methods have limited understanding of spatial affordances—the “where” and “how” of object interactions—which matters for complex tasks.
Method
A0 hierarchically separates high-level spatial-affordance reasoning from low-level action execution using object-centric contact points and post-contact trajectories.
Results
A0 outperforms state-of-the-art methods on wiping and stacking tasks across Franka, Kinova, Realman, and Dobot systems.
Takeaways & Limitations
The object-centric, embodiment-agnostic design supports manipulation across diverse robotic platforms and complex spatial-affordance tasks.
Takeaways & Limitations
A0 focuses on high-level spatial understanding, while long-horizon planning and precise orientation-sensitive 6D manipulation remain limitations or require additional components.
Abstract
from arXiv · showhide
Robotic manipulation faces critical challenges in understanding spatial affordances--the "where" and "how" of object interactions--essential for complex manipulation tasks like wiping a board or stacking objects. Existing methods, including modular-based and end-to-end approaches, often lack robust spatial reasoning capabilities. Unlike recent point-based and flow-based affordance methods that focus on dense spatial representations or trajectory modeling, we propose A0, a hierarchical affordance-aware diffusion model that decomposes manipulation tasks into high-level spatial affordance understanding and low-level action execution. A0 leverages the Embodiment-Agnostic Affordance Representation, which captures object-centric spatial affordances by predicting contact points and post-contact trajectories. A0 is pre-trained on 1 million contact points data and fine-tuned on annotated trajectories, enabling generalization across platforms. Key components include Position Offset Attention for motion-aware feature extraction and a Spatial Information Aggregation Layer for precise coordinate mapping. The model's output is executed by the action execution module. Experiments on multiple robotic systems (Franka, Kinova, Realman, and Dobot) demonstrate A0's superior performance in complex tasks, showcasing its efficiency, flexibility, and real-world applicability.
1. Introduction
A0 addresses limited spatial-affordance understanding in robotic manipulation with a hierarchical model that links high-level affordance reasoning to low-level execution. It uses object-centric contact points and trajectories, extensive pretraining, and cross-platform evaluation.
- Motivation: Existing modular and end-to-end methods remain limited in understanding the “where” and “how” of object interactions.This limitation can produce incomplete or inefficient execution in tasks such as whiteboard wiping.
- Approach: A0 decomposes manipulation into high-level spatial affordance understanding and low-level action execution.Its affordance representation predicts object-centric contact points and post-contact trajectories to guide execution.
- Training and representation: A0 is pre-trained on 1 million contact-point localization data points and fine-tuned on annotated spatial trajectories.The representation captures spatial affordance through object-centric contact points and trajectories.
- Evaluation: A0 achieves average success rates of 62.50% on Franka and 53.75% on Kinova, outperforming the strongest baselines on both platforms.The model is evaluated on multiple robotic platforms and complex spatial-affordance tasks.
- Architecture: The model incorporates Position Offset Attention, DiT blocks, and a Spatial Information Aggregation Layer for spatial affordance understanding.These components support the model’s affordance-learning architecture.
2. Related Works
Related work spans spatial-affordance representations, vision-language-action models, and hierarchical control. A0 differs by explicitly combining hierarchical execution with object-centric spatial affordances for cross-platform manipulation.
- Affordance representations: Affordance methods use point clouds or RGB images, while language integration adds multimodal information to spatial reasoning.Output representations include heatmaps, bounding boxes, and keypoints, each balancing localization, efficiency, or articulated-object robustness.
- VLA methods: VLA research includes transformer-based, VLM-based, and diffusion-based approaches for multimodal action prediction and generalization.These categories differ in how they model action sequences and use pretrained visual-language knowledge.
- Hierarchical models: Hierarchical models organize control across abstraction levels, combining high-level planning with low-level motor control.DexGraspNet and Helix exemplify hierarchical approaches for grasping and adaptable manipulation.
- A0’s distinction: A0 combines hierarchical affordance-aware control with explicit object-centric spatial affordances rather than latent semantic representations alone.The paper positions this design as reducing computational overhead and improving adaptability for real-world deployment.
3. The A0 Model
A0 models robotic manipulation as hierarchical spatial-affordance prediction followed by action execution. It represents object-centric contact points and trajectories, conditions diffusion-based waypoint prediction on images and language, and trains from localization before task-specific trajectory supervision.
- Hierarchical Model: The model hierarchically predicts high-level spatial affordances before low-level action execution, focusing on contact positions and post-contact trajectories.This object-centric design supports deployment across platforms while requiring only limited task-specific annotation for fine-tuning.
- Embodiment-Agnostic Affordance Representation: A0 uses an Embodiment-Agnostic Affordance Representation containing object-centric 2D contact points, waypoints, and natural-language manipulation instructions.The representation draws on diverse robotic, hand-object interaction, and custom data sources.
- A0 Model Structure: A0 is a diffusion-transformer model that conditions waypoint generation on visual observations and language instructions through cross-attention.A pre-trained vision encoder processes current and previous images, while a text encoder processes the instruction; the previous image can provide motion information.
- A0 Model Structure: The model predicts T normalized two-dimensional waypoints, with the first point representing the initial contact position and later points encoding future trajectory steps.A final nonlinear MLP projects latent representations back to physical space, while inference obtains waypoints through diffusion ODE solving.
- Training: A0 is pre-trained to localize objects from textual descriptions, then fine-tuned to predict full waypoint sequences for dynamic robotic manipulation.Pre-training supervises the first waypoint using one image; fine-tuning extends the output to T waypoints and incorporates motion information.
4. Action Execution
The action execution module converts predicted 2D affordance keypoints into 3D grasp and waypoint trajectories. It combines depth deprojection, grasp-candidate selection, height-category sampling, and SE(3) trajectory generation.
- 2D-to-3D Projection: Depth-based deprojection maps predicted 2D keypoints into 3D coordinates using pixel depth, camera intrinsics, and homogeneous image coordinates.The transformation is Xi = D(xi)K−1˜xi.
- 2D-to-3D Projection: The first predicted keypoint identifies the contact point, while the remaining T−1 keypoints provide post-contact directional cues.These directional keypoints are also projected into 3D for waypoint execution.
- Grasp Pose Estimation: Grasp samplers generate candidates from local geometry, and execution selects the candidate closest to the projected grasp point.The procedure can query GraspNet or other grasp samplers.
- Waypoint Execution: A VLM selects waypoint height categories, after which the system samples final 3D waypoints and generates a smooth feasible trajectory in SE(3).Height choices include target level or above target.
5. Experiment
The experiments evaluate A0’s architecture, pre-training, real-world manipulation performance, and transfer across robotic platforms. Results examine waypoint error, component ablations, task success, and comparisons with affordance and VLA methods.
- Experimental Setup: The evaluation combines offline MAE analysis with deployment on multiple robotic platforms and comparisons against state-of-the-art methods.The study uses DROID-3k, HOI4D-22k, ManiSkill-5k, and PixMo-One-Point, then tests real-world tasks across Franka, Kinova, Realman, and Dobot.
- Effectiveness of Pre-training: Pre-training on 1 million contact-point localization samples decreases waypoint MAE and improves generalization across unseen objects and environments.The pre-training strategy is evaluated using MAE of ground-truth and predicted waypoints.
- Effectiveness of Network Structure: Removing Position Offset Attention increases MAE by 0.8 on ManiSkill-5k, while removing the Spatial Information Aggregation Layer increases MAE by 13.2 pixels on HOI4D-22k.The ablation results associate these components with motion-aware reasoning and waypoint prediction in complex or occluded environments.
- Real-World Evaluation: The real-world evaluation covers opening drawers, placing objects, pressing buttons, and wiping whiteboards using 2D affordance predictions and action execution.Each task uses 20 trials to calculate success rate across standardized camera and object-placement settings.
- Comparison with Vision-Language-Action Methods: Against VLA methods on Kinova, A0 improves average success by 33.75% over the second-best method and improves Wipe Board success by 40%.The comparison uses RDT-1B and π0, which rely on step-by-step model inference, whereas A0 uses single-shot inference.
6. Conclusion
A0 is a hierarchical affordance-aware diffusion model that combines spatial affordance reasoning with low-level action execution and generalizes across robotic platforms. Experiments validate its robustness and applicability in complex manipulation tasks.
- Conclusion: A0 decomposes manipulation into high-level spatial affordance reasoning and low-level action execution.It predicts object-centric contact points and trajectories to guide execution.
- Conclusion: The Embodiment-Agnostic Affordance Representation supports generalization across robotic platforms.A0 is pretrained on 1 million contact points and fine-tuned on annotated trajectories.
- Conclusion: A0 outperforms state-of-the-art methods in tasks including wiping and stacking.The conclusion attributes this performance to its affordance-aware design and spatial reasoning components.
- Conclusion: Experiments on Franka, Kinova, Realman, and Dobot validate A0’s robustness and real-world applicability.The model is also used with VLM-based decomposition for long-horizon and orientation-sensitive tasks.
A. Social Impact
The datasets used for evaluation and training are described as publicly available and transparent, while collected real-robot data contain no personal information.
- A. Social Impact: PixMo-Points, HOI4D, and DROID are publicly available and transparent datasets.The authors state that the real-robot datasets they collected contain no personal information.
- A. Social Impact: The authors report no ethical risk from dataset usage or privacy violations because the benchmarks are publicly available and transparent.
B. Limitations and Future work
The paper identifies limitations in grasp-pose estimation, height estimation under occlusion, orientation-sensitive manipulation, and long-horizon planning. It proposes VLM-assisted planning and improved geometric conditioning as future directions.
- B. Limitations and Future work: A0 relies on gripper samplers for grasp poses, but existing samplers may generalize poorly across tasks.This constrains the action-execution component rather than the high-level affordance representation.
- B. Limitations and Future work: Depth-based height estimation refined with a VLM may perform poorly when objects are occluded.
- B. Limitations and Future work: Orientation-sensitive tasks requiring precise 6D manipulation remain a discussion point for modular execution and viewpoint selection.The paper mentions liquid pouring and revolute-drawer opening as examples.
- B. Limitations and Future work: Long-horizon planning remains a limitation shared by affordance-based and modular approaches.The authors propose using VLMs to decompose long tasks into shorter subtasks executed stage by stage with A0.
- B. Limitations and Future work: Future work targets improved grasp-pose estimation using VLM-assisted gripper selection or direct grasp-pose generation.
- B. Limitations and Future work: Future height-estimation improvements would condition prediction on depth, gripper length, and related information.
D. Compare with Robopoint
A0 is compared with Robopoint using first-interaction-pixel MAE on HOI4D and DROID. A0 achieves lower error on both datasets under the stated comparison protocol.
- D. Compare with Robopoint: The comparison measures MAE between the first predicted interaction pixel and ground truth on HOI4D and DROID.A0 uses its first waypoint, whereas Robopoint uses the average of its predicted interaction points.
- D. Compare with Robopoint: 54.46 versus 121.09 MAE on HOI4D gives A0 a 55.2% reduction in error relative to Robopoint.
- D. Compare with Robopoint: 14.13 versus 27.47 MAE on DROID gives A0 a 40.4% improvement relative to Robopoint.
E. Real World Experiment
The real-world experiments use a multi-source annotation platform to generate standardized spatial affordance data and evaluate the system across multiple robot platforms.
- Experiments span Kinova, Franka, and Realman Robot platforms.
- The annotation platform combines robotic interactions, human-object interaction datasets, simulations, and internet-sourced data.
- The platform is semi-automated and generates standardized Embodiment-Agnostic Affordance Representation data.
G. Additional Qualitative Result on Franka
A0 generalizes across different visual backgrounds on the Franka robot, indicating robustness beyond a single scene appearance.
- A0 generalizes well to different backgrounds on the Franka robot.