PaperlayerView exact revision on arXiv

Source-linked AI summary

AFUN: Towards an Affordance Foundation Model for Functionality Understanding

Zhaoning Wang, Yi Zhong, Jiawei Fu, Henrik I. Christensen, Jun Gao

arXiv:2606.02551v1cs.ROcs.CV

TL;DR

Affordance foundation models must jointly understand where and how interactions should occur while generalizing across diverse environments, objects, and tasks. AFUN addresses this with a standardized heterogeneous-data pipeline and a model that predicts task-conditioned masks and 3D post-contact motion from RGB-D observations and language. It outperforms baselines across segmentation and motion evaluations and supports open-world and real-robot manipulation, while remaining limited on completely novel motions.

  • Problem

    Existing affordance methods often localize task-relevant regions without executable motion or predict motion with limited scalability, hindering unified open-world functionality understanding.

  • Method

    AFUN converts heterogeneous interaction data into a shared schema and jointly predicts a task-conditioned functional mask and Bézier-spline 3D post-contact motion from RGB-D input and language.

  • Results

    AFUN outperforms baselines across eight segmentation test sets and achieves the best reported 3D motion performance across three motion test sets; it reaches 69.3 mean segmentation gIoU versus 45.4 for the strongest baseline.

  • Takeaways & Limitations

    AFUN can generalize to open-world images and support real-robot manipulation without robot-specific finetuning or task-specific heuristics.

  • Takeaways & Limitations

    Adaptation remains limited for completely novel motions without similar objects for the model to relate the action to.

Abstract

from arXiv · show

Affordance understanding bridges visual perception and physical action, serving as an explainable interface for robot manipulation in open and unstructured real-world environments. Yet, building an affordance foundation model that not only understands where and how the interaction should happen, but also generalizes across diverse environments, objects, and tasks, remains a long-standing research challenge. Existing methods typically address only part of this challenge, either localizing task-relevant regions without specifying executable motion, or predicting motion but with limited scalability. In this paper, we present ourmodel, a step towards an affordance foundation model for functionality understanding. From a single RGB-D observation and a language task description, ourmodel predicts a task-conditional functional mask (where to interact) and a 3D post-contact motion curve (how to interact). To support open-world generalization, we build a large-scale standardized data pipeline that converts heterogeneous robot, human, simulation, and real-world scan data into a shared affordance schema with language, masks, and object-centric 3D motion labels. We evaluate ourmodel from three aspects: for affordance segmentation, ourmodel outperforms all baselines by a large margin across 8 test sets from 4 benchmarks, improving mean gIoU/cIoU by +23.9/+26.3; for contact-point prediction, it predicts substantially more accurate points, with a 12.7--61.3% hit-rate gain over the best baseline; and for 3D motion, it achieves the best performance on all three test sets. ourmodel can be deployed for real-world robot manipulation without finetuning for robot embodiment or using task-specific heuristics, demonstrating the ability to adapt to open-world affordance tasks. Project page: https://www.zhaoningwang.com/AFUN

1 Introduction

AFUN targets open-world affordance understanding by jointly predicting where interaction should occur and how the object should move, addressing limitations in scalability and actionable motion.

  • Motivation: Affordance understanding connects visual perception with physical action and must generalize across diverse environments, objects, and tasks.It is positioned as an explainable interface for robot manipulation in unstructured real-world environments.
  • Requirements: An affordance foundation model requires diverse task data, instruction-conditioned functional masks, and expressive yet executable 3D motion representations.The same object can afford different interaction regions under different tasks.
  • Limitations of Prior Work: Existing methods often localize interaction regions without modeling post-contact object motion, while some motion methods remain 2D or depend on heuristic object localization.These limitations leave the full interaction process incompletely specified for robot execution.
  • AFUN: AFUN converts heterogeneous robot, human, and simulation data into standardized affordance annotations and jointly predicts functional masks and Bézier-spline 3D motion from RGB-D observations and language.The predicted mask can be unprojected into 3D points for downstream robot grasping modules.
  • Results: 69.3 mean segmentation gIoU versus 45.4 for the strongest segmentation baseline, with strong open-world generalization and real-robot deployment without robot-specific finetuning.The model also surpasses standalone motion baselines by a substantial margin and predicts precise mask and motion for manipulation.

2 Related Work

Prior affordance research separately studies interaction localization, motion representation, and dataset construction, motivating AFUN’s unified treatment of these components.

  • Affordance Localization: Affordance localization methods represent task-specific interaction possibilities using dense masks, weak supervision, grounding, keypoints, or 3D point-cloud approaches.These approaches span instance- and part-level segmentation, language-conditioned grounding, and contact-point prediction.
  • Research Landscape: The related work spans separate solutions for where interaction is possible, how objects move after contact, and how affordance supervision is collected.This division frames the need for a shared affordance schema combining localization and motion.
  • Motion Representations for Affordance: Motion-focused affordance methods model post-contact object movement through discrete action types, articulation parameters, or continuous interaction geometry.Representations vary in granularity and structure.
  • Affordance Data Pipelines and Datasets: Affordance datasets differ in annotation targets, motion sources, and labeling costs, covering direct labels, robot manipulation, human videos, and 3D functional annotations.The literature includes both static affordance labels and trajectory or motion supervision.

3 Data Pipeline for AFUN

AFUN builds a unified pipeline that standardizes heterogeneous interaction data, tracks object motion rather than hand or gripper motion, and fits noisy 3D trajectories into executable curves.

  • Data Curation: The pipeline combines public robot, human, simulation, and scan data under a common affordance schema containing RGB-D observations, language, masks, and motion.This addresses the need for diverse open-world supervision covering both what and how to manipulate.
  • Data Curation: 321,190 raw videos from 10 public sources are split into 1,242,740 action intervals for standardized processing.The sources span human demonstrations, robot demonstrations, simulation data, and real-world scans.
  • Preprocessing: Each action interval is normalized into an RGB-D frame, task language, camera parameters, and corresponding video clips, with dataset-specific adapters resolving storage and camera formats.The interval-based format enables consistent downstream annotation.
  • Object Tracks and Masks: Object tracking is used as the post-contact motion signal because hand or gripper trajectories can entangle unwanted pre-contact movement with affordance-relevant object motion.The pipeline uses a vision-language model to generate a manipulable-part query for tracking.
  • 3D Motion Curves: The tracked masks are back-projected into 3D, and the resulting noisy, non-uniform path is fit with a smooth Bézier spline as the canonical motion representation.This produces a structured curve for downstream motion prediction and execution.
  • Filtering and Dataset Statistics: 59,867 training samples remain after filtering 1,242,740 action intervals and retaining 223,334 samples with valid motion labels.Filtering removes low-quality clips involving poor task grounding, occlusion, unreliable segmentation, or insufficient motion.

4 Method

AFUN jointly predicts task-conditioned functional masks and 3D post-contact motion from RGB-D observations and task phrases. Its architecture connects frozen vision-language reasoning with segmentation and motion decoders, using an anchored Bézier representation and staged training.

  • Network Architecture: AFUN takes an RGB-D observation and task phrase to jointly predict a task-conditioned functional mask and 3D post-contact motion curve.Both outputs are produced in a single forward pass.
  • Network Architecture: MetaQuery uses two learnable token sets to pass frozen VLM representations separately to the segmentation and motion models.The token hidden states provide shared VLM reasoning capabilities to both downstream models.
  • Segmentation and Motion Decoding: SAM3 decodes functional masks, while a pretrained 3D feature encoder supplies point-cloud geometry for the transformer-based motion decoder.The motion decoder attends to geometric features, SAM3 object features, and motion MetaQuery tokens.
  • Curved Motion Representation: AFUN represents post-contact motion as an anchored 3D Bézier spline whose contact-centroid start point and predicted control points define executable waypoints.Uniformly sampling normalized time produces the robot-followable trajectory.
  • Training Scheme: Three-stage training first aligns MetaQuery with SAM3, then learns affordance segmentation, and finally fine-tunes motion prediction after segmentation becomes reliable.Point-sampling supervision is reported as more effective than directly regressing control-point locations.
  • Qualitative Results: Qualitative examples show task-conditioned masks capturing complex scissor handles and intent-dependent shovel-blade and hammer-handle regions.The examples illustrate segmentation that follows the task instruction rather than only object appearance.

5 Experiments

AFUN is evaluated across segmentation, contact-point, 3D-motion, ablation, and real-robot settings, consistently outperforming baselines and achieving reliable deployment without task-specific heuristics.

  • Evaluation Overview: AFUN is evaluated on affordance segmentation, contact-point prediction, 3D motion, ablations, and real-robot manipulation.The evaluation spans eight segmentation test sets, three motion test sets, and four real-world tasks.
  • Affordance Segmentation: AFUN improves mean gIoU/cIoU by 23.9/26.3 points over the strongest baseline across eight test sets from four benchmarks.It achieves the best gIoU and cIoU on every test set, including with the smaller Qwen3-VL-2B variant.
  • Contact-Point Prediction: 12.7%–61.3% is AFUN’s contact-point hit-rate advantage over the best baseline across evaluated datasets.The predicted point is the Pole of Inaccessibility of AFUN’s predicted mask, evaluated by whether it lies inside the ground-truth affordance mask.
  • 3D Motion Evaluation: AFUN achieves the best ADE and FDE in absolute and relative scale on all three 3D-motion test sets and significantly outperforms baselines in CIM.This advantage remains when General Flow receives AFUN’s predicted mask and start anchor under a favorable evaluation protocol.
  • Ablations: Ablations show that Qwen3-VL-8B, the default 3D feature encoder, and AFUN’s curve parameterization each outperform their tested alternatives.The authors associate the results with dense vision–language suitability, stronger geometric cues, and reduced ambiguity across motion types.
  • Real-Robot Demonstration: AFUN achieves a 90% average success rate on four Franka Research 3 tasks using RGB-D input and no additional task-specific heuristics.The tasks include screwdriver pickup, pot-lid removal, drawer opening, and microwave opening.

6 Conclusion

AFUN unifies task-conditioned affordance localization with executable 3D motion prediction from RGB-D observations and language. It outperforms baselines across segmentation, contact-point, and motion evaluations and transfers to real-robot manipulation without embodiment-specific finetuning.

  • AFUN predicts a task-conditional functional mask and a 3D post-contact motion curve from one RGB-D observation and a language task description.The mask specifies where to interact, while the motion curve specifies how the interaction should happen.
  • AFUN outperforms all baselines on affordance segmentation across eight test sets from four benchmarks.
  • AFUN predicts substantially more accurate contact points and achieves the best 3D motion performance on all three motion test sets.
  • Without embodiment-specific finetuning, AFUN can be directly deployed for real-robot manipulation.The paper presents this deployment as a practical path toward open-world affordance models combining functionality perception with executable action.

Technical Appendices and Supplementary Material

The supplementary appendix supports the main paper with limitations and failure analysis, data-schema construction details, dataset galleries, and additional experimental information.

  • Appendix A documents AFUN’s limitations, failed cases, and social responsibility considerations.
  • Appendix B explains how heterogeneous sources are converted into the unified affordance-data schema.It covers source-specific preprocessing, SAM3 query generation, curve fitting, dataset galleries, and HOVA-500K conversion for segmentation training.
  • Appendix C provides additional experimental details.

A Limitations, Failed Case Analysis and Social Responsibility

AFUN’s open-world adaptation remains limited when the required motion is completely novel and no similar objects are available for action transfer. The paper also emphasizes that deployment requires safety controls and task-specific validation.

  • Limitations and Failure Cases: AFUN cannot accurately predict motion for completely novel actions when no similar objects exist for the model to relate the action to.The appendix identifies larger-scale, truly open-world data as a next step.
  • Limitations and Failure Cases: Figure 8 shows failed cases involving Spray Bottles and Sun Visors, which are absent or not sufficiently represented in the training dataset.
  • Social Impact and Responsibility: AFUN is not a standalone policy or safety system and requires human supervision, collision checking, physical limits, emergency stops, and task-specific validation.These safeguards are especially important near people, fragile objects, or hazardous materials.
  • Social Impact and Responsibility: Downstream users should evaluate bias and generalization in their own environments rather than assuming benchmark performance transfers uniformly.

B Dataset Pipeline Details

The dataset pipeline converts heterogeneous demonstrations into a unified affordance schema through source-specific preprocessing followed by dataset-agnostic training-data generation. Per-source statistics distinguish recording episodes, action intervals, camera views, and successfully fitted motion curves.

  • Dataset Pipeline Details: A two-phase pipeline converts robot teleoperation, human egocentric video, simulation, and real-world scan data into a unified affordance dataset.
  • Dataset Pipeline Details: The dataset-specific preprocessing module handles raw-format differences and exports a shared per-interval schema.
  • Dataset Pipeline Details: The dataset-agnostic main process generates a SAM3 task query, tracked object mask, depth, 3D object trajectory, and fitted Bézier spline curve.
  • Dataset Statistics: Table 7 reports per-source statistics for episodes, intervals, views, and successfully fitted curves.Episodes are top-level recordings, intervals are action-segmented sub-clips, views are per-camera observations, and fitted curves are views with successful Bézier fitting.

B.1 Cross-Dataset Preprocess

AFUN standardizes heterogeneous robot, human, simulation, and scan data into interval-level affordance records, then derives task-conditioned masks and canonical 3D motion curves.

  • Shared preprocessing: Every source adapter writes a shared schema containing observation/contact frames, RGB-D, task language, camera calibration, and video spans.
  • Source-specific handling: Human and scanned-room sources require specialized handling for moving cameras, geometric annotations, or absent object motion.VITRA uses SLAM poses for trajectory geometry, whereas SceneFun3D bypasses tracking and curve fitting because its objects do not move.
  • Interval recovery: Dataset-specific adapters recover manipulation intervals from gripper signals, language boundaries, action events, or source-provided annotations.Calvin combines language windows with nearby gripper-closing segments, while HOI4D uses events such as Reachout, Grasp, and Pickup.
  • Shared preprocessing: The pipeline enforces common units and transform conventions, using millimeters for depth, meters for 3D positions, and Tbase→cam transforms.
  • Query generation: AFUN generates concise segmentation queries naming the smallest visible manipulable part, using observation and contact frames to resolve target identity and spatial qualifiers.The query is passed to SAM3 to obtain the affordance mask.
  • Motion processing: The motion pipeline robustly smooths back-projected positions, fits a planar constant-curvature primitive, snaps near-linear motion to a line, and converts it to a cubic Bézier curve.Support points are resampled by cumulative arc length before robust nonlinear fitting.

B.3 Gallery on the AFUN dataset

The AFUN dataset gallery visualizes task instructions together with predicted affordance masks and fitted 3D trajectories across sampled examples.

  • AFUN dataset gallery: 48 AFUN samples are shown as observation frames with SAM3 affordance masks and fitted Bézier-spline 3D trajectories.Samples are divided between Figures 9 and 10, with each language instruction displayed below its image.
  • Visualization: The gallery uses red masks and yellow bounding boxes to present the predicted affordance region and trajectory context.
  • Training-data conversion: HOVA-500K point annotations are converted into masks by selecting confident SAM3 predictions whose centroids lie within 0.07W pixels of the contact point.Samples without a qualifying mask are rejected.
  • Training-data conversion: A second language-generation pass rewrites converted samples into task-level instructions without naming the highlighted contacted part.The part-level prompt is used only to obtain the mask, avoiding direct answer leakage in the training query.

C Extended Evaluation Qualitative Results

The extended qualitative results compare AFUN with baselines on segmentation, 3D motion, and a robotic screwdriver-pickup demonstration using shared visual overlays.

  • Affordance segmentation: Figures 12–15 compare AFUN and three baselines against ground-truth masks across four affordance segmentation evaluations.Rows show the input query, model predictions, and ground-truth mask; red denotes predictions, green ground truth, and yellow boxes mask boxes.
  • 3D motion evaluation: Figures 16–20 compare AFUN with four baselines and ground truth for 3D motion evaluation.Each cell includes projected trajectory-and-mask imagery above a back-projected 3D point cloud.
  • 3D motion evaluation: Motion trajectories are colored yellow to blue from start to end, while ground-truth trajectories are rendered as green curves.
  • Robotic demonstration: The robotic demo shows AFUN providing trajectories for a screwdriver-pickup task.An additional result is presented in Figure 11.
Loading 2606.02551v1…