PaperlayerView exact revision on arXiv

Source-linked AI summary

3D AffordanceNet: A Benchmark for Visual Object Affordance Understanding

Shengheng Deng, Xun Xu, Chaozheng Wu, Ke Chen, Kui Jia

arXiv:2103.16397v2cs.CV

TL;DR

Visual affordance understanding lacks a geometry-rich 3D benchmark despite its importance for robotic interaction. The paper introduces 3D AffordanceNet, evaluates three affordance-estimation settings with point-cloud networks, and finds that the benchmark remains challenging, especially under SO(3) rotation.

  • Problem

    Existing visual affordance research mainly uses 2D or 2.5D data, while geometry-rich 3D learning and prediction remain insufficiently benchmarked for robotic affordance understanding.

  • Method

    The paper builds 3D AffordanceNet with probabilistic point-wise affordance annotations and evaluates full-shape, partial-view, and rotation-invariant estimation using point-cloud networks plus semi-supervised learning.

  • Results

    SO(3)/SO(3) rotation causes a 5–10% drop in all metrics, while z/z rotation causes about a 1% drop across networks.

  • Takeaways & Limitations

    The benchmark encourages research on 3D affordance estimation, with future work needed for difficult affordance categories and SO(3) rotation.

Abstract

from arXiv · show

The ability to understand the ways to interact with objects from visual cues, a.k.a. visual affordance, is essential to vision-guided robotic research. This involves categorizing, segmenting and reasoning of visual affordance. Relevant studies in 2D and 2.5D image domains have been made previously, however, a truly functional understanding of object affordance requires learning and prediction in the 3D physical domain, which is still absent in the community. In this work, we present a 3D AffordanceNet dataset, a benchmark of 23k shapes from 23 semantic object categories, annotated with 18 visual affordance categories. Based on this dataset, we provide three benchmarking tasks for evaluating visual affordance understanding, including full-shape, partial-view and rotation-invariant affordance estimations. Three state-of-the-art point cloud deep learning networks are evaluated on all tasks. In addition we also investigate a semi-supervised learning setup to explore the possibility to benefit from unlabeled data. Comprehensive results on our contributed dataset show the promise of visual affordance understanding as a valuable yet challenging benchmark.

1. Introduction

Visual affordance understanding is important for robotic interaction, but existing benchmarks largely use 2D or 2.5D data rather than geometry-rich 3D shapes. 3D AffordanceNet addresses this gap with large-scale annotations and three affordance-estimation benchmarks.

  • Visual affordance understanding supports robotic interaction with objects and environments, including sitting, grasping, and lifting.
  • 2D and 2.5D datasets support visual affordance research, but they do not convey the detailed geometry needed for interactions such as grasping and sitting.
  • The dataset construction uses affordance definitions for 3D objects, sparse point annotations, and label propagation to produce point-wise probabilistic affordance scores.
  • Three point-cloud networks are benchmarked, alongside a semi-supervised method that uses unlabeled data for affordance estimation.
  • 3D AffordanceNet contains 56,307 affordance annotations for 22,949 shapes across 18 affordance classes and 23 semantic object categories.
  • The benchmark defines full-shape, partial-view, and rotation-invariant affordance-estimation tasks for evaluating 3D visual affordance understanding.

2. Related Work

Prior visual affordance datasets focus on RGB or RGBD images and relatively small object and affordance sets. 3D AffordanceNet connects affordance understanding to direct 3D point-cloud annotations with richer object geometry.

  • Visual affordance research includes recognition, detection, segmentation, reasoning, activity recognition, and social-affordance tasks.
  • UMD, CAD120, and IIT-AFF provide RGBD affordance segmentation datasets covering 10–20 objects, 6–9 affordance types, and 3k–10k labeled images.
  • ShapeNet and PartNet provide large-scale 3D shape and part-structure resources, but PartNet did not establish a dedicated affordance benchmark.
  • 3D AffordanceNet provides visual affordance estimation on 3D point clouds, with affordance types selected from prior work and annotations made directly on 3D data.

3. Dataset Construction

3D AffordanceNet combines human-selected affordance annotations with label propagation to create dense, probabilistic score maps for 3D shapes. Its construction covers 22,949 shapes across 23 categories, 18 affordance classes, and 56,307 affordance annotations.

  • Affordance Type: The dataset defines 18 affordance categories and associates suitable affordances with each PartNet object category.Annotators can freely localize an affordance on supported regions, and some shapes may lack affordances defined for their category.
  • Annotation Interface: A web-based 3D question-answering interface collects raw annotations while allowing annotators to rotate, translate, and rescale each shape.Questions are tailored to individual affordances to guide perception during annotation.
  • Ground-Truth Construction: Labels are propagated from selected affordance keypoints to densely sampled mesh-surface points using a kNN graph restricted to parts supporting each affordance.The graph is symmetrized and normalized before propagation, and the resulting scores are linearly normalized to [0, 1].
  • Ground-Truth Construction: The propagation solution uses one-hot labels over 18 affordances and sets the decay hyper-parameter α to 0.998 throughout the experiments.The propagated scores are computed with the closed-form solution S = (I−αf W)^−1Y before linear normalization.
  • Statistics: The final dataset contains 22,949 shapes from 23 categories, 56,307 affordance annotations from 18 classes, and multi-label point annotations.Each category has at most five defined affordance types.

4. Tasks and Benchmarks

The benchmark evaluates affordance estimation from full, partial, and rotated 3D point clouds, alongside a semi-supervised setting. Results show that existing networks produce reasonable full-shape predictions but face substantial difficulty with affordance localization, partial views, and arbitrary rotations.

  • Benchmark tasks: The benchmark covers full-shape, partial-view, and rotation-invariant affordance estimation, plus semi-supervised learning with unlabeled data.Rotation experiments include z/z and SO(3)/SO(3) settings.
  • Full-Shape Affordance Estimation: Full-shape estimation predicts point-wise affordance scores from complete 3D point clouds using PointNet++, DGCNN, and U-Net.Evaluation reports mAP, MSE, AUC, and aIoU; dice loss addresses the large proportion of zero-score points.
  • Full-Shape Affordance Estimation: The three networks achieve relatively low aIoU scores, indicating that full-shape affordance estimation remains challenging and that PointNet++ and DGCNN still underfit.Qualitative results include successful handle and webbing localization for bag grasp, but bottle pour is misplaced toward the lid.
  • Partial-View Affordance Estimation: Partial-view estimation uses point clouds synthesized from four camera viewpoints and evaluates affordances only on visible points.The same backbone networks and training strategies are used, with 2048 points sampled per viewpoint.
  • Partial-View Affordance Estimation: Partial-view performance decreases because geometric information is lost, although qualitative predictions remain reasonable for several affordances.Contain on storage furniture is partially missing when the furniture top is not fully observed.
  • Rotation-Invariant Affordance Estimation: Rotation-invariant performance drops by around 1% for z/z and 5−10% for SO(3)/SO(3) across metrics, with SO(3)/SO(3) more challenging.The experiments randomly rotate training shapes and test each shape under five sampled poses.
  • Semi-Supervised Affordance Estimation: With only 1% labeled data, the semi-supervised approach outperforms the fully supervised baseline on all three metrics.Gains are especially high for some affordances, such as open, that perform poorly in full-shape estimation.

5. Conclusion

The conclusion presents 3D AffordanceNet as a 3D point-cloud benchmark with three affordance-estimation tasks and reports remaining challenges under difficult affordance categories and SO(3) rotation.

  • 3D AffordanceNet contains 22,949 shapes from 23 semantic object categories, with 56,307 annotations spanning 18 visual affordance categories.
  • The study defines full-shape, partial-view, and rotation-invariant affordance-estimation tasks and benchmarks three state-of-the-art point-cloud networks.
  • The results indicate that better performance is still needed for difficult affordance categories and under SO(3) rotation.
  • The authors propose a semi-supervised affordance-estimation method intended to use large amounts of unlabeled data.

B. Data Annotation

The annotation analysis finds that human-perceived affordances do not fully overlap with individual PartNet parts, while annotator disagreement remains sufficiently patterned for learning.

  • Human-perceived affordances often do not fully overlap with individual PartNet parts.Low maximal IoU between affordances and combinations of fine-grained parts indicates that affordances should be labeled separately.
  • Three annotators per object provide annotation diversity and partially address affordance ambiguity.
  • Relatively small average variance across annotators indicates patterns in the annotations that models can learn.

C. More Ground-Truth Visualizations

Ground-truth visualizations reinforce that affordances frequently extend beyond individual PartNet parts, supporting separate affordance annotation.

  • Ground-truth visualizations show that human-perceived affordances often do not fully overlap with individual PartNet parts.
  • This mismatch justifies annotating affordances separately from existing part annotations.

D. More Qualitative Examples

Qualitative comparisons of PointNet++ and DGCNN show similar predicted affordance locations but different confidence tendencies, producing different error patterns.

  • The qualitative experiments cover full-shape, partial-view, and rotation-invariant affordance estimation with PointNet++ and DGCNN backbones.
  • PointNet++ and DGCNN often predict nearby affordance locations, but their confidence scores differ.
  • PointNet++ tends toward low confidence, causing more false negatives, whereas DGCNN predicts more aggressively, leading to more false positives.

E. Training Details

The training setup formulates affordance estimation as 18 binary classification problems using PointNet++ and DGCNN segmentation backbones, with a separate semi-supervised configuration. Annotation uses a web-based 3D interface that guides keypoint selection and affordance assignment.

  • Model and optimization: Affordance estimation is formulated as 18 binary classification heads, one for each affordance category, on PointNet++ and DGCNN point-wise features.The point-wise feature dimensions are 128 for PointNet++ and 256 for DGCNN.
  • Model and optimization: PointNet++ and DGCNN are trained for 200 epochs with batch size 16 using separate optimizer and learning-rate schedules.PointNet++ uses Adam with an initial learning rate of 0.001, while DGCNN uses SGD with an initial learning rate of 0.1.
  • Semi-supervised training: The semi-supervised setup uses DGCNN with balanced batches of 8 labeled and 8 unlabeled examples and Virtual Adversarial Training.The VAT hyperparameters are ξ=1e-6 and ϵ=2.0, with one iteration for the virtual adversarial direction.
  • Annotation interface: The annotation interface presents 3D shapes that annotators can translate, rotate, and rescale while selecting surface keypoints through guided questions.Annotators can also inspect supported affordances selected from the affordance list.

G. PointContrast Fine-Tune

The PointContrast experiment fine-tunes a pretrained U-Net on 3D AffordanceNet and compares it with training the same architecture from scratch. Fine-tuning outperforms scratch training on full-shape affordance estimation, suggesting pretrained representations can benefit this task.

  • Method: A PointContrast-pretrained U-Net is fine-tuned on 3D AffordanceNet and compared against the same architecture trained from scratch.Fine-tuning uses SGD for 60 epochs with learning rate 0.1, weight decay 1e-4, and momentum 0.9.
  • Results: Fine-tuned U-Net performance surpasses training from scratch on the full-shape affordance estimation task.The comparison is reported in Table 9.
  • Interpretation: The authors state that pretraining on a rich source dataset can benefit fine-tuning on affordance estimation.They suggest the same possibility may apply to PointNet++ and DGCNN.

H. Affordance Understanding in 3D

The paper argues that 3D reasoning is needed for affordances involving functional surface properties that may be hidden in a single view. Experiments compare full-3D and partial-view training, while qualitative figures visualize predictions across full-shape, partial-view, and rotation-invariant settings.

  • Motivation: Many affordances require 3D surface reasoning because relevant interaction points can be self-occluded in a single-view observation.The paper uses grasping a mug as an example where grasp points may be hidden from a 2.5D view.
  • Experimental comparison: The experiments compare single-view partial, multi-view partial, and full-3D training by testing partial-view models on full-shape data.Single-view partial uses one sampled view, while multi-view partial uses four sampled views per object.
  • Experimental comparison: The results in Table 10 verify the comparison between 2.5D and 3D training, while single-view ground truth still relies on 3D annotation.PointNet++ and DGCNN are both included in the comparison.
  • Qualitative analysis: Qualitative figures organize predictions by semantic object category and affordance category across full-shape, partial-view, and two rotation-invariant settings.The rows include ground truth, full-shape estimates, partial-view estimates, z/z rotation-invariant estimates, and SO(3)/SO(3) rotation-invariant estimates.
  • Annotation context: The annotation interface provides a 3D GUI for observing and manipulating shape geometry during affordance labeling.The interface supports translation, rotation, scaling, guided keypoint selection, and supported-affordance inspection.
Loading 2103.16397v2…