PaperlayerView exact revision on arXiv

Source-linked AI summary

AffordanceNet: An End-to-End Deep Learning Approach for Object Affordance Detection

Thanh-Toan Do, Anh Nguyen, Ian Reid

arXiv:1709.07326v3cs.CVcs.RO

TL;DR

AffordanceNet addresses the need to detect object locations, categories, and pixel-level affordances together for robotic interaction. It uses an end-to-end two-branch network with deconvolutional layers, robust mask resizing, and multi-task learning, achieving state-of-the-art results on public datasets and approximately 150ms inference per image.

  • Problem

    Robotic affordance detection must handle abstract multiclass pixel labels while also localizing and recognizing relevant objects efficiently.

  • Method

    AffordanceNet jointly detects object locations and classes and assigns each object pixel an affordance label using an end-to-end two-branch network with deconvolution, robust resizing, and multi-task loss.

  • Results

    AffordanceNet achieves state-of-the-art public-dataset performance, including a F w β score of 73.35 on IIT-AFF and a 2.9% improvement over ED-RGBD on UMD.

  • Takeaways & Limitations

    The framework provides object locations, categories, and affordances end to end and runs at around 150ms per image, supporting robotic applications.

Abstract

from arXiv · show

We propose AffordanceNet, a new deep learning approach to simultaneously detect multiple objects and their affordances from RGB images. Our AffordanceNet has two branches: an object detection branch to localize and classify the object, and an affordance detection branch to assign each pixel in the object to its most probable affordance label. The proposed framework employs three key components for effectively handling the multiclass problem in the affordance mask: a sequence of deconvolutional layers, a robust resizing strategy, and a multi-task loss function. The experimental results on the public datasets show that our AffordanceNet outperforms recent state-of-the-art methods by a fair margin, while its end-to-end architecture allows the inference at the speed of 150ms per image. This makes our AffordanceNet well suitable for real-time robotic applications. Furthermore, we demonstrate the effectiveness of AffordanceNet in different testing environments and in real robotic applications. The source code is available at https://github.com/nqanh/affordance-net

I. INTRODUCTION

Affordance detection models object functionality at pixel level to support robotic interaction, but must handle abstract, multiclass labels while also localizing objects efficiently. AffordanceNet addresses this need with an end-to-end architecture that jointly detects objects and affordances.

  • I. INTRODUCTION: Pixel-level affordances represent groups of pixels sharing the same object functionality.This formulation avoids requiring additional human-interaction information and connects affordance detection with semantic segmentation.
  • I. INTRODUCTION: Affordance labels can assign the same functionality to object parts with different appearances, making detection harder than classical semantic segmentation.The method must also run in real time and generalize to unseen objects.
  • I. INTRODUCTION: Robotic tasks require both affordance recognition and object localization and classification, such as identifying a bottle and bowl before pouring.A prior sequential two-network approach was time-consuming during testing.
  • I. INTRODUCTION: AffordanceNet overcomes the sequential design by jointly detecting objects and affordances in an end-to-end architecture.The framework builds on shared detection and segmentation features.

II. RELATED WORK

Prior affordance research used hand-designed features, RGB-D data, human demonstrations, or sequential deep networks. AffordanceNet instead jointly predicts object information and multiple pixel-level affordance classes in an end-to-end framework.

  • II. RELATED WORK: Affordance understanding is called object part labelling in computer vision and affordance detection in robotics.The robotics literature focuses more on real-world objects with which robots can interact.
  • II. RELATED WORK: Earlier methods inferred affordances from 3D point clouds, global object poses, local appearances, or object-action interactions from demonstrations.Other work detected affordances at pixel level using hand-designed geometric features and RGB-D images.
  • II. RELATED WORK: Deep learning methods improved affordance detection, including approaches using RGB or RGB-D inputs, multi-scale CNNs, weak supervision, and object detectors.A prior object-detector-based method still used two deep networks sequentially.
  • II. RELATED WORK: AffordanceNet jointly detects object locations, object labels, and associated affordances using an end-to-end architecture.Unlike binary instance-segmentation masks, it predicts multiple affordance classes within each object.

B. AffordanceNet Architecture

AffordanceNet combines shared region-based feature extraction with separate object and affordance branches. RoIAlign preserves alignment, while high-resolution deconvolutional processing addresses multiclass affordance masks.

  • B. AffordanceNet Architecture: AffordanceNet begins with a VGG backbone and RPN, then uses RoIAlign to convert each proposal into a fixed 7 × 7 feature map.The object branch predicts location and category, while the affordance branch outputs a multiclass mask through convolutional-deconvolutional layers and softmax.
  • B. AffordanceNet Architecture: RoIAlign replaces RoIPool’s coordinate rounding with bilinear interpolation, reducing misalignment between regions of interest and extracted features.This alignment is important for pixel-level prediction.
  • B. AffordanceNet Architecture: Small fixed masks used by instance-segmentation networks are inadequate when each object contains multiple affordance classes.AffordanceNet therefore uses deconvolutional layers to produce a higher-resolution affordance mask.
  • B. AffordanceNet Architecture: Three deconvolutional layers upsample the RoI feature map from 7 × 7 to 244 × 244.The intermediate map reaches 30 × 30 before the final high-resolution output.

3) Robust Resizing Affordance Mask:

The training mask is resized to a fixed resolution while preserving multiple affordance labels. A value-based remapping strategy thresholds resized pixels near the original label values rather than using one foreground threshold.

  • 3) Robust Resizing Affordance Mask:: Resizing an RoI groundtruth mask to 244 × 244 produces values between 0 and 1, but one threshold cannot reliably separate multiple affordance classes.The problem arises because each object may contain several affordance labels.
  • 3) Robust Resizing Affordance Mask:: The resizing strategy retains a pixel label when its resized value ρ(x, y) lies within α of a label value p̂, and otherwise assigns zero.The hyperparameter α is set to 0.005 in the experiments.
  • 3) Robust Resizing Affordance Mask:: After thresholding, the method remaps retained values to the original affordance labels to form the target training mask.An alternative separately resizes each affordance label, but the paper describes that strategy as less practical.

4) End-to-End Architecture:

AffordanceNet jointly detects objects and their pixel-level affordances through shared features and two coordinated branches. Its multi-task formulation combines object classification, box localization, and multiclass affordance prediction.

  • Network structure: The network uses a VGG16 backbone, RPN-generated RoIs, and RoIAlign features for two branches: object detection and affordance detection.The detection branch localizes and classifies objects, while the affordance branch predicts pixel-level functionality within each RoI.
  • Affordance prediction: The affordance branch outputs a softmax distribution over C+1 affordance labels, including background, for every pixel inside each RoI.
  • Multi-task loss: The multi-task loss jointly trains object classification, bounding-box localization, and affordance-map prediction.The three components are denoted Lcls, Lloc, and Laff.
  • Training targets: Each RoI is supervised with an object class, bounding-box offset, and affordance mask formed by intersecting the RoI with its associated groundtruth mask.Pixels outside the intersection are labeled background, and the target mask is resized to 244 × 244.
  • Loss components: Classification uses multinomial cross entropy, localization uses Smooth L1 loss, and affordance prediction uses multinomial cross entropy.The affordance loss is defined over pixels in the RoI, with si representing the softmax output for the true label and N the number of pixels.
  • Loss components: Localization and affordance losses apply only to positive RoIs, whereas classification loss applies to both positive and negative RoIs.

D. Training and Inference

The network is trained end to end with stochastic gradient descent and a scheduled learning rate. At inference, detected objects are filtered, their affordance masks are predicted, resized, and resolved for overlaps.

  • Training: Training uses stochastic gradient descent with 0.9 momentum and 0.0005 weight decay for 200k iterations on a Titan X GPU.The learning rate is 0.001 for the first 150k iterations and reduced by 10 for the final 50k.
  • Training: Input images are resized so the shorter edge is 600 pixels and the longer edge is capped at 1000 pixels.
  • Inference: Inference selects the top 1000 RPN RoIs, applies non-maximum suppression, and retains detections with classification scores above 0.9.If none exceed the threshold, the highest-scoring box is retained.
  • Inference: For each detected object, the affordance branch assigns the highest-probability class per pixel and resizes the predicted 244×244 mask to the object-box size.Overlapping objects are resolved using affordance priority.

A. Dataset and Baseline

The evaluation uses IIT-AFF and UMD, covering real-world or cluttered imagery and RGB-D object scenes, with affordance performance compared against segmentation, detector-based, and geometric baselines.

  • Datasets: IIT-AFF contains 8,835 real-world images, including ImageNet images and author-collected cluttered scenes.
  • Datasets: IIT-AFF includes 10 object categories, 9 affordance classes, 14,642 object boxes, and 24,677 pixel-level affordance regions.The standard split uses 70% of the data for training and 30% for testing.
  • Datasets: UMD contains around 30,000 RGB-D images of kitchen, workshop, and garden objects, with 7 affordance classes and 17 object categories.The images were captured in clutter-free setups, and only RGB images are used here.
  • Baselines: Evaluation uses the Fwβ metric and compares AffordanceNet with DeepLab, encoder-decoder CNNs, detector-based CNNs, CRF variants, and geometric-feature methods.

B. Results

AffordanceNet improves affordance detection across the IIT-AFF and UMD datasets while jointly producing object detections and affordance predictions without extra post-processing.

  • IIT-AFF Dataset: 73.35 F wβ score improves 3.7% over BB-CNN-CRF on IIT-AFF.AffordanceNet also achieves the best results for all 9 affordance classes.
  • IIT-AFF Dataset: Object-detector-based methods significantly outperform deep networks alone on cluttered IIT-AFF scenes.The comparison includes AffordanceNet and BB-CNN versus DeepLab and ED-RGB.
  • UMD Dataset: AffordanceNet achieves the highest average results on UMD, outperforming ED-RGBD by 2.9%.UMD contains clutter-free scenes, so the improvement is smaller than on IIT-AFF; AffordanceNet uses RGB only, whereas ED-RGBD also uses depth.
  • UMD Dataset: Deep learning methods significantly outperform hand-designed geometric approaches on UMD.The comparison is between AffordanceNet, DeepLab, and ED-RGB versus HMP and SRF.
  • Overall Results: AffordanceNet reaches state-of-the-art results without extra post-processing or data augmentation.Its end-to-end outputs include object locations, object categories, and object affordances.

C. Effect of Affordance Map Size

Larger affordance maps improve multiclass affordance detection, although gains diminish as resolution increases and network size grows.

  • Map-Size Comparison: Affordance detection accuracy increases as the affordance map becomes larger.The experiments compare networks producing 14 × 14, 28 × 28, and higher-resolution affordance maps.
  • Map-Size Comparison: The 14 × 14 AffordanceNet performs poorly because its map is too small to represent multiclass affordances.Accuracy improves significantly with a 28 × 28 map.
  • Map-Size Comparison: Accuracy gains slow as larger mask sizes are used.Thus, improvement is not linear with affordance map size.
  • Design Choice: AffordanceNet uses a 244 × 244 map because it balances accuracy with training feasibility on a Titan X GPU.Larger maps improve accuracy but increase the number of network parameters.
  • Convolutional Layers: Additional convolutional layers improve accuracy but still require upsampling to a high-resolution affordance map.The AffordanceNet14 6Conv variant adds six convolutional layers before deconvolutional upsampling.

D. Affordance Detection in The Wild

AffordanceNet generalizes beyond benchmark images to simulated and artistic inputs, and its outputs support real-time humanoid-robot manipulation.

  • Generalization: AffordanceNet successfully detects objects and affordances in artwork images and Gazebo simulation images.These results are qualitative demonstrations of generalization to other testing environments.
  • Robotic Applications: In the WALK-MAN pouring task, object detections localize the bottle and pan, while affordance predictions indicate grasping and pouring locations.The two branches provide complementary visual information for task execution.
  • Robotic Applications: 150ms per image makes AffordanceNet suitable for robotic applications.The system detects both objects and their affordances at this speed.
  • Conclusion: The framework simultaneously detects objects and their affordances using an end-to-end architecture.The paper identifies deconvolutional layers, robust resizing, and a new loss function as key components for multiclass affordance detection.
Loading 1709.07326v3…