PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Affordance Grounding from Exocentric Images

Hongchen Luo, Wei Zhai, Jing Zhang, Yang Cao, Dacheng Tao

arXiv:2203.09905v1cs.CV

TL;DR

Diverse human-object interactions make it difficult to link object parts to affordances. This paper transfers affordance knowledge from exocentric interactions to egocentric images, and reports that its method outperforms representative models on AGD20K.

  • Problem

    Diverse interactions make it difficult to connect objects’ intrinsic properties with affordance regions, although understanding how objects can be used matters for applications such as robot grasping and action prediction.

    Source: §1 · pp. 1–2

  • Method

    A cross-view framework learns affordance-specific features from exocentric interactions and transfers them to egocentric object images using affordance-label supervision.

    Source: §1 · p. 2

  • Results

    The method outperforms representative models on AGD20K; on KLD, it improves 17.2% over the best saliency model.

    Source: §1 · p. 2, §5.2 · p. 7, §6 · p. 9

  • Takeaways & Limitations

    AGD20K provides a testbed for exocentric-view affordance grounding, and the method is presented as a strong baseline for future research.

    Source: §1 · p. 2, §6 · p. 9

    Predicted affordance maps may include intermediate background regions when multiple objects appear, or activate irrelevant background regions for slender objects.

    Source: §5.4 · p. 8

Abstract

from arXiv · show

Affordance grounding, a task to ground (i.e., localize) action possibility region in objects, which faces the challenge of establishing an explicit link with object parts due to the diversity of interactive affordance. Human has the ability that transform the various exocentric interactions to invariant egocentric affordance so as to counter the impact of interactive diversity. To empower an agent with such ability, this paper proposes a task of affordance grounding from exocentric view, i.e., given exocentric human-object interaction and egocentric object images, learning the affordance knowledge of the object and transferring it to the egocentric image using only the affordance label as supervision. To this end, we devise a cross-view knowledge transfer framework that extracts affordance-specific features from exocentric interactions and enhances the perception of affordance regions by preserving affordance correlation. Specifically, an Affordance Invariance Mining module is devised to extract specific clues by minimizing the intra-class differences originated from interaction habits in exocentric images. Besides, an Affordance Co-relation Preserving strategy is presented to perceive and localize affordance by aligning the co-relation matrix of predicted results between the two views. Particularly, an affordance grounding dataset named AGD20K is constructed by collecting and labeling over 20K images from 36 affordance categories. Experimental results demonstrate that our method outperforms the representative models in terms of objective metrics and visual quality. Code: github.com/lhc1224/Cross-View-AG.

1. Introduction

Affordance grounding must link object parts to action possibilities despite diverse human interactions. The paper proposes learning that link from exocentric interactions, transferring affordance knowledge to egocentric images, and evaluates the approach on AGD20K.

1. Introduction

  • Diverse interaction habits make it difficult to link object properties to affordances, while humans can infer invariant affordance regions from exocentric interactions.The example describes people holding a racket at different positions while observers identify its swingable region from the long handle.Source: §1 · pp. 1–2
  • The proposed task learns affordance knowledge from exocentric human-object interactions and egocentric object images using affordance labels alone as supervision.At test time, it predicts an affordance region from an egocentric object image and a particular affordance label.Source: §1 · p. 2
  • The cross-view framework uses AIM to extract affordance-specific cues from interaction diversity and ACP to preserve affordance correlations when localizing regions.AIM uses low-rank matrix decomposition to reduce intra-class differences; ACP aligns the correlation matrices of predictions from the two views.Source: §1 · p. 2
  • On AGD20K, the method outperforms state-of-the-art methods; the dataset contains 20,061 exocentric and 3,755 egocentric images across 36 affordance categories.The authors present AGD20K as a benchmark for affordance grounding from exocentric views.Source: §1 · p. 2

2. Related Works

Prior work grounds affordances, builds datasets, and learns transformations between viewpoints, but this paper distinguishes its approach through exocentric-to-egocentric affordance transfer. Its related-work discussion contrasts the method and AGD20K with those earlier efforts.

2.1. Visual Affordance Grounding

  • Earlier affordance-grounding work maps object regions to affordances, including weakly supervised detection and video-based grounding; this paper transfers affordance knowledge from exocentric interactions to egocentric images.The cited video approach uses affordance labels, whereas this paper explicitly transfers knowledge determined by objects’ intrinsic properties.Source: §2.1 · pp. 2–3

2.2. Visual Affordance Dataset

  • AGD20K is contrasted with prior datasets for its larger scale, richer affordance and object categories, exocentric-to-egocentric viewpoint transformation, and part-level annotations.The cited datasets are described as limited in scale or category diversity, lacking human-action reasoning, or lacking part-level affordance labels.Source: §2.2 · p. 3

2.3. Learning View Transformations

  • Unlike prior view-transformation work for tasks such as action recognition and video summarization, this work transfers affordance knowledge amid interaction diversity and multiple affordance regions.The related work describes earlier approaches as generating perspective-invariant representations from paired data or distilling egocentric signals from exocentric datasets.Source: §2.3 · p. 3

3. Method

The method transfers affordance knowledge from exocentric interactions to egocentric object images using affordance-label supervision, with separate modules for mining invariant features and preserving cross-view correlations.

3. Method

  • Table 1 summarizes related and AGD20K datasets by part-level and high-quality annotation and by object, affordance, and image counts.Source: Table 1 · p. 3
  • Training transfers affordance knowledge from exocentric interactions to egocentric images using affordance-label supervision; testing predicts a region from an egocentric image and affordance label.The framework uses an exocentric branch and an egocentric branch, aligning their pooled features before prediction.Source: §3 · p. 3

3.1. Affordance Invariance Mining Module

  • NMF factors nonnegative exocentric features into a shared dictionary W and per-image coefficients H, reconstructing features for residual addition to the inputs.The initial dictionary is updated using the mini-batch average to capture common interaction subfeatures.Source: §3.1 · p. 4

3.2. Affordance Co-relation Preserving Strategy

  • ACP aligns class-correlation matrices from the two branches with cross-entropy, while the total loss balances classification, ACP, and knowledge-transfer terms.T controls attention to correlations between negative labels.Source: §3.2 · p. 5

3.3. Inference

  • At inference, class activation mapping combines last-layer feature maps to produce the affordance heatmap from an egocentric image and affordance label.Table 2 reports comparisons for seen and unseen object categories; Table 3 covers ablations of AIM, ACP, and LKT, without values supplied here.Source: §3 · p. 3, §3.3 · pp. 5–6, Table 2 · p. 6

4. Dataset

AGD20K combines exocentric interaction images and egocentric object images across 36 affordance classes. Its category statistics show a multi-to-multi relationship between affordance and object categories, challenging affordance grounding.

4. Dataset

  • Exocentric images come mainly from HICO and COCO, with 2,112 additional images collected from free-license websites; 3,755 egocentric images were collected online.Ambiguous interactions were manually removed from the selected exocentric images.Source: §4 · p. 6
  • The dataset covers 36 commonly used affordance classes, with exocentric labels assigned from human-object interactions and egocentric labels assigned by object class.Source: §4 · p. 6
  • Interaction regions are annotated as points, with denser points marking more frequent interactions, and heatmaps are derived from those points.Source: §4 · p. 6
  • AGD20K’s category statistics show a multi-to-multi relationship between affordance and object categories, posing a significant grounding challenge.The dataset also spans a wide range of affordance and object categories in diverse scenarios.Source: §4 · p. 6

5. Experiments

The method leads across Seen and Unseen settings on KLD, with qualitative, category, scale, and source analyses extending the comparison. Ablations favor ACP’s correlation preservation, while limitations remain for multi-object and slender-object scenes.

5.1. Benchmark Setting

  • Evaluation uses KLD, SIM, and NSS across Seen and Unseen splits, against saliency, weakly supervised localization, and affordance-grounding models.Seen training and test sets share object categories; Unseen sets do not.Source: §5.1 · pp. 6–7

5.2. Quantitative and Qualitative Comparisons

  • KLD gains were 17.2%, 16.5%, and 13.3% over best saliency, WSOL, and affordance models in Seen; Unseen gains were 10.2%, 15.1%, and 10.4%, respectively.The evaluation compares Seen settings with overlapping training and test object categories against Unseen settings with no category overlap.Source: §5.2 · p. 7
  • Heatmaps show more accurate outputs across Seen and Unseen; examples span “Sit on” across beds and chairs and distinct “Open,” “Hold,” and “Cut with” regions.Figure 5 presents Hotspots, EIL, and DeepGazeII predictions alongside the affordance maps.Source: §5.2 · pp. 7–8, Figure 5 · p. 7

5.3. Ablation Study

  • AIM learns affordance-specific features from diverse interactions, while ACP improves more obviously than LKT.The ablation attributes a significant role in performance improvement to AIM and indicates stronger gains from preserving affordance correlation than from LKT.Source: §5.3 · p. 8
  • Performance is more affected by ACP’s T than dictionary rank r; increasing exocentric images from N=2 to 3 has a larger impact, while N=1 still beats most contenders.Figure 6 presents Seen and Unseen hyper-parameter studies for T, r, and N.Source: Figure 6 · p. 7, §5.3 · p. 8

5.4. Performance Analysis

  • The model exceeds other methods across representative classes with varied appearances, overlapping object categories but different affordance regions, and differently sized regions.Table 4 reports KLD for selected affordance categories.Source: §5.4 · p. 8, Table 4 · p. 8
  • The model outperforms all other methods on Big, Middle, and Small mask-to-image scale splits in both settings.Figure 7 reports NSS results for the three scale subsets.Source: §5.4 · p. 8, Figure 7 · p. 8
  • Using both exocentric and egocentric images improves most methods only modestly, yet the proposed method still surpasses all models.Table 5 compares Exo-only training with Exo&Ego training.Source: §5.4 · p. 8, Table 5 · p. 8

6. Conclusion

The paper presents exocentric-view affordance grounding, a cross-view transfer framework, and AGD20K. It reports that the model outperforms representative related models and can serve as a strong baseline.

6. Conclusion

  • The paper introduces affordance grounding from exocentric views and a framework to transfer invariant affordance knowledge from diverse interactions to egocentric views.Source: §6 · p. 9
  • AGD20K is a testbed for the task, containing 20K well-annotated images.Source: §6 · p. 9
  • The model outperforms representative models from related areas and is presented as a strong baseline for future research.Source: §6 · p. 9
  • The authors state that risky human demonstrations could provide negative guidance to agents and should be prohibited by strict legislation.Source: §6 · p. 9
Loading 2203.09905v1…