PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Act Properly: Predicting and Explaining Affordances from Images

Ching-Yao Chuang, Jiaman Li, Antonio Torralba, Sanja Fidler

arXiv:1712.07576v2cs.CV

TL;DR

The paper studies how an agent can identify safe and socially appropriate action-object affordances from an egocentric scene, including exceptions and their likely consequences. It builds an annotated dataset and a graph-based model; evaluations report benefits from contextual graph information and multi-task models comparable to independent models.

  • Problem

    Agents need to identify objects where an action is safe and appropriate, explain scene-specific exceptions, and anticipate their likely consequences.

    Source: §1 · p. 1

  • Method

    The authors build an ADE20k-based dataset with affordance, exception, explanation, and consequence annotations, and use a graph neural network to reason over objects and their spatial relations.

    Source: §1 · p. 1, §4.2 · p. 5, §6 · pp. 8–9

  • Results

    Context aggregation improves relationship prediction over Unaries, the spatial-relation graph uses context better than fully connected and chain graphs, and shared multi-task models perform comparably to independent models, sometimes surpassing them.

    Source: §5.1 · pp. 6, 8

  • Takeaways & Limitations

    Evaluation highlights the value of contextual graph information while identifying affordance reasoning as a task with challenges going forward.

    Source: §5.1 · p. 6, §6 · p. 9

    The dataset's size limits the diversity of generated explanations and consequences, leading the model to describe similar situations with similar sentences.

    Source: §5.1 · p. 7

Abstract

from arXiv · show

We address the problem of affordance reasoning in diverse scenes that appear in the real world. Affordances relate the agent's actions to their effects when taken on the surrounding objects. In our work, we take the egocentric view of the scene, and aim to reason about action-object affordances that respect both the physical world as well as the social norms imposed by the society. We also aim to teach artificial agents why some actions should not be taken in certain situations, and what would likely happen if these actions would be taken. We collect a new dataset that builds upon ADE20k, referred to as ADE-Affordance, which contains annotations enabling such rich visual reasoning. We propose a model that exploits Graph Neural Networks to propagate contextual information from the scene in order to perform detailed affordance reasoning about each object. Our model is showcased through various ablation studies, pointing to successes and challenges in this complex task.

1. Introduction

The paper frames affordance reasoning as deciding which actions are safe and socially appropriate in a dynamic scene, and explaining exceptions and their likely consequences. It introduces a dataset and a graph-based model for this task.

1. Introduction

  • Physical constraints and social norms both shape affordances, which depend on the complete dynamic scene rather than a simple knowledge base.Examples include avoiding obstacles, leaving a purse-occupied chair unused, and not running during a lecture.Source: §1 · p. 1
  • The goal is to identify scene objects where a chosen action is safe and appropriate, and explain exceptions and their likely consequences.The reasoning takes an egocentric view, treating the photo as the agent’s current view.Source: §1 · p. 1
  • The ADE20k-based dataset contains 20,000 diverse-scene images and annotations for typical action-object pairs, exceptions, explanations, and likely consequences.Exceptions include cases that are not safe, not physically possible, or socially awkward.Source: §1 · pp. 1–2
  • A graph neural network propagates information among adjacent object nodes to predict affordances, explanations, and consequences for each object.The graph is built from instance-level segmentation; an RNN language model generates explanations and consequences.Source: §1 · p. 2
  • The paper uses ablation studies to investigate which information matters for affordance reasoning, while noting the task’s successes and challenges.The dataset is reported as publicly available.Source: §1 · p. 2

2. Related Work

The section reviews affordance reasoning, visual description generation, and graph neural networks. It contrasts prior work focused on typical affordances or image-level descriptions with scene-dependent, object-level reasoning.

2. Related Work

  • Most prior affordance approaches [26] treat relations as static, whereas this work explains scene-dependent physical and social exceptions.The section distinguishes reasoning over the whole scene from inferring object state alone.Source: §2 · p. 2
  • Image captioning describes whole images, while this work generates per-object explanations and exploits dependencies between objects.The section also notes explanation generation for classification decisions [13] and graph-based explanations for human interactions [27].Source: §2 · p. 2
  • Prior vision work uses GGNNs for situation prediction and semantic segmentation, while this work models object dependencies to produce affordances and explanations.The paper adopts Gated Graph Neural Networks with GRU-like propagation.Source: §2 · p. 2

3. ADE-Affordance Dataset

ADE-Affordance adds action-object affordance annotations to ADE20K imagery. Its collection pipeline moves from typical action-object pairs to image-level exceptions and their explanations and likely consequences.

3. ADE-Affordance Dataset

  • ADE-Affordance builds on ADE20K’s densely instance-annotated images, which span 900 scenes from varied indoor and outdoor settings.Source: §3 · p. 2

3.1. Dataset Collection

  • The knowledge base covers sit, run, and grasp using typical actions gathered for 150 ADE20K object classes, but may omit classes outside that vocabulary.Source: §3.1 · p. 3
  • Annotations distinguish positive affordances, firm class-level negatives, and five instance-specific exception types for otherwise applicable action-object pairs.Source: §3.1 · p. 3
  • Workers mark positive objects and exceptions, and may add objects missed by the knowledge base, producing labels for why an action should not be taken.Source: §3.1 · p. 3
  • Exception instances receive a natural-language explanation and likely consequence, such as the social prohibition against grasping another person’s purse.Source: §3.1 · p. 3
  • Test images have three annotators per image, compared with one for train and validation, to support more robust evaluation of this subjective task.Source: §3.1 · p. 3

3.2. Dataset Analysis

  • Run exceptions occur in more than 50% of images; socially awkward exceptions are higher for sit and grasp, while run has more physical obstacles.Source: §3.2 · p. 4
  • Sentence statistics show sit explanations and consequences are more complex than run and grasp; the split contains 8,011 training, 1,000 validation, and 1,000 test images.Source: §3.2 · p. 4

4. Affordance Reasoning with Graph Neural Models

The section formalizes affordance reasoning as predicting scene-dependent action–object relationships and, for exceptions, explaining why an action should not be taken and what may follow. It proposes Spatial GGNN, which uses spatial relations among scene objects to encode context for this reasoning.

4.1. Problem Definition

  • The task predicts affordances for each action–object-instance pair, including the relationship, exception reasons, and likely negative consequences.Relationships distinguish possible actions, actions that cannot be taken under any condition, and special circumstances that prevent an action.Source: §4.1 · pp. 4–5

4.2. Our Model

  • Spatial GGNN represents image objects as graph nodes and connects adjacent objects whose boundaries touch, using spatial relations to encode scene context.The model adopts GGNN propagation, which updates node states from their prior states and messages received from neighboring nodes.Source: §4.2 · p. 5
  • Each node’s initial state combines object-class information with image features extracted from the object’s masked image region.The class may be ground truth or predicted by an instance-segmentation network; object features come from ResNet-50.Source: §4.2 · p. 5
  • At each propagation step, nodes aggregate neighbors’ hidden states through a shared linear layer and update their states with a GRU-like gate.After T steps, the model uses each node’s resulting hidden state.Source: §4.2 · p. 5
  • For each object, the output combines its propagated state, initial state, and a global image feature; negative relationships also trigger generation of an explanation and consequence.The global feature comes from ResNet-50 applied to the full image, while a captioning RNN generates the two sentences.Source: §4.2 · pp. 5–6

5. Experiments

The experiments evaluate affordance prediction and language generation with ground-truth and inferred segmentation, then test contextual modeling choices and joint training. Results favor spatially structured GGNNs on several tasks, while also revealing limits in segmentation and sentence diversity.

5. Experiments

  • The experiments report relationship mean accuracy metrics and captioning metrics, select models on validation accuracy, and evaluate on the test set.Training uses ground-truth segmentation-derived graphs, with separate models for each task except in the joint-training experiment.Source: §5 · p. 6, §5.1 · p. 6

5.1. Results

  • Neighbor aggregation improves relationship prediction over Unaries, and the spatial graph uses context better than fully connected and chain graphs.The evaluation uses ground-truth segmentation.Source: §5.1 · p. 6, Table 4 · p. 7
  • Qualitative examples show spatial context helping exception prediction, while generated explanations and consequences can depend on inter-object context.Figure 7 illustrates relationship predictions; Figure 8 shows explanation and consequence generation with ground-truth segmentation.Source: §5.1 · pp. 6–7, Figure 7 · p. 7, Figure 8 · p. 8
  • GGNN structured models consistently outperform other baselines for explanation generation across actions, with spatial information improving performance in various settings.The evaluation uses ground-truth instance segmentation.Source: §5.1 · p. 7, Table 6 · p. 8
  • For consequence generation, the model outperforms baselines on most metrics, but some baselines also perform well; contextual information helps for grasp.The authors suggest uncertain ground-truth consequences may not always depend on neighboring objects.Source: §5.1 · p. 7, Table 7 · p. 8
  • Performance saturates around three propagation steps; removing object-class information harms prediction most, while the effect of removing other features depends on the model.Without global image features, GGNN still predicts successfully while Unaries performance decreases; removing object features largely harms Unaries.Source: §5.1 · p. 7, Table 9 · p. 8
  • With real segmentation results, the model achieves reasonable performance; accuracy evaluation focuses only on objects among the 60 of 150 ADE classes predicted by its instance-segmentation model.Source: Table 5 · p. 7, §5.1 · p. 8
  • Joint models using single-action or multi-action parameter sharing perform comparably to independent models and sometimes surpass them, saving inference time.The comparison uses ground-truth instance segmentation.Source: §5.1 · p. 8, Table 8 · p. 8

6. Conclusion

The conclusion presents action-object affordance reasoning from images as a scene-dependent task, addressed with a new annotated dataset and a graph-based model. The evaluation identifies both demonstrated capabilities and challenges ahead.

6. Conclusion

  • The paper targets identifying objects to which an action applies while accounting for scene-dependent physical and implied social constraints.Source: §6 · p. 8
  • ADE-Affordance builds on ADE20K and annotates object exceptions with types, explanations, and likely consequences.Source: §6 · pp. 8–9
  • The proposed model performs visual reasoning from raw images using neural networks defined on graphs, and evaluation points to challenges ahead.Source: §6 · p. 9
Loading 1712.07576v2…