PaperlayerView exact revision on arXiv

Source-linked AI summary

AffordanceLLM: Grounding Affordance from Vision Language Models

Shengyi Qian, Weifeng Chen, Min Bai, Xiong Zhou, Zhuowen Tu, Li Erran Li

arXiv:2401.06341v2cs.CVcs.RO

TL;DR

Affordance grounding requires locating where to interact with an object, but the task draws on scene and interaction knowledge that limited training labels do not capture. AffordanceLLM combines pretrained vision-language knowledge with depth-based geometry, and the authors report strong AGD20K results and generalization to Internet images, including some novel actions.

  • Problem

    Affordance grounding requires understanding object parts, 3D geometry, functionality, actions, and scene context, much of which lies beyond limited training labels.

    Source: §1 · p. 1

  • Method

    AffordanceLLM builds on a pretrained VLM, adds an affordance-map decoder, and supplies depth maps alongside RGB to incorporate 3D geometry.

    Source: §1 · pp. 1–2

  • Results

    AffordanceLLM outperforms state-of-the-art baselines on AGD20K and produces reasonable maps for Internet images, with some evidence of generalization to novel objects and actions.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors identify potential use in helping intelligent robots manipulate in-the-wild objects.

    Source: §5 · p. 8

    The method can fail on ambiguous action questions and sometimes selects the wrong object in images containing multiple objects.

    Source: §4.7 · p. 8

Abstract

from arXiv · show

Affordance grounding refers to the task of finding the area of an object with which one can interact. It is a fundamental but challenging task, as a successful solution requires the comprehensive understanding of a scene in multiple aspects including detection, localization, and recognition of objects with their parts, of geo-spatial configuration/layout of the scene, of 3D shapes and physics, as well as of the functionality and potential interaction of the objects and humans. Much of the knowledge is hidden and beyond the image content with the supervised labels from a limited training set. In this paper, we make an attempt to improve the generalization capability of the current affordance grounding by taking the advantage of the rich world, abstract, and human-object-interaction knowledge from pretrained large-scale vision language models. Under the AGD20K benchmark, our proposed model demonstrates a significant performance gain over the competing methods for in-the-wild object affordance grounding. We further demonstrate it can ground affordance for objects from random Internet images, even if both objects and actions are unseen during training. Project site: https://jasonqsy.github.io/AffordanceLLM/

1. Introduction

Affordance grounding is difficult to generalize because it requires understanding object geometry, functionality, actions, and scene context beyond limited training labels. AffordanceLLM uses pretrained vision-language knowledge and 3D geometry, and the paper reports improved performance on AGD20K and generalization to Internet images.

1. Introduction

  • Grounding affordance from a single image is challenging because it requires understanding object parts, 3D geometry and functionality, actions, human-object interaction, and spatial configuration beyond limited heatmap labels.Prior approaches trained with human demonstrations perform well on seen objects and actions but struggle on novel objects.Source: §1 · p. 1
  • Pretrained VLMs supply world and human-object-interaction knowledge; for example, LLaVA answers a motorcycle-riding question by identifying the handlebars.The authors argue that this knowledge may help affordance models generalize beyond limited demonstrations.Source: §1 · pp. 1–2, Figure 2 · p. 2
  • AffordanceLLM introduces 3D geometry because object shape can indicate functionality, such as cylindrical forms for grasping and flat surfaces for sitting or lying.The paper proposes relating geometry to actions to facilitate generalization across visual appearance variations.Source: §1 · p. 2
  • AffordanceLLM extends a LLaVA backbone with a mask decoder and special token to predict affordance maps, and adds depth maps alongside RGB input.The model is trained end-to-end; the authors report that both designs significantly improve performance.Source: §1 · p. 2
  • On AGD20K, AffordanceLLM outperforms all state-of-the-art baselines by a large margin and produces reasonable maps for varied Internet objects, with some evidence of novel-action generalization.The paper also reports generalization to novel object categories.Source: §1 · p. 2

2. Related Work

Related work connects vision-language knowledge and 3D cues to affordance grounding, while tracing the development of affordance datasets and applications. AffordanceLLM applies VLM world knowledge to this setting and uses AGD20K as its primary benchmark.

  • VLMs use visual-language knowledge for open-world detection and segmentation, while multimodal LLMs support reasoning in vision, 3D scenes, and robotics; AffordanceLLM applies this approach to affordance grounding.The cited work describes visual-language correspondence as world knowledge and identifies affordance grounding as a novel setting for transferring VLM knowledge.Source: §2 · p. 2
  • AGD20K is the first large-scale affordance dataset, and LOCATE is its state-of-the-art approach; the paper uses AGD20K to compare against prior methods.Prior work also extended affordance grounding to scene understanding, 3D models, egocentric video, hand-pose generation, and human-part association.Source: §2 · p. 3
  • Object-part 3D shape can hint at utility and interaction, and the paper uses DPT-generated pseudo-depth maps to provide 3D information to VLMs.The related work cites depth estimation, surface-normal estimation, 3D reconstruction, and few-image NeRF as ways to estimate 3D from images.Source: §2 · p. 3
  • Affordance learning is connected to manipulation as a possible solution, although manipulation itself is not the paper’s goal.Prior robotics methods address tabletop objects and mobile manipulation.Source: §2 · p. 3

3. Approach

AffordanceLLM maps an image and an interaction query to an affordance map by decoding a special LLM token, with pseudo-depth added as a 3D input. Its architecture replaces LLaVA’s standard image encoder and adapts the decoder for higher-resolution output.

3.1. Overview

  • LLaVA-7B encodes the image and interaction prompt into features for the LLM, using a query template asking which object part to interact with for an action.Source: §3 · p. 3, §3.1 · p. 3
  • The model trains an LLM to predict a special mask token whose hidden state becomes a query for decoding a dense affordance map.The query and image features are inputs to the decoder.Source: §3.1 · p. 3, §3 · p. 3
  • DPT generates a pseudo-depth map that the same image encoder converts to depth features, concatenated with image features as additional LLM input.Source: §3.1 · p. 4
  • Training combines binary focal affordance loss and text-output cross-entropy, with λ set to 0.01 to balance the losses.Positive and negative affordance examples are weighted 0.95 and 0.05, respectively.Source: §3.1 · p. 4

3.2. Network Architecture

  • OWL-ViT replaces CLIP for higher-resolution, location-aware image features, while token projection and an added transposed convolution adapt the encoder and decoder.OWL-ViT uses 768×768 input resolution versus CLIP’s 224×224; neighboring tokens are concatenated after projection, and the added layer raises decoder output resolution.Source: §3.2 · p. 4
  • The implementation initializes from pretrained Llama-7B weights, freezes the image encoder during projection pretraining, then tunes the language model and the full model on AGD20K.Source: §3.2 · p. 4

4. Experiments

The experiments benchmark AffordanceLLM against state-of-the-art methods and probe prompts, visual encoders, and depth. They use a harder AGD20K split to test generalization to less-similar unseen objects.

4.1. Experimental Setup

  • The evaluation uses KLD, SIM, and NSS and compares weakly supervised methods with fully supervised 3DOI and adapted LOCATE baselines.Source: §4.1 · pp. 4–5

4.2. Dataset

  • To address similar train/test counterparts in the original Unseen split, the authors create a Hard split with disjoint classes and generally less-similar test objects; it is harder than Easy.Source: §4.2 · p. 6

4.3. Results

  • On AGD20K’s Hard split, AffordanceLLM beats all baselines by a large margin; on Easy it is comparable to LOCATE and significantly outperforms LOCATE-Sup on both splits.Source: Table 3 · p. 6, §4.3 · p. 7
  • On the Hard split, LOCATE tends to cover whole objects, 3DOI often selects small or incorrect regions, while AffordanceLLM gives the best-quality maps.Source: Figure 4 · p. 5, §4.3 · p. 7

4.4. Ablation

  • A complete-question Full prompt outperforms simpler prompts; the authors say correct prompts typically focus predictions and may activate affordance knowledge in the VLM.Source: §4.4 · p. 7, Figure 5 · p. 7

4.5. Pseudodepth as Inputs

  • With DPT-generated pseudodepth, the model predicts better affordance maps, supporting the value of 3D information in affordance reasoning.Source: §4.5 · p. 8

4.6. Generalization to Internet Images

  • On random Internet images, AffordanceLLM handles novel objects and novel actions, while LOCATE cannot infer novel actions.Source: §4.6 · p. 8, Figure 6 · p. 8

4.7. Failure Examples

  • Failures include confusing the intended object in multi-object scenes and interpreting “cut with” as holding a knife handle rather than using its blade.Source: §4.7 · p. 8, Figure 7 · p. 8

5. Conclusion

AffordanceLLM grounds affordances for in-the-wild objects unseen during training and generalizes better than state-of-the-art affordance grounding models. The authors note both potential benefits for robot manipulation and risks from adversarial misuse.

5. Conclusion

  • AffordanceLLM generalizes better to in-the-wild objects than state-of-the-art affordance grounding models.The conclusion describes grounding affordances for objects unseen during training.Source: §5 · p. 8
  • The approach could help build intelligent robots that manipulate in-the-wild objects.Source: §5 · p. 8
  • Adversarial misuse could cause physical damage or harm.Source: §5 · p. 8

A. Metrics

The section describes KLD, SIM, and NSS as metrics for evaluating the model. They assess differences or correspondence between predicted and ground-truth affordance maps.

A. Metrics

  • KLD measures distribution difference between the predicted affordance map and the ground truth.Source: A. Metrics · p. 12
  • SIM, also called histogram intersection, measures intersection between predicted and ground-truth affordance maps on a 0-to-1 range.Source: A. Metrics · p. 12
  • NSS measures correspondence between the prediction map and the ground truth.Source: A. Metrics · p. 12

B. Details of the Data Splits

The data-split details specify class and image counts for easy and hard AGD20K settings, with separate counts for dense and weak supervision.

B. Details of the Data Splits

Loading 2401.06341v2…