PaperlayerView exact revision on arXiv

Source-linked AI summary

AFFORDANCE20Q: Evaluating Affordance Reasoning from Physical Properties

Yifan Jiang, Meige Yang, Zitong Li, Jay Pujara

arXiv:2606.14240v1cs.AI

TL;DR

Existing affordance benchmarks often reveal object identities, making it difficult to assess reasoning from physical properties rather than object-affordance recall. AFFORDANCE20Q removes object identity in a 20-Questions benchmark, and experiments find a substantial human-performance gap while KARI improves open-source LLMs by up to 15.2 points.

  • Problem

    Existing affordance benchmarks often expose object identities, limiting direct evaluation of affordance reasoning from physical properties rather than object-affordance recall.

  • Method

    AFFORDANCE20Q has models identify hidden-object affordances through multi-turn yes/no questions about physical properties, while KARI induces knowledge-base-grounded affordance rules with LLMs.

  • Results

    Across 15 LLMs, performance remains substantially below humans by approximately 20 points, while KARI improves open-source LLMs by up to 15.2 points.

  • Takeaways & Limitations

    AFFORDANCE20Q exposes shortcomings in LLM affordance reasoning from physical properties and provides a benchmark for progress beyond memorized object-affordance mappings.

  • Takeaways & Limitations

    AFFORDANCE20Q is limited to a text-only question-answering setting rather than visual or richer contextual affordance reasoning.

Abstract

from arXiv · show

Affordance reasoning, the inference of an object's action possibilities from its physical properties (e.g., shape and material), is fundamental to human physical understanding and increasingly critical for Large Language Models (LLMs). However, existing affordance benchmarks largely expose explicit object identities in the evaluation setup, allowing models to rely on memorized object-affordance mappings rather than reasoning over physical properties. To address this gap, we introduce Affordance20Q, a novel affordance reasoning benchmark formulated as a 20-Questions game without exposing the object's identity. In each game, the model identifies a hidden object's affordance from a candidate set by asking yes/no questions about its physical properties. Affordance20Q comprises 1,009 games over 454 objects and 59 affordances, all manually filtered, refined, and annotated. We conduct comprehensive experiments with 15 state-of-the-art LLMs and find a substantial gap (~20 points) compared to human performance. A KL-based information-gain (IG) analysis further shows that models fail to ask discriminating questions as the game progresses. To close the gap, we develop KB-Anchored Rule Induction (KARI), a pipeline based on LLMs that generates affordance rules grounded in evidence from knowledge bases (KBs). KARI improves open-source LLMs by up to 15.2 points, while the limited coverage of KBs hinders further gains. We release all our code and data at https://github.com/1171-jpg/Affordance20Q.git

1 Introduction

AFFORDANCE20Q evaluates whether models can infer affordances from physical properties without object identities, separating physical reasoning from memorized object-affordance mappings. Across 1,009 games and 15 LLMs, it reveals a substantial human-performance gap and introduces KARI, which improves open-source models using knowledge-base-grounded rule induction.

  • Motivation: Existing affordance benchmarks expose object identities, allowing models to answer through memorized object-affordance mappings rather than physical-property reasoning.This setup conflates recall with reasoning from properties such as shape, material, and structure.
  • Benchmark: AFFORDANCE20Q frames affordance identification as a 20-Questions game over hidden objects, using yes/no questions about physical properties such as material and shape.The benchmark provides candidate affordances while withholding the object’s identity.
  • Evaluation: ~20 points separate 15 state-of-the-art LLMs from humans, with the strongest model reaching only 45.9%.KL-based information-gain analysis finds that models fail to ask discriminating questions as games progress.
  • Method: Up to 15.2 points improve open-source LLMs with KARI, which uses knowledge bases to inspire and post-hoc validate rules grounded in physical commonsense.Remaining performance shortfalls trace to the coverage limits of current commonsense knowledge bases.

2 Related Work

Prior affordance benchmarks span visual inputs grounded in object geometry, parts, or images, while 20-Questions research studies information-seeking through sequential yes/no questioning. DANCE20Q applies this game format to affordance reasoning and evaluates question effectiveness with a KL-based information-gain metric.

  • Affordance Reasoning Benchmarks: Affordance benchmarks have evaluated reasoning from 3D shape, part geometry, and 2D images across diverse input formats and modalities.Early vision benchmarks grounded affordances in 3D shape and part geometry, while later work used 2D image inputs.
  • 20-Questions Games and Active Question-Asking: 20-Questions games identify hidden targets through sequential yes/no questions within a fixed number of turns.The format originated in cognitive science for studying information-seeking behavior and has relevance to active question-asking in human-computer interaction.
  • Novelty: DANCE20Q is the first affordance-reasoning benchmark to adopt the 20-Questions game, aligning with Gibson’s active-perception view of affordance.The benchmark evaluates affordance reasoning through active questioning rather than only passive inputs.
  • Novelty: A KL-based information-gain metric evaluates how effectively models ask questions across game turns.The metric is designed to assess question effectiveness as questioning progresses.

3 AFFORDANCE20Q Construction

AFFORDANCE20Q frames affordance reasoning as a 20-turn game in which a Questioner identifies a hidden object’s affordance from eight candidates using questions about physical properties. A three-stage construction pipeline produces a manually refined benchmark of 454 objects, 59 affordances, and 1,009 balanced test games.

  • Game setup: Each game presents eight candidate affordances, including one possessed by the hidden object and seven distractors.The Questioner observes only the candidate set and asks yes/no questions about the hidden object’s physical properties.
  • Game setup: The game succeeds when the Questioner identifies the target affordance within T = 20 turns and fails after an incorrect guess or exhausted budget.A Checker validates questions, while an Oracle returns answers.
  • Collection pipeline: The benchmark’s three-stage pipeline filters unsuitable objects, refines affordance descriptions and object properties, then annotates every object–affordance pair.Filtering removes non-discrete physical objects and objects whose functions depend on hidden internal mechanisms.
  • Annotation quality: 85.2% majority agreement was achieved on 1,298 re-annotated pairs, with Fleiss κ = 0.82 against the released labels.Three additional annotators re-annotated a random subset of pairs.
  • Data statistics: The resulting dataset contains 454 objects, 59 affordances, and labels for every object–affordance pair, with 1,009 balanced object–target-affordance games sampled as the test set.The games are balanced across target affordances so that no single affordance dominates.

4 KB-Anchored Rule Induction

KARI combines external knowledge-base evidence with LLM reasoning to induce grounded, expressive affordance rules for the Questioner. Its rules encode physical conditions compositionally and are generated through iterative proposal, auditing, and validation over KB-derived object properties.

  • Motivation and contribution: KARI combines knowledge-base evidence with LLM reasoning to induce one compositional rule per affordance, addressing fixed-schema limits and hallucination risks.The pipeline is designed to produce rules that are both grounded and expressive.
  • Rule representation: KARI rules are tree expressions of static physical conditions, using MADE_OF, SHAPE, SIZE, SURFACE, AND, and OR operators.The four atomic predicates represent physical properties, while AND and OR combine multiple predicates and can nest.
  • Pipeline components: KARI collects affordance verbs and positive objects with materials, parts, and other physical properties from CSKG and the Aristo Tuple KB.The proposer is the only component that writes rule expressions, drafting or extending rules with new OR branches.
  • Pipeline components: KARI iterates over three data groups: the proposer drafts a seed rule, the auditor updates values and vocabulary, and later rounds add validation and rules for uncovered objects.In rounds 2 and 3, the validator checks the previous rule before the proposer handles uncovered objects, followed by another auditor update.
  • Inference: 2,223 rules are produced with QWEN3-14B; during inference, candidate affordances match their highest-scoring rule only when sentence similarity exceeds 0.7.Matched rules are verbalized in the Questioner’s system prompt, while candidates without a match above the threshold receive no rule.

5 Experiment Setup

The experiment evaluates 15 LLM Questioners on a 1,009-game AFFORDANCE20Q test split, comparing open- and closed-source models with human and non-LLM reference points. Performance is measured by success, efficiency, and per-turn information gain under fixed game conditions.

  • Evaluation protocol: The evaluation uses the 1,009-game test split, with human performance measured on a sampled subset and additional non-LLM reference points.The Questioner, Oracle, and Checker run at temperature 0; five annotators play a 30% subset under the same game rules, and the average result is reported.
  • Models: 15 LLMs serve as Questioners, grouped into open-source and closed-source models.The evaluation includes ten open-source and five proprietary models.
  • Models: 10 open-source models span dense models from 8B to 14B parameters and two mixture-of-experts models.The dense models include QWEN3, QWEN3.5, PHI-4, LLAMA-3.1, MINISTRAL, NEMOTRON, and GEMMA-3; the mixture-of-experts models are DEEPSEEK-V4-PRO and DEEPSEEK-V4-FLASH.
  • Models: 5 proprietary models are accessed through their official APIs: GPT-5, GPT-5-MINI, GEMINI-2.5-PRO, GEMINI-2.5-FLASH, and MINIMAX-M2.5.The proprietary models provide the closed-source evaluation group.
  • Metrics: Three metrics are reported: Success Rate, Turns, and Information Gain.Success Rate is the fraction of games solved within 20 turns; Turns is the average number of turns in solved games; Information Gain measures how quickly questions narrow the candidate affordances using KL divergence between consecutive distributions.

6 Results

AFFORDANCE20Q exposes a substantial gap between human and LLM affordance reasoning, with performance varying by affordance complexity and question quality. KARI improves smaller open-source models, but its benefits are inconsistent across model families.

  • Overall performance: 100%: OPTIMAL succeeds in 2.5 turns, whereas FIX20Q reaches 24.8% using all 20 questions and humans reach 64.2% in 10.7 turns.The theoretical 2.5-turn minimum requires optimally ordered questions, while extensive questioning without game-specific reasoning does not ensure success.
  • Overall performance: 14.9% to 27.4%: most open-source LLMs perform near FIX20Q, while all LLMs fall roughly 20 to 50 points below humans.DEEPSEEK-V4-FLASH and DEEPSEEK-V4-PRO are clear exceptions, likely aided by their mixture-of-experts architecture.
  • Affordance-specific performance: 3.1% to 56.0%: affordance success varies widely, with single-property cases such as conduct_heat reaching 55.2% and multi-property sink_in_water reaching 3.8%.Closed-source LLMs outperform open-source models on affordances requiring multiple physical properties despite similar turn usage.
  • Question behavior: Around 40%: shape questions dominate later questioning, after models typically begin with material questions and use size and surface questions more afterward.The study evaluates whether this behavior is meaningful using per-turn information gain.
  • KARI rule integration: 4.1 to 15.2 points: KARI increases success rates for 8B-to-14B models, but its rules can reduce performance for most closed-source and DeepSeek-V4 models.KARI also changes turn usage inconsistently: QWEN3-8B/14B decreases substantially, NEMOTRON-9B doubles its turns, and LLAMA-3.1-8B uses all 20 turns.
  • Success and failure modes: KARI can resolve uncertainty by supplying explicit affordance knowledge, whereas unassisted models may overthink, exhaust the turn budget, or guess incorrectly despite sufficient evidence.The example games show these failure modes for DEEPSEEK-V4-FLASH and MINIMAX-M2.5.

7 Conclusion

AFFORDANCE20Q evaluates affordance reasoning from physical properties rather than object-identity recall, revealing a substantial performance gap between 15 state-of-the-art LLMs and humans. KARI is proposed to close this gap by generating affordance rules through an LLM-based pipeline.

  • Benchmark and findings: 1,009 games over 454 objects and 59 affordances comprise AFFORDANCE20Q, which tests affordance reasoning from physical properties rather than object-identity recall.The benchmark is formulated around physical-property reasoning instead of memorized object-affordance mappings.
  • Benchmark and findings: ∼20 points separate the performance of 15 state-of-the-art LLMs from human performance.The experiments reveal a substantial gap between current LLMs and humans.
  • Benchmark and findings: KL-based information-gain analysis shows that models fail to ask discriminating questions as the game progresses.The analysis examines question selection during the 20-Questions game.
  • KARI: KARI is an LLM-based pipeline that generates affordance rules to close the gap between model and human performance.The proposed method targets the benchmark’s observed reasoning deficit.

8 Limitations

AFFORDANCE20Q focuses on affordances deducible from physical properties in a text-only, question-answering setting, leaving non-physical affordances, visual reasoning, and richer contextual inference unexplored.

  • Scope of physical affordances: The benchmark covers 59 affordances and 454 objects but filters out affordances not deducible from physical-property dimensions alone.Non-physical examples, such as a bulb providing light through electricity, remain outside its scope.
  • Text-only domain: AFFORDANCE20Q evaluates affordance reasoning in a text-only setting without exposing object identities, leaving visual-domain performance for future study.A visual extension would better align with many real-world affordance-reasoning tasks.
  • Question-Answering Setting: The benchmark formulates affordance reasoning as question answering, whereas real-world applications often embed implicit affordance cues in narrative understanding or open-ended planning.Extending the benchmark to richer contexts is identified as future work.

9 Ethical Considerations … A.2 Rule Examples

The paper uses public, non-sensitive physical-property data and confines AFFORDANCE20Q and KARI to research on everyday-object affordances. Its prompts formalize rule induction, validation, vocabulary auditing, and structured physical-property examples.

  • 9 Ethical Considerations: AFFORDANCE20Q uses publicly released CSKG evidence and contains only physical-property descriptions of everyday objects, with no personal or sensitive data.
  • 9 Ethical Considerations: The benchmark is intended for physical-affordance research, excludes offensive content, and closed-source LLM scores may change as providers update their models.
  • A.1 Prompts: The proposer generates rules describing physical prerequisites from affordance verbs and positive KB objects with physical properties.
  • A.1 Prompts: Rules combine subexpressions with AND or OR, introduce named parts using PART, and attach MADE_OF, SHAPE, SIZE, or SURFACE predicates.
  • A.1 Prompts: The prompt requires canonical static physical terms, forbids affordance or dynamic-behavior words, and enforces fresh, previously declared part variables.
  • A.1 Prompts: The validator labels each rule atom YES or NO using raw evidence and world knowledge, including properties present in any object part.
  • A.1 Prompts: The auditor maps proposed predicate values to same-dimension synonyms or marks genuinely new concepts as NEW, choosing NEW when uncertain.
  • A.2 Rule Examples: The pierce_through example gives three part-specific physical pathways, whereas fold_flat gives four whole-object alternatives.

B Experiment Setup Details · B.1 Fix20Q Baseline Question Selection · B.2 Prompts

The experiment uses a fixed 20-question baseline and a constrained 20-Questions interaction in which models infer affordances from hidden objects’ physical properties. Prompts separately define the Questioner, Oracle, and Checker roles to enforce atomic physical-property questions, yes/no answers, and valid final guesses.

  • B.1 Fix20Q Baseline Question Selection: The Fixed-Q baseline asks 20 yes/no questions in a fixed order curated from all evaluated LLM questioners on the test split.The curated questions are reported in Table 2.
  • B.2 Prompts: The Questioner asks one yes/no question per turn about exactly one static physical property, without directly probing affordance, function, use, purpose, activity, or category.Questions must target one of four dimensions: material, shape, size, or surface.
  • B.2 Prompts: The Questioner must avoid multi-property questions, such as asking whether an object is both metal and pointed or flexible and smooth.The prompt also provides function, use, and category questions as examples to avoid.
  • B.2 Prompts: The Questioner outputs a final guess only when confident or after 20 questions, using the exact format FINAL_GUESS: <affordance label>.Each turn uses the format QUESTION: <your yes/no question>.
  • B.2 Prompts: The Oracle sees the hidden object’s full description and answers each question with exactly one yes/no response about observable physical characteristics.The Oracle is instructed not to reveal the object’s name.
  • B.2 Prompts: The Checker classifies each question into every explicitly probed DIMENSION:value pair and rejects questions probing function, use, or category.It preserves the question’s wording without inference, expansion, paraphrase, or added synonyms, and emits one pair per line.
  • B.2 Prompts: Checker mappings cover material, shape, size, and surface properties, while preserving comparative size relations and emitting multiple pairs for multi-property questions.Examples include SIZE:larger_than_hand, SIZE:smaller_than_coin, and separate SIZE:long and SIZE:narrow outputs.

B.3 Implementation Details … D.2 Recruitment and payment.

The paper evaluates 15 LLM questioners, validates Oracle answers against human labels, reports affordance difficulty breakdowns, and describes human annotation recruitment and participation conditions.

  • B.3 Implementation Details: 15 LLM questioners are evaluated in the benchmark experiments.The evaluated models include open-source, closed-source, and two DeepSeek-V4 variants.
  • B.3 Implementation Details: Open-source models run locally on 8 × NVIDIA RTX A6000 GPUs, while closed-source and DeepSeek-V4 models use commercial APIs.All decoding uses temperature 0, with per-call maximum-token budgets of 512 for the Questioner, 20 for the Oracle, and 50 for the remaining component.
  • B.4 Oracle Validation: 300 sampled yes/no questions are labeled by three human annotators to assess Oracle agreement with human answers.Three candidate Oracle models are run on the same set, with agreement reported in Table 3.
  • C Success Rate Breakdown: Top-5 easiest and bottom-5 hardest affordances are reported for all 15 LLMs, open-source models, and closed-source models.These breakdowns appear in Tables 4, 5, and 6, respectively.
  • D Human Annotation: Human annotation involved volunteers for the baseline, Oracle-answer verification, manual annotation and refinement, and object-affordance label verification.The reported groups included five human-baseline volunteers, three Oracle verifiers, six manual annotators, and three additional label verifiers.
  • D.1 Participants and Instructions.: Annotators were instructed to base object and affordance labels on observable or physically inferable properties, excluding unsupported functional, social, electronic, chemical, or mechanism-specific information.Physical descriptions focused on shape, material, surface characteristics, dimensions, and structural components.
  • D.2 Recruitment and payment.: All participants were university students recruited voluntarily through internal channels and received no financial compensation.This condition applied to the reported annotation participation.

D.3 Data Consent

Participants were told that their annotations would be used only for academic research and dataset construction, and no demographic or personally identifying information was retained beyond annotation outputs.

  • Participants were informed that annotations would be used solely for academic research and dataset construction, with no demographic or personally identifying information retained beyond annotation outputs.
Loading 2606.14240v1…