A4-Agent: An Agentic Framework for Zero-Shot Affordance Reasoning

1. Introduction

• Affordance prediction identifies object regions for language-instructed interactions, supporting task planning, robotic grasping, and human–robot collaboration.For “open the refrigerator,” the handle is the actionable region.Source: §1 · pp. 1–2
• Traditional grounding-focused approaches lack high-level reasoning, while trained unified models face trade-offs, limited generalization, and reduced flexibility.These limitations motivate questioning whether reasoning and grounding should remain entangled in end-to-end systems.Source: §1 · p. 2
• A4-Agent decouples affordance prediction into Dreamer for imagined interactions, Thinker for actionable-part descriptions, and Spotter for spatial localization.The stages use generative models, vision-language models, and vision foundation models, respectively.Source: §1 · p. 2
• A4-Agent significantly outperforms state-of-the-art supervised methods across multiple benchmarks and demonstrates robust generalization to real-world settings.The framework coordinates pre-trained models without task-specific training.Source: §1 · p. 2
• The paper highlights decoupling reasoning and grounding and using explicit imagination as contributions to affordance reasoning.It also presents A4-Agent as a training-free framework with zero-shot generalization capabilities.Source: §1 · p. 2

2. Related Work

• Many affordance methods jointly train reasoning and grounding, face trade-offs between reasoning complexity and spatial precision, and generalize poorly to novel scenarios.Prior work includes learning from human-object images, demonstrations, point clouds, 3D Gaussian Splatting, and multimodal language models.Source: §2 · p. 2
• Multimodal reasoning work extends text-based reasoning to vision, while other approaches explore visual representations such as textual imagery for dynamic reasoning.The related work describes these as emerging approaches to multimodal reasoning.Source: §2 · pp. 2–3
• Vision foundation models provide fine-grained grounding but weak reasoning, whereas vision-language models reason well but ground poorly; fine-tuning has left both abilities underwhelming.The comparison motivates combining complementary model capabilities.Source: Figure 2 · p. 3
• Prior visual-imagination approaches report that generated intermediate visuals can enhance reasoning and interpretability, informing affordance reasoning that needs spatial and interaction understanding.A4-Agent differs from end-to-end affordance methods by decoupling reasoning from grounding.Source: §2 · p. 3

3. Motivation

• Vision foundation models localize finely but lack semantic understanding of complex instructions, while multimodal language models reason well but often predict coarse or inaccurate locations.Affordance prediction requires both capabilities.Source: §3 · p. 3
• End-to-end models trained on grounding data face limited generalization, capability trade-offs, costly whole-system upgrades, and restricted access to closed-source models.The stated limits include brittleness on novel objects and environments.Source: §3 · p. 3
• A4-Agent instead decouples reasoning and grounding into specialized agents coordinated at test time, with components designed independently using foundation models.The framework argues affordance prediction is inherently multi-stage.Source: §3 · p. 3
• The proposed paradigm is presented as training-free and modular, with independently upgradeable components and interpretable intermediate steps for diagnosis and refinement.It is also framed as avoiding task-specific fine-tuning and expensive data collection.Source: §3 · pp. 3–4
• Dreamer imagines an interaction, Thinker selects the actionable part, and Spotter localizes it through boxes and keypoints refined into pixel-accurate masks.The framework describes this as a coarse-to-fine grounding process.Source: Figure 3 · p. 4

4.1. Problem Definition

• Affordance prediction maps an image and task description to interaction regions, represented primarily as segmentation masks for pixel-level precision.The paper notes that boxes are another representation, depending on downstream applications.Source: §4.1 · p. 4

4.2. Framework Overview

• A4-Agent decouples reasoning about which object part to interact with from grounding that part's location, using Dreamer, Thinker, and Spotter in sequence.Dreamer imagines how the operation could look; Thinker decides what part to operate; Spotter localizes where to operate through boxes and key points refined into masks.Source: §4.2 · p. 4

4.3. Dreamer: Imagine how to Operate

• Dreamer uses a VLM to create an actionable interaction prompt and a generative model to edit the original image into a plausible interaction scenario.The imagined image depicts contact and motion cues that can guide evaluation of whether the action pattern is reasonable; Figure 4 reports qualitative ReasonAff predictions close to ground truth and surpassing Affordance-R1.Source: §4.3 · pp. 4–5, Figure 4 · p. 5

4.4. Thinker: Decide what to Operate

• Thinker considers the original image, imagined interaction image, and task, then returns a machine-readable specification naming the task, object, and actionable part.It uses the imagined image for contact and motion cues, then grounds the selected part back in the original image without assigning spatial coordinates.Source: §4.4 · p. 5

4.5. Spotter: Locate where to Operate

• Spotter converts Thinker's textual description into localization by pairing open-vocabulary detection with segmentation-model refinement.Detection supplies coarse boxes and key points as prompts for pixel-accurate masks, yielding multi-granular representations for downstream use.Source: §4.5 · pp. 5–6

5.1. Experimental Settings

• Tests span ReasonAff, RAGNet, UMD, and open-world images; A4-Agent is zero-shot on the benchmarks and coordinates GPT-4o, Qwen-Image-Editing, Rex-Omni, and SAM2-Large.Source: §5.1 · p. 6

5.2. Quantitative Results

• 71.83 gIoU on ReasonAff and 63.9 gIoU on RAGNet-3DOI establish state-of-the-art results in these zero-shot evaluations.Table 2 covers RAGNet-3DOI and RAGNet-HANDAL.Source: §5.2 · pp. 6–7, Table 2 · p. 6
• 15.53 gIoU above baselines on UMD, A4-Agent also surpasses fine-tuned methods without training.Source: §5.2 · p. 7, Table 3 · p. 7

5.3. Qualitative Results on Open-World Images

• Open-world examples cover novel objects, complex scenes, and reasoning about tool use; the authors report consistent performance where baselines often fail on out-of-distribution objects.Source: §5.3 · p. 7

5.4. Ablation Study

• Imagination improves all metrics across base models, and Qwen-2.5-VL (7B) with imagination outperforms GPT-4o using text-only reasoning.The ablation is on RAGNet-3DOI.Source: §5.4 · p. 7, Table 4 · p. 8
• GPT-4o improves reasoning performance, while smaller SAM2 variants cause slight drops that are smaller than Affordance-R1’s, preserving effectiveness with weaker segmentation.The component ablation is on RAGNet-3DOI.Source: §5.4 · p. 8, Table 5 · p. 8

• A4-Agent decouples high-level reasoning from low-level grounding, using vision-language models for semantic interpretation and vision foundation models for localization.Source: §6 · p. 8
• Its imagination mechanism uses a generative model to visualize potential interactions during reasoning.Source: §6 · p. 8
• Experiments report that the zero-shot approach outperforms supervised methods on challenging benchmarks and generalizes well to open-world scenarios.Source: §6 · p. 8

7.1. Details of Baseline Methods

• Baselines cover open-vocabulary segmentation, MLLM-enhanced end-to-end segmentation, and MLLM grounding followed by SAM segmentation.Examples include VLPart [42], OVSeg [25], AffordanceLLM [36], and Seg-Zero [28].Source: §7.1 · p. 12

7.2. Evaluation Metrics

• Four complementary metrics assess prediction quality: gIoU, cIoU, P@50, and P@50:95.They measure average image-level IoU, cumulative dataset-level IoU, the share of predictions above IoU 0.5, and average precision over IoU thresholds 0.5–0.95, respectively.Source: §7.2 · p. 12

7.3. System prompt of our Agent

• The system prompt asks an image-editing model to depict the task interaction while preserving the existing object and scene, then uses the interaction image to identify and ground the relevant part.The Thinker prompt returns the task, object name, and object part in structured form.Source: §7.3 · pp. 12–13

8. More Exploratory Experiments

• The exploratory variant uses Rex-Omni and SAM alone, omitting Dreamer and Thinker to test Spotter-only operation.Rex-Omni is an MLLM-based object detector with stronger language understanding than traditional detection models, according to the authors.Source: §8 · p. 13
• Although Spotter has some reasoning ability, its backbone is only a small MLLM with limited reasoning capacity, which the authors say leads to suboptimal performance.Table 6 reports more exploratory results on RAGNet-3DOI.Source: §8 · p. 13, Table 6 · p. 13

9. More Intermediate Results

• In the bottle example, the predicted grip region is the bottle’s body.The accompanying reasoning identifies the body as the area to grip securely.Source: §9 · p. 14
• Figures 8–9 show sampled intermediate results from the ReasonAff dataset.The section identifies these figures as ReasonAff examples.Source: §9 · p. 13, Figure 8 · p. 14, Figure 9 · p. 15
• For the ladle’s grasp affordance, the predicted part is its handle.The example’s structured output identifies the ladle handle as the object part.Source: §9 · p. 16
• Figures 10–11 show sampled intermediate results from the UMD dataset.The section identifies these figures as UMD examples.Source: §9 · p. 13, Figure 10 · p. 16, Figure 11 · p. 17
• Figures 12–13 show sampled intermediate results from the RAGNet dataset.The section identifies these figures as RAGNet examples.Source: §9 · p. 13, Figure 12 · p. 18, Figure 13 · p. 19