PaperlayerView exact revision on arXiv

Source-linked AI summary

AffordBot: 3D Fine-grained Embodied Reasoning via Multimodal Large Language Models

Xinyi Wang, Xun Yang, Yanlong Xu, Yuchen Wu, Zhen Li, Na Zhao

arXiv:2511.10017v1cs.CV

TL;DR

Fine-grained embodied reasoning requires locating actionable elements and inferring how to interact with them, while existing approaches may stop at object-level grounding or separate grounding from motion estimation. AffordBot applies MLLM reasoning to 3D point-cloud scenes through instruction-conditioned triplet prediction and achieves state-of-the-art results on SceneFun3D.

  • Problem

    Existing approaches often stop at object-level recognition and grounding or treat affordance grounding and motion estimation separately, with instruction-agnostic motion prediction limiting instruction-conditioned scenarios.

    Source: §1 · p. 2

  • Method

    AffordBot uses MLLM reasoning over 3D point-cloud scenes for instruction-conditioned prediction of each referenced element’s affordance mask, motion type, and motion axis.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    AffordBot achieves state-of-the-art results on SceneFun3D and reports higher scores in affordance grounding and motion estimation than the listed existing methods and enhanced Fun3DU (+motion) baseline.

    Source: §1 · p. 2, §4.2 · p. 7

  • Takeaways & Limitations

    The results demonstrate the feasibility of efficient, task-coherent embodied reasoning that bridges scene perception, affordance localization, and motion synthesis.

    Source: §5 · p. 10

    Performance depends heavily on initial element-segmentation accuracy; missing or inaccurate elements impair subsequent reasoning.

    Source: D Limitations and Future Directions · p. 17

Abstract

from arXiv · show

Effective human-agent collaboration in physical environments requires understanding not only what to act upon, but also where the actionable elements are and how to interact with them. Existing approaches often operate at the object level or disjointedly handle fine-grained affordance reasoning, lacking coherent, instruction-driven grounding and reasoning. In this work, we introduce a new task: Fine-grained 3D Embodied Reasoning, which requires an agent to predict, for each referenced affordance element in a 3D scene, a structured triplet comprising its spatial location, motion type, and motion axis, based on a task instruction. To solve this task, we propose AffordBot, a novel framework that integrates Multimodal Large Language Models (MLLMs) with a tailored chain-of-thought (CoT) reasoning paradigm. To bridge the gap between 3D input and 2D-compatible MLLMs, we render surround-view images of the scene and project 3D element candidates into these views, forming a rich visual representation aligned with the scene geometry. Our CoT pipeline begins with an active perception stage, prompting the MLLM to select the most informative viewpoint based on the instruction, before proceeding with step-by-step reasoning to localize affordance elements and infer plausible interaction motions. Evaluated on the SceneFun3D dataset, AffordBot achieves state-of-the-art performance, demonstrating strong generalization and physically grounded reasoning with only 3D point cloud input and MLLMs.

1 Introduction

Fine-grained embodied reasoning requires identifying actionable scene elements and how to manipulate them, but prior work often separates grounding from motion estimation or ignores task context. AffordBot combines 3D scene information with MLLM reasoning to predict instruction-conditioned affordance and motion triplets.

1 Introduction

  • Existing approaches often stop at object recognition and spatial grounding, while SceneFun3D treats grounding and motion estimation separately and predicts motion without instruction context.This requires estimating motion parameters for all functional parts regardless of the task.Source: §1 · p. 2
  • Fine-grained 3D Embodied Reasoning jointly predicts an affordance mask, motion type, and motion axis direction for each referenced element from a natural-language instruction.The formulation couples spatial grounding and interaction reasoning under task guidance.Source: §1 · p. 2
  • AffordBot integrates 3D geometric information with MLLM reasoning and operates directly on 3D point clouds rather than video-based inputs.The approach addresses the challenge of applying 2D-native MLLMs to physically grounded 3D spatial tasks.Source: §1 · p. 2
  • AffordBot renders surround-view images from the point cloud and projects structured 3D affordance candidates onto them to give the MLLM spatially aligned visual context.This establishes explicit 3D-to-2D correspondences without redundant video streams.Source: §1 · p. 2
  • Its chain-of-thought process selects an informative viewpoint before guiding the MLLM through affordance localization and motion type and axis inference.The steps are conditioned on spatial input and task intent.Source: §1 · p. 2
  • AffordBot achieves state-of-the-art results on SceneFun3D for physically grounded, instruction-conditioned 3D reasoning.Its overview combines multimodal representation construction with a tailored chain-of-thought process.Source: §1 · p. 2, Figure 2 · p. 3

2 Related Work

Related work spans affordance grounding, 3D motion estimation, and MLLMs for 3D understanding. The paper positions its approach as combining MLLM reasoning with fine-grained grounding and motion estimation.

Affordance Understanding

  • SceneFun3D introduced fine-grained affordance grounding in complex indoor scenes, while Fun3DU uses VLMs and universal segmentation to localize target elements in video frames.Earlier affordance work focused on simpler settings.Source: §2 · p. 3

3D Motion Estimation

  • Earlier motion-estimation approaches inferred articulated motion from individual objects’ geometric structures, whereas SceneFun3D evaluates motion across scene-level affordance elements.Source: §2 · p. 3

MLLMs for 3D Understanding

  • MLLM approaches to 3D understanding either develop native 3D-aware models or adapt 2D VLMs through 2D representations; this work uses MLLM reasoning for joint affordance grounding and motion estimation.Source: §2 · p. 3

3 Methodology

The section formulates instruction-conditioned 3D embodied reasoning as joint affordance grounding and motion prediction, then presents AffordBot’s point-cloud-to-MLLM representation and staged observe-then-infer pipeline.

3 Methodology

  • Given a 3D point cloud and task instruction, the task predicts each referenced affordance element’s 3D mask, motion type, and motion-axis direction.It jointly reasons over geometry, semantics, and language intent, including ambiguous references and physically plausible interaction cues.Source: §3 · p. 3

3.1 AffordBot Overview

  • AffordBot bridges 3D perception and 2D-native MLLMs with a holistic multimodal representation and a chain-of-thought reasoning paradigm.The overview describes the representation and reasoning pipeline as its two key components.Source: §3.1 · p. 4

3.2 Holistic Multimodal Representation

  • Projected 3D elements are associated with surround views using labeled bounding boxes, while adaptive label placement avoids overlaps and keeps views legible.Projected boxes carry unique IDs linked to the original 3D elements and their affordance types.Source: §3.2 · pp. 4–5
  • A 360° panoramic scan generates surround views, addressing limited video coverage while avoiding per-instruction keyframe extraction and redundant-frame processing.The scan captures more scene information than traditional video, which can miss targets and their anchors within a frame.Source: §3.2 · pp. 4–5
  • AffordBot extracts affordance elements and geometry-semantic descriptors, using progressive mask dilation and combined Dice and cross-entropy losses for segmentation.The dilation radius changes by curriculum stage; the losses balance region-level alignment and point-wise classification.Source: §3.2 · p. 5

3.3 Chain-of-thought Reasoning

  • The MLLM selects an informative view, grounds instruction-referenced elements, then infers their motion type and axis direction.Continuous direction vectors are discretized into interpretable categories, with refinements for translational and rotational motions.Source: §3.3 · p. 6

4 Experiments

On SceneFun3D, AffordBot outperforms the reported comparison methods in affordance grounding and motion estimation. Ablations attribute gains to enriched visual synthesis and active view selection, while analyses identify segmentation noise and category variation as important constraints.

4.1 Experimental Setup

  • SceneFun3D provides 230 annotated scenes—200 for training and 30 for validation—with point clouds and element-level affordance masks, motion types, and axis directions.The evaluation reports mIoU, mAP, AP, AP25, and AP50, with motion-constrained AP25 variants for motion type and direction.Source: §4.1 · p. 7

4.2 Quantitative Results

  • AffordBot significantly outperforms the reported methods, including Fun3DU (+motion), across all reported metrics and scores higher in both affordance grounding and motion estimation.Its higher AP also indicates precision under stricter IoU thresholds.Source: §4.2 · pp. 7–8

4.3 Ablation Studies

  • Adding ALR yields +0.4% AP25, EVS raises AP25 from 16.1% to 22.1% (+6.0%), and AVS reaches 23.3%, the highest overall accuracy.EX4 is AffordBot; the component variants are compared in Table 2.Source: §4.3 · p. 8, Table 2 · p. 8
  • Dynamic view selection reaches 23.3% AP25, an absolute gain of 7.6% over query-aligned selection, by choosing an instruction-matched view from a dense 360° sweep.Table 3 compares viewpoint-selection approaches.Source: §4.3 · p. 8, Table 3 · p. 8
  • Replacing Mask3D proposals with ground-truth masks boosts AP25 by 22.1%, while adding ground-truth views improves it by a further 2.0%.The analysis identifies instance-segmentation noise as the primary bottleneck; active view selection is not the dominant bottleneck.Source: §4.3 · pp. 8–9, Table 4 · p. 9
  • GPT-o1 raises AP25 to 33.4% from Qwen2.5-VL-72B’s 23.3%, and the evaluated larger deployable models also show stronger performance.Table 5 compares different MLLMs.Source: §4.3 · p. 9, Table 5 · p. 9
  • AP50 varies widely by affordance type, from 100% for foot_push to 0% for rotate; results also favor Multiple over Unique target tasks.The analysis links type variation partly to class imbalance and initial segmentation quality, and Unique-task difficulty to Mask3D’s noisier descriptors for small, weakly textured objects.Source: §4.3 · p. 9, Table 6 · p. 9, Table 7 · p. 9

4.4 Qualitative Results

  • Qualitative examples include remote, drawer, door, and window interactions; AffordBot generally grounds targets more accurately and consistently, especially for multiple or small targets.The paper reports qualitative improvement over prior SOTA method.Source: §4.4 · p. 9, Figure 5 · p. 10

5 Conclusion

AffordBot unifies affordance grounding and motion estimation through instruction-aware triplet prediction and a tailored chain-of-thought framework. The authors report state-of-the-art performance and identify stronger multimodal understanding as future work.

5 Conclusion

  • AffordBot unifies instruction-aware affordance localization and motion synthesis through structured triplet prediction.The conclusion frames this as bridging scene perception to action.Source: §5 · p. 10
  • The framework uses an MLLM and tailored chain-of-thought reasoning to advance from scene perception to localization and motion synthesis.Source: §5 · p. 10
  • The authors report state-of-the-art performance and the feasibility of efficient, task-coherent embodied reasoning.Source: §5 · p. 10
  • Future work will give AffordBot more advanced multimodal understanding ability.Source: §5 · p. 10

Appendix

The appendix supplements the main paper with implementation details, additional results, limitations, impact discussion, and asset licensing information.

Appendix

  • The appendix provides implementation details on geometric descriptors, view synthesis, label refinement, motion-axis discretization, and chain-of-thought prompts.Source: A Overview · p. 15
  • It includes additional qualitative results and discussion of current limitations and future research directions.Source: A Overview · p. 15
  • The supplementary materials also discuss real-world impact and provide licensing information for the assets used.Source: A Overview · p. 15

B Implementation Details

The implementation details describe affordance-aware segmentation, surround-view synthesis, label refinement, motion-axis discretization, and a three-step MLLM prompting procedure.

Geometric Descriptor Generation

  • Mask3D segmentation uses affordance-type labels, curriculum-based mask dilation, 80 instance queries, and combined region-level and point-wise losses.The nine labels are rotate, key_press, tip_push, hook_pull, pinch_pull, hook_turn, foot_push, plug_in, and unplug; dilation uses δ0 = 0.1, β = 0.5, and τ = 200 iterations, with loss weights λ1 = 5 and λ2 = 2.Source: B Implementation Details · p. 15

Enriched Visual Synthesis

  • The view synthesis positions a virtual camera at the scene observation center and renders eight evenly spaced views over 360°.The camera uses a 90° field of view and 680 × 680 pixel output.Source: B Implementation Details · p. 15

Adaptive Label Refinement

  • Adaptive label refinement tests four anchor positions, placing each identifier without image-boundary violations or overlap, then uses a top-right fallback if none fits.The procedure annotates projected proposal boxes in each rendered image.Source: B Implementation Details · pp. 15–16

Motion Axis Discretization

  • Motion axes are discretized into six primitives: four translational directions and horizontal or vertical rotation.Translation distinguishes horizontal or vertical motion and inwards or outwards direction; rotation retains only horizontal or vertical categories.Source: B Implementation Details · p. 15

Prompts for Chain-of-Thought Reasoning

  • The MLLM first selects a view, then identifies affordance-element IDs, and finally predicts motion type and axis direction for those elements.The prompts pair the task description with surround views for selection, then use the selected image and spatial information for grounding and motion reasoning.Source: B Implementation Details · pp. 15–16

C Additional Qualitative Results

Additional qualitative results compare AffordBot with the current state-of-the-art approach on affordance grounding and motion estimation. The authors report stronger localization and crisper element boundaries, alongside more reliable motion predictions.

C Additional Qualitative Results

D Limitations and Future Directions

AffordBot’s reasoning depends on accurate initial segmentation, while scene-centric rendering can limit visibility; the authors identify view synthesis and selection as future improvement areas.

D Limitations and Future Directions

E Broader Impact

The paper describes potential benefits for human-agent collaboration and possible societal risks from automation and MLLM reliance.

E Broader Impact

F Licenses for Used Assets

The listed dataset and model have stated licenses, and the accompanying prompt materials cover viewpoint selection and motion analysis.

F Licenses for Used Assets

Related papers

Loading 2511.10017v1…