PaperlayerView exact revision on arXiv

Source-linked AI summary

AffordAny: Open-World 3D Affordance Grounding from Monocular RGB Images via Vision-Language-Guided Geometric Reasoning

Junqi Wu, Kaihua Tang, Xuanwen Chen, Hongzhi Li, Jianqiang Huang, Xian-Sheng Hua

arXiv:2608.20720v1cs.CV

TL;DR

Open-world 3D affordance grounding must localize functional object parts from free-form language without relying on pre-built 3D assets or closed ontologies. AffordAny constructs text-conditioned 3D supervision from one monocular RGB image, grounds queries using a frozen VLM-guided geometry decoder, and applies pseudo-label self-training. It achieves strong generalization across unseen objects, categories, and instruction paraphrases, with self-training improving unseen-category mIoU by 6.3% relative.

  • Problem

    Open-world 3D affordance grounding lacks broad support for raw RGB observations, diverse part taxonomies, and unconstrained language queries.

  • Method

    AffordAny automatically constructs text-conditioned 3D part supervision and fuses frozen Cosmos-2B features with 3D geometry for language-conditioned grounding, followed by pseudo-label self-training.

  • Results

    6.3% relative unseen-category mIoU improvement is obtained after self-training, with 0.428 IoU on unseen objects, 0.305 IoU on unseen categories, and a 0.105 instruction sensitivity gap.

  • Takeaways & Limitations

    The framework supports open-world affordance grounding from a single monocular RGB image while improving category-level generalization without additional human annotation.

  • Takeaways & Limitations

    Benchmark transfer to safety-critical manipulation is limited by unverified automatically fused labels, long-tailed data, reconstruction failures, and frozen offline VLM features.

Abstract

from arXiv · show

Open-world 3D affordance grounding requires localizing functional object parts in 3D given free-form language queries. Existing methods typically assume pre-built object-centric 3D geometry and closed affordance ontologies, limiting deployment from raw RGB observations. We present AffordAny, an end-to-end framework that uses one monocular RGB image to construct large-scale text-conditioned 3D part supervision, ground affordances with a frozen vision-language model (VLM) guided decoder, and improve open-world generalization through pseudo-label self-training. Our automated pipeline produces a benchmark of 5,334 objects and 10,633 part-level samples spanning 473 categories, an order-of-magnitude increase in categorical diversity over prior work. The decoder progressively fuses frozen Cosmos-2B features with 3D geometry through spatial projection, instruction-conditioned semantic compression, and bidirectional geometry-semantics interaction. Minimal-perturbation pseudo-label self-training further adds new objects without human annotation. Under a systematic generalization protocol evaluating unseen objects, unseen categories, and unseen instruction paraphrases, our approach achieves 0.428 IoU on unseen objects and 0.315 IoU on unseen categories after self-training, with unseen-category mIoU improving by 6.3% relative (p<0.01) and an instruction sensitivity gap of only 0.105, demonstrating effectiveness and robustness of our method.

Introduction

AffordAny targets open-world 3D affordance grounding from raw RGB images by building text-conditioned 3D supervision and combining VLM features with geometry. Its evaluation covers unseen instructions, objects, and categories, with self-training improving category-level generalization.

  • Motivation: 3D affordance grounding localizes functional interaction regions from free-form language or action descriptions for embodied-agent interaction.The motivation includes manipulation, navigation, and task planning.
  • Approach: AffordAny automatically constructs text-conditioned 3D part supervision from LVIS images through filtering, single-image reconstruction, language-guided prompts, and annotation fusion.The pipeline starts from 1,270,141 LVIS training instances and produces a benchmark from retained interaction-relevant objects.
  • Evaluation: The evaluation separately measures unseen instructions, unseen objects, and unseen categories under a systematic generalization protocol.Inference uses one monocular RGB image and object masks, without pre-built 3D assets or multi-view captures.
  • Results: 0.428 IoU is achieved on unseen objects and 0.305 IoU on unseen categories, while the instruction sensitivity gap is 0.105.After self-training, unseen-category mIoU improves by 6.3% relative with p<0.01.

Related Work

Prior 3D affordance resources span synthetic objects, real interactions, and language-conditioned methods, but language-based approaches generally remain tied to fixed affordance taxonomies. AffordAny instead emphasizes open-vocabulary grounding from real images.

  • 3D Affordance Learning: Prior work includes fine-grained part annotations, physically grounded articulated-object interaction, probabilistic point labels, and real interaction-image affordance grounding.Examples include PartNet, SAPIEN, 3D AffordanceNet, and AGD20K.
  • Vision-Language Models for 3D Understanding: Vision-language models have transferred image-text priors to open-world point-cloud representations, with PartSLIP specializing this transfer for low-level part understanding.The cited methods include ULIP, OpenShape, Uni3D, and PartSLIP.
  • Open-Vocabulary Gap: Even language-conditioned affordance methods generally map free-form instructions to fixed part or affordance taxonomies.The paper characterizes these methods as closed under Open Part.

AffordAny Benchmark

AffordAny builds a large real-image benchmark for open-vocabulary, text-conditioned 3D part affordance grounding through automated reconstruction, prompting, segmentation, and label lifting. The benchmark combines broad category coverage with multi-view instruction annotations and a multi-axis generalization protocol.

  • Benchmark Scale: 5,334 objects, 10,633 part-level samples, 473 categories, and 678 unique part types comprise the benchmark.The paper describes this as an order-of-magnitude increase in categorical diversity over prior work.
  • Construction: The pipeline filters LVIS instances and reconstructs each retained object from a single image into 3D Gaussians.It retains 7,407 candidate objects from 473 interaction-relevant categories.
  • Construction: Multi-view rendering, LLM-guided part discovery, text-conditioned 2D segmentation, and multi-view 3D label lifting produce the part annotations.These stages convert reconstructed objects into text-conditioned 3D supervision.
  • Instruction Annotations: Three synonymous natural-language interaction instructions are generated for each validated part, yielding 31,899 instruction-level samples.The instruction annotations support evaluation beyond a single wording of an interaction query.
  • Evaluation Protocol: The evaluation design separates seen categories from held-out categories and measures unseen objects, categories, and instruction paraphrases.The supplied passage begins the 70%/15%/15% category split specification.

Method

AffordAny grounds free-form affordance instructions in 3D by combining reconstructed point clouds with frozen VLM features and geometry-aware decoding. Its training pipeline adds text-conditioned supervision, pseudo-labels, and calibration to support open-vocabulary generalization.

  • Input and task: Given an RGB image, object mask, and free-form instruction, AffordAny reconstructs an object-centric point cloud and predicts per-point functional-region scores.The point cloud is produced by SAM 3D, and predictions are sigmoid scores in [0, 1].
  • VLM-guided decoder: The decoder fuses frozen Cosmos-2B image–language features with 3D geometry through projection injection, semantic compression, and bidirectional interaction.Projection preserves point-to-image correspondence, while geometry constrains the spatial extent of image–language evidence.
  • Data construction: The automated data pipeline reconstructs LVIS images from six views, uses an LLM and segmentation model for part annotations, and lifts labels to 3D by multi-view voting.The resulting benchmark covers an unrestricted part vocabulary and compares unseen instances and categories as open evaluation axes.
  • VLM-guided decoder: Instruction-conditioned semantic compression converts 145 VLM tokens into K = 16 task-aware prototypes before point–prototype interaction.FiLM modulation makes the fixed-size semantic bottleneck instruction-dependent and bounds later attention independently of VLM sequence length.
  • VLM-guided decoder: Group-Mix-Ungroup lets prototypes absorb 3D context and return geometry-refined semantic information to points, supporting transfer across geometric realizations.The enriched point tokens then query the affordance token and refined prototypes through a four-layer Transformer cross-decoder.
  • Self-training and calibration: Pseudo-label self-training filters uncertain predictions, mixes pseudo-labels with real data, and applies Platt calibration to separate overlap gains from probability calibration.The teacher ignores probabilities in [0.15, 0.65], while the calibrated transform adjusts logit sharpness and positive-rate bias.

Experiments

AffordAny evaluates open-world generalization across unseen objects, categories, and instruction paraphrases, comparing decoder variants, baselines, and pseudo-label self-training. Results show strong thresholded localization and instruction robustness, while revealing category-shift trade-offs and dataset limitations.

  • Generalization results: 0.428 IoU on unseen objects and 0.305 IoU on unseen categories lead AffordAny’s decoder comparison, while unseen-instruction IoU reaches 0.680.AffordAny also leads unseen-object AUC and SIM and unseen-category SIM, but LASO retains the best unseen-category AUC and MAE.
  • Generalization results: LMAffordance3D falls to 0.244 IoU on unseen categories, so AffordAny does not uniformly dominate every metric or baseline.The reported main gain is better thresholded localization and spatial concentration under category shift.
  • Instruction robustness: 0.105 absolute IoU gap between unseen and seen instructions is the smallest among compared methods.AffordAny also has the smallest absolute mIoU, AUC, and MAE gaps, although OpenAD has a slightly smaller SIM gap of 0.065 versus 0.078.
  • Ablation study: Semantic compression contributes the largest ablation margin, with a 0.060 unseen-category IoU drop when removed.Removing GPBlock, projection injection, and instruction dropout produces drops of 0.035, 0.030, and 0.017, respectively.
  • Pseudo-label self-training: 49/71 unseen categories improve after pseudo-label self-training, with larger gains for tail categories than head categories.Tail categories with ≤50 training samples gain +0.012, compared with +0.003 for head categories; pseudo-labels add geometric diversity without new language vocabulary.
  • Limitations: 2,064 of 7,398 completed objects have no validated parts, while automatic labels contain projection and segmentation noise.The authors also note long-tailed distributions, frozen offline VLM features, and limited direct transfer to safety-critical manipulation without further validation.

Conclusion

AffordAny grounds open-world 3D affordances from a single monocular RGB image by combining automatically constructed text-conditioned supervision, frozen VLM features, and 3D geometry. It generalizes to unseen objects and categories, and pseudo-label self-training further improves category-level performance.

  • Conclusion: AffordAny constructs a 5,334-object benchmark across 473 categories with open-vocabulary text-conditioned part supervision from real images.The framework targets open-world grounding from a single monocular RGB image.
  • Conclusion: The VLM-guided decoder fuses frozen Cosmos-2B features with 3D geometry through projection injection, semantic compression, and bidirectional fusion.This design supports generalization to unseen objects and categories with notably small instruction sensitivity.
  • Conclusion: Unseen-category mIoU improves by 6.3% relative after pseudo-label self-training, with the largest gains on tail categories.The reported significance is p<0.01.

Supplementary Material

The supplementary material documents benchmark construction, quality controls, split semantics, 2D-to-3D label lifting, instruction generation, and reproducibility metadata. The pipeline combines automated filtering and reconstruction with deterministic fusion and provenance tracking.

  • Benchmark construction: 1,270,141 LVIS instances across 1,203 categories are filtered to 7,407 candidates in 473 categories before object-level processing.A square-root category quota reduces head-category dominance while retaining at least one candidate per retained category.
  • 3D reconstruction: Each masked RGB instance is reconstructed into an object-centric 3D Gaussian representation and rendered from six canonical views.Views cover front, right, back, left, top, and bottom using fixed camera settings.
  • Part annotation: Part discovery proposes at most five candidate parts, while downstream segmentation retains at most three successfully localized parts per object.The prompt requires visually localizable physical parts and rejects whole-object or abstract affordance descriptions.
  • 2D-to-3D lifting: Per-Gaussian part scores equal positive observations divided by visible views after projection, depth ownership, and mask-overlap voting.Fusion applies a score threshold of 0.3 and an ambiguity margin of 0.05; unresolved regions remain unknown and are excluded from supervision.
  • Instruction generation: Each packaged part receives exactly three generated English commands, producing 31,899 instruction pairs.Commands are limited to 14 words and require lexical and syntactic variation without explicit dataset labels.
  • Reproducibility: Structured rejection reasons, camera metadata, masks, visibility counts, votes, unknown masks, and manifests support deterministic rebuilding of accepted samples.The pipeline excludes unknown regions to preserve label reliability.

B Dataset Statistics and Split Semantics

The supplementary statistics specify the benchmark’s vocabulary, tensor organization, split semantics, frozen VLM cache, and decoder configuration. These details separate category, object, and instruction novelty while fixing the training interface and computational setup.

  • Dataset statistics: The benchmark contains 5,334 objects, 10,633 validated part samples, 678 normalized part names, and 31,899 generated instructions.Experimental manifests contain 22,767 rows because not all generated instructions are evaluated for every validation and test part.
  • Split semantics: Category splits assign 70% of categories to training and 15%/15% to unseen-category validation and test partitions.Within training categories, object instances use 80%/10%/10% train, validation, and test partitions for unseen-object evaluation.
  • Split semantics: Unseen-instruction evaluation holds wording changes constant in object and part identity while testing a reserved instruction index.This design isolates wording changes from instance and category novelty.
  • VLM cache: The frozen Cosmos-Reason2-2B encoder produces a cached instruction-conditioned marker state with hidden width D = 2,048.Only downstream projections, prototypes, fusion blocks, and the prediction head are learned.
  • Inference interface: At inference, one RGB image, a known binary object mask, and a free-form instruction produce object-centric Gaussian geometry and affordance predictions.Training-time geometry, camera metadata, and frozen Cosmos states are computed offline and cached, while the decoder is optimized from those records.
  • Decoder configuration: The decoder uses 16 prototypes, one Group–Mix–Ungroup block, and a four-layer point-to-semantic Transformer cross-decoder.Its point encoder contains three residual MLP blocks with width 256.

C.4 Pseudo-Label Self-Training

Pseudo-label self-training expands supervision from disjoint objects while calibration separates ranking preservation from threshold-dependent segmentation metrics. The section also specifies evaluation and computational details for the decoder and metrics.

  • Pseudo-label generation: 5,325 disjoint LVIS objects yield pseudo-labels, with probabilities in [0.15, 0.65] ignored and the remainder binarized.A fixed subsample is mixed approximately 1:1 with real training data.
  • Minimal perturbation training: Self-training starts from the supervised checkpoint and applies low-weight pseudo-label losses during one additional training round.Pseudo BCE and focal contributions receive an additional factor of 0.5.
  • Computational cost: 121.63 GFLOPs quantify decoder matrix operations for one 4,096-point object, while measured latency and memory include other operators.The profile uses batch size 1, 144 visual tokens, 16 prototypes, and width 256.
  • Computational cost: 16.5-ms median latency corresponds to about 60 cached-decoder evaluations per second, with self-attention dominating the four point-decoder layers.The measurement isolates the learned localization decoder under cached-feature evaluation.
  • Metric definitions: IoU and mIoU use thresholded binary predictions, whereas AUC, MAE, and SIM provide complementary ranking, error, and distributional views.mIoU averages category-level IoUs with equal weight for head and tail categories; SIM retains soft targets.

D.3 Human-Verified Evaluation Subsets

Human-verified subsets provide a matched evaluation of unseen categories and unseen objects, while complementary metrics and targeted examples clarify how thresholded overlap can diverge from ranking and probability quality.

  • Subset construction: 95 human-verified rows comprise 41 unseen-category and 54 unseen-object examples retained only when reconstruction, part localization, and instructions were usable and unambiguous.Rows were manually checked using masked source images and six canonical renders.
  • Evaluation protocol: All frozen checkpoints use threshold 0.5 and the same pooled IoU, category-macro mIoU, global ROC AUC, and pointwise MAE definitions.Dense evaluation maps ignore labels conservatively to background.
  • Self-training results: Self-training significantly improves category-macro mIoU on both splits and unseen-category IoU while maintaining unseen-object IoU.Paired bootstrap reports p < 0.001 for the first gains and p = 0.461 for unseen-object IoU.
  • Category distributions: AffordAny achieves the highest central tendency on unseen instructions and unseen categories across heterogeneous category distributions.Figure 6 pools intersection and union within each category before plotting.
  • Metric complementarity: IoU measures thresholded overlap, while AUC, MAE, and SIM capture ranking, probability error, and normalized distributional agreement.Joint reporting avoids relying on a single favorable threshold.
  • Ranking–overlap discrepancy: The toaster-oven and drawer examples show high AUC and low MAE despite near-zero IoU because few target scores exceed 0.5.Reported examples include IoU 0.000 or 0.020, AUC 0.934 or 0.944, and MAE 0.012 or 0.014.
  • Qualitative comparison: Qualitative comparisons show more concentrated predictions on intended functional parts across unseen instructions, objects, and categories.The examples complement aggregate tables and category distributions.

G.1 Robustness Considerations

Robustness is strongest when reconstruction, visibility, semantic applicability, geometry lifting, and instruction grounding are reliable. The evaluation includes qualitative examples across three generalization conditions.

  • Upstream uncertainty: Thin, reflective, transparent, or occluded parts can reduce reconstruction quality and create uncertainty before or during decoding.Missing canonical-view coverage can also occur when a valid part is absent from all rendered views.
  • Geometric lifting: 2D-to-3D lifting is sensitive to depth ownership and closely adjacent boundaries.These factors affect localization before decoding consumes the lifted geometry.
  • Instruction grounding: Instruction grounding is most challenging for related functional parts, spatially broad targets, and ambiguous commands admitting multiple valid targets.Unknown masks reduce missing-evidence effects by excluding uncertain regions from supervision.
  • Qualitative robustness: Figure 8 compares source image/instruction, ground truth, and predictions across unseen instructions, unseen objects, and unseen categories.Red points mark thresholded affordance regions and numbers report per-row IoU.

H Limitations and Release Constraints

The benchmark targets perception and language-conditioned object-part localization rather than downstream physical execution. Its single-view, LVIS-derived construction remains bounded by reconstruction, annotation, licensing, and coverage constraints.

  • Scope boundary: The evaluation focuses on perception and grounding, leaving grasp planning, collision-aware motion, and physical execution for future work.The benchmark is intended for language-conditioned object-part localization and open-world 3D affordance perception.
  • Data limitations: Single-view reconstruction may be less reliable on unobserved surfaces, and label quality depends on upstream reconstruction and segmentation.The benchmark retains the naturally long-tailed distribution of object interactions.
  • Coverage: The LVIS provenance motivates future expansion toward broader geographic and cultural coverage.This is a stated direction rather than a current coverage guarantee.
  • Release constraints: Release artifacts must preserve LVIS identifiers and provenance and comply with applicable LVIS and source-image redistribution terms.The package includes preprocessing scripts, split manifests, configuration snapshots, checkpoint hashes, and reproduction commands.
Loading 2608.20720v1…