PaperlayerView exact revision on arXiv

Source-linked AI summary

UniRAP: Towards Unified Part-level Physical Affordance Reasoning and Actionable Perception

Linfei Li, Ruining Hu, Lin Zhang, Zhong Wang, Fengyi Zhang, Ying Shen, Binqiang Wang, Xin Zhang

arXiv:2610.29768v1cs.CV

TL;DR

Vision-language perception has advanced in semantic alignment and localization, but fine-grained affordances and contact geometry remain difficult to ground from language. UniRAP combines multimodal prompting, a dense affordance decoder, and curriculum-based transfer training; experiments report gains across perception and interaction alignment. The authors note that compute constraints limited scaling the model and training data.

  • Problem

    Vision-language systems can localize scene elements, but fine-grained part-level affordances and contact representations needed to ground instructions in physical interaction remain limited.

    Source: §1 · pp. 1–2

  • Method

    UniRAP unifies visual, textual, and optional reference prompts with a shared token interface and decoder for affordance masks, detections, and 4-DoF interaction poses.

    Source: §1 · pp. 2–3, §3 · p. 4

  • Results

    UniRAP reports consistent gains in perception and interaction alignment, with a favorable accuracy–stability trade-off across general segmentation, affordance perception, and part-level interaction grounding.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    The authors conclude that UniRAP establishes a more robust foundation for language-guided robotic systems in dynamic environments.

    Source: §5 · p. 10

    Compute constraints prevented further scaling of training data or model size, limiting potential performance gains.

    Source: §D.1 · p. 26

Abstract

from arXiv · show

Vision-language perception has achieved impressive progress in aligning natural language with visual observations, yet grounding high-level semantics into part-level physical interaction remains challenging. To address this gap, we propose UniRAP, a unified model for inferring part-level physical affordances and mapping language instructions to actionable geometric representations. UniRAP formulates this problem as a conditional multimodal generation task, integrating visual inputs, textual instructions, and optional prompts into a shared spatiotemporal representation space through a unified interface token mechanism. To predict executable contact geometry, we introduce a Unified Affordance Decoder (UAD), which jointly performs object detection, part-level affordance segmentation, and 4-DoF interaction pose estimation by leveraging intermediate segmentation features. In addition, we propose a curriculum-based transfer training strategy that progressively adapts the model from general visual parsing to interaction-aware perception, improving data efficiency under limited textual annotations. Experiments show that UniRAP achieves state-of-the-art performance on referring expression segmentation, affordance grounding, and interaction pose estimation, while maintaining strong spatiotemporal consistency in dynamic video scenarios. These results demonstrate the effectiveness of UniRAP as a unified perception framework for language-guided physical manipulation.

1 Introduction

Vision-language systems can align language with visual scenes, but lack the part-level affordances and contact geometry needed to ground instructions in physical interaction. UniRAP combines multimodal prompting, actionable geometric outputs, and transfer training to address this gap.

1 Introduction

  • Existing vision-language perception localizes scene elements but remains largely semantic, while embodied systems also need to reason about how to make physical contact.This requires interaction-relevant affordances, pixel-level alignment, and part-level structural understanding.Source: §1 · pp. 1–2
  • Prior approaches remain limited by static or object-centric representations that do not explicitly capture part-level affordances and contact regions for interaction feasibility.The cited discussion also identifies precision and stability concerns in autoregressive grasp-pose generation.Source: §1 · p. 2
  • UniRAP maps multimodal prompts to fine-grained contact geometry across images and videos, including object detections, part-level affordance masks, and interaction representations.It uses a 4-DoF grasp pose as an explicit representation linking linguistic semantics with physical interaction.Source: §1 · pp. 2–3
  • A unified prompting and decoding pipeline shares localization signals through <REF> and <SEG>, while a Unified Affordance Decoder reuses intermediate features across tasks.Curriculum-based transfer training progressively develops actionable perception while aiming to preserve general segmentation and reasoning performance.Source: §1 · p. 3
  • UniRAP reports a favorable accuracy–stability trade-off across general segmentation, affordance perception, and part-level interaction grounding.The stated contributions include consistent multi-granularity affordance representations in image and video domains.Source: §1 · p. 3

2 Related Work

Prior vision-language perception progressed from coarse localization to affordance and interaction tasks, but language understanding and interaction-centric modeling remain limited. UniRAP is presented as unified actionable affordance perception across images and videos.

Non-LLM Vision-Language Perception

  • Non-LLM vision-language perception moved from object detection and semantic segmentation toward part-level affordance segmentation, contact prediction, and 4-DoF grasp estimation.These methods remain restricted to simple category-level language and struggle with compositional, reasoning-intensive instructions.Source: §2 · p. 3

LLM-based Vision-Language Perception

  • LLM-based methods support localization and dense segmentation, but cited affordance approaches largely lack interaction-centric modeling or video support.The passage identifies RAGNet as extending LISA to affordance segmentation, while noting its lack of interaction-centric modeling and video capability.Source: §2 · p. 3
  • UniRAP’s overview combines visual input, text, and optional visual references in a shared <SEG> representation before decoding masks, detections, and 4-DoF poses.The figure caption also describes memory attention and an MLP projector in this pipeline.Source: Figure 2 · p. 4
  • Compared with RealVLG-R1, UniRAP is presented as end-to-end reasoning-aware actionable affordance modeling across images and videos.The passage characterizes RealVLG-R1 as limited by unstable text representations, weak reasoning, and lack of video support.Source: §2 · p. 4

3 UniRAP

UniRAP unifies textual instructions, optional visual prompts, and image or video features around shared interface tokens, then predicts affordance masks, boxes, and 4-DoF interaction poses. Its decoder shares intermediate representations across these tasks, while curriculum training progresses from general visual parsing to interaction-aware perception.

3 UniRAP

  • UniRAP processes images or videos, textual prompts, and optional visual references to jointly predict affordance masks, region detections, and 4-DoF interaction poses.For videos, memory attention incorporates historical frame features to maintain temporal consistency.Source: §3 · p. 4

3.1 Unified Vision-Language Spatiotemporal Modeling

  • Visual and textual references are aligned through a shared <SEG> representation, with random Fourier mapping encoding visual prompts’ spatial and temporal information.The projected <SEG> representation is fused with memory-enhanced visual features for decoding.Source: §3.1 · pp. 4–5

3.2 Unified Affordance Decoder

  • UAD transfers intermediate representations from abundant vision-language segmentation data to fine-grained interaction tasks with scarce annotations.It integrates segmentation, detection, and part-level contact reasoning in a shared representation space.Source: §3.2 · pp. 5–6
  • The mask head modulates upsampled visual features with mask-token projections and selects among four candidate logits to produce the final segmentation mask.A stability-based selection strategy iteratively refines the candidates.Source: §3.2 · p. 5
  • The bbox head refines a mask-derived initial box using bbox-token modulation, mask-guided pooling, and residual regression.A consistency constraint between the predicted box and mask suppresses mask noise.Source: §3.2 · pp. 5–6
  • The grasp head samples foreground anchors, pools FiLM-conditioned features with spatial queries, and predicts grasp geometry and a quality score.The grasp task is framed as regression for part-level contact-region estimation.Source: §3.2 · p. 6

3.3 Curriculum Transfer Training

  • Four training stages progress from language-prompted general pre-training through affordance and visual-referring pre-training to actionable-affordance fine-tuning.The final stage jointly predicts masks, boxes, and 4-DoF poses using limited vision-language grasp data.Source: §3.3 · pp. 6–7
  • Training combines language-modeling and task-specific losses, with mask-in-box and grasp-center alignment constraints enforcing spatial consistency.Task losses include mask, IoU, objectness, box-regression, and grasp-pose terms.Source: §3.3 · p. 7

4 Experiments

Experiments evaluate UniRAP on general and affordance segmentation, actionable perception, and real-world grasping. Results show strengths across these tasks, with performance boundaries on ReasonSeg and coarse detection and a direct-grasping advantage in robotic trials.

4.1 General Segmentation Performance

  • UniRAP achieves state-of-the-art referring expression segmentation, strong ReVOS results against 3B-scale baselines, and competitive but not best ReasonSeg performance [14].On ReVOS it remains competitive with larger models, while the ReasonSeg shortfall is attributed to stronger multi-step reasoning demands.Source: §4.1 · pp. 7–8, Table 1 · p. 8, Table 2 · p. 8

4.3 Actionable Perception Performance

  • Up to a threefold gAcc improvement over RealVLG-R1 in novel settings accompanies UniRAP’s advantages in end-to-end segmentation and 4-DoF grasp estimation.RealVLG-R1 retains an advantage in coarse object detection; segmentation baselines use a two-stage bbox + SAM2 pipeline.Source: Table 5 · p. 9, §4.3 · p. 9

4.2 Referring and Reasoning Affordance Segmentation Performance

  • 17.6 gIoU and 15.1 cIoU gains on zero-shot 3DOI accompany stronger referring-affordance performance and generalization to unseen scenarios.The evaluation covers annotated HANDAL, GraspNet, and 3DOI datasets; gray marks zero-shot benchmarks.Source: Table 3 · p. 8, Table 4 · p. 8, §4.2 · p. 9

4.4 Ablation Study

  • 9.9 and 15.4-point referring and reasoning gIoU gains on 3DOI follow affordance pre-training; actionable fine-tuning raises grasp accuracy from 19.6 to 63.3.Visual referring pre-training with the Grasp Head raises grasp accuracy from 26.4 to 63.3; the Bbox Head improves ReasonSeg, ReVOS, and 3DOI, while a heatmap-based Grasp Head reduces accuracy from 63.3 to 13.4.Source: §4.4 · pp. 9–10, Table 6 · p. 9

4.5 Application: Actionable Perception for Robotic Grasping

  • 88.5% average success with direct UniRAP 4-DoF grasping exceeds RealVLG-R1 (62.3%) and UniRAP+GraspNet (55.4%).The Franka FR3 setup used objects 1–6 for referring grasping and 7–13 for reasoning-based grasping; RAGNet+GraspNet achieved 43.8%.Source: Table 7 · p. 10, §4.5 · p. 10, Figure 5 · p. 10

5 Conclusion

UniRAP connects vision-language reasoning with fine-grained physical contact perception through a token-based interface and dense decoder. The conclusion reports gains across perception and interaction alignment and describes the model as an end-to-end actionable affordance framework for images and videos.

5 Conclusion

  • UniRAP maps language instructions to affordance segmentation, target-region detection, and 4-DoF interaction pose estimates, with reported gains in perception and interaction alignment.The authors describe it as a more robust foundation for language-guided robotic systems in dynamic environments and, to their knowledge, the first end-to-end reasoning-aware unified model for actionable affordance perception across images and videos.Source: §5 · p. 10

A.1 Architecture

The section argues that conventional semantic-to-dense prediction is insufficient for physical interaction, which requires reasoning about feasible, task-relevant contact regions. UniRAP extends the paradigm through a token-level interface that aligns language and visual prompts for structured interaction-oriented prediction.

A.1 Architecture

  • Prior embedding-as-mask approaches map semantic representations directly to pixel space, with differences chiefly in task formulation and supervision granularity.The section characterizes this as an implicit semantic-to-dense prediction assumption.Source: §A.1 · p. 20
  • Physical contact prediction requires identifying a local contact region that is geometrically and functionally feasible, beyond recognizing an object and locating it.The section identifies local geometry, interaction affordances, and task-relevant functional semantics as dimensions beyond conventional segmentation formulations.Source: §A.1 · p. 20
  • UniRAP uses the <SEG> representation to encode potential interaction intent and localize contact regions subject to physical and functional constraints.This extends the unified paradigm from perception-driven dense prediction toward interaction-oriented reasoning.Source: §A.1 · p. 20

A.2 Visual Referring Encoder

  • UniRAP represents point and box prompts as spatiotemporal coordinates and uses random Fourier features to embed them.The encoding is intended to capture high-frequency spatial and temporal variations for precise localization and interaction-sensitive reasoning.Source: §A.2 · p. 21
  • Prompt embeddings are projected into discrete tokens and interact with the LLM, allowing prompt features to enter the semantic representation of the <SEG> token.Unlike direct fusion with visual encoder features, this design aligns prompt features within a unified semantic space.Source: §A.2 · p. 21
  • Unlike mask-based prompt conversion for region-based visual question answering, UniRAP maps structured prompts into a text-guided <SEG> token representation space.The authors state that this reduces representation-level distributional discrepancies and mitigates cross-task transfer complexity.Source: §A.2 · p. 21

B.1 Implementation Details

The implementation uses Qwen2.5-VL and SAM 2.1, with center padding to preserve geometry-sensitive coordinates. Grasp poses use a 4-DoF representation with angle periodicity encoded through doubled-angle sine and cosine.

B.1 Implementation Details

  • UniRAP is built on Qwen2.5-VL (3B) and SAM 2.1, with the visual prompt encoder hidden dimension set to 256.Source: §B.1 · p. 21
  • Annotations are center-padded to 768 × 768 rather than directly resized to avoid geometric distortion in bounding-box and grasp-pose coordinate inference.The authors note that naive resizing can exacerbate distortion and degrade prediction accuracy.Source: §B.1 · p. 21
  • Grasp poses use (cx, cy, cos 2θ, sin 2θ, w), with 32 candidate poses and doubled-angle encoding for angular periodicity.Video experiments randomly sample 8 frames per instance; training uses LoRA and runs for one epoch on 8 A800 GPUs.Source: §B.1 · p. 21

B.2 Training Recipe

The training recipe progressively moves from general segmentation pretraining through affordance and visual-referring pretraining to actionable affordance fine-tuning. The stages train different components to adapt the model while preserving its textual reasoning capability.

B.2 Training Recipe

  • Stage 1 uses 461K segmentation samples to train LoRA adapters, the <SEG> projection, SAM2 parameters, and the decoder segmentation head.Other decoder components are frozen or excluded from training.Source: §B.2 · p. 22, Table 8 · p. 22
  • Stage 2 uses a 253K affordance-segmentation dataset, selecting half the HANDAL samples per scene to mitigate bias from redundant scenes.Source: §B.2 · p. 22, Table 8 · p. 22
  • Stage 3 trains the visual prompt encoder, bounding-box and grasp heads, and shared Two-way Transformer using segmentation and grasp datasets.Its objective is to align <REF> prompt features with the <SEG> feature space.Source: §B.2 · p. 22, Table 8 · p. 22
  • Stage 4 initializes from Stage 3 and trains only the Unified Affordance Decoder, using RealVLG, while preserving the VLM's textual reasoning capability.Source: §B.2 · p. 22, Table 8 · p. 22

B.3 Training Objective

The training objective combines task-specific losses with tuned weights, using affordance masks as the primary spatial signal. It also adds asymmetric spatial constraints to promote coherence.

B.3 Training Objective

  • The objective is a weighted sum of task-specific losses: Ltxt, 100 Lmask, 5 Ldice, 5 Liou, 5 Lcls, 0.5 LL1, 0.5 LGIoU, 2 Lgrasp, and 0.1 each for Lmib and Lgcm.The weighting coefficients were determined through validation-based tuning.Source: §B.3 · pp. 22–23
  • Grasp-pose supervision greedily matches predicted configurations to ground truth, combining Huber losses for center, width, and angle with binary cross-entropy for quality.Source: §B.3 · p. 23
  • The mask-in-bounding-box loss penalizes predicted-mask activations outside the predicted object box.Source: §B.3 · p. 23
  • The grasp-center-in-mask loss encourages predicted grasp centers to fall within the predicted affordance mask.The predicted mask is bilinearly interpolated at the grasp-center coordinates; both spatial losses are normalized per object and accumulated over decoding steps.Source: §B.3 · p. 23

B.4 Evaluation Metrics

Evaluation uses task-specific metrics for image and video segmentation, hallucination robustness, and real-world vision-language grasping.

B.4 Evaluation Metrics

  • Image segmentation uses gIoU and cIoU, while video segmentation uses J, F, and J&F; hallucination robustness is measured with the R2VOS robustness score R.Source: §B.4 · p. 23
  • RealVLG-Benchmark evaluation uses gIoU and cIoU for detection, Fβ and Sα for segmentation, and mIoU and gAcc for grasping.A grasp counts as successful for gAcc when the predicted grasp box has IoU greater than 0.25 and angular deviation less than 30°.Source: §B.4 · p. 23

B.5 Qualitative Results

The section presents qualitative results across referring, reasoning, affordance, and video tasks, alongside a motivation for actionable perception as an intermediate representation for robotic manipulation.

B.5 Qualitative Results

  • Qualitative results cover referring and reasoning segmentation on RefCOCO/+/g and ReasonSeg.Source: §B.5 · p. 24
  • The paper also presents qualitative affordance-segmentation results on 3DOI, HANDAL, and GraspNet Novel.Source: §B.5 · p. 24
  • Additional qualitative results show referring and reasoning video object segmentation on ReVOS and actionable perception on RealVLG-Benchmark.Source: §B.5 · p. 24
  • UniRAP maintains consistent actionable predictions across sampled video frames despite viewpoint, occlusion, and scene-configuration changes.The pseudo-clips contain six randomly sampled frames from each RealVLG-GraspNet scene.Source: §B.5 · p. 24

C.1 Motivation

  • VLA systems still face challenges with fine-grained localization and generalization, while language-to-actionable-region correspondences remain difficult in cluttered or contact-dense scenes.Source: §C.1 · p. 24
  • UniRAP models executable vision-language localization as interaction cues for classical geometric planners rather than as a directly learned end-to-end control policy.The authors describe perception-action decoupling as potentially below the theoretical upper bound of end-to-end approaches, while offering transfer across environments, embodiments, cameras, and instructions.Source: §C.1 · p. 24

C.2 UniRAP for Reasoning-Aware Robotic Grasping

UniRAP supports hybrid robotic grasping pipelines through semantic region outputs or language-conditioned 4-DoF poses. In real-world tests, it outperforms RealVLG-R1, with a larger advantage on reasoning prompts.

C.2 UniRAP for Reasoning-Aware Robotic Grasping

  • UniRAP's bounding boxes or masks can filter task-relevant regions for downstream geometry-based grasp synthesis, including GraspNet.This hybrid setup retains classical pipeline efficiency and robustness while adding task-level semantic awareness.Source: §C.2 · p. 24
  • Its language-conditioned 4-DoF grasp poses encode location and orientation constraints and can be mapped to robot coordinates through hand-eye calibration without additional policy learning.Predictions depend on visual context and language-driven intent, reducing ambiguity in grasp selection.Source: §C.2 · pp. 24–25
  • 26.2% higher overall grasp success rate than RealVLG-R1 (3B), with the advantage particularly pronounced on reasoning-based prompts.Evaluation covered referring and reasoning instructions across 13 objects, with 10 independent trials per object.Source: Table 10 · p. 25, §C.2 · p. 25
  • Real-world pose execution transforms predictions into the robot base frame using hand-eye calibration before direct manipulator execution.The setup uses a Franka FR3 arm with an eye-in-hand RealSense D435i camera and 13 household objects.Source: §C.2 · p. 25, Figure 6 · p. 25

D.1 Limitations & Future Work

This section summarizes UniRAP’s stated limitations and future directions, including computational constraints, study scope, and investigation of smaller models and embodied applications. It also covers disclosures on asset and code availability, human-subjects research, core LLM components, and societal impacts.

D.1 Limitations & Future Work

  • Computational constraints prevented scaling the training data or model, limiting potential performance gains, and PixelQA was not included; future work should investigate smaller models for real-time actionable perception.The authors note that smaller-model capability could support edge-device deployment and reduce real-world system computational demands.Source: §D.1 · p. 26

D.2 Potential Societal Impacts

  • The framework is presented as relevant to interaction-intensive embodied applications such as robotic home assistance, security, and healthcare, with no specific negative societal impacts identified for disclosure.The authors also describe it as a practical alternative for other multimodal embodied applications.Source: §D.2 · p. 26

E Licenses

NeurIPS Paper Checklist

  • The authors state that experimental details and cited assets support reproducibility, and plan to release the code publicly once the paper’s status is confirmed.The checklist answers the reproducibility question [Yes].Source: NeurIPS Paper Checklist · pp. 35–36
  • The authors report that each experiment was conducted only once because of limited computational resources, so they did not report error bars or other statistical significance information.The checklist marks the statistical-significance question [No].Source: NeurIPS Paper Checklist · p. 37
  • The authors say they will disclose code after the paper’s status is confirmed, but no new assets are currently available.The checklist marks documentation of new assets as N/A.Source: NeurIPS Paper Checklist · p. 39
  • The paper reports no crowdsourcing or research with human subjects.The checklist marks both the human-subjects instructions/compensation question and IRB question as N/A.Source: NeurIPS Paper Checklist · p. 40
  • The core method development does not use LLMs as important, original, or non-standard components.Source: NeurIPS Paper Checklist · p. 40

Related papers

Loading 2610.29768v1…