PaperlayerView exact revision on arXiv

Source-linked AI summary

RoboRefer: Towards Spatial Referring with Reasoning in Vision-Language Models for Robotics

Enshen Zhou, Jingkun An, Cheng Chi, Yi Han, Shanyu Rong, Chi Zhang, Pengwei Wang, Zhongyuan Wang, Tiejun Huang, Lu Sheng, Shanghang Zhang

arXiv:2506.04308v4cs.ROcs.AIcs.CV

TL;DR

Spatial referring requires robots to understand complex 3D scenes and reason about instruction-indicated locations, capabilities that existing VLM methods inadequately support. RoboRefer combines depth-aware SFT with RFT for multi-step reasoning, supported by RefSpatial and RefSpatial-Bench, and reports strong benchmark and real-world robotic results.

  • Problem

    Existing VLM approaches inadequately handle complex 3D scenes and multi-step spatial reasoning for robotic interaction.

  • Method

    RoboRefer uses a dedicated depth encoder during SFT, followed by RFT with metric-sensitive process rewards for multi-step spatial referring.

  • Results

    RoboRefer achieves state-of-the-art single-step spatial understanding and surpasses all baselines on RefSpatial-Bench, exceeding Gemini-2.5-Pro by 17.4% average accuracy.

  • Takeaways & Limitations

    RoboRefer generalizes to long-horizon dynamic tasks across diverse robots and tasks in cluttered real-world scenes.

  • Takeaways & Limitations

    Current models mainly use qualitative spatial relations and 2D image-plane coordinates, requiring depth-based conversion to 3D.

Abstract

from arXiv · show

Spatial referring is a fundamental capability of embodied robots to interact with the 3D physical world. However, even with the powerful pretrained vision language models (VLMs), recent approaches are still not qualified to accurately understand the complex 3D scenes and dynamically reason about the instruction-indicated locations for interaction. To this end, we propose RoboRefer, a 3D-aware VLM that can first achieve precise spatial understanding by integrating a disentangled but dedicated depth encoder via supervised fine-tuning (SFT). Moreover, RoboRefer advances generalized multi-step spatial reasoning via reinforcement fine-tuning (RFT), with metric-sensitive process reward functions tailored for spatial referring tasks. To support SFT and RFT training, we introduce RefSpatial, a large-scale dataset of 20M QA pairs (2x prior), covering 31 spatial relations (vs. 15 prior) and supporting complex reasoning processes (up to 5 steps). In addition, we introduce RefSpatial-Bench, a challenging benchmark filling the gap in evaluating spatial referring with multi-step reasoning. Experiments show that SFT-trained RoboRefer achieves state-of-the-art spatial understanding, with an average success rate of 89.6%. RFT-trained RoboRefer further outperforms all other baselines by a large margin, even surpassing Gemini-2.5-Pro by 17.4% in average accuracy on RefSpatial-Bench. Notably, RoboRefer can be integrated with various control policies to execute long-horizon, dynamic tasks across diverse robots (e,g., UR5, G1 humanoid) in cluttered real-world scenes.

1 Introduction

RoboRefer addresses the underexplored challenge of combining precise single-step 3D spatial understanding with generalized multi-step reasoning for spatial referring. It introduces a dedicated training strategy, dataset, benchmark, and evaluations demonstrating strong spatial performance and robotic generalization.

  • Motivation: Existing work emphasizes single-step spatial understanding, while compositional multi-step spatial reasoning remains underexplored.The gap limits investigation of how precise spatial understanding can support complex spatial referring.
  • Method: RoboRefer combines a dedicated depth encoder trained with SFT and an RFT stage using metric-sensitive process rewards.The design preserves the RGB branch while decomposing complex references into sequential analytical steps.
  • Data and benchmark: RefSpatial provides 20M QA pairs across 31 spatial relations and reasoning processes of up to 5 steps.The dataset combines 2D, 3D embodied-video, and simulated data, with RGB-D inputs supporting depth alignment.
  • Data and benchmark: RefSpatial-Bench contains 200 real-world images, with over 70% of samples requiring multi-step reasoning up to 5 steps.The benchmark includes manually annotated object-location and placement tasks with precise masks.
  • Results: 89.6% average success rate is achieved by SFT-trained RoboRefer on existing single-step spatial reasoning benchmarks.The model is reported to achieve state-of-the-art performance on benchmarks including CV-Bench and BLINK.
  • Results: RoboRefer surpasses Gemini-2.5-Pro by 17.4% average accuracy on RefSpatial-Bench and generalizes across robots and tasks.Reported deployments include long-horizon dynamic tasks with UR5 and G1 humanoid robots in cluttered real-world scenes.

2 Related work

Related work develops spatial understanding and referring through tool-based or data-driven VLM approaches, but robotics requires precise 3D localization and reasoning beyond conventional 2D cues. RoboRefer responds with depth-enhanced SFT followed by RFT for multi-step spatial referring.

  • Spatial Understanding with VLMs: Spatial understanding concerns object properties and inter-object relations, whereas spatial reasoning draws higher-level inferences from them.Existing VLM approaches use either integrated vision tools or fine-tuning with spatial annotations.
  • Referring with VLMs for Robotics: Robotic referring requires unique target localization in 3D, extending beyond 2D attributes and image-plane positions.Prior approaches incorporating basic spatial cues often struggle with complex environments and instructions.
  • Reinforcement Fine-tuning for VLMs: Most VLM reinforcement-fine-tuning methods rely on 2D perception, while RoboRefer first strengthens depth-aware understanding with SFT before RFT reasoning.The two-stage strategy uses intermediate perception outputs from SFT to support multi-step spatial referring.

3 Method

RoboRefer formulates spatial referring as point prediction and combines separate RGB-depth processing with sequential SFT and RFT. RefSpatial supplies diverse, fine-grained data for spatial understanding and multi-step reasoning.

  • Task formulation: Spatial referring predicts a single 2D image point from RGB or RGB-D observations and a spatially constrained instruction.The point specifies a target location or destination and can encode properties, relations, and multi-step reasoning.
  • VLM Architecture: RoboRefer uses separate RGB and depth encoders whose features are aligned with the language model for visual question answering or point prediction.This design avoids sharing the RGB encoder with depth inputs, thereby addressing modality interference concerns described for prior approaches.
  • Supervised Fine-tuning: SFT first aligns the depth projector, then fine-tunes the model on fine-grained spatial annotations and explicit multi-step reasoning data.Only the depth projector is updated during depth alignment; the subsequent stage fine-tunes all parameters on RefSpatial.
  • Reinforcement Fine-tuning: RFT follows SFT with GRPO and metric-sensitive outcome rewards to improve point prediction and generalization to novel spatial constraints.SFT initialization provides a spatial-understanding prior, while normalized group rewards and KL regularization guide and stabilize policy updates.
  • RefSpatial Dataset: RefSpatial integrates 2D images, 3D embodied videos, and simulated scenes with fine-grained referents and annotated multi-step reasoning processes.The dataset includes multiple same-category objects, hierarchical captions, and scalable sources designed for spatial referring.

4 Experiments

Experiments show that RoboRefer achieves strong single-step spatial understanding and substantially improves multi-step referring, unseen-case generalization, and real-world robotic execution. Ablations further support its depth encoder, mixed-source data recipe, and process reward design.

  • Single-step Spatial Understanding: Depth input improves 3D benchmarks by 1.5% relative over 2D benchmarks, while RGB-only inference remains effective.The model benefits from RGB-D training and can reuse the RefSpatial dataset for strong RGB-based spatial understanding.
  • Multi-step Spatial Referring: 17.4% absolute improvement over Gemini-2.5-Pro makes RoboRefer-RFT the strongest model on RefSpatial-Bench.The benchmark evaluates cluttered real-world scenes, with Location and Placement subsets and more than 70% of samples requiring multi-step reasoning.
  • Multi-step Spatial Referring: 9.1% absolute improvement over RoboRefer-SFT on unseen relation combinations indicates stronger generalization from RFT.These combinations were omitted during SFT/RFT training, and the result is consistent with RFT leveraging learned spatial knowledge beyond the training distribution.
  • Real-world Evaluation: RoboRefer supports long-horizon manipulation and navigation in cluttered, dynamic environments when integrated with control policies.The system updates at 2.5 Hz and adapts to object movement and rotation during real-world tasks, including G1 humanoid navigation with spatially constrained pick-and-place.
  • Ablation Study: A combined 2D, 3D, and simulated-data recipe yields optimal ablation performance, while a dedicated depth encoder preserves image understanding.The data sources provide complementary outdoor, embodied-indoor, and spatially diverse coverage; the dedicated encoder outperforms a shared encoder under limited RGB-only data.
  • Ablation Study: Process reward produces a 5-point improvement by refining intermediate perception and improving point predictions for complex spatial relations.The reward uses key-step annotations from RefSpatial to supervise step-wise perception.

5 Conclusion and Future work

The paper presents RoboRefer as a 3D-aware VLM combining SFT-based spatial understanding with RFT-based multi-step reasoning, supported by RefSpatial and RefSpatial-Bench. Future work targets human intent understanding and more precise 3D perception.

  • Conclusion: RoboRefer combines a separate depth encoder trained with SFT and metric-sensitive process-reward RFT for spatial referring.The approach addresses both single-step accurate understanding and generalized multi-step spatial reasoning.
  • Conclusion: RefSpatial supports SFT and RFT training, while RefSpatial-Bench evaluates multi-step spatial referring.The paper reports broad experimental effectiveness and potential for robotic applications.
  • Future Work: Future work must improve understanding of brief, ambiguous human instructions and direct quantitative 3D reasoning.Current models mainly use qualitative relations and predict 2D image-plane coordinates before depth-based conversion to 3D.

A Discussion

The discussion positions RoboRefer as a spatial-referring system distinct from simpler VQA approaches, and explains RFT, backbone choice, real-world depth assumptions, and the multi-source dataset construction.

  • Distinction from Related Work: RoboRefer addresses spatially constrained instructions and step-by-step 2D point localization, unlike SpatialRGPT’s simpler VQA setting with externally provided regions.Its training pipeline also adds process-based RFT after SFT to improve multi-step reasoning and generalization.
  • RFT Justification: 9.1% accuracy improvement over SFT on unseen relation combinations supports RFT’s generalization benefit.RFT also consistently outperforms SFT across reasoning steps, especially at larger step counts.
  • Backbone Choice: NVILA is selected because it outperforms comparable open-source VLMs in spatial understanding, while RefSpatial remains transferable to other backbones.Qwen2.5-VL-7B also shows notable improvement after partial RefSpatial training.
  • Real-world Assumption: Real-world evaluation assumes accurate depth-to-3D mapping from 2D predictions given known camera intrinsics and extrinsics.The paper acknowledges depth noise and partial observations as real-world challenges and uses prior-work assumptions for the mapping.
  • Depth Robustness: DepthAnything V2 provides the highest robustness among evaluated depth inputs under real-world depth noise.The model uses relative depth estimation to reduce noise from real cameras and evaluates success rates under noisy depth conditions.
  • Dataset Construction: RefSpatial combines 2D web images, 3D embodied videos, and simulated scenes in a scalable pipeline for spatial referring.Its construction includes image filtering, scene-graph generation, 3D annotation processing, simulation, and QA generation.

B.1.1 Multi-Stage Image Filtering

The dataset pipeline combines coarse SigLIP2 filtering with fine-grained Qwen2.5-VL filtering to retain images with spatial semantics, then enriches retained data with spatially precise object descriptions.

  • 1.7M training images from OpenImages require filtering because text-only graphics, QR codes, medical scans, and abstract art lack relevance for spatial referring.
  • SigLIP2 efficiently removes low-quality or off-theme images using positive and negative textual labels, preserving object diversity, depth cues, and scene context.
  • Qwen2.5-VL-7B performs slower but more precise fine-grained filtering, retaining 846k of 934k SigLIP2-filtered images.
  • The two-stage pipeline removes unsuitable content such as artwork, dim scenes, grayscale images, geometric distortions, and collages while retaining images rich in spatial relationships.
  • GroundingDINO combined with RAM generates more concise one-to-one object detections than Florence-2, supporting precise object referring.
  • Spatially enriched object descriptions append relative ordering, such as ordinal positions along the axis with the largest spatial variation, to produce discriminative referring expressions.

B.2.2 Inherent Challenges and Limitations in CA-1M

CA-1M presents annotation-quality and semantic-label limitations that can compromise scene-graph construction and spatial reasoning.

  • CA-1M contains ambiguous or semantically insignificant annotations with unclear boundaries or unidentifiable content.
  • Such annotations introduce training noise, hamper spatial understanding, and may mislead the model’s perception of object relationships.
  • Most CA-1M objects lack semantic categories, preventing category-dependent queries such as identifying a red chair relative to a table.

B.2.3 Addressing Limitations: Object Annotation and Bounding Box Filtering

The pipeline improves CA-1M usability by adding semantic labels, refining bounding-box correspondences, and filtering ambiguous annotations before downstream 3D processing. It also filters scenes for visible placement platforms and samples unoccupied spatial regions for object placement.

  • Object Annotation: The annotation pipeline combines GroundingDINO, RAM, and Florence-2 to assign semantic labels and predict visually coherent 2D bounding boxes.
  • Bounding Box Filtering: Bidirectional matching uses IoU and one-to-one assignment to remove redundant or weakly aligned CA-1M boxes.For each retained prediction, the CA-1M box with the highest IoU is selected as its unique match.
  • Downstream Processing: The refined model-predicted boxes better align with visible object extents and support more accurate instance-mask extraction.
  • Platform Filtering: Qwen2.5-VL filters out images without plausible horizontal platforms while retaining scenes containing floors, tabletops, beds, or similar surfaces.
  • Unoccupied-Point Sampling: Free-space sampling searches directional sectors, shrinks vertical projections to 80%, and evaluates regions between object projections for occupancy.The directional sectors use a radius based on footprint diagonal or 20 cm, whichever is larger.

C Implementation Details and Samples of RefSpatial-Bench

RefSpatial-Bench evaluates location and placement referring in complex indoor scenes, including unseen spatial-relation combinations for generalization testing. Its implementation uses a base VLM with dynamic-resolution vision processing and separate image, depth, and language connectors.

  • Benchmark Design: RefSpatial-Bench contains 100 Location samples and 100 Placement samples, each with an image, referring caption, and precise masks.
  • Generalization Evaluation: The Unseen set contains 77 samples with novel spatial-relation combinations absent from RefSpatial training data.
  • Reasoning Complexity: Reasoning-step annotations count anchor objects and disambiguating directional relations, cap complexity at five steps, and treat values above five as yielding diminishing returns.
  • Benchmark Expansion: RefSpatial-Expand-Bench extends evaluation to additional indoor settings such as shops and factories and to outdoor scenes including streets, parking lots, and parks.
  • Model Implementation: The model uses NVILA with dynamic-resolution visual encoding, Qwen2, and separate linear connectors for image and depth representations.

D.2 Training Data

Training proceeds from supervised depth alignment and spatial-understanding enhancement to reinforcement fine-tuning with structured, metric-sensitive rewards. The data and objectives support both direct point prediction and explicit intermediate reasoning.

  • SFT Data: The SFT stage uses RefSpatial RGB-D data for depth alignment, then combines RGB-D, RGB, and benchmark data for full-parameter spatial-understanding enhancement.
  • RFT Data: The RFT stage uses 100k RefSpatial samples containing detailed intermediate reasoning steps and final answers.
  • SFT Objective: SFT maximizes the likelihood of generating answers conditioned on sensor observations and textual questions, with outputs optionally containing reasoning steps.
  • RFT Optimization: RFT samples multiple actions from the SFT-initialized policy and compares their rewards to update relative advantages.
  • Reward Design: The reward combines outcome-format, point-accuracy, process-format, and key-step accuracy components, with process rewards scaled by α=0.25.
  • Process Rewards: Rule-based process rewards evaluate intermediate perceptual results with attribute-specific metrics and without requiring a Process Reward Model.

E.1 Experiments Compute Resources

The paper reports substantial compute requirements for data preparation and model training, with the RFT implementation especially costly because its RGB-D architecture cannot use common inference accelerators.

  • Data Preparation: Data preparation includes 8.5 hours for coarse OpenImages filtering, 2.5 days for fine-grained filtering, and one week for simulator generation.
  • Data Preparation: Pseudo-3D scene-graph construction requires 10 hours for depth estimation, 10 hours for additional geometric processing, and 18 hours for object-caption generation.
  • Model Training: SFT spatial-understanding enhancement runs for two days for 2B models and nearly one week for 8B models across 10 nodes.
  • Model Training: RFT for the 2B variant runs on one node for three days and is over twice as slow as other Qwen 2/2.5-VL-based methods.
  • Computational Constraint: The RFT slowdown is attributed to RGB-D processing, architectural modifications, and incompatibility with vLLM or SGLang acceleration.

E.2 Spatial Understanding Benchmarks

The evaluation covers public single-step spatial understanding benchmarks and compares proprietary, general open-source, and spatially specialized VLMs.

  • The benchmarks cover 2D spatial relation, 3D depth order, 3D distance, relative depth, configuration, and related spatial understanding tasks.
  • Non-spatial tasks, including 2D counting, art style, and IQ tests, are excluded from evaluation.
  • The comparison includes proprietary VLMs, general-VQA open-source VLMs, and spatially specialized models.

E.3 Spatial Referring Benchmarks

The spatial referring evaluation extends prior robotic benchmarks with a more demanding real-world benchmark, while also examining perception, ablations, and robot integration.

  • E.3 Spatial Referring Benchmarks: Existing robotic referring benchmarks RoboRefIt, Where2Place, and RoboSpatial are limited to two reasoning steps.RoboRefIt focuses on object location, while Where2Place and RoboSpatial address placement relative to anchor objects.
  • E.3 Spatial Referring Benchmarks: The evaluation compares Gemini-2.5-Pro with spatially specialized models possessing basic spatial referring abilities.
  • E.3 Spatial Referring Benchmarks: On Open6DOR V2, the model is evaluated only on the position track because the work focuses on location and placement rather than 6DOF manipulation.Perception succeeds nearly 100% of the time, while failures are primarily attributed to motion-planning errors such as IK failures or collisions.
  • E.3 Spatial Referring Benchmarks: The robot demonstrations use predicted 2D points for grasping, placement, navigation, and dynamic interruption or redirection.Depth, camera intrinsics, calibration, segmentation, grasping, and SLAM components convert predictions into robot actions.
  • E.3 Spatial Referring Benchmarks: Combining RGB and RGB-D data during SFT improves image-encoder spatial understanding, whereas RGB-D-only training can cause over-reliance on the depth encoder.

G More Discussion on Limitations and Future Work

The discussion identifies a boundary in ambiguous human instructions and notes broader VLM risks, while presenting visual examples and dataset licensing information.

  • Limitations: The model relies on precise descriptions and accurate anchor references, limiting performance when real-world instructions are ambiguous.Resolving such instructions may require visual-linguistic reasoning, process of elimination, and human prior knowledge.
  • Limitations: Human interpretation can use probabilistic preference and spatial compatibility to resolve otherwise ambiguous references.Examples include choosing the plate most likely aligned with multiple drinks and selecting a pair with enough physical space for placement.
  • Future Work: RefSpatial lacks procedurally generated multi-step data embedding human priors and intent understanding.Suggested directions include intent-aware procedural synthesis and co-training with intent-comprehension datasets such as PixMo-Points.
  • Broader Considerations: RoboRefer inherits VLM risks including hallucinations, base-model biases, and increased energy consumption from model upscaling.
  • Broader Considerations: The paper argues that broader release could support continued development of spatial referring with reasoning across robotic arms and humanoids.
Loading 2506.04308v4…