PaperlayerView exact revision on arXiv

Source-linked AI summary

RoboRefer: Towards Spatial Referring with Reasoning in Vision-Language Models for Robotics

Enshen Zhou, Jingkun An, Cheng Chi, Yi Han, Shanyu Rong, Chi Zhang, Pengwei Wang, Zhongyuan Wang, Tiejun Huang, Lu Sheng, Shanghang Zhang

arXiv:2506.04308v4cs.ROcs.AIcs.CV

TL;DR

RoboRefer addresses the limited multi-step spatial reasoning of current vision-language models for complex 3D referring tasks. It combines dedicated depth encoding, reinforcement fine-tuning, and RefSpatial training resources, outperforming all baselines and surpassing Gemini-2.5-Pro by 17.4% on RefSpatial-Bench.

  • Problem

    Multi-step spatial referring remains underexplored, while existing methods mainly target single-step understanding and face depth-modality and dataset limitations.

    Source: §1 · p. 2

  • Method

    RoboRefer combines supervised fine-tuning with a dedicated depth encoder and reinforcement fine-tuning using metric-sensitive process rewards, supported by RefSpatial and RefSpatial-Bench.

    Source: §1 · pp. 2–3, §5 · p. 10

  • Results

    RoboRefer outperforms all baselines across single-step and multi-step spatial referring, surpassing Gemini-2.5-Pro by 17.4% in average RefSpatial-Bench accuracy.

    Source: §1 · p. 3

  • Takeaways & Limitations

    RoboRefer supports long-horizon, dynamic spatial interaction across diverse robots and cluttered real-world scenes when integrated with control policies.

    Source: §1 · p. 3

    Current models predominantly use qualitative spatial relations and predict 2D image coordinates, requiring depth-based conversion to 3D.

    Source: §5 · p. 10

Abstract

from arXiv · show

Spatial referring is a fundamental capability of embodied robots to interact with the 3D physical world. However, even with the powerful pretrained vision language models (VLMs), recent approaches are still not qualified to accurately understand the complex 3D scenes and dynamically reason about the instruction-indicated locations for interaction. To this end, we propose RoboRefer, a 3D-aware VLM that can first achieve precise spatial understanding by integrating a disentangled but dedicated depth encoder via supervised fine-tuning (SFT). Moreover, RoboRefer advances generalized multi-step spatial reasoning via reinforcement fine-tuning (RFT), with metric-sensitive process reward functions tailored for spatial referring tasks. To support SFT and RFT training, we introduce RefSpatial, a large-scale dataset of 20M QA pairs (2x prior), covering 31 spatial relations (vs. 15 prior) and supporting complex reasoning processes (up to 5 steps). In addition, we introduce RefSpatial-Bench, a challenging benchmark filling the gap in evaluating spatial referring with multi-step reasoning. Experiments show that SFT-trained RoboRefer achieves state-of-the-art spatial understanding, with an average success rate of 89.6%. RFT-trained RoboRefer further outperforms all other baselines by a large margin, even surpassing Gemini-2.5-Pro by 17.4% in average accuracy on RefSpatial-Bench. Notably, RoboRefer can be integrated with various control policies to execute long-horizon, dynamic tasks across diverse robots (e,g., UR5, G1 humanoid) in cluttered real-world scenes.

1 Introduction

RoboRefer addresses the gap between single-step spatial understanding and underexplored multi-step spatial reasoning in embodied robotics. It combines a dedicated depth encoder, sequential SFT-RFT training, RefSpatial data, and RefSpatial-Bench evaluation to improve spatial referring and real-world robot control.

Task Definition

  • Spatial referring predicts a precise point satisfying complex spatial relations from sensor observations and constrained instructions.The predicted point can support navigation waypoints, manipulation targets, and placement locations in complex 3D environments.Source: §1 · p. 2

Motivation

  • Existing methods mainly target single-step understanding, while multi-step spatial referring remains underexplored because suitable datasets are scarce.Prior approaches either require costly multi-view 3D reconstruction or process depth through shared RGB encoders, risking modality gaps, interference, and degraded pretrained image representations.Source: §1 · p. 2

Method

  • RoboRefer combines a dedicated depth encoder for precise spatial perception with sequential SFT-RFT training for explicit multi-step reasoning.The depth encoder avoids interfering with the RGB branch, while metric-sensitive process rewards refine intermediate reasoning and point predictions.Source: §1 · pp. 2–3

Dataset

  • RefSpatial provides 2.5M examples and 20M QA pairs covering 31 spatial relations and reasoning processes of up to 5 steps.Its 2D, 3D, and simulation sources support bottom-up spatial learning, and RGB-D samples enable depth alignment for SFT.Source: §1 · pp. 2–3

Evaluation

  • 89.6% average success rate on existing single-step benchmarks, while surpassing Gemini-2.5-Pro by 17.4% on RefSpatial-Bench.RefSpatial-Bench contains 200 real-world images, over 70% requiring multi-step reasoning of up to 5 steps; RoboRefer also executes long-horizon dynamic tasks across UR5 and G1 robots.Source: §1 · p. 3

Real-World Generalization

  • RoboRefer generalizes across cluttered real-world robots and tasks, including manipulation and navigation.The model integrates with various control policies for long-horizon, dynamic execution on platforms such as UR5 and the G1 humanoid.Source: §1 · p. 3

2 Related work

Prior work advances VLM-based spatial understanding and robotic referring through tool integration, data-driven fine-tuning, and reinforcement fine-tuning, but remains limited in multi-step 3D spatial reasoning. RoboRefer addresses these gaps with a 3D-aware framework, dataset and benchmark, and two-stage SFT–RFT training strategy.

Spatial Understanding with VLMs

  • Existing spatial-understanding methods use tool-based integration or data-driven fine-tuning, but lack multi-step reasoning annotations and a dedicated spatial-referring benchmark.These methods draw on pseudo-3D, real-world 3D, or simulated data to improve spatial understanding and reasoning.Source: §2 · p. 3

Referring with VLMs for Robotics

  • Robotic referring requires single-target 3D localization beyond 2D attributes and image-plane positions, motivating RoboRefer’s multi-step reasoning framework.Unlike phrase localization and generalized visual grounding, referring expression comprehension emphasizes uniquely identifying one target for precise robotic interaction.Source: §2 · p. 3

Reinforcement Fine-tuning for VLMs

  • RFT extends SFT for VLM reasoning, but existing approaches mostly rely on 2D perception; RoboRefer instead adds depth during SFT before RFT-based multi-step spatial referring.The two-stage strategy uses intermediate perception outputs from SFT to support reasoning in the RFT stage.Source: §2 · p. 3

3 Method

RoboRefer formulates spatial referring as predicting a single image-space point from visual inputs and instructions, then combines dedicated depth encoding, staged SFT/RFT, and the RefSpatial dataset to support precise and multi-step spatial reasoning. RefSpatial integrates diverse 2D, 3D, and simulated data with fine-grained annotations, spatial relations, and reasoning processes for training and evaluation.

3.1 Problem Formulation

  • RoboRefer predicts a single 2D image-space point from RGB or RGB-D inputs and instructions containing object attributes, spatial relations, and multi-step constraints.This point-based formulation maps naturally to 3D through depth and supports navigation, grasping, placement, and localizing visible object parts under occlusion.Source: §3.1 · p. 4

3.2 RoboRefer: A 3D-aware reasoning VLM for spatial referring

  • RoboRefer uses separate RGB and depth encoders with projectors to preserve RGB representations while independently learning depth cues for spatial understanding.Two-stage SFT first aligns the depth projector, then jointly fine-tunes the model on fine-grained spatial annotations, explicit reasoning data, and instruction-following datasets, providing an RFT cold start.Source: §3.2 · pp. 4–5
  • RFT applies GRPO with outcome and metric-sensitive process rewards that enforce reasoning format and measure step-specific spatial prediction errors.The combined reward includes format, point L1, process-format, and accuracy terms, while group-normalized advantages and KL regularization stabilize policy updates; α is set to 0.25.Source: §3.2 · p. 5

3.3.1 Overview

  • RefSpatial combines 2D images, 3D embodied videos, and simulated scenes into 2.5M samples and 20M QA pairs covering fine-grained references, quantitative spatial queries, point prediction, and 31 spatial relations.It supports multiple same-category objects, hierarchical captions, multi-step reasoning annotations, indoor and outdoor scenes, and scalable data generation.Source: §3.3.1 · pp. 5–6, Figure 3 · p. 6

3.3.2 Data Recipe

  • RefSpatial’s data recipe progressively builds spatial understanding from filtered OpenImages, 3D CA-1M frames with bounding boxes and occupancy maps, and procedurally generated simulations with verified assets.The simulated data translates code-based reasoning into textual steps and structured intermediate results, yielding annotated multi-step referring QA pairs.Source: §3.3.2 · pp. 6–7

3.4 Training Details

  • Training uses NVILA 2B/8B for SFT, applies RFT only to the 2B model, reuses RefSpatial with RGB and RGB-D inputs, and supports RGB-only or RGB-D inference with optional inferred depth.The two SFT stages use RefSpatial alone and then a mixture with instruction-tuning and referring datasets, while RFT uses RefSpatial multi-step reasoning data.Source: §3.4 · p. 7

4 Experiments

RoboRefer achieves strong single-step and multi-step spatial referring, with RFT improving benchmark accuracy, generalization, and robotic execution. Ablations show that multimodal data, dedicated depth encoding, and process rewards are central to these gains.

4.1 Single-step Spatial Understanding

  • 5% absolute: RoboRefer-SFT surpasses spatial specialist models and Gemini-2.5-Pro [9] on average, while its 2B variant exceeds NVILA-2B by 21.7%.Depth inference improves 3D benchmarks by 1.5% relative to 2D benchmarks, although RGB-only input remains strong.Source: §4.1 · p. 8

4.2 Multi-step Spatial Referring

  • 17.4% absolute: 2B-RFT surpasses Gemini-2.5-Pro [9] and all baselines on RefSpatial-Bench, while generalizing better to unseen spatial-relation combinations.The 2B-RFT model exceeds 2B-SFT by 9.1% absolute on unseen combinations, which contain relations omitted during training.Source: §4.2 · pp. 8–9

4.3 Public vision-language benchmarks

  • RefSpatial improves 2D referring beyond baselines while joint RGB/RGB-D training preserves or slightly improves general VQA performance versus NVILA-2B [38].On ReCOCO/+/g [87], point predictions are judged within ground-truth boxes; the VQA comparison uses RoboRefer-2B-SFT against NVILA-2B [38].Source: §4.3 · p. 9

4.4 Simulator and Real-world Evaluation for Robotics

  • 6.8% absolute: integrating RoboRefer into Open6DOR improves success rate over 2D-box-based alternatives, while reducing execution time versus GPT-4o by 27.5%.RoboRefer’s point predictions produce more accurate masks and grasp poses under occlusion; its referring also supports long-horizon manipulation and navigation in dynamic clutter.Source: §4.4 · pp. 9–10

4.5 Ablation Study

  • Ablations identify multimodal data, dedicated depth encoding, and process rewards as key components: the full 2D/3D/simulated recipe is optimal, dedicated encoding preserves image understanding, and process rewards add 5 points.Depth improves both single-step and multi-step referring, with larger multi-step gains because precise coordinates and cumulative reasoning amplify depth’s value.Source: §4.5 · p. 10

5 Conclusion and Future work · Appendix Table of RoboRefer

RoboRefer combines 3D-aware perception, accurate single-step understanding, and generalized multi-step spatial reasoning for robotic spatial referring. Future work targets human intent understanding and more precise direct 3D perception.

5 Conclusion and Future work

  • RoboRefer combines a separate depth encoder via SFT with metric-sensitive process rewards via RFT for single- and multi-step spatial referring.The model is supported by RefSpatial for training and RefSpatial-Bench for evaluation.Source: §5 · p. 10
  • RefSpatial provides a large-scale dataset for SFT and RFT training, while RefSpatial-Bench evaluates spatial referring.Source: §5 · p. 10
  • Extensive experiments demonstrate RoboRefer’s effectiveness and potential across a broad range of robotic applications.Source: §5 · p. 10
  • Future work will improve understanding of brief, ambiguous human instructions through intent-aware synthesis or co-training with intent-rich datasets.These directions address cases where the correct location is unique despite underspecified instructions.Source: §5 · p. 10
  • Current models rely mainly on qualitative spatial relations and 2D image-plane coordinates converted into 3D using depth.This dependence limits direct quantitative geometric reasoning and precise 3D prediction.Source: §5 · p. 10
  • Future 3D-perception work will directly model quantitative geometry or predict 3D points and visual traces under spatially constrained instructions.Source: §5 · p. 10

A Discussion · B Implementation Details and Samples of RefSpatial Dataset · B.1 2D Web Image

RoboRefer’s discussion highlights RFT’s gains in unseen-case generalization and multi-step reasoning, while its RefSpatial dataset uses a staged pipeline spanning 2D web images, 3D embodied video, and synthetic simulation. The implementation overview emphasizes progressive spatial learning and the technical details underlying each data source.

A Discussion

  • RoboRefer differs from SpatialRGPT through a harder instruction-driven referring task, textual object references, staged 2D-to-3D-to-simulation data, and process-based RFT after SFT.SpatialRGPT instead addresses simpler VQA, relies on externally provided regions or masks, uses only 2D-generated data, and trains with SFT only.Source: A Discussion · p. 21
  • 2B-RFT surpasses 2B-SFT by 9.1% in RefSpatial-Bench Unseen raw accuracy, while consistently improving performance across reasoning steps, especially at larger steps.The comparison is reported in Tab. 2, and step-wise success rates are reported in Table 8.Source: A Discussion · p. 21, Table 8 · p. 22
  • Real-world evaluation assumes accurate depth-to-3D mapping from known camera intrinsics and extrinsics, while depth noise and partial views are mitigated with depth methods and pixel-level target points.FoundationStereo is used under severe noise, and RoboRefer could support active perception as a spatially aware planner.Source: A Discussion · p. 21
  • DepthAnything V2 provides the highest robustness and success under real-camera depth noise, while mixed RGB and RGB-D SFT training preserves performance by leveraging RGB priors.The real-world experiments use relative depth estimates rather than raw camera depth.Source: A Discussion · p. 22

B Implementation Details and Samples of RefSpatial Dataset

  • The RefSpatial implementation uses a multi-source pipeline designed to adapt general VLMs to spatial referring through bottom-up spatial understanding and reasoning.The dataset construction covers 2D web images, 3D embodied video, and synthetic simulator data, with models, prompts, processing, filtering, and examples detailed for each source.Source: B Implementation Details and Samples… · p. 22

B.1 2D Web Image

  • The 2D Web Image pipeline filters images, constructs pseudo-3D scene graphs, generates hierarchical referential descriptions, and creates diverse QA pairs.Descriptions progress from coarse categories to fine-grained spatial referents.Source: B Implementation Details and Samples… · p. 22

B Implementation Details and Samples of RefSpatial Dataset

  • The 3D Embodied Video pipeline selects data from CA-1M [136], enriches scene graphs with depth maps and oriented 3D boxes, and generates spatial and free-space placement QA pairs.The section also discusses source limitations and mitigation strategies.Source: B Implementation Details and Samples… · p. 22
  • Synthetic simulator data provides synthesized 3D scenes with selected and annotated digital assets, efficient assembly and rendering, and scene-grounded QA generation.This component complements the web-image and embodied-video sources with controlled simulated scenes.Source: B Implementation Details and Samples… · p. 22

B.1.1 Multi-Stage Image Filtering … B.2.1 Why Use CA-1M and How to Pre-process It

This section describes the data-processing pipeline for spatial QA generation, from filtering 2D images and reconstructing pseudo-3D scene graphs to enriching descriptions and diversifying QA pairs. It also details the use and preprocessing of annotated 3D data to support efficient processing.

B.1.1 Multi-Stage Image Filtering

  • The two-stage filter reduces OpenImages to 934k SigLIP2-qualified images and 846k Qwen2.5-VL-retained images by removing irrelevant or visually unsuitable content.SigLIP2 performs fast coarse filtering using positive and negative semantic labels, while Qwen2.5-VL-7B performs slower fine-grained filtering for artworks, dim scenes, grayscale images, distortions, and collages.Source: §B.1.1 · pp. 23–25

B.1.2 Pseudo-3D Scene Graphs Construction

  • Pseudo-3D scene graphs combine RAM labels, GroundingDINO boxes, SAM 2.1 masks, and UniDepth V2 with WildeCamera to represent objects and metric-aware geometry.GroundingDINO+RAM yields concise labels and one-to-one object-box mappings, unlike Florence-2’s ambiguous or redundant detections.Source: §B.1.2 · pp. 26–27

B.1.3 Hierarchical Object Description Generation

  • Hierarchical descriptions disambiguate repeated object categories by appending relative spatial order along the axis with greatest variation, producing a final set of 466k images.Single-instance objects retain dense captions, whereas repeated categories receive ordinal references such as left-to-right or front-to-back positions; low-variance cases are discarded.Source: §B.1.3 · pp. 27–28

B.1.4 Generating Diverse QA Pairs via Pseudo-3D Scene Graphs

  • QA generation uses pseudo-3D scene graphs and hierarchical descriptions to create template, choice, and fact questions focused mainly on reliable qualitative spatial reasoning.Templates cover relative position, relative size, and quantitative cues from estimated depth, coordinates, or coarse monocular depth while accounting for single-view ambiguity.Source: §B.1.4 · pp. 29–30
  • Templates generate QA pairs for 2D object coordinates, depth retrieval, pointwise object identification, approximate depth, and precise spatial relations.Fact templates provide structured declarative statements that support richer QA generation beyond fixed formats.Source: §B.1.4 · pp. 30–31
  • QwQ-32B transforms scene descriptions, object labels, spatial facts, initial QA pairs, and optional multiple-choice questions into natural, complex spatial reasoning QA pairs.Question generation preserves exact object labels, uses scene context, avoids leaking descriptive relations, and answers remain concise, factual, and description-grounded.Source: §B.1.4 · p. 31

B.2.1 Why Use CA-1M and How to Pre-process It

  • CA-1M [136] is selected because its per-frame 2D/3D annotations, depth maps, camera parameters, and scale support egocentric 3D scene graphs and complex spatial data generation.Compared with ARKitScenes [147], ScanNet V2 [148], EmbodiedScan, and 3RScan [149], alternatives lack combinations of per-frame annotations, precise projections, orientation support, or image quality.Source: §B.2.1 · pp. 32–33
  • Every 20th frame is sampled from CA-1M videos to reduce temporal redundancy and computational cost while preserving scene and viewpoint transitions.Figure 11 contrasts minimally varying consecutive frames with sampled frames showing greater temporal variation for downstream scene-graph construction and QA generation.Source: §B.2.1 · p. 33, Figure 11 · p. 33

B.2.2 Inherent Challenges and Limitations in CA-1M

CA-1M’s scale, egocentric perspective, and rich annotations are offset by ambiguous object annotations and widespread missing semantic labels, undermining scene graph construction and spatial understanding.

Overall Limitation

  • These annotation limitations undermine CA-1M’s use for scene graph construction if they are not addressed.The dataset’s large scale, egocentric perspective, and rich annotations do not eliminate these inherent limitations.Source: §B.2.2 · pp. 33–34

Ambiguous or Meaningless Object Annotations

  • Ambiguous or semantically insignificant annotations introduce noise, hamper spatial understanding, and may mislead models about object relationships.Bounding boxes can enclose indiscernible regions such as wall patches or vague background elements, as illustrated in Figure 12.Source: §B.2.2 · pp. 33–34, Figure 12 · p. 34

Widespread Absence of Semantic Labels

  • Most annotated objects lack semantic categories beyond a few structural labels, preventing category-dependent spatial queries such as identifying a chair relative to a table.Unlike ARKitScenes and ScanNet V2, CA-1M generally labels objects only as “object,” leaving categories such as “chair” and “table” unknown.Source: §B.2.2 · p. 34

B.2.3 Addressing Limitations: Object Annotation and Bounding Box Filtering … B.3.5 Question-Answer Pair Generation

This section presents data-generation pipelines that refine object annotations, construct grounded 3D scenes and relations, generate physically valid placement queries, and produce diverse spatial-reasoning QA pairs. These procedures support scene-grounded 3D understanding and multi-step referring through structured relations, expressions, and reasoning processes.

B.2.3 Addressing Limitations: Object Annotation and Bounding Box Filtering

  • A two-stage annotation pipeline combines GroundingDINO, RAM, and Florence-2 with IoU-based bidirectional matching to assign semantic labels and remove noisy CA-1M boxes.Low confidence thresholds maximize candidate recall, while one-to-one refinement retains strongly aligned matches and provides better boxes for instance-mask extraction.Source: §B.2.3 · pp. 34–35

B.2.4 3D Object Description Generation and Scene Graph Construction

  • The resulting 3D scene graphs exploit ground-truth depth, camera parameters, and 3D oriented boxes to support quantitatively grounded QA across 28 spatial relation types.Compared with 2D-based counterparts, the enhanced graphs are described as improving object localization and spatial-relation accuracy in indoor scenes.Source: §B.2.4 · p. 35

B.2.5 Free Space QA Generation for Object Placement

  • The method filters scenes for viable placement platforms, samples free space around target objects, and projects candidate points for visibility filtering before selecting representative target locations.Free-space regions must exceed 0.036m2, and points are discarded when depth differs by more than 2.5 cm; visibility requirements vary by spatial relation.Source: §B.2.5 · pp. 35, 38, 39

B.2.6 Generating Diverse QA via precise 3D annotations

  • Template-based generation augmented with QwQ-32B uses depth maps, camera poses, and per-object 3D boxes to create qualitative and quantitative QA for relations, geometry, free space, and placement.The generated queries distinguish world- and camera-frame above/below relations and include orientation, metric comparisons, free-space reasoning, and language-conditioned coordinate prediction.Source: §B.2.6 · p. 40
  • The pipeline generates diverse 3D QA covering spatial reasoning, geometric comparison, viewpoint inference, environmental understanding, and coordinate-level localization.QwQ-32B is used to generate structurally diverse and semantically rich complex-reasoning QA pairs.Source: §B.2.6 · p. 40

B.3.1 Indoor Scene Generation

  • Indoor scenes are generated with Infinigen, filtered for tabletop area, lighting, realism, and camera accessibility, then adjusted through lighting, viewpoint, height, and distance variation.The initial generation produces over 3k unique indoor scenes.Source: §B.3.1 · p. 41

B.3.2 3D Asset Selection and Preparation

  • More than 3k high-quality 3D assets remain after category, attribute, and manual filtering, including checks for size, geometry, visual quality, and distinguishable views.Assets are sourced from Objaverse’s LVIS dataset, with manual filtering removing irregular geometries that undermine bounding-box-based scaling.Source: §B.3.2 · pp. 41–42

B.3.3 3D Asset Annotation with LLM

  • GPT-4o converts orientation and caption data into structured asset annotations including object descriptions, directional phrases, dominant colors, and category-consistency flags.The generated outputs are constrained to concise phrases and a valid JSON format for downstream template use.Source: §B.3.3 · p. 42

B.3.4 Scene Population and Data Rendering

  • Curated assets are placed predominantly on identified tabletop surfaces in plausible physical arrangements, with 3–9 assets per scene, and rendered in Blender Cycles at 960 × 540 pixels.Rendering uses 2048 samples.Source: §B.3.4 · p. 44

B.3.5 Question-Answer Pair Generation

  • QA pairs combine attribute- and relation-based referring expressions with templates for positions, orientations, distances, betweenness, surface locations, object identification, and empty-space localization.Each QA pair also records a structured reasoning process derived from the scene graph and ground-truth information.Source: §B.3.5 · p. 45

C Implementation Details and Samples of RefSpatial-Bench · D Implementation Details for RoboRefer · D.1 Architecture

RefSpatial-Bench evaluates complex 3D spatial referring through location and placement prediction, including an unseen generalization set and capped multi-step reasoning. RoboRefer combines a dynamic-resolution visual encoder with a dedicated depth encoder, Qwen2 language model, and modality-specific projectors for fine-grained spatial understanding.

C Implementation Details and Samples of RefSpatial-Bench

  • RefSpatial-Bench contains 200 manually annotated samples across Location Prediction and Placement Prediction, plus 77 unseen samples testing novel spatial-relation combinations.Each task has 100 samples with images, referring captions, and precise masks; the unseen set evaluates generalization beyond RefSpatial training combinations.Source: C Implementation Details and Samples… · pp. 45–46
  • Location Prediction maps expressions about attributes, order, or spatial anchors to a target-object point, while Placement Prediction identifies free space from complex spatial references.Placement queries may involve multiple anchors, hierarchical relations, or implied placements.Source: C Implementation Details and Samples… · p. 46
  • Benchmark reasoning steps count disambiguating anchors or linked directional phrases, exclude intrinsic target attributes and weak relations, and cap complexity at 5 steps.Instructions with step ≥3 are already substantially complex, while additional qualifiers beyond five provide diminishing search-space reduction.Source: C Implementation Details and Samples… · p. 46
  • RefSpatial-Expand-Bench broadens evaluation with indoor shops and factories and outdoor streets, parking lots, and parks.Its statistics and RoboRefer evaluation results are reported in Tables 12–14.Source: C Implementation Details and Samples… · p. 46

D.1 Architecture

  • RoboRefer adopts NVILA with a visual encoder, Qwen2 language model, and multimodal projector as its base architecture.The language model is fully fine-tuned during supervised training, providing visual knowledge for downstream 3D spatial reasoning.Source: §D.1 · p. 47
  • Its visual encoder supports dynamic 448 × 448 processing, producing more tokens for higher-resolution inputs to improve fine-grained point prediction.The encoder uses the siglip-so400m-patch14-448 design from NVILA.Source: §D.1 · p. 47
  • A dedicated depth encoder mirrors the image encoder and uses image-encoder initialization to encode relative depth maps as spatial cues for 3D understanding.This disentangled depth pathway supplements visual perception with depth-specific information.Source: §D.1 · p. 47
  • Separate linear connectors align image and depth embeddings with language, enabling modality-specific processing and reducing cross-modal interference.The connectors follow NVILA and are preferred over Q-Former for visual understanding and generalization.Source: §D.1 · p. 47

D.2 Training Data … E Experimental Setting and Details

RoboRefer trains through staged SFT and RFT, combining depth alignment, spatial-understanding enhancement, and reinforcement learning with metric-sensitive, order-invariant process rewards. The pipeline uses large RefSpatial-derived datasets, grouped action sampling, and constrained policy updates to improve spatial referring.

D.2 Training Data

  • The SFT data pipeline uses 3.4M depth-alignment samples and 8.5M spatial-enhancement samples after conversation slicing and auxiliary-data integration.Depth alignment uses 2.5M RefSpatial (RGB-D) samples; enhancement combines RefSpatial RGB/RGB-D with instruction, referring, SAT, and EmbSpatial data.Source: §D.2 · pp. 47–48
  • RFT trains on 100k moderately difficult RefSpatial examples annotated with detailed reasoning processes, typically involving three reasoning steps.The annotations include key intermediate steps and final answers.Source: §D.2 · p. 48

D.3 SFT Training Details

  • SFT maximizes answer likelihood from RGB or RGB-D observations and initializes the subsequent RFT policy; it first aligns depth, then fine-tunes all parameters.Depth alignment updates only the depth projector, while spatial-understanding enhancement fine-tunes the full model using the datasets from D.2.Source: §D.3 · pp. 48–49

D.4 RFT Training Details

  • RFT refines the SFT policy with GRPO, training for two epochs using batch size 1 per GPU and eight sampled outputs.GRPO avoids PPO’s value network by estimating relative advantages from intra-group rewards, improving efficiency and scalability.Source: §D.4 · p. 49

D.4.1 Sampling Action Groups

  • GRPO samples diverse action groups from the current policy, encouraging exploration and preventing premature convergence during reinforcement learning.Each action is sampled conditionally on the RGB or RGB-D observation and textual question.Source: §D.4.1 · p. 49

D.4.2 Reward Design and Policy Update

  • The reward integrates strict output formatting, final-point L1 accuracy against ground truth, and key-step process accuracy: position error below 50 pixels, orientation cosine similarity above 0.8, or size within ±15%, with process rewards scaled by α=0.25.Source: §D.4.2 · pp. 49–50
  • RoboRefer replaces unreliable PRM-based process evaluation with rule-based rewards that use ground-truth intermediate annotations, metric-sensitive criteria, and order-invariant assessment.This addresses LLMs’ inability to process images and VLMs’ imprecision with textual coordinates, while accommodating distinct position, orientation, and size representations.Source: §D.4.2 · p. 49

E.1 Experiments Compute Resources

Experiments run on multi-GPU infrastructure spanning web and 3D data processing, synthetic generation, and model training. The RFT implementation is slower than comparable methods because it uses RGB-D inputs and architecture changes that prevent inference acceleration.

Data Construction

  • Web-data preparation filters 1.7M OpenImages to 933k samples in 8.5 hours, then to approximately 845k samples over 2.5 days using SigLIP2 and Qwen 2.5-VL 7B.Source: §E.1 · pp. 50–51
  • Pseudo-3D construction for 845k samples uses 3 nodes for depth and scene processing plus 4 nodes for object captions, while reasoning-QA generation uses QwQ-32B for 3.75 days on 4 nodes.Source: §E.1 · p. 51
  • Synthetic simulator data generation uses 4 RTX 4090 GPUs for one week, whereas 3D-data filtering and scene-graph construction require 3 hours on 1 node.Source: §E.1 · p. 51

Training Resources

  • Spatial Understanding Enhancement in SFT uses 10 nodes for 2 days on 2B variants and nearly 1 week on 8B variants, with ZeRO-3 for both.Source: §E.1 · p. 51
  • Spatial Referring in RFT takes 3 days on 1 node for 2B variants and is over twice as slow as other Qwen 2/2.5-VL-based methods [122] [130].The slowdown is attributed to RGB-D training inputs and modified NVILA architecture, which are incompatible with vLLM or SGLang acceleration.Source: §E.1 · p. 51

E.2 Spatial Understanding Benchmarks

This section evaluates RoboRefer on public single-step spatial understanding benchmarks using official protocols and accuracy, comparing proprietary, general-purpose open-source, and spatially specialized VLMs.

Benchmark Protocol

  • The evaluation covers CV-Bench [15], BLINK [16], RoboSpatial [2], SAT, and EmbSpatial [22], excluding non-spatial tasks and reporting multiple-choice accuracy.CV-Bench includes 2D Spatial Relation, 3D Depth Order, and 3D Distance; BLINK includes Spatial Relation and Relative Depth.Source: §E.2 · p. 51

Compared Models

  • Models are grouped into proprietary VLMs such as Gemini-2.5-Pro [9], general-VQA-trained open-source VLMs, and spatially specialized models trained on spatially relevant datasets.The categories represent strong spatial perception, general-purpose visual question answering, and basic spatial understanding, respectively.Source: §E.2 · p. 51

E.3 Spatial Referring Benchmarks … F More Demonstrations

RoboRefer is evaluated across established and multi-step spatial-referring benchmarks, simulation, and real-world manipulation settings. Additional ablations and demonstrations examine its data design, operational robustness, and deployment across UR5 and G1 platforms.

E.3 Spatial Referring Benchmarks

  • Evaluation covers RoboRefIt [140], Where2Place [5], RoboSpatial [2], and the more complex RefSpatial-Bench using mask-based success rates averaged across samples.The first three benchmarks are limited to two reasoning steps, whereas RefSpatial-Bench targets multi-step referring in cluttered real-world scenes.Source: §E.3 · p. 51
  • The benchmark comparison distinguishes proprietary Gemini-2.5-Pro from spatially specialized models trained on spatially relevant datasets.The supplied passage defines these as the two main model categories used for comparison.Source: §E.3 · p. 51

E.4 Simulation Evaluation

  • The Open6DOR V2 position track achieves nearly 100% perception-stage success, with remaining failures primarily caused by inverse-kinematics or collision-prone motion-planning errors.The evaluation follows the official Open6DOR V2 protocol introduced in SoFar [7] and excludes 6DOF manipulation execution.Source: §E.4 · p. 52

E.5.1 UR5 Manipulation

  • On UR5, RoboRefer supports disturbance-triggered interruption and replanning at 2.5Hz, while voice commands halt the current task and redirect it through Whisper [158] transcription.Grasping uses SAM2 [78], AnyGrasp [159], depth sensing, and eye-to-hand calibration; placement converts predicted 2D points into transformed 3D robot coordinates.Source: §E.5.1 · p. 52

E.5.2 G1 Humanoid Mobile Manipulation

  • On the G1 humanoid, head-mounted RGB-D localization supports grasping and placement, while chest-camera landmarks are projected into a global SLAM map for navigation.The pipeline uses SAM2 [78], AnyGrasp [159], depth and intrinsics, camera-to-robot calibration, and FAST_LIO_LOCALIZATION_HUMANOID.Source: §E.5.2 · pp. 52–53

E.6 More Ablation Studies

  • Combining RGB and RGB-D RefSpatial data during second-stage SFT enhances image-encoder spatial understanding, whereas RGB-D-only training risks over-reliance on depth cues.This comparison is reported in Table 16 as an ablation of RGB/RGB-D combinations for spatial-understanding enhancement.Source: §E.6 · p. 53, Table 16 · p. 53

F More Demonstrations

  • Additional demonstrations visualize RefSpatial’s 31 spatial-relation types, simulation rollouts, RefSpatial-Bench location and placement predictions, simulator behavior, and real-world evaluations.The demonstrations are provided in Figures 36–51, including benchmark, simulator, and real-world examples.Source: F More Demonstrations · p. 54

G More Discussion on Limitations and Future Work · H Broader Impacts · I Licenses

RoboRefer remains limited by ambiguous instructions requiring human priors, intent understanding, and visual-linguistic reasoning, while its LLM-based robotic deployment introduces hallucination, bias, energy, and safety risks. The authors propose intent-aware data and co-training as future directions, support open research release, and document licenses for the underlying datasets and assets.

G More Discussion on Limitations and Future Work

  • RoboRefer struggles with ambiguous instructions because it relies on precise object and anchor descriptions, whereas resolving implied locations requires visual-linguistic reasoning and human prior knowledge.The limitation includes ambiguous utterances that may still specify a unique location but require process-of-elimination reasoning.Source: G More Discussion on Limitations and… · p. 54
  • Human intent can depend on probabilistic preference and spatial compatibility, such as selecting the plate most likely aligned with multiple drinks or the only pair offering enough placement space.These representative cases are discussed in Figure 35.Source: G More Discussion on Limitations and… · p. 54, Figure 35 · p. 55
  • RefSpatial lacks procedurally generated multi-step referring data that embeds human priors and intent understanding, motivating intent-aware synthesis or co-training with datasets such as PixMo-Points.The authors identify scalable incorporation of these characteristics as challenging.Source: G More Discussion on Limitations and… · p. 54

H Broader Impacts

  • RoboRefer inherits LLM-related hallucinations, biases, and increased energy consumption, while its use as a high-level planner for manipulation and navigation creates safety challenges with control policies.These risks accompany its benefits as a visual assistant and spatially capable robotic planner.Source: H Broader Impacts · p. 54
  • Open access to RoboRefer is presented as beneficial for continued spatial-referring research across robotic arms, humanoids, manipulation, and navigation tasks.The proposed release is intended to support broader robotics development and refinement.Source: H Broader Impacts · p. 54

I Licenses

  • The underlying resources use Apache License 2.0 for OpenImages and Objaverse-XL [138], CC-BY-NC-ND for CA-1M [136], and BSD 3-Clause for Infinigen [137].These licenses cover the cited 2D web images, 3D embodied videos, procedural scene generation, and 3D digital assets.Source: I Licenses · pp. 54–55

Related papers

Loading 2506.04308v4…