PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Spatial Trace with Reasoning in Vision-Language Models for Robotics

Enshen Zhou, Yibo Li, Jingkun An, Jiayuan Zhang, Shanyu Rong, Mengzhen Liu, Yi Han, Yuheng Ji, Huajie Tan, Jiawei He, Pengwei Wang, Zhongyuan Wang, Cheng Chi, Lu Sheng, Shanghang Zhang

arXiv:2512.13660v4cs.ROcs.CV

TL;DR

Spatial tracing asks robots to turn spatially constrained instructions into 3D positional sequences, requiring multi-step reasoning that combines referring and real-world measurement. RoboTracer combines scale-aware supervised learning with reinforcement fine-tuning and metric-sensitive process rewards, supported by TraceSpatial and TraceSpatial-Bench. It reports strong cross-task benchmark results and execution of long-horizon tasks across robots in cluttered real-world scenes.

  • Problem

    Existing methods struggle with spatial tracing because 2D traces lack 3D grounding and absolute metric understanding, while the task requires multi-step spatial referring and real-world measurement.

    Source: §1 · p. 2

  • Method

    RoboTracer combines supervised 3D referring and measuring with a scale decoder and universal spatial encoder, then uses reinforcement fine-tuning with metric-sensitive process rewards for multi-step tracing.

    Source: §1 · p. 3, §3.2 · pp. 5–6

  • Results

    RoboTracer achieves a 79.1% SOTA average success rate, exceeds Gemini-2.5-Pro by 11%, and surpasses it by 36% on TraceSpatial-Bench.

    Source: §1 · p. 3

  • Takeaways & Limitations

    RoboTracer executes long-horizon, dynamic tasks in cluttered real-world scenes across robots including UR5 and G1 humanoid.

    Source: §1 · p. 3

    VLM inference is often slow, making the architecture less suitable for highly dynamic environments.

    Source: G Discussion on Limitations and Future… · p. 63

Abstract

from arXiv · show

Spatial tracing, as a fundamental embodied interaction ability for robots, is inherently challenging as it requires multi-step metric-grounded reasoning compounded with complex spatial referring and real-world metric measurement. However, existing methods struggle with this compositional task. To this end, we propose RoboTracer, a 3D-aware VLM that first achieves both 3D spatial referring and measuring via a universal spatial encoder and a regression-supervised decoder to enhance scale awareness during supervised fine-tuning (SFT). Moreover, RoboTracer advances multi-step metric-grounded reasoning via reinforcement fine-tuning (RFT) with metric-sensitive process rewards, supervising key intermediate perceptual cues to accurately generate spatial traces. To support SFT and RFT training, we introduce TraceSpatial, a large-scale dataset of 30M QA pairs, spanning outdoor/indoor/tabletop scenes and supporting complex reasoning processes (up to 9 steps). We further present TraceSpatial-Bench, a challenging benchmark filling the gap to evaluate spatial tracing. Experimental results show that RoboTracer surpasses baselines in spatial understanding, measuring, and referring, with an average success rate of 79.1%, and also achieves SOTA performance on TraceSpatial-Bench by a large margin, exceeding Gemini-2.5-Pro by 36% accuracy. Notably, RoboTracer can be integrated with various control policies to execute long-horizon, dynamic tasks across diverse robots (UR5, G1 humanoid) in cluttered real-world scenes. Please see the project page at https://zhoues.github.io/RoboTracer.

1 Introduction

Spatial tracing is needed to translate spatially constrained robot instructions into 3D positional sequences, but it demands multi-step reasoning that combines spatial referring with real-world metric measurement. RoboTracer pairs supervised metric perception with reinforcement fine-tuning, supported by TraceSpatial, and achieves strong benchmark and real-world robot results.

1 Introduction

  • Existing VLMs can perform 2D spatial reasoning and generate 2D traces, but lack intermediate-object reasoning, 3D grounding, and absolute metric understanding.Spatial traces are proposed as 3D positional sequences that bridge instructions and robot action trajectories.Source: §1 · p. 2
  • RoboTracer uses SFT for 3D spatial referring and measuring, then RFT with metric-sensitive process rewards to supervise key perceptual objects for spatial traces.A regression-supervised scale decoder enhances metric perception, while a universal spatial encoder accepts geometric inputs such as camera intrinsics and absolute depth.Source: §1 · p. 3
  • TraceSpatial provides 4.5M examples and 30M QA pairs across indoor, outdoor, and tabletop scenes, with step-wise reasoning annotations up to 9 steps.It includes 48.2% absolute-scale data, reported as 14x prior [75], and spatial traces for three single-arm and dual-arm robot configurations.Source: §1 · p. 3
  • RoboTracer achieves a 79.1% SOTA average success rate, exceeds Gemini-2.5-Pro by 11%, and surpasses it by 36% on TraceSpatial-Bench.It also executes long-horizon, dynamic tasks in cluttered real-world scenes across robots including UR5 and G1 humanoid.Source: §1 · p. 3
  • The paper introduces TraceSpatial-Bench to evaluate spatial tracing with 100 real-world images and tasks requiring metric-grounded multi-step reasoning up to 8 steps.Annotations include start-point masks, end-point 3D bounding boxes, geometry, and diverse evaluation metrics.Source: §1 · p. 3
  • The authors report that RoboTracer generalizes across spatial understanding, measuring, referring, tracing, and real-world robot control.The contribution summary characterizes its robot control as efficient across diverse real-world tasks.Source: §1 · p. 3

2 Related work

Prior VLM work addresses spatial reasoning and trace generation, but the cited approaches are limited in metric grounding, 3D tracing, or intermediate process supervision. The paper positions its dataset, benchmark, and metric-sensitive process rewards as responses to these gaps.

Spatial Reasoning with VLMs

  • Metric-grounded spatial reasoning requires absolute-scale measurements, while prior methods still struggle with complex absolute-scale scenes despite using explicit or implicit 3D modalities.The passage distinguishes metric-agnostic reasoning about object properties and relations from metric-grounded reasoning about physical-world distance, depth, and size.Source: §2 · p. 4

Trace Generation with VLMs for Robotics

  • Prior VLM trace methods predict 2D point sequences, then either lift them to 3D using depth and camera intrinsics or overlay them on images to guide control.The cited approaches [33] [80] [92] [99] use lift-to-3D; [41] [43] [110] use overlap-on-2D.Source: §2 · p. 4

Reinforcement Fine-tuning for VLMs

  • Most prior RFT approaches remain limited to 2D-relative perception and outcome-based rewards, whereas spatial tracing needs 3D multi-step metric-grounded reasoning.The paper responds with metric-sensitive process rewards that supervise key perception steps.Source: §2 · p. 4

3 Method

The method represents spatial traces as metric-grounded keypoint sequences and pairs a geometry-adaptive VLM with staged supervised and reinforcement fine-tuning. TraceSpatial supplies diverse, fine-grained data for spatial referring, measuring, and tracing.

3.1 Problem Formulation and Representation Advantages

  • Spatial traces are ordered sequences of 6–12 sparse 3D keypoints predicted from visual inputs, optional camera geometry, and a textual instruction.Each (u, v, d) point combines image-plane coordinates with absolute depth; traces can serve as spatial plans alongside motion planning.Source: §3.1 · pp. 4–5
  • The decoupled (u, v, d) representation converts to 3D via camera intrinsics and can yield 2D traces or referring data by dropping depth or retaining endpoints.It avoids requiring VLMs to implicitly learn camera geometry and supports co-training with existing 2D datasets.Source: §3.1 · p. 5
  • Spatial traces capture intermediate execution more fully than start- or end-keypoint plans and can support zero-shot generalization without task-specific training.The paper presents traces as an intermediate representation that handles spatially and metrically constrained tasks compared with direct VLAs.Source: §3.1 · p. 5

3.2 RoboTracer

  • RoboTracer adds a regression-supervised scale decoder for absolute metric awareness and a universal spatial encoder that integrates available geometric cues.Its two-stage SFT aligns the encoder and decoder using geometric annotations, then trains other components on RGB-only and RGB-plus-geometry inputs.Source: §3.2 · pp. 5–6
  • RFT uses metric-sensitive process rewards to supervise annotated perceptual steps, alongside outcome rewards for structured outputs, endpoint consistency, and trajectory alignment.Process rewards use type-specific error metrics, allow flexible step ordering, and are scaled by 0.25 in the final reward.Source: §3.2 · p. 7

3.3 TraceSpatial Dataset

  • TraceSpatial contains 4.5M samples and 30M QA pairs across indoor, outdoor, and tabletop data, with 48.2% metric-grounded QA and detailed reasoning annotations.Its pipeline combines web images, 3D scans, and manipulation videos; generated reasoning annotations assume the code reflects optimal reasoning.Source: Figure 3 · p. 7, §3.3 · pp. 8–9

4 Experiments

Across evaluations, RoboTracer outperforms baselines on TraceSpatial-Bench 3D metrics and generalizes to unseen robot tasks, with real-world task execution and data-generation applications. Ablations report benefits from multi-step process RFT, geometric cues, and the training-data recipe.

4.1 Spatial Understanding and Measuring

  • 85.7%: RoboTracer-8B-SFT achieves average success rate, exceeding Gemini-2.5-Pro by 8.58% and NVILA-8B by 20.3%.Its gains are larger on 3D-related and measurement tasks than on 2D tasks (23.6% vs.14.7%).Source: §4.1 · p. 10

4.3 Multi-Step Spatial Tracing

  • 36%: RoboTracer-RFT exceeds Gemini-2.5-Pro on TraceSpatial-Bench 3D metrics.RFT without the spatial encoder still exceeds SFT and remains robust to noisy geometric input; more precise geometry yields up to 6% absolute gain.Source: Table 5 · p. 11, §4.3 · p. 12

4.2 2D Spatial Referring and Visual Trace

  • RoboTracer achieves the best performance on both 2D spatial referring and visual trace benchmarks.The decoupled (u, v, d) formulation supports dimensionality reduction and co-training with existing 2D datasets.Source: §4.2 · p. 11

4.4 Public vision-language Benchmarks

  • Joint training with TraceSpatial RGB and RGB+X data and general visual-instruction datasets leaves the model on par with or slightly above its base model.The data pipeline also scales to RGB-only datasets without ground-truth metadata, with comparable performance from the newly RFT-trained model.Source: §4.4 · p. 13, §4.6 · p. 14

4.5 Simulator and Real-world Evaluation

  • 32.8%: RoboTracer outperforms the best baseline on unseen RoboTwin tasks without task-specific training; generated traces also raise MolmoAct’s 3D reasoning and spatial success rate from 0% to 25%.RoboTracer with motion planning and Code-as-Monitor handles long-horizon tasks in cluttered, dynamic scenes, including across UR5 and G1 robots.Source: §4.5 · pp. 13–14

4.6 Ablation Study

  • Combining 2D, 3D, and video data yields optimal SFT performance, while the universal spatial encoder enhances metric-grounded reasoning and multi-step tracing.The three data types provide scale supervision, gripper-motion learning, and greater spatial-instruction diversity.Source: §4.6 · p. 14
  • Regression-based scale supervision performs best, and metric-sensitive process rewards raise overall success rate by 4% over outcome-only rewards.Outcome-only rewards yield smaller gains on 3D metrics and overall success than on 2D metrics relative to SFT.Source: §4.6 · p. 15

5 Conclusion

RoboTracer addresses spatial tracing through multi-step, metric-grounded reasoning over 3D spatial referring and measuring. The paper reports strong performance and potential for robotic applications, while noting that motion-planning pose recovery is less effective for rotation-rich manipulation.

5 Conclusion

  • RoboTracer combines flexible geometric inputs, scale supervision, and metric-sensitive rewards for multi-step, metric-grounded spatial tracing.TraceSpatial supports SFT and RFT, and TraceSpatial-Bench is tailored to evaluate spatial tracing.Source: §5 · p. 15
  • The paper reports that RoboTracer performs strongly in experiments and has potential across robotic applications.Source: §5 · p. 15
  • Motion planning recovers the 6D end-effector pose, but is less effective for rotation-rich manipulation with critical orientation constraints.Dense spatial traces are suggested as a direction for future research.Source: §5 · p. 15

A More Discussions

The discussion adds robustness checks, evaluation results, and robot-task evidence, then outlines how TraceSpatial data is assembled from several sources. It also describes filtering and the broader use of the dataset for spatial reasoning.

A More Discussions

  • Relabeling TraceSpatial-Bench and expanding it to 800 samples both preserve or increase RoboTracer’s margin over Gemini-2.5-Pro; three added real-robot tasks use 20 human-annotated trials each.MolmoAct is fine-tuned using 50 successful rollouts; the reported RFT gain is larger for the 8B than the 2B model (12% vs. 8%).Source: A More Discussions · p. 16
  • 39% for lifting to (u, v, d) 3D labels exceeds 34% for direct output lifting with oracle-lift and 30% for lifting to (x, y, z) labels under the same training data.The passage attributes this comparison to the superiority of a 3D-native formulation.Source: A More Discussions · p. 17
  • The (u, v, d) formulation also supports navigation by providing 2D object references or waypoints that can be paired with SLAM.The passage gives RoboPoint [97] as an example of this downstream use.Source: A More Discussions · p. 17
  • Compared with RoboRefer [112], RoboTracer predicts metric-grounded 3D traces and uses task-level temporal sequences rather than single-image, single-point features.The passage says those sequences capture intermediate dynamics and temporal consistency.Source: A More Discussions · p. 17
  • RoboTracer also surpasses specialized models on SpatialRGPT-Bench and SAT VQA results.The supplied table caption identifies the evaluation as VQA performance on SpatialRGPT-Bench and SAT.Source: A More Discussions · p. 17, Table 11 · p. 17

B TraceSpatial Details

  • TraceSpatial combines 2D web images, 3D scanning data, simulation-generated object-centric traces, and manipulation videos across three robot embodiments and single- and dual-arm configurations.The pipeline overview describes distinct data-generation procedures for each source.Source: B TraceSpatial Details · pp. 17–18

B.1 2D Web Image

  • OpenImages is filtered in stages before pseudo-3D scene-graph construction, object-description enrichment, and QA generation from the resulting spatial information.The 2D pipeline uses image filtering, pseudo-3D graphs, hierarchical descriptions, and diverse QA-pair creation.Source: Dataset Construction · p. 18, §B.1 · p. 18

Dataset Construction

TraceSpatial’s construction refines pseudo-3D scene graphs from web images and combines them with spatial descriptions and QA generation. It also uses annotated 3D scans to support spatial referring, measuring, and multi-step tracing.

Dataset Construction

  • The dataset pipeline refines pseudo-3D scene-graph construction beyond prior work [112] while retaining a multi-step process of filtering, graph construction, object descriptions, and QA generation.The stated refinement is a more accurate method for constructing pseudo-3D scene graphs.Source: Dataset Construction · p. 19

Multi-Stage Image Filtering.

  • 934k of OpenImages’ 1.7M images remain after SigLIP2 coarse filtering, and Qwen2.5-VL fine-grained filtering retains 846k.The second stage addresses categories that SigLIP2 handles less reliably, including artwork, low-light scenes, grayscale images, distortions, and collages.Source: Multi-Stage Image Filtering · pp. 19–20

Pseudo-3D Scene Graphs Construction

  • RAM++ supplies semantic labels, GroundingDINO localizes objects, and MoGe-2 reconstructs metric-scale point clouds and camera intrinsics from a single image.SAM 2.1 generates instance masks, which join the labels, boxes, and object-level point clouds in each pseudo-3D scene graph.Source: Pseudo-3D Scene Graphs Construction · p. 22

Hierarchical Object Description Generation

  • Spatially oriented descriptions disambiguate repeated object categories by sorting instances along the principal axis with the greatest variation.The pipeline applies this enhancement only when multiple same-category instances occur and filters low-variance cases, yielding 466k images.Source: Hierarchical Object Description… · pp. 24–25

Generating Diverse QA Pairs via 3D Scene Graphs

B.2 3D Scanning Data

  • ScanNet and CA-1M support 3D spatial QA and object-centric, multi-step metric-grounded tracing, but ScanNet requires added 2D boxes because its source annotations lack them.The construction uses comprehensive 3D annotations alongside completed 2D information.Source: §B.2 · p. 27, ScanNet Data Processing · p. 27

ScanNet Data Processing

This section describes processing 3D scans and manipulation videos into spatial QA and trace-aligned training data, including metric-grounded measurement and referring, task formulation, trace planning and refinement, and instruction generation and formatting.

ScanNet Data Processing

  • ScanNet processing projects sampled 3D-box surface points into images, filters by depth consistency and image bounds, and takes the remaining points' extrema as 2D boxes; 3D scene graphs add geometric cues for spatial QA.Each box contributes 5,000 uniformly sampled surface points; the 3D graphs support QA across 28 spatial-relation types.Source: ScanNet Data Processing · pp. 28–29

Human-Like Measuring Descriptions Generation

  • Measurement descriptions use object- and dimension-specific phrasing and dynamically selected units, which experiments show improves performance over fixed-unit or bare-number formats.Values below 1 meter are more likely to be expressed in centimeters or inches, while larger values are typically in meters but may be expressed in feet.Source: Human-Like Measuring Descriptions… · pp. 29–30

3D Spatial Measuring Data Generation

3D Spatial Referring Data Generation

B.3 Object-centric Spatial Tracing Generation

  • Because static scans lack manipulation trajectories, the authors synthesize object-centric traces from scene graphs with occupancy information through initialization, planning, and reasoning.The pipeline first standardizes the environment and defines object roles to support plausible task formulation.Source: §B.3 · p. 31, Scene Initialization & Task Formulation · p. 31

Scene Initialization & Task Formulation

  • Scenes are gravity-aligned and objects assigned task roles; captioned objects support unambiguous moving and via targets, stable supports serve as references, and all raw instances remain obstacles.The pipeline also defines five manipulation primitives, including active bypass-and-place and bypass-and-stack tasks.Source: Scene Initialization & Task Formulation · pp. 31–32

Simulation-based Spatial Trace Generation

  • The trace pipeline uses sampling-based planning, centroid-outward endpoint selection, RRT* paths, and escape handling for constrained starts, with hierarchical collision checks for scanned geometry.Collision checking first filters with AABBs, then uses OBBs and the Separating Axis Theorem for precise intersection tests.Source: Simulation-based Spatial Trace… · pp. 33–35

Advanced Spatial Reasoning Logic

  • The pipeline combines heuristic-guided via-object bypass and retroactive relation labeling with smoothed, compressed, depth-grounded traces and visibility-based quality filtering.Paths are compressed to at most 8 keypoints; traces with more than 30% occlusion are rejected.Source: Advanced Spatial Reasoning Logic · pp. 35–37
  • Spatial-trace instructions cover 2D, 3D, and lifting variants, with adaptive trace filtering and a semantic blocklist for immovable fixtures.The filtering also requires traces to remain within camera-frame boundaries and uses a volume-adaptive minimum-length threshold.Source: Advanced Spatial Reasoning Logic · p. 37, VQA Data Generation · p. 37

VQA Data Generation

  • Approximately 20% of instruction templates include metric placeholders populated with the ground-truth trace displacement.This makes instructions combine relative directions with explicit distances in meters.Source: VQA Data Generation · p. 37
  • Each trace generates three VQA tasks: 2D visual tracing, 3D spatial tracing, and 2D-to-3D trace lifting.The 3D task predicts depth from monocular input, while lifting provides the ground-truth 2D trace to train trace-conditioned depth estimation.Source: VQA Data Generation · pp. 39–40
  • Trace outputs use normalized 2D pixel coordinates in [0, 1000] and absolute depth in meters.This hybrid representation leverages pretrained 2D visual grounding while training precise 3D metric structures.Source: VQA Data Generation · p. 40

B.4 Manipulation Video

  • 20.5k Droid manipulation trajectories yielded 46.8k end-effector-centric and 58.4k object-centric QAs.The pipeline uses raw stereo videos to extract camera intrinsics and depth with the ZED SDK, filters for valid SVO files, and checks extrinsics for depth alignment; the data supports 2D and 3D trace generation.Source: Droid Data Processing · p. 42

Droid Data Processing

Droid preprocessing validates camera extrinsics through depth alignment, then extracts and simplifies end-effector- and object-centric traces from gripper-closure intervals.

Droid Data Processing

  • Extrinsics are validated by projecting the end-effector into the image and comparing projected-pixel depth with its camera-frame z-coordinate; only in-bounds projections are checked.A frame is consistent when the depth difference is under 5 cm, and extrinsics are accepted when more than one-third of tested frames are consistent; the paired images contrast incorrect and correct extrinsics.Source: Droid Data Processing · p. 42, Figure 14 · p. 43
  • The gripper position is defined 0.15 m from the end-effector base, and traces are extracted around the single gripper-closure interval.The closure interval and a base frame up to 60 frames earlier define object-centric and end-effector traces, which are RDP-downsampled; tasks retaining too many points are removed.Source: Droid Data Processing · pp. 42, 44

AgiBot Data Processing

AgiBot preprocessing addresses extrinsic errors, inaccurate task timing, and occlusions before extracting traces. It validates depth alignment, redesigns task segmentation, filters occluded samples, and generates trace-description QA pairs.

AgiBot Data Processing

  • The pipeline filters demonstrations with invalid extrinsics, inaccurate motion segmentation, or occlusions that can make traces meaningless.Original Place intervals may continue after the object reaches its target, impeding reliable object-centric traces; the preprocessing removes problematic demonstrations and redesigns segmentation.Source: Figure 17 · p. 46, AgiBot Data Processing · p. 46
  • AgiBot videos are accepted when the proportion of depth-aligned frames exceeds a threshold, set to 0.83 in practice.The visualizations show retained-sample proportions across thresholds and spatially contiguous clusters of misaligned frames, consistent with temporally coherent extrinsic failures.Source: AgiBot Data Processing · p. 46, Figure 18 · p. 47, Figure 19 · p. 47
  • Redesigned segmentation merges Pick and subsequent Place labels into Pick-and-Place subtasks and extracts traces according to closure intervals and which arm moves.Traces use the first subtask frame as the base frame and the gripper point 0.15 m along the local z-axis; the redesign supports end-effector- and object-centric traces across varied manipulation behaviors.Source: AgiBot Data Processing · pp. 46, 48
  • Occlusion filtering checks gripper visibility at a task-dependent frame and discards samples when more than 60% of nearby depth values are zero.Depth is inspected within a 30-pixel-radius neighborhood after projecting the gripper into the base frame.Source: AgiBot Data Processing · p. 48
  • RDP simplification removes trace redundancy while preserving geometric structure, and valid traces are converted into three categories of QA pairs.Traces with out-of-bounds projections or too many points after compression are discarded; the resulting dataset contains 977.5k samples.Source: AgiBot Data Processing · p. 48
  • The modified RoboTwin simulation environment generates spatial traces across 16 manipulation tasks.The statistics cover episodes, total end-effector- and object-centric traces, and collection time.Source: Table 13 · p. 49

B.5 Simulation Data

The simulation pipeline collects spatial traces for both robot grippers and manipulated objects, then generates referring expressions and QA pairs. TraceSpatial-Bench evaluates object-centric 3D traces using manually annotated scenes, with reference paths that are feasible but not unique.

B.5 Simulation Data

  • The RoboTwin pipeline records gripper and manipulated-object motion as separate end-effector-centric and object-centric traces.It re-engineers 16 tasks, maps grasp points into camera coordinates, and aligns traces to each subtask’s start frame.Source: §B.5 · p. 49
  • Cluttered scenes and scene-graph rules generate unambiguous target descriptions across four referring-expression categories.The categories cover spatial relations, proximity, ordinal position, and attribute comparisons.Source: §B.5 · p. 50
  • 443.1k unique simulation traces yield 1.329M QA pairs, split between 914.1k end-effector-centric and 415.1k object-centric samples.The traces are downsampled using the RDP algorithm before template-based QA generation.Source: §B.5 · p. 50

C TraceSpatial-Bench Details

  • TraceSpatial-Bench is a manually annotated, object-centric suite designed to evaluate 3D spatial trace prediction, using ScanNet and CA-1M scenes with precise geometry.Its geometry annotations include absolute depth maps, camera geometry, and 3D occupancy.Source: C TraceSpatial-Bench Details · p. 50

C.1 Benchmark Composition and Data

  • Each benchmark scene includes an image, camera parameters, an object mask, a destination 3D box, and a feasible reference trace that is not unique ground truth.Multiple collision-free paths to the destination may be valid.Source: §C.1 · pp. 50–51
  • Depth and camera intrinsics support reconstruction of occupancy maps and target point clouds, while projection enables analogous 2D path and box evaluation.The 3D trace and destination box can be projected into the image plane.Source: §C.1 · p. 51

C.2 Benchmark Statistics

TraceSpatial-Bench contains 100 manually annotated scenes. Its prompts average 21.73 words, and prompt length increases with the defined step complexity.

C.2 Benchmark Statistics

  • The benchmark contains 100 manually annotated scenes, with source and task-category distributions summarized in Table 14.The supplied passage does not report the table’s distribution values.Source: §C.2 · p. 51, Table 14 · p. 51
  • Benchmark prompts average 21.73 words.Source: §C.2 · p. 51
  • A “step” counts spatial relations and associated objects in a prompt, and higher step counts correlate with longer prompts.The passage describes a clear correlation between step count and average prompt word count.Source: §C.2 · p. 51, Table 15 · p. 52

D.1 Architecture

RoboTracer extends an NVILA base with metric-grounded spatial representations and a regression-supervised scale decoder, trained through staged SFT and GRPO-based RFT. Its rule-based process rewards assess intermediate spatial cues with metric-specific criteria and give process feedback less weight than final outcomes.

D.1 Architecture

  • RoboTracer builds on NVILA and adds a frozen MapAnything spatial encoder that accepts geometric inputs such as depth and camera geometry.The base visual encoder uses dynamic resolution, and the Qwen2 backbone supplies the language model.Source: §D.1 · p. 52
  • A trainable MLP scale decoder maps the <SCALE> token embedding to a scale factor and uses regression supervision to improve metric awareness.It is initialized from MapAnything’s scale head.Source: §D.1 · p. 52

D.2 Training Data

  • SFT uses 4.5M RGB+X samples for metric alignment, then 9M samples for metric enhancement; the full SFT stage totals 14M samples.RFT uses 120k TraceSpatial samples, typically with 4–6 reasoning steps.Source: §D.2 · p. 53, Table 16 · p. 53

D.3 SFT Training Details

  • SFT first trains the spatial and scale projectors and decoder, then fine-tunes all model parameters except the universal spatial encoder.The two steps are metric alignment and metric enhancement.Source: §D.3 · p. 54

D.4 RFT Training Details

  • RFT refines the SFT model with GRPO, sampling groups of responses and comparing their rewards without a separate value network.The setup uses two epochs and eight outputs per sampled group.Source: §D.4 · p. 54, Sampling Action Groups · p. 54

Reward Design and Policy Update

  • Outcome rewards assess output format, trace endpoints, and trace consistency, while the process-reward design evaluates intermediate perceptual cues without a PRM.The authors make process evaluation metric-sensitive and order-invariant because relevant spatial reasoning steps need not be strictly sequential.Source: Reward Design and Policy Update · pp. 55–56
  • Process accuracy rewards check annotated key steps, applying distinct criteria to referring coordinates and depth, measurements, and scale.For measurement and scale, predictions within ±30% of ground truth receive a reward of 1; referring uses separate coordinate and depth thresholds.Source: Reward Design and Policy Update · p. 57
  • Process rewards are weighted by 0.25, prioritizing final outcomes; group-normalized advantages guide policy updates constrained by KL divergence.The combined reward includes outcome-format, point, process-format, and process-accuracy components.Source: Reward Design and Policy Update · p. 57

E.1 Experiments Compute Resources

The reported compute requirements vary across data-generation and training stages. Spatial tracing RFT is over twice as slow as comparable Qwen 2.5-VL-based methods and cannot use vLLM or SGLang acceleration.

E.1 Experiments Compute Resources

  • Trace-generation and refinement take approximately 32 hours for CA-1M and 6 hours for ScanNet, using a 96-core CPU server with 180 concurrent processes.The pipeline includes obstacle-aware RRT* planning and visualization rendering.Source: §E.1 · p. 58
  • Droid depth processing takes 46 hours on an H100 GPU; parallel processing takes 5 hours for Droid and 27 hours for AgiBot.RoboTwin evaluation uses eight H100 GPUs for 36 hours.Source: §E.1 · p. 58
  • Metric alignment takes 20 hours for 2B variants and 50 hours for 8B variants; metric enhancement takes 2 days and nearly 1 week, respectively.Spatial-tracing RFT takes 3 days for 2B variants, runs over twice as slowly as cited Qwen 2.5-VL-based methods, and is incompatible with vLLM or SGLang acceleration.Source: §E.1 · p. 58

E.2 Spatial Understanding Benchmarks

On TraceSpatial-Bench, RoboTracer substantially exceeds strong VLM baselines on overall 3D success; in RoboTwin simulation, its predicted traces support high-success execution across the evaluated tasks.

E.2 Spatial Understanding Benchmarks; E.3 Spatial Measuring Benchmarks

  • Spatial-understanding evaluation covers spatial relations, depth order, distance, configuration, and measuring; multiple-choice benchmarks use accuracy, while measurement success follows the QSpatial protocol.The cited benchmarks include CV-Bench, BLINK, RoboSpatial, EmbSpatial, QSpatial, and MSMU.Source: §E.2 · pp. 58–59, §E.3 · p. 59

E.4 2D Spatial Referring Benchmarks; E.5 2D Visual Trace Benchmarks

  • Spatial-referring benchmarks score predicted points by their inclusion in ground-truth masks, while visual-trace benchmarks compare predicted and reference traces using distance and error metrics.The visual-trace metrics include Discrete Fréchet Distance, Hausdorff Distance, and Root Mean Square Error; lower is better.Source: §E.4 · p. 59, §E.5 · pp. 59–60

2D Evaluation

  • 2D tracing evaluation checks whether the start point falls in the target mask and whether one of the final three projected points falls inside the destination box.The 2D evaluation projects 3D ground-truth attributes into the image plane.Source: 2D Evaluation · p. 60

3D Evaluation

  • Overall 3D Success requires correct start and end positions, then a simulated path with no more than 20% of object points intersecting environmental occupancy.The stated start and end thresholds are within 20 cm of the target point cloud and destination box, respectively.Source: 3D Evaluation · p. 60
  • On this challenging benchmark, the authors’ model exceeds a 30% success rate, while Gemini-2.5-Pro and Qwen-3VL each achieve less than 10% Overall 3D Success.Source: Baseline Performance · p. 60

E.7 Simulation Evaluation

  • RoboTwin simulation evaluates 19 tasks, including 12 with variants in TraceSpatial and 7 entirely novel tasks; the authors report high execution success under RoboTracer’s traces.The results table reports success rates for RoboTwin hard tasks, with gray rows marking variants absent from TraceSpatial.Source: §E.7 · p. 61, Table 17 · p. 62

E.8 Real-world Evaluation; UR5 Manipulation; G1 Humanoid Manipulation

  • For UR5 manipulation, RoboTracer runs at 1.5 Hz and predicted endpoint shifts can trigger interruption and replanning; G1 watering uses predicted 3D waypoints for movement.The UR5 pipeline uses observed depth for contact points, while the G1 pipeline directly follows predicted mid-air waypoints.Source: UR5 Manipulation · pp. 61–62, G1 Humanoid Manipulation · pp. 62–63

F More Demonstrations

  • The supplementary demonstrations show TraceSpatial-Bench examples, spatial-tracing data, and RoboTracer rollouts in the simulator.The cited figure ranges are 20–22, 23–26, and 27–32, respectively.Source: F More Demonstrations · p. 63

G Discussion on Limitations and Future Work

The authors identify slow VLM inference and limited 3D metric-grounded capability as remaining constraints. They point to better representations and richer scale-aware supervision as future directions.

G Discussion on Limitations and Future Work

  • Slow VLM inference makes the approach less suitable for highly dynamic environments.The authors suggest quantization or parallel inference may help mitigate this limitation.Source: G Discussion on Limitations and Future… · p. 63
  • Improvements in 3D metric-grounded understanding, especially from scaling data, remain limited relative to gains in 2D spatial reasoning.The paper notes that maintaining language understanding while enhancing metric-grounded 3D capability remains challenging.Source: G Discussion on Limitations and Future… · p. 63
  • The current method supervises a simple metric scale factor but omits dense geometric outputs such as depth maps because of computational overhead.The authors propose better input representations and richer supervision at both input and output levels.Source: G Discussion on Limitations and Future… · p. 64
Loading 2512.13660v4…