PaperlayerView exact revision on arXiv

Source-linked AI summary

Urbex: Agentic Spatial Grounding for City-Scale 3D Scenes

Yu Wang, Rui Dai, Wei Chen, Yi Wang, Bo Dang, Kaikui Liu, Xiangxiang Chu, Yansheng Li

arXiv:2610.18356v1cs.CVcs.AI

TL;DR

City-scale grounding must handle large scenes and abstract spatial descriptions, while candidate-ranking methods rely on precomputed proposals. Urbex turns grounding into reinforcement-learned active multi-view search, achieving strong CityRefer results and faster online inference, with zero-shot robustness on STPLS3D-Refer.

  • Problem

    City-scale grounding must resolve small objects from abstract, long-range spatial cues, while candidate-ranking approaches depend on precomputed proposals and can be inefficient at scale.

    Source: §1 · pp. 1–2

  • Method

    Urbex uses a VLM agent to search an interactive city scene through landmark queries, local zooms, and oblique views, and trains its tool-use policy with reinforcement learning.

    Source: §1 · p. 2, §3.3 · p. 6

  • Results

    Urbex achieves strong grounding accuracy, candidate-free localization, and improved online efficiency on CityRefer; CityAnchor takes about 32.1 seconds per example versus about 3.2 seconds for Urbex, and zero-shot results on STPLS3D-Refer suggest robustness.

    Source: §1 · p. 2, §4.2 · p. 8, §5 · p. 11

  • Takeaways & Limitations

    Active spatial search provides a robust alternative to static candidate retrieval for large-scale 3D scene understanding.

    Source: §5 · p. 11

    Urbex depends on reliable BEV rendering and view-to-world calibration; missing or ambiguous landmarks can reduce search efficiency.

    Source: E Limitations · p. 22

Abstract

from arXiv · show

City-scale 3D grounding is commonly formulated as candidate ranking, which requires precomputed instance proposals and becomes inefficient in large outdoor scenes. We propose Urbex, an agentic vision-language framework that instead casts grounding as active spatial search. Urbex represents a city-scale 3D scene as an interactive multi-view environment, allowing a VLM agent to query landmarks, zoom into local regions, render oblique views, and commit a grounded bounding box. We optimize this tool-use policy with reinforcement learning on CityRefer, using smooth localization rewards and lightweight shaping rewards for efficient exploration. Experiments show that Urbex achieves the best Hit@1 and IoU on CityRefer, supports candidate-free grounding without ground-truth boxes, and is substantially faster than the strongest city-scale ranking baseline. Zero-shot results on STPLS3D-Refer further indicate improved cross-scene robustness.

1 Introduction

City-scale grounding must resolve small objects through abstract, long-range cues, while downsampling loses fine geometry and candidate ranking depends on proposals and scales scoring cost with candidate count. Urbex instead uses active multi-view search with city-aware tools, reinforcement learning, and strong benchmark results.

1 Introduction

  • Urban referring expressions can identify small objects through named buildings, road directions, or distant landmarks, requiring fine-grained recognition and long-range spatial reasoning.City-scale scenes often contain hundreds of thousands to tens of millions of points.Source: §1 · p. 1
  • Downsampling city-scale point clouds discards fine geometry, while candidate ranking depends on proposal quality and incurs greater scoring cost as candidate counts grow.Ranking also treats localization as static retrieval rather than sequential exploration.Source: §1 · p. 1
  • Urbex casts grounding as active spatial search in which a VLM parses cues, chooses where to inspect, gathers evidence, and commits when ready.The agent operates through a task-specific interface that moves between global context and local detail without enumerating every instance.Source: §1 · pp. 1–2
  • Urbex combines a global BEV view, landmark queries, local zooms, and oblique verification, then projects the committed box to world coordinates.Reinforcement learning on CityRefer uses overlap and center-distance signals with tool-level shaping rewards for intermediate search behavior.Source: §1 · p. 2
  • Urbex achieves the best Hit@1 and IoU on CityRefer, improves online efficiency over the strongest ranking baseline, and shows zero-shot robustness on STPLS3D-Refer.The introduction reports strong cross-scene robustness alongside the CityRefer results.Source: §1 · p. 2

2 Related work

Prior 3D grounding work largely targets indoor or object-centric settings; city-scale grounding adds large search spaces, repetitive instances, landmarks, and online-efficiency demands. Agentic methods offer sequential interaction, but Urbex specializes that approach for urban scenes.

2.1 3D Scene Grounding

  • Indoor grounding methods encode point clouds or multi-view images and rank instances, while later work adds relation modeling, multi-view context, transformers, proposal-free localization, and open-world methods.These approaches do not directly address city-scale landmark reasoning, very large search spaces, or online efficiency over dense urban instances.Source: §2.1 · p. 3
  • City-scale methods typically rank precomputed boxes, avoiding direct processing of the full point cloud but remaining constrained by proposal quality, preprocessing costs, and candidate-scoring efficiency.Urbex contrasts this one-shot retrieval formulation with sequential search through targeted observations.Source: §2.1 · p. 3

2.2 Agentic Vision-Language Models

  • Agentic VLMs use observation, reasoning, action, and feedback loops, but much prior work focuses on 2D tasks or indoor navigation.Related sequential 3D grounding methods use indoor RGB-D or graph-structured settings; Urbex adapts agentic tools and an RL objective to city-scale exploration.Source: §2.2 · p. 3

3 Method

Urbex recasts city-scale grounding as active spatial search through coordinate-aligned multi-view observations, then trains a VLM to navigate them with reinforcement learning.

3 Method

  • Urbex exposes city-scale scenes through coordinate-aligned views and optimizes a VLM policy to search them actively rather than rank precomputed candidates.The method comprises a multi-view environment, a tool-use reasoning loop, and a reinforcement-learning objective.Source: §3 · p. 3, Figure 2 · p. 4

3.1 Interactive environment for city-scale 3D grounding

  • Without an inference-time candidate set, Urbex lets the agent extend available views and maps its predicted box from a BEV-aligned view into world coordinates.Landmark queries anchor global search, zoom-ins recover detail from the original high-resolution BEV, and oblique views support 3D verification.Source: §3.1 · p. 4

3.2 Agentic reasoning with multi-view tool use

  • The VLM conditions on the current observation and interaction history, calling tools or submitting a final answer as it searches, inspects locally, and confirms candidates.Accumulated tool responses let it refer back to earlier views and maintain coordinate consistency.Source: §3.2 · p. 5

3.3 Reinforcement learning for active spatial search

  • Reinforcement learning trains the tool-use policy with a smooth terminal localization reward and tool-level shaping, using GRPO without supervised trajectory demonstrations.IoU rewards precision, center distance provides a signal even when overlap is zero, and a step cost discourages long trajectories; tool rewards guide zooms and oblique rendering while penalizing invalid calls.Source: §3.3 · pp. 5–6

4 Experiments

Experiments compare Urbex with prompted VLMs, indoor grounding pipelines, and city-scale ranking methods on CityRefer, then test zero-shot transfer and analyze tools, rewards, and search behavior. Urbex leads the reported CityRefer and STPLS3D-Refer accuracy comparisons while enabling candidate-free grounding and faster online inference.

4.1 Experimental setup

  • The evaluation covers CityRefer in-domain performance, STPLS3D-Refer transfer, and component sensitivity, with both datasets converted to the same BEV-based interactive environment and action protocol.CityRefer supplies the main accuracy and efficiency benchmark; STPLS3D-Refer tests transfer to different geometry and visual appearance.Source: §4.1 · pp. 6–7

4.2 Main results on CityRefer

  • Urbex achieves the best Hit@1 and Hit@3 on both CityRefer test splits and the best Hit@5 on NO, while CityAnchor has the highest Hit@5 on ND.The CityRefer comparison also includes candidate-free grounding and inference-efficiency results.Source: §4.2 · p. 7, Table 1 · p. 8
  • About 3.2 seconds per example for Urbex versus 32.1 seconds for CityAnchor yields an order-of-magnitude online inference-speed improvement.The comparison excludes offline scene preparation; on-demand oblique rendering is estimated at 4.4 seconds per query.Source: §4.2 · p. 8

4.3 Zero-shot generalization on STPLS3D-Refer

  • Urbex achieves the best score on every reported accuracy metric in zero-shot STPLS3D-Refer evaluation after training only on CityRefer.Without further training, it reaches 23.3 Hit@1 on 30 manually verified landmark-free expressions.Source: §4.3 · pp. 8–9, Table 2 · p. 9

4.4 Ablation studies

  • With all tools, Urbex reaches 36.9 Hit@3 and 14.6 2D IoU, versus 10.5 and 2.1 without tools; Landmark Query supplies the largest tool gain.Adding Landmark Query raises Hit@3 from 15.5 to 28.2 and 2D IoU from 4.1 to 8.9; eight turns provide the best accuracy-efficiency trade-off in the ablation.Source: Table 3 · p. 9, §4.4 · pp. 9–10
  • Terminal IoU alone reaches 22.8 Hit@3 and 6.7 2D IoU, while adding distance shaping raises them to 30.6 and 10.9, and the full reward performs best.Distance shaping helps when early predictions have no target overlap, while tool-level rewards stabilize tool use.Source: Table 4 · p. 9, §4.4 · p. 10

4.5 Analysis

  • A fixed Landmark Query → Zoom-in → Oblique Rendering → Answer workflow raises Qwen's Hit@1 from 2.7 to 10.0 with RL-trained Urbex, while GPT-5.4 uses fewer tokens with the workflow.On CityRefer-NO, 10 m coordinate noise lowers Hit@1 by 1.2 points, whereas missing or incorrect landmarks cause drops exceeding ten points.Source: §4.5 · p. 10
  • Correct-target identification can coexist with loose boxes: requiring BEV IoU ≥0.25 reduces Hit@1 from 23.53 to 18.71, and the predicted-to-target area ratio has median 1.40.The P90 area ratio is 9.92; direct IoU and point F1 are reported alongside Hit@k to assess spatial extent.Source: §4.5 · p. 10

5 Conclusion

Urbex replaces ranking precomputed candidates with active spatial search in an interactive multi-view environment, optimized with reinforcement learning for accurate, efficient exploration. On CityRefer it supports candidate-free localization and improves online inference efficiency over ranking-based baselines, while zero-shot STPLS3D-Refer results suggest transfer to new scenes.

5 Conclusion

  • CityRefer experiments show strong grounding accuracy, candidate-free localization, and substantially improved online inference efficiency over ranking-based baselines.Zero-shot results on STPLS3D-Refer further suggest active spatial search is a robust alternative to static candidate retrieval for large-scale 3D scene understanding.Source: §5 · p. 11
  • Urbex casts grounding as active spatial search in an interactive multi-view environment instead of ranking precomputed candidates.The tool-use policy is optimized with reinforcement learning under a limited interaction budget to encourage accurate and efficient exploration.Source: §5 · p. 11

A Implementation Details

Urbex is trained with reinforcement learning and evaluated through a shared city-tool interface. The implementation also standardizes tool-using VLM baselines and adapts indoor and city-scale ranking methods for comparison.

A.1 Hyperparameters and Hardware

  • Urbex is initialized from Qwen3-VL-4B-Instruct and trained on CityRefer with GRPO using 300 optimization steps and a reward combining final IoU, center-distance shaping, step cost, and tool-level shaping.Trajectories allow at most eight assistant turns; local inference timing uses one H20 GPU and excludes offline scene preprocessing.Source: §A.1 · p. 13

A.2 Tool Interface

  • The shared city-aware tool interface defines tool inputs and outputs, and validates tool names, argument types, box bounds, and view identifiers before execution.The prompt template specifies the task, tools, coordinates, action format, and final-box format for prompted VLM baselines and Urbex.Source: §A.2 · p. 13, Table 5 · p. 13, Figure 5 · p. 14

A.3 Baseline Settings

  • Prompted GPT-5.4 and Qwen3.6-plus use the same tool environment without task-specific policy optimization, while InstanceRefer and EDA use indoor preprocessing with 40,000-point downsampling.InstanceRefer is omitted from candidate-free evaluation because it requires provided instances; CityRefer and CityAnchor are retrained, with online candidate scoring included in CityAnchor timing.Source: §A.3 · pp. 13–14

B.1 CityRefer Data Preparation

CityRefer is prepared with cleaned annotations and separate description-generalization and held-out-scene splits. STPLS3D-Refer is constructed as a semi-automatic, cross-domain zero-shot benchmark to test transfer to different urban geometry, appearance, and geography.

B.1 CityRefer Data Preparation

  • CityRefer is split into train, validation, ND, and NO sets, with ND using unseen descriptions in training scenes and NO evaluating held-out scenes.After removing duplicate scene-query rows and low-quality annotations, the split contains 2,744 training, 549 validation, 1,098 ND, and 1,396 NO queries.Source: §B.1 · p. 14, Table 6 · p. 14

B.2 STPLS3D-Refer Construction

  • STPLS3D-Refer uses two STPLS3D photogrammetry scenes with reliable geographic context and limits targets to buildings and vehicles.The benchmark is built for zero-shot evaluation of transfer from CityRefer to different urban geometry, appearance, and geography.Source: §B.2 · p. 15
  • Because STPLS3D lacks instance annotations, the benchmark clusters semantic masks and filters more than 1,000 pseudo-instances to produce verified object pools.The verified pools contain 59 buildings and 62 vehicles in USC, and 11 buildings and 17 vehicles in WMSC; the two scenes include 278 real OSM landmarks after geographic registration.Source: §B.2 · p. 15, Table 7 · p. 15
  • For each target, VLM-assisted expression generation uses BEV and oblique views, target attributes, distractors, and nearby OSM landmarks, with automatic filtering and manual inspection.The process reduces more than 1,000 preliminary target-expression samples to 445 high-confidence landmark-grounded samples.Source: §B.2 · p. 15

C Additional Qualitative Examples

The appendix illustrates agent trajectories and reports training dynamics, alongside repeated-decoding results for one trained checkpoint. Tool use converges toward a one-call-per-tool sequence, while the repeated-run interval is described as evidence relative to decoding variation—not as a paired significance test.

C Additional Qualitative Examples

  • The qualitative examples and trajectory figure trace the agent from a query and global BEV observation through landmark lookup, local zoom, oblique rendering, and final box prediction.The examples include descriptions grounded in landmarks, object attributes, and spatial relations.Source: C Additional Qualitative Examples · p. 16, Figure 8 · p. 17

D Controlled Experiments and Diagnostic Analysis

  • Training reward and train/validation 2D IoU improve over 300 steps, with a small stable train-validation gap, while tool use converges to roughly one call per tool per episode.Early training overuses zoom_in (peak ≈4.7 calls; first-30-step mean 2.6) and underuses oblique_render (mean ≈0.6).Source: C Additional Qualitative Examples · p. 16, Figure 7 · p. 16

D.1 Repeated Decoding Runs

  • 25.62 ± t4,0.975(0.33/√5) = [25.21, 26.03] is the approximate 95% Student-t interval for the repeated-run ND mean, above CityAnchor’s reported 23.4.The interval reflects five stochastic decoding runs of one checkpoint and is not a paired significance test; it omits test-sample and training or baseline-evaluation variation.Source: §D.1 · p. 18, Table 8 · p. 18

D.2 Fixed Tool Workflows and Learned Execution

The appendix tests whether Urbex’s learned execution policy adds value beyond a fixed tool sequence, examines landmark dependence, and compares proposal-based grounding with proposal-free localization.

D.2 Fixed Tool Workflows and Learned Execution

  • Under the fixed workflow, RL-trained Urbex reaches 10.0 Hit@1 versus 2.7 for the base model, while free Urbex reaches 23.5.Most free trajectories already follow L-Z-O, so the gap does not isolate tool ordering from argument generation or prompting effects.Source: Table 9 · p. 18, §D.2 · p. 18

D.3 Landmark Availability and Construction Bias

  • Missing or wrong landmark matches reduce landmark-guided Hit@1 by 10.8 and 12.2 points, compared with 1.2 points for 10 m coordinate noise.Expressions without named landmarks reach 31.2 Hit@1; this does not show that removing necessary landmarks is harmless.Source: Table 10 · p. 19, §D.3 · p. 19

D.4 Grounding with Predicted Instance Proposals

  • On CityRefer-NO, Urbex exceeds the proposal pipeline in direct localization: BEV IoU 14.6 vs 4.2, 3D IoU 7.7 vs 2.4, and point F1 17.8 vs 4.7 (percent).Source: §D.4 · p. 19, Table 11 · p. 19

D.5 Candidate Matching and Box-Extent Diagnostics

These diagnostics distinguish candidate identification from box localization: ranking results are stable under tested matching rules, but requiring target overlap and examining box extents reveal localization weaknesses.

Sensitivity to the ranking rule

  • Hit@1 varies by at most 0.29 points and Hit@3 by at most 0.22 across the tested matching rules.The rules include the original IoU ranking, excluding zero-overlap candidates, and CIoU; stability does not establish tight localization.Source: Table 12 · p. 20, §D.5 · p. 20

Target-IoU-thresholded Hit@1

  • Requiring at least 0.25 target IoU reduces thresholded Hit@1 by 4.82 points on NO and 5.24 points on ND.These are thresholded Hit@1 diagnostics, not thresholded Hit@3/5 results.Source: Table 13 · p. 20, §D.5 · p. 20

Candidate overlap and area ratio

  • Positive-overlap candidate counts average 6.5 on NO and 4.6 on ND, while the median count reaching 0.25 IoU is one on both splits.Predicted-to-target area-ratio medians are 1.40 and 1.56, with P90 values of 9.92 and 7.21 on NO and ND, respectively, indicating a tail of oversized predictions.Source: Table 14 · p. 20, Table 15 · p. 20, §D.5 · p. 20

D.6 City-Scale Search and Disambiguation Failures

The failure analysis links harder regional search and local disambiguation with landmark distance and nearby distractors, while latency measurements compare online inference costs under a stated accounting scope.

D.6 City-Scale Search and Disambiguation Failures

  • On ND, local-disambiguation success falls from 64.7% with no nearby same-class distractors to 25.1% with at least ten.Regional-search success decreases with landmark distance across all three datasets; the observed group trends are associations, with no bin-level uncertainty reported.Source: Table 16 · p. 21, §D.6 · p. 21
  • Two saved CityRefer-NO trajectories show distinct zero-IoU failure stages: a first zoom excluding the target and a wrong-row selection despite including it.Both runs retain the proposed box after oblique inspection; these selected cases do not estimate failure frequencies or establish aggregate causes.Source: §D.6 · p. 22

D.7 Inference efficiency

  • Urbex takes 3.2 s/query versus CityAnchor’s 32.1 s/query under warm online accounting, an approximately 10× ratio.On-demand oblique rendering raises Urbex’s estimated total to 4.4 s/query, still about 7.3× faster; scene-cache construction and other preparation steps are outside these online ratios.Source: Table 17 · p. 22, §D.7 · p. 22

E Limitations

Urbex relies on reliable scene rendering and calibration, outputs a BEV-aligned box rather than a refined 3D instance mask, and presents risks associated with city-scale language-driven localization.

System dependencies

  • Calibration or rendering errors can map a plausible 2D box to the wrong world location, while unavailable or ambiguous landmarks increase reliance on visual search and local context and reduce efficiency.Source: E Limitations · p. 22

Output scope

  • The current policy outputs a BEV-aligned bounding box, leaving fine-grained 3D segmentation accuracy as a limitation.The paper proposes adding multi-view 3D instance-segmentation tools to refine localized regions into masks or boxes.Source: E Limitations · p. 22

Broader impacts and mitigation

  • City-scale localization risks surveillance, persistent tracking, sensitive-site identification, and automated monitoring; proposed mitigations include public-benchmark training and a research-only license.The paper also recommends privacy-preserving preprocessing such as blurring faces and license plates.Source: E Limitations · p. 22

Related papers

Loading 2610.18356v1…