PaperlayerView exact revision on arXiv

Source-linked AI summary

Open-Set 3D Scene Graphs for Field Robotics: An Outdoor Case Study

Chad R. Samuelson, Gabriel R. Slade, Joshua G. Mangelson

arXiv:2609.04607v1cs.RO

TL;DR

Outdoor 3DSG behavior with open-set VLMs remains poorly understood in realistic field conditions. This report evaluates Terra across five outdoor datasets, finding feasible navigation and compact, relatively consistent maps alongside multimodal embeddings, weak region reasoning, and routing limitations.

  • Problem

    The behavior and reliability of open-set VLM-based 3DSGs in complex real-world outdoor deployments remain insufficiently evaluated.

  • Method

    The study analyzes Terra across five outdoor datasets, covering VLM point embeddings, graph navigation, region understanding, memory, and repeated-run consistency.

  • Results

    Outdoor 3DSGs support navigation-based object retrieval near 70% success and maintain less than 600MB memory footprints, while region understanding averages F1 around 0.359.

  • Takeaways & Limitations

    Outdoor 3DSGs show potential for large-scale, multi-session semantic mapping, but robust deployment requires handling multiple semantic modes, traversability, and higher-level region understanding.

Abstract

from arXiv · show

Three-dimensional scene graphs (3DSGs) have emerged as a promising approach for building geometrically grounded, semantically informed, hierarchical general-purpose maps to support high-level robotic reasoning. However, the behavior of 3DSGs in real-world outdoor deployments remains poorly understood, particularly when combined with open-set vision-language models (VLMs). In this field report, we analyze the components common to most 3DSG representations across five outdoor robotic datasets to characterize challenges that arise in complex outdoor environments. Using the recently proposed Terra 3DSG as a case study, we investigate semantic point embeddings, place-node graph navigation, region-level understanding, and memory size across the five diverse datasets. We additionally introduce novel consistency metrics to evaluate whether semantic and structural graph properties remain stable across repeated traversals of the same environment. Our analysis reveals that outliers and multiple modes are common in VLM point embeddings across all tested datasets with outlier ratios above $0.1$ for around $30\%$ of points. We demonstrate the feasibility of outdoor 3DSGs for navigation-based object retrieval, achieving success rates near $70\%$, though performance is limited by traversability failures and inefficient routing, with trajectories averaging approximately $66\%$ suboptimal path efficiency. Region-level understanding remains challenging in complex natural environments, with low average F1 scores around $0.359$. Overall, our results show that outdoor 3DSGs can maintain compact (less than $600$MB for multi-kilometer trajectories) and relatively consistent large-scale environment representations, while highlighting open challenges in handling multiple semantic modes, incorporating traversability into graph structures, and improving higher-level region understanding.

I. INTRODUCTION

Outdoor 3DSGs extend hierarchical geometric-semantic mapping toward field robotics, but their behavior with open-set VLMs remains insufficiently characterized in complex environments. This report studies those behaviors and failure modes across diverse outdoor datasets using Terra as a representative case study.

  • 3DSGs combine geometry, semantics, and hierarchy to support high-level robotic reasoning and task-level decision making.
  • Outdoor 3DSG behavior with open-set VLMs remains predominantly unexplored despite stronger prior evaluation in indoor settings.
  • Complex outdoor scale, unstructured geometry, and ambiguous object-terrain-region boundaries make systematic evaluation difficult but essential.
  • The field report evaluates a representative terrain-aware VLM-based 3DSG across five outdoor robotic datasets and analyzes embeddings, navigation, region reasoning, memory, and consistency.
  • The study addresses prior gaps by analyzing multimodal and outlier VLM embeddings and measuring geometric and semantic stability across repeated deployments.

C. EVALUATION PRACTICES FOR OUTDOOR SEMANTIC MAPPING SYSTEMS

The evaluation broadens outdoor semantic-mapping assessment beyond isolated task metrics by examining five diverse environments and multiple layers of Terra’s representation. It combines common task measures with embedding-level and repeated-run consistency analyses.

  • Evaluation scope: Prior outdoor evaluations commonly measure detection, retrieval or navigation, memory, and region identification, but provide limited insight into robustness, scalability, and repeat-session consistency.
  • Evaluation scope: The evaluation extends these metrics across five diverse natural outdoor environments while adding analyses of embedding outliers, multiple semantic modes, and multi-run graph consistency.
  • Terra representation: Terra builds a voxelized LiDAR- and image-semantic MS-Map using terrain labels, FastSAM masks, CLIP embeddings, and cross-frame aggregation.
  • Terra representation: Place nodes and connectivity derive from terrain-specific generalized Voronoi diagram edges, while region nodes hierarchically cluster place nodes using semantic and geometric consistency.
  • Terra representation: Terra supports natural-language object retrieval, region querying, and terrain-aware navigation through CLIP-based cosine similarity over the 3DSG.
  • Field testbed: The five datasets vary in terrain ambiguity, traversability, lighting, vegetation, trajectory scale, object diversity, and region structure.
  • Field testbed: Rock Canyon Campground emphasizes long-trajectory consistency in a trail-dominated environment with ambiguous regions and few distinctive objects.

B. ROBOT PLATFORM

The field evaluation uses a manually driven robotic wheelchair with synchronized cameras, LiDAR, and IMU sensors across diverse outdoor trajectories. The analysis first examines how outdoor conditions affect VLM embeddings before downstream evaluation.

  • Robot platform: The custom robotic wheelchair carries three OAK-D LR cameras, an Ouster OS1 128-beam LiDAR, and an integrated 6-axis IMU.
  • Datasets: The dataset overview pairs satellite imagery showing trajectory extent with representative camera images showing encountered visual conditions.
  • Datasets: The five datasets span diverse environments, terrain complexities, trajectory lengths, and lighting conditions, with seven terrain types and 7615.17 m of combined trajectory.
  • Robot platform: The cameras and LiDAR recorded at 10Hz, while the IMU recorded at 100Hz.
  • Robot platform: Sensor timing and calibration used PTP synchronization, DepthAI clock synchronization below 200µs, camera intrinsics, and manually computed camera-LiDAR extrinsics.
  • Embedding analysis: The analysis therefore examines VLM embedding structure before assessing downstream robotic task performance.
  • Embedding analysis: VLM semantics are assigned to 3D map points through image segmentation and CLIP encoding, but inconsistent segmentation causes substantial embedding variation across observations.

1) Metrics

The evaluation converts each map point’s associated semantic embeddings into cosine-distance distributions, then uses modified z-scores and HDBSCAN clustering to assess outliers and multiple semantic modes. Across five datasets, substantial outlier prevalence and consistently separated clusters show that outdoor point embeddings often violate a unimodal assumption.

  • Outlier metrics: The modified z-score procedure computes cosine distances from each embedding to its point-wise mean before applying a 3.5 outlier threshold.The resulting outlier ratio is the number of outliers divided by the embedding-set size.
  • Outlier metrics: Around 30% of point-cloud points have outlier ratios above 0.1 across the datasets.Although most points have relatively few outliers, this prevalence indicates that outliers or non-unimodal embeddings must be considered.
  • Multiple embedding modes: Viewpoint, distance, segmentation boundaries, occlusions, and lighting changes can associate multiple semantic modes with one 3D map point.The paper identifies leveraging these modes instead of averaging all embeddings into one representation as future work.
  • Multiple embedding modes: HDBSCAN clustering identifies multiple dense embedding clusters associated with individual map points as evidence of multiple semantic modes.Silhouette scores assess cluster separation, with values near 1 indicating well-separated clusters.
  • Multiple embedding modes: 0.493, 0.500, 0.460, 0.479, and 0.496 are the average silhouette scores for River Park, Nunns Park, Marina Part 1, Marina Part 2, and Rock Canyon Campground, respectively.The scores remain consistently near 0.5 across all five datasets.
  • Multiple embedding modes: Most River Park points contain fewer than 10 embedding clusters, but some contain up to 100 clusters.A single point is illustrated with 21 representative clusters, while paired cluster visualizations show consistency within clusters despite variation across clusters.

C. GENERAL TAKEAWAYS

Outdoor open-set 3DSGs must account for semantic embeddings that are not reliably unimodal and for lightweight VLMs’ limited higher-level context. The paper therefore motivates contextualized object querying while identifying multimodal semantics as an unresolved direction.

  • Embedding behavior: Outdoor VLM embeddings for a given point may contain meaningful outlier behavior that reflects multiple semantic modes rather than noise.This variation is associated with viewpoint, observation distance, segmentation boundaries, occlusions, and environmental factors.
  • Embedding behavior: Future work should use multiple embedding modes for semantic reasoning instead of naively averaging all embeddings into one point representation.The recommendation follows the observed multimodal associations in outdoor 3DSGs.
  • Evaluation boundary: The navigation evaluation used static objects and omitted Rock Canyon because it lacked distinct static objects months after graph construction.This constrains the evaluation setting for contextualized object navigation.
  • Contextualized querying: Lightweight Terra-like 3DSGs rely on small VLMs such as CLIP, which lack higher-level contextual understanding.The paper addresses this limitation with contextualized object querying.
  • Contextualized querying: Contextualized querying first identifies object bounding boxes, then ranks their associated place nodes by relevance to a supplied context description.The approach is compared with object-only and full-sentence query strategies.

B. PLACE NODE NAVIGATION ANALYSIS

The place-node layer supports navigation to natural-language object queries in outdoor scenes, but success is constrained by VLM grounding, traversability, endpoint choice, and inefficient graph routes.

  • Object-query grounding: Object+Context prompts produce the best object-detection success rates, motivating their use in real-world navigation experiments.Full Sentence prompts perform lowest, likely because contextual object names challenge small VLMs such as CLIP.
  • Experimental setup: 42 navigation trials evaluate 14 correctly detected object prompts from three starting locations per dataset.The trials use GPS localization and A* paths through the place-node graph, with manually executed waypoint following.
  • Navigation results: 54.8–69.1% success rates show feasible but imperfect navigation to detected objects.The reported metrics distinguish strict and relaxed success, with strict success requiring the object in view and within 10 m.
  • Navigation limitations: Selecting the closest place node does not guarantee visual coverage or a place node within 10 m of the object, separating relaxed and strict outcomes.The endpoint criteria explain the discrepancy between SS-1 and RS-1 without causing navigation failures themselves.
  • Navigation limitations: Untraversable terrain causes navigation failures when planned waypoints direct the robot over ledges or through deeper water.These failures are the sole cause of relaxed top-1 success rates below 1.0 across the datasets.
  • Navigation limitations: Roughly 66% path efficiency indicates that the place-node graph produces navigable but suboptimal routes.The paper suggests path smoothing or improved graph connectivity, while noting that denser connectivity could increase navigation failures.

C. GENERAL TAKEAWAYS

Outdoor 3DSGs support contextualized object navigation, but their broader utility is limited by traversability representation, route efficiency, and weak region-level semantic abstraction in complex scenes.

  • Navigation takeaways: Outdoor 3DSGs enable navigation to contextualized object queries, but performance remains bounded by the underlying VLM’s semantic capabilities.This limitation is especially relevant to systems such as Terra that use small, lightweight VLMs.
  • Navigation takeaways: Traversability must be embedded in the graph structure to guarantee successful navigation, although this limitation may not apply to graphs that encode traversability directly.The evaluated terrain-based place-node structure does not explicitly encode traversability.
  • Navigation takeaways: GVD-based graphs can provide safe path-planning structure while still producing suboptimal routes, motivating better graph algorithms for complex outdoor scenes.The paper identifies route efficiency as a separate concern from basic navigability.
  • Region-level understanding: Region querying evaluates semantic and geometric region retrieval using micro and macro precision, recall, and F1 across clustering methods and k values.The experiments use human-described regions and compare agglomerative with spectral clustering.
  • Region-level understanding: Agglomerative and spectral clustering perform on par, while dataset structure affects region-querying scores.Marina scenes have clearer man-made boundaries, whereas Rock Canyon’s more natural scenery yields lower scores.
  • Region-level understanding: The choice of k heavily affects region-querying results, and the best k differs substantially across datasets.Lower k is preferred when regions are well separated semantically and geometrically, but hierarchical node selection can favor lower-level regions.
  • Region-level understanding: Only 2 of 68 ground-truth region nodes are more similar to the query than all their children, indicating weak higher-level semantic abstraction.Parent-node similarities usually fall between those of their child nodes rather than removing image-specific noise.
  • Region-level understanding: Region-querying results report micro-F1 and macro-F1 across k values for each dataset, with example region-node similarities shown for Marina Part 2.The figures provide complementary views of clustering performance and hierarchical region selection.

B. PROMPT TUNING

Outdoor region querying is highly sensitive to prompt wording because region descriptions and boundaries are ambiguous. The analysis also finds that averaging embeddings can dilute information relevant to distinguishing regions.

  • B. PROMPT TUNING: Prompt sensitivity makes region querying unstable because distinct descriptions of the same outdoor region can produce substantially different results.The paper evaluates multiple descriptions against the same ground-truth place nodes.
  • B. PROMPT TUNING: 0.152 to 0.707 F1 scores were obtained across five prompts for the same region.All prompts used agglomerative clustering with k = 11.
  • B. PROMPT TUNING: The clarification prompt improved precision, recall, and F1, while the valid negation prompt achieved the lowest result, F1 = 0.152.The paper attributes this behavior to CLIP struggling with negation.
  • C. GENERAL TAKEAWAYS: Outdoor-region ambiguity affects both descriptions and boundaries, making region detection and querying a major challenge for outdoor scene graphs.Homogeneous scenery further reduces VLM-based region clustering performance.
  • C. GENERAL TAKEAWAYS: Multi-annotator region labeling is proposed as one way to address ambiguity revealed during region labeling and evaluation.The paper notes that descriptions sometimes required relabeling or clarification when they were insufficiently distinct.
  • C. GENERAL TAKEAWAYS: Averaging embeddings often decreases similarity to region prompts relative to some constituent embeddings, potentially diluting critical distinguishing information.The observation arises from Terra’s aggregation approach, but averaging is also common in VLM applications.
  • C. GENERAL TAKEAWAYS: The largest evaluated 3DSG remains under 600 MB, indicating compact memory use for large-scale outdoor scenes.Figure 12 compares mapped area size with Terra memory size and node count.

VIII. ANALYSIS OF FULL 3DSG UTILITY FOR OUTDOOR FIELD ROBOTICS

The integrated Terra 3DSG remains compact across large outdoor environments, while repeated-traversal tests show strong semantic and structural consistency but weaker terrain and clustering consistency.

  • Memory size: 600 MB stores a Terra 3DSG covering 30000 m2, while its place-and-region graph occupies under 10 MB across datasets.Memory size follows a relatively clear linear trend with covered area.
  • Memory size: Node count varies ambiguously with area because scene structure changes place-node density.Open RV Park terrain and narrow Marina Part 2 breakwaters produce different graph densities at similar areas.
  • Consistency evaluation: Four repeated RV Park traversals vary trajectories and start/end locations to test graph consistency across views and positions.The datasets are aligned to the RV Park V1 frame using point-cloud registration before comparisons.
  • Consistency evaluation: The consistency framework compares graph structure, semantic embeddings, terrain labels, and geometric place-node locations using nearest-neighbor associations.It defines graph nodes, place and region subsets, semantic embeddings, terrain IDs, and Euclidean place-node distances.
  • Consistency results: NCR and NCS remain close to 1.0, while TMC and NCC exceed 0.5 across the repeated-traversal datasets.NCS values are interpreted through cosine similarity, where values nearer 1 indicate more consistent associated embeddings.
  • Consistency results: Worst consistency occurs mainly in RV Park V4 comparisons and between V2 and V3, associated with inconsistent terrain groupings.The reported discrepancies involve purple dirt nodes and blue leaf nodes classified differently across datasets.

C. GENERAL TAKEAWAYS

Outdoor 3DSGs combine compact memory usage with relatively consistent geometric, semantic, and structural representations across repeated traversals.

  • General takeaways: Outdoor 3DSGs maintain a small memory footprint suitable for large-scale field robotics applications.The graph and memory analyses support this conclusion across the evaluated outdoor datasets.
  • General takeaways: Graph consistency across sessions supports potential applications in long-term mapping, robot relocalization, and multi-session outdoor autonomy.The reported consistency spans geometric, semantic, and structural properties.
  • General takeaways: Most memory usage comes from the MS-Map representation, while place-node and region-node graph structures require comparatively little memory.This motivates lightweight representations centered on place and region hierarchies.

IX. LESSONS LEARNED AND FUTURE DIRECTIONS

This section summarizes the primary lessons learned about deploying 3DSGs in complex outdoor environments across the report’s experiments and datasets.

  • The section consolidates lessons about the challenges and opportunities of deploying 3DSGs in complex outdoor environments.
  • The lessons synthesize findings from analyses of individual outdoor 3DSG components across all experiments and datasets.

A. OUTDOOR VLM EMBEDDINGS CONTAIN OUTLIERS AND MULTIPLE SEMANTIC MODES

Outdoor 3DSGs face semantic and navigational challenges but retain compact, relatively consistent representations across repeated traversals.

  • Future directions: Embedding averaging for regions may suppress semantic information, while low-level MS-Map representations dominate memory use.Compact place-node and region-node hierarchies are identified as a possible future direction.
  • Outdoor VLM embeddings: Outlier ratios above 0.1 occur for around 30% of outdoor semantic-map points, alongside multiple distinct semantic modes.The reported causes include viewpoint variation, segmentation inconsistencies, occlusions, and environmental ambiguity.
  • Outdoor 3DSG navigation: Around 70% navigation success and approximately 66% path efficiency demonstrate object-navigation feasibility but expose routing limitations.More robust navigation requires traversability-aware and path-efficient graph structures.
  • Region-level reasoning: A low average F1 score around 0.359 indicates that region-level reasoning remains particularly challenging.Simple embedding averaging often fails to recover the higher-level semantics needed for reliable region understanding.
  • Scalability and consistency: Terra 3DSGs remain under 600 MB for multi-kilometer trajectories and show relatively consistent geometric, semantic, and structural representations across repeated traversals.These properties support potential long-term autonomy, relocalization, and multi-session mapping applications.

APPENDIX

The appendix documents Terra 3DSG construction parameters, dataset-specific prompts and context, and supporting ablations for experimental choices.

  • Terra 3DSG construction parameters are explicitly listed and described in Table 6.
  • A coarse α sweep on River Park identifies 0.25–0.27 as the range yielding the high...The ablation supports the choice of α = 0.26 used throughout the experiments, but the passage is truncated after “high”.
  • Object prompts and dataset-specific context used in the VI.B experiments are listed in Table 8.
  • An ablation examines the object and context weights used by the Object+Context method.
  • Region prompts used in the VII.A experiments are listed for each dataset in Table 10.
Loading 2609.04607v1…