PaperlayerView exact revision on arXiv

Source-linked AI summary

Physico-Geospatial Grounded Scene Interpretation for Mobile Robotics

Nicolas Schuler, Janik Kurtz, Lea Dewald, Marcel Sauber, Félicia Teferle, Jürgen Graf

arXiv:2609.06629v1cs.CVcs.RO

TL;DR

Robotic scene interpretation can benefit from incorporating physical and geospatial knowledge, but the paper focuses on augmenting descriptions from pre-trained, unmodified VLMs. Its pipeline fuses sensory measurements and mapped information using LLMs; on a campus pilot set, end-to-end F1-Score was 0.83 for buildings and 0.64 for paths.

  • Problem

    Robotic scene interpretation includes tasks such as object detection, tracking, description, and reasoning, with interest in integrating physical and geospatial knowledge.

    Source: §I · p. 1

  • Method

    A modular pipeline augments pre-trained, unmodified VLM scene descriptions with physical measurements, detected objects, person re-identification, and geospatial information.

    Source: §I · p. 1, §V · p. 7

  • Results

    End-to-end F1-Score was 0.83 for identified buildings and 0.64 for paths.

    Source: B. Results · p. 5

  • Takeaways & Limitations

    For simple scenes with few nearby buildings, one defined path, and a single person, the augmentation is reliable and can add correct building names and path surfaces.

    Source: C. Discussion · p. 7

    Stereo depth estimation adds substantial noise when detected buildings are up to 200 meters away, and occlusion adds further noise.

    Source: C. Discussion · p. 6

Abstract

from arXiv · show

Recent advancements in deep learning allow robotic agents to interact with dynamic and unstructured environments. Of special interest is the integration of physico-geospatial world knowledge into such systems, either by using physics-aware machine learning models, knowledge graphs to model relationships or spatio-temporal and logical reasoning. In the present work, we introduce an approach to augment the output of pre-trained, unmodified VLMs used for scene interpretation by integrating semantic descriptions, OpenStreetMap building data and street information with positional, temporal and metric information obtained from our sensory systems, fusing this information using LLMs. We apply this concept to an outdoor recording within a university campus, achieving an F1-Score of 0.83 in the task of grounding buildings and 0.64 for path surface grounding on our pilot evaluation set. The results demonstrate the conceptual capability of the proposed solution to deliver physico-geospatial grounded natural language descriptions. Code and results are available at https://datahub.rz.rptu.de/hstr-csrl-public/publications/physico-geospatial-grounded-scene-interpretation

I. INTRODUCTION

Robotic scene interpretation spans detection, tracking, description, and reasoning, with interest in grounding agents’ reasoning in physical and geospatial data. This paper introduces a pipeline that enriches pre-trained, unmodified VLM descriptions with physico-geospatial information.

I. INTRODUCTION

  • Physical and geospatial data let agents reason beyond their local frame of reference about geo-located, physically grounded objects.Source: §I · p. 1
  • The pipeline augments pre-trained, unmodified VLM scene descriptions with speed and semantic enrichment such as path surfaces and building names.Source: §I · p. 1

II. FOUNDATIONS AND RELATED WORK

  • The paper’s stated structure moves from theoretical foundations and related work to the pipeline, an application, and future work.Source: §I · pp. 1–2, §II · p. 2

A. Vision Language Models

The background covers transformer-based language and vision-language models, geospatial reasoning, and data sources for location-aware analysis. It contrasts model deployment settings and describes approaches connecting local and geospatial information.

A. Vision Language Models

  • Transformer-based LLMs and VLMs are considered state of the art for reasoning, scene description, state prediction, planning, and behavior-tree construction.Source: A. Vision Language Models · p. 2
  • Foundational models may be too large for edge devices, while edge-capable models can avoid computational, connectivity, bandwidth, and privacy challenges.Source: A. Vision Language Models · p. 2
  • Cosmos3 combines VLM, reasoning, world, action, and generative multimodal models; Cosmos-Reason2 supports lower-budget deployment, while Gemini Robotics ER 1.6 is cloud-only.Source: A. Vision Language Models · p. 2
  • General-purpose image embeddings such as DINO3 can also be applied to satellite imagery, connecting local and geospatial data for geospatial reasoning.Source: A. Vision Language Models · p. 2

B. Geospatial Reasoning

  • Geospatial reasoning is problem solving over diverse cross-modal datasets tied to a location, and interest in it has increased with foundation models.Source: B. Geospatial Reasoning · p. 2
  • Geospatial systems include natural-language interfaces to geographical knowledge and retrieval combining vector, raster, and geospatial data; OSM and Google Maps are possible data sources.Source: B. Geospatial Reasoning · p. 2

C. Knowledge Representation

The related-work discussion covers representations that combine geometric, semantic, and geospatial information. The paper distinguishes local physical integration from geospatial integration and positions its approach as combining both for VLM scene descriptions.

C. Knowledge Representation

  • Multimodal geospatial representations must accommodate geospatial data, 3D point clouds, and their semantic hierarchy; foundation-model features can be embedded in 3D representations.Source: C. Knowledge Representation · p. 2
  • 3D scene graphs combine local geometric and semantic information in hierarchical representations, with recent work extending beyond indoor settings and DAAAM adding spatio-temporal memory for LLM queries.Source: C. Knowledge Representation · p. 2

D. Related Work

  • The paper distinguishes local integration using agent imagery and reference points from geospatial integration that can include satellite images.Source: D. Related Work · p. 2
  • Local approaches include depth-based spatial reasoning, shared world embeddings for action planning, and grounded reasoning that encodes semantic, spatial, and temporal information.Source: D. Related Work · p. 2
  • EarthAI uses geospatial data without a direct connection to an agent’s local sensory world model, whereas OPEN links local localization and planning to georeferenced objects.Source: D. Related Work · p. 2
  • The proposed approach combines local physical properties and 3D-reconstructed coordinates with geospatial positions of buildings and streets to augment VLM descriptions.Source: D. Related Work · pp. 2–3

III. PHYSICO-GEOSPATIAL GROUNDING PIPELINE

The pipeline section introduces the agents, sensory systems, models, data modalities, and workflow used to generate physico-geospatial grounded descriptions. The laboratory uses several mobile platforms with custom modular sensor configurations.

III. PHYSICO-GEOSPATIAL GROUNDING PIPELINE

  • The pipeline section introduces the agents and an overview of its models, data modalities, and workflow.Source: §III · p. 3

A. Our Mobile Platforms and Sensory Systems

B. Physico-Geospatial Grounding for Mobile Robotics

The pipeline records multimodal sensory data, derives semantic and physical information, projects detections into world coordinates, and matches tracked objects to OpenStreetMap objects. An LLM then uses retrieved information to augment scene descriptions; this paper focuses on grounded interpretation rather than downstream tasks.

Pipeline overview

  • Sensory calibration and recording capture RGB stereo images, LiDAR scans, and agent position from GNSS/INS; this paper performs processing post-recording.The pipeline can also run live in a more limited capacity with smaller models.Source: B. Physico-Geospatial Grounding for… · p. 3
  • Semantic processing generates descriptions, detects and tracks objects, re-identifies people, and estimates depth from stereo images or LiDAR.Cosmos-Reason2 generates descriptions; Grounded DINO and SAM provide segmentation, while OSNet-based tracking handles people and ByteTrack handles other objects.Source: B. Physico-Geospatial Grounding for… · p. 3
  • Detected masks are projected into the world using calibrated camera data and the agent’s estimated position to calculate object coordinates in ECEF.The projected point uses either the segmentation-mask centroid or bounding-box center, depending on inference-time constraints.Source: B. Physico-Geospatial Grounding for… · p. 3
  • Physical properties and nearby OSM matches are calculated for relevant objects, with linked observations passed through a temporal Bayesian filter.The filter uses projected-coordinate distances and a Gaussian distance model; its final belief state supplies the best OSM match for each track.Source: B. Physico-Geospatial Grounding for… · pp. 3–4

Grounded description generation

  • An LLM augments the original scene description with retrieved physico-geospatial information, including object properties linked during earlier stages.The paper serves models through an Open WebUI instance on a local network.Source: B. Physico-Geospatial Grounding for… · p. 4

Scope

  • Although the data may support downstream tasks such as natural-language report generation, this paper evaluates grounded scene interpretation only.The paper explicitly limits its focus to the interpretation produced by the pipeline.Source: B. Physico-Geospatial Grounding for… · p. 4

IV. EVALUATION

The evaluation examines the proposed pipeline’s feasibility on a recording intended to illustrate its capabilities and limitations.

Evaluation scope

  • The evaluation is designed to assess feasibility using a recording chosen to illustrate the pipeline’s capabilities and limitations.Source: §IV · p. 4

Evaluation structure

  • The experimental setup, implementation details, and metrics are covered in Sect. IV-A.Source: §IV · p. 4
  • Results and discussion are presented in Sects. IV-B and IV-C.Source: §IV · p. 4

A. Experimental Setup

The study evaluates a seven-minute campus recording and supplements baseline scene descriptions with object, path, building, and person information. Its metrics separately assess OSM grounding, detection, and errors in the augmented descriptions.

Recording and implementation

  • The evaluation uses 8,464 frames at 20 Hz from a wheelchair-mounted stereo platform on the Trier University of Applied Sciences campus.The stereo base is 46 cm; recordings used a Jetson AGX Orin, with RTX 5090 and Ryzen 5 5600X for post-processing.Source: A. Experimental Setup · p. 4

Baseline descriptions

  • Baseline descriptions cover 142 three-second intervals, and baseline-generation errors are excluded from the evaluation.Descriptions are generated with Cosmos-Reason2-8B; the evaluation focuses on errors introduced by the augmentation approach.Source: A. Experimental Setup · p. 4

Speed estimation

  • Person-speed estimates use Kalman filtering and RTS smoothing, but their validity is not evaluated because speed ground truth is unavailable.The filter assumes a 10-meter distance standard deviation and a 1 m/s2 acceleration deviation.Source: A. Experimental Setup · p. 4

Description augmentation

  • For each interval, the LLM receives identified buildings and paths, car counts, and tracked-person names and speeds to augment the baseline description.Llama4:16x17b and Llama4:128x17b generate the augmented descriptions with temperature 0.0 and 4-bit quantization.Source: A. Experimental Setup · p. 4

Grounding metrics

  • Building and path identification are assessed with F1-Score and Subset Accuracy across OSM-versus-ground-truth, OSM-versus-detection, and ground-truth-versus-detection comparisons.The comparisons distinguish end-to-end results, OSM identification relative to detections, and detector quality; car counts use MAE, and person re-identification is compared with ground truth.Source: A. Experimental Setup · p. 4

Description-error assessment

  • Description errors include four categories rated by two independent raters and four categories extracted automatically from generated text.The study reports error prevalence and Wilson 95% confidence intervals; Cohen’s κ and agreement ratios summarize inter-rater reliability, with disagreements resolved by consensus.Source: A. Experimental Setup · pp. 4–5

B. Results

The pilot evaluation reports building and path grounding scores, errors in LLM-augmented descriptions, and inter-rater reliability for those error annotations.

Grounding results

  • 0.84 person re-identification F1-Score is reported against detections, while car-count MAE is 3.04.The ground-truth descriptions contain an average of 8.17 and median of 5 unique cars per three-second interval.Source: B. Results · p. 5
  • 0.83 building F1-Score and 0.64 path F1-Score are achieved end to end against ground truth.Table I reports accuracy and F1-Score for building and path identification and person re-identification, with 95% confidence intervals.Source: Table I · p. 5, B. Results · p. 5

Augmented-description errors

  • 30% of Llama4:128x17b descriptions and 52% of Llama4:16x17b descriptions contain at least one categorized error.Wrong Entities is the most frequent category for both models, at 0.13 and 0.15, respectively; Table II presents error prevalence with 95% confidence intervals.Source: Table II · p. 5, B. Results · p. 5

Supplementary materials

  • Raw outputs, individual ratings, ground truths, and additional sequence metrics are available in the supplementary repository.Source: B. Results · p. 5

Annotation reliability

  • Annotation agreement ranges from chance-level in two model-category pairs to κ = 0.88 for Wrong Entities with Llama4:16x17b.Table III reports Cohen’s κ and overall agreement; Input Description Left Out for Llama4:128x17b has fair agreement.Source: B. Results · p. 5, Table III · p. 6

C. Discussion

Across the campus examples, the pipeline adds empirically grounded names and path surfaces to baseline descriptions, especially in simple scenes. Building and path matching and LLM augmentation remain vulnerable to projection noise, object ambiguity, and incomplete inclusion.

C. Discussion

  • 0.56 Subset Accuracy for ‘OSM vs GT’ (‘Buildings’) increased to 0.66 when detection-model errors were excluded, while paths scored 0.40.The passage attributes building errors to occlusion, distance, and depth estimation; path errors involve merged paths and OSM's representation of paths as ways.Source: C. Discussion · p. 6
  • Building matching is affected by occlusion and noisy stereo depth at distances up to 200 meters, while continuous paths and OSM centerlines complicate path matching.The nearest OSM path element can differ from the physically closest path because OSM represents paths as centerlines.Source: C. Discussion · p. 6
  • 0.30 versus 0.52 error rates were reported for Llama4:128x17b and Llama4:16x17b, respectively, with the larger model performing better.For Llama4:128x17b, ‘Wrong Entities’ was the largest error category at 0.13, while ‘Not Included All Buildings’ was 0.09.Source: C. Discussion · p. 6
  • Rater agreement was at least moderate for most categories, but three categories were low or chance-level; hallucination κ was 0.0.For hallucination ratings, one rater marked no cases and the other marked four, highlighting difficulty evaluating semantic text without well-defined ground truth.Source: C. Discussion · pp. 6–7
  • The end-to-end pipeline adds empirically grounded information to baseline descriptions, and augmentation is reliable in simple scenes with few nearby buildings, one clear path, and one person.Even when relevant buildings are only partially detected, correct supplied information can enhance the baseline; the example adds a named building, person identity, speed, path surface, and object count.Source: C. Discussion · p. 7

V. CONCLUSION AND FUTURE WORK

The paper presents a pipeline that augments pre-trained, unmodified VLM scene descriptions with physical measurements and geospatial information. It identifies direct prompt augmentation as ineffective in preliminary tests and points to entity-to-description matching and model fine-tuning as future-work directions.

V. CONCLUSION AND FUTURE WORK

  • The pipeline augments pre-trained, unmodified VLM descriptions with physical measurements, geospatial information, detected objects, and reidentified persons.The demonstration uses descriptions generated by Cosmos-Reason2-8B [18].Source: §V · p. 7
  • Preliminary tests found that adding augmentation data directly to the prompt did not improve pre-trained, unmodified VLM outputs.Source: §V · p. 7
  • Future work proposes fine-tuning a model to use the provided data directly without an additional LLM.Source: §V · p. 7
  • Connecting detected entities to the baseline description remains a key challenge, with relative distances, semantic descriptions, or localized VLM outputs proposed as possible supports.Source: §V · p. 7

DATA AVAILABILITY AND PROTECTION

The data was recorded at Trier University of Applied Sciences, with raw data access restricted and published materials anonymized. Code, example media, metrics, and results are made publicly available.

DATA AVAILABILITY AND PROTECTION

Loading 2609.06629v1…