PaperlayerView exact revision on arXiv

Source-linked AI summary

Assisted Spatial Cognition Through Vision-Language Models

H. Riaz, J. B. Fernandez, I. Mills, D. Hickey, F. Cleary, M. I. Ali

arXiv:2609.12747v1cs.AI

TL;DR

Traditional assistive navigation tools often lack the depth and 3D spatial information needed for wayfinding. This paper processes smartphone video through 3D reconstruction, scene structuring, and local language-model reasoning. Across described environments, the system provides object localization, spatial guidance, and safety-relevant descriptions, while alignment is reported to increase object detections.

  • Problem

    Traditional 2D navigation aids lack depth, precise object positioning, and 3D scene structure needed for wayfinding.

    Source: §1 · p. 2

  • Method

    The framework processes smartphone video with SLAM3R, aligns point clouds, structures scenes with SpatialLM, and uses GPT-OSS-20B to generate natural-language spatial descriptions and reasoning [8, 9, 12].

    Source: §1 · pp. 2–3, §4 · p. 6

  • Results

    Across the described environments, GPT-OSS-20B provides contextual spatial descriptions and localization, and aligned inputs yield more object detections than misaligned inputs.

    Source: §5.3 · p. 10, §5.4 · p. 10

  • Takeaways & Limitations

    The integrated pipeline demonstrates the feasibility of transforming smartphone video into 3D spatial descriptions for assistive navigation.

    Source: §6 · p. 11

Abstract

from arXiv · show

Multimodal AI, powered by Large Language Models (LLMs) and Vision-Language Models (VLMs), is transforming assistive technologies by enabling simultaneous processing of visual and textual data. This advancement holds significant promise for over 43 million visually impaired and neuro-divergent individuals worldwide who face persistent challenges in navigating indoor and outdoor environments due to limited spatial awareness and insufficient environmental cues. Existing navigation aids often lack comprehensive 3D scene understanding, relying on constrained route-based strategies that hinder user autonomy. In this paper, we introduce a novel end-to-end framework that integrates LLMs, VLMs and digital twin technologies to deliver a spatially cognitive navigation support for visually impaired and neuro-divergent users. Our system captures video input via standard mobile phone cameras, and employs SLAM3R to generate dense 3D point clouds from monocular RGB sequences in real-time. Our custom post-processing algorithm ensures accurate point cloud alignment across multiple viewpoints without requiring predefined reference points. This enhances the capabilities of SpatialLM to produce structured 3D representations, including architectural elements and oriented object bounding boxes. The enriched spatial data is then processed by a locally deployed LLM, which interprets 3D contexts to generate detailed scene descriptions and precise distance measurements between users and surrounding objects. We evaluated our approach across diverse video scenarios featuring various perspectives, looped walking views and captured in multiple environments. The evaluation results demonstrate consistent accuracy in 3D scene interpretation and object localisation, underscoring the potential of our system as a transformative assistive navigation solution that combines advanced visual perception with spatial reasoning

1 Introduction

The introduction identifies a gap between navigation needs for people with visual and cognitive impairments and the limited depth information provided by traditional 2D aids. It proposes an end-to-end framework that turns smartphone video into structured 3D scenes and natural-language spatial descriptions.

1 Introduction

  • People with visual and cognitive impairments face barriers navigating complex environments, including reduced mobility, limited autonomy, and reliance on external assistance.The introduction cites 43 million people living with blindness and 295 million with moderate-to-severe visual impairment, and estimates that 15–20% of the global population is neuro-divergent.Source: §1 · p. 2
  • Traditional 2D navigation aids cannot convey the depth, precise object positions, and 3D structure needed for informed wayfinding.The introduction gives distinguishing an obstacle 2 meters away from one 10 meters away as an example of missing depth information.Source: §1 · p. 2
  • Recent LLM and VLM advances offer opportunities for multimodal scene understanding that processes visual and textual information together [15].The introduction presents this as a recent development in AI for spatial intelligence.Source: §1 · p. 2
  • The proposed framework combines 3D reconstruction, spatial understanding, and natural-language processing for assistive navigation.It uses SLAM3R on smartphone footage, aligns point clouds, processes them with SpatialLM, then uses GPT-OSS-20B for descriptions and spatial relationships [8, 9, 12].Source: §1 · pp. 2–3

2 Background

The background reviews assistive technologies, 3D reconstruction, and vision-language approaches. It notes that existing assistive tools remain limited in conveying precise spatial relationships, while newer methods address reconstruction and 3D scene reasoning through different techniques.

2.1 Assistive Technologies for Visually Impaired and Neurodivergent Users

  • Assistive tools such as Seeing AI identify objects and describe scenes, but their 2D perspective does not convey precise 3D relationships needed for safe navigation in complex settings.The background also describes AI applications in healthcare, education, and daily living [4].Source: §2.1 · p. 3

2.2 SLAM-based 3D Scene Reconstruction

  • SLAM-based reconstruction approaches include monocular dense mapping, Gaussian splatting, and fusion of CNN outputs with multiview stereo; FlashSLAM reports 90% less tracking time than SplaTAM while maintaining superior accuracy in sparse-view settings.The passage also notes MASt3R-SLAM’s performance on in-the-wild video sequences without fixed camera-model assumptions [12].Source: §2.2 · p. 4

2.3 Vision-Language Models for 3D Scene Understanding

  • 3D scene-understanding approaches include object-centric agents, monocular-video instruction tuning [3], object identifiers for grounding, and Bird’s Eye View prompting [15].The passage states that VLM-3R uses over 200K curated 3D reconstructive instruction-tuning question-answer pairs.Source: §2.3 · p. 4

3 Data Collection and Preparation

Data preparation uses smartphone video and SLAM3R to create dense point clouds, then aligns and scales them for SpatialLM. The preparation pipeline emphasizes stable, complete footage and correctly oriented, metric-scale inputs.

3 Data Collection and Preparation

  • Videos are captured on an iPhone 16 or Android phone with a wide-angle lens, using smooth camera motion, constant pacing, and complete space coverage to reduce reconstruction gaps.Shaky or blurry footage can add point-cloud noise and artifacts that reduce downstream layout- and object-detection performance.Source: §3 · pp. 4–5
  • SLAM3R generates dense 3D point clouds from monocular video, while preprocessing removes optional statistical outliers, fixes orientation and floor tilt, and scales scenes to metric units [9].SpatialLM expects z-up point clouds with walls aligned to the x-y planes; the scale factor can be based on an estimated room height such as 2.5 m.Source: §3 · p. 5
  • Figure 1 summarizes the process of producing a clean point cloud from mobile-camera input through pretrained SLAM and post-processing for alignment and scaling.The described pipeline includes point-cloud alignment and scaling.Source: Figure 1 · p. 5

4 Methodology

The methodology converts smartphone video into structured 3D scene representations and natural-language spatial descriptions. Its stages comprise video preprocessing, SLAM3R reconstruction and alignment, SpatialLM scene structuring, and local GPT-OSS-20B processing.

4 Methodology

  • The pipeline has four stages: video capture and preprocessing, SLAM3R-based reconstruction, SpatialLM scene understanding, and GPT-OSS-20B natural-language description and spatial reasoning.The system captures short smartphone videos, typically 30 seconds to 2 minutes, in indoor and outdoor environments.Source: §4 · p. 6, §4.1 · p. 6

4.2 SLAM3R-based Point Cloud Generation: Aligned & Scaled Point Clouds

  • SLAM3R generates point clouds from consecutive video frames using pretrained weights, with inputs assumed to be axis-aligned and metric-scale.The preparation convention is z-up, with walls aligned to the x-y planes, and metric scale supports distance-aware reasoning.Source: §4.2 · p. 6

4.3 SpatialLM Inference for Structured Scene Understanding

  • SpatialLM converts unstructured 3D geometry into structured indoor representations with architectural primitives and oriented object boxes containing categories, positions, sizes, and yaw.The described outputs include walls, doors, windows, floor and ceiling planes, as well as object bounding boxes.Source: §4.3 · p. 6
  • SpatialLM’s output is a text file encoding object classes and 3D geometry in a format specified by its paper [9].The text file contains structured entries; the passage names object class, position, size, and orientation as format fields.Source: §4.3 · p. 6

4.4 GPT-OSS-20b Integration for NLP

  • A locally deployed GPT-OSS-20b model turns structured outputs into scene descriptions, distance-aware narration, directional guidance, and responses to spatial questions.The passage reports 31.8% higher decode throughput for GPT-OSS-20b and describes queries covering object localization, navigation assistance, and contextual awareness.Source: §4.4 · p. 7, 3D Scene Description Generation · p. 7

5 Use Cases & Results

The evaluation spans indoor environments and varied capture views, reporting bird’s-eye layouts and locally generated spatial descriptions; more objects are detected after alignment than before it.

5 Use Cases & Results

  • The system was tested in a laboratory, conference room, and large building using varied views and video lengths.Capture conditions included front, 180° and 360° views, looped walking, and videos of 30, 60, and 120 seconds.Source: §5 · p. 7

5.1 Testing Environments

5.2 Results In Terms of Bird Eye View

  • Bird’s-eye visualisations show a dense laboratory workspace, a more open meeting room, and interconnected rooms and corridors in the large building.The building view also captures varied furniture arrangements.Source: §5.2 · p. 9, Figure 4 · p. 9

5.3 Results from Locally Deployed Open Source GPT-OSS-20B

  • In the meeting room, GPT-OSS-20B identifies 6 chairs among 15 objects and notes the absence of entry and exit points.The paper describes the missing entry/exit information as relevant to orientation in enclosed spaces.Source: §5.3 · p. 10
  • Across settings, responses identify hazards, distinguish seating availability, and describe furniture positions relative to walls and doors.Examples include wet surfaces as slip hazards, “no suitable seating,” and a desk’s position relative to a door and wall.Source: §5.3 · p. 10

5.4 Unaligned Vs Aligned Results

  • SpatialLM detects fewer objects before alignment than after it across all environments; Table 1 reports the before-and-after counts.The supplied text gives no count values.Source: §5.4 · p. 10, Table 1 · p. 11

6 Conclusion & Future Research Directions

The paper presents an end-to-end system combining 3D reconstruction, scene understanding, and language-based spatial reasoning for assistive navigation, evaluated across several indoor environments. Future work targets additional navigation-relevant object categories and an annotation benchmark.

6 Conclusion & Future Research Directions

  • The system combines SLAM3R reconstruction, point-cloud alignment, SpatialLM scene understanding, and GPT-OSS-20B language processing in a four-stage assistive navigation pipeline.The paper describes experiments in laboratories, conference facilities, and architectural buildings.Source: §6 · p. 11
  • Future SpatialLM fine-tuning will add navigation-relevant categories such as stairs, elevators, ramps, institutional signage, crowded places, and accidents.The proposed settings include hospitals, universities, and transportation hubs.Source: §6 · pp. 11–12
  • Future evaluation will include an annotation benchmark to verify, compare, and improve results from the integrated technologies.Source: §6 · p. 12
Loading 2609.12747v1…