PaperlayerView exact revision on arXiv

Source-linked AI summary

Lightweight and Resource-Efficient Perception for Robotic Guide Dogs

Jinse Kwon, Yoojin Lim, Choonghan Lee, Yongseung Yu, Yongin Kwon, Jemin Lee

arXiv:2610.03187v2cs.DCcs.CV

TL;DR

Robotic guides must detect hazards relevant to people despite strict platform constraints, and access to guide dogs is limited. This paper combines on-device 360-camera and 2D-LiDAR perception with human-centric object descriptions and guidance. Its deployed system reports 83.8% GuideDogQA relative-depth-comparison accuracy, compared with 67.1% for GPT-4o.

  • Problem

    Guide dogs are inaccessible to most potential beneficiaries, while robotic guides must detect human-relevant hazards under strict sensing, size, weight, power, and cost constraints.

    Source: §1 · p. 2

  • Method

    The system fuses 360-camera and 2D-LiDAR perception on-device, adding moving-object descriptions and event-driven VLM–TTS explanation for human-centric guidance.

    Source: §1 · p. 2, §3.3 · p. 6, §5 · p. 14

  • Results

    83.8% GuideDogQA relative-depth-comparison accuracy was achieved, compared with 67.1% for GPT-4o.

    Source: §1 · p. 2, §5 · p. 14

  • Takeaways & Limitations

    The results demonstrate real-time on-device inference without sacrificing perception accuracy in the evaluated quadruped system.

    Source: §5 · p. 14

    The system retains monocular mid-range depth bias, uses a closed COCO vocabulary for static infrastructure, and lacks validation in user studies with blind or low-vision people.

    Source: §5 · p. 14

Abstract

from arXiv · show

Robotic guide dogs should understand their surroundings, objects, and potential risks. Prior research has focused on raw sensor data from cameras and 2D or 3D LiDAR, which precisely measure distance points rather than provide a semantic understanding of the scene. While these physical measurements are effective for robot-centric collision avoidance and robot safety, they are not suitable for human-centric guidance. The system should recognize the type and relevance of obstacles and explain them, clearly and actionably, in terms of their spatial relation to the user. We present complete on-device perception modules that fuse a 360 camera and a 2D LiDAR for reliable collision avoidance, with moving-object detection and tracking for human-centric guidance. Finally, in walking-impossible situations, a vision--language model delivers pathway explanations as a safety mechanism to reduce user anxiety. In experiments, verification of fused 360 camera--LiDAR depth shows reliable near-range perception but inherent mid-range bias, while the system as a whole sustained real-time performance under 55 W. On the real-world egocentric GuideDogQA benchmark, our system achieved 83.8\% accuracy, compared with 67.1\% for GPT-4o. These results demonstrate that practical human-centric guidance with real-time on-device inference is feasible even on quadrupeds.

1 Introduction

Robotic guides must recognize hazards relevant to people, but need omnidirectional sensing and reliable distance measurement within strict platform constraints. This section introduces an on-device system combining 360-camera and 2D-LiDAR perception for human-centric guidance.

1 Introduction

  • Only about 2–3% of people who could benefit are matched with a guide dog, amid training costs and limited working lifespans.The passage attributes the limited access to years of training, tens of thousands of dollars, and short working lives.Source: §1 · p. 2
  • Robotic guides must detect human-relevant hazards, including overhead, ground-level, and dynamic obstacles, while meeting size, weight, power, and cost limits.The passage also calls for omnidirectional sensing, accurate distance measurement, and stable performance.Source: §1 · p. 2
  • We develop omnidirectional perception modules for compact quadruped guide dogs, detailing algorithmic design and on-device performance while providing omnidirectional geometry and human-centric scene understanding for their operation.The handler expresses high-level intent, while the robot performs local path following and collision avoidance.Source: §1 · p. 2
  • 83.8% GuideDogQA relative-depth-comparison accuracy versus 67.1% for GPT-4o is reported for the system.The comparison is stated for the deployed perception modules.Source: §1 · p. 2
  • The complete stack sustains depth estimation at 25.3 FPS and object detection at 11.4 FPS per cube-map face within a 55 W power budget.The system is deployed on a Jetson AGX Orin with multiple Hailo-8 NPUs.Source: §1 · pp. 2–3

2 Related Work

Prior robotic-guide perception has faced deployment constraints and has often lacked semantic descriptions for human-centric guidance. The paper positions its approach among efficient depth methods and work on VLM spatial reasoning.

2 Related Work

  • Prior robotic-guide systems either required tethered laptop-GPU inference or paired accurate 3D geometry with high weight, power, cost, and distance-only output.The cited 3D-LiDAR and stereo-depth system cost nearly ten thousand dollars.Source: §2 · p. 3
  • Monocular estimators provide relative or scale-ambiguous depth, so the paper uses SC-DepthV3 with per-face 2D-LiDAR affine fitting for metric scale.The selected estimator has native Hailo-8 acceleration; four cube-map faces are used instead of many overlapping tangent-image inferences.Source: §2 · p. 3
  • Current VLMs show limited spatial reasoning across several settings, so the paper generates moving-object descriptions geometrically and reserves language models for event-driven pathway explanation.The cited evidence spans embodied question answering, 3D language interfaces, spatially fine-tuned models, and frontier models,,, [7].Source: §2 · p. 3
  • The GuideDogQA benchmark supplies the egocentric BLV-guidance evaluation used to compare deployed perception modules with larger VLMs.The benchmark is introduced by the closest related work identified in the passage.Source: §2 · p. 3

3 Omnidirectional Perception Modules

The system combines 360-camera imagery and 2D LiDAR through online fusion for omnidirectional perception. It then adds moving-object descriptions and event-driven pathway explanations for guidance.

3 Omnidirectional Perception Modules

  • Because no single affordable sensor meets all requirements, the system combines a 360 camera with 2D LiDAR through online fusion.Source: §3 · p. 4

3.1 Equirectangular to Cube map decomposition

  • Each panorama is remapped into four horizontal perspective faces; top and bottom views are discarded, and cached mappings support efficient parallel processing.Source: §3.1 · p. 4

3.2 Synthesizing 3D Depth with AI-based Estimation

  • Mid-row LiDAR ranges anchor a six-coefficient angle-dependent affine fit that rescales monocular depth across each face, with smoothing and gated updates stabilizing it.Source: §3.2 · p. 5
  • Overlapping views provide shared support across adjacent faces, and blended estimates with cosine-ramped seam weights smooth transitions at boundaries.Source: §3.2 · pp. 5–6

3.3 Moving Object Description

  • Four-face detections are tracked into object class, position, and motion cues, then reduced to one safety-prioritized spoken sentence per cycle.Source: §3.3 · pp. 6–7

3.4 VLM-TTS Based Explanation

  • When walking is impossible, a VLM explains forward obstacles and options through streamed TTS while depth monitoring continues and normal guidance resumes once clear.Source: §3.4 · p. 8

4 Experiments

Experiments evaluate the deployed perception stack, its runtime and power, and depth performance across controlled and real-world tests. The results show strong GuideDogQA relative-depth accuracy and near-range depth reliability, alongside a mid-range bias and a resolution choice constrained by real-time processing.

4.1 On-device Platform and Configuration

  • The deployed platform uses a 360 camera split into four 90° cube-map faces, five Hailo-8 NPUs, and a locally run VLM triggered by walking_impossible events.Four NPUs run depth estimation, one runs object detection, and the perception core communicates asynchronously over a ZMQ publish–subscribe bus.Source: §4.1 · p. 9

4.2 Runtime, Power, and Thermal Behavior

  • The system sustains 25.3 FPS per face for depth and 11.4 FPS per face for detection below 55 W, with stable operating temperatures.At maximum load, board power is about 53 W without thermal throttling; decoding adds about 1.2 W, and two PCIe switches contribute about 19 W regardless of workload.Source: §4.2 · p. 10

4.3 Validation of Corrected Depth Estimation

  • Depth error is lowest at 1–3 m (MAE 0.34 m, RMSE 0.47 m), rises at 4–7 m (MAE 3.41 m, RMSE 3.95 m), and falls at 8–10 m (MAE 1.69 m, RMSE 1.79 m).The mid-range bias is attributed to background dominance: a distant wall can pull the affine fit toward its distance, especially when the foreground occupies few columns or is offset from the LiDAR scan plane.Source: Figure 6 · p. 10, §4.3 · p. 11

4.4 Perception Accuracy on the GuideDogQA Benchmark

  • 83.8% accuracy on GuideDogQA relative distance comparison exceeds GPT-4o’s 67.1%, while object recognition reaches 63.0% overall and 95.4% on COCO-80-covered questions.The COCO-80 vocabulary covers 240 of the 435 object-recognition questions; remaining errors primarily involve objects outside that vocabulary.Source: §4.4 · pp. 12–13, Table 2 · p. 12, §4.3 · p. 12

4.5 Depth Resolution and LiDAR-Grounded Evaluation

  • 320×256 is deployed as the practical balance between ordering accuracy and real-time processing; correction improves accuracy at most tested resolutions, with its largest gain of 3.1 percentage points at 192×160.The 480×384 corrected result is slightly higher than the deployed setting but exceeds the real-time processing budget; the LiDAR-grounded test reports ordering accuracy rising from 72.8% to 74.7% and MAE falling from 37.29 m to 31.93 m when using physical LiDAR rather than DA-V2 as anchor.Source: §4.5 · pp. 13–14, Figure 8 · p. 13, Table 3 · p. 14

5 Conclusions

The system combines a 360 camera and 2D LiDAR for on-device perception, with accurate near-range metric depth and strong GuideDogQA performance. Its limitations include mid-range depth bias, a closed vocabulary for static street infrastructure, and no validation through BLV user studies.

5 Conclusions

  • The system combines a 360 camera with 2D LiDAR using angle-conditioned affine correction and seam-aware blending.Controlled experiments verified accurate metric depth estimation in the near range.Source: §5 · p. 14
  • 83.8% accuracy on GuideDogQA relative depth comparison exceeded GPT-4o’s 67.1%, while the COCO coverable-subset detector reached 95.4% accuracy.The deployed INT8 implementation also matched full-precision accuracy.Source: §5 · p. 14
  • The system remains limited by monocular-estimation mid-range depth bias, the closed COCO vocabulary for static street infrastructure, and the absence of BLV user studies.Future work targets open-vocabulary detection within the NPU resource budget and real-world field evaluations with BLV users.Source: §5 · p. 14
Loading 2610.03187v2…