PaperlayerView exact revision on arXiv

Source-linked AI summary

Onboard Lightweight LLM Reasoning for Human-Aware Safe UAV Navigation

Siwei Cai, Yuwei Wu, Vijay Kumar, Lifeng Zhou

arXiv:2610.22092v1cs.RO

TL;DR

Conventional UAV planning treats obstacles as geometry, leaving category-specific safety constraints unavailable and pedestrian constraints vulnerable to temporary occlusion. This paper combines visual grounding, retrieval-stabilized LLM reasoning, and persistent obstacle mapping onboard. Its Gazebo evaluations report fewer urban collision events and higher landing success among moving actors.

  • Problem

    Conventional obstacle maps encode geometry but not how object categories should change clearance or motion response, and temporary occlusions can remove pedestrian constraints.

    Source: §I · p. 1

  • Method

    An onboard pipeline grounds RGB-D observations in 3D, uses retrieval-stabilized and grammar-constrained LLM verdicts cached per class, and preserves obstacle identities through occlusions.

    Source: §I · p. 2, A. Perception: Shared VLE,… · p. 2

  • Results

    Urban collision events fall roughly threefold, while landing success among moving actors rises from 34% to 96%.

    Source: §V · p. 4

  • Takeaways & Limitations

    The semantic planner maintains larger human-aware separation through additional detours, at the cost of longer paths and peak jerk.

    Source: A. Urban Navigation Among Pedestrians… · p. 4

    The 224-pixel encoder input limits reliable perception of small, distant objects, especially pedestrians.

    Source: C. Perception Under Limited Input… · p. 4

Abstract

from arXiv · show

Aerial robots operating near people must treat a pedestrian, a parked car, and a wall differently, yet conventional perception--planning stacks reduce all of them to occupied geometry---a \emph{human-aware safety gap} in which category-specific clearance is unavailable to the planner and temporary occlusions remove pedestrian-specific constraints from its map. We present a modular, fully onboard pipeline that closes this gap by turning visual observations into context-dependent safety constraints for real-time UAV navigation. A shared BLIP encoder drives scene captioning and text-conditioned segmentation, which is fused with depth to localize obstacles in 3D. Obstacle-specific risk is inferred by a lightweight LLM (Qwen-2.5-0.5B) stabilized with retrieval-augmented generation and grammar-constrained decoding that restricts its output to a planner-compatible JSON schema. The LLM reasons once per object category rather than per frame: its verdicts are cached per class, so previously seen categories are resolved without inference and new ones trigger a fresh retrieval-grounded query; detections are additionally given persistent identities and held in the map across occlusions. The perception pipeline runs at 10\,Hz on a Jetson Orin NX, while fallback LLM queries take approximately 0.8\,s and remain off the control-critical path. In Gazebo, low-altitude urban navigation among pedestrians and vehicles shows a roughly $3\times$ reduction in collision events over a geometric planner, and semantics-aware landing among moving actors raises success from 34\% to 96\%. The 224-pixel encoder input limits reliable small-object perception at range; conservative pedestrian margins and short-term persistence mitigate this limitation during navigation.

I. INTRODUCTION

Conventional UAV planning treats obstacles as geometry, leaving category-specific risk and clearance unavailable. The paper proposes an onboard pipeline that turns visual observations into context-dependent safety constraints while preserving pedestrian awareness through short occlusions.

I. INTRODUCTION

  • Conventional occupancy maps and detections locate obstacles but do not specify how they should alter clearance, priority, or motion response.This is consequential in semantically heterogeneous environments where pedestrians, vehicles, and other obstacles imply different risks.Source: §I · p. 1
  • The human-aware safety gap is the absence of category-specific clearance and short-term awareness of people during temporary occlusions.The proposed goal is to provide category- and context-dependent clearance, dynamism, and persistence as real-time onboard planner constraints.Source: §I · pp. 1–2
  • The system separates a lightweight vision-language encoder from a text-only LLM, grounds outputs with depth, and stabilizes small-model reasoning with retrieval and grammar-constrained JSON.The LLM adjusts planner constraints rather than commanding the vehicle; persistent identities address constraints disappearing during occlusions.Source: §I · p. 2
  • The paper contributes semantic-to-spatial grounding, cached per-class LLM verdicts, persistent obstacle identities, and onboard closed-loop evaluations.New labels can trigger retrieval-grounded reasoning, while verdicts for previously handled categories are reused.Source: §I · p. 2

II. METHOD

The method has perception, reasoning, and action subsystems. Perception uses a shared visual encoder to produce semantic masks and depth-grounded 3D obstacle detections.

II. METHOD

  • The method divides the system into perception, reasoning, and action, with action maintaining persistent obstacles and injecting them into a model-based planner.Perception converts RGB-D frames into semantic and spatial primitives, and reasoning maps these to obstacle-specific safety parameters.Source: §II · p. 2

A. Perception: Shared VLE, Segmentation, and 3D Grounding

  • A frozen BLIP ViT-B/16 encoder serves both scene captioning and text-conditioned segmentation, avoiding redundant visual computation onboard.The segmentation decoder fuses multi-scale features and uses text prompts such as “person” or “car” to condition the visual map.Source: A. Perception: Shared VLE,… · p. 2
  • Connected mask components above an area threshold are backprojected through time-aligned depth, with depths more than 2σ from the in-mask median rejected.The resulting class, 3D center, and confidence tuples are produced at 10 Hz.Source: A. Perception: Shared VLE,… · p. 2

B. Reasoning: From Semantics to Safety Parameters

The reasoning module assigns semantic safety verdicts by class rather than by frame. Retrieval and constrained decoding support new classes, while caching avoids repeated inference for known classes.

B. Reasoning: From Semantics to Safety Parameters

  • The LLM reasons on the first appearance of a class, then caches and reuses its dynamic-status and bounding-box verdict for later detections.Unknown labels can trigger retrieval-grounded reasoning; in the 20-run urban benchmark, all observed labels hit the warm cache and required no online query after initialization.Source: B. Reasoning: From Semantics to Safety… · p. 2
  • For unknown classes, hybrid retrieval combines dense synonym matching and sparse keyword search, fuses ranks, and selects top-k knowledge-base segments for the prompt.The retrieval setup is intended to limit attention dilution in small models.Source: B. Reasoning: From Semantics to Safety… · pp. 2–3
  • A quantized Qwen-2.5-0.5B model uses grammar-constrained decoding for a JSON verdict and is restricted to semantic risk rather than waypoint generation.The grammar reduces malformed outputs and removes a separate free-form parsing stage; the planner handles geometry.Source: B. Reasoning: From Semantics to Safety… · p. 3

C. Action: Persistent Obstacles and Planner Constraints

The action layer preserves obstacle identities and holds their last known boxes through short detection gaps. It turns semantic boxes into unsafe regions for planner back ends.

C. Action: Persistent Obstacles and Planner Constraints

  • Greedy nearest-neighbor association assigns stable class-specific IDs, and the map retains each object's last known box for 3 s after it is last seen.The hold-in-place policy is deliberate for pedestrians; vehicles may be extrapolated when tracked velocity is available.Source: C. Action: Persistent Obstacles and… · p. 3
  • The persistence policy keeps pedestrian constraints in the map through temporary occlusions instead of allowing fixed voxel decay to remove them within seconds.The method tracks object IDs across detection gaps and releases held voxels only after the hold time.Source: C. Action: Persistent Obstacles and… · p. 3
  • Dynamic objects become axis-aligned unsafe boxes around their 3D centers, which the planner excludes from safe trajectories.The landing planner uses these boxes in kinodynamic A* and nonlinear MPC, while urban navigation injects them as inflated voxels into an EGO-Planner grid map.Source: C. Action: Persistent Obstacles and… · p. 3

III. ONBOARD FEASIBILITY

The onboard design uses a compact modular vision-language pipeline and selects a lightweight LLM that fits alongside perception and VIO. Its per-class cache keeps query latency off the frame-by-frame path.

Why not the cloud

  • A single 384×384 JPEG takes ≈8 s to transmit at the field uplink rate, and hosted-model generation adds 1.6–2.8 s on average, motivating onboard execution.The passage estimates a roughly 10 s round trip could correspond to about 10 m of travel at 1 m/s.Source: §III · p. 3

Platform

  • The Orin NX is the deployment platform; the 8 GB Orin Nano could not accommodate the VLE, KB, LLM, VIO, and ROS stack simultaneously.The compared Orin NX has 16 GB memory and 25 W power draw.Source: §III · p. 3

Modular VLE+LLM versus monolithic small VLMs

  • The modular compressthen-reason design sends ≈30 text tokens through a 0.5B model behind an 86M encoder, reducing language-model input tokens by more than 19× versus compared monolithic small VLMs.The same encoder output is reused by the segmentation head.Source: §III · p. 3

LLM selection

  • Qwen-2.5-0.5B answers in ≈0.75–0.79 s using 5.6 GB of total system memory, while perception maintained 9.96 Hz over 20 runs.With per-class verdict caching, this latency is incurred once per new category rather than per frame.Source: §III · p. 3

IV. EXPERIMENTS

The closed-loop experiments use simulated RGB-D sensing in Gazebo to evaluate urban navigation and autonomous landing. The landing study spans three distinct environments, while the urban study targets pedestrians and vehicles.

IV. EXPERIMENTS

  • All closed-loop experiments use ROS Noetic with Gazebo physics and simulated RGB-D sensing.Urban navigation uses an EGO-Planner back end; autonomous landing is a second case study with a kinodynamic-A*/MPC back end.Source: §IV · pp. 3–4
  • The primary evaluation is low-altitude urban navigation among pedestrians and vehicles, and autonomous landing serves as a second case study.Source: §IV · pp. 3–4
  • The landing environments include an open field with moving humanoid actors and a vehicle, an urban town, and a grassland/forest edge.The urban town includes buildings, streets, and parked cars; the grassland/forest edge includes trees and underbrush.Source: Figure 2 · p. 4

A. Urban Navigation Among Pedestrians and Vehicles

In low-altitude urban navigation, the semantic planner reduced collision events relative to the geometric planner, while adding path length and peak jerk. Persistent identities and cached verdicts were exercised during the runs.

A. Urban Navigation Among Pedestrians and Vehicles

  • The comparison used 20 runs per condition on a fixed low-altitude, multi-goal route with two scripted SUVs and two walkers.The perception front end ran a fixed {person, car, truck} prompt set at 10 Hz; the semantic condition used persistent ids and Th=3 s.Source: A. Urban Navigation Among Pedestrians… · p. 4
  • Collision events per run dropped from 1.35 to 0.45 (p=0.038), the only safety metric reaching significance at n=20.Worst-case clearance improved in mean from 0.16 to 0.52 m but not significantly; crash counts were 3 vs. 1 of 20 and within noise.Source: A. Urban Navigation Among Pedestrians… · p. 4
  • The semantic planner maintained larger human-aware separation through additional detours, with path length rising from 54.5 to 68.7 m and peak jerk reaching 2.6×.At n=20, worst-clearance differences below ≈0.2 m are not resolvable given per-run standard deviations of 0.5–0.8 m.Source: A. Urban Navigation Among Pedestrians… · p. 4
  • All observed classes were served from the warm verdict cache, no online LLM query was issued, and perception maintained 9.96 Hz.Per run, 20 object identities were carried across detection gaps of ≥0.5 s; the longest bridged gap equaled the 4 s time-to-live.Source: A. Urban Navigation Among Pedestrians… · p. 4

B. Case Study: Semantics-Aware Landing

The landing study compares geometric avoidance, an LLM using captions without RAG, and the full RAG-stabilized pipeline across three environments. The reported results contrast geometric and unaugmented-LLM performance with the reliability of RAG and grammar-constrained decoding.

B. Case Study: Semantics-Aware Landing

  • The evaluation compares Baseline MPC with depth-only geometric avoidance, LLM–MPC using captions without RAG, and Ours, the full RAG-stabilized pipeline.Trials span three environments; success requires landing within 0.5 m of the target without entering an actor’s 1 m safety buffer, with 50 trials per cell.Source: B. Case Study: Semantics-Aware Landing · p. 4
  • Table II reports landing success rates (%) with 50 trials per entry.Source: Table II · p. 4
  • Geometric avoidance alone reaches 34% landing success among moving actors, while the unaugmented LLM achieves 6–12%.The authors attribute the unaugmented LLM's weaker performance to hallucinated or malformed verdicts that destabilize the planner.Source: B. Case Study: Semantics-Aware Landing · p. 4
  • RAG plus grammar-constrained decoding turns the same 0.5B model into a reliable source of safety constraints.Source: B. Case Study: Semantics-Aware Landing · p. 4

C. Perception Under Limited Input Resolution

The 224-pixel encoder input limits effective resolution for small, distant objects, especially pedestrians. Larger pedestrian margins and short-term persistence help maintain conservative safety behavior; higher-resolution perception within the same compute budget is future work.

C. Perception Under Limited Input Resolution

V. CONCLUSION

The system reduces collision events roughly threefold in urban navigation among pedestrians and vehicles, and raises landing success among moving actors from 34% to 96%, with longer paths and sharper maneuvers. Its retrieval-grounded 0.5B-parameter LLM supplies category-specific safety constraints using per-class caching and persistent map objects; novel-category reasoning and small-object perception remain limitations.

V. CONCLUSION

  • Collision events fall roughly threefold in urban navigation among pedestrians and vehicles, while landing success among moving actors rises from 34% to 96%.The navigation improvement comes at the cost of longer paths and sharper maneuvers.Source: §V · p. 4
  • A retrieval-grounded 0.5B-parameter LLM with a machine-parseable schema and per-class verdict caching supplies context-dependent safety constraints off the per-frame critical path.Persistent objects in the planner map preserve pedestrian-specific constraints through short occlusions.Source: §V · p. 4
  • Pedestrians and parked cars receive different clearance, while pedestrian-specific constraints persist through short occlusions.Source: §V · p. 4
  • All observed categories in the evaluated closed-loop scenarios used the warm verdict cache; retrieval-grounded reasoning for unseen categories and comparison with fixed semantic policies remain future work.Source: §V · p. 4
  • Small-object perception at range remains limited primarily by the 224-pixel encoder input.Future work includes higher-resolution perception within the same compute budget, improved motion models for persistent objects, and cache-miss reasoning for novel categories.Source: §V · p. 4

Related papers

Loading 2610.22092v1…