PaperlayerView exact revision on arXiv

Source-linked AI summary

CAVEAT: Recurrent Multimodal Diffusion Planning for Mapless Aerial Exploration

Steven Visch, Nicolò Botteghi, Antonio Franchi, Barbara Bazzana

arXiv:2609.06623v1cs.ROcs.LG

TL;DR

Aerial exploration in unknown spaces commonly uses explicit spatial representations, raising the question of whether a fixed-dimensional recurrent state can support useful mapless deployed planning. CAVEAT conditions diffusion waypoint generation on multimodal observations and evaluates its inference mechanisms in simulation and proof-of-concept Elios 3 experiments.

  • Problem

    Can useful UAV exploration be generated from onboard observations and a compact recurrent state instead of a persistent global spatial representation in the deployed policy?

    Source: §I · p. 1

  • Method

    CAVEAT uses fused LiDAR, visual, and pose features to update a recurrent state conditioning diffusion waypoint generation, with trajectories imitated from map-based FUELv2.

    Source: §I · pp. 1–2

  • Results

    Rolling inference improves coverage per traveled distance, while CAVEAT reaches high final coverage but FUELv2 is more distance-efficient; Elios 3 tests demonstrate partial exploration and visual servoing.

    Source: 1) Rolling-inference ablation · p. 5, 3) Comparison with the FUELv2 expert · p. 6, §IV · p. 7

  • Takeaways & Limitations

    The study provides proof-of-concept evidence for recurrently conditioned aerial exploration without a persistent global map in the deployed policy.

    Source: §IV · p. 7

    Real-world experiments demonstrate partial exploration, and future work will evaluate more diverse and geometrically distinct environments.

    Source: §IV · p. 7

Abstract

from arXiv · show

Can exploratory UAV waypoint sequences be generated from multimodal onboard observations and a fixed-dimensional recurrent internal state without maintaining a persistent global map in the deployed policy? We investigate this question through CAVEAT, a diffusion policy conditioned on a recurrent internal state updated from fused LiDAR, visual, and pose features and trained from trajectories generated by the map-based FUELv2 expert. Rolling inference partially warm-starts consecutive predictions, while a temporary local signed distance field provides heuristic obstacle guidance. Simulation results evaluate both inference mechanisms and compare CAVEAT with its demonstration-generating expert. Proof-of-concept experiments on a Flyability Elios 3 demonstrate partial exploration of a previously unseen indoor environment and target-directed visual servoing using a separately trained policy.

I. INTRODUCTION

Aerial exploration in unknown spaces must make use of onboard sensing, while established planners commonly rely on explicit spatial representations. CAVEAT investigates whether a fixed-dimensional recurrent state can support mapless deployed waypoint generation and evaluates its inference mechanisms against the map-based expert that supplies demonstrations.

I. INTRODUCTION

  • CAVEAT asks whether a compact recurrent state can represent the spatial and historical information needed for useful exploration without a persistent global representation in the deployed policy.This question arises because established methods explicitly represent environments as maps, frontiers, trees, graphs, or coverage structures.Source: §I · p. 1
  • CAVEAT conditions diffusion-generated waypoint sequences on fused LiDAR, visual, and pose features summarized by a recurrent internal state.FUELv2 supplies the training trajectories using an explicit global spatial representation, while the deployed policy uses current observations and its recursively updated state.Source: §I · pp. 1–2
  • Rolling inference warm-starts consecutive denoising processes, while a temporary local signed distance field guides nearby waypoints toward greater obstacle clearance.Sparse waypoints are passed to an NMPC tracking layer in training and simulation, or to polynomial-based trajectory generation and tracking in Elios 3 experiments.Source: §I · p. 2
  • The evaluation studies rolling inference and local SDF guidance, compares CAVEAT with FUELv2 across four simulated environment families, and includes Elios 3 proof-of-concept tests.A separately trained instance of the architecture also demonstrates target-directed visual servoing as a secondary behavior.Source: §I · p. 2

II. METHOD

CAVEAT processes current LiDAR, camera, and pose observations together with its recurrent state to generate high-level waypoint sequences, then executes the first waypoint and replans. Its multimodal encoders fuse observation features for recurrent conditioning of the diffusion model.

II. METHOD

  • At each decision step, CAVEAT generates odometry-frame waypoints, executes the first, and replans from updated observations after interpolation subject to motion constraints.Training and simulation use an NMPC tracking layer; real-world deployment uses a trajectory-generation and tracking module with PID control.Source: §II · p. 2

A. Recurrent Multimodal Diffusion Policy

  • Modality-specific encoders process LiDAR, visual, and pose observations; fused features update a GRU state that conditions diffusion-based waypoint generation.The state recursively represents observation history and conditions the diffusion model generating the next waypoint sequence.Source: Figure 2 · p. 2, A. Recurrent Multimodal Diffusion Policy · p. 2

II. METHOD

  • A separately trained instance of the same architecture is evaluated for target-directed visual servoing.This behavior is distinct from the exploration policy discussed in the method section.Source: §II · p. 2

A. Recurrent Multimodal Diffusion Policy

  • The LiDAR encoder samples the point cloud to 1024 points and uses shared pointwise transformations with symmetric max pooling to produce a 512-dimensional feature.Symmetric pooling makes the output independent of point ordering.Source: A. Recurrent Multimodal Diffusion Policy · p. 2
  • The visual encoder uses three ResNet-style stages and an 8 × 8 pooled feature map before a linear projection produces a 512-dimensional feature.The coarse grid preserves approximate image-location information, relevant to the separately trained target-directed behavior.Source: A. Recurrent Multimodal Diffusion Policy · p. 2
  • A two-layer MLP maps seven-dimensional VIO pose estimates to 128 features; concatenated modality features are compressed into the 512-dimensional fused observation feature.The learned fusion determines the modalities’ relative contributions from training data.Source: A. Recurrent Multimodal Diffusion Policy · p. 2

1) Multimodal observation encoding:

CAVEAT recursively updates a fixed-dimensional GRU state to summarize observation history without constructing a geometric map. That state conditions a diffusion model that iteratively generates waypoint sequences.

2) Recurrent internal state:

  • The fused feature recursively updates a 512-dimensional GRU state that summarizes observation history without being constructed or supervised as a geometric map.The state dimension is independent of episode duration, and it is initialized to zero for each observation sequence.Source: 2) Recurrent internal state · p. 3

3) Conditional waypoint diffusion:

  • A temporal convolutional network predicts noise along ordered waypoint sequences, with the recurrent state conditioning intermediate features through FiLM.The convolutions provide an inductive bias toward local dependence between adjacent waypoints.Source: 3) Conditional waypoint diffusion · p. 3
  • Starting from an initially noisy waypoint sequence, iterative noise prediction conditioned on ht yields the sequence; only xt|t is executed before updated observations are processed.The observation horizon is 33 observations (32 history steps plus the current observation). The model is trained to predict added noise over 100 diffusion steps.Source: 3) Conditional waypoint diffusion · p. 3

B. Rolling Inference and Local Obstacle Guidance

Rolling initialization carries part of the previous predicted trajectory into the next denoising process, while the updated recurrent state continues to encode observation history. Later waypoints receive progressively more noise, with the final waypoint resampled.

B. Rolling Inference and Local Obstacle Guidance

  • Rolling initialization reuses the preceding prediction to address abrupt changes between independently generated plans, while the GRU state separately summarizes observation history.The reused trajectory represents previous-plan information; the GRU state represents encoded observation history.Source: B. Rolling Inference and Local… · p. 3, 1) Rolling inference · p. 3

1) Rolling inference:

  • After the first waypoint is executed, the remaining sequence is shifted and its first three waypoints are retained for the next denoising process.Later waypoints receive progressively increasing Gaussian noise, and the final waypoint is fully resampled.Source: 1) Rolling inference · p. 3
  • The partially perturbed sequence initializes the next denoising process, which is conditioned on the updated recurrent state.The noise amplitude schedule increases linearly from 0.1 to 1.0 across the later waypoints.Source: 1) Rolling inference · p. 3

2) Local SDF guidance:

CAVEAT uses a temporary SDF derived from currently observed LiDAR geometry to guide nearby waypoints toward clearance. This local heuristic neither builds a persistent global map nor guarantees collision-free plans or trajectories.

2) Local SDF guidance:

  • A temporary two-dimensional SDF is built from LiDAR points within ±0.5 m of flight altitude, projected onto the horizontal plane.The field uses only the current observation and is not accumulated into a persistent global map.Source: 2) Local SDF guidance · p. 3
  • During the final four denoising steps, waypoint planar positions below the 0.5 m clearance threshold are adjusted using the SDF gradient, while vertical position and yaw remain unchanged.The method also checks whether straight-line segments to predicted waypoints are blocked and displaces blocked planar positions toward the UAV.Source: 2) Local SDF guidance · p. 3
  • The local SDF guidance is a heuristic, not a hard collision constraint, and does not guarantee collision-free waypoint sequences or executed trajectories.It relies only on currently observed LiDAR geometry.Source: 2) Local SDF guidance · p. 3

C. Expert Demonstrations and Training

FUELv2 generates the map-based expert trajectories used to train CAVEAT by imitation in Gazebo. Training maps observation histories to expert waypoint sequences, with a separate same-architecture policy trained for target-directed visual servoing.

C. Expert Demonstrations and Training

  • FUELv2 generates Gazebo demonstrations using a global TSDF map, then is replaced by the learned policy at deployment.It was selected for its open-source ROS implementation supporting range measurements and odometry while accounting for imperfect state estimation.Source: C. Expert Demonstrations and Training · p. 4
  • Ray-based free-space carving was added during demonstration generation because sparse LiDAR measurements otherwise caused free space ahead of observed surfaces to be classified as occupied.The modification provides positive signed-distance evidence to voxels between the sensor origin and each LiDAR return, and is used only for expert demonstrations.Source: C. Expert Demonstrations and Training · p. 4
  • The diffusion model is trained to predict added Gaussian noise after component-wise trajectory normalization, using a squared-cosine noise schedule and a mean-squared-error objective.The modality encoders, fusion network, GRU, and conditional diffusion U-Net are optimized jointly for 50 epochs with batch size 96.Source: C. Expert Demonstrations and Training · p. 4
  • Each training sample maps 33 multimodal observations to an eight-waypoint expert trajectory, using 32 history steps plus the current observation.The eight commands are consecutive samples after the current observation, each containing three position coordinates and yaw.Source: C. Expert Demonstrations and Training · p. 4
  • A separate model with the same architecture is trained on simulated target-approach trajectories for visual servoing, distinct from the exploration policy.Its training trajectories approach a predefined visual target from randomized initial conditions.Source: C. Expert Demonstrations and Training · p. 4

III. EXPERIMENTAL EVALUATION

The evaluation tests simulation exploration, inference-mechanism effects on occupancy coverage per distance, and physical deployment. It uses FUELv2 demonstrations and measures forward-looking LiDAR coverage across simulated trials.

III. EXPERIMENTAL EVALUATION

  • The study asks whether CAVEAT explores effectively in simulation, how rolling inference and local SDF guidance affect coverage per distance, and whether it deploys on the target platform.The physical target is the Flyability Elios 3.Source: §III · p. 4

A. Experimental Setup and Metrics

  • The simulated sensing setup uses a 360-degree Ouster LiDAR, a forward-facing visual-inertial camera, an additional forward-facing LiDAR, and an available pose estimate.The additional forward-facing LiDAR supports demonstration generation, local geometric guidance, and coverage evaluation; the Ouster LiDAR is not an input to the learned observation encoder.Source: A. Experimental Setup and Metrics · p. 4
  • FUELv2 demonstrations were collected in four Gazebo environment families spanning connected rooms to layouts with multiple rooms, occlusions, and more complex connectivity.Fig. 3 identifies the training families Atrain–Dtrain.Source: A. Experimental Setup and Metrics · p. 4, Figure 3 · p. 5
  • Simulation reports forward-looking occupancy coverage against UAV traveled distance, with mean curves and standard-deviation shading across randomized trials.Trials terminate when the evaluation-side FUELv2 frontier set empties or the planner reaches a local minimum; that map is not provided to CAVEAT.Source: A. Experimental Setup and Metrics · p. 5
  • The demonstration dataset contains 2,674 trajectories and 199,603 recorded time steps.Trajectories were split into training and validation sets at the trajectory level; evaluation episodes were generated separately rather than replayed from the demonstrations.Source: Table I · p. 5, A. Experimental Setup and Metrics · p. 5

B. Simulation Results

The simulation study examines rolling inference, local SDF clearance, and comparison with the FUELv2 expert. In the rolling-inference ablation, rolling initialization improves occupancy coverage per traveled distance in both test environments.

B. Simulation Results

  • The simulation study evaluates rolling inference, varies the local SDF clearance threshold, and compares the complete policy with its FUELv2 demonstration-generating expert.Learned-policy parameters remain fixed within each comparison.Source: B. Simulation Results · p. 5

1) Rolling-inference ablation:

  • The rolling-inference ablation changes initialization while holding observations, learned-policy parameters, diffusion settings, and local SDF guidance unchanged.It compares independent Gaussian initialization with rolling initialization.Source: 1) Rolling-inference ablation · p. 5
  • Rolling inference reaches approximately 80% coverage within 80 m in Atest and approximately 95% within 70 m in Btest, outperforming independent initialization on coverage per traveled distance in both.Independent initialization remains near 60–65% in Atest over substantial additional distance and reaches approximately 95% at 175 m; in Btest it reaches approximately 85% after 140 m.Source: 1) Rolling-inference ablation · p. 5

2) SDF-threshold sensitivity:

The tested SDF clearance threshold changes coverage efficiency, especially in Btest. Against FUELv2, CAVEAT achieves high final coverage in unseen test environments but needs more travel, while both methods complete coverage in Btrain.

3) Comparison with the FUELv2 expert:

  • FUELv2 reaches approximately 85–90% coverage within 40 m in Atest and within 45 m in Btest, while CAVEAT approaches 90% after approximately 70–75 m.The comparison evaluates transfer from FUELv2 demonstrations to a learned policy that does not execute the expert's mapping and frontier-planning pipeline at deployment.Source: Figure 5 · p. 6, 3) Comparison with the FUELv2 expert · p. 6

2) SDF-threshold sensitivity:

  • With dmin = 0.5 m, coverage reaches approximately 90% after 70 m in Atest and approximately 95% within 75 m in Btest; dmin = 1.0 m is slower, especially in Btest.In Atest, the 1.0 m threshold requires approximately 100 m for comparable coverage; in Btest it stays near 55–65% over an extended interval and approaches completion after approximately 160–170 m.Source: 2) SDF-threshold sensitivity · pp. 5–6, Figure 5 · p. 6

3) Comparison with the FUELv2 expert:

C. Real-World Proof of Concept

Physical trials tested CAVEAT’s exploration pipeline in a previously unseen indoor environment, using DDPM and DDIM sampling; the section also reports target-directed visual-servoing evaluation with a separately trained model.

C. Real-World Proof of Concept

  • The physical proof of concept evaluated exploration with DDPM and DDIM and target-directed visual servoing using a separately trained instance of the architecture.Source: C. Real-World Proof of Concept · p. 6

1) Exploration:

  • 10 DDIM steps took approximately 1.5 s per waypoint sequence, versus 100 DDPM steps and approximately 3.5 s; faster updates yielded the longer displayed trajectory.The measurements were obtained on an off-board workstation.Source: 1) Exploration · pp. 6–7, Figure 8 · p. 7
  • Both samplers produced partial exploratory motion in the previously unseen indoor environment, with displayed flights progressing from the lower-left toward the upper-right of the mapped region.Fig. 8 depicts representative trajectories, headings, and start and end positions.Source: 1) Exploration · pp. 6–7, Figure 8 · p. 7
  • Battery lifetime limited flight duration, but the trials executed the complete perception-to-waypoint pipeline and continued exploration on the Elios 3.Source: 1) Exploration · p. 7

2) Target-Directed Visual Servoing:

With augmented training images, the target-directed visual-servoing policy produced three marker-approach trajectories. The discussion and conclusion frame the work as a feasibility demonstration with bounded evidence and identify broader evaluation and task integration as future work.

2) Target-Directed Visual Servoing:

  • Image augmentation was followed by three trajectories that approached the wall-mounted marker from different initial positions, without the target-loss behavior observed before augmentation.The augmented images included noise, blur, brightness and contrast variations, and radial lens distortion; an approach counts as successful when the target enters close view.Source: 2) Target-Directed Visual Servoing · p. 7, Figure 9 · p. 7

D. Discussion

  • The results do not establish general equivalence to FUELv2, guaranteed collision avoidance, or broad real-world generalization; exploration and visual servoing use separately trained policies.Source: D. Discussion · p. 7

IV. CONCLUSION

  • Future work will examine recurrent-state information, more diverse and geometrically distinct environments, and integration of exploration with target-directed behavior.A unified dataset is proposed as a route toward an integrated exploration-and-inspection paradigm.Source: §IV · p. 7, D. Discussion · p. 7
Loading 2609.06623v1…