PaperlayerView exact revision on arXiv

Source-linked AI summary

Information-Guided Safe Reinforcement Learning for Autonomous Gas Source Localization using sUAS

Sachin Giri, Thomas Zhao, Matthew Huynh, YangQuan Chen

arXiv:2609.05569v1cs.RO

TL;DR

Turbulent, intermittent concentrations make sUAS gas-source localization an ill-posed inverse problem, and deterministic planners can stall on flawed early estimates. The paper combines EMGR planning with KL-triggered learned exploration under an RCBF safety filter. In simulation, RL reaches roughly 79% success compared with baselines around 30%, with zero barrier violations when plume was detected.

  • Problem

    Sparse, noisy measurements make estimating a gas source location an ill-posed inverse problem, while deterministic planners can stall on flawed early estimates.

    Source: §I · p. 1, B. Empirical Observability Gramian… · p. 4

  • Method

    A KL-monitoring meta-supervisor blends EMGR planning with Meta-SAC exploratory actions, with an RCBF safety layer accounting for worst-case disturbances.

    Source: §I · p. 1

  • Results

    79.3% Stage 2 success versus baselines around 30%, while Stage 3 maintains 78.6% with mobile sources; zero barrier violations occurred across episodes where plume was detected.

    Source: B. Results and Discussions · p. 7

  • Takeaways & Limitations

    The simulation results provide a rigorous foundation for robust deployment of information-guided safe reinforcement learning for trace-gas localization.

    Source: §VI · p. 7

    The convergence criterion relies on oracle distance unavailable in deployment, and calibrating a deployable Gramian-based stopping rule remains ongoing work.

    Source: B. Results and Discussions · p. 7

Abstract

from arXiv · show

The autonomous localization of fugitive gas emissions using small Unmanned Aircraft Systems (sUAS) constitutes a fundamentally ill-posed inverse problem. In turbulent atmospheric boundary layers, highly intermittent scalar concentration fields violate the assumptions of classical gradient-based navigation, causing data-driven estimators to suffer from severe noise and spurious local minima. To address these challenges, we introduce an Information-Guided Safe Reinforcement Learning framework evaluated within a custom, GPU-accelerated 3D simulation environment coupling an Eulerian wind solver with a Lagrangian puff dispersion model. We identify a critical vulnerability in deterministic information-seeking planners - a Gramian bias where agents act greedily upon flawed early estimates, starving the estimator of spatial diversity. To systematically break this degeneracy, our architecture integrates a classical empirical observability Gramian (EMGR) planner with a learned Soft Actor-Critic (SAC) exploratory policy. A deterministic meta-supervisor actively monitors estimator reliability via Kullback-Leibler (KL) divergence, dynamically blending deterministic exploitation with learned exploration to steer the sUAS into high-information zones. Trained via a progressive curriculum and safeguarded by a strictly enforced Robust Control Barrier Function (RCBF), our RL framework achieves nearly 80% localization success on complex, mobile sources - drastically outperforming classical baselines (~30%) - while ensuring zero safety violations.

I. INTRODUCTION

Turbulent, sparse measurements make gas-source localization ill-posed, while deterministic planners can stall around flawed early estimates. The paper combines EMGR planning with learned exploration and develops a GPU-accelerated 3D simulation testbed.

I. INTRODUCTION

  • The framework blends deterministic EMGR velocities with Meta-SAC exploration under KL-divergence monitoring and applies an RCBF safety layer accounting for worst-case disturbances.The supervisor monitors mismatch between observations and estimator predictions and dynamically blends the actions.Source: §I · pp. 1–2

II. GPU-ACCELERATED 3D SIMULATION ENVIRONMENT

  • The custom GPU-accelerated 3D testbed couples an Eulerian wind solver with a Lagrangian puff model to capture macroscopic wind and sub-grid scalar intermittency.The environment is built on NVIDIA Warp.Source: §II · p. 2, §I · p. 2, Figure 1 · p. 2

A. Eulerian Stable Fluids Solver

  • The Eulerian solver uses Stable Fluids operator splitting, a Helmholtz-Hodge projection, and GPU Red-Black SOR to maintain incompressibility at real-time rendering rates.An Ornstein-Uhlenbeck process supplies external forcing to mimic gusty atmospheric boundary-layer behavior.Source: A. Eulerian Stable Fluids Solver · p. 2
  • The solver enforces zero velocity inside obstacles and no vertical flow at the ground, while open domain limits use free-outflow boundaries.Solid interfaces impose a homogeneous Neumann condition for the Poisson solver; lateral faces and sky use a Dirichlet pressure condition.Source: A. Eulerian Stable Fluids Solver · p. 2

B. Lagrangian Gaussian Puff Model

The paper models continuous gas leaks as discrete Lagrangian puffs to preserve intermittent concentration structure. It formulates source recovery from sparse measurements as an ill-posed inverse problem.

B. Lagrangian Gaussian Puff Model

  • A continuous leak is represented by fixed-mass puffs released every τe seconds, with centroid motion combining Eulerian wind integration and sub-grid Langevin noise.Methane buoyancy is modeled as an initial acceleration that decays as the puff ages.Source: B. Lagrangian Gaussian Puff Model · pp. 2–3
  • Each puff follows a 3D Gaussian concentration distribution whose anisotropic variance grows with age, and total concentration sums the active puffs with a reflected image at the ground.The image method enforces a physical no-flux boundary at the earth’s surface.Source: B. Lagrangian Gaussian Puff Model · p. 3
  • Decoupling microscopic gas transport from the macroscopic grid avoids Eulerian numerical smearing and preserves sharp concentration gradients for realistic navigation.The paper presents this as narrowing the reality gap for sim-to-real transfer.Source: B. Lagrangian Gaussian Puff Model · pp. 2–3

A. The Ill-Posed Inverse Problem

  • Recovering the 3D source location from sequential measurements is severely ill-posed because sparse concentrations make distinct hypotheses indistinguishable and small measurement perturbations can yield unbounded estimate changes.In turbulent conditions, concentration gradients are dominated by advection and point predominantly downwind, away from the source.Source: A. The Ill-Posed Inverse Problem · p. 3

1) Maximum Likelihood Estimation:

The section defines an MLE point estimate from a sliding measurement window and complements it with a particle filter that maintains a weighted ensemble of source hypotheses.

1) Maximum Likelihood Estimation:

  • The MLE maximizes mean-subtracted projected correlation between observed and predicted concentrations over a sliding window of measurements.The wind history is fractionally filtered to capture long-range dependence in predicted concentrations.Source: 1) Maximum Likelihood Estimation · p. 3
  • A Grünwald–Letnikov fractional filter with order α ∈ [−0.9, 0] preprocesses wind history using fractional integration for long-memory smoothing.The order is selected by grid search over deterministic survey trajectories.Source: 1) Maximum Likelihood Estimation · p. 3

2) Particle Filter for Hypothesis Diversity:

  • Because the MLE cannot represent multimodal uncertainty, a Bayesian particle filter maintains weighted source hypotheses and updates their importance weights upon each measurement.Systematic resampling with jitter is triggered when effective sample size falls below a threshold.Source: 2) Particle Filter for Hypothesis… · p. 3
  • The particle filter’s effective sample size indicates weight degeneracy, while the MLE and PF provide competing but complementary estimates.When only a few particles carry most of the mass, effective sample size is much smaller than the particle count.Source: 2) Particle Filter for Hypothesis… · p. 3

3) KL Divergence Model-Mismatch Detector:

The KL detector compares observed and predicted concentrations to assess MLE reliability, while the EMGR uses sensitivity-based information measures to guide source-seeking trajectories. In 3D turbulence, Gramian bias can draw a planner toward an inaccurate estimate and reduce the spatial diversity needed for correction.

3) KL Divergence Model-Mismatch Detector:

B. Empirical Observability Gramian (EMGR)

  • The EMGR estimates nonlinear-system observability through forward simulations that perturb the source estimate, extending prior guidance to 3D turbulence.Its sensitivity vectors compare predicted concentrations under positive and negative parameter perturbations.Source: B. Empirical Observability Gramian… · p. 4
  • A forgetting factor γw < 1 accommodates a drifting estimate in 3D turbulence, while inverse-trace information provides a scalar summary for the RL observation.The reciprocal of the Gramian’s smallest singular value is also identified as an unobservability index.Source: B. Empirical Observability Gramian… · p. 4
  • When the estimate is inaccurate, Gramian bias can attract the planner toward that estimate and deprive estimators of spatial diversity for self-correction.The deterministic planner’s planning Gramian perturbs candidate sensor positions.Source: B. Empirical Observability Gramian… · p. 4

IV. INFORMATION-GUIDED SAFE RL FRAMEWORK

The framework wraps EMGR planning in an RL architecture whose deterministic meta-supervisor blends a classical planner with learned exploration. The agent is formulated as an MDP and receives a 15-dimensional state that includes perception and estimator-reliability information.

IV. INFORMATION-GUIDED SAFE RL FRAMEWORK

  • The framework uses a learned exploratory policy and deterministic meta-supervisor to blend EMGR planning and address the planner’s tendency to follow a flawed estimate.That tendency can starve estimators of the spatial diversity needed for correction.Source: §IV · p. 4

A. Markov Decision Process Formulation

1) State Space (S):

  • The agent observes a 15-dimensional state whose first eight features encode raw perception, including normalized position and a log-compressed, bias-corrected sensor signal.The state also includes a plume-detection flag and an EMGR directional hint attenuated by estimator reliability.Source: 1) State Space (S) · p. 4
  • The state includes a normalized KL alarm and a sigmoidal compression of the unobservability index derived from the Gramian trace.Source: 1) State Space (S) · pp. 4–5

2) Action Space (A):

The SAC policy outputs a three-dimensional velocity command, which is blended with the classical planner and safety-filtered before the sUAS position is updated.

2) Action Space (A):

  • The Soft Actor-Critic policy outputs a squashed-Gaussian action in [-1, 1]^3, decoded as a full 3D velocity vector scaled by vmax.Blending with the classical planner is handled by the meta-supervisor.Source: 2) Action Space (A) · p. 5

Transition Dynamics (P):

  • The sUAS position evolves using the safety-filtered velocity plus a bounded disturbance, with projection enforcing domain and altitude constraints.Source: 2) Action Space (A) · p. 5

B. Deterministic Meta-Supervisor

After deterministic search detects the plume and activates the MLE, a meta-supervisor blends EMGR planning with learned SAC exploration. The SAC policy uses entropy-regularized learning, with design choices intended to mitigate overestimation and exploration collapse.

B. Deterministic Meta-Supervisor

  • After plume detection activates the MLE, the meta-supervisor augments deterministic EMGR planning with RL through four handover signals.When active, the MLE and particle-filter estimates are fused according to their disagreement.Source: B. Deterministic Meta-Supervisor · p. 5
  • A KL-triggered signal responds to estimator mismatch, while proximity counters Gramian bias near the estimate and a void signal responds to plume loss.Source: B. Deterministic Meta-Supervisor · p. 5
  • The composite blending weight sets the velocity command, with a minimum exploration floor and a stagnation override that forces full RL authority.The override applies when displacement remains below δstag for nstag consecutive steps, for nesc steps.Source: B. Deterministic Meta-Supervisor · p. 5

C. Soft Actor-Critic Policy

  • The SAC policy maximizes entropy-regularized return, and twin Q-networks mitigate overestimation through a soft Bellman target.The critic uses Huber loss, while automatic temperature tuning has a hard floor to prevent exploration collapse.Source: C. Soft Actor-Critic Policy · p. 5

D. Robust Control Barrier Functions

The RCBF filters velocity commands for obstacle safety, while a three-stage curriculum trains the policy from a restricted static-source setting to mobile sources. Reward shaping and convergence rules use simulation information, with deployment stopping-rule calibration still ongoing.

D. Robust Control Barrier Functions

  • The RCBF applies a minimal-correction quadratic program with an inflated obstacle keepout margin and disturbance-robust constraints; the agent trains on the executed safe action.The executed action teaches the policy to anticipate safety corrections.Source: D. Robust Control Barrier Functions · p. 5

E. Curriculum Training Strategy

  • Training progresses from a central static source, to static sources across the full domain, then to stochastic source drift, initializing each stage from the previous policy.The stages target denser plume encounters first, then prolonged search and recovery for non-stationary sources.Source: E. Curriculum Training Strategy · pp. 5–6

F. Reward Design and Convergence

  • The reward combines particle-filter posterior shift with proximity shaping that uses the true source only as privileged simulation reward information, not observable state.The passage states potential-based shaping theoretically leaves the optimal policy unchanged.Source: F. Reward Design and Convergence · p. 6
  • Training terminates episodes using an oracle-distance threshold, while a proposed cumulative-Gramian stopping rule for deployment remains under calibration.Timeouts bootstrap from successor states rather than being treated as true terminal states.Source: F. Reward Design and Convergence · p. 6

V. SIMULATION EXPERIMENTS AND RESULTS

The simulation evaluates localization in a large three-dimensional domain with staged source conditions and a common convergence criterion. The execution loop separates initial search from RL-augmented tracking.

A. Experimental Setup

  • The simulated domain is 256 × 128 × 128 m3, with 3 to 5 randomly placed spherical obstacles per episode.The sUAS operates between 4 and 60 m altitude, at a maximum speed of 8 m/s.Source: A. Experimental Setup · p. 6
  • Algorithm 1 describes information-guided residual SAC with a search budget and horizon, perimeter-sweep or Lévy-based velocity, an RL-augmented tracking phase, RCBF safety filtering, and measurement-based MLE and particle-filter updates.Source: A. Experimental Setup · p. 6
  • Each curriculum stage trains for 500 episodes; Stage 1 uses a central 80 × 40 m source region, Stage 2 expands to 226 × 98 m, and Stage 3 adds stochastic drift.All five configurations are evaluated over 100 episodes with the same 5 m convergence criterion.Source: A. Experimental Setup · p. 6

B. Results and Discussions

The RL stages substantially outperform the classical baselines on localization success, including with mobile sources, while reported safety violations remain zero. The results also report faster convergence and identify oracle-distance evaluation and difficult plume detection as boundaries.

B. Results and Discussions

  • 79.3% (Stage 2) and 78.6% (Stage 3) success, versus about 30% for both baselines; Stage 3 uses mobile sources.Stage 3’s median convergence time is 276 steps.Source: Table I · p. 6, B. Results and Discussions · p. 7
  • RL agents reached terminal success rates three to four times faster than baselines, while Fig. 4 shows their median distance falling below 5 m within 300 steps.Baselines plateaued above 20 m in Fig. 4.Source: Figure 5 · p. 7, Figure 4 · p. 7, B. Results and Discussions · p. 7
  • 98.8% success in Stage 1 partly reflects overfitting to its restricted source region; expanding the domain in Stage 2 reduced success to 79.3%.The paper attributes the Stage 2 setting to distant sources and weaker initial signals across a seven times larger area.Source: B. Results and Discussions · p. 7
  • Zero barrier violations were reported in episodes with plume detection; RL triggered the RCBF on 84 to 89% of steps versus 7 to 13% for baselines.The minimum barrier margin was 2.54 m.Source: Table II · p. 7, B. Results and Discussions · p. 7
  • The 20% failure rate in Stages 2 and 3 primarily stems from boundary sources and occluded plumes that prevent reliable detection within the search budget.The evaluation convergence criterion relies on oracle distance, which is unavailable in deployment.Source: B. Results and Discussions · p. 7

VI. CONCLUSION

The paper presents a curriculum-trained Meta-SAC framework that blends EMGR planning with learned exploration under KL-divergence monitoring and an RCBF safety filter. Sim-to-real transfer and multi-source, multi-agent scenarios remain future work.

VI. CONCLUSION

  • The proposed framework combines an EMGR planner, curriculum-trained Meta-SAC exploration, KL-divergence monitoring, and an RCBF safety filter for trace-gas localization.The conclusion describes validation in a GPU-accelerated 3D simulation.Source: §VI · p. 7
  • Future work includes sim-to-real hardware transfer and multi-source, multi-agent scenarios; code and trained models are also provided online.The reproducibility note identifies a GitHub environment and Kaggle models for all three RL stages.Source: §VI · p. 7, REPRODUCIBILITY · p. 7

Related papers

Loading 2609.05569v1…