Source-linked AI summary
CIG-RL: Curiosity-Driven Information-Guided Reinforcement Learning for Source Term Estimation in Uncertain Environments
Junhee Lee, Seunghwan Kim, Hongro Jang, Hyungjin Kim, Hyoungho Park, Changseung Kim, Hyondong Oh
TL;DR
Hazardous-gas STE must operate with noisy measurements, while information-theoretic action selection is computationally costly and existing DRL exploration can be inadequate. CIG-RL combines curiosity-driven exploration of novel belief transitions with uncertainty-adaptive active perception, and simulations plus real-world experiments support robust, efficient, feasible source search.
Problem
Hazardous-gas STE requires robust source-property estimation under noise, but information-theoretic planning is costly and existing DRL methods rely on random exploration or current-belief uncertainty reduction.
Method
CIG-RL uses a curiosity network to explore novel belief-state transitions and an uncertainty-adaptive reward balancing mutual information with source-directed approach.
Results
Simulations under highly noisy conditions and real-world experiments show robust, efficient, feasible source search with practical computational cost for real-time deployment.
Takeaways & Limitations
Coupling belief-space exploration with active-perception guidance supports robust source search under noisy conditions.
Abstract
from arXiv · showhide
Source term estimation (STE), which aims to estimate key properties of the gas source, is essential for identifying hazardous gas releases. Information-theoretic approaches have been adopted for autonomous STE using mobile sensors due to robustness in noisy environments, yet their online action selection incurs substantial computational cost. Deep reinforcement learning (DRL) provides a promising alternative with its fast decision-making capability. In DRL-based STE, the agent selects actions based on belief states of the source term updated from noisy measurement sequences. However, existing methods rely on random exploration or solely on belief uncertainty reduction without an effective exploration strategy in DRL, which can limit policy robustness in noisy environments. To address this, we propose a curiosity-driven information-guided reinforcement learning for robust and efficient STE. The proposed method promotes active exploration of novel belief state transitions that have not been sufficiently explored during training. We further introduce an uncertainty-adaptive active perception reward to guide efficient source search under uncertainty. Simulations under high-noise conditions and real-world experiments demonstrate the robustness and feasibility of the proposed framework, highlighting its potential for practical STE problems.
I. INTRODUCTION
STE uses mobile sensors to estimate hazardous gas-source properties, but existing approaches face computational and exploration challenges under noisy, turbulent conditions. CIG-RL addresses these issues through curiosity-driven belief-transition exploration and uncertainty-adaptive active perception.
- STE estimates key gas-source properties, such as source location and release strength, to support hazardous-leak identification.
- Information-theoretic STE selects actions by maximizing information gain but incurs high computational cost when evaluating action candidates.
- End-to-end DRL enables fast, non-myopic search but lacks explicit source estimators for determining completion or reliably estimating source location.
- Under severe noise, particle-filter beliefs may favor incorrect hypotheses, making effective exploration of belief-state transitions important during training.
- CIG-RL explores novel belief transitions with a curiosity network and balances information gathering with source approach using particle-filter variance.
B. End-to-End DRL
DRL-based STE methods combine learned search policies with explicit or implicit source-state representations. CIG-RL uses particle-filter source estimation and GMM-derived belief features within a noisy gas-dispersion setting.
- End-to-End DRL: End-to-end DRL methods use learned policies and memory networks to leverage historical measurements during source search.
- End-to-End DRL: Explicit-estimator DRL frameworks improve source-search reliability by representing source uncertainty with particle-filter beliefs.
- End-to-End DRL: Information-guided DRL methods address noisy search through information-theoretic rewards, but concentration-based exploitation can be vulnerable to spurious measurements.
- End-to-End DRL: CIG-RL focuses on localizing the source position while updating its source-term distribution from gas measurements using a particle filter.
- End-to-End DRL: The framework uses an isotropic plume model to describe gas concentration at the sensing position.
B. Sensor Model
The sensor model represents measurements as gas concentration corrupted by environmental turbulence and sensor noise. A particle filter then estimates the source-term distribution and uses resampling procedures to mitigate particle degeneracy and impoverishment.
- Sensor Model: Sensor measurements combine modeled gas concentration with wind-turbulence noise and sensor-measurement noise.
- Sensor Model: Both noise terms are modeled as Gaussian, with sensor-noise scale linked to the modeled concentration through β.
- Sensor Model: The total noise standard deviation combines the environmental and sensor-noise variances.
- Sensor Model: A particle filter represents the source-term probability distribution with weighted particles and updates weights sequentially after new measurements.
- Sensor Model: Resampling addresses particle degeneracy, while MCMC is applied afterward to reduce particle impoverishment.
IV. CURIOSITY-DRIVEN INFORMATION-GUIDED REINFORCEMENT LEARNING
CIG-RL represents particle-filter beliefs compactly for reinforcement learning. GMM clustering extracts structured belief features, which are assembled into the state used by the policy.
- IV. CURIOSITY-DRIVEN INFORMATION-GUIDED REINFORCEMENT LEARNING: The framework introduces state and action representations before explaining curiosity-driven belief-space exploration and active-perception rewards.
- 1) State:: GMM clustering extracts features from the particle filter to improve source-estimation efficiency.
- 1) State:: The belief state is approximated by GMM component means, diagonal covariance terms, mixture weights, and the mean of all particles.
- 1) State:: The GMM uses L = 3 clusters, whose components are consistently ordered to avoid label-switching ambiguity.
- 1) State:: The state representation is built from the extracted belief-state features for downstream policy learning.
2) Action:
CIG-RL uses curiosity-driven exploration in latent belief space to seek insufficiently learned belief transitions during STE policy training. The resulting prediction-error reward is combined with active perception signals to support robust and efficient learning.
- The action space is parameterized by heading direction, while each movement uses a fixed distance Δd.
- The exploration module predicts the next latent belief representation and uses prediction error as an auxiliary reward for exploring novel belief transitions.It combines a feature extractor, forward network, and inverse network.
- The feature extractor encodes particle-filter belief states, while the forward network predicts the next latent representation from the current representation, position, and action.
- The inverse network predicts the action responsible for a transition, helping retain action-relevant information in the latent belief representation.
- The curiosity network minimizes a weighted sum of forward and inverse losses, with ε balancing the two terms.
- Large prediction error indicates an insufficiently learned belief transition and encourages the policy to explore it.The error is the Euclidean distance between actual and predicted next latent representations and generally decreases as training progresses.
C. Uncertainty-Adaptive Active Perception Reward
The uncertainty-adaptive active perception reward combines mutual information with a distance-to-source term. Particle-filter variance adjusts their relative influence, prioritizing uncertainty reduction early and source approach as estimates become more reliable.
- The reward augments mutual information with a distance-based source approach term and adaptively balances them using particle-filter variance.This design supports robust search under uncertainty while facilitating rapid approach to the source.
- Mutual information based on Shannon entropy reduction rewards actions that reduce uncertainty in the estimated source term.
- The distance term measures separation between the agent position and the estimated source location to accelerate source search.
- Each source-term dimension is normalized before variance computation, and tr(Cov(θt)) is clipped below by (0.1)^2 for numerical stability.
- High particle-filter variance gives mutual information greater influence, whereas decreasing variance lets the distance term contribute more strongly.The transition reflects increasing reliability of source estimation as measurements accumulate.
- The total extrinsic reward includes +10 for successfully finding the source and −5 for leaving the search boundary.
D. Learning Framework for CIG-RL
CIG-RL combines SAC with curiosity-driven exploration and active perception rewards for robust source search. Its replay strategy preserves extrinsic uncertainty reduction while updating novelty rewards online.
- Learning Framework: CIG-RL uses soft actor-critic with auxiliary learning to improve exploration in source term estimation.The framework combines SAC with a curiosity-driven learning signal.
- Learning Framework: The replay buffer stores interaction tuples and reuses sampled mini-batches to support diverse experiences and efficient training.Stored tuples include state, action, reward, and next state.
- Learning Framework: The extrinsic active perception reward represents uncertainty reduction in the estimated source term.This reward is stored and sampled from the replay buffer.
- Learning Framework: The auxiliary reward captures belief-transition novelty and is updated by the curiosity network rather than replayed.This decoupling mitigates repeated exploration of previously uncertain regions.
- Learning Framework: Extrinsic and auxiliary rewards are combined using a weight parameter η to train the critic and target Q-network.The resulting target Q-value incorporates the combined reward and SAC terms.
- Learning Framework: The actor is optimized with an entropy-regularized SAC objective, while automatic entropy tuning adjusts α to target policy stochasticity.The actor objective balances entropy against estimated action values.
A. Simulation Setup
The simulations vary gas and wind conditions across two high-noise environments, with environment 2 imposing more severe environmental and sensor noise. Training uses fixed episode and step settings, while performance is compared against information-theoretic and DRL baselines.
- Simulation Setup: Two simulation environments vary gas properties and wind conditions, with environmental noise levels of 0.4 and 0.5.These settings exceed the 0.2 environmental noise used in cited prior studies.
- Simulation Setup: Environment 2 is configured as the more challenging setting because it incorporates higher environmental and sensor noise.The resulting measurements exhibit more severe fluctuations.
- Simulation Setup: Each simulation ends after 300 steps or when the 2,000-particle filter standard deviation falls below 0.1.Training runs for 80,000 episodes, with source and agent locations randomized at episode start.
B. Evaluation Metrics
Evaluation uses success rate and mean travel distance on unseen scenarios, with a stricter one-meter success criterion than training. CIG-RL shows robust performance, and ablations indicate that curiosity, uncertainty adaptation, and decoupled reward sampling contribute to search quality and efficiency.
- Evaluation Metrics: Success rate measures the percentage of episodes that estimate the source location successfully, while mean travel distance averages distances over successful episodes.Both metrics are evaluated on 1,000 random scenarios unseen during training.
- Performance Comparison: CIG-RL achieves success rates of 98.3% in environment 1 and 95.0% in environment 2.It is reported as the most robust performer in both environments.
- Performance Comparison: Training curves show the highest success rate and lowest success-rate variability among DRL baselines.The result is reported as evidence of robust policy learning under noisy conditions.
- Ablation Analysis: Removing both curiosity and active perception reward guidance significantly degrades performance, while removing either component individually causes smaller declines.The ablation supports coupling belief-space exploration with active perception guidance.
- Ablation Analysis: Mean travel distance increases without particle-filter variance in the active perception reward, and replaying auxiliary rewards causes less efficient search.The findings support uncertainty adaptation and decoupled sampling.
2) Sensitivity Analysis of CIG-RL:
CIG-RL’s curiosity reward supports exploration of novel belief transitions and converges under high-noise conditions. Real-world experiments further assess source-search performance and computation in two environments.
- Sensitivity to η: When η exceeds 5, excessive early exploration increases mean travel distance, whereas insufficient curiosity degrades success rate.The auxiliary reward must balance extrinsic and curiosity-driven exploration.
- Curiosity convergence: The curiosity auxiliary reward starts high and consistently decreases to a low value in both high-noise environments.This convergence indicates stable training of the belief-based curiosity mechanism despite noisy measurements.
- Experimental setup: Real-world experiments use a CO2 source, a mobile gas-sensing platform, LiDAR, and onboard computation in an 8 m×8 m gym.Each algorithm is tested 10 times, with source-search termination based on particle-filter variance and a 0.5 m localization criterion.
- Real-world comparison: Infotaxis has the highest one-step computation time, while MVG-RDDPG and CIG-RL require substantially less computation through learned DRL policies.The experiments compare Infotaxis, MVG-RDDPG, and CIG-RL in both real-world environments.
- Overall findings: CIG-RL couples curiosity-driven belief exploration with uncertainty-adaptive active perception for robust source search under noisy conditions.Simulations and real-world experiments report improved success rate and search efficiency while retaining practical computational cost.