Source-linked AI summary
Optimizing Search and Rescue UAV Connectivity in Challenging Terrain through Multi Q-Learning
Mohammed M. H. Qazzaz, Syed A. R. Zaidi, Desmond C. McLernon, Abdelaziz Salama, Aubida A. Al-Hameed
TL;DR
SAR UAVs must navigate challenging terrain while maintaining cellular connectivity, and standard trajectory models do not account for changing obstacles and conditions. The paper uses multi Q-learning with separate agents for path planning and base-station connectivity. Evaluation reports strong arrival and obstacle-avoidance success in the first two scenarios, alongside connectivity outages and greater failure at higher obstacle density.
Problem
Standard trajectory models do not account for dynamic SAR obstacles and conditions, making continuous connectivity during complex-terrain navigation an unresolved challenge.
Source: A. Motivation · p. 1Method
Multi Q-learning trains a Strategic Planning Agent for path planning and collision awareness and a Real-time Adaptive Agent for base-station connectivity.
Source: §V · p. 7Results
The first two scenarios had a 100% destination-and-obstacle-avoidance success rate; outages affected approximately 1% of low-frequency and 10% of middle-frequency flights, while 30% obstacle-density testing had a 10% destination failure rate.
Source: §IV · p. 7Takeaways & Limitations
The findings highlight multi Q-learning's potential to enhance UAV autonomy and decision-making in complex environments.
Source: §V · p. 7
Abstract
from arXiv · showhide
Using Unmanned Aerial Vehicles (UAVs) in Search and rescue operations (SAR) to navigate challenging terrain while maintaining reliable communication with the cellular network is a promising approach. This paper suggests a novel technique employing a reinforcement learning multi Q-learning algorithm to optimize UAV connectivity in such scenarios. We introduce a Strategic Planning Agent for efficient path planning and collision awareness and a Real-time Adaptive Agent to maintain optimal connection with the cellular base station. The agents trained in a simulated environment using multi Q-learning, encouraging them to learn from experience and adjust their decision-making to diverse terrain complexities and communication scenarios. Evaluation results reveal the significance of the approach, highlighting successful navigation in environments with varying obstacle densities and the ability to perform optimal connectivity using different frequency bands. This work paves the way for enhanced UAV autonomy and enhanced communication reliability in search and rescue operations.
I. INTRODUCTION
SAR UAVs need robust connectivity in rugged environments, where existing path-planning approaches do not fully address dynamic obstacles and conditions. The paper proposes multi Q-learning to adapt trajectories, building on prior work in path planning, collision avoidance, and connectivity.
A. Motivation
- Cellular connectivity is promising, but SAR UAV trajectory planning still needs to adapt to evolving obstacles and environmental conditions while maintaining communication.Satellite channels have high latency and cost, while UAV-specific channels require substantial infrastructure investment.Source: A. Motivation · p. 1
- Multi Q-learning is proposed to dynamically adjust SAR UAV trajectories for connectivity while navigating rugged landscapes.The paper presents this as a way to integrate connectivity optimization with trajectory planning.Source: A. Motivation · p. 1
B. Related Works
- Prior work spans SAR path planning and collision avoidance, connectivity-aware multi-UAV strategies, 5G network functions, and multi-robot coordination.Methods include fuzzy logic, particle swarm optimization, genetic algorithms, and simultaneous inform and connect path planning.Source: B. Related Works · pp. 1–2
C. Contributions and Problem Statement
The paper targets connectivity for SAR UAVs in rugged terrain, proposing multi Q-learning for connectivity-aware trajectories and short travel time. The section also previews the paper's organization.
C. Contributions and Problem Statement
- SAR UAVs in rugged terrain need reliable communication with ground base stations, especially in remote or adverse conditions.The paper links communication gaps to mission effectiveness and the safety of rescuers and victims.Source: C. Contributions and Problem Statement · p. 2
- The proposed multi Q-learning approach plans UAV trajectories for connectivity while seeking the shortest travel time.It aims to adapt trajectories in real time to obstacles, terrain variations, and communication dynamics.Source: C. Contributions and Problem Statement · p. 2
D. Organisation
- The paper introduces the RL model, then the proposed model and policies, reports results, and closes with conclusions.The sections are organized as RL model, proposed model, results and discussion, then conclusions.Source: D. Organisation · p. 2
II. REINFORCEMENT LEARNING (RL) ALGORITHM
The section describes reinforcement learning as iterative decision-making through state transitions and rewards or penalties. It then explains Q-learning's Q-table and the variables used in its formal update.
Reinforcement learning
- An RL agent learns through actions that move it between states, reinforcing successful transitions with rewards and avoiding unsuccessful ones through penalties.The described process iterates so the agent can distinguish positive and negative actions.Source: §II · p. 2
Q-learning
- Q-learning stores state-and-action rewards in a Q-table after each episode, which guides later choices toward maximizing rewards and avoiding penalties.The table accumulates information about optimal actions and rewards.Source: §II · pp. 2–3
Q-learning equation
- The Q-learning equation uses state, action, and subsequent reward, with α setting the learning rate and γ weighting future rewards against immediate ones.The passage defines St, At, Rt+1, α, and γ.Source: §II · p. 3
III. SYSTEM MODEL
The system model concerns autonomous UAV navigation between origin and target in remote SAR settings. It combines path planning with crash-risk reduction and connectivity to the remote controller or pilot.
System model
- The scenario deploys UAVs on SAR missions in remote, difficult-to-access areas, potentially over long distances and tough terrain.Some mission areas may be inaccessible to human intervention.Source: §III · p. 3
- Path planning must mitigate crash risk while maintaining seamless connectivity with the remote controller or pilot.Source: §III · p. 3
- The proposed system places an autonomous model on the UAV to navigate from origin to target and optimize operational efficiency.Source: §III · p. 3
A. Model Environement
The model uses a grid-based 3D environment to represent rugged terrain and obstacles. The UAV must remain within cellular coverage, avoid obstacles, and find an efficient route between endpoints.
A. Model Environement
- A grid-based 3D environment represents rugged terrain and potential obstacles for training and testing the UAV agent.The agent is configured with parameters listed in Table I.Source: A. Model Environement · p. 3
- The UAV must satisfy coverage constraints, avoid obstacles, and find the most efficient route between its initial and destination endpoints.Source: A. Model Environement · p. 3
- The simulation uses the COST Hata propagation model to estimate rural wireless path loss, accounting for terrain roughness and vegetation characteristics.The described inputs include frequency, base-station antenna height, UAV-to-base-station distance, and a terrain roughness correction factor; multiple frequency ranges are explored.Source: A. Model Environement · p. 3
B. Model Impelementation
The implementation trains two agents using exploration and exploitation strategies. An ϵ-Greedy policy shifts action selection from random exploration toward choices with higher estimated Q-values as training progresses.
B. Model Impelementation
- The implementation uses multiple policies to train two agents for the SAR UAV connectivity optimization model.Source: B. Model Impelementation · p. 3
- Early in training, ϵ-Greedy encourages random actions so agents can gather experience about rewards and penalties.Source: B. Model Impelementation · p. 4
- As training progresses and Q-values converge, the agents increasingly select actions with higher estimated Q-values; ϵ begins high and decreases over time.The adaptive adjustment is intended to balance exploring new possibilities with exploiting learned strategies.Source: B. Model Impelementation · p. 4
1) Strategic Planning Agent:
The Strategic Planning Agent handles relatively static path-planning and obstacle-location tasks, learning a route from an initial point to randomly selected destinations. It rewards progress toward the destination and penalizes deviations and collisions.
1) Strategic Planning Agent:
- The agent plans a shortest route from a fixed initial location to randomly selected destinations, assuming path and obstacle information remains relatively static.Source: 1) Strategic Planning Agent · p. 4
- Actions that reduce distance to the destination receive rewards, while actions that deviate from the optimal path receive penalties.Source: 1) Strategic Planning Agent · p. 4
- Encountering an obstacle incurs a substantial penalty intended to discourage collision-producing actions and favor obstacle-free paths.Source: 1) Strategic Planning Agent · p. 4
- Through repeated exploration and feedback, the agent refines its navigation strategy to identify paths while avoiding obstacles.Source: 1) Strategic Planning Agent · p. 4
2) Real-time Adaptive Agent:
The Real-time Adaptive Agent addresses changing connectivity conditions by identifying coverage locations that maintain communication with the base station. It learns from coverage feedback across frequency bands, and is described as improving UAV connectivity robustness and reliability.
2) Real-time Adaptive Agent:
- Because connectivity conditions vary dynamically, the independent Real-time Adaptive Agent adapts coverage choices to maintain UAV-base-station connectivity.Its objective includes selecting paths that maximize signal strength and minimize the risk of signal loss or degradation.Source: 2) Real-time Adaptive Agent · p. 4
- The agent receives rewards for traversing locations with accepted coverage and penalties for areas with insufficient coverage, across 900, 1800, and 2100 MHz bands.Source: 2) Real-time Adaptive Agent · p. 4
- Prioritizing optimal coverage locations across frequency bands is described as improving cellular-connected UAV robustness and reliability and supporting operations in challenging environments.Source: 2) Real-time Adaptive Agent · p. 5
3) Dual Model Decision-Making Approach:
The dual-agent approach combines offline-trained path-planning and connectivity agents, then reconciles their recommendations using their Q-values. When recommendations conflict, obstacle avoidance takes priority, with an extra step available to preserve connectivity.
3) Dual Model Decision-Making Approach:
- Both agents train offline for separate missions: strategic path planning and collision awareness, and adaptive coverage selection for base-station connectivity.The adaptive agent may also train online to respond to changing channel conditions.Source: 3) Dual Model Decision-Making Approach · p. 5
- At each time step, each agent selects its best action from its Q-values for the current state, and the UAV determines its next move from both recommendations.When actions differ, the algorithm compares each agent’s Q-value for the other agent’s proposed action.Source: 3) Dual Model Decision-Making Approach · pp. 5–6
- When recommendations conflict, the UAV prioritizes the Strategic Planning Agent’s action to avoid obstacles, even if that route crosses weak coverage.It may take an additional step, lengthening the path, to maintain connectivity with the ground base station.Source: 3) Dual Model Decision-Making Approach · pp. 5–6
IV. RESULTS AND DISCUSSION
Evaluation reports successful arrival and obstacle avoidance in the first two scenarios, alongside connectivity outages and worse outcomes at higher obstacle density. Training results also show that obstacle complexity and frequency band shape learning dynamics.
IV. RESULTS AND DISCUSSION
- Early training episodes incur significant penalties while the Q-model is unfamiliar with states, and more episodes may be needed to explore the state space effectively.The training discussion describes learning through repeated trials using rewards and penalties.Source: §IV · p. 6
- The Strategic Planning Agent reaches positive rewards more quickly with fewer obstacles, while denser environments make optimal-action selection and collision avoidance more difficult.Average rewards and penalties accumulated during training are used to gauge agent effectiveness.Source: §IV · p. 6
- Higher-frequency conditions, including 2100 MHz, bring greater path loss and longer learning with higher penalties, whereas 900 MHz supports faster learning and more favorable connectivity.The band effect is apparent early in training, when the agent encounters many weak-coverage locations.Source: §IV · pp. 6–7
- Table II reports 100% destination-reaching and on-path obstacle-avoidance success in the first two scenarios, while outages affected approximately 1% of low-frequency-band flights and 10% of middle-frequency-band flights.The evaluation prioritized successful delivery over navigating weak-coverage points.Source: Table II · p. 7, §IV · p. 7
- At 30% obstacle density, testing found a 10% destination-reaching failure rate, while approximately 25% of locations had unsatisfactory connectivity efficiency, especially near borders and at high frequencies.The passage attributes the higher crash likelihood to increased obstacle density despite sufficient training.Source: §IV · p. 7
V. CONCLUSIONS
The paper uses two multi Q-learning agents to coordinate SAR UAV path planning and cellular connectivity, with evaluations showing navigation across obstacle densities and frequency-dependent connectivity trade-offs. The authors identify real-time weather, dynamic obstacle detection, and multi-agent coordination as directions for future work.
V. CONCLUSIONS
- Lower frequencies ensured higher arrival rates, while higher frequencies offered more path-loss characteristics in far areas, revealing a trade-off between arrival success and continuous connectivity.The Real-time Adaptive Agent learned to adapt to different frequency bands and learned faster with lower frequencies due to stronger signals around the environment.Source: §V · p. 7
- The Strategic Planning Agent achieved a high success rate across environments with varying obstacle densities, while the Real-time Adaptive Agent maintained connectivity with a ground base station.The agents were trained for efficient path planning and collision awareness, and to maintain optimal connectivity, respectively.Source: §V · p. 7
- The authors suggest exploring real-time weather data, dynamic obstacle detection, and multi-agent coordination for collaborative search patterns in future work.They describe the findings as promising for real-world SAR and highlight multi Q-learning's potential to enhance UAV autonomy and decision-making in complex environments.Source: §V · p. 7