Source-linked AI summary
Autonomous UAV Navigation Using Reinforcement Learning
Huy X. Pham, Hung M. La, David Feil-Seifer, Luan V. Nguyen
TL;DR
UAV navigation in unknown environments is difficult when accurate environmental models or prior knowledge are unavailable. This paper applies Q-learning with a UAV position controller and evaluates the approach in simulation and a real implementation. The UAV learns to reach a goal in both settings, with the real implementation reducing the route from 100 steps initially to 8 steps in later episodes.
Problem
UAV navigation in unknown environments is challenging when environmental models or prior knowledge are limited or unavailable.
Source: §I · p. 1Method
The paper combines Q-learning over discretized UAV states with a position controller that translates chosen next states into flight control.
Source: §III · pp. 2–3, §IV · p. 3Results
100 steps in the first real-implementation episode fell to 8 steps in later episodes as the UAV learned to reach the goal.
Source: §VI · p. 6Takeaways & Limitations
Simulation and implementation show that a UAV can learn to navigate an environment without a mathematical model, providing a simple framework for such applications.
Source: §VII · p. 6Real-world deployment should account for stochastic learning and environmental uncertainties such as wind and other dynamics.
Source: §VII · p. 6
Abstract
from arXiv · showhide
Unmanned aerial vehicles (UAV) are commonly used for missions in unknown environments, where an exact mathematical model of the environment may not be available. This paper provides a framework for using reinforcement learning to allow the UAV to navigate successfully in such environments. We conducted our simulation and real implementation to show how the UAVs can successfully learn to navigate through an unknown environment. Technical aspects regarding to applying reinforcement learning algorithm to a UAV system and UAV flight control were also addressed. This will enable continuing research using a UAV with learning capabilities in more important applications, such as wildfire monitoring, or search and rescue missions.
I. INTRODUCTION
Prior UAV reinforcement-learning work largely addresses control and trajectory tasks, while high-level navigation and practical implementation on physical UAVs remain less covered. This paper presents a detailed implementation framework for learning tasks in unknown environments.
I. INTRODUCTION
- UAVs support unknown-environment missions including wildfire monitoring [1], target tracking –, and search and rescue, but many approaches rely on accurate environment models or prior knowledge,.The paper notes that such knowledge and data are often limited or unavailable in realistic implementations.Source: §I · p. 1
- Earlier UAV reinforcement-learning work includes motion planning, stable maneuver trajectories, PID tuning for tracking, and adaptation to airflow disturbances.The cited examples address suspended-load trajectory planning, stable trajectories, tracking, and airflow disturbances.Source: §I · p. 1
- Other cited approaches target computational time through parallel action selection, model learning, and planning, or use distance information in Q-learning to reduce time to a target.These are presented as efforts to improve reinforcement-learning performance in UAV applications.Source: §I · p. 1
- The paper identifies limited work on high-level UAV tasks such as navigation and monitoring, and limited practical implementation detail for physical UAV systems.Its stated contribution is a detailed implementation of a UAV that learns to accomplish tasks in an unknown environment.Source: §I · p. 1
- The framework uses a simple reinforcement-learning algorithm to navigate from an arbitrary start to a goal by a shortest route in an unknown environment.The paper presents this as its main framework contribution.Source: §I · p. 1
II. PROBLEM FORMULATION
The problem is to move a UAV from an arbitrary starting position to a known goal when the environment is insufficiently known for model-based planning. The proposed setup uses reinforcement learning to choose actions, with a position controller translating the chosen next state into flight control.
II. PROBLEM FORMULATION
- Figure 1 depicts the discretized environment: the red circle is the current UAV state, green circles are next-iteration options, and a red flag marks the goal.The figure presents the state and possible choices in a closed environment.Source: Figure 1 · p. 1
- The UAV must reach a pre-described goal from an arbitrary position, while limited environmental information can make model-based motion planning unavailable.The UAV observes its position, and reinforcement learning can use data obtained directly from the system.Source: §II · p. 2
- The learning agent maps encountered situations to actions to maximize a numerical reward measuring its performance.The setup defines the reward as the signal guiding the agent’s choices.Source: §II · p. 2
- The learning model selects a desired next state from the current state, then the position controller computes input for a lower-level propeller controller to generate thrust.The position controller must handle the UAV’s nonlinear dynamics to support stable trajectories and hovering at the new state.Source: §II · p. 2
III. REINFORCEMENT LEARNING AND Q LEARNING
The paper models navigation as an agent–environment learning problem and uses Q-learning to select actions from discrete states. It defines state, action, transition, and reward components, with a grid representation and goal-based rewards.
III. REINFORCEMENT LEARNING AND Q LEARNING
- The model treats learning as agent–environment interaction in which the agent accumulates experience, assuming the environment has the Markov property.Under that assumption, the next state and reward depend only on the current state.Source: §III · p. 2
- The learning model comprises states, actions, transition probabilities, and rewards, with this paper treating transitions as deterministic.The transition probability is set to 1 for the modeled move.Source: §III · p. 2
- Q-learning iteratively estimates state–action values in a Q-table, which the agent uses to choose actions that maximize total reward.The objective is to derive an optimal policy from the state–action value function.Source: §III · p. 2
- The UAV’s state is its approximate position at a grid-cell center; holding altitude constant reduces the environment to a two-dimensional grid.The environment is represented by equal-radius spheres whose centers form a grid, becoming circles when altitude is held constant.Source: §III · p. 3
- The UAV chooses among four lateral directions; reaching the landmark-identified goal earns a big reward, while other locations incur a small penalty.An action that would take the UAV beyond the environment’s border leaves it in its current state.Source: §III · p. 3
IV. CONTROLLER DESIGN AND ALGORITHM
The controller moves the UAV between discrete states and uses position feedback to hold it near each desired state. Because proportional control produced overshoot, the design uses a tuned PID controller alongside Q-learning.
IV. CONTROLLER DESIGN AND ALGORITHM
- The position controller translates the UAV from state s_k to s_{k+1} and holds it within error radius d.It uses the real-time position and desired position to compute the control input.Source: §IV · p. 3
- Proportional control produced excessive overshoot and instability after state transitions, motivating use of a standard PID controller.The cited work describes PID as providing relatively good hovering stabilization despite not effectively regulating quadrotor nonlinearity.Source: §IV · p. 3
- Tuning increased derivative gain and eliminated integral gain to achieve a stable trajectory.The integral term can reduce steady-state error but may increase overshoot; the derivative term can reduce overshoot and settling time.Source: §IV · p. 3
- After tuning, the UAV remained within d = 0.3m of the desired state.The controller combines PID with Q-learning in the stated algorithm.Source: §IV · pp. 3–4, Figure 6 · p. 4
V. SIMULATION
The simulation tests Q-learning on a 5 by 5 grid, with a fixed altitude and a goal at (5, 5). It finds an 8-step optimal route after 39 training episodes.
V. SIMULATION
- The simulated environment was a 5 by 5 board with 25 states, and the UAV navigated from (1, 1) to (5, 5) at constant altitude.Reaching the goal earned +100; other outcomes received a penalty of -1.Source: §V · p. 4
- Figure 7 depicts the simulated environment at time step t = 17, marking the original start S and goal G.The task was to reach the goal in the shortest possible way.Source: Figure 7 · p. 4, §V · p. 4
- 8 steps was the optimal route from the selected starting position to the goal, found after 39 training episodes.The episode-by-episode time steps are shown in Figure 8.Source: §V · p. 4, Figure 8 · p. 4
VI. IMPLEMENTATION
The real-world implementation used a quadrotor, motion capture, and PID control to test Q-learning in a discretized room. The UAV learned a route that reached the goal in fewer steps.
VI. IMPLEMENTATION
- A Parrot AR Drone 2.0 used Motion Analysis motion capture for relative position, while PID control helped it move between states.Source: §VI · pp. 4–5
- After 38 episodes, steps fell from 100 in the first episode to 8 in later episodes, and the final trajectory reached the target by the shortest possible way.Source: Figure 11 · p. 5, §VI · pp. 5–6, Figure 12 · p. 6
- A MATLAB GUI displayed the UAV position, steps, Q-table values, and episode comparisons, and saved data so learning could resume after a failure.Source: §VI · p. 5, Figure 10 · p. 5
- The closed-room test used a 5 by 5 board with 25 states, goal-only environment knowledge, a +100 goal reward, and a -1 penalty otherwise.The learning rate was α = 0.1 and the discount rate was γ = 0.9.Source: §VI · p. 5
VII. CONCLUSION
The paper presents PID+ Q-learning for quadrotor navigation in an unknown environment and reports similar successful outcomes in simulation and implementation. It frames the approach as a simple framework while noting that real-world deployment must account for uncertainty.
VII. CONCLUSION
- The PID+ Q-learning technique let a quadrotor learn to navigate to a target in an unknown environment, with simulation and implementation showing similar successful outcomes without a mathematical model.Source: §VII · p. 6
- The authors present the work as a simple framework for UAVs operating where an environment model is unavailable.Source: §VII · p. 6
- For real-world deployment, the authors say stochastic learning should account for uncertainties such as wind and other environmental dynamics.Source: §VII · p. 6
- Future directions include wildfire monitoring, search and rescue, and extending the work to multi-agent systems for coordination and effectiveness.Source: §VII · p. 6