PaperlayerView exact revision on arXiv

Source-linked AI summary

Computer Vision Navigation for an Autonomous Vehicle Model Using Reinforcement Learning

Dinis Fernandes, Tiago Ribeiro, António Ribeiro, Carolina Lopes, Gil Lopes, A. Fernando Ribeiro

arXiv:2610.33755v1cs.CV

TL;DR

Autonomous driving motivates learning-based control rather than hard-coded solutions. This work trains a camera-based reinforcement-learning policy to control a vehicle in simulation, where it navigates a simplified track but does not converge in a more visually complex scenario.

  • Problem

    The project addresses how to learn autonomous driving rather than rely on hard-coded solutions.

    Source: §1 · p. 2

  • Method

    A reinforcement-learning policy uses processed camera frames as states and outputs steering and rear-wheel speed commands for the vehicle.

    Source: §2 · p. 3

  • Results

    The policy navigated the simplified track, but additional visual noise such as parking spots and pedestrian crossings affected performance.

    Source: §6.2 · p. 13, §7 · p. 14

  • Takeaways & Limitations

    The results demonstrate the potential of reinforcement learning for visual-input vehicle navigation, while complex scenes require more robust feature extraction.

    Source: §7 · p. 14

    Physical-vehicle testing was not completed, and training did not converge in the more complex scenario.

    Source: §1 · p. 2, §7 · p. 14

Abstract

from arXiv · show

This work aims to assess the capabilities of the reinforcement learning algorithm, Proximal Policy Optimization (PPO), in developing a policy capable of controlling the speed and steering angle of a vehicle model to navigate within simulated road lines. The control system's input is a camera, and its outputs are the rotational speed of the rear wheels and the steering angle of the front wheels, controlling the vehicle's speed and direction, respectively. The algorithm is trained and tested in a simulated environment, where it receives image frames from the vehicle's camera, processes them through a feature extraction algorithm, and feeds them into the policy. For each frame, the policy outputs speed and direction commands for the vehicle. The policy is iteratively updated to enhance its performance. The reinforcement learning algorithm successfully generates a policy capable of navigating the vehicle in a simplified environment. Potential improvements are suggested to enhance the system's robustness in more complex scenarios. Creating a simulation environment suitable for testing an advanced reinforcement learning algorithm is a complex task. This project offers a framework for other developers to test different approaches and algorithms. The tested algorithm, PPO, uses proximity intervals - an innovative approach in reinforcement learning. In some scenarios, the algorithm achieved better performance than predecessor algorithms, which are typically harder to implement and require tuning more hyperparameters.

1. Introduction

The project applies reinforcement learning to camera-based autonomous navigation on a simulated robotics-championship track. It trains a policy to control steering and speed, but physical-vehicle testing was not completed.

1. Introduction

  • Reinforcement learning maps states to actions through a policy that is iteratively adjusted using rewards or punishments.The agent must understand its state relative to the environment to choose an action.Source: §1 · p. 1
  • Autonomous driving motivates learning-based approaches because the goal is to learn the task rather than use a hard-coded solution.The introduction compares this motivation with reinforcement learning's demonstrated success in other domains.Source: §1 · pp. 1–2
  • LAR's earlier autonomous-driving projects included road-sign recognition, sign-responsive control, and policies for three parking maneuvers.The cited projects used YOLOv3, YOLOv3 with DDPG, and policy training for parallel, angular, and perpendicular parking, respectively.Source: §1 · p. 2
  • The project trains a camera-driven policy to navigate the Portuguese National Robotics Championship track by controlling front-wheel steering and rear-wheel speed.The vehicle model is based on a design developed by LAR at the University of Minho.Source: §1 · p. 2
  • Training was conducted in simulation because real-world training requires continuous supervision and poses hardware risks; physical-vehicle testing was not completed.The time needed to develop the simulation, implement the algorithm, and train the policy exceeded expectations.Source: §1 · p. 2
  • The simulation uses Gazebo and ROS for the vehicle, actuators, and camera data, with OpenCV for feature extraction and TensorFlow 2 for PPO.The project also provides code and a demonstration video in its supplementary materials.Source: §1 · pp. 2–3

2. Algorithm overview

The algorithm turns camera frames into a state and maps that state to wheel-control actions. Training repeatedly gathers experience and updates the neural-network policy when its buffer is full.

2. Algorithm overview

  • A processed camera frame forms the state, and the policy outputs front-wheel axle angle and rear-wheel rotation speed to control direction and speed.The angle and rotation speed are expressed in radians and radians per second, respectively.Source: §2 · p. 3
  • During operation after training, extracted features define the state fed to the neural-network policy, whose action values control the wheels through actuators.Figure 3 is identified as an overview of the algorithm after training.Source: §2 · p. 3, Figure 3 · p. 3
  • Training repeats experience collection and policy updates until navigation performance is satisfactory; gradient updates begin when the experience buffer reaches capacity.The buffer stores tuples gathered from sensor states and actuator actions for use in updating the neural network.Source: §2 · p. 3

3. Feature extraction algorithm

Feature extraction focuses the camera input on road lanes and converts it into a form suited to the policy's neural-network input. The processing removes irrelevant image content and reduces the frame to a one-dimensional array.

3. Feature extraction algorithm

  • The feature-extraction flow first crops camera frames to isolate the road area and remove the unnecessary horizon.The method is designed to extract relevant environmental data with a focus on road lanes.Source: §3 · p. 4, Figure 5 · p. 4
  • Frames are converted to grayscale, downscaled to reduce input and buffer memory demands, and thresholded so pixels become binary lane indicators.The downscaled images retain enough detail to identify road lanes, while thresholding assigns zero below the threshold and one above it.Source: §3 · p. 4
  • The final frame is flattened into a one-dimensional array to match the neural network's input layer.Figure 6 is identified as showing feature-extraction algorithm layers.Source: §3 · p. 4, Figure 6 · p. 5

4. Reward system

The reward system encourages lane-centred, forward motion while penalizing leaving the track or following an incorrect section sequence. Speed reward was added to avoid a policy that stays still or ends episodes quickly to avoid lane-deviation penalties.

4. Reward system

  • The reward uses vehicle position and policy speed, rewarding proximity to the lane centre with a negative reward proportional to centre deviation.When displacement stays within the threshold and the vehicle follows the correct section, reward is speed reward minus centre displacement.Source: §4 · p. 5
  • Leaving track boundaries or entering an incorrect track section ends the episode and incurs a -10 penalty.Boundary detection compares vehicle displacement with a threshold; track sections are ordered and checked against the previous section.Source: §4 · p. 5
  • Adding speed to the reward discourages a stationary policy and the earlier strategy of crashing quickly to avoid accumulating lane-centering penalties.Without speed in the reward, ending the episode early could be preferable to remaining centred and incurring penalties.Source: §4 · p. 5

5. Proximal policy optimization

PPO is presented as an actor–critic method whose actor selects actions and whose critic estimates state values to evaluate the actor. Its training combines stochastic exploration, advantage estimation, constrained policy updates, and mini-batch gradient steps.

5.1. Actor-critic

  • The actor maps state values to actions, while the critic estimates state value and helps evaluate the actor; both networks are updated with gradient methods.The critic loss is the mean squared error between its value estimate and that estimate plus the advantage; repeated updates can make the value estimate differ from its initial value.Source: §5.1 · p. 6

5.2. Exploration

  • Actions are sampled from a normal distribution centered on policy outputs, with deviation controlling exploration and adjusted during training.The account describes higher deviation early in training and reduced deviation as the policy improves.Source: §5.2 · pp. 6–7

5.3. Policy actions

  • A sigmoid constrains policy outputs to zero-to-one values, which are transformed to the vehicle’s actuator ranges: 0–6 rad/s for rear-wheel speed and −0.27–0.27 rad for front-axle angle.The action values correspond to speed and steering angle, respectively.Source: §5.3 · p. 7

5.4. Advantage

  • Generalized Advantage Estimation combines advantage estimates across n values to balance dependence on the critic’s value estimate against increased variance.The implementation stores rewards and states, then uses critic value estimates with the stored rewards to calculate the advantage.Source: §5.4 · p. 9

5.5. Loss Function

  • PPO’s clipped surrogate loss approximates a trust-region update, limiting policy changes through a clipped probability ratio; the total loss adds an entropy term scaled by a user-defined coefficient.The clipping and minimum operations bound the loss on one side relative to the parameter-change ratio, encouraging conservative changes.Source: §5.5 · pp. 9–10

5.6. Mini-batch

  • The algorithm divides collected experience into randomized mini-batches for successive gradient updates, aiming to make each subset representative of the full batch.The full batch is computationally intensive to process at once, while an unrepresentative mini-batch can yield unreliable updates.Source: §5.6 · p. 11

5.7. Algorithm implementation

  • Each iteration collects states, actions, rewards, next states, episode-completion flags, and action probabilities; the critic then estimates values for advantage calculation before randomized mini-batch updates.The implementation flow is depicted in Figure 13.Source: §5.7 · p. 11, Figure 13 · p. 11

6. Results

The experiments tested PPO using average training reward, with both runs sharing hyperparameters. PPO navigated the simplified track but did not converge on the original track containing parking spaces and pedestrian crossings.

6. Results

  • The experiments compared performance using average reward during training, and both runs used the same hyperparameters.Source: §6 · p. 12, Table I · p. 12

6.1. Experiment 1

  • Experiment 1 used a simplified track without parking spaces or pedestrian crossings, with episodes starting at one of four locations and training in two stages.The second stage began with policy parameters learned in the first stage.Source: §6.1 · p. 12, Figure 14 · p. 12, Figure 15 · p. 13

6.1. Experiment 1; 6.2. Experiment 2

  • PPO converged and navigated the simplified track without crashing, but failed to converge when parking spaces and pedestrian crossings added noisy state data.Testing of the simplified-track policy covered five episodes of 2048 iterations each.Source: §6.1 · p. 12, Figure 16 · p. 13, §6.2 · pp. 13–14

6.2. Experiment 2

7. Conclusions

The study evaluates PPO for camera-based vehicle navigation in simulation. PPO learned to navigate the simplified track when the state contained road lines, but performance was affected by additional visual noise.

  • The study evaluated PPO for controlling an autonomous vehicle model in simulation, using camera input processed through feature extraction to inform policy actions.Source: §7 · p. 14
  • PPO learned to navigate a simplified track when the state representation contained only road lines, while parking spots and pedestrian crossings affected performance.The authors identify robust feature extraction that filters irrelevant visual information as important for navigation.Source: §7 · p. 14
  • Training did not converge in the more complex scenario, and the authors recommend improving feature extraction and investigating transfer to real-world implementations.Source: §7 · p. 14
  • The study contributes evidence of the potential for reinforcement learning to control vehicles based on visual input.Source: §7 · p. 14

8. Further Work

The proposed future work addresses robustness, transfer beyond simulation, and broader system capabilities. It also calls for comparisons with alternative algorithms and further study of training strategies, validation, and safety.

  • Investigate transfer of the trained policy from simulation to real-world scenarios, including sensor noise, lighting variability, and hardware constraints.Source: §8 · p. 14
  • Develop more robust feature extraction, including techniques such as convolutional neural networks, for environments with crossings and parking lots.Source: §8 · p. 14
  • Compare PPO with SAC, DDPG, and A3C on training time, convergence, and robustness.Source: §8 · p. 15
  • Extend the system to multiple tasks, including obstacle avoidance and dynamic path planning, using hierarchical reinforcement learning frameworks.Source: §8 · p. 15
  • Explore reward shaping and curriculum learning to accelerate training and improve policy generalization.Source: §8 · p. 15
  • Validate the approach on a physical vehicle platform to assess real-world performance and simulation-to-deployment discrepancies.Source: §8 · p. 15
  • Investigate policy explainability with a focus on safety-critical autonomous driving applications.Source: §8 · p. 15

Related papers

Loading 2610.33755v1…