PaperlayerView exact revision on arXiv

Source-linked AI summary

Federated Transfer Reinforcement Learning for Autonomous Driving

Xinle Liang, Yang Liu, Tianjian Chen, Ming Liu, Qiang Yang

arXiv:1910.06001v1cs.LGcs.AIcs.RO

TL;DR

Simulator-to-robot RL commonly uses a time-consuming sequential transfer process, with little simulator knowledge available to robot models during fine-tuning. This paper proposes federated online transfer across agents and environments, and reports improved collision-avoidance results when simulator agents transfer knowledge to RC cars. The FTRL-DDPG-SIM results include a 27.2% increase in average distance with obstacles and a 42.5% decrease in collision number compared with DDPG.

  • Problem

    Simulator-to-robot RL transfer is commonly sequential and time-consuming, and simulator experience is unavailable to final robot models after transfer and fine-tuning.

    Source: §I · p. 1, A. Related Work · p. 1

  • Method

    FTRL combines federated learning and transfer learning to let agents in different environments learn asynchronously while transferring RL knowledge online.

    Source: §I · p. 1

  • Results

    Compared with DDPG, FTRL-DDPG-SIM increases average distance with obstacles by 27.2% and decreases collision number by 42.5% on test experimental race tasks.

    Source: B. Comparison Results · p. 6

  • Takeaways & Limitations

    The authors conclude that online simulator-to-car knowledge transfer lets FTRL-DDPG-SIM outperform single-agent RL and federation with identical RL agents in training speed and performance.

    Source: B. Comparison Results · p. 6

Abstract

from arXiv · show

Reinforcement learning (RL) is widely used in autonomous driving tasks and training RL models typically involves in a multi-step process: pre-training RL models on simulators, uploading the pre-trained model to real-life robots, and fine-tuning the weight parameters on robot vehicles. This sequential process is extremely time-consuming and more importantly, knowledge from the fine-tuned model stays local and can not be re-used or leveraged collaboratively. To tackle this problem, we present an online federated RL transfer process for real-time knowledge extraction where all the participant agents make corresponding actions with the knowledge learned by others, even when they are acting in very different environments. To validate the effectiveness of the proposed approach, we constructed a real-life collision avoidance system with Microsoft Airsim simulator and NVIDIA JetsonTX2 car agents, which cooperatively learn from scratch to avoid collisions in indoor environment with obstacle objects. We demonstrate that with the proposed framework, the simulator car agents can transfer knowledge to the RC cars in real-time, with 27% increase in the average distance with obstacles and 42% decrease in the collision counts.

I. INTRODUCTION

Prior distributed RL and simulator-to-robot transfer approaches leave gaps in asynchronous learning and continued knowledge exchange across environments. The paper proposes an asynchronous federated transfer process and motivates it against sequential transfer that leaves simulator knowledge unavailable during robot fine-tuning.

I. INTRODUCTION

  • Existing distributed RL frameworks are described as synchronous and assuming a constant environment, while offline model transfer can be time-consuming and lacks feedback from fine-tuned models in different real-life scenarios..Source: §I · p. 1
  • The proposed process combines federated and transfer learning to support asynchronous learning by agents in different environments.Source: §I · p. 1
  • The framework aims to keep training data local on edge vehicles while enabling companies to pool robotic-car resources for RL training.Source: §I · p. 1

A. Related Work

  • Earlier work investigated simulator-to-robot transfer for collision avoidance, interaction anticipation, and nonprehensile rearrangement,,.Source: A. Related Work · p. 1
  • The usual simulator-to-robot path pre-trains, transfers, and fine-tunes sequentially, with no meaningful simulator experience available to the final robot models.The paper asks whether transfer and fine-tuning can instead run in parallel.Source: A. Related Work · p. 1

B. Our Proposal

FTRL combines federated learning with online transfer of RL-agent knowledge. It supports asynchronous learning across different environments, and its collision-avoidance validation is reported to improve training speed and evaluation performance.

B. Our Proposal

  • FTRL transfers RL-agent knowledge in real time on a federated-learning foundation.Source: B. Our Proposal · p. 2
  • The framework supports source and target RL tasks in simulator or real-life environments with non-identical robots, obstacles, sensors, and control systems.Source: B. Our Proposal · p. 2
  • Federated learning provides a knowledge-aggregation process described as operating in nearly real time.Source: B. Our Proposal · p. 2
  • Collision-avoidance experiments on JetsonTX2 RC cars and Airsim simulators report better training speed and evaluation performance for FTRL.Source: B. Our Proposal · p. 2

II. HARDWARE PLATFORM AND TASKS

The validation setup combines JetsonTX2 RC cars, Microsoft Airsim, and a PC used for pre-training and federated learning. Indoor collision-avoidance tasks use a race layout that varies between runs but remains fixed within each run.

II. HARDWARE PLATFORM AND TASKS

  • The validation system uses three JetsonTX2 RC cars, Microsoft Airsim, and a PC server for pre-training and federated learning.Source: §II · p. 2
  • Each RC car carries a JetsonTX2 computer, LIDAR sensor, battery, USB hub, and onboard Wi-Fi module.Source: §II · p. 2
  • The PC server has an 8-core, 32G Intel i9-9820X CPU and four NVIDIA 2080 Ti GPUs.Source: §II · p. 2
  • Airsim's coastline map is used for pre-training and federation in the simulator.Source: §II · p. 2
  • A fence-like indoor race varies in overall shape and sometimes obstacle placement, while remaining unchanged during each specific RL-task run.Source: §II · p. 2

III. PROPOSED FRAMEWORK

The framework is presented as independent of any specific RL method, with DDPG selected for its description and validation.

III. PROPOSED FRAMEWORK

  • FTRL is not designed for a specific RL method.Source: §III · p. 2
  • The paper uses DDPG to describe the framework and validate its effectiveness.Source: §III · p. 2

A. DDPG Algorithm

The section sets out a discrete-time RL setting and describes DDPG as an actor-critic method with separate actor and critic networks.

A. DDPG Algorithm

  • The RL setting has agents interact with stochastic environments in discrete time, receiving rewards after observations and actions.The observations and actions are real-valued.Source: A. DDPG Algorithm · p. 2
  • A deterministic policy maps each observation to an action, while the state-action value function gives expected return after taking an action and then following that policy.The value function uses a discount factor γ.Source: A. DDPG Algorithm · p. 3
  • DDPG uses an actor network and a critic network, updating the critic from temporal-difference error and the actor with a deterministic policy gradient.The policy gradient is attributed to Silver et al..Source: A. DDPG Algorithm · p. 3

B. Steering Control RL Settings

The collision-avoidance task uses steering control driven by LIDAR observations and a reward based on the next observation, penalizing collisions and nearby obstacles.

B. Steering Control RL Settings

  • The task is steering control for autonomous agents, with LIDAR distance data as the observations.The reward is conditioned on the next observation rather than the current observation and action.Source: B. Steering Control RL Settings · p. 3
  • The reward is designed to favor policies with no collisions and larger values for the smallest fraction of obstacle distances.The reward's distance component uses the ascending sequence of next-state distances.Source: B. Steering Control RL Settings · p. 3
  • The reward uses next-step LIDAR distances, penalizing a detected collision and applying an exponential penalty to the average of the smallest fraction of distances.The collision condition is a minimum distance below the safe-distance threshold; the distance term encourages larger clearances.Source: B. Steering Control RL Settings · p. 3

C. FTRL Framework

FTRL combines local RL agents in different environments with federated model aggregation and online transfer between non-identical environments. Agents and server update asynchronously.

C. FTRL Framework

  • The framework comprises online transfer between environments, standard RL-agent training and inference, and FedAvg aggregation of distributed agents' knowledge.The server's federation cycle controls the regular collection and aggregation of models.Source: C. FTRL Framework · p. 4
  • The server regularly collects agent models and forms a federation model by element-wise arithmetic averaging of their parameters.The agents share an identical model structure, enabling FedAvg aggregation.Source: C. FTRL Framework · pp. 3–4
  • FTRL agents transfer observations and actions to a standard environment scale so models can be shared across environments with different scales.The observation transfer uses a scale-ratio parameter; steering actions are standardized and mapped to each car's steering range.Source: C. FTRL Framework · p. 4
  • Each agent transfers its observation when needed, updates from the server asynchronously, and trains its local RL model with DDPG.The agent procedure uses a synchronization interval to control when it retrieves the federation model.Source: C. FTRL Framework · p. 4
  • Local training and server aggregation run asynchronously, so an agent's intervening local updates may be superseded by a later federation-model update.The paper notes that local training after one federation snapshot can have no impact when the agent next synchronizes to a newer snapshot.Source: C. FTRL Framework · p. 4

IV. EXPERIMENTS

The experiments use RC cars and Airsim to assess online transfer from simulators to real-life environments and compare FTRL with a single DDPG run.

IV. EXPERIMENTS

  • Real-life RC-car and Airsim experiments are set up to assess simulator-to-real-world online knowledge transfer.The section also frames training speed and performance versus a single DDPG run as evaluation goals.Source: §IV · p. 5
  • One stated evaluation goal is whether FTRL transfers knowledge online from simulators to real-life environments.The supplied passage states this as an experimental aim, not an outcome.Source: §IV · p. 5
  • A second evaluation goal is whether FTRL improves training speed and performance compared with a single DDPG run.The supplied passage does not report the comparison result.Source: §IV · p. 5

A. Application Details

This section specifies the DDPG, Airsim, RC-car, and reward settings used in the experiments.

  • The DDPG actor and critic each use three 128-unit fully connected layers, with γ=0.99, τ=0.02, learning rates of 1e-4, a 2,500-entry buffer, and batch size 32.The actor has a continuous output layer and the critic has a state-action output layer.Source: A. Application Details · p. 5
  • Airsim uses its coastline map and a front-view LIDAR field of view from −90° to 90°, divided into 60 dimensions.The sensor setting was chosen to maintain transferability to the RC cars.Source: A. Application Details · p. 5
  • RC-car LIDAR is collected at 40 Hz; agents make decisions every 0.25 seconds, with 2-minute federation and 3-minute local synchronization cycles.All RC cars use β_i=6.67.Source: A. Application Details · p. 5
  • The reward uses a base value of 8, collision penalty of 60, minimum safe distance of 1.1, and exponential distance penalty value of 7.Source: A. Application Details · p. 5

B. Comparison Results

In the test race, FTRL-DDPG-SIM outperformed DDPG on average distance and collision count, while the relative-performance comparisons also showed gains from federated training. The comparisons use trained models in a race with more obstacles and tighter spacing than the training environments.

  • Stage I spans 625–1,250 seconds and stage II spans 1,250–1,875 seconds; the first 2,500-step stage is excluded because it uses only pre-trained-model inference before the buffer fills.Source: B. Comparison Results · p. 5
  • Relative performance compares corresponding rewards across stages I and II, with values bounded from −1 to 1 and positive values indicating better stage-II performance.The metric is used because reward scales may differ across cars.Source: B. Comparison Results · p. 5
  • Most cumulative relative-performance values for single-car DDPG were below zero, so 2,500 training steps did not guarantee local-agent improvement.Source: B. Comparison Results · p. 5
  • With three-car FTRL-DDPG, most cumulative relative-performance values were positive for cars 1 and 3, while car 2 improved after the first 1,500 time steps.Source: B. Comparison Results · p. 5
  • 27.2% increase in average distance with obstacles and 42.5% decrease in collision number for FTRL-DDPG-SIM versus DDPG; car-only FTRL-DDPG improved those averages by 20.3% and 30.7%.The test race used 50 cycles per car; Table I reports average distance and collision number.Source: Table I · p. 6, B. Comparison Results · p. 6
  • FTRL-DDPG-SIM achieved greater cumulative relative performance than car-only FTRL-DDPG on most recorded time steps.The authors attribute the faster training to online transfer of knowledge from the Airsim simulator.Source: B. Comparison Results · p. 6, Figure 3 · p. 6
  • The trained-model race used more obstacles and tighter distances than the training environments, and compared DDPG, FTRL-DDPG, and FTRL-DDPG-SIM.Average LIDAR distance and collision counts were recorded for each car.Source: B. Comparison Results · p. 6, Figure 4 · p. 6
  • The paper concludes that online simulator-to-car transfer performed better than single-agent execution and federation of identical RL agents in training speed and performance.Source: B. Comparison Results · p. 6

V. CONCLUSIONS AND FUTURE WORK

FTRL enables online transfer of knowledge across different RL tasks in non-identical environments, using a simple human-knowledge-based transfer model. Online transfer of already learned knowledge to new tasks remains a research frontier.

  • FTRL conducts online transfer of knowledge across different RL tasks in non-identical environments.Source: §V · p. 6
  • The transfer model used in FTRL is simple and based on human knowledge.Source: §V · p. 6
  • Online transfer of knowledge from already learned tasks to new ones remains a research frontier.Source: §V · p. 7
Loading 1910.06001v1…