PaperlayerView exact revision on arXiv

Source-linked AI summary

Thinking While Moving: Deep Reinforcement Learning with Concurrent Control

Ted Xiao, Eric Jang, Dmitry Kalashnikov, Sergey Levine, Julian Ibarz, Karol Hausman, Alexander Herzog

arXiv:2004.06089v4cs.LGcs.AIcs.ROstat.ML

TL;DR

Concurrent control requires choosing an action while the system continues to evolve, unlike the frozen-state assumption common in DRL. The paper extends value-based learning with concurrent-action information and evaluates it in simulation and robotic grasping. In simulated grasping the concurrent models recover blocking-baseline success while acting 31.3% faster, and real-robot grasping achieves comparable success while acting 49% faster.

  • Problem

    Most DRL approaches are evaluated in blocking environments, where the state is assumed not to change between observation and action execution, unlike dynamic real-time settings.

    Source: §1 · p. 1

  • Method

    The paper derives concurrent Bellman formulations and extends value-based Q-learning with information such as previous action, action-selection time, or Vector-to-go.

    Source: §1 · p. 1, §3.6 · p. 6

  • Results

    Concurrent models recovered blocking-baseline success in simulated robotic grasping while acting 31.3% faster, and real-robot grasping achieved comparable success while acting 49% faster.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    Concurrent value-based control can match blocking-baseline grasp success while reducing execution time in the reported simulated and real-robot grasping tasks.

    Source: §1 · p. 2, §5 · p. 9

    The work focuses on model-free value-based methods, selected for their sample-efficiency and off-policy properties in real-world vision-based robotics.

    Source: §3 · p. 3

Abstract

from arXiv · show

We study reinforcement learning in settings where sampling an action from the policy must be done concurrently with the time evolution of the controlled system, such as when a robot must decide on the next action while still performing the previous action. Much like a person or an animal, the robot must think and move at the same time, deciding on its next action before the previous one has completed. In order to develop an algorithmic framework for such concurrent control problems, we start with a continuous-time formulation of the Bellman equations, and then discretize them in a way that is aware of system delays. We instantiate this new class of approximate dynamic programming methods via a simple architectural extension to existing value-based deep reinforcement learning algorithms. We evaluate our methods on simulated benchmark tasks and a large-scale robotic grasping task where the robot must "think while moving".

1 INTRODUCTION

Most deep reinforcement learning benchmarks assume the environment stays still while an agent computes an action, an assumption that breaks in dynamic real-time settings. The paper develops concurrent-control methods and reports comparable grasp success with faster execution.

1 INTRODUCTION

  • Blocking DRL freezes the environment during action computation, but real-time environments evolve before the computed action is executed.Catching a ball illustrates why such tasks cannot pause while an agent decides; concurrent action may also support smoother motion and planning during execution.Source: §1 · p. 1
  • The paper introduces a framework for concurrent environments and derives a modified Bellman operator for them.The introduction also identifies minimal added state-observation information for recovering blocking performance with Q-learning.Source: §1 · pp. 1–2
  • Real-world robotic grasping achieved comparable task success to a blocking baseline while acting 49% faster.The paper also introduces simulated concurrent-action environments and a minimal observation augmentation intended to recover blocking performance with Q-learning.Source: §1 · p. 2

2 RELATED WORK

Prior approaches either engineer systems toward blocking behavior or modify learning to address concurrency. The paper distinguishes its approach by directly handling concurrent execution in the learning algorithm.

Minimizing Concurrent Effects

  • Low-latency hardware, lightweight controllers, or completion-oriented feedback control can make robotics systems approximately blocking.The cited feedback-control approach decelerates system velocity to zero before recording a state.Source: §2 · p. 2

Algorithmic Approaches

  • Algorithmic approaches address latency through robustness, history, future-state prediction, delayed-MDP formulations, or temporally aware architectures.The paper contrasts its approach as optimizing for a specific latency regime, using latency-source properties, avoiding explicit high-dimensional forward models, and handling interrupted actions.Source: §2 · p. 2

Continuous-time Reinforcement Learning

  • Continuous-time control and reinforcement-learning formulations motivate concurrent DRL, but their application to image-based DRL problems remains untested in the cited discussion.The paper builds on these theoretical formulations to apply deep reinforcement learning to more complex vision-based robotics tasks.Source: §2 · p. 2

3 VALUE-BASED REINFORCEMENT LEARNING IN CONCURRENT ENVIRONMENTS

The paper formulates concurrent control as value-based reinforcement learning in which states evolve during observation and policy inference. It extends Bellman updates with concurrent-action information and shows the resulting operators retain contraction properties.

3 VALUE-BASED REINFORCEMENT LEARNING IN CONCURRENT ENVIRONMENTS

  • The method adds information about the previous action to a Q-function through small modifications to standard value-based algorithms.The paper focuses on model-free value-based methods because of their sample-efficiency and off-policy properties for real-world vision-based robotics.Source: §3 · p. 3

3.1 CONCURRENT ACTION ENVIRONMENTS

  • In a concurrent environment, the state can evolve during state capture and policy inference rather than remaining fixed as in a blocking environment.The figure contrasts a frozen state during observation and action selection with continuous-time state and action dynamics.Source: §3.1 · p. 3, Figure 1 · p. 4

3.2 DISCRETE-TIME REINFORCEMENT LEARNING PRELIMINARIES

  • The discrete-time preliminaries define state, action, policy, transitions, reward, and discounted return for standard reinforcement learning.The blocking formulation assumes the environment state does not change between observation and action execution.Source: §3.2 · p. 3

3.3 VALUE FUNCTIONS AND POLICIES IN CONTINUOUS TIME

  • The continuous-time formulation describes stochastic environment dynamics and defines continuous-time value and Q-functions.H is the constant interval between state captures and the duration of an action trajectory; a denotes the continuous action function applied over that interval.Source: §3.3 · pp. 3–4

3.4 CONCURRENT ACTION MARKOV DECISION PROCESSES

  • Concurrent MDPs capture state while the previous action is still executing, then begin the newly selected action without waiting for completion.The paper describes this in continuous time as translating the action along the time dimension; experiments use discrete-time DRL implementations.Source: §3.4 · p. 4

3.5 VALUE-BASED CONCURRENT REINFORCEMENT LEARNING ALGORITHMS IN CONTINUOUS AND DISCRETE-TIME

  • The continuous-time concurrent Q-function accounts for returns from the previous and new action trajectories, and the concurrent Bellman operator remains a contraction.The discrete-time operator is also a contraction; its Q-function can be extended with action-selection delay and the previous action in addition to standard inputs.Source: §3.5 · pp. 5–6

3.6 DEEP Q-LEARNING WITH CONCURRENT KNOWLEDGE

  • The method conditions the Q-function on previous action, action-selection time, or Vector-to-go, the remaining action when the state is measured.The analysis assumes these quantities are obtainable and holds the sampling period H constant; accurately predicting action-selection time can be difficult on complex robotics systems.Source: §3.6 · p. 6

4 EXPERIMENTS

The experiments test concurrent knowledge representations in toy control tasks and robotic grasping, comparing them with unconditioned blocking baselines. VTG is robust across toy-task hyperparameters, while robotic models recover blocking performance and the real-robot concurrent policy is faster at comparable grasp success.

4.1 TOY FIRST-ORDER CONTROL PROBLEMS

  • The toy-task ablation varied action-execution and action-selection delays alongside network architecture, concurrent-knowledge features, frame-stacking, and discretized actions.The evaluation used DQN and recurrent DQN on concurrent 3D MuJoCo Cartpole and Pendulum tasks.Source: §4.1 · p. 6, §4 · p. 6
  • VTG was the clearest source of improved Cartpole performance and the majority of Pendulum gains, with greater robustness across hyperparameters.Frame-stacking helped nominally on Pendulum; Figure 2 sorts reward outcomes across hyperparameter sweeps, with larger area-under-curve indicating greater robustness.Source: §4.1 · pp. 6–7, Figure 2 · p. 7

4.2 CONCURRENT QT-OPT ON LARGE-SCALE ROBOTIC GRASPING

  • The simulated grasping task used a 7 DoF arm, camera images, continuous gripper displacements, and procedurally generated objects with binary lift success.Blocking actions finish before the next observation; concurrent actions proceed without waiting, so the next state is observed while the robot is moving.Source: §4.2 · pp. 7–8
  • 31.3% shorter episode duration was achieved by simulated concurrent-knowledge models versus blocking unconditioned models, while recovering baseline task performance.The duration comparison also held against blocking models using a shaped timestep penalty rewarding faster policies.Source: §4.2 · p. 8, Table 1 · p. 8
  • 49% faster policy duration with comparable grasp success was reported for the real-robot concurrent model versus the blocking model.Policy duration excludes infrastructure setup and teardown.Source: §4.2 · p. 8, Table 2 · p. 8

5 DISCUSSION AND FUTURE WORK

The discussion presents a continuous-time framework for concurrent control and identifies required concurrent information for Q-learning. Experiments support VTG's robustness and faster grasping policies, while broader algorithms and dynamic environments remain future directions.

5 DISCUSSION AND FUTURE WORK

  • The framework preserves contraction and Q-learning convergence guarantees when the Q-function uses action-selection delay and previous-action information; VTG represents the remaining previous action.The discussion identifies delay and previous action, or their representations, as the theoretical requirements for good learning performance.Source: §5 · p. 9
  • Experiments support VTG's low hyperparameter sensitivity and blocking-level performance in concurrent settings, including robotic grasping policies reported as 31.3% and 49% faster.The simulated grasping result concerns concurrent models versus blocking execution; the real-world result reports comparable grasp success to a blocking baseline.Source: §5 · p. 9
  • Future work includes testing off-policy data across latency regimes, value-based alternatives such as policy gradients, and control in dynamic environments.The paper presents these as possible extensions rather than evaluated capabilities.Source: §5 · p. 9

A.1 DEFINING BLOCKING BELLMAN OPERATORS

The section defines continuous- and discrete-time Q-function formulations for concurrent actions. In continuous time, the delayed portion of the update corresponds to an expected future Q-function.

A.1 DEFINING BLOCKING BELLMAN OPERATORS

  • The continuous-time estimator’s delayed term is itself a Q-function at time t + tAS, so the update takes an expectation over unknown future state, action, and reward values.The resulting Q-function represents an expected future Q-function, not just an expected sum of discounted future rewards.Source: §A.1 · p. 12
  • The discrete-time concurrent Q-function is parameterized and rewritten using the concurrent-action formulation.The passage provides the parameterization and subsequent rewritten Q-function, but does not supply enough text to recover their full form.Source: §A.1 · p. 12

A.2 CONTRACTION PROOFS FOR THE BLOCKING BELLMAN OPERATORS

The proofs establish contraction properties for the traditional Bellman operator and the updated operators used for concurrent settings.

A.2 CONTRACTION PROOFS FOR THE BLOCKING BELLMAN OPERATORS

  • The traditional Bellman operator is a contraction, with its update difference bounded by a factor c between 0 and 1 times the Q-function difference.The operator is defined as reward plus discounted expected maximum next-state action value.Source: §A.2 · p. 12
  • The updated Bellman operators introduced for the concurrent setting are also contractions.The appendix states that the discrete-time proof extends to the continuous-time operator.Source: §A.2 · p. 13

A.3 CONCURRENT KNOWLEDGE REPRESENTATION

The section describes three concurrent-knowledge representations—previous action, action-selection time, and vector-to-go—and explains what information they encode and require.

A.3 CONCURRENT KNOWLEDGE REPRESENTATION

  • Previous action records the action executed at the prior timestep, while action-selection time tAS measures how long action selection takes.In the experiments, tAS is normalized using known bounds on latency.Source: §A.3 · p. 13
  • Vector-to-go encodes how much of the previous action remains when the current state is observed, combining the previous action with the current state.For first-order control, it can be computed from synchronized proprioceptive state and provides information also represented by previous action and current state.Source: §A.3 · pp. 13, 15
  • Using tAS relies on knowledge of system concurrency, whereas computing vector-to-go requires a measure of action completion synchronized with state observation.The figure contrasts blocking stages, assumed instantaneous, with concurrent state capture and policy inference proceeding during action execution.Source: §A.3 · pp. 13, 15, Figure 4 · p. 14

A.4.1 CARTPOLE AND PENDULUM ABLATION STUDIES

The appendix gives implementation details for simulated control and robotic grasping experiments, then shows reward sweeps for concurrent Cartpole and Pendulum settings.

A.4.1 CARTPOLE AND PENDULUM ABLATION STUDIES

  • Cartpole-Swingup and Pendulum-Swingup use discretized joint-position actions, ground-truth position and velocity observations, and simulated action-selection delays.Execution and selection-step latency settings are sampled from specified millisecond values; frame-stacking and concurrent-knowledge settings are varied in the sweeps.Source: §A.4.1 · p. 15

A.4.2 LARGE SCALE ROBOTIC GRASPING

  • The grasping setup uses a 7-DoF arm and image observations; its QT-Opt Q-network is augmented with vector-to-go and/or previous-action features.In simulated concurrent mode, the next state is observed while the robot remains in motion; the physical setup uses common household objects.Source: §A.4.2 · pp. 15–16

A.5 FIGURES

  • On concurrent Cartpole, critic input with vector-to-go yields more robust performance across hyperparameters; the Pendulum figure displays reward sweeps over frame-stacking and concurrent-knowledge settings.Both figures sort results by reward across hyperparameter combinations; the Pendulum caption states no winning configuration.Source: Figure 6 · p. 17, Figure 7 · p. 17

Related papers

Loading 2004.06089v4…