Source-linked AI summary
Memory-based control with recurrent neural networks
Nicolas Heess, Jonathan J Hunt, Timothy P Lillicrap, David Silver
TL;DR
Partially observed control requires policies to retain useful information from observation histories, but model-free methods commonly assume fully observed states. The paper extends deterministic and stochastic value-gradient algorithms with recurrent networks trained by backpropagation through time. Recurrent agents solve diverse memory, exploration, and pixel-control tasks, while deterministic and stochastic policies perform similarly in the tested setup.
Problem
Model-free continuous-control methods generally assume fully observed states, despite real tasks requiring memory of noisy, temporary, hidden, or visually incomplete information.
Method
The paper extends DPG and SVG(0) to partially observed domains by replacing feedforward networks with recurrent networks trained using backpropagation through time.
Results
Recurrent DPG and RSVG(0) solve partially observed physical-control tasks spanning short-term integration, system identification, long-term memory, water-maze exploration, and pixel observations.
Takeaways & Limitations
Bootstrapped actor-critic algorithms can be viable for partially observed control, with little performance difference between stochastic and deterministic policies in the tested setup.
Takeaways & Limitations
The methods may suffer bias from off-policy state distributions, potentially more severely because policies and value functions depend on entire action-observation histories.
Abstract
from arXiv · showhide
Partially observed control problems are a challenging aspect of reinforcement learning. We extend two related, model-free algorithms for continuous control -- deterministic policy gradient and stochastic value gradient -- to solve partially observed domains using recurrent neural networks trained with backpropagation through time. We demonstrate that this approach, coupled with long-short term memory is able to solve a variety of physical control problems exhibiting an assortment of memory requirements. These include the short-term integration of information from noisy sensors and the identification of system parameters, as well as long-term memory problems that require preserving information over many time steps. We also demonstrate success on a combined exploration and memory problem in the form of a simplified version of the well-known Morris water maze task. Finally, we show that our approach can deal with high-dimensional observations by learning directly from pixels. We find that recurrent deterministic and stochastic policies are able to learn similarly good solutions to these tasks, including the water maze where the agent must learn effective search strategies.
1 Introduction
Partially observed control is difficult because relevant state information may be noisy, temporary, hidden, or visually unavailable, while existing model-free methods often assume full observability. The paper extends deterministic and stochastic policy-gradient methods with recurrent networks and demonstrates them across diverse memory-dependent control tasks.
- Motivation: Partial observability arises from temporary information, noisy sensors, unobserved system variation, state aliasing, and limitations of visual observations.Examples include remembering a navigation sign, identifying plant parameters, estimating velocity from images, and handling occlusions or restricted fields of view.
- Existing approaches: Belief-state methods require an environment model and typically incur substantial computational cost during belief updates.
- Existing approaches: Model-free memory learning is challenging because relevant features are unknown and associations may need to span many time steps.Consequently, most model-free approaches assume full observability or use hand-crafted multiple-frame inputs.
- Approach: The paper extends deterministic policy gradients and stochastic value gradients to partially observed continuous-action control.Both methods use action-value gradients from an actor-critic framework; the deterministic method updates a deterministic policy, while its counterpart learns a stochastic policy.
- Contributions: Recurrent DPG and recurrent SVG(0) use recurrent networks trained with backpropagation through time for tasks requiring short-term integration, system identification, long-term memory, exploration, and pixel-based control.Demonstrations include velocity-free pendulum and cartpole swing-up, variable pole length, a returning robot arm, a simplified water maze, and direct control from pixels.
- Findings: The results suggest bootstrapped actor-critic methods can learn control policies under partial observability, with little observed difference between deterministic and stochastic policies in the tested setup.The comparison includes the water maze, where agents learn search strategies.
2 Background
The paper formulates partial observability as a POMDP in which the agent must act from observation-action history rather than directly observed state. Because full histories are generally intractable, the central background challenge is scalable history summarization.
- POMDP formulation: A POMDP is defined by states, actions, an initial-state distribution, transition dynamics, rewards, and observations conditioned on underlying states.
- POMDP formulation: Under partial observability, the agent receives observations instead of directly observing the underlying state.
- History dependence: An optimal agent may require the entire preceding observation-action history to choose actions.The history contains observations and prior actions up to the current observation.
- Objective: The policy maps history to a distribution over actions and seeks to maximize expected discounted reward.The framework considers both stochastic and deterministic policies.
- Trajectory distribution: Trajectories combine states, observations, actions, transitions, and policy-conditioned action selection over time.For deterministic policies, stochastic action sampling is replaced by a deterministic function mapping history to actions.
- Action values: The action-value function is defined using history because the agent lacks direct access to state, with expectations over the inferred current state and future trajectory.The text notes that this is equivalent to conditioning on the belief state because history is a sufficient statistic.
- Core challenge: Conditioning directly on the full observation sequence is generally intractable, making scalable summarization of the past a central challenge.
3 Algorithms
The paper extends deterministic and stochastic policy-gradient methods to partially observed control by replacing feedforward function approximators with recurrent networks trained through trajectories. The algorithms use recurrent histories, learned critics, replay, target networks, and backpropagation through time.
- Recurrent DPG: RDPG extends deterministic policy gradients to partially observed domains by updating the actor through the gradient of a learned differentiable critic.The exact action-value function is replaced by an approximate recurrent critic Qω that can be learned with Q-learning.
- Recurrent DPG: Recurrent networks let the policy and action-value function preserve limited information from the observation-action history needed to solve partially observed problems.The method replaces feedforward networks with recurrent networks and conditions µ and Q on history rather than directly on state.
- Recurrent SVG(0): SVG(0) applies the analogous backpropagation update to stochastic policies by reparameterizing actions as a deterministic function of fixed independent noise.The stochastic policy is written as a = πθ(h, ν), with ν drawn independently from a fixed distribution at each time step.
- Off-policy learning and stabilization: The algorithms store experienced trajectories in a replay buffer and use recurrent target networks to support off-policy learning and improve stability.RDPG and RSVG(0) may have greater state-distribution bias because policies and critics depend on entire action-observation histories.
- Off-policy learning and stabilization: Off-policy learning with experience replay remained effective in the partially observed case, although on-policy learning with SVG(0) was not explored.SVG(0) potentially allows on-policy learning, but the paper reports results only for the off-policy approach used here.
- Optimization procedure: Actor and critic updates are computed with backpropagation through time, followed by Adam updates and delayed target-network updates.Algorithm 1 initializes recurrent actor, critic, and target networks, samples episode minibatches, computes recurrent target values, and updates both networks.
4 Results
The recurrent algorithms were evaluated on partially observed physical-control tasks requiring sensor integration, system identification, explicit memory, exploration, and pixel-based control. RDPG with LSTM networks solved diverse memory tasks, while recurrent methods also succeeded in the water maze and high-dimensional observations.
- Sensor integration and system identification: Both recurrent algorithms learned pendulum swing-up policies without direct velocity observations.
- Sensor integration and system identification: RDPG with LSTM reliably solved variable-length cartpole swing-up requiring velocity estimation and hidden pole-length identification, whereas feedforward DDPG failed entirely.A simple recurrent network performed considerably worse, plausibly because the episodes lasted T=350 steps and gradients were difficult to propagate.
- Memory tasks: RDPG with LSTMs reliably solved both the hidden-target reacher and return-to-start gripper tasks, while feedforward agents failed on their memory components.The reacher supplied the target only initially, and the gripper required returning the payload to the hand’s initial position.
- Water maze: Both recurrent algorithms learned good water-maze solutions, and recurrent agents returned to the platform more quickly after resets by recalling its position.The non-recurrent DDPG agent learned only a limited search strategy and failed to exploit memory after random resets.
- High-dimensional observations: RDPG learned effective policies directly from high-dimensional renderings, integrating multiple timesteps for velocity estimation and remembering a disappearing visual target.The evaluated tasks were pendulum control and two-choice reaching with a target that disappeared after five frames.
5 Discussion
The discussion situates recurrent continuous-control methods alongside prior model-free recurrent approaches for partially observed problems. It also highlights architectural choices involving memory, action stochasticity, and actor–critic parameter sharing.
- Variants: The experiments keep actor and critic networks entirely disjoint, leaving shared convolutional or recurrent layers for future investigation.Sharing could improve computational efficiency and learning speed but may also cause instability when updates to one network damage the other.
- Related work: Prior model-free recurrent methods addressed discrete-action POMDPs, while this work targets continuous-action control with recurrent policy-gradient algorithms.
- Related work: Most prior approaches learn memory inside the policy with BPTT, whereas guided policy search treats memory as additional state dimensions and requires a well-defined future cost.
6 Conclusion
The paper concludes that model-free policy-gradient algorithms can be extended with recurrent neural networks to learn effectively across partially observed control problems, including pixel-based tasks. Standard BPTT also permits use of LSTM methods for challenging long-term memory problems such as the Morris water maze.
- Conclusion: Two related model-free approaches were extended with recurrent neural networks and learned effectively across partially observed problems, including directly from pixel observations.
- Conclusion: Standard backpropagation through time enables these algorithms to benefit from LSTM innovations for challenging memory problems such as the Morris water maze.
7 Supplementary
The supplementary algorithm describes recurrent stochastic value-gradient training with replay, recurrent target networks, and backpropagation through time for critic and actor updates.
- Algorithm 2 RSVG(0): RSVG(0) initializes recurrent critic and actor networks, corresponding target networks, and a replay buffer before training episodes.
- Algorithm 2 RSVG(0): At each timestep, the algorithm appends the observation and previous action to the history and selects an action from the recurrent stochastic policy.
- Algorithm 2 RSVG(0): The algorithm stores complete episodes, samples minibatches, and computes target values using recurrent target networks.
- Algorithm 2 RSVG(0): Critic and actor updates are computed with BPTT, followed by Adam optimization and target-network updates.