PaperlayerView exact revision on arXiv

Source-linked AI summary

Memory-based Deep Reinforcement Learning for POMDPs

Lingheng Meng, Rob Gorbet, Dana Kulić

arXiv:2102.12344v5cs.LGcs.AIcs.RO

TL;DR

Real-world robotics can involve partial observability, while many DRL approaches assume fully observable MDPs. The paper introduces LSTM-TD3, a memory-based DRL algorithm, and reports improved POMDP performance while retaining state-of-the-art MDP performance.

  • Problem

    Many DRL approaches assume fully observable MDPs, although robotics may involve partial observations from sensing limits or incomplete observation designs.

    Source: §I · p. 1

  • Method

    LSTM-TD3 adds recurrent memory to TD3, using history in a recurrent actor-critic for continuous control.

    Source: §I · p. 1, A. Recurrent Actor-Critic Framework · p. 3

  • Results

    LSTM-TD3 significantly outperforms baselines on POMDPs while retaining state-of-the-art performance on MDPs.

    Source: §VIII · p. 7

  • Takeaways & Limitations

    The ablation study reports that DC and TPS help stabilize learning, CFE is especially important for MDP performance, and PA benefits tasks where past actions inform the current state.

    Source: §VIII · p. 7

    History length is fixed per run; length 5 performs well on the devised POMDPs, but may not suit tasks where state depends on less recent memory, and longer histories increase computation.

    Source: §VIII · p. 8

Abstract

from arXiv · show

A promising characteristic of Deep Reinforcement Learning (DRL) is its capability to learn optimal policy in an end-to-end manner without relying on feature engineering. However, most approaches assume a fully observable state space, i.e. fully observable Markov Decision Processes (MDPs). In real-world robotics, this assumption is unpractical, because of issues such as sensor sensitivity limitations and sensor noise, and the lack of knowledge about whether the observation design is complete or not. These scenarios lead to Partially Observable MDPs (POMDPs). In this paper, we propose Long-Short-Term-Memory-based Twin Delayed Deep Deterministic Policy Gradient (LSTM-TD3) by introducing a memory component to TD3, and compare its performance with other DRL algorithms in both MDPs and POMDPs. Our results demonstrate the significant advantages of the memory component in addressing POMDPs, including the ability to handle missing and noisy observation data.

I. INTRODUCTION

DRL commonly assumes fully observable MDPs, while robotics often involves partial observability. The paper proposes LSTM-TD3 for continuous control and compares it with other algorithms on MDP and POMDP tasks.

I. INTRODUCTION

  • LSTM-TD3 significantly improves performance over other DRL algorithms on POMDPs with reduced observability, according to the paper’s comparison.The study also examines MDP versions of tasks and includes an ablation study.Source: §I · p. 1

II. RELATED WORK

  • Earlier recurrent DRL addressed POMDPs, but cited methods [24],, [25] used discrete action spaces rather than continuous control.Prior continuous-control studies also examined limited task or action-space settings, [27],.Source: §I · p. 1, §II · p. 2

A. Decision Process

  • An MDP assumes a fully observable environment, whereas a POMDP adds observations and an observation model without assuming the underlying state is fully observable.At each step, the POMDP agent receives an observation associated with the next state.Source: A. Decision Process · p. 2

B. Reinforcement Learning (RL)

The section introduces reinforcement learning and the LSTM and TD3 components, then describes LSTM-TD3’s use of history in a recurrent actor-critic. Its history consists of recent observation-action pairs.

B. Reinforcement Learning (RL)

  • Reinforcement learning learns policies through trial-and-error interaction, while deep reinforcement learning uses neural networks for end-to-end policy and value learning.The passage defines state- and action-value functions as expected returns under a policy.Source: B. Reinforcement Learning (RL) · p. 2

C. Long-Short-Term-Memory (LSTM)

D. Twin Delayed Deep Deterministic Policy Gradient (TD3)

  • TD3 addresses actor-critic overestimation with two critics and bootstraps current Q-values using the lower predicted future return.The passage describes TD3 as a variant of DDPG.Source: D. Twin Delayed Deep Deterministic… · p. 2

IV. PROPOSED APPROACH

  • LSTM-TD3 uses recurrent networks in both actor and critic, with history formed from the preceding l observation-action pairs.When history is unavailable, zero-valued dummy observation and action vectors are used as memory input.Source: §IV · pp. 2–3

A. Recurrent Actor-Critic Framework

The recurrent actor-critic framework uses LSTM to extract information from history for both actor and critic. Each combines history-derived memory with features from the current input.

A. Recurrent Actor-Critic Framework

  • LSTM extracts information from past history for the actor and critic, and the framework can handle history of any length.The passage describes the framework as recurrent actor-critic.Source: A. Recurrent Actor-Critic Framework · p. 3
  • The critic combines memory extracted from history, current features based on observation and action, and a perception-integration component.The current critic feature uses the current observation and action.Source: A. Recurrent Actor-Critic Framework · p. 3
  • The actor likewise combines history-derived memory and current-observation features through memory-extraction, current-feature, and perception-integration components.The actor’s current feature is based on the current observation.Source: A. Recurrent Actor-Critic Framework · p. 3

B. Optimization of the Recurrent Actor-Critic

The recurrent actor-critic follows TD3 optimization: critics fit estimated target values, and the actor is optimized against a critic while that critic’s parameters are fixed.

B. Optimization of the Recurrent Actor-Critic

  • Each critic is optimized to minimize mean-squared error between its predicted Q-value and the estimated target.The critic parameters are updated for this objective.Source: B. Optimization of the Recurrent… · p. 3
  • The actor’s parameters are optimized to maximize an approximated Q-value while the selected critic’s parameters remain fixed.Either Q1 or Q2 may provide the actor’s Q-value objective.Source: Figure 2 · p. 3, B. Optimization of the Recurrent… · p. 3
  • The target Q-value uses target actor and critic networks, clipped target-action noise, and the minimum of the two estimated target-critic values.The target bootstraps from the next observation and its preceding history.Source: B. Optimization of the Recurrent… · p. 3

V. EXPERIMENT SETTINGS

The experiments compare continuous-control tasks under fully observable MDP settings and four POMDP variants, using standard DRL baselines and LSTM-TD3. Baseline results average 10 evaluation episodes across four random seeds.

V. EXPERIMENT SETTINGS

  • Each PyBullet-Gym task has an original fully observable MDP version and four POMDP versions modeling potential causes of partial observability.The variants include removing velocity observations, flickering observations, random sensor loss, and sensor noise.Source: §V · pp. 3–4
  • Fig. 3 identifies the five PyBulletGym tasks used in the experiments.They are HalfCheetah, Ant, Walker2D, Hopper, and InvertedDoublePendulum.Source: §V · p. 3, Figure 3 · p. 4
  • The comparison includes DDPG, SAC, TD3, TD3 with an observation window, and TD3 with an observation window plus past actions.The observation-window baselines concatenate observations within a history window; the second also concatenates past actions.Source: §V · p. 4
  • Baseline hyperparameters use OpenAI Spinning Up defaults; LSTM-TD3 uses TD3 hyperparameters and a network sized similarly to TD3.Reported results average 10 evaluation episodes over four random seeds.Source: §V · p. 4

A. Performance Comparison

LSTM-TD3 is competitive with baselines on MDPs and generally performs better on POMDPs, with a notable exception on velocity-removal tasks. Its clearest advantage appears when observations flicker, disappear, or contain noise.

A. Performance Comparison

  • Across the sampled tasks, LSTM-TD3 is competitive on MDPs and outperforms all baselines on POMDPs except HalfCheetah POMDP-RV, where LSTM-TD3(5) is slightly worse than TD3.Fig. 4 shows the learning curves; LSTM-TD3(0) is also reported as sensitive to random seeds.Source: A. Performance Comparison · p. 4, Figure 4 · p. 5
  • On POMDP-FLK, baselines basically fail while LSTM-TD3 reaches performance comparable to MDP, with similarly dramatic gains on POMDP-RN and POMDP-RSM.These settings represent flickering, noisy, and randomly missing sensor observations.Source: A. Performance Comparison · pp. 4–5
  • Memory does not always help on POMDP-RV: HalfCheetah and Ant show exceptions, while Hopper and InvertedDoublePendulum improve with LSTM-TD3 over TD3.The authors hypothesize that similar speeds across the history window can make velocity difficult to infer.Source: A. Performance Comparison · p. 5
  • Adding past actions to TD3-OW consistently harms performance relative to TD3-OW, including worse-than-TD3 performance on MDPs.The authors suggest that larger concatenated inputs and equal treatment of older and newer observations may make learning harder.Source: A. Performance Comparison · p. 5

B. Policy Generalization

The paper tests whether policies generalize across MDP and POMDP versions of AntPyBulletEnv-v0. LSTM-TD3 is the most robust of the compared policies when MDP-trained policies are evaluated under partial observability.

B. Policy Generalization

  • MDP-trained LSTM-TD3, TD3, and TD3-OW outperform DDPG, SAC, and TD3-OW-AddPastAct; under POMDP evaluation, LSTM-TD3 is more robust than TD3 and TD3-OW.Performance decreases for all three when evaluated on POMDPs.Source: B. Policy Generalization · p. 6
  • Across the last three cross-evaluation panels, LSTM-TD3 significantly outperforms TD3-OW on different environments, while each algorithm changes little in performance.When trained on POMDP-FLK and evaluated on MDP, LSTM-TD3 performs better than on POMDP-FLK.Source: B. Policy Generalization · p. 6

VII. ABLATION STUDY

The ablation study examines double critics, target policy smoothing, current feature extraction, and past actions in LSTM-TD3. Removing double critics or current feature extraction reduces performance in the reported settings.

VII. ABLATION STUDY

  • The ablation compares full LSTM-TD3 with variants removing current feature extraction, past actions, double critics, target policy smoothing, or both critics and smoothing.The learning curves average 10 evaluation episodes over 4 random seeds.Source: §VII · p. 6, Figure 7 · p. 7

A. Effect of Double Critics and Target Policy Smoothing

  • Removing double critics significantly reduces performance on all MDPs and most POMDPs; removing both double critics and target policy smoothing also causes a significant decrease.Target policy smoothing alone appears less important for the full model's best performance.Source: A. Effect of Double Critics and Target… · p. 6
  • Without double critics and target policy smoothing, LSTM-TD3 becomes LSTM-DDPG, which outperforms DDPG on every matched task and exceeds DDPG on MDP even on POMDP tasks.Table II compares DDPG and LSTM-DDPG by maximum average return.Source: A. Effect of Double Critics and Target… · p. 6, Table II · p. 7

B. Effect of Current Feature Extraction

C. Including Past Action Sequence in Memory

The ablation finds that past actions can benefit LSTM-TD3 when they inform the agent's current state. The discussion also identifies limits in interpreting extracted memory and in choosing history length across tasks.

C. Including Past Action Sequence in Memory

  • Removing past actions reduces LSTM-TD3 performance, notably on POMDP-RV InvertedDoublePendulum, unlike the harm from adding past actions to TD3-OW; this suggests greater robustness to whether past actions are useful.The authors consider this desirable when designers lack prior knowledge about whether past actions are needed to infer state.Source: C. Including Past Action Sequence in… · p. 7

VIII. DISCUSSION AND CONCLUSION

  • LSTM-TD3 significantly outperforms baselines on POMDPs while retaining state-of-the-art MDP performance; the ablation attributes benefits to double critics, target policy smoothing, current feature extraction, and informative past actions.Double critics and target policy smoothing help stabilize learning, current feature extraction particularly supports MDP performance, and past actions help when informative about the agent's state.Source: §VIII · p. 7
  • Without specific constraint terms in the cost functions, the extracted memory cannot be properly interpreted.The authors identify interpretation as a challenge and call for future work on it.Source: §VIII · pp. 7–8
  • History length is fixed as a hyperparameter for each run; length 5 may not work as well on tasks where the underlying state depends on less recent memory.Longer histories increase computation resources and time during both training and decision making.Source: §VIII · p. 8

I. Algorithms Implementation

The implementation describes separate memory and current-feature extraction before their integration. Reported comparisons cover benchmark performance, robustness to partial observability, and the effects and costs of history length.

I. Algorithms Implementation

  • The implementation separates the memory component from current feature extraction, then combines the extracted memory and current feature in the perception integration structure.The supplementary material describes the first network row as memory, the second as current-feature extraction, and the third as integration.Source: §I · p. 9

A. Performance Comparison

  • LSTM-TD3 either matches or outperforms other baselines on MDP tasks and significantly outperforms them on the POMDP versions of each task.Table II reports maximum average returns; its POMDP-FLK, POMDP-RN, and POMDP-RSM settings use pflk = 0.2, σrn = 0.1, and prsm = 0.1.Source: A. Performance Comparison · p. 9, Table II · p. 10

B. Robustness to Partial Observability

  • As observability decreases, LSTM-TD3 is generally more robust than TD3-OW, which is more robust than TD3; severe reductions still degrade LSTM-TD3 performance.The figure varies σrn and prsm, with higher values indicating lower observability.Source: B. Robustness to Partial Observability · p. 9, Figure 1 · p. 10

C. Effect of History Length

  • At pflk = 0.05, LSTM-TD3(3) and LSTM-TD3(5) similarly outperform LSTM-TD3(0) and LSTM-TD3(1); at pflk = 0.1, length 5 maintains performance while length 3 drops sharply.At pflk = 0.2, length 5 also degrades, and all tested history lengths fail at pflk = 0.5 and 0.8.Source: C. Effect of History Length · p. 10, Figure 2 · p. 10
  • Long history lengths consume much extra computation; consequently, LSTM-TD3 runs with history lengths 10, 50, and 100 were not fully run up to 1 million steps.Source: C. Effect of History Length · p. 11, Figure 3 · p. 11

D. Policy Generalization

This section tests history-length generalization and examines how actor and critic memories relate to return and predicted Q-values. Longer evaluation histories preserved performance, while shorter histories were inconsistent and memory interpretation remained challenging.

D. Policy Generalization

  • History-length mismatch is tested in both directions because shorter evaluation histories may cut inference time, while shorter training histories may save computation if longer evaluation restores performance.The authors also note that saving training computation may matter when onboard resources are limited.Source: D. Policy Generalization · pp. 11–12
  • Longer evaluation histories preserved performance, whereas shorter histories were inconsistent and sometimes caused dramatic drops.The longest training history tested was 5, limiting the scope of the generalization results.Source: 1) Evaluation with Different History… · p. 12, Figure 5 · p. 13

1) Evaluation with Different History Lengths:

  • After a few thousand steps, LSTM-TD3(0)'s actor memory stayed near 0.0034 ± 0.0024, consistent with histories mapping to a roughly fixed value.The authors note that near-zero memory does not establish that the policy ignores history.Source: 1) Evaluation with Different History… · p. 12

Predicted Q-value, and the Extracted Memory of the Actor-Critic

F. Supplementary results for the Ablation Study

Loading 2102.12344v5…