PaperlayerView exact revision on arXiv

Source-linked AI summary

Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPs

Tianwei Ni, Benjamin Eysenbach, Ruslan Salakhutdinov

arXiv:2110.05038v3cs.LGcs.AIcs.RO

TL;DR

Many RL problems can be viewed as POMDPs, yet recurrent model-free RL has often been reported as weaker than specialized methods. This paper evaluates whether careful implementation choices change that comparison and finds that a recurrent implementation performs competitively across diverse POMDP benchmarks.

  • Problem

    Prior work reported that recurrent model-free RL often performs poorly relative to specialized algorithms across several POMDP settings, despite its general applicability.

  • Method

    The paper develops and evaluates recurrent model-free RL with careful choices of architecture, RL algorithm, context length, and input information across multiple POMDP benchmarks.

  • Results

    The implementation performs on par with or better than prior specialized methods across benchmarks corresponding to different POMDP types.

  • Takeaways & Limitations

    Recurrent model-free RL is a strong general baseline for POMDPs when its implementation and hyperparameters are carefully designed.

  • Takeaways & Limitations

    Oracle policies may underperform because they use the same hyperparameters as the recurrent implementation, so further oracle tuning could change the comparison.

Abstract

from arXiv · show

Many problems in RL, such as meta-RL, robust RL, generalization in RL, and temporal credit assignment, can be cast as POMDPs. In theory, simply augmenting model-free RL with memory-based architectures, such as recurrent neural networks, provides a general approach to solving all types of POMDPs. However, prior work has found that such recurrent model-free RL methods tend to perform worse than more specialized algorithms that are designed for specific types of POMDPs. This paper revisits this claim. We find that careful architecture and hyperparameter decisions can often yield a recurrent model-free implementation that performs on par with (and occasionally substantially better than) more sophisticated recent techniques. We compare to 21 environments from 6 prior specialized methods and find that our implementation achieves greater sample efficiency and asymptotic performance than these methods on 18/21 environments. We also release a simple and efficient implementation of recurrent model-free RL for future work to use as a baseline for POMDPs.

1. Introduction

Many RL settings are POMDPs, but prior work reported that general recurrent model-free RL often underperformed specialized methods. This paper argues that careful implementation choices can make recurrent model-free RL competitive across diverse POMDP benchmarks.

  • Many RL subareas, including meta-RL, robust RL, generalization, and temporal credit assignment, can be formulated as special cases of POMDPs.
  • Specialized algorithms often rely on assumptions that limit their use across different POMDP settings, such as objective or stationarity assumptions.
  • Recurrent model-free RL combines model-free RL with recurrent policies or value functions, offering a simple and general approach applicable to any POMDP.
  • Prior studies reported poor performance for recurrent model-free RL in meta-RL, general POMDPs, robust RL, generalization, and temporal credit assignment.
  • Careful choices of actor-critic architecture, RL algorithm, and RNN context length are critical to recurrent model-free RL performance.
  • The paper contributes a performant, easy-to-use, and memory-efficient recurrent model-free RL implementation, supported by ablations of key design decisions.

2. Background

An MDP specifies fully observable states and dynamics, whereas a POMDP adds observations generated from hidden states and permits policies conditioned on observation history. The background introduces observable and hidden state components and average- versus worst-case objectives.

  • An MDP is defined by states, actions, transitions, initial-state distribution, rewards, horizon, and discount factor, with a policy maximizing expected discounted return.
  • For an MDP, an optimal policy can be memoryless, so it need not condition on the full interaction history.
  • A POMDP extends an MDP with observations and an emission function that maps states and actions to observation distributions.
  • In a POMDP, the general memory-based policy conditions actions on the observable trajectory, including observations, actions, and rewards.
  • The state can be divided into an observable component revealed by the current observation and a hidden component not directly revealed.
  • The background distinguishes average-case and worst-case objectives for POMDPs.

3. Related Work

The paper situates meta-RL, robust RL, generalization, and temporal credit assignment as specialized POMDP settings with distinct hidden-state, objective, and evaluation characteristics. Prior methods exploit these structures but may not transfer across settings.

  • Meta-RL: Meta-RL varies reward or dynamics parameters across episodes while keeping them fixed within each episode, making those values latent task identities.Meta-RL differs from multi-task RL because the latter observes task parameters.
  • Robust RL: Robust RL seeks policies maximizing worst-case returns against perturbations to dynamics, observations, or actions.Its objective distinguishes it from average-case generalization settings.
  • Generalization in RL: Generalization in RL evaluates performance on unseen test domains, using an average-case rather than worst-case objective.Benchmarks include shifts in initial states, procedurally generated games, and dynamics parameters.
  • Cross-setting comparison: Table 1 organizes POMDP subareas by hidden-state effects, within-episode changes, agent inputs, objective type, and domain shift.The table marks deviations from typical trends with an asterisk.
  • Temporal credit assignment: Temporal credit assignment becomes a POMDP when current rewards depend on earlier observations or actions, including delayed or trajectory-defined rewards.Proposed solutions include decomposing episodic rewards and redistributing rewards across time.

4. Design Considerations for Recurrent Model-Free RL

Recurrent model-free RL depends on several interacting design choices, including encoder sharing, agent inputs, the underlying algorithm, and recurrent context. The paper compares these choices across prior implementations and selects benchmark-level configurations rather than task-specific tuning.

  • Architecture: Separate actor and critic encoders are preferred because shared encoders increase gradient norms and hinder learning.The paper adopts the separate architecture after comparing the alternatives.
  • Agent inputs: Adding previous rewards and other information to recurrent inputs can increase return by up to 30%.Prior baselines often condition only on previous observations and actions.
  • Implementation comparison: Table 2 shows that prior implementations rarely share the same design, while the paper reports benchmark-specific choices that performed best.The table attributes poor performance in many prior implementations to their design decisions.
  • Model-free RL algorithm: The underlying model-free RL algorithm is paramount because recurrent model-free RL applies that algorithm to actors and Q-functions conditioned on input sequences.The discussion contrasts off-policy algorithms such as TD3 and SAC with their limited use in recurrent settings.
  • RNN variants and context length: LSTM and GRU can mitigate recurrent-training instability, but the paper finds RNN architecture has only a minor effect on final performance.The experiments compare LSTM and GRU variants.
  • RNN variants and context length: Optimal context length is task-specific, with experiments comparing short length 5, medium length 64, and long lengths larger than 100.Inferring hidden velocities from consecutive positions can theoretically require only a short context.

5. Experiments

Across POMDP benchmarks, a carefully tuned recurrent model-free RL implementation often matches or exceeds specialized methods, with performance depending strongly on architecture, algorithm, and context length. It achieves broad gains in sample efficiency and performance while retaining lower memory use, though some task-specific failures and slower wall-clock time remain.

  • Benchmark comparisons: 18 out of 21 environments show recurrent model-free RL at least comparable to, and sometimes substantially better than, prior specialized methods.The evaluation uses one best variant per benchmark without individually tuning each task.
  • Benchmark comparisons: 6 out of 8 environments favor the recurrent model-free implementation over VRM, while training 5× faster and using 200× less RAM.The largest gains occur on Cheetah-P and Hopper-P, exceeding 80% of oracle performance.
  • Benchmark comparisons: On generalization, the method matches EPOpt-PPO-FF in interpolation, surpasses it in extrapolation, and does not require dynamics parameters.It is comparable to the oracle in the more challenging extrapolation setting.
  • Benchmark comparisons: 100× fewer samples than IMPALA+SR are required to solve the Delayed-Catch and Key-to-Door temporal credit-assignment tasks.Both tasks use sparse rewards dependent on the whole trajectory.
  • Design choices: Separate recurrent actor and critic weights can improve performance, while context length and RL algorithm choices require task-dependent tuning.A medium context length of 64 is common among best variants, but both longer and shorter contexts can help different environments.

6. Conclusion and Future Work

The paper concludes that careful recurrent model-free RL implementations can match or exceed specialized POMDP methods, while emphasizing practical efficiency and tunable design choices.

  • Conclusion: Carefully designed recurrent model-free RL performs on par with or better than specialized methods across diverse POMDP benchmarks.Ablations identify the underlying RL algorithm and RNN context length as important design decisions.
  • Usability: The code supports API-based use or framework-based tuning of recurrent architecture and input choices.Users can configure the RNN encoder, actor-critic sharing, and inclusion of previous observations, actions, and rewards.
  • Efficiency: The replay buffer is designed for sequential off-policy data, improving memory efficiency relative to three-dimensional trajectory storage.The implementation samples subtrajectories using the selected context length and handles shorter episodes with zero-padding.
  • Efficiency: Training speed can be adjusted through the ratio of gradient updates to environment rollout steps.The ratio is typically at most 1.0 when higher training speed is desired.
  • Efficiency: The implementation is reported as memory-efficient in both RAM and GPU usage compared with specialized methods.The comparison includes system usage across environments, with different process counts and hardware settings affecting time costs.
  • Design factors: The implementation exposes actor-critic sharing, RL algorithm, encoder, inputs, context length, and entropy temperature as tunable factors.Available choices include shared or separate architectures, TD3 or SAC, LSTM or GRU, multiple input spaces, and short, medium, or long contexts.

B. Training Details

Training uses recurrent actor-critic architectures with task-dependent architectural adjustments, fixed benchmark hyperparameters, and sequence replay with zero-initialized hidden states.

  • Network architecture: Separate actor and critic RNNs are used by default, while temporal credit assignment tasks use an adjusted architecture for image observations and discrete actions.The architecture includes optional previous-action and reward embedders and a shortcut from current observation embeddings to the MLP.
  • Hyperparameters: The reported benchmark hyperparameters were not tuned, except for adjusting gradient steps so experiments finished within 72 hours.The benchmark-specific settings are summarized separately from the architectural choices.
  • Network architecture: A shared-RNN variant feeds both actor and critic heads from the same recurrent representation.The critic heads and actor head consume the shared RNN output.
  • Sequence training: Training samples subtrajectories from a replay buffer using the selected context length and zero-pads episodes shorter than that length.RNN hidden states are initialized with zeros under the zero start state strategy.
  • Baselines: Markovian SAC and TD3 remove the embedders and RNNs while retaining the recurrent-policy hyperparameters.For each task, the higher-return Markovian result between SAC and TD3 is reported.
  • Comparison settings: The recurrent implementation shares exactly the same settings as off-policy variBAD and VRM, whereas MRPO and EPOPT use different on-policy Markovian settings with ground-truth states.These setting differences affect how the specialized comparisons should be interpreted.

C. Evaluation Details

Evaluation covers six benchmarks and 21 environments, using four random seeds and normalized final performance to select the best recurrent variant per benchmark.

  • Evaluation protocol: Each implementation variant and comparison method is evaluated with 4 random seeds.This seed count is used throughout the experiments.
  • Variant selection: Best-variant selection first averages performance over the final 20% of environment steps across seeds.The selected variant is then chosen using normalized returns across all methods, including oracle and random policies.
  • Reporting: Final normalized performance is reported in bar charts and Table 3.The normalized results summarize each method or variant at the end of evaluation.
  • Evaluation settings: The hyperparameter summary reports module sizes, RL settings, and training hyperparameters for each benchmark.For meta-RL, the table uses the Cheetah-Vel model as an example following off-policy variBAD’s architecture design.
  • Evaluation settings: RNN weight initialization uses orthogonal matrices, while environment and gradient-step budgets vary across experiments.The listed environment-step budgets are 5M, 1.5M, 3M, and 5M, with corresponding gradient-step budgets of 0.1M, 1.5M, 0.6M, and 1.25M.
  • Evaluation scope: The experiments evaluate 6 benchmarks containing 21 environments in total.This provides the cross-benchmark scope for comparisons among recurrent variants and specialized methods.

D.1. “Standard” POMDP Benchmark from VRM

The standard POMDP benchmark removes selected observation components from locomotion environments, creating position- or velocity-hidden tasks, with Pendulum included as a sanity check.

  • Benchmark construction: The benchmark follows VRM’s occlusion setup and replaces deprecated Roboschool with PyBullet.Position- or angle-related entries are removed for “-V” environments, while velocity-related entries are removed for “-P” environments.
  • Additional evaluation: Pendulum is also evaluated as a sanity check in the appendix.This supplementary evaluation is separate from the main standard POMDP benchmark description.
  • Environment variants: The “-P” environments retain position observations while hiding velocity information.The benchmark includes Pendulum, Ant, Cheetah, Hopper, and Walker variants.
  • Comparison settings: The specialized-method settings used for the comparisons are summarized in Table 6.The table provides the reference configurations for the methods compared in the main paper.

D.2. Meta-RL Benchmark from Off-Policy VariBAD

This benchmark evaluates recurrent model-free RL on meta-RL tasks with hidden goals, dynamics parameters, or target velocities, using settings adopted from prior variBAD work. The environments require policies to act under task-relevant information that is not fully observed.

  • Experimental setting: The evaluation follows the prior benchmark’s training-task protocol, limiting training tasks for off-policy variBAD and not limiting them for on-policy variBAD.These choices are made to preserve the respective prior methods’ training settings.
  • Benchmark environments: The off-policy variBAD benchmark includes Semi-Circle, Wind, and Cheetah-Vel tasks with hidden or task-specific goal, noise, or target-speed information.Semi-Circle hides the goal in the reward; Wind includes fixed hidden dynamics noise; Cheetah-Vel uses target speed as the hidden state.
  • Benchmark environments: The on-policy variBAD benchmark includes Ant-, Cheetah-, and Humanoid-Dir tasks whose hidden state is the target velocity direction.Ant-Dir and Cheetah-Dir use two directional tasks, while Humanoid-Dir samples directions uniformly from the unit circle.

D.4. Robust RL Benchmark from MRPO

The robust RL benchmark evaluates policies across environments with hidden physical dynamics parameters. Performance is summarized by both average returns and the returns from the worst 10% of tasks.

  • Benchmark environments: The benchmark uses Hopper-Robust, Walker-Robust, and Cheetah-Robust environments adopted from MRPO.Each environment is based on simulated robots whose hidden parameters include density and friction coefficients.
  • Evaluation metrics: Each environment is evaluated on 100 tasks, with average returns and worst returns computed across those tasks.Worst returns are defined as the average of the worst 10% of tasks, following MRPO.
  • Related evaluation setting: The generalization benchmark uses hidden density, friction, and power parameters to evaluate interpolation and extrapolation across task distributions.Interpolation and extrapolation settings differ in whether training and testing use the same or disjoint hidden-state supports.

D.6. Temporal Credit Assignment Benchmark from IMPALA+SR

The temporal-credit-assignment benchmark uses two sparse-reward pixel environments in which useful rewards depend on actions taken much earlier in an episode. The evaluation compares learning curves across the paper’s design factors and prior methods.

  • Benchmark environments: Delayed-Catch and Key-to-Door use discrete actions, pixel observations, and trajectory-level sparse rewards.The environments have 3 and 4 actions, with observations of 1 × 7 × 7 and 3 × 5 × 5 pixels, respectively.
  • Delayed-Catch: Delayed-Catch provides non-zero reward only at episode end, where the optimal terminal reward is 40 successful runs.Each episode contains 40 runs and lasts around 280 steps.
  • Key-to-Door: Key-to-Door makes the final door-opening bonus depend on picking up a key during an earlier phase when the key is no longer visible.The episode also includes an intermediate apple-collection phase with rewards.
  • Evaluation: The paper presents learning curves for temporal credit assignment alongside curves for standard POMDPs, meta-RL, robust RL, and generalization in RL.The compared plots include oracle, Markovian, and random policies where applicable.
  • Design analysis: The analysis ablates actor-critic architecture, inputs, model-free RL algorithm, RNN encoder, and context length.These factors are evaluated because recurrent model-free performance can depend on implementation choices.

E.4. Additional Results on Comparison with VRM

Additional results compare the recurrent model-free implementation with VRM and other specialized or recurrent baselines across POMDP benchmarks. The comparisons use matched or explicitly adjusted training budgets and report final performance and learning curves.

  • Reporting: The final-performance table reports means and standard deviations averaged over the last 20% of environment steps across four seeds.The table’s best variant uses the notation defined in the appendix.
  • Comparison with VRM: VRM is approximately 5× slower than the implementation, so the standard-POMDP comparison gives VRM 0.5M environment steps under a matched time budget.The same budget adjustment is described for both the -P and -V benchmark figures.
  • Meta-RL comparisons: The implementation is at least comparable to, and sometimes greatly surpasses, off-policy variBAD across all three environments in its benchmark.The comparison is based on learning curves with better sample efficiency reported for the implementation.
  • Cross-benchmark comparisons: The benchmark figures compare recurrent model-free RL with specialized methods across meta-RL, robust RL, generalization, and temporal credit assignment settings.The compared methods include on-policy variBAD, MRPO, EPOpt-PPO-FF, and IMPALA+SR, alongside recurrent baselines.
  • Ablations: Ablation figures analyze RL algorithm, encoder, context length, inputs, and actor-critic architecture across standard POMDP and specialized benchmarks.The actor-critic comparison also examines performance and encoder-gradient norms for shared versus separate recurrent architectures.
  • Comparison with VRM: The implementation surpasses VRM on 7 of 8 standard POMDP environments when both are trained for 0.5M simulation steps.The figure reports normalized returns and includes Markovian policies as lower bounds and an oracle policy as a reference.
Loading 2110.05038v3…