PaperlayerView exact revision on arXiv

Source-linked AI summary

Does Learning Recurrent Networks Matter in Partially Observable RL?

Zekang Wang, Zhe He, Gaojie Jin

arXiv:2610.13197v1cs.LGcs.AI

TL;DR

Partially observable RL commonly trains recurrent memory jointly with the policy or value function, leaving open whether learning that memory is necessary. The paper tests frozen recurrent networks with trainable output modules and finds competitive results across diverse POMDPs, with substantially fewer trainable parameters.

  • Problem

    It remains unclear whether a frozen recurrent network can provide effective memory for general POMDPs, despite prior findings on untrained or frozen recurrent networks.

    Source: §1 · p. 1

  • Method

    The paper uses frozen recurrent networks as memory and trains downstream policy or value modules, including visual encoders for pixel-based tasks.

    Source: §1 · pp. 1–2

  • Results

    Frozen recurrent networks match trainable counterparts across varied POMDPs, including long-horizon recall, discrete and continuous control, and pixel-based tasks, while using more than half fewer trainable parameters.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results suggest learning the recurrent module is not always necessary, and frozen recurrent networks offer a simple, parameter-efficient baseline for partially observable RL.

    Source: §7 · p. 8

    The evaluation covers a limited set of benchmarks and baselines, and the GRC analysis relies on specific assumptions whose conditions may not hold exactly in practical RL.

    Source: §6 · p. 8

Abstract

from arXiv · show

Reinforcement learning (RL) under partial observability typically re- lies on recurrent networks to encode temporal information, with the recurrent module trained jointly with the policy or value function. This raises a natural question: is learning the recurrent network actually necessary? In this paper, we investigate whether a frozen recurrent network can provide sufficient memory for solving par- tially observable Markov decision processes (POMDPs). Motivated by the theoretical foundations of Reservoir Computing (RC) and Generalized Reservoir Computing (GRC), we study a simple archi- tecture that combines a fixed recurrent network with a trainable nonlinear output module. We evaluate this approach across tasks with varying memory demands, observation modalities, and action spaces. Surprisingly, frozen recurrent networks support recall over hundreds of steps and achieve competitive control performance across diverse POMDPs, while using less than half the number of trainable parameters required by their fully trainable counterparts. These results suggest that learning the recurrent memory module is not always necessary for solving POMDPs. Frozen recurrent networks therefore provide a simple, parameter-efficient, and sur- prisingly strong baseline for partially observable RL. Our source code is available at https://github.com/Aphlios/FRN-POMDP

1 Introduction

Partially observable RL needs memory to recover information missing from current observations, but recurrent memory is usually trained jointly with the policy or value function. The paper asks whether frozen recurrent networks can provide effective memory for POMDPs and reports competitive performance across varied tasks with fewer trainable parameters.

1 Introduction

  • Memory lets an agent summarize interaction history into a latent state that captures information missing from partial or noisy observations, while RNNs, Transformers, and SSMs are commonly trained jointly with policy or value functions.This motivates asking whether learning the memory module itself is necessary.Source: §1 · p. 1
  • The paper investigates whether randomly initialized frozen recurrent networks can serve as POMDP memory while only the policy or value function is learned.The question is motivated by prior findings on untrained or frozen recurrent networks, but their effectiveness for general POMDPs remains unclear.Source: §1 · p. 1
  • Reservoir Computing uses a randomly initialized recurrent reservoir with frozen weights and a readout, while Generalized Reservoir Computing suggests a suitable readout can recover input history without the Echo State Property.The paper argues that the Echo State Property is therefore not necessary for designing frozen recurrent memory in recurrent agents.Source: §1 · pp. 1–2
  • Frozen recurrent networks match trainable counterparts across long-horizon recall, discrete and continuous control, and pixel-based POMDPs, with modest losses in a few task-specific cases.The paper also reports that the approach reduces trainable parameters by more than half.Source: §1 · p. 2
  • The proposed architecture combines frozen recurrent memory with a trainable nonlinear output module and trains visual CNN encoders through gradients passing through the recurrent network.The contributions describe a reduction of more than half in trainable parameters and report that visual encoders can be learned for high-dimensional partially observable control.Source: §1 · p. 2, Figure 1 · p. 2

2 Related Work

Prior work studies learned recurrent, attention-based, structured, and external-memory approaches, as well as reservoir computing and frozen recurrent networks. This paper distinguishes its approach by freezing recurrent units from conventional architectures and evaluating them across a broader range of POMDPs.

2 Related Work

  • Existing recurrent, attention-based, structured-memory, and external-memory agents process past information, but still learn many parameters within their recurrent memory.Examples include LSTM-based agents [12], attention or recurrent-memory approaches [2] [14] [34] [51] [53], structured recurrence [36] [37] [45], and external-memory systems [15] [64].Source: §2 · p. 2
  • Reservoir computing separates a fixed recurrent backbone from a trainable readout, and ESN-based methods have achieved competitive performance on memory-intensive tasks [41].Prior studies primarily focus on reservoir-specific architectures, whereas this paper freezes recurrent units from conventional architectures.Source: §2 · p. 2
  • Earlier frozen-network studies cover limited POMDP environments, and reported frozen RNN or GRU agents failed on several counting tasks under one A2C setup [9].The paper uses GRC to guide its design and tests frozen recurrent networks on complex POMDPs.Source: §2 · p. 2

3 Background

The section defines the Markov and partially observable settings, then explains recurrent memory and reservoir-based alternatives. It distinguishes forgetting initial conditions from preserving input history, and shows how a readout can recover history-dependent outputs even when hidden states retain initialization dependence.

3.1 Reinforcement Learning (RL)

  • An MDP models state, action, transitions, rewards, and discounting, with the next state depending only on the current state and action.The agent seeks a policy maximizing expected discounted returns.Source: §3.1 · p. 3

3.2 Partially Observable RL and Recurrent Memory

  • A POMDP hides the latent state, so a single observation may not be Markov and the agent instead uses its trajectory as context.A recurrent model compresses that history into a hidden state and a readout maps the state to action values or policy parameters.Source: §3.2 · p. 3

3.3 Reservoir Computing and the Echo State Property

  • For a leaky ESN, κ < 1 is sufficient for contraction, whereas recurrent spectral radius below one alone does not guarantee the required norm bound.The Echo State Property concerns forgetting initialization under the same input sequence; control still requires distinguishing histories with different inputs.Source: §3.3 · p. 3

3.4 Generalized Reservoir Computing

  • Generalized Reservoir Computing permits hidden states to retain initialization dependence when a time-invariant readout extracts a history-dependent output.A two-coordinate example shows a readout can discard the initialization-dependent coordinate while preserving a nonconstant function of input history.Source: §3.4 · pp. 3–4
  • A frozen LRU is a stable linear reservoir satisfying the Echo State Property, while a frozen GRU need not satisfy state ESP.The section gives a frozen-GRU construction without state ESP but with a readout that has output ESP.Source: §3.4 · p. 4

4 Experiments

Across recall, discrete and continuous control, and visual tasks, frozen recurrent networks perform competitively with trainable counterparts. The experiments also find that learned mixers improve frozen-network performance.

4.1 Experimental Setup

  • Experiments compare frozen GRU and LRU agents with trainable baselines across recall, discrete and continuous control, and pixel-observation POMDPs.The recurrent units remain fixed while trainable mixers and readout networks are learned; visual observations can pass through a learned CNN encoder.Source: §4 · p. 4, §4.1 · pp. 4–6

4.2 RQ1: Long-Horzion Recall

  • 100% success persists to 256-step delays for Frozen LRU and 128-step delays for Frozen GRU, versus 512 steps for trainable LRU.Trainable GRU also reaches 100% at 128 steps.Source: §4.2 · p. 6

4.3 RQ2: Vector Observation Discrete POMDPs

  • Frozen LRU returns 0.464 versus 0.458 for LRU while training approximately 40% of total agent parameters on POPGym.Frozen GRU trains approximately 30% of total agent parameters and achieves 88.70% of GRU’s final evaluation return.Source: Figure 3 · p. 5, §4.3 · p. 7

4.4 RQ3: Vector Observation Continuous POMDPs

  • Frozen LRU retains 95% of LRU’s return, and Frozen GRU retains 92% of GRU’s return across seven continuous-control observation settings.The respective final evaluation returns are 791.8 versus 831.9 and 772.9 versus 842.8.Source: Figure 4 · p. 6, §4.4 · p. 7

4.5 RQ4: Partially Observable Visual Control

  • Frozen LRU scores 0.694 versus 0.624 for LRU across four visual tasks, a relative increase of 11.2%.Frozen GRU scores 0.643, retaining 95.2% of GRU’s 0.676 return; the agents use learned visual representations.Source: §4.5 · p. 7, Figure 5 · p. 7

4.6 RQ5: Learning the Mixers Matter

  • Training mixers raises final evaluation returns from 0.392 to 0.464 for Frozen LRU and from 0.341 to 0.386 for Frozen GRU.These correspond to relative improvements of 18.5% and 13.2%, respectively.Source: §4.6 · p. 8, Figure 6 · p. 8

5 Discussion

The experiments challenge the assumption that recurrent RL requires a trained memory model. The paper presents frozen recurrent networks with nonlinear MLPs and mixers as an alternative for tested POMDPs.

5 Discussion

  • Frozen recurrent networks paired with nonlinear MLPs and mixers offer an alternative to training the memory model in the tested POMDPs.The authors also suggest possible benefit for recurrent RL in robotic learning.Source: §5 · p. 8

6 Limitations and Future Work

The study’s evaluation is limited in benchmark breadth and ablation coverage, while its GRC analysis depends on assumptions that may not hold exactly in practical RL.

6 Limitations and Future Work

  • The evaluation covers a limited benchmark set and lacks a large-scale exhaustive ablation study.Computational constraints also restrict the range of baselines considered.Source: §6 · p. 8
  • The GRC analysis relies on specific assumptions and establishes its result under a relaxed ESP condition.Random initialization, optimization dynamics, and finite-sample effects may keep trained agents from satisfying theoretical conditions exactly.Source: §6 · p. 8
  • Future work should broaden ablations, benchmarks, and baselines and examine how practical deviations from GRC conditions affect performance.Source: §6 · p. 8

7 Conclusion

The paper concludes that frozen recurrent memories can retain information over long horizons and perform competitively across diverse POMDPs. They offer a simple, parameter-efficient baseline, including for high-dimensional visual control.

7 Conclusion

  • Across diverse POMDPs, frozen recurrent memories retain information over long horizons and achieve competitive performance with substantially fewer trainable parameters.Source: §7 · p. 8
  • The architecture uses frozen recurrent networks as memory modules, while visual encoders can be trained through them for high-dimensional visual control.Source: §7 · p. 8
  • The results suggest that learning the recurrent module is not always necessary, and frozen recurrent networks provide a simple, strong, parameter-efficient recurrent RL baseline.Source: §7 · p. 8

AI Use Statement

This material discloses AI assistance and describes the PPO agent and three deep reservoir variants. It outlines their recurrent updates and how their features reach trainable readouts.

AI Use Statement

  • OpenAI Codex with GPT-6 Astra supported language and LaTeX polishing, plotting code, and execution scripts; complete prompt and conversation records were not retained.The authors state they take responsibility for the content, code, and results.Source: AI Use Statement · p. 8

A PPO Agent

  • The MuJoCo Playground recurrent PPO agent follows the PQN architecture, using separate actor and critic MLPs; PPO updates its trainable parameters.The actor produces the policy and the critic estimates state value.Source: A PPO Agent · p. 11

B Deep Reservoir Computing

B.1 Deep Echo State Network (DeepESN)

  • DeepESN stacks fixed recurrent reservoir layers with leaky state updates, then gives a trainable readout concatenated features from different depths.The leak rate controls how much of each candidate state replaces the previous state.Source: §B.1 · p. 11

B.2 Structured Reservoir Computing (SRC)

  • SRC replaces dense random projections with structured transforms built from normalized Hadamard and random-sign diagonal matrices.The fast Hadamard transform evaluates the projection in O(m log m) operations without storing a dense matrix; sequence updates remain stepwise.Source: §B.2 · p. 11

B.3 Parallel Echo State Network (ParalESN)

  • ParalESN separates a fixed complex-valued recurrence from nonlinear mixing that produces real-valued features for later blocks and the trainable readout.Because the recurrence remains affine, updates compose associatively and can be evaluated by a parallel scan.Source: §B.3 · pp. 11–12

C Reservoir Training on POPGym

The section compares DeepESN, SRC, and ParalESN with trainable variants on POPGym tasks, using learning curves and final evaluation returns.

C Reservoir Training on POPGym

  • DeepESN, SRC, and ParalESN are compared with their trainable variants on 18 POPGym tasks.Figure A2(a) presents learning curves, while Figure A2(b) summarizes final evaluation returns across the tasks.Source: C Reservoir Training on POPGym · p. 12

D Detailed Evaluation Results

The appendix reports final evaluations across POPGym, POPGym Arcade, and MuJoCo Playground, with a higher mean return for ParalESN than its trainable counterpart on six of seven selected POPGym settings.

D Detailed Evaluation Results

  • Each seed’s trained agent is evaluated over 10 episodes; heatmap entries give mean and sample standard deviation across 10 seeds for POPGym and MuJoCo, and five for Arcade.Source: D Detailed Evaluation Results · p. 12

D.1 POPGym

  • POPGym heatmaps cover 18 tasks and 11 model configurations, excluding four tasks because of vector-environment implementation bugs.The excluded tasks are Autoencode, Concentration, CountRecall, and MultiArmedBandit.Source: §D.1 · p. 12
  • Frozen GRU and Frozen LRU retain trainable mixers and outer projections, and the POPGym heatmaps share a color scale centered at zero.Source: §D.1 · p. 12
  • ParalESN achieves a higher mean final evaluation return than its trainable counterpart on six of the seven selected learning-curve settings.Source: §D.1 · p. 12

D.2 POPGym Arcade and MuJoCo Playground

  • Final returns are reported for four POPGym Arcade tasks and seven MuJoCo Playground settings, using PQN and PPO, respectively.CountRecall is retained in POPGym Arcade because it uses a separate implementation from vector POPGym.Source: §D.2 · p. 12

E Model and Training Parameters

This section specifies recurrent-agent and CNN architecture details, training settings, and the appendix tables and figures documenting configurations and evaluations.

E Model and Training Parameters

  • The architecture appendix describes recurrent agents and the POPGym Arcade CNN encoder, including observation, action, and input dimensions and readout configurations.PPO uses separate actor and critic readouts with a shared recurrent trunk; recurrent-block and readout-block counts vary by benchmark.Source: E Model and Training Parameters · p. 12, Table A1 · p. 13, Table A2 · p. 14
  • Training uses Adam with linear learning-rate decay, and Tables A3–A6 list benchmark-specific training hyperparameters.For MuJoCo Playground, the reported initial learning rates are ranges across models and tasks.Source: E Model and Training Parameters · p. 12, Table A3 · p. 15, Table A4 · p. 15
  • Figures A2–A5 provide learning curves and final evaluation returns for reservoir models, recurrent baselines, and frozen variants across POPGym, Arcade, and MuJoCo.Figure A2 covers selected-task learning curves and final returns across all 18 POPGym tasks; Figures A3–A5 provide detailed returns.Source: Figure A2 · p. 13, Figure A3 · p. 14, Figure A4 · p. 15
Loading 2610.13197v1…