PaperlayerView exact revision on arXiv

Source-linked AI summary

Shared Actors Need Not Share Critics: Effects of Value Mismatch in Parallel Reinforcement Learning

Zhenya Liu, Yang Meng, Zhuokai Zhao, Xuefeng Liu, Yuxin Chen

arXiv:2608.26481v1cs.LG

TL;DR

Parallel reinforcement learning can suffer when a shared critic assigns one value to inputs whose returns differ across environments, altering realized updates despite unchanged fixed-policy expected gradients. The paper analyzes this value mismatch and conditions only the critic on an environment index; across benchmark families, this improves learning while preserving one shared actor, including a 40.8% Procgen gain.

  • Problem

    A shared critic must reconcile different environment-specific values for the same visible input, but fixed-policy expected-gradient equivalence does not preserve online learning dynamics.

  • Method

    The paper uses multi-environment bandit analysis and experiments with critics conditioned only on a logged categorical environment index, leaving the actor unchanged.

  • Results

    Across four benchmark families, conditioning the critic improves learning; the multihead critic improves aggregate normalized Procgen return on unseen levels by 40.8%.

  • Takeaways & Limitations

    Value mismatch is a direct mechanism by which critic sharing can degrade sampled learning dynamics, motivating critic-only environment conditioning in parallel reinforcement learning.

  • Takeaways & Limitations

    The method uses recurring environments with stable logged identifiers and does not address genuine task conflict, critic estimation error, or every cost of privileged information.

Abstract

from arXiv · show

When a single policy is trained in parallel across multiple environments of the same task, such as procedurally generated levels, randomized dynamics, or curricula, implementations commonly use one critic across all sampled environments. Yet different environments can assign different expected returns to the same input visible to the critic. A critic without environment information must then reconcile distinct value targets, systematically shifting the sampled advantages within individual environments. Using illustrative bandit models with multiple environments and a common optimal arm, we characterize how this value mismatch redistributes sampled policy updates, reinforcing unhelpful actions while attenuating or even reversing useful ones. The oracle processes using no baseline, the shared value, or the value specific to the sampled environment have the same mean logit update at a fixed policy and converge to the same optimal policy, yet their realized learning paths can differ sharply. The analysis motivates a minimal intervention: give only a logged environment index to the critic so that it can separate the value targets. Controlled CartPole and MuJoCo experiments expose the predicted shifted values, advantages, and performance gaps. In the more complex BipedalWalker and Procgen settings, the same intervention yields more stable learning and higher returns. Across all $16$ Procgen games, the multihead conditional critic improves aggregate normalized return on $600$ unseen levels per game by $40.8\%$. In conclusion, the theory identifies value mismatch as a direct mechanism through which critic sharing can degrade stochastic learning dynamics, not captured by scalar estimator variance alone, and the experiments show that conditioning on an index is broadly effective in parallel reinforcement learning.

1 Introduction

Parallel reinforcement learning can use one shared actor across environment variants, but an unconditioned critic may assign different continuation values to the same visible input. Although baseline choices share the same fixed-policy expected update and asymptotic destination, they can produce sharply different sampled learning paths.

  • Parallel training repeatedly samples environment variants while optimizing one shared policy, and standard implementations commonly share one value function across levels.
  • Different dynamics or horizons can create different continuation values for the same critic-visible input, producing value mismatch under a shared critic.
  • A shared value can remain an unbiased baseline while systematically miscentering samples within individual environments.
  • The no-baseline, shared-value, and environment-specific-value processes have the same fixed-policy expected update and asymptotic destination but sharply different sampled paths.
  • The proposed intervention gives only a logged environment index to the critic, allowing distinct environment-specific values without changing the actor.
  • 40.8% improvement in aggregate normalized Procgen return was achieved by one conditioned-critic design on unseen levels over the shared critic.

2 Related work

Prior work addresses heterogeneous values, privileged critics, and conditioning, but this paper focuses specifically on how shared marginal values redistribute realized updates along the closed-loop learning path.

  • Baseline research studies variance reduction and control variates for policy-gradient estimators, including state- and action-dependent designs.
  • PopArt and multitask PopArt address heterogeneous value targets while retaining shared policy structure and task-indexed value outputs.
  • Conditional and asymmetric critics use information available only to the critic, including conditional values and privileged signals under partial observability.
  • AACC conditions the critic on continuous simulator factors, whereas PAMDP uses a dual critic conditioned on persona profiles.
  • This paper contributes a complementary learning-dynamics account: a shared prediction can redistribute the same mean update across sampled environment branches and alter the realized path.

3 Problem setting: one policy, many environments

The theory analyzes one softmax actor sampling deterministic rewards across finitely many environments that share a strict optimal arm. It compares oracle baselines and shows that their paths differ despite a common expected update and destination.

  • The explanatory model samples an environment index independently from a finite distribution and an arm from a single shared softmax policy.
  • Every environment has a deterministic bounded reward table, and all environments share one strict optimal arm.
  • The analyzed oracle baselines are no baseline, shared value, and sampled-environment value, isolating centering effects from critic fitting.
  • Each iteration samples an environment and arm, observes its reward, evaluates a baseline, and applies a softmax REINFORCE logit update.
  • The bandit abstraction isolates value mismatch from state visitation, critic estimation error, and function approximation.
  • All three processes converge to the same optimal policy with an O(log T/T) time-averaged suboptimality bound, while finite-horizon differences arise from their paths.
  • All three processes share the same expected update at a fixed policy because action-independent baselines cancel after averaging over arms and environments.

4 Baselines change the online update dynamics

Baselines leave the expected policy-gradient update unchanged at a fixed policy, but they can sharply alter realized learning paths. In parallel environments, an unconditioned critic shifts advantages by environment-specific value mismatch, whereas conditioning removes that offset and improves update direction.

  • Expected versus realized updates: At a fixed policy, action-independent baselines produce identical expected updates, but realized single-sample updates can differ in magnitude and sign.This distinction separates common asymptotic behavior from different online learning trajectories.
  • One environment: an eventual ratchet: A value baseline reinforces an arm only when its sampled reward exceeds the current value, unlike no-baseline updates that can reinforce positive-reward rivals.In the one-environment setting, value centering eventually turns every sample into progress toward the optimal arm.
  • Multiple environments: the shared offset: Value mismatch shifts the score contribution by eπ_z(s), allowing above-average environments to reinforce suboptimal draws and below-average environments to suppress optimal draws.The shared critic places one averaged bar across environments; conditioning removes the environment-specific offset.
  • Multiple environments: the shared offset: Earlier reversals and larger reversal magnitudes can compound into less favorable finite-horizon trajectories even when shared and conditional processes have the same asymptotic destination.Figure 2 illustrates a conditional path that rises after an early transient while the shared path continues stepping backward.
  • Multiple environments: the shared offset: Conditional centering eventually moves every sampled branch toward the optimum, while shared centering can keep producing backward steps from sufficiently hard environments.Mismatch magnitude and learning rate jointly affect when reversals begin and how strongly they act over finite horizons.
  • Conditioning only the critic: A minimal intervention gives only a logged categorical environment index to the critic, preserving the shared actor and sampling process while separating value targets.The actor does not receive the index, and the critic and index are discarded at deployment.

5 Experiments

Experiments test whether conditioning only the critic on logged environment identity separates mismatched value targets while leaving the shared actor and sampling process unchanged. Across controlled and complex benchmarks, conditioned critics recenter advantages, avoid shared-critic plateaus, and improve returns on unseen environments.

  • Experimental design: Conditioned critics receive the logged environment index while the actor, sampling protocol, and PPO pipeline remain shared across comparisons.The intervention tests value separation without supplying environment information to the actor.
  • CartPole: CartPole’s heterogeneous gravities produce different futures from the same observation, with multihead values separating by gravity while the shared value remains between them.The identical-dynamics control shows no systematic separation, isolating heterogeneous dynamics as the source of mismatch.
  • CartPole: Under heterogeneous gravity, the shared critic keeps the hard level’s mean advantage negative, whereas the multihead critic recenters it and reaches a narrow return band near 200 after its fitting transient.The hard-level multihead loss initially peaks near 780 before the advantage recenters.
  • MuJoCo: In MuJoCo, FiLM improves returns on HalfCheetah, Hopper, and Walker2d while conditioning largely removes the shared critic’s light-positive and heavy-negative advantage ordering.Reported gains are roughly 2600 versus 2050 on HalfCheetah and 2450 versus 2000 on Walker2d.
  • BipedalWalker: On BipedalWalker, shared-critic performance plateaus below 100, while FiLM reaches roughly 150 and multihead reaches 165–190 on unseen terrains.Both conditioning architectures escape the same plateau, including when critic architectures are enlarged to roughly five times their original size.
  • Ablations: Ablations attribute the gains to environment-specific value conditioning rather than critic capacity or target normalization, while conditioning the actor collapses training and test return.Adding PopArt reduces the pinned-level multihead gain from +24.2% to +10.0%.

6 Discussion, limitations, and conclusion

The discussion frames value mismatch as a mechanism that can degrade shared-critic learning dynamics and motivates conditioning only the critic on environment identity. It also distinguishes when conditioning helps, where the theory and method are bounded, and how the intervention performs across benchmark families.

  • Statistical sharing versus target separation: Conditioning is most useful when environment identity explains substantial value variation, but finite-data estimation can make target separation costly when values are similar or environments are rare.The practical tradeoff is between statistical sharing and separating conflicting value targets.
  • Mismatch severity: Mismatch severity depends on value dispersion and sampling, not merely the number of environments; rare environments can produce infrequent but large miscentered updates.Tightly clustered environments may be benign, whereas separated values can suffice to create mismatch.
  • Scope of the theory: The theory isolates a pathwise mechanism rather than a uniform finite-horizon ordering, using oracle baselines, deterministic bandits, a common optimal arm, and a fixed environment mixture.Deep RL adds state visitation, function approximation, learned critics, GAE, and PPO transformations, so experiments test transfer of the mechanism rather than a general convergence theorem.
  • Scope of the method: The method uses recurring environments with stable logged identifiers; continuous or unseen variants may require simulator parameters or learned or inferred representations.Conditioning does not address genuine task conflict, critic estimation error, or every cost of privileged information.
  • Conclusion: Across four benchmark families, conditioning only the critic improves learning while preserving one shared actor.The conclusion proposes conditioning selectively where mismatch is large, combining conditional value fitting with statistical sharing elsewhere.

AI use statement

The authors state that generative AI tools were used for manuscript polishing and mathematical-proof verification, while research conception, experiments, analyses, and conclusions remained the authors’ responsibility.

  • AI use statement: Generative AI was used only to polish the manuscript and assist in verifying mathematical proofs.The authors attribute the research ideas, claims, results, designs, analyses, and conclusions to themselves.

Reproducibility statement

The paper identifies where theoretical assumptions, proofs, architectures, benchmark environments, hyperparameters, seeds, evaluation protocols, aggregation rules, and uncertainty conventions are documented.

  • Documentation: Section 3 gives the theoretical setting, assumptions, and update rule, while Appendices A–D provide complete proofs and auxiliary results.Appendix E.1 covers conditioning architectures and parameter counts.
  • Experimental details: Appendix E documents benchmark environment pools, hyperparameters, seed counts, evaluation protocols, and aggregation rules.Figure and table captions state seed counts and uncertainty conventions.

A What variance can and cannot explain

The appendix distinguishes aggregate variance guarantees from the sampled branch allocation that governs learning paths. It derives the richer-information variance certificate, then uses a bandit example to show that lower local covariance can accompany slower and less stable finite-horizon learning.

  • Variance framework: Conditioning the baseline on richer information improves the variance certificate by a quantity proportional to the expected squared value mismatch.The gain is exact under the stated finite-moment assumptions.
  • Variance framework: The exact covariance-trace minimizer is a score-weighted baseline, whereas the ordinary value critic minimizes an unweighted prediction error.Ordinary value prediction remains a valid variance-related surrogate.
  • What variance misses: The aggregate variance certificate does not reveal which environment–action branches are reinforced or how those allocations alter future sampling.Baseline placement can therefore matter for the realized learning path even when expected updates agree.
  • Numerical counterexample: After 1,000 updates, the conditional-value process reaches 0.9998 ± 0.0004 versus 0.9777 ± 0.1468 for the covariance-trace minimizer.The conditional ensemble mean crosses 0.99 after 38 updates, while the covariance-trace process remains below 0.99 through update 1,000.
  • Interpretation: The two processes share the same asymptotic destination, so variance remains an aggregate certificate but cannot explain the sampled path’s branch allocation.The example shows why finite-horizon learning dynamics need not follow covariance-trace rankings.

B Proof of Proposition 1

The proof reduces the three baseline processes to a common-shift stochastic update with identical mean direction, then uses concentration and exploration arguments to establish convergence to the shared optimum.

  • Effective reward reduction: All three baseline choices induce the same conditional mean update after averaging over sampled environments, differing only through a predictable common shift in realized rewards.The shift preserves pairwise arm gaps while changing the sampled noise law.
  • Exploration and drift: Infinite exploration follows from conditional Borel–Cantelli, ensuring that every arm continues to be sampled sufficiently often for the optimal arm’s positive drift to accumulate.The argument combines exploration with concentration so noise cannot indefinitely cancel the drift.
  • Concentration: The proof uses a stopped Freedman–Bernstein concentration bound to control martingale deviations uniformly over time.The bound supports the later divergence and trapping arguments.
  • Conclusion: Consequently, each process reaches the same optimal policy, while its finite-time trajectory remains sensitive to the realized baseline-dependent updates.The asymptotic conclusion does not imply identical learning paths.
  • Elimination: A supermartingale argument eventually traps the process in a region where the optimal logit separates from all rivals, causing every rival logit to diverge negatively.The proof handles the common strict optimum simultaneously across all rival arms.

C Proof of Proposition 2

The proof shows that softmax policy-gradient dynamics with bounded updates eliminate suboptimal arms under a strict common optimum, with value centering preserving positive margins once the optimal arm dominates.

  • Value baseline: When the optimal-arm probability exceeds a threshold, the value baseline gives positive optimal advantages and negative rival advantages on every sampled branch.Every possible action then strictly increases the margins between the optimal arm and each rival.
  • Invariant region: The region in which the optimal arm has probability above the threshold is invariant under future value-baseline updates.The optimal probability therefore converges to one after finite entry into that region.
  • No baseline: With no baseline, positive-reward rivals still generate strict drawdowns in the aggregate rival-margin process, while softmax exploration samples every arm infinitely often.These repeated drawdowns support elimination despite the absence of value centering.
  • Elimination: The proof combines convergence and infinite exploration to show that every rival logit tends to negative infinity while the optimal probability tends to one.The conclusion applies simultaneously to the finitely many rival arms.

D Proofs for Section 4.2

These proofs characterize shared-critic mismatch: conditional values minimize environment-aware prediction error, while shared values can reverse or attenuate useful updates and cause recurring drawdowns in mismatched environments.

  • Critic targets: The population squared-error minimizer is the marginalized value without environment information and the conditional value when the critic observes the environment index.The two targets differ by the value mismatch across environments.
  • Easy environments: In easy environments, shared centering can assign positive residuals to rival actions, promoting unhelpful branches that conditional centering would correct.The resulting signs depend on the environment-specific offset relative to the pooled value.
  • Hard environments: In hard environments, shared centering flips the optimal-arm update when the sampled return falls below the shared value and attenuates it when the return exceeds that value.The reversal threshold is determined by the environment’s mismatch from the pooled value.
  • Scope of comparison: The ordering by mismatch severity applies along the same shared-baseline trajectory, not necessarily to hitting times across separately trained processes.Changing offsets can also change which environments belong to the mismatched set.
  • Persistent drawdowns: Mismatched hard environments produce optimal-arm drawdowns infinitely often, while non-mismatched environments increase the optimal probability on those branches.The recurrence follows from conditional Borel–Cantelli and the asymptotic shared residual signs.
  • Path effects: At a fixed policy, shared and conditional baselines have identical expected updates, so their divergence arises from different realized branches and subsequent policy states.The same-sample comparison is local and does not equate later updates after trajectories diverge.

E.1 Conditioning architectures: FiLM, multihead, and PopArt

The conditioning architectures give the critic environment-indexed value capacity while leaving the actor unchanged, using either independent multihead values, shared FiLM modulation, or per-head PopArt normalization.

  • Architecture choices: FiLM and multihead critics represent environment-specific value corrections through different parameter-sharing schemes.Multihead fits independent readout rows per level, whereas FiLM jointly fits low-dimensional embeddings and shared generators.
  • PopArt: PopArt augments the multihead critic by normalizing each head’s targets while preserving its denormalized value output.Running per-head moments determine the normalized prediction and rescaling.
  • Initialization: All conditioned critics are initialized to match the shared critic exactly, so later differences arise from learning rather than initialization.FiLM modulation starts at zero and multihead rows copy the shared readout.
  • Information asymmetry: The intervention supplies only an arbitrary identity label to the critic; the actor never observes it and deployment runs only the actor.Thus critic conditioning adds no test-time actor input or deployment cost.
  • CartPole: CartPole retains two logged level identities, with homogeneous gravity rewards of (10, 10) or heterogeneous rewards of (10, 50), while gravity remains absent from observations.The setup isolates learning-signal effects rather than unseen-level generalization.
  • MuJoCo: MuJoCo varies body mass across ten paired levels, and its levelwise advantage heatmaps order environments from lightest to heaviest.Shared and conditioned runs use the same level collection for each seed.

E.4 BipedalWalker

BipedalWalker evaluates critic-conditioning choices on pinned and unseen procedural terrains, finding that asymmetric conditioning preserves a shared actor while improving returns. Additional diagnostics and comparisons show that gains are not explained by value-network capacity or a systematic fitting penalty.

  • Experimental setup: BipedalWalker trains on 100 pinned terrains and evaluates on 100 unseen terrains generated with seed offset 500, preserving the same normal/hardcore split.The training set contains 10 normal and 90 hardcore terrains; testing uses one episode per unseen terrain.
  • Capacity control: Widening every value network by roughly five times leaves the shared critic at the same plateau while conditioned critics again reach roughly 150 or higher.The comparison holds conditioning mechanisms and all other settings fixed, indicating the performance gap is not a capacity artifact.
  • Conditioning choices: Separate value networks for every terrain learn substantially more slowly than FiLM and multihead conditioning.This comparison removes the shared value representation, unlike the successful conditioned designs.
  • Conditioning choices: Supplying the environment index to the actor as well as the multihead critic collapses both training return and return on unseen terrains.The unseen-terrain evaluation cannot use a valid index, motivating the asymmetric design.
  • Conditioning choices: Conditioning only the critic is the successful intervention, while the actor and critic representations remain shared.The environment index is used for value prediction without changing the actor.
  • Value loss diagnostics: Conditioned critics do not impose a systematic value-fitting penalty: early multihead excess closes on BipedalWalker, while losses are lower through most MuJoCo training and often favorable on Procgen.On Procgen, the ordering varies by game rather than uniformly favoring conditioning.
Loading 2608.26481v1…