PaperlayerView exact revision on arXiv

Source-linked AI summary

Single or Multiple Policies for Phase-Structured Reinforcement Learning?

Guilhem Loussouarn, Nancy Nayak, Kin K. Leung

arXiv:2610.03475v1cs.LGcs.AIcs.SYeess.SY

TL;DR

The paper asks why phase-specific policies can outperform a shared policy even though the shared policy can represent the same solutions. It develops a regime-based comparison of transient handoff costs and quasi-stationary specialization gains, then finds that the preferred architecture depends on learning constraints and phase coupling.

  • Problem

    Although shared and phase-specific policies can have equivalent representational capability, their practical performance can differ under finite learning and optimization.

    Source: §1 · p. 1, (a) Phase structure and within-phase… · p. 2, §3 · p. 5

  • Method

    The paper decomposes phases into transient and quasi-stationary regimes and uses handoff costs, settled gains, heterogeneity, and phase-local data to diagnose architecture choice.

    Source: (a) Phase structure and within-phase… · pp. 2–3

  • Results

    Across the three environments, the ranking of shared and specialized architectures can reverse as phase count, duration, heterogeneity, budget, or boundary dependence changes.

    Source: (a) Phase structure and within-phase… · p. 2, §6 · p. 10

  • Takeaways & Limitations

    Sharing is favored by behavioral overlap, constrained learning budgets, or substantial boundary-state coordination; specialization is not always preferable.

    Source: §6 · p. 10

    The study focuses on observable phase structures; unknown phase structures and coordinated multiple policies remain future-work directions.

    Source: §3 · p. 4, §6 · p. 10

Abstract

from arXiv · show

Many reinforcement-learning (RL) problems are non-stationary yet structured and can be decomposed into phases, each with its own transition probabilities and reward functions. When the phase sequence is known, the common solution augments the state with information to satisfy the Markovian property and applies standard RL techniques. However, prior work finds that the multi-policy approach for different phases can outperform a single state-augmented policy shared among the phases, for reasons that remain unclear. In this work, we first show that the shared policy can theoretically achieve performance of any multi-policy solution. However, whether a multi-policy solution can perform better than the corresponding single shared policy in practice depends on function approximation, learning and optimization processes, as well as, for multi-policy solutions, the sample efficiency and loss of continuity from one policy to another. We propose a regime-based phase decomposition method to identify which policy can provide better performance. The method is based on consideration of the duration of transient system dynamics relative to the duration of the quasi-stationary period. Numerical experiments are conducted with different non-stationary RL problems to validate our four major hypotheses: (a) longer phase durations favor multi-policies, (b) the heterogeneity between phases increases the burden on single policy, (c) multi-policies need sufficient data for each phase, and (d) environment-specific transition dynamics between phases can affect which policy is preferable.

1 Introduction

Phase-structured MDPs have deterministic, observed phase schedules, with each phase governed by its own transitions and rewards. The paper asks when to share policy parameters across phases and proposes a transient–quasi-stationary framework for comparing sharing with specialization.

1 Introduction

  • A shared phase-conditioned policy can represent any collection of phase-specific policies, so practical performance differences are learning effects rather than representational limits.The paper identifies function approximation, learning, optimization, sample efficiency, and continuity across phase changes as relevant factors.Source: §1 · p. 1, (a) Phase structure and within-phase… · p. 2

(a) Phase structure and within-phase regimes

  • The proposed regime decomposition separates transient handoff dynamics from quasi-stationary learning and relates architecture choice to heterogeneity, phase-local data, and coordination.The diagnostic framework weighs handoff cost against settled specialization gain, alongside task heterogeneity and the data budget available to each phase.Source: (a) Phase structure and within-phase… · p. 2, Figure 1 · p. 2

1 Introduction

  • The paper formalizes phase-structured MDPs and identifies scheduler information that restores a time-homogeneous MDP.The scope is deterministic scheduled phases whose active phase is observed; latent or history-dependent phase processes are left for future work.Source: §1 · p. 1, (a) Phase structure and within-phase… · p. 2

(a) Phase structure and within-phase regimes

  • The paper tests its predictions with shared, multihead, and multipolicy PPO across HAPS coverage control, PhasePendulum, and PhaseGridWorld.The experiments also probe capacity, gradients, warm starts, distillation, and controlled handoff interventions.Source: (a) Phase structure and within-phase… · p. 3

2 Related Work

Prior work addresses structured changes, parameter sharing, transfer, and model capacity, but does not fully capture how consecutive phases couple through the trajectory state. This paper uses that distinction to motivate studying cross-phase coordination alongside sharing and transfer.

Structured time-varying MDPs

  • Phase-structured RL is distinguished from periodic and hidden-change models by its deterministic, observed change structure.Periodic MDPs commonly augment the state with a period index, while switching non-stationary MDPs may use a latent Markov chain.Source: §2 · p. 3

Periodic multi-policy reinforcement learning

  • Prior multi-policy RL studies report gains over shared-policy baselines but do not explain the performance gap.The cited examples include periodic network resource allocation and HAPS control [16].Source: §2 · p. 3

Multi-task learning, modularity, and transfer

  • Multi-task sharing can improve data efficiency through transfer but may cause negative transfer when tasks require incompatible updates.The paper evaluates PCGrad and CAGrad and uses phase-specific gradient geometry as a local diagnostic [18] [19].Source: §2 · p. 3
  • Existing multi-task formulations do not necessarily address cross-phase coordination because phases are consecutive parts of one trajectory.The policy in one phase determines the state passed into the next.Source: §2 · p. 3

Capacity and irrelevant features

  • Capacity and irrelevant features can affect finite-sample deep-RL learning, motivating controls for parameter count and phase-information extraction.The paper also uses multihead and warm-start baselines motivated by modular and transfer methods.Source: §2 · pp. 3–4

3 Phase-Structured MDPs

The paper formalizes deterministic phase schedules and explains how scheduler information can restore Markovianity. It also establishes that shared and specialized policy classes are equivalent in representational capability, while finite-resource training can yield different outcomes.

Phases and schedules

  • A phase-structured MDP consists of stationary phase-specific dynamics and rewards organized by a deterministic schedule.Phase occurrences can have different durations, and the schedule may revisit the same phase.Source: §3 · p. 4, Table 1 · p. 4

Scheduler state and Markovianity

  • A sufficient scheduler state augments the physical state to recover a time-homogeneous MDP and Markovianity.For periodic phases, the phase index and within-phase clock suffice; global time can restore Markovianity but prevent data reuse across cycles.Source: §3 · p. 4

Phase-local models and handoffs

  • A phase's entry-state distribution depends on preceding policies, so phases are not generally independent.This cross-phase dependence is the handoff effect.Source: §3 · p. 5

Shared policy and specialized policies

  • Shared and specialized policies have equivalent representational capability, but finite networks, data, and gradient-based optimization can lead them to different trained solutions.The shared policy conditions on phase, while specialized policies use a separate policy for each phase.Source: §3 · p. 5

4 When to Share and When to Specialize

The paper compares specialization's quasi-stationary learning gains with its transient handoff costs. The resulting architecture choice depends on phase duration, heterogeneity, phase-local data, and the degree of coupling across phase boundaries.

4.1 Transient and Quasi-Stationary Regimes

  • Transient periods retain sensitivity to the inherited entry state, whereas quasi-stationary periods are governed mainly by the current phase's dynamics and objective.Quasi-stationarity requires only that the entry-state effect on the return quantities becomes small, not convergence to an invariant distribution.Source: §4.1 · p. 5
  • The transient length can differ across occurrences and may span an entire phase, so periodicity alone does not guarantee recovery before the next boundary.The remaining phase duration is the quasi-stationary window.Source: §4.1 · p. 5
  • Specialization may improve phase-local approximation in the quasi-stationary regime, while sharing can account for continuation value and pool data across phases.With a balanced allocation, each of M specialists receives about B/M of total budget B.Source: §4.1 · pp. 5–6

4.2 Settled Gains and Transient Handoff Costs

  • Specialization is locally advantageous when its quasi-stationary gain exceeds its transient deficit.Handoff cost is larger when returns are sensitive to entry states and recovery is slow; it is smaller when phases reset independently or dynamics contract quickly.Source: §4.2 · p. 6
  • Heterogeneity can widen the shared learner's learning gap, while transferable representations and pooled data can narrow it.A specialist's gain depends on receiving enough phase-local data to avoid data starvation.Source: §4.2 · pp. 6–7

4.3 From Regime Quantities to Architecture Choice

  • The architecture diagnostic favors specialization when settled gains outweigh handoff and budget penalties, and sharing when transfer, scarce data, or boundary sensitivity dominates.A multihead architecture is suggested for unresolved cases, sharing features while allowing phase-specific action mappings.Source: §4.3 · p. 7

5 Experiments and Results

Across three phase-structured environments, the experiments test when shared, specialized, and partially shared policies perform best. Rankings depend on phase duration, behavioral diversity, specialist data, and handoff dependence.

5 Experiments and Results

  • Experiments compare Single+clock, independent Multipolicy specialists, and a shared-trunk Multihead policy across HAPS, PhasePendulum, and PhaseGridWorld using PPO.The environments vary in phase count, duration, and cycles; all methods use the same PPO hyperparameters unless stated otherwise, and results report raw reward per environment step.Source: §5 · pp. 7–8

5.1 H1: Longer Phases Favour Specialization with multi-policy

  • At M = 20, Single+clock leads at T = 100, methods are comparable at T = 200, and Multipolicy leads at T = 500 and T = 2000.PhasePendulum shows a similar pattern: the single policy leads at T ∈{100, 200}, while multi-policy has the best mean at T = 500.Source: §5.1 · p. 8, Figure 2 · p. 8

5.2 H2: Behavioural Diversity Increases the Burden on Single Shared Policy

  • Greater phase diversity can widen the shared policy’s performance gap, while architectures perform comparably when adjacent phases share an optimal policy.In HAPS, increasing phase count degrades Single+Clock but leaves Multipolicy flat; in PhaseGridWorld, Multipolicy outperforms when phases require distinct optimal policies. The shared policy’s cosine similarity falls from 0.675 to 0.195 as M grows from 2 to 20, while alignment remains close to one, indicating that destructive interference is not the main burden.Source: §5.2 · p. 9

5.3 H3: Specialization Requires Sufficient Phase-Local Data

  • With a fixed interaction budget, dividing samples among specialists can leave each below the data needed to learn its phase, allowing Single+clock to outperform Multipolicy.At B = 5M timesteps in PhasePendulum, each specialist receives about 500k interactions; Multipolicy approaches Single+clock’s asymptotic performance when the total budget reaches B ≥10M.Source: §5.3 · p. 9

5.4 H4: Handoff Dependence Can Reverse the Ranking

  • GridWorld handoff dependence can reverse rankings: trapped carryover penalizes Multipolicy, reachable resets favor it, and Multipolicy dominates in recoverable bottlenecks under either condition.In the Closed Trap, Single+clock and Multihead choose the safe goal in almost all episodes, while Multipolicy chooses the trap goal in 90%.Source: §5.4 · pp. 9–10

6 Conclusion

The conclusion frames policy choice as a trade-off shaped by learning dynamics and phase coupling, despite equal representational capability with sufficient scheduler information. It also points to adaptive architectures and broader phase settings as future work.

6 Conclusion

  • With sufficient scheduler information, shared and multiple policies have equal representational capability, so performance differences arise from environment and learning dynamics.Sharing is favored by behavioral overlap, constrained budgets, or costly boundary coordination; specialization is favored by heterogeneity, sufficient specialist data, and quasi-stationary gains exceeding handoff costs.Source: §6 · p. 10
  • Multihead networks offer an intermediate design between fully shared and phase-specific policies.Source: §6 · p. 10
  • Future work should study adaptive architectures that share transferable features, specialize where local optimization is hard, and propagate continuation values across phase boundaries.The conclusion also identifies unknown phase structures and coordinated multiple policies as future directions.Source: §6 · p. 10

A Appendix

The appendix organizes supporting definitions, derivations, experimental details, and additional results into four parts.

A Appendix

  • Appendix A.1 defines quantities and assembles the predictive certificate; A.2 groups derivations, A.3 specifies environments and protocols, and A.4 presents additional results and diagnostics.Source: A Appendix · p. 13

A.1 Supporting Mechanisms and Bounds

This section provides definitions and results used across the paper’s sections.

A.1 Supporting Mechanisms and Bounds

  • The section supplies definitions and results used by different parts of the work.Source: §A.1 · p. 13

A.1.1 Schedule Sufficiency and Markovian Augmentation

The scheduler state augments the physical state with phase and schedule context so that transitions satisfy a formal sufficiency condition. For periodic schedules, a phase index and within-phase clock provide compact Markovian augmentation, with extra occurrence information needed when a phase repeats differently.

A.1.1 Schedule Sufficiency and Markovian Augmentation

  • The scheduler state z_t=(σ(t), i_t) includes the active phase and auxiliary schedule context to represent phase transitions in the augmented MDP.The context can include elapsed phase time or the global step index.Source: §A.1.1 · p. 13
  • The scheduler state is sufficient when the joint transition dynamics factor into the active phase’s physical transition kernel and an exogenous schedule update kernel.Under this condition, the augmented state forms a time-homogeneous MDP.Source: §A.1.1 · p. 13
  • For a deterministic periodic cycle, the phase index and within-phase clock form a compact scheduler state that makes the augmented process Markovian and time-homogeneous.The within-phase clock is defined from the global step modulo the period and the active phase’s entry offset.Source: §A.1.1 · p. 13
  • If a phase recurs within one macro-cycle with different durations or successors, the scheduler context must also encode its occurrence index.Table 3 illustrates the unique sequence of phase-and-clock pairs for a periodic four-phase schedule.Source: §A.1.1 · p. 13, Table 3 · p. 13

A.1.2 A Transient–Quasi-Stationary View

The analysis separates each phase occurrence into an early transient and a later quasi-stationary period, then decomposes the learned return gap between specialized and shared policies. Specialization is favored only when settled local gains dominate transient handoff costs; phase length alone is not sufficient.

A.1.2 A Transient–Quasi-Stationary View

  • The ε-recovery time is the first step after which the state distribution remains within ε of a possible near-invariant law, and it depends on both policy and entry law.It is therefore not a fixed property of the phase.Source: §A.1.2 · p. 14
  • The return-gap decomposition is evaluated over phase occurrences using normalized evaluation weights that sum to one.Each occurrence’s normalized return difference is weighted before summing across occurrences.Source: §A.1.2 · p. 14
  • The learned-policy return gap is decomposed into a settled specialization gain and a transient handoff cost, so longer phases favor specialists only when a quasi-stationary region yields positive local gains.The comparison is between learned shared and learned specialized policies, not the best policies in unrestricted classes.Source: §A.1.2 · pp. 14–15

A.1.3 Hand off Costs with Two Goals

The two-goal example contrasts a phase-local choice that optimizes immediate phase return with a coordinated choice that accounts for continuation value. It formalizes how a handoff-sensitive goal choice can create a return loss when the stated inequalities hold.

A.1.3 Hand off Costs with Two Goals

  • When F_00(s)>F_01(s), a phase-local specialist prefers the goal with higher immediate phase-0 reward, while a full-trajectory controller accounts for phase-1 continuation value.The passage states that the coordinated controller prefers the safe-handoff goal if and only if a further condition holds, but that condition is not included here.Source: §A.1.3 · p. 15
  • When both stated inequalities hold, the phase-local choice incurs a return loss relative to the coordinated choice.The comparison concerns two consecutive phases and the entry-state distribution induced for phase 1.Source: §A.1.3 · p. 15

A.1.4 Sample Allocation and Sample Thresholds

Specialists divide the total interaction budget into phase-local allocations, potentially leaving each with fewer steps than a shared model. A local sample threshold specifies the allocation needed to preserve positive specialization gain, and balanced sampling requires a total budget at least M times the largest threshold.

A.1.4 Sample Allocation and Sample Thresholds

  • An ensemble partitions total budget B into local allocations B_m=ζ_mB, so each specialist receives fewer environment steps than a shared model trained on B.Balanced sampling assigns ζ_m=1/M.Source: §A.1.4 · p. 15
  • The local sample threshold B_0,m(δ) is the minimum allocation needed for specialist m to recover its asymptotic return within tolerance δ.It is used to ensure local learning starvation does not erase settled specialization gain.Source: §A.1.4 · p. 15
  • If unlimited-sample specialization gain exceeds δ and the total budget satisfies the proposition’s condition, every specialist receives enough samples to retain strictly positive local gain.The condition simplifies under balanced sampling to B ≥ M max_m B_0,m(δ).Source: §A.1.4 · p. 15

A.1.5 Gradient Geometry as an Optimization Mechanism Test

The section tests whether shared-policy optimization is limited by destructive interference or by gradient dilution. Its diagnostics identify gradient dilution as the primary bottleneck in HAPS.

A.1.5 Gradient Geometry as an Optimization Mechanism Test

  • The test evaluates phase-restricted gradients for phase-local objectives at a common parameter vector under a fixed reference data distribution.The empirical check differentiates the phase-restricted clipped PPO actor surrogate at commonly trained checkpoints using global advantage normalization.Source: §A.1.5 · p. 16
  • A phase’s local objective is harmed only when its gradient has negative inner product with the aggregate update; negative pairwise inner products alone are insufficient.The first-order decomposition separates phase self-improvement from cross-phase transfer.Source: §A.1.5 · p. 16
  • Under mutually orthogonal phase gradients, Ag equals 1 for any phase count, while Cg > 0 indicates negative cross-phase terms dominate the aggregate update.Source: §A.1.5 · p. 16
  • A 1/M cosine-similarity decline indicates the shared update spreads across orthogonal task directions—gradient dilution, rather than destructive multi-task interference [34].Source: §A.1.5 · p. 16
  • Ag ≈1.0, fewer than 2% of phases have negative inner product with the aggregate gradient, and cosine alignment tracks 1/M, identifying gradient dilution as HAPS’s primary optimization bottleneck.The empirical evaluation rejects destructive interference.Source: §A.1.5 · p. 16

A.2 Proofs and Derivations

This appendix develops formal proofs and derivations for the paper’s structural and policy-equivalence claims, including conditions involving handoffs, data budgets, and gradient alignment. It also documents baselines and experimental protocols and presents extended results and controls across HAPS, PhasePendulum, and PhaseGridWorld concerning phase duration, heterogeneity, phase-local data, boundary dynamics, and policy handoffs.

A.2.1 Structural Results

  • A sufficient scheduler state makes the physical state and scheduler state together a time-homogeneous MDP, restoring Markovianity and defining stationary transitions and rewards.For deterministic periodic phases, the phase index and within-phase clock form a compact sufficient scheduler state that recurs every cycle.Source: §A.2.1 · pp. 17–19
  • Unrestricted shared phase-conditioned policies and ensembles of phase-specialized policies are bijectively equivalent and can induce identical trajectories and expected returns.Their functional expressivity is the same, but finite-data optimization can yield different realized solutions.Source: §A.2.1 · p. 19, §A.2.2 · p. 19

A.2.2 Geometric Intuition of Policy Class Equivalence

  • Specialists optimize phase-local returns without cross-phase coordination, so their assembled performance can improve with compatible handoffs or worsen with handoff mismatch or data starvation.The geometric account attributes this to independent local optimization and phase-boundary handoffs, despite zero projection loss in assembling specialists.Source: §A.2.2 · pp. 19–20

A.2.3 Transient Handoff, Recovery, and Regime Bounds

  • When the locally higher-reward goal creates a worse phase-1 entry state, a full-trajectory controller chooses the safer handoff if its continuation value outweighs the immediate-reward advantage.Under that condition, the uncoordinated specialist’s choice produces a return loss relative to the coordinated controller.Source: §A.2.3 · p. 21

A.2.4 Sample Allocation and Sample Thresholds

  • If each specialist’s unlimited-sample advantage exceeds δ, a sufficient total interaction budget ensures every specialist receives enough samples to retain positive local gain.With balanced sampling, the condition is B ≥ M max_m B0,m(δ).Source: §A.2.4 · p. 21

A.2.5 Gradient Geometry and Optimization Proofs

  • A shared gradient update can decrease a phase’s local objective when its aggregate update has negative inner product with that phase’s gradient.Under the stated orthogonality assumptions, the appendix also derives a scaling baseline for aggregate-gradient alignment, including a generalization to unequal gradient norms.Source: §A.2.5 · pp. 22–23
  • With balanced weights, the aggregate gradient is defined across phase gradients; mutual orthogonality yields a baseline that generalizes to unequal gradient norms.The proof begins from equal gradient norms and orthogonal cross-terms, then states the generalized orthogonal baseline for unequal norms.Source: §A.2.5 · p. 23

A.3.1 Architectures and Common Training Protocol

  • Architectures differ in phase conditioning and parameter sharing, and all are evaluated on complete cyclic trajectories; standard multipolicies train separately from phase-local resets.Single+clock receives state, phase code, and local clock; Multihead shares its body and critic but uses phase-specific action heads; Multipolicy uses independent actor–critics. Final returns are unweighted cross-seed means with matched-seed differences and paired 95% percentile bootstrap intervals.Source: §A.3.1 · pp. 23–24, §A.3.5 · p. 30

A.3.2 HAPS Coverage Environment

  • The HAPS experiments vary phase count and dwell time, evaluating complete cycles that retain state across boundaries and therefore include handoff effects.The main phase-count sweep uses M = 2, 5, 10, 20 at T = 500; the dwell-time sweep uses T = 100, 200, 500, 2000, with five seeds.Source: §A.3.2 · p. 26

A.3.3 PhasePendulum Environment and Protocol

  • PhasePendulum varies phase-target heterogeneity while preserving continuous physical state at boundaries; its heterogeneity sweep compares architectures within each setting rather than interpreting rewards across settings causally.The protocol fixes M = 10 and T = 1000, with h from 0 to 1; each specialist receives 0.5M interactions.Source: §A.3.3 · p. 26

A.3.4 PhaseGridWorld Environment and Protocol

  • PhaseGridWorld protocols test carried versus reset boundary states and controlled handoffs, including whether a locally rewarding goal leaves the agent unable to reach the next phase’s goal.The two-goal construction contrasts a higher-reward goal that creates an unreachable phase-1 entry with a lower-reward goal that preserves reachability.Source: §A.3.4 · pp. 27, 29

A.4.1 HAPS Experiments Results

  • At M = 20, rankings shift with dwell time: Single+clock leads at T = 100, methods are comparable at T = 200, and Multipolicy leads at T = 500 and 2000.The reported rewards are 31.75 vs. 20.88 at T = 100; 32.26 and 31.90 at T = 200, with no consistent seed-level winner; 42.36 vs. 34.15 at T = 500; and 45.76 vs. 33.16 at T = 2000.Source: §A.4.1 · p. 30
  • Specialists retain a high-M advantage despite increased shared-policy budgets, while PCGrad and CAGrad improve on the single-policy baseline but remain below Multipolicy.Specialists outperform the shared baseline at B0 = 250k per phase, or 5M total interactions, with no late-budget crossover. At 5M steps, PCGrad and CAGrad score 28.87 and 28.05 versus 25.54 for the standard single-policy baseline and 43.33 for Multipolicy.Source: §A.4.1 · p. 30
  • On HAPS, teacher-guided specialists match unconstrained specialists, whereas distilling specialists into a shared student performs poorly even with added capacity and training.Teacher-guided specialists score 46.58 versus 46.72 for standard specialists; the distilled student scores 12.95, or 27.73 after scaling the architecture and adding 5M student steps, below the 30.10 shared baseline and 46.72 specialist teachers.Source: §A.4.1 · pp. 30–31
  • Widening or deepening the shared HAPS policy does not close its gap to specialists, and gradient diagnostics find little aggregate harm to phase objectives.At M = 20 and T = 500, deeper PPO networks also showed optimization instability; across phase counts, aggregate alignment stayed near Ag = 1 and at most 2% of phases were harmed, indicating gradient dilution rather than destructive aggregate interference.Source: §A.4.1 · pp. 31–32, Table 9 · p. 33
  • HAPS phase ordering affects returns: adjacent phases are easiest, while random and maximum-jump schedules widen the multi-policy advantage over multi-head baselines.A within-phase clock leaves multi-policy performance essentially unchanged, but explicit handoff and handoff-warm-start objectives degrade it; budget-matched shared initialization also reduces specialist performance, consistent with negative transfer during fine-tuning.Source: §A.4.1 · pp. 31–32, Table 7 · p. 32, Table 8 · p. 32

A.4.2 PhasePendulum Experiments Results

  • At high PhasePendulum target heterogeneity, Multipolicy outperforms Multihead and Single+clock, while identical targets favor sharing unless specialists receive more data.At h = 0.75 and 1.00, Multipolicy wins across five matched seeds; at h = 0, extending the total budget from 5M to 20M lets it match the shared baseline, while higher entropy destabilizes action variance without restoring performance.Source: §A.4.2 · p. 33, Table 10 · p. 33, Figure 10 · p. 39
  • Multipolicy beats Single+clock in 19 of 20 PhasePendulum runs, with mean advantages of +0.136, +0.119, +0.135, and +0.242 at M = 2, 5, 10, and 20.At M = 20, Multihead’s mean score is slightly higher than Multipolicy’s (−2.236 vs. −2.260). Multipolicy leads at T = 1000 and 2000 for M = 10, but Multihead leads at T = 500 and 1000 for M = 20; concentrated or small-jump targets and dominant Phase 0 rewards favor sharing, while maximum-jump targets favor specialization.Source: §A.4.2 · pp. 33–34, Table 11 · p. 34

A.4.3 GridWorld Experiment Results

  • At Z = 128 distractors, Multihead and warm-started clock-aware specialists score 0.0968 and 0.1178, exceeding Single+clock’s 0.0629.Single+clock retains high performance through Z = 32. In the single-phase budget calibration, it reaches threshold across all three seeds at 500k interactions, versus 750k for Multipolicy; later experiments therefore allocate at least 1M interactions per specialist. Clock information may also help avoid repetitive behaviors.Source: §A.4.3 · pp. 34–35, Table 13 · p. 35, Table 12 · p. 35
  • At GridWorld density 0.40, Single+clock leads under a 3M total budget, while Multi+clock beats it at B0 = 1.5M and M = 10.At density 0.25, Multi+clock matches sharing (0.0992 vs. 0.0991) at that per-phase budget; Warm+clock helps at low density, but direct transition-aware warm starts underperform across settings. Thus sample starvation alone does not explain all modular failures.Source: §A.4.3 · pp. 35–36, Table 14 · p. 36, Table 15 · p. 36
  • At ρ = 0.9, Single+clock and Multihead choose the safe GridWorld handoff in 96% and 98% of episodes, versus 10% for independent specialists.Specialists optimize phase-local returns without downstream continuation-value training, so they tend to pursue the larger immediate Phase 0 reward rather than preserve Phase 1 reachability.Source: §A.4.3 · p. 36, Figure 12 · p. 37
  • In the Closed trap, specialist-minus-shared returns shift from −0.0798 under state carryover to +0.0060 after reachable-state resets; Bottleneck favors specialization under both entry conditions.The interventions support the regime decomposition: recoverable transitions incur finite transient costs that settled specialist gains can overcome, whereas unrecoverable transitions can make transient costs span the phase. Aligned phases show no specialist advantage.Source: §A.4.3 · p. 37, Table 16 · p. 37, Figure 13 · p. 40
Loading 2610.03475v1…