PaperlayerView exact revision on arXiv

Source-linked AI summary

Dynamic Context Scheduling: Learning Beyond the Static Universe

Martin Mráz, André Biedenkapp

arXiv:2608.20799v1cs.AI

TL;DR

Policies trained under static context can be brittle when environment parameters shift, motivating richer contextual reinforcement-learning training signals. The paper introduces scheduled intra-episode context variation through DYNAMICCARLENV and finds that dynamic schedules generally match or outperform static baselines across three CARL environments. Automated multi-stage search improves over static training, but its benefit beyond the best single-stage schedule remains unresolved.

  • Problem

    Policies trained under narrow, stationary dynamics can fail when deployment parameters shift, while contextual RL usually treats context as static within an episode.

  • Method

    DYNAMICCARLENV trains policies with deterministic, parameterized intra-episode context schedules while evaluating them under predefined static contexts.

  • Results

    Across CartPole, BipedalWalker, and CarRacing, dynamic schedules consistently match or outperform static baselines, with largest gains on BipedalWalker observed and CarRacing blind conditions.

  • Takeaways & Limitations

    State-space coverage is indistinguishable between dynamic and static schedulers, indicating that the reported benefit derives from temporal training structure rather than broader state exploration.

  • Takeaways & Limitations

    Multi-stage search uses restricted scheduler families and few trial seeds, so its benefit beyond a well-chosen single-stage schedule cannot be concluded.

Abstract

from arXiv · show

We study dynamic context scheduling as a training instrument for contextual re- inforcement learning. Rather than treating intra-episode context variation as a deployment reality, we treat it as a controlled shaping mechanism. Thereby, context evolves within each training episode according to a predetermined schedule, expos- ing the policy to a richer and more temporally structured region of the environment parameter space. We introduce DYNAMICCARLENV, a framework that wraps contextual environments with pluggable schedule families, such as sinusoidal off- sets or cosine annealing. Across CartPole, BipedalWalker and VehicleRacing with CARL contextualization, we show that dynamic schedules match or outperform static context baselines in the out-of-distribution (OOD) regimes. Interestingly, for the more complex BipedalWalker and VehicleRacing environments we also achieve higher in-distribution (ID) evaluation performance. Preliminary findings indicate that automatic search for multi-stage curricula can successfully discover schedules that improve generalization, performing comparably to extensive grid search over single-stage schedulers.

1 Introduction

Policies trained under narrow, stationary dynamics can fail when environment parameters shift, while contextual RL typically treats context as static within episodes. This paper proposes dynamic context variation as a controlled training instrument for broader generalization.

  • Motivation: Stationary training can produce large return drops when deployment dynamics differ in friction, payload, actuator gain, or gravity.The mismatch arises because training exposes the policy to a single-dynamics regime while testing may involve different dynamics.
  • Related approaches: Domain randomization broadens training experiences by sampling related environments without explicitly informing agents about environment changes.The approach is described as unstructured exposure to varied environment instances.
  • Contextual reinforcement learning: Contextual RL conditions policies on explicit context variables whose values parameterize transitions and rewards.This setting makes relationships among environments available to the agent, unlike approaches that hide environment changes.
  • Motivation: Most contextual RL approaches sample context once per episode, discarding temporal variation that could provide structured training signal.Static context improves training stability and credit assignment but restricts exposure to continuously changing dynamics.
  • Proposed direction: Dynamic schedules evolve context during training episodes for regularization and exploration, while evaluation remains under static contexts.The schedules include sinusoidal sweeps, piecewise regimes, linear drift, random walks, cosine annealing, and hybrid compositions.
  • Contributions: The work evaluates scheduling families, extends CARL with dynamic contexts, and analyzes whether improved generalization requires broader state-space coverage.Its stated contributions include a training paradigm, empirical schedule comparisons, an open-source extension, and coverage analysis.

2 Related Work

Related work frames contextual RL as learning across a family of parameterized MDPs, while dynamic contextual MDPs allow context to evolve over time. Existing contextual RL research nevertheless mostly assumes static episode-level context.

  • Contextual MDPs: A contextual MDP augments an MDP with context variables that parameterize transition, reward, and initial-state functions.A cMDP represents a family of related MDPs and can be viewed as a subclass of partially observable MDPs.
  • Dynamic contextual MDPs: Dynamic contextual MDPs extend contextual MDPs by allowing contexts to be history-dependent and evolve over time.This formalism provides the conceptual setting for time-varying context.
  • Benchmarks and evaluation: The cMDP setting supports zero-shot generalizability evaluation, and CARL implements this protocol with physical contexts such as gravity and friction.CARL extends common reinforcement-learning benchmarks with contextualized environment parameters.
  • Research gap: Most contextual RL studies treat context as monolithic and static, while relatively few works investigate dynamic context changes.Prior dynamic-context work includes learning when environmental shifts occur, but context-estimation approaches may assume changes between observations.

3 Dynamic Context Scheduling

DYNAMICCARLENV replaces static episode context with a parameterized schedule that updates context during training, while evaluation uses fixed contexts. It supports multiple observability modes and schedule families, including smooth, stochastic, abrupt, and composite trajectories.

  • Framework: Training uses evolving context, but evaluation uses predefined static contexts to measure generalization to fixed dynamics rather than tracking ability.The scheduled context governs transitions during training while static contexts define evaluation conditions.
  • Framework: DYNAMICCARLENV advances a parametric context schedule at every step, applies the updated context to the simulator, and optionally exposes it to the policy.The wrapper separates scheduling from environment internals through context getter and setter callables.
  • Observability modes: The framework distinguishes Live observability, which exposes evolving context, from None observability, which hides context and isolates temporal structure.None corresponds to dynamic changes without explicit context conditioning; observable context is normalized before being appended to the state.
  • Framework: The scheduler produces per-step context from parameters such as amplitude, period, drift rate, dwell times, and change points, while the base environment receives c0 at reset.Adapters control simulator physics, policy observations, and logging of states, actions, rewards, contexts, and visible context.
  • Schedule families: Implemented families include sinusoidal sweeps, cosine annealing, continuous increments, piecewise-constant shifts, and bounded random walks.Composite families add structures such as sinusoidal jumps, Ornstein–Uhlenbeck dynamics, and phased OU schedules.
  • Schedule families: Context trajectories can also include stochastic jumps, whose random timing and magnitude produce heavy-tailed changes.Composite schedules combine families, and amplitude or step-size parameters are expressed relative to the full context range.
  • Curriculum scheduling: Multi-stage schedules divide training into fixed-length stages, each using an independently configured scheduler from the available families.The design supports changing temporal structure across training phases rather than applying one family throughout the run.

4 Experiments

The experiments compare dynamic context schedules with static training across CARL environments, observability modes, and evaluation regimes. They also examine multi-stage curriculum search and whether dynamic scheduling improves generalization through broader state-space coverage.

  • Evaluation regimes: CarRacing shows a strong observability reversal: blind dynamic policies outperform observed policies, while both observed conditions produce negative IQM.The best blind dynamic scheduler reaches 592.3 IQM, a +489 IQM improvement over static blind training; observed policies later degrade after learning during training.
  • Evaluation regimes: CartPole dynamic schedules limit the OOD-high performance drop that static policies develop as training progresses.Dynamic training provides modest gains in ID and OOD-low regions while preventing or accelerating recovery from static-training degradation in OOD-high evaluation.
  • Multi-stage schedulers: Optuna multi-stage curricula generally improve over static baselines but do not consistently exceed the best single-stage dynamic schedulers.Top configurations often use an idle first stage followed by an active sinusoidal stage, and the number of active stages is essentially uncorrelated with final performance.
  • Experimental limitations: The multi-stage comparison is limited by restricted scheduler and observability choices, few search seeds, and a shorter BipedalWalker training budget.After retraining with more seeds, configuration rankings shift substantially, so the benefit of multi-stage structure beyond a well-chosen single stage remains unresolved.
  • State-space coverage: Coverage remains statistically indistinguishable across dynamic and static schedules, suggesting that dynamic scheduling restructures temporal training signals rather than expanding visited observation states.CartPole coverage clusters between 6.54% and 6.83% with ANOVA p > 0.05; BipedalWalker projections show the same null result, and most cells are discovered within the first 50 k steps.
  • Cross-environment results: Dynamic schedulers outperform static training across CartPole, BipedalWalker, and CarRacing, with the largest gains in observed Walker and blind CarRacing.The best dynamic scheduler adds +39 IQM observed and +53 blind on CartPole, +120 IQM for observed Walker, and +489 IQM for blind CarRacing.

5 Discussion and Conclusion

Dynamic context scheduling improves zero-shot generalization and OOD robustness by varying physical parameters within training episodes, while its benefits depend on context observability and introduce tuning costs.

  • Discussion and Conclusion: Dynamic schedules match or outperform static baselines across CartPole, BipedalWalker, and CarRacing, improving zero-shot generalization and OOD robustness.The approach treats continuously evolving physical parameters as a training instrument rather than a deployment model.
  • Discussion and Conclusion: Coverage analysis indicates that these gains arise from restructuring the temporal training signal rather than broader state-space exploration.Dynamic scheduling is described as a regularizer against narrow, over-specialized behavioral manifolds.
  • Discussion and Conclusion: Context observability benefits simpler tasks, whereas blind dynamic scheduling is superior in complex environments such as CarRacing.The conclusion emphasizes that observability must be selected with care rather than assumed universally beneficial.
  • Schedulers: The framework evaluates eleven schedule families, including sinusoidal oscillation, stochastic jumps, Ornstein-Uhlenbeck drift, phased OU, and identity.These families vary temporal structure through oscillation, jumps, drift, or fixed context.
  • Schedulers: Identity holds context at its episode-initial value and provides the within-experiment static reference for schedule-induced variation.It is defined as the degenerate special case of a dynamic schedule.

B Hyperparameters and Training Details

Experiments use fixed-default PPO configurations across contextualized environments, with explicitly specified context ranges, observation bounds, normalization intervals, and payload parameters.

  • Training setup: All agents use PPO through Stable-Baselines3 with untuned default hyperparameters, isolating scheduling strategy from incidental optimizer tuning.CartPole and BipedalWalker use two 64-unit fully connected layers; CarRacing uses a CNN plus context MLP branch.
  • Training setup: The main experiments use one parallel training environment, while evaluation uses four parallel workers across conditions.Auxiliary parallel-environment runs are excluded from the main comparison.
  • Context configuration: CartPole pole-length normalization uses the manually specified interval (0.05, 2.0) m for live context observation.The interval is chosen as a conservative physically plausible operating range and motivated by PPO stability.
  • Context configuration: The CartPole pole-length pool-7 contains seven midpoint-inserted values from 0.35 to 0.75, while force magnitude and gravity use pool-5 values from 5.0 to 15.0.The pool construction provides discrete training contexts for the main CartPole experiments.
  • Payload contexts: BipedalWalker varies COM_X using a 2.0 kg payload over [−0.6, +0.6] m, while CarRacing varies COM_X and COM_Y over [−0.8, +0.8] m with a 1.0 kg payload.Both payload configurations use conservatively wider normalization bounds of (−2.0, +2.0) m per applicable axis.

D.1 Cross-Environment Scheduler Analysis

Across environments, dynamic schedulers generally outperform static baselines, but scheduler rankings and the value of context observability vary by environment and mode.

  • Baseline comparisons: 89–100% of context-observed scheduler families outperform the corresponding static observed baseline across environments.Blind scheduling is more nuanced, with CartPole and Walker showing roughly mixed-to-favorable results.
  • Scheduler ranking: No single scheduler family dominates all environments and modes; Sinusoidal, Lévy Walk, and Sudden Jump rank most consistently above static on average.Piecewise Constant ranks highest overall but was evaluated only on CartPole in the cited analysis.
  • Context observability: CartPole generally benefits from observing context, CarRacing benefits strongly from blindness, and Walker has mixed observability results by family.The environment-level reversal persists across nearly all scheduler families, indicating an environment-driven pattern.
  • Context observability: Cosine Annealing, Random Walk, and Sinusoidal consistently prefer observed mode in CartPole and Walker, while Sudden Jump and Cosine Annealing show the largest blind advantages.These comparisons are reported as scheduler-family patterns in normalized blind−observed IQM gaps.

D.2 Best-Anytime Checkpoint Results for Scheduler Search

Best-anytime evaluation shows dynamic schedulers outperform static baselines across environments and modes, with especially large gains for Walker and CarRacing; coverage remains largely unchanged.

  • Best-anytime results: Dynamic schedulers improve over static baselines across all environments and modes at peak performance, with generally larger margins than at the final checkpoint.The results indicate that the best policies are often learned earlier and partly lost through later instability or overfitting.
  • Best-anytime results: 193.0 vs. 134.2 (+59 IQM) is achieved by the best observed Walker scheduler relative to static observed.Walker shows one of the most pronounced best-anytime gains.
  • Best-anytime results: 778.9 vs. 617.7 (+161 IQM) is achieved by the best blind CarRacing scheduler relative to static blind.The CarRacing blind result is the largest reported gain in this comparison.
  • Coverage analysis: 4D coverage has negligible correlation with final evaluation score (r = 0.21, p = 0.003), with dynamic and static seeds interleaved across coverage.The metric discretizes four CartPole observation dimensions into 12 bins each, yielding 20,736 cells.
  • Coverage analysis: ≈6.5% coverage remains flat across pool sizes and schedulers, with stages 2–3 contributing less than 1% and at most 0.03% newly discovered cells after stage 1.Pool size therefore changes the distribution of training signal across context values rather than the breadth of visited states.

D.3.4 Parallel vs. Sequential Training

Parallelizing environments does not reproduce the benefits of dynamic intra-episode context variation. Across CartPole and BipedalWalker, parallel training neither reliably improves generalization nor expands state-space coverage.

  • Score differences ranged from −21 to +28 IQM, while coverage differences stayed below 0.5% across parallel and sequential CartPole training conditions.Summed deltas across eight conditions were +13.1 IQM for score and −0.56% for coverage, both negligible.
  • Parallel training neither expanded state-space coverage nor reliably improved generalization relative to sequential training.The comparison used multiple scheduler families, pool sizes, and 10 seeds per condition.
  • Dynamic scheduling differs from parallel rollout diversity by exposing a policy to a structured temporal sequence of dynamics changes within one episode.Parallel environments provide transitions from multiple regimes in a single policy update, whereas dynamic scheduling varies context during one rollout.
  • Across four BipedalWalker state-space projections, scheduler families showed no significant coverage differences.The projections covered inter-leg phase, 4D joint angles, 3D posture, and leg-1 phase.
  • BipedalWalker coverage declined from approximately 10–13% in stage 1 to 3–5% in stage 3 for all schedulers.This indicates convergence toward a narrow behavioral manifold regardless of scheduler.

D.4 Additional Bar Plots per Eval Context

Per-context evaluation shows dynamic scheduling is especially beneficial for OOD performance in CartPole and BipedalWalker, while CarRacing exhibits checkpoint-dependent differences and late-training degradation.

  • CartPole: CartPole OOD-low IQM was 313 for static observed and 456 for dynamic blind at the last checkpoint.Dynamic observed led at the last checkpoint with combined IQM 464, while dynamic blind led at best-anytime with 488.
  • BipedalWalker: BipedalWalker static observed achieved ID IQM 114 but OOD IQM −48 high and −10 low, yielding combined IQM 31.Dynamic schedulers substantially outperformed both static conditions, including dynamic blind versus static blind.
  • CarRacing: CarRacing dynamic blind was strongest at the last checkpoint on both COM_X and COM_Y, but best-anytime leadership differed by axis.At best-anytime, static blind led COM_X with IQM 842.9 versus 803.7 for dynamic blind, while dynamic blind led COM_Y with 760.7 versus 636.4 for static blind.
  • CarRacing: CarRacing context-observed conditions showed significant late-training policy degradation, motivating comparison of last-checkpoint and best-anytime results.

E.1 Best-Anytime Results

Best-anytime evaluation and multi-stage retraining analyses examine whether optimized dynamic schedules improve performance and how the discovered curricula are structured across environments.

  • Best-anytime IQM reports the highest combined evaluation score achieved at any training stage boundary.BipedalWalker results use a matched 1.5 M-step training budget.
  • The retraining analysis reports top-3 multi-stage Optuna trials per environment at the best-anytime checkpoint.Values are reported as IQM [Q1, Q3].
  • CartPole: CartPole retraining examines learning curves, scheduler-mode frequencies, active-stage hyperparameters, and rank shifts between search and retraining.Stage 2 is active in nearly all configurations, while Stage 1 is most often constant.
  • BipedalWalker: BipedalWalker retraining similarly analyzes scheduler patterns, learning curves, rank shifts, mode frequencies, and parameter distributions.
  • Per-scheduler-family tables report last-checkpoint and best-anytime IQM [Q1, Q3] across context-observability modes and all three environments.The best-performing configuration for each family and mode is selected by last-checkpoint combined IQM.

G Scheduler Training Progression

Scheduler progression figures compare representative training trajectories across evaluation regions and environments, separating dynamic observed and dynamic blind schedules from static pool baselines.

  • Each progression figure plots IQM episode return averaged over training checkpoints, with one best-representative line per scheduler family.Shaded bands show Q1–Q3 across seeds, and the dashed line marks the static pool baseline.
  • CartPole: CartPole progression is separated into combined, ID, OOD-low, and OOD-high IQM evaluations.
  • BipedalWalker: BipedalWalker progression is likewise reported for combined, ID, OOD-low, and OOD-high IQM evaluations.
  • CarRacing: CarRacing progression covers combined, ID, OOD-low, and OOD-high IQM for both longitudinal COM_X and lateral COM_Y contexts.
  • CarRacing: The CarRacing progression also includes OOD-low and OOD-high evaluations for the lateral COM_Y context.
Loading 2608.20799v1…