PaperlayerView exact revision on arXiv

Source-linked AI summary

Structure Detection for Contextual Reinforcement Learning

Tianyue Zhou, Jung-Hoon Cho, Cathy Wu

arXiv:2601.08120v1cs.LG

TL;DR

CRL must train policies across related but structurally diverse CMDPs without the cost of independent training or the negative transfer of universal policies. The paper introduces SD-MBTL, whose M/GP-MBTL instantiation detects MOUNTAIN structure and switches between clustering-based M-MBTL and GP-MBTL. Across synthetic data and CRL benchmarks, M/GP-MBTL improves the aggregated metric over the previous best method by 12.49%.

  • Problem

    Related CMDPs can have different generalization structures, so a single task-selection strategy may not suit all problems.

  • Method

    SD-MBTL detects the underlying CMDP structure and selects a matching MBTL algorithm; M/GP-MBTL switches between M-MBTL for MOUNTAIN cases and GP-MBTL otherwise.

  • Results

    12.49%: M/GP-MBTL outperforms the previous best method on the aggregated metric across synthetic data and CRL benchmarks.

  • Takeaways & Limitations

    Online structure detection can guide source-task selection for scalable and robust transfer learning in complex CRL environments.

  • Takeaways & Limitations

    The detector focuses on one structure, relies on transfer evaluations, and has only been tested in three-dimensional context spaces.

Abstract

from arXiv · show

Contextual Reinforcement Learning (CRL) tackles the problem of solving a set of related Contextual Markov Decision Processes (CMDPs) that vary across different context variables. Traditional approaches--independent training and multi-task learning--struggle with either excessive computational costs or negative transfer. A recently proposed multi-policy approach, Model-Based Transfer Learning (MBTL), has demonstrated effectiveness by strategically selecting a few tasks to train and zero-shot transfer. However, CMDPs encompass a wide range of problems, exhibiting structural properties that vary from problem to problem. As such, different task selection strategies are suitable for different CMDPs. In this work, we introduce Structure Detection MBTL (SD-MBTL), a generic framework that dynamically identifies the underlying generalization structure of CMDP and selects an appropriate MBTL algorithm. For instance, we observe Mountain structure in which generalization performance degrades from the training performance of the target task as the context difference increases. We thus propose M/GP-MBTL, which detects the structure and adaptively switches between a Gaussian Process-based approach and a clustering-based approach. Extensive experiments on synthetic data and CRL benchmarks--covering continuous control, traffic control, and agricultural management--show that M/GP-MBTL surpasses the strongest prior method by 12.49% on the aggregated metric. These results highlight the promise of online structure detection for guiding source task selection in complex CRL environments.

1 Introduction

CRL addresses related CMDPs whose contexts vary while sharing underlying dynamics, but existing training paradigms face computational costs or negative transfer. SD-MBTL detects generalization structure and adapts task selection, with M-MBTL exploiting MOUNTAIN structure through clustering.

  • Motivation: Independent training is computationally expensive for large task families, whereas multi-task training can suffer limited capacity or negative transfer when tasks are dissimilar.These trade-offs motivate selecting a smaller subset of source tasks for training and transfer.
  • Framework: SD-MBTL detects CMDP generalization structure and adapts source-task selection to the detected structure.Its M/GP-MBTL instantiation switches between Gaussian Process-based and clustering-based MBTL algorithms.
  • MOUNTAIN structure: M-MBTL reduces the GSTS problem to sequential clustering under MOUNTAIN structure and uses clustering loss to guide training-task search in continuous spaces.The method is designed for settings where generalization degrades with increasing context difference.
  • Empirical validation: 12.49%: M/GP-MBTL outperforms the previous best method on the aggregated metric normalized between the random baseline and the oracle.Experiments cover synthesized multi-dimensional CMDPs and continuous-control, traffic-control, and crop-management benchmarks.

2 Preliminaries

The preliminaries define multi-dimensional CMDPs, zero-shot transfer, and greedy source-task selection, then describe GP-MBTL as a Bayesian optimization approach for choosing expensive source-task trainings.

  • Contextual Markov Decision Processes: A CMDP is a family of MDPs indexed by multi-dimensional context, with each context specifying transition dynamics, rewards, and an initial-state distribution.The task family contains a finite target set Y embedded in a continuous task space X.
  • Contextual Markov Decision Processes: Zero-shot generalization evaluates a policy trained on source task x within target task y, with the transfer gap defined as ΔJ(πx, y) = J(πx, x) − J(πx, y).Transfer can therefore perform worse than direct training on the target task.
  • Greedy source task selection: GSTS greedily selects source tasks for full RL training to maximize expected performance across the target-task set using the best available trained policy.Exact optimization can be intractable in large or continuous task spaces, while solving each task MDP is expensive.
  • Model-Based Transfer Learning: GP-MBTL discretizes the continuous source-task space, models source-task returns with Gaussian-process regression, and selects tasks using predicted performance, uncertainty, and estimated transfer gaps.After evaluating the selected policy, it updates the Gaussian-process posterior to improve coverage of target tasks.

3 Structure Detection Model-Based Transfer Learning

The paper decomposes generalization performance into source-policy quality, target-task difficulty, task dissimilarity, and a constant, then uses detected structure to select a matching MBTL algorithm. SD-MBTL embeds this detection inside the task-selection loop.

  • Motivation: Multi-dimensional CMDPs can require more data for accurate GP-MBTL predictions, motivating structure detection to reduce unnecessary exploration when recognizable structure exists.The framework is intended to address the exploration–exploitation challenge in such settings.
  • Generalization Performance Structure Decomposition: The generalization-performance decomposition separates policy quality f(x), task difficulty g(y), task dissimilarity h(x, y), and a constant component.The decomposition is inspired by Sobol–Hoeffding functional ANOVA and gives h(x, y) = 0 when source and target tasks coincide.
  • Generalization Performance Structure Decomposition: The decomposition assigns f(x) to intrinsic source-policy quality, g(y) to target-task difficulty, and h(x, y) to dissimilarity that can reduce transfer performance.Large context differences are described as usually yielding poorer generalization.
  • Structure Detection: SD-MBTL detects a CMDP structure from observed source-to-target generalization performances and selects the corresponding MBTL algorithm from a matched library.The framework treats structurally constrained components as known and learns remaining components for task selection.

4 M/GP Model-Based Transfer Learning

M/GP-MBTL detects whether a CMDP follows MOUNTAIN structure and switches between clustering-based M-MBTL and general GP-MBTL accordingly. Under MOUNTAIN, source-task selection becomes sequential clustering over task distances.

  • 4.1 MOUNTAIN Structure: MOUNTAIN assumes constant source-policy quality and task dissimilarity equal to the negative of a distance metric, yielding J(πx, y) = J(πy, y) − dist(x, y).The paper uses the L1 norm as the distance metric; generalization therefore decreases with source–target distance.
  • 4.2 Structure Detection: Structure detection removes estimated target-task difficulty before testing small policy-quality variance and consistent decreasing slopes on both sides of each source task.The relative performance is defined by subtracting the mean performance over currently observed training tasks.
  • 4.3 MOUNTAIN Model-Based Transfer Learning: MOUNTAIN structure reduces greedy source-task selection to sequential clustering that minimizes distances between target tasks and their selected sources.This reduction enables clustering loss to guide search efficiently in the continuous task space.
  • 4.3 MOUNTAIN Model-Based Transfer Learning: M-MBTL extends K-Means with random restarts, refining sampled target tasks as centroids and selecting the lowest-loss candidate at each training round.The procedure repeats for K rounds, training on each selected task and evaluating the resulting policy across targets.
  • 4.4 M/GP-MBTL: M/GP-MBTL dynamically selects M-MBTL for MOUNTAIN CMDPs and GP-MBTL otherwise, combining structure-specific efficiency with general robustness.The detector returns MOUNTAIN when both structural criteria hold and NONE otherwise.

5 Experiments

Experiments compare adaptive and fixed task-selection strategies on synthetic CMDPs and four diverse benchmarks. Results show that M/GP-MBTL preserves the stronger specialized method across structures, while experiments are limited to three-dimensional settings.

  • 5.1 Setup: The evaluation uses synthetic CMDPs plus continuous-control, traffic-control, and agricultural-management benchmarks to test structure-specific selection.The benchmark suite comprises CartPole, BipedalWalker, IntersectionZoo, and CyclesGym with three context variables each.
  • 5.4 Additional Results: After only a small number of decision rounds, M/GP-MBTL maintains the best overall performance across five- and seven-dimensional synthetic datasets.The reported curves vary K to show that the decision-round setting is not cherry-picked.
  • 5.1 Setup: Experiments are confined to three-dimensional settings because the number of tasks and training time grow exponentially with context dimensionality.Higher-dimensional generalization is examined through five- and seven-dimensional synthetic datasets rather than the full benchmark suite.
  • 5.2 Synthetic Data: On synthetic data, M-MBTL performs best when MOUNTAIN assumptions hold, while GP-MBTL performs best when those assumptions are violated.M-MBTL closely approaches the myopic oracle under MOUNTAIN structure, whereas GP-MBTL is more robust when noise or structural violations weaken those assumptions.
  • 5.3 CMDP Benchmarks: M/GP-MBTL consistently matches the stronger of M-MBTL and GP-MBTL across CartPole, BipedalWalker, IntersectionZoo, and CyclesGym.M-MBTL leads on MOUNTAIN-like BipedalWalker and CyclesGym, whereas GP-MBTL leads on structure-violating IntersectionZoo; both perform well on CartPole.

6 Related Works

Related work frames CRL as learning across context-parameterized task families and contrasts universal policies with multi-policy transfer. Structure detection remains relatively nascent in CRL.

  • Contextual Reinforcement Learning and Multi-Policy Approaches: CRL represents variations in dynamics, rewards, or initial states through context variables, producing a family of related MDP tasks.Prior approaches condition a single policy on context or encode context latently.
  • Contextual Reinforcement Learning and Multi-Policy Approaches: Universal context-conditioned policies can suffer negative transfer and capacity limits when task variation is large, motivating expert-policy alternatives.Policy committees and represented-MDP methods instead cluster or reuse a smaller set of policies.
  • Structure Detection: Structure detection is well established in other machine-learning settings but remains relatively nascent for CRL.Existing CRL examples include context-change detection, model selection, and policy-reuse selection.
  • Structure Detection: The cited CRL structure-detection literature addresses deciding when source policies can transfer safely or how multiple source policies can be combined.Examples include mixture-of-experts approaches and online tests for rejecting over-complex dynamics models.

7 Conclusion

The paper presents structure detection for CRL, adapting task-selection strategies to inferred CMDP structure while reporting limitations in structure coverage, evaluation cost, and dimensional scalability.

  • 7 Conclusion: M/GP-MBTL switches between clustering for MOUNTAIN cases and a GP-based strategy otherwise, improving transfer across synthetic and real benchmarks.The conclusion describes consistent improvements over prior methods and frames structure detection as a route toward scalable, robust transfer learning.
  • 7 Conclusion: The current detector focuses on a single structure, and structure inference can be costly when context spaces are large or policies are expensive to evaluate.Inference relies on transfer evaluations of training tasks, which creates the stated practical cost boundary.
  • 7 Conclusion: Experiments were conducted only in three-dimensional context spaces, leaving scalability to higher-dimensional settings for future work.The authors propose exploring richer CMDP structures and reducing required source-task trainings in future work.
  • Additional Related Works: M-MBTL incrementally adds clustering centroids while preserving previously learned ones to guide efficient GSTS search.This adapts hierarchical clustering to the iterative source-task-selection setting.
  • Additional Related Works: Unlike multi-task, curriculum, UED, and meta-learning approaches, the framework targets zero-shot coverage through source-task selection from a fixed CMDP.The related-work passages distinguish fixed-CMDP selection from task invention, single-policy curricula, and rapid adaptation to individual new tasks.

D Proof of Equivalence in Definition 4.3

The proof derives an equivalent relationship between the objective values of policies π_x and π_y using the specified forms of f and h.

  • With f(x) = C1 and h(x, y) = −dist(x, y), the proof obtains J(π_y, y) = g(y) + C1 + C.These substitutions establish the objective value for policy π_y evaluated at context y.
  • Substituting the relation for J(π_y, y) yields J(π_x, y) = J(π_y, y) − dist(x, y).The objective difference is therefore determined by the distance between contexts x and y.
  • Thus, the objective for π_x at y is equivalent to the objective for π_y at y minus dist(x, y).

E Proof of Lemma 4.4

The proof uses h(x, y) = 0 when x = y to reduce same-context performance to f(x) and cross-context variation to h(x, y), establishing the stated standard-deviation condition.

  • Because h(x, y) = 0 when x = y, the proof obtains J(π_x, x) = f(x).This identifies same-context objective values with the function f(x).
  • Therefore, std_x∈x1:k(J(π_x, x)) equals std_x∈x1:k(f(x)), while std_y∈Y(J(π_x, y)) equals std_y∈Y(h(x, y)).
  • If the standard deviation of f(x) across x1:k is below the expected cross-context standard deviation of h(x, y), the corresponding inequality for J holds.

F Proof of Lemma 4.5

M-MBTL reduces task selection under MOUNTAIN structure to sequential clustering, while GP-MBTL provides a more general alternative when that structure is not detected.

  • F Proof of Lemma 4.5: The clustering loss assigns target tasks to their nearest selected centroid, making the next training task the candidate that most reduces aggregate distance.
  • H.1 Main Algorithm: M-MBTL sequentially selects training tasks as new centroids while keeping previously selected centroids fixed, then minimizes clustering loss.Random restarts address local optima by evaluating multiple initial candidates and selecting the optimized candidate with minimum loss.
  • H.2 Reducing Time Complexity of M-MBTL: M-MBTL accelerates updates by reusing unaffected candidate losses and recomputing distances only for task-centroid pairs influenced by the new centroid.
  • H.3 Time comparison: As the number of training tasks increases, M-MBTL becomes substantially faster than GP-MBTL because GP-MBTL spends considerable time estimating its Gaussian Process.Reported runtimes exclude policy-training time, and M/GP-MBTL follows the runtime of the algorithm it selects.
  • I.1 Gaussian Process Model-Based Transfer Learning (Cho et al. 2024): GP-MBTL selects tasks using estimated source performance, uncertainty, and transfer gaps, but its original approximation can be inaccurate when task difficulty is not constant.The revised version uses actual generalization performance for acquisition, reducing the effect of performance-estimation errors and retaining advantages when MOUNTAIN structure is violated.

M Aggregated Metric

The aggregated metric normalizes performance across four benchmarks between the Random baseline and Myopic Oracle, then averages the normalized scores.

  • M Aggregated Metric: Each benchmark score subtracts the Random baseline and divides by the gap between Myopic Oracle and Random performance.
  • M Aggregated Metric: Aggregated performance averages each algorithm’s normalized performance across four benchmarks.
  • M Aggregated Metric: Under this metric, 0 equals Random-baseline performance and 1 equals Myopic-Oracle performance.
  • M Aggregated Metric: Transfer matrices are N × N arrays whose entries represent generalization performance from each source task to each target task.

N.2 Benchmarks

The benchmarks evaluate multi-dimensional contextual task transfer across continuous control, traffic control, agricultural management, and synthetic transfer settings.

  • CartPole: CartPole varies pole length, cart mass, and pole mass across 9 × 10 × 10 contexts, with five million simulation timesteps per independent training run.
  • BipedalWalker: BipedalWalker varies friction, gravity, and scale across 8 × 8 × 8 tasks, with five million timesteps per independent run and zero-shot transfer evaluation.
  • IntersectionZoo: IntersectionZoo models urban traffic using inflow, autonomous-vehicle penetration, and green-phase duration, producing 63 distinct contexts.
  • CyclesGym: CyclesGym varies precipitation, temperature, and sunlight through predefined scaling factors to form a three-dimensional agricultural-context grid.
  • Synthetic Transfer Matrices: Synthetic transfer experiments use 512 tasks represented by three-dimensional integer contexts and visualize 512 × 512 source-target performance matrices.Brightness indicates higher performance, allowing structured effects from source bias, target bias, interaction terms, and noise to be compared.

O.1 Synthetic Data with Different Noise Terms

Across synthetic and benchmark experiments, M/GP-MBTL detects CMDP structure and selects between M-MBTL and GP-MBTL, achieving strong performance across noise levels, dimensions, and environments.

  • Synthetic data with different noise terms: Under substantially increased synthetic noise, violations of the MOUNTAIN assumptions cause the structure to deviate, while GP-MBTL remains effective because it uses observed generalization performance.
  • Synthetic structure detection: The detector selects M-MBTL for MOUNTAIN configurations and usually selects GP-MBTL for non-MOUNTAIN configurations across repeated synthetic trials.
  • Synthetic performance versus decision rounds: M/GP-MBTL improves rapidly as decision rounds increase, surpasses GP-MBTL around K = 12 on the 3D synthetic dataset, and maintains the best performance thereafter.
  • CMDP benchmark results: Across four CMDP benchmarks, M/GP-MBTL approaches the myopic oracle quickly and performs almost as well as the better standalone strategy.
  • CMDP benchmark results: Benchmark analyses show high overall performance with lower variability, near-highest AUC, and structure-dependent switching between M-MBTL and GP-MBTL.M-MBTL dominates in CartPole and CyclesGym, GP-MBTL dominates in IntersectionZoo, and BipedalWalker produces mixed switching; M/GP-MBTL also outperforms relevant hybrid baselines in the ablation study.
  • Higher-dimensional synthetic experiments: M/GP-MBTL performs best overall on 5- and 7-dimensional synthetic datasets, with relatively small confidence intervals compared with GP-MBTL.GP-MBTL exhibits larger confidence intervals, indicating greater instability across different CMDP structures.
Loading 2601.08120v1…