PaperlayerView exact revision on arXiv

Source-linked AI summary

How to Find and Reuse Policies for Continuous Adaptation in Lifelong Reinforcement Learning

Saptarshi Nath, Inish M. D'Souza, Antonio Carta, Soheil Kolouri, Andrea Soltoggio

arXiv:2610.03119v1cs.LGcs.AIcs.RO

TL;DR

Retaining earlier policies does not guarantee positive transfer, and useful knowledge may be spread across prior policies. AMSC uses online task similarity to select and weight prior masks; on CT28 and MiniGrid it outperforms modular baselines in mean AUC and forward transfer without forgetting.

  • Problem

    Retaining prior policies does not ensure positive transfer, and reinforcement-learning agents must identify useful knowledge among potentially conflicting priors.

    Source: §1 · p. 1

  • Method

    AMSC uses online task similarity to select a variable-size support of prior policy masks and determine their composition weights.

    Source: §1 · p. 2

  • Results

    On CT28 and MiniGrid, AMSC outperforms modular baselines in mean AUC and forward transfer while retaining prior policies without forgetting.

    Source: §7 · p. 9

  • Takeaways & Limitations

    Ablations indicate that selecting relevant policies and determining how to reuse them are central to more effective forward transfer and successful lifelong learning.

    Source: §6 · p. 9

    On CW10, AMSC's shared similarity-derived weights underperform learned mask composition, suggesting that relevant-policy selection and layer-specific composition can be distinct challenges.

    Source: §5.3 · p. 6

Abstract

from arXiv · show

In lifelong reinforcement learning, retaining previously learned policies is not sufficient for effective transfer to a new task. Useful knowledge may be distributed across several prior policies, and its relevance may change as the learner acquires experience. One hypothesis is that task similarity can be effectively used in a continual learning setting to find and combine previously learned policies. To test it, Adaptive Mask Selection and Composition (AMSC) is designed to estimate similarity from online experience via non-parametric Wasserstein task embeddings from state-action-reward samples. The z-score-normalized sparsemax of the similarity scores are used to derive a variable-size support to periodically choose and weight policies to form a prior when learning a new task. On CT-graph and MiniGrid, AMSC achieves higher mean performance and forward transfer than the evaluated modular composition baselines while exhibiting no forgetting. Results on Continual World suggest that identifying relevant prior knowledge and determining its layer-specific composition may require additional layer-specific tuning. Ablations show that selecting relevant sources and determining how strongly to reuse them are central to these gains. Independently measured pairwise transfer is also positively associated with task-embedding similarity. These results indicate that task similarity can be an effective criterion to select and weight specific knowledge for reuse in lifelong reinforcement learning.

1 INTRODUCTION

AMSC uses online task similarity to select and weight prior policies for new tasks. On CT28 and MiniGrid, it reports strong transfer with no measured forgetting, while results on Continual World qualify the approach.

1 INTRODUCTION

  • Retaining prior policies alone does not ensure positive transfer; lifelong learners must identify relevant knowledge without interference from irrelevant or conflicting policies.In reinforcement learning, unsuitable priors can bias exploration, and sparse rewards may delay evidence that a policy is harmful.Source: §1 · p. 1
  • Selective reuse remains comparatively underexplored, while shared or aggregated knowledge can admit interfering priors and learned reuse weights may arrive too late to improve early learning.The useful number of sources is unknown, and both selected knowledge and its contribution may change as experience accumulates.Source: §1 · pp. 1–2
  • AMSC uses online task similarity to select a variable-size support of prior masks and weight their composition with the current task mask.It learns task-specific masks over a fixed backbone without task descriptions, task labels, or transfer probes.Source: §1 · p. 2
  • AMSC achieves AUC/FWT/FGT of 0.94/0.94/0.00 on CT28 and 0.73/0.54/0.00 on MiniGrid.In CT28 and MiniGrid, similarity-derived weights perform comparably to learned composition weights.Source: §1 · p. 2
  • On Continual World, similarity-derived weights underperform learned mask composition, suggesting that shared layer weights may not suit every task.The introduction distinguishes selecting relevant policies from determining how to compose their knowledge.Source: §1 · p. 2

2 RELATED WORK

Related work addresses selective reuse through data, capacity, structures, modular routing, and composition. Other work uses task similarity and optimal transport to compare tasks and support transfer-related operations.

Figure 1

  • Figure 1 plots mean forward transfer against AUC for CT28 and MG16; marker colour denotes forgetting and error bars show 95% confidence intervals.Each marker summarizes seven seeds.Source: Figure 1 · p. 2

Selective reuse in lifelong reinforcement learning

  • Prior selective-reuse methods target data, capacity, or structures to determine what knowledge is worth retaining.Examples include selective replay, adaptive structure allocation, subnetworks, and related or unrelated task sequences.Source: §2 · p. 2

Modular routing and composition

  • Modular lifelong RL transfers task-specific knowledge by routing among modules or composing masks, modules, policies, or task vectors.Methods also activate modules, select policy heads, infer contexts, and route adapters.Source: §2 · p. 2

Task similarity and optimal transport

  • Task similarity has been used to predict transfer and select policies, while optimal transport compares task distributions through OTDD and Wasserstein embeddings.These tools also support memory, selection, distillation, context detection, and expert diversity.Source: §2 · p. 3

3 PRELIMINARIES

The paper frames lifelong learning as sequential task learning with both retention and forward transfer objectives. Its modular-mask preliminaries describe task-specific subnetworks and compositions of current and prior task scores.

3.1 PROBLEM SETTING

  • Tasks arrive sequentially with known boundaries, and the learner interacts only with the current task for a fixed training budget.The setting aims to maintain earlier-task performance while continuing to learn new tasks through forward transfer.Source: §3.1 · p. 3

3.2 MODULAR POLICY MASKS

  • Score-based masking learns task-specific subnetworks over a shared fixed backbone, and modular methods can compose current and frozen prior-task scores using non-negative weights.Prior approaches may use the complete prior library and learn layer-specific weights; when no priors are admitted, the current mask starts with weight 1.Source: §3.2 · p. 3

4 ADAPTIVE MASK SELECTION AND COMPOSITION

AMSC retrieves prior masks from online task representations, then combines selected masks with the current mask using similarity-derived weights. Retrieval and composition are updated as experience accumulates.

4 ADAPTIVE MASK SELECTION AND COMPOSITION

  • AMSC has three operations: normalized sparsemax retrieval, scale-normalized mask scores, and composition using retrieval weights.The current policy-mask scores are optimized while prior masks remain available in the reusable library.Source: §4 · p. 3, §4.1 · p. 4, §4.3 · p. 5

4.1 MASK LIBRARY AND ONLINE TASK REPRESENTATIONS

  • Online state-action-reward experience is embedded in a common vector space using a non-parametric linear Wasserstein task embedding.Completed task representations remain fixed, while the current representation is updated and smoothed online.Source: §4.1 · p. 4

4.2 SPARSE RETRIEVAL

  • Cosine similarities between the current and completed-task representations are z-score normalized before sparsemax assigns exact-zero weights and selects a variable-size prior support.The main experiments use τ = 1.Source: §4.2 · p. 4

4.3 SCALE-NORMALIZED FIXED COMPOSITION

  • Selected prior and current scores are normalized to unit ℓ2 norm before composition, preventing raw score magnitudes from overriding the combination weights.A straight-through estimator preserves gradients for the current scores.Source: §4.3 · p. 4
  • For m selected priors, the current mask receives mass 1/(m + 1), and the priors share m/(m + 1) according to retrieval weights across layers.AMSC recomputes retrieval and composition after sufficient experience is collected, then freezes the final composed scores at the task boundary.Source: §4.3 · p. 5

5 RESULTS

Across CT28 and MG16, AMSC leads modular methods in mean AUC and forward transfer; CW10 results expose a boundary for shared similarity-derived layer weights.

5.1 EXPERIMENTAL SETUP

  • Evaluation spans sparse-reward CT28 and MG16 plus robotic CW10, comparing replay, modular, scratch, and fine-tuning methods using seven-seed AUC, FWT, and FGT.CT28 and MG16 use PPO; CW10 uses SAC.Source: §5.1 · p. 5

5.2 COMPARISON TO BASELINES

  • AMSC has the highest modular-method mean AUC and FWT on CT28 and MG16, including CT28 0.94/0.94 versus MOSAIC 0.81/0.79 and MG16 FWT 0.54.On MG16, both MOSAIC and CompoNet have negative FWT; AMSC’s gains over Mask-LC, CompoNet, and MOSAIC are significant on both metrics and curricula (p ≤0.023).Source: §5.2 · p. 5, Table 1 · p. 6

5.3 LIMITS OF SIMILARITY WEIGHTING

  • On CW10, AMSC has 0.48 AUC, −1.21 FWT, and zero forgetting versus Mask-BLC’s 0.53 AUC and 0.12 FWT; shared layer proportions may be too restrictive.AMSC applies the same similarity-derived proportions across masked layers, whereas Mask-BLC learns layer-specific weights.Source: Table 2 · p. 6, §5.3 · p. 6

5.5 RETRIEVAL, COMPOSITION, AND CAUSAL TRANSFER

  • Online similarity tracks pairwise transfer (Pearson r = 0.67 on CT28 and r = 0.48 on MG16); regularly selected sources average ΔAUC 0.66 versus 0.02 on CT28, but cover only 19 of 47 positive-transfer MG16 pairs.Figure 2 also shows composition concentrating on related hierarchical masks while retaining a current-task component.Source: Figure 2 · p. 7, §5.5 · pp. 7–8

5.4 TASK REPRESENTATION SENSITIVITY

  • Representation leaders differ by curriculum: WTE-transition yields CT28 0.93/0.92 AUC/FWT, while Fréchet moments lead MG16 at 0.70/0.47.No single representation is best across both curricula.Source: Table 3 · p. 7, §5.4 · p. 7

5.6 ABLATION STUDIES

  • Relevant-source selection is central: keeping support size and weights fixed while randomizing sources reduces CT28/MG16 AUC/FWT to 0.35/0.13 and 0.36/0.08.Freezing retrieval, using only one source, uniform weights, or removing L2-STE also lowers performance in the reported ablations.Source: Table 4 · p. 8, §5.6 · p. 8

5.7 ADDRESSING PER-TASK MASK MEMORY COSTS

  • Ternary masks cut storage 16× and retain CT28 AUC 0.94 versus Float32 0.941, with FWT/FGT of 0.91/0.04 versus 0.935/0.000.Ternary quantization maps scores to {−1, 0, 1}, preserving sign while suppressing near-zero scores.Source: §5.7 · p. 9

6 DISCUSSION AND FUTURE WORK

The discussion presents task similarity as a way to guide selective policy reuse when relevant prior knowledge is available, while identifying composition, representation choice, and library growth as open challenges. It also points to retrieval updates and transfer beyond pairwise comparisons as directions for further study.

6 DISCUSSION AND FUTURE WORK

  • Similarity-guided reuse can avoid directly optimizing retrieval or composition weights when relevant knowledge is available, while random selection shows that sparsity alone is insufficient.On CT28 and MG16, AMSC is competitive with replay methods in AUC and forward transfer while retaining prior policies without forgetting; periodic retrieval improves reuse as experience accumulates and representations update.Source: §6 · p. 9
  • Embedding choice may depend on the distributions, and future work could test whether task similarity predicts transfer from combinations of priors rather than pairs.The discussion also proposes testing task embeddings for layer-specific reuse or combining similarity-based selection with learned composition, including adaptive balancing of prior and current knowledge.Source: §6 · p. 9
  • Managing policy-library growth remains a complementary challenge, including assessing how consolidation and compression affect future transfer.Extending AMSC beyond known task boundaries also requires detecting distribution changes and deciding when to retrieve, adapt, or create a new mask.Source: §6 · p. 9

7 CONCLUSION

The conclusion presents AMSC as a task-similarity approach for selectively reusing prior knowledge, with strong results on CT28 and MG16 but a layer-composition caveat on CW10.

7 CONCLUSION

  • On CT28 and MG16, AMSC outperforms modular baselines in mean AUC and forward transfer, remains competitive with replay methods, and retains prior policies without forgetting.The conclusion cautions that identifying relevant knowledge and adapting its composition across layers may require different mechanisms, as similarity-guided weights shared across layers may be restrictive.Source: §7 · p. 9
  • AMSC periodically selects and composes a sparse set of policy masks using task similarity, without directly optimizing selection or composition.Ablations support the importance of selecting relevant sources and updating how they are reused during learning.Source: §7 · p. 9
  • The conclusion frames task similarity as a practical approach to selective reuse, but says its effectiveness depends on how prior knowledge is composed per layer.Source: §7 · p. 9

A MASK-SCORE SENSITIVITY AND QUANTIZED STORAGE

The section tests how mask-score magnitude affects gate stability and whether quantized masks can preserve transfer while reducing storage. Ternary masks retain much of Float32 performance at a fraction of its storage, but aggressive thresholding harms performance.

Perturbation results

  • At α = 0.5, perturbing the lowest-magnitude quartile changes 37.3% of selected CT28 gates and 39.3% of selected MG16 gates, versus 0.2% and 0.5% for the highest-magnitude quartile.Low-magnitude perturbations also change selected actions on 6.5% of CT28 states and 6.2% of MG16 states, versus 0.8% on both benchmarks for high-magnitude perturbations.Source: A MASK-SCORE SENSITIVITY AND QUANTIZED… · p. 15

Quantization results

  • At δ = 0.50, ternary masks achieve AUC/FWT/FGT of 0.935/0.912/0.041 versus 0.941/0.935/0.000 for Float32 masks.The paired tests do not reject zero AUC or FWT difference, but FGT increases by 0.041 (p = 0.031); this does not establish equivalence.Source: A MASK-SCORE SENSITIVITY AND QUANTIZED… · p. 16
  • At δ = 1, AUC/FWT/FGT falls to 0.624/0.568/0.265, showing that aggressive thresholding can delete gates required by completed policies.At δ = 0.625, the metrics are 0.915/0.908/0.056, with 41.4% of stored scores set to zero; AUC reduction is significant (p = 0.023), while FWT reduction is not (p = 0.086).Source: A MASK-SCORE SENSITIVITY AND QUANTIZED… · p. 16

Quantized mask composition

  • Jointly normalizing prior and current scores is needed for sparsemax weights to control mask contributions; omitting it yields AUC/FWT/FGT of 0.283/0.070/0.291 at δ = 0.50.Normalizing only prior masks also fails, yielding 0.269/0.049/0.005.Source: A MASK-SCORE SENSITIVITY AND QUANTIZED… · p. 17

Storage comparison

  • Packed ternary masks require 26.8 KiB per task, compared with 24.2 KiB for rank-4 Float32 LoRA and 48.4 KiB for rank-8 Float32 LoRA.Packed-value storage excludes container and index overhead.Source: A MASK-SCORE SENSITIVITY AND QUANTIZED… · pp. 17–18, Figure 4 · p. 17
  • Normalized-sparsemax composition of rank-4 LoRA deltas achieves AUC/FWT/FGT of 0.885/0.870/0.000, below ternary masks by 0.050 AUC and 0.042 FWT.The section attributes this gap to LoRA recompressing transferred updates within a rank-4 subspace, whereas mask composition can combine frozen backbone connections independently.Source: A MASK-SCORE SENSITIVITY AND QUANTIZED… · p. 18

B.1 WASSERSTEIN TASK EMBEDDING CONSTRUCTION

AMSC builds Wasserstein task embeddings from sampled online experience and compares the evolving current-task representation with fixed completed-task representations. Its training procedure periodically updates embeddings and retrieval while optimizing only the current task’s mask scores.

B.1 WASSERSTEIN TASK EMBEDDING CONSTRUCTION

  • AMSC samples state-action-reward entries from the current task’s replay buffer to construct Wasserstein task embeddings, with discrete actions one-hot encoded and continuous actions kept as vectors.The flattened state dimension is ds, the action-feature dimension is da, and the input dimension is ds + da + 1.Source: §B.1 · p. 19
  • A fixed ordered random reference is transported to the empirical batch distribution, and the embedding records each reference point’s displacement.The construction is the expanded form of the paper’s earlier equation and uses no learned encoder or embedding loss.Source: §B.1 · p. 19
  • The current task’s embedding evolves through normalized updates, while completed-task embeddings remain fixed and are compared against the evolving estimate at retrieval events.The reported CT28 and MG16 experiments use ρ = 0.5; task boundaries freeze completed embeddings and clear the current buffer.Source: §B.1 · p. 20

B.2 PSEUDO-CODE

  • The training procedure collects rollouts, updates only current mask scores with the RL loss, and periodically updates embeddings and retrieval support.At task boundaries, completed masks and embeddings are frozen into their libraries; retrieval weights are not optimized by the RL loss.Source: §B.2 · p. 20

C BENCHMARKS

The evaluation spans sparse-reward graph and grid-world curricula and a continuous-control curriculum, with replay, routing, isolation, and composition baselines. AMSC learns earlier than MOSAIC on CT28, while CW10 results show task-specific differences between AMSC and learned composition.

C BENCHMARKS

  • CT28 tests sparse-reward decision trees with family-shared prefixes; MG16 tests room-chain growth under symbolic remappings; CW10 adds robotic tasks with shaped rewards and continuous actions.CW10 uses ten tasks, trained for 1M environment steps each, with binary success evaluation.Source: C BENCHMARKS · pp. 21–22

D BASELINES

  • Baselines span replay, parameter isolation and routing, learned policy composition, and MOSAIC-style similarity-guided composition; comparisons use shared seeds, task orders, and interaction budgets.Architectures are matched where direct substitution is possible; some methods retain their released architectures and use matched no-reuse references for FWT.Source: D BASELINES · p. 23

E HYPERPARAMETERS

  • Principal AMSC settings and benchmark-specific baseline settings are tabulated separately; each CT28 or MG16 PPO iteration collects 512 environment samples.The CW10 configuration uses 39-dimensional MetaWorld-v2 observations.Source: E HYPERPARAMETERS · p. 24, Table 5 · p. 24, Table 6 · p. 25

F.1 EARLY ACQUISITION

  • AUC20 = 0.86 versus 0.35 for MOSAIC (p = 0.016), and first positive training reward arrives at 2.34k versus 5.83k steps (p = 0.016); SER is not significantly different.The early advantage is clear against MOSAIC, but neither SER variant differs significantly from AMSC on early AUC or time to first reward.Source: §F.1 · p. 27, Table 8 · p. 27

F.2 SIMILARITY-DERIVED COMPOSITION VS. LEARNED COMPOSITION IN CW10

  • On CW10, Mask-BLC exceeds AMSC on peg-unplug-side (0.92 versus 0.63) and push-back (0.41 versus 0.23), but AMSC leads on push (0.51 versus 0.22).AMSC shares similarity-derived weights across masked actor layers, unlike Mask-BLC’s layer-specific weights; the coefficients describe saved training-time allocation, not effective contribution after consolidation.Source: §F.2 · p. 27

F.3 HYPERPARAMETER SENSITIVITY

Sensitivity sweeps show that acquisition varies with representation and retrieval settings, especially on MG16, while retention remains unchanged. Per-task results reveal variation hidden by curriculum averages.

F.3 HYPERPARAMETER SENSITIVITY

  • CT28 remains near AUC/FWT 0.94/0.94 for reference sizes 10, 25, and 50, while τ = 4 and N = 100 each yield 0.89/0.88.The CT28 reference remains stable across the tested settings; these are one-factor sweeps, not evidence of robustness to joint changes.Source: §F.3 · pp. 27–28, Table 11 · p. 30, Table 12 · p. 30
  • On MG16, increasing N from 128 to 1024 changes AUC/FWT from 0.54/0.23 to 0.73/0.54, but 2048 samples reduce FWT to 0.30; τ = 0.5 yields 0.67/0.50.Reference-distribution sizes are also non-monotonic: R = 10 obtains 0.70/0.52, while R = 25 obtains 0.55/0.35.Source: §F.3 · pp. 28–30, Table 12 · p. 30, Table 13 · p. 31
  • On CT28, raising either embedding or selection update frequency from 1 to 10 lowers AUC/FWT from 0.94/0.94 to 0.84/0.83 and 0.83/0.81; EMA is not required for peak point estimates.On MG16, raising embedding frequency from 1 to 10 changes 0.62/0.43 to 0.57/0.34, while selection frequency and smoothing show less ordered peaks.Source: §F.3 · p. 30, Table 14 · p. 31, Table 15 · p. 31

F.4 PER-TASK ACQUISITION AND RETENTION

  • Per-task results show CT28 gains diminish on some late, high-depth tasks, while AMSC improves acquisition across later MG16 R3–R5 tasks; both parameter-isolation methods have zero measured FGT.The displayed MG16 task table is from an earlier result set averaging approximately 0.62 AUC, not a decomposition of the principal 0.73 result.Source: §F.4 · p. 30, Table 18 · p. 32, Table 19 · p. 33

G EVALUATION METRICS

AUC, FWT, and FGT capture different aspects of learning across a task curriculum, and comparisons use paired-seed testing. Pairwise-transfer and utilization matrices show how transfer and mask reuse are represented across tasks.

G EVALUATION METRICS

  • AUC averages normalized performance during a task’s training block, FWT compares its learning curve with a seed-matched from-scratch reference, and FGT compares end-of-task with final-curriculum performance.AUC measures acquisition rather than retention; normalized FWT can be substantially negative and need not rank methods like AUC.Source: G EVALUATION METRICS · p. 34

H SIGNIFICANCE TESTING

  • Comparisons use seven shared seeds and exact two-sided sign-flip tests of paired differences, with unadjusted p-values and paired bootstrap confidence intervals.Positive AMSC-minus-comparison differences favor AMSC for AUC and FWT, while negative differences favor AMSC for FGT; colors encode evidence for a difference, not which method is better.Source: H SIGNIFICANCE TESTING · p. 35

H.1 PAIRWISE TRANSFER MATRICES

  • Pairwise-transfer matrix cells give source-policy target-task ∆AUC relative to a no-source run; positive values indicate improved learning over the target budget and negative values indicate interference.The complete matrices include self-pairs and sources unavailable at the target’s curriculum point; the curriculum-restricted analysis excludes those future sources.Source: §H.1 · p. 37

H.2 PER-TASK POLICY-MASK UTILIZATION

  • Mask-utilization matrices place target tasks in rows and current or retrieved masks in columns; their mean coefficients quantify reuse, not the fraction of backbone parameters activated.Diagonal cells are current masks, below-diagonal colored cells are prior masks, and grey cells mark available priors never selected.Source: §H.2 · p. 37

H.3 INITIAL RETRIEVAL AND TEMPORAL REFINEMENT

Initial AMSC retrieval is aligned with measured pairwise transfer, and later retrieval changes are associated with additional aggregate transfer. Controls that retain the initial retrieval show lower acquisition scores, though pairwise transfer is only a proxy for time-local benefit.

H.3 INITIAL RETRIEVAL AND TEMPORAL REFINEMENT

  • Initial retrieval’s mean ∆AUC exceeds random expectation by 0.42 on CT28 and 0.27 on MG16, and includes the best available source in 98% and 84% of task runs.Refinement trades some positive-transfer mass recall for greater precision, consistent with pruning an initially over-inclusive support.Source: §H.3 · p. 37
  • From first to final retrieval, source-normalized weighted improvement is 0.13 on CT28 (95% CI [0.11, 0.15]) and 0.03 on MG16 (95% CI [0.01, 0.05]).Only 10% of consecutive CT28 retrieval events and 7% of MG16 events change support, but the changes accumulate over training.Source: §H.3 · pp. 37, 42, Table 26 · p. 42
  • Retaining the first retrieval lowers AUC/FWT from 0.94/0.94 to 0.87/0.85 on CT28 and from 0.73/0.54 to 0.63/0.42 on MG16.These results support updating retrieval as experience accumulates, but pairwise transfer summarizes complete target-learning curves and is a proxy, not direct evidence of instantaneous benefit.Source: §H.3 · p. 42

Related papers

Loading 2610.03119v1…