PaCo: Parameter-Compositional Multi-Task Reinforcement Learning

• MTRL trains one policy across related tasks, motivated by the efficiency of learning similar skills together rather than solving each task separately.Source: §1 · p. 1
• Parameter sharing is difficult because tasks may differ substantially even when they use the same robot or share some similarities.Source: §1 · p. 1
• PaCo learns a task-agnostic parameter set forming a policy subspace, then infers each task-specific policy using a compositional vector.Source: §1 · p. 2
• The framework composes policies directly in parameter space, allowing tasks to use identical or independent policies and accommodating different subspace dimensions.Source: §1 · p. 2
• PaCo separates task-specific and task-agnostic parameters to improve multi-task training stability, and reports state-of-the-art Meta-World performance.Source: §1 · p. 2

• A discrete-time MDP comprises states, actions, transitions, rewards, initial-state distribution, and a discount factor, with a policy maximizing accumulated discounted return.Source: §2.1 · p. 2
• PaCo uses Soft Actor-Critic, an off-policy maximum-entropy actor-critic method with policy, soft Q-function, and temperature parameters.Source: §2.2 · p. 2
• For multi-task learning, the task skill identifier and goal are appended to the state, while multiple tasks may have different learning dynamics.Source: §2.2 · p. 2

3.1 Multi-Task Reinforcement Learning Setting

• MTRL uses a universal policy over a family of task MDPs, assuming a shared state space with task-specific subsets that let the policy identify the current task.Source: §3.1 · pp. 2–3
• MTRL task families are assumed to share similarities and may vary by goals within one environment or by skills across related environments such as Meta-World.Source: §3.1 · p. 3

3.2 Challenges in Multi-Task Reinforcement Learning

• Parameter sharing must balance exploiting task similarities against differences in task content and difficulty, because poor sharing can reduce success on tasks that might succeed independently.Source: §3.2 · p. 3
• Conflicting skills and exploding losses can let one task adversely affect others through shared parameters, especially as the number of tasks grows.Source: §3.2 · p. 3

4.1 Formulation

• PaCo composes each task’s parameters from a shared parameter set Φ and a task-specific vector wτ, placing task policies in a learned parameter subspace.Each basis vector in Φ matches the dimensionality of the task parameter vector, while wτ is a trainable embedding for task τ.Source: §4.1 · p. 3, Figure 2 · p. 4
• All tasks update the shared parameters Φ, whereas each task loss updates only its own compositional vector wτ.This separates task-agnostic learning from task-specific adaptation within the multi-task loss.Source: §4.1 · p. 4, Figure 2 · p. 4
• The compositional parameterization provides flexible sharing by learning task-specific combinations over a shared parameter set.The same formulation can represent dedicated single-task parameters, fully shared multi-task parameters, and shared-trunk multi-head models.Source: §4.1 · p. 4, §4.3 · pp. 5–6
• The shared task-agnostic parameter set Φ can also serve as a reusable pretrained policy basis in transfer scenarios, although this benefit is beyond the current work’s scope.The paper notes initial transfer attempts in Appendix A.3 rather than presenting this as the main evaluation.Source: §4.1 · p. 4

4.2 Stable Multi-Task Reinforcement Learning

• When a task loss exceeds threshold ϵ, PaCo masks that loss from the aggregate update and reinitializes its w-vector without changing other tasks’ parameters.The combined Reset scheme uses loss maskout and w-reset to let the unstable task continue learning while protecting non-exploding tasks.Source: §4.2 · p. 5
• w-reset is specific to PaCo’s task-agnostic/task-specific decomposition, whereas baselines use the more expensive stop-relaunch procedure.Stop-relaunch discards the whole training run after instability is detected; PaCo can reset only the affected task’s compositional vector.Source: §4.2 · p. 5, §4.3 · p. 5

5 Related Work

• Prior MTRL methods seek to improve single-task reinforcement learning by exploiting similarities among tasks, often through task-conditional policies.The literature includes goal-conditional formulations and many other approaches to multi-task policy learning.Source: §5 · p. 6
• Conflicting gradients from multiple task losses can degrade performance and destabilize training when shared parameters are updated inconsistently.This issue is identified as a general multi-task learning problem and is especially consequential under parameter sharing.Source: §5 · pp. 6–7
• Existing approaches address interference by discarding parameter sharing, selecting shared information with context-based attention, or balancing losses and gradients.Gradient projection may be less reliable when reinforcement-learning gradients are noisy, while context-based grouping requires additional context information.Source: §5 · p. 7
• PaCo differs by leveraging a decomposition into task-agnostic and task-specific parameters to improve training stability.Its stability schemes operate on the structure of the parameterization rather than solely on gradient or loss manipulation.Source: §5 · p. 7

6.2 Stable MTRL Training

• The complete PaCo design improves both average success rate and training stability over PaCo-Vanilla and PaCo-Maskout.PaCo-Vanilla is more variable because of exploding losses, while loss maskout can reduce adverse effects but may compromise learning opportunities.Source: §6.2 · p. 8, Table 2 · p. 8

6.3 Ablation Study

• K=5 provides a good performance–complexity trade-off, whereas increasing K can reduce performance through added over-parameterization and lower sample efficiency.With K=10 and one-hot initialization, PaCo reaches 88.9±1.9; the parameter-set size grows with task number n at a lower rate.Source: §6.3 · pp. 8–9, Table 3 · p. 8
• Applying compositional structure to both actor and critic networks gives the best performance, while output-only structure is comparable to Multi-Head SAC with fewer parameters.Actor-only improves over output-only, and full actor–critic composition further unleashes the method’s potential.Source: Table 3 · p. 8, §6.3 · p. 9

6.4 Qualitative Results on Parameter Set and Compositional Vectors

• A two-dimensional compositional basis represents three distinct skills on a low-dimensional policy manifold, with all basis parameters contributing through task-specific weights.Sampled points on the unit circle produce varied behaviors, including potential object interactions in the peg-insert-side environment.Source: §6.4 · pp. 9–10
• PCA shows that compositional vectors cluster skills by behavioral similarity, including peg-insert near several object-interaction tasks despite literal task differences.Reach is separated from other skills, while window-open and door-open, and window-open and window-close, appear close in w-space.Source: §6.4 · p. 10

• PaCo separates task-agnostic and task-specific components, supporting flexible learning and training stabilization.Source: §7 · p. 10
• The authors report clear improvement over current state-of-the-art methods on standard benchmarks without additional metadata or more complicated design.Source: §7 · p. 10

Limitation

• A globally applied simple linear compositional form may limit representation power in some cases.Source: §7 · p. 10

Future Work

• Future work includes extending PaCo beyond global linear composition and adjusting task distributions according to task difficulty or progress.Source: §7 · p. 10
• PaCo’s separation of task-agnostic and task-relevant parameters may support transfer learning.Source: §7 · p. 10

• The paper reports that it accurately describes its contributions and scope and discusses its limitations.Source: Checklist · p. 13
• Theoretical-results requirements are marked not applicable because the paper does not present theoretical results or proofs.Source: Checklist · p. 13
• The paper reports training details, error bars, and computational-resource information, with some resource details provided in the supplementary file.Source: Checklist · p. 13
• The paper reports citing existing assets and describing their licenses, while new-asset release is marked not applicable.Source: Checklist · p. 13
• Human-subject, participant-risk, compensation, personally identifiable information, and offensive-content requirements are marked not applicable.Source: Checklist · p. 13

• The appendix provides additional results in Section A.1, implementation details in Section A.2, and initial future-extension attempts in Section A.3.Source: A Appendix · p. 14

• PaCo-Vanilla shows large variation and lower stability because exploding losses affect some random seeds.At 20M steps, performance ranges from 0.90 for some seeds to 0.28 for others.Source: §A.1.1 · p. 14, Figure 5 · p. 14
• PaCo-Maskout can mitigate adverse effects from exploding losses but may compromise performance by reducing learning opportunities.Source: §A.1.1 · p. 14
• Figure 5 compares average success-rate curves for PaCo, PaCo-Maskout, and PaCo-Vanilla under ablated stabilization schemes.Source: Figure 5 · p. 14

Benchmark results

• PaCo reaches higher success rates than baseline methods on Meta-World-V2 MT10 with fixed goals, as reported in Table 4.The fixed-goal setting is simpler than random-goal evaluation because it avoids learning a goal-conditioned policy for each skill.Source: §A.1.2 · p. 14, Table 4 · p. 15
• PaCo is evaluated on the more complex MT50-rand benchmark with 20 parameter groups over 100M total environment steps.MT50 contains 50 manipulation tasks, and random-goal training requires more samples and steps because of increased task-combination randomness.Source: §A.1.2 · p. 15, Table 5 · p. 15

Compositional vector visualization

• The MT10 visualizations show 90% average success for random-goal tasks, with failure on pick-place, and 100% success for fixed-goal tasks.Figures 6 and 7 display the compositional vectors used for the ten skills in each setting.Source: §A.1.4 · p. 15, Figure 6 · p. 16, Figure 7 · p. 16
• PCA reveals that random-goal task vectors are more scattered, whereas fixed-goal task vectors approximately occupy a one-dimensional low-dimensional space.The projections also place behaviorally related skills near one another, including peg-insert and several opening tasks.Source: §A.1.4 · pp. 15, 17, Figure 8 · p. 17

Implementation details

• PaCo replaces fully connected layers with compositional layers that combine K parameter groups using task-specific compositional vectors.Identical initialization across parameter groups makes task-specific parameters initially identical and empirically helps PaCo find interpolated policies faster.Source: §A.1.4 · p. 15, §A.2.1 · pp. 17–18

Training and transfer

• PaCo uses a built-in stabilization scheme based on its decompositional structure, avoiding the baseline procedure of discarding runs after instability appears.The paper also proposes extending the parameter subspace to new tasks, but verifying this on larger skill sets remains future work.Source: §A.2.4 · p. 19, §A.3.1 · p. 19