PaperlayerView exact revision on arXiv

Source-linked AI summary

MATES: Learning Multi-Agent Interactions by Transforming Observations for Frozen Single-Agent Policies

Elie Abboud, Oren Gal

arXiv:2609.26010v1cs.MAcs.ROcs.SYeess.SY

TL;DR

MARL often trains agents from scratch to learn individual competence and coordination together, although many tasks have single-agent counterparts. MATES adapts observations for a frozen solo policy using a lightweight adapter. Across evaluated settings, it uses far fewer parameters than full-policy training, outperforms training from scratch, and generalizes to unseen team sizes.

  • Problem

    MARL commonly trains decentralized policies from scratch, requiring agents to acquire individual task competence and coordination simultaneously despite the availability of compatible single-agent counterparts.

    Source: §I · p. 1

  • Method

    MATES learns a small adapter from multi-agent experience that transforms observations for a frozen single-agent policy while leaving its architecture and the MARL algorithm’s objectives and updates unchanged.

    Source: §I · p. 1

  • Results

    3.5–7.3% as many parameters as full-policy training; MATES consistently outperforms MARL training from scratch across evaluated settings and retains strong performance at unseen team sizes.

    Source: §I · p. 2

  • Takeaways & Limitations

    Under the evaluated observation structure, effective multi-agent behavior can be learned without modifying the policy that encodes individual competence.

    Source: §VI · p. 8

Abstract

from arXiv · show

Multi-agent reinforcement learning (MARL) commonly trains decentralized policies from scratch, requiring agents to acquire individual task competence and coordination simultaneously. Yet many multi-agent problems admit a compatible single-agent counterpart in which the underlying task can be learned in isolation. We introduce Multi-Agent Observation Transformation for Existing Single-Agent Policies (MATES), an input-side adaptation framework for tasks whose multi-agent observations preserve the solo-task information while exposing separately identifiable neighbor information. From multi-agent experience, MATES learns a small adapter that maps this observation into the format expected by a frozen single-agent policy, inducing actions suited to the shared environment without updating the single-agent policy itself. MATES leaves the pretrained policy's internal architecture unchanged and retains the objectives and update procedures of the underlying MARL algorithm. We evaluate MATES using both on- and off-policy algorithms on lifelong pathfinding, navigation, and cooperative discovery, spanning discrete and continuous observation and action spaces. Across all evaluated settings, MATES optimizes only 3.5-7.3% as many parameters as full-policy training while consistently outperforming MARL training from scratch. It approaches the performance of full fine-tuning, remains competitive overall with demonstration-based baselines, and retains strong task performance at team sizes not encountered during training. These results provide evidence that, under this observation structure, effective multi-agent behavior can be learned without modifying the policy that encodes individual competence.

I. INTRODUCTION

MATES addresses the challenge of learning individual competence and coordination simultaneously by adapting a frozen solo policy to multi-agent observations. It learns a lightweight input adapter from multi-agent experience and performs strongly across the evaluated tasks and team sizes.

I. INTRODUCTION

  • MATES is motivated by the challenge that agents trained from scratch must learn task competence and coordination simultaneously, despite many tasks having easier single-agent counterparts.In shared environments, agents’ actions change the conditions under which others must succeed, creating non-stationarity during learning.Source: §I · p. 1
  • The approach targets tasks where the multi-agent observation preserves solo-task information while adding separately identifiable local information about neighboring agents.The solo observation contains information used to learn individual competence; neighbor information supplies context for responding to teammates.Source: §I · p. 1
  • MATES learns a small adapter from multi-agent experience to transform each agent’s observation for a frozen single-agent policy, leaving that policy unchanged.The framework retains the underlying MARL algorithm’s objectives and update procedures.Source: §I · p. 1
  • The evaluation spans lifelong pathfinding, navigation, and cooperative discovery, with discrete and continuous observation and action spaces.The framework is evaluated with both on-policy and off-policy MARL algorithms.Source: §I · p. 2
  • Across lifelong pathfinding, navigation, and cooperative discovery, MATES optimizes 3.5–7.3% as many parameters as full-policy training while consistently outperforming MARL training from scratch.It approaches full fine-tuning performance, remains competitive overall with PegMARL and R2BC, and retains strong performance at team sizes not encountered during training.Source: §I · p. 2

II. RELATED WORK

Related work covers transfer learning and demonstration-guided MARL. MATES differs by retaining a pretrained single-agent policy as a frozen controller and learning observation-side adaptation from multi-agent experience.

A. Transfer and Demonstration-Guided Multi-Agent Learning

  • Transfer learning in reinforcement learning reuses knowledge from related source tasks through policies, value functions, experience, models, or other representations,.Source: A. Transfer and Demonstration-Guided… · p. 2
  • R2BC accumulates per-agent demonstrations for behavior-cloning updates, while PegMARL uses personalized demonstrations to train discriminators that shape rewards during MARL training,.Source: A. Transfer and Demonstration-Guided… · p. 2
  • MATES uses neither demonstration data nor imitation-derived signals; it preserves the pretrained single-agent policy as a frozen action-producing controller and learns observation adaptation from multi-agent experience.Source: A. Transfer and Demonstration-Guided… · p. 2

B. Adaptation of Frozen Models and Policies

MATES adapts a frozen policy by changing its input rather than its internal computation or actions. The paper’s MARL setting uses local information under decentralized training and execution.

B. Adaptation of Frozen Models and Policies

  • Unlike adapter tuning and side-tuning, MATES leaves the pretrained policy’s internal computation unchanged and adapts the observation presented to it.Source: B. Adaptation of Frozen Models and… · p. 2
  • Residual reinforcement learning corrects a fixed controller’s actions, whereas MATES instead adapts its input; VR-Goggles transforms images for a pretrained policy, while MATES targets multi-agent interaction.Source: B. Adaptation of Frozen Models and… · p. 2

A. Decentralized Cooperative Multi-Agent Reinforcement Learning

B. Single-Agent Counterpart and Frozen Policy

MATES assumes a compatible solo task and an observation split between solo-task information and neighbor context. It transfers a frozen solo policy and learns an adapter that changes its input using multi-agent experience.

B. Single-Agent Counterpart and Frozen Policy

  • A solo policy is trained on a compatible counterpart, then frozen during transfer; its parameters are not updated by multi-agent training.Gradients may still pass through the frozen policy to components before its input.Source: B. Single-Agent Counterpart and Frozen… · p. 3, §IV · p. 3
  • MATES focuses on tasks where the multi-agent observation explicitly separates solo-task information from local neighbor information, and it does not evaluate broader settings.The architecture could in principle apply whenever a compatible single-agent counterpart exists, but that broader setting is not evaluated.Source: B. Single-Agent Counterpart and Frozen… · p. 3

IV. METHOD

  • A trainable adapter maps each agent’s full observation into a solo-compatible observation before the frozen policy selects actions.Neighbor information therefore changes the policy’s effective input rather than replacing or modifying its learned controller.Source: §IV · p. 3
  • The adapter is optimized with the underlying MARL algorithm, with gradients passing through the frozen policy to update the adapter alone.Multi-agent experience trains the adapter to transform observations in ways that elicit effective behavior from the preserved solo controller.Source: §IV · p. 3

A. Implementation Variants

MATES permits shared or agent-specific adapters and adapts transferred critics with separate observation adapters. The critic output head may also be trained, but this flexibility had no interpretable performance effect.

A. Implementation Variants

  • MATES can use one shared adapter across homogeneous agents or independently parameterized adapters for agent-specific transformations.Shared adapters keep trainable parameter count independent of team size; the experiments use shared adapters.Source: A. Implementation Variants · p. 3
  • Actor–critic methods transfer and freeze the solo critic, placing a separately initialized observation adapter before it; each critic’s objective optimizes its adapter.When there are multiple critics, each receives its own independently initialized adapter.Source: A. Implementation Variants · p. 3
  • Training the transferred critic’s output head had no interpretable effect on performance compared with keeping it frozen.The remainder of the transferred critic stays frozen in this variant, allowing its output prediction to be recalibrated without relearning its internal representation.Source: A. Implementation Variants · p. 3

V. EXPERIMENTS

The experiments are organized around questions of parameter efficiency, team-size generalization, the source of MATES’ performance, and whether critic value-head adaptation is necessary.

A. Research Questions

  • The experiments ask whether MATES achieves competitive performance while training few transfer parameters, generalizes across team sizes, benefits from a competent solo policy, and needs value-head adaptation.Figure 1 depicts the two-phase action-selection pathway: train the solo policy, freeze it, then train the adapter with the underlying MARL method; critic-side components are omitted.Source: A. Research Questions · p. 4, Figure 1 · p. 4

B. Tasks and Metrics

The evaluation covers one discrete grid pathfinding task and two continuous-control tasks. Each task is assessed using a measure of task completion rather than optimization reward.

B. Tasks and Metrics

  • POGEMA measures lifelong grid pathfinding by throughput, the number of goals reached per environment step, with higher values indicating better performance.Agents avoid obstacles and one another, and receive another goal after reaching one.Source: B. Tasks and Metrics · p. 4
  • VMAS Navigation measures the steps needed to finish an episode in which agents reach assigned goals while avoiding collisions; fewer steps indicate faster completion.Episodes end when all agents simultaneously reach their goals.Source: B. Tasks and Metrics · p. 4
  • VMAS Discovery measures targets covered per episode as agents locate and cover targets while avoiding collisions; higher counts indicate better performance.Covered targets respawn, requiring agents to redistribute themselves continually.Source: B. Tasks and Metrics · p. 4
  • The reported metrics measure task completion directly rather than the rewards used for optimization.Source: B. Tasks and Metrics · p. 4

C. Learning Algorithms

The study tests MATES with distinct on-policy and off-policy MARL algorithms, then compares it with direct-training, fine-tuning, and expert-guided baselines. The evaluation also includes MATES-NH, which keeps the critic’s value head frozen.

C. Learning Algorithms

  • MATES is instantiated with on-policy IPPO and off-policy ISAC to test observation-side transfer under distinct optimization and data-reuse regimes.POGEMA uses IPPO; Navigation and Discovery use both algorithms, with matching solo and conventional MARL network architectures.Source: C. Learning Algorithms · p. 4

D. Methods and Baselines

  • MATES and MATES-NH freeze the transferred solo actor and critic while training observation adapters; MATES additionally trains the critic’s value head.MATES-NH keeps that head frozen.Source: D. Methods and Baselines · p. 4
  • The direct baselines are Solo, Classic MARL, and full fine-tuning, which expands the input layer and optimizes the transferred network during multi-agent training.Solo deploys the pretrained policy without neighbor information or subsequent multi-agent training; Classic MARL starts from random initialization.Source: D. Methods and Baselines · pp. 4–5
  • PegMARL shapes rewards using discriminators trained on solo-expert demonstrations, while R2BC uses a frozen expert to provide online action labels in round-robin order.PegMARL shares discriminators and policy parameters across homogeneous agents; R2BC uses the corresponding Classic MARL policy as its expert.Source: D. Methods and Baselines · pp. 5–6

E. Architectures and Parameter Efficiency

MATES uses compact adapters with decentralized, parameter-shared networks, freezing transferred parameters while training the adapters and, in one variant, the critic’s value head. Its trainable parameter budget is a small fraction of full-network optimization.

E. Architectures and Parameter Efficiency

  • Decentralized, parameter-shared networks serve all homogeneous agents, and observation dimensionality does not grow with team size.POGEMA uses a shared 64-channel ResNet encoder, while Navigation and Discovery use two-layer width-256 MLP backbones.Source: E. Architectures and Parameter… · p. 6
  • Adapters are smaller than the backbones: POGEMA uses a two-layer CNN, while VMAS uses one-hidden-layer width-64 MLP adapters.For VMAS, IPPO has separate actor and critic adapters; ISAC has an actor adapter and a joint observation–action adapter for each Q-network.Source: E. Architectures and Parameter… · p. 6
  • MATES freezes transferred single-agent parameters and trains adapters, with the IPPO MATES variant also updating the critic’s value head.MATES-NH keeps that head frozen and trains only the adapters.Source: E. Architectures and Parameter… · pp. 6–7
  • 3.5%–7.3% of the corresponding full MARL network’s parameters are optimized by MATES across evaluated settings.MATES-NH optimizes 1.0%–5.7%; Table I reports percentages relative to the corresponding fully trainable network.Source: E. Architectures and Parameter… · pp. 6–7

F. Results

Across evaluated tasks and algorithms, MATES performs competitively with direct and expert-guided baselines while optimizing far fewer parameters. It also remains effective at unseen team sizes, and ablations indicate that both solo competence and observation adaptation contribute to performance.

F. Results

  • 3.5%–7.3% of the corresponding full-network parameter count is optimized by MATES while its performance is competitive with Classic MARL, full fine-tuning, PegMARL, and R2BC.No baseline consistently dominates across all settings.Source: F. Results · p. 7
  • MATES remains competitive at team sizes beyond training, frequently matching or exceeding fully trained baselines as its advantage over the unadapted solo policy generally widens.Policies trained with 64 agents in POGEMA and 8 in Navigation and Discovery are evaluated without additional training.Source: F. Results · p. 7
  • The solo policy transfers useful task competence but loses relative performance as team size increases; the adapter-only policy consistently underperforms MATES.A frozen random backbone with a trained adapter does not approach MATES and degrades toward random-policy performance as team size increases.Source: F. Results · p. 7
  • Under IPPO, MATES and MATES-NH perform similarly in Navigation and Discovery, while value-head updates slightly benefit POGEMA, particularly at larger team sizes.The passage characterizes value-head adaptation as not generally necessary, though beneficial in some settings.Source: F. Results · pp. 7–8

VI. CONCLUSION

MATES transfers single-agent policies through compact input-side adapters while preserving the pretrained actor and critic. Across the evaluated tasks, it performs competitively with full MARL training, fine-tuning, and expert-guided baselines using 3.5%–7.3% of the corresponding full-network parameter count; adapters remain effective at unseen team sizes.

VI. CONCLUSION

  • Across discrete and continuous environments with IPPO and ISAC, MATES performs competitively with full MARL training, full fine-tuning, and expert-guided baselines using 3.5%–7.3% of the corresponding full-network parameter count.The conclusion reports that learned adapters remain effective at team sizes not encountered during training.Source: §VI · p. 8

Related papers

Loading 2609.26010v1…