PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Decentralized Multi-task Multi-Agent Reinforcement Learning under Partial Observability

Shayegan Omidshafiei, Jason Pazis, Christopher Amato, Jonathan P. How, John Vian

arXiv:1703.06182v4cs.LGcs.AIcs.MA

TL;DR

The paper addresses decentralized multi-agent learning across related partially observable tasks, where local views and teammate exploration create non-stationarity and task identity is unavailable at execution. It combines hysteretic recurrent Q-learning with concurrent experience replay, then distills specialized policies into one recurrent multi-task policy. The resulting approach coordinates agents across target-capture Dec-POMDP tasks without communication assumptions or execution-time task identities.

  • Problem

    Multi-task multi-agent reinforcement learning lacks a formulation for cooperative decentralized learners in partially observable environments where task identity is unavailable during execution.

  • Method

    The approach combines cautious hysteresis, Deep Recurrent Q-Networks, Concurrent Experience Replay Trajectories, and distillation from specialized action-value networks into a generalized recurrent policy.

  • Results

    The method achieves multi-agent coordination using a single joint policy across Dec-POMDP target-capture tasks without task identities during execution.

  • Takeaways & Limitations

    A unified decentralized policy can perform across related partially observable multi-agent tasks while learning and executing without communication assumptions.

  • Takeaways & Limitations

    The formalization assumes agents observe intermediate rewards during learning, unlike Dec-POMDP planning approaches that omit them.

Abstract

from arXiv · show

Many real-world tasks involve multiple agents with partial observability and limited communication. Learning is challenging in these settings due to local viewpoints of agents, which perceive the world as non-stationary due to concurrently-exploring teammates. Approaches that learn specialized policies for individual tasks face problems when applied to the real world: not only do agents have to learn and store distinct policies for each task, but in practice identities of tasks are often non-observable, making these approaches inapplicable. This paper formalizes and addresses the problem of multi-task multi-agent reinforcement learning under partial observability. We introduce a decentralized single-task learning approach that is robust to concurrent interactions of teammates, and present an approach for distilling single-task policies into a unified policy that performs well across multiple related tasks, without explicit provision of task identity.

1. Introduction

The paper formulates multi-task multi-agent reinforcement learning under partial observability, where decentralized agents must coordinate across related tasks without task identity at execution. It combines robust single-task learning with policy distillation into one generalized recurrent policy.

  • Motivation: Multi-task reinforcement learning seeks performance generalization across several related tasks rather than specialization on one task.The motivating example is autonomous underwater vehicles detecting and repairing faults under varying environmental conditions.
  • Motivation: Partial observability, limited communication, and teammates’ concurrent actions make local learning difficult because agents perceive non-stationary environments.Agents must learn from local noisy viewpoints while teammates explore and interact with the environment.
  • Problem formulation: The proposed MT-MARL problem uses cooperative independent learners in partially observable stochastic environments with local observations and joint rewards, without explicit task identity during execution.Each task is formalized as a Dec-POMDP, and the objective is execution-time performance across related tasks.
  • Approach: The first phase combines cautiously optimistic learners, DRQNs, and CERTs to obtain sample-efficient and stable decentralized coordination for single tasks.CERTs extend experience replay with concurrently stored and sampled multi-agent trajectories.
  • Approach: The second phase distills specialized action-value networks into a generalized recurrent multi-task network that performs across related tasks without communication assumptions.Both specialized and distilled policies achieve good performance on multi-agent target-capture Dec-POMDP domains.

2. Background

The background develops reinforcement learning, partial observability, and cooperative multi-agent learning, emphasizing how local views and teammate interactions complicate decentralized control and motivate multi-task transfer.

  • Reinforcement learning: An MDP models states, actions, transitions, rewards, and discounting, while model-free reinforcement learning directly learns Q-values or policies without an explicit environment model.Model-free methods can be more space- and computation-efficient than model-based approaches.
  • Reinforcement learning: Q-learning updates action values using a temporal-difference backup, but tabular learning is unsuitable for large state and action spaces.Deep Q-learning addresses scale with neural-network function approximation and replay-based training.
  • Partial observability: POMDPs represent settings where agents observe noisy signals of latent states, making recurrent networks useful for compressing observation histories.Policies map observation histories to actions rather than relying on fully observed states.
  • Multi-agent reinforcement learning: Cooperative MARL agents share a joint return, and independent learners use only local actions or observations rather than observing all agents’ joint actions.The paper adopts independent learners because joint-action observability is a strong assumption in partially observable domains.
  • Multi-agent reinforcement learning: Teammate interactions make local multi-agent domains non-stationary and can produce shadowed equilibria in which locally optimal actions form a globally sub-optimal joint action.Effective coordination must address this mismatch between local decisions and joint outcomes.
  • Transfer and multi-task learning: Transfer learning generalizes knowledge from source tasks to a target task, whereas multi-task reinforcement learning learns policies across related target tasks.The paper situates its problem at the intersection of multi-task learning and decentralized partial observability.

3. Related Work

Prior work covers partially observable MARL, independent Q-value learning, hysteretic coordination, and transfer learning, but is dominated by single-agent or fully observable settings.

  • Partially observable MARL: Partially observable MARL has received limited attention, with prior methods including gradient ascent, simulator-supported linear programs, and model-based interleaved learning.These approaches address destabilization or policy improvement through different centralized or model-dependent mechanisms.
  • Independent learners: Independent-learning approaches include decentralized Q-learning, distributed Q-learning, policy hill climbing, and frequency-based action-selection heuristics.Most existing independent learners assume full observability, limiting their direct fit to the paper’s setting.
  • Coordination under non-stationarity: Hysteretic Q-learning uses cautious optimism to stabilize policies while teammates explore and has shown empirical success against more complex methods.This record motivates its use as the foundation for the proposed partially observable approach.
  • Transfer and multi-task learning: Transfer and multi-task reinforcement learning surveys largely target single-agent, fully observable settings, while earlier work develops task-similarity metrics, policy libraries, and lifetime-reward objectives.The related literature therefore provides transfer mechanisms without fully addressing decentralized partial observability.

4. Multi-task Multi-agent RL

The paper formalizes decentralized multi-task MARL with task variation, local observations, and joint rewards, then characterizes the non-stationarity faced by independent learners.

  • Dec-POMDP formalization: A Dec-POMDP models cooperative agents, states, joint actions, transitions, joint observations, observation probabilities, and discounted joint rewards.Each agent selects actions from local observation histories, while the team receives a shared reward.
  • Decentralized policies: The Dec-POMDP framework uses decentralized independent learners whose local policies map each agent’s available information to actions.The approach assumes decentralized learning and execution rather than observing all joint actions.
  • Assumptions: The framework assumes agents observe intermediate rewards during reinforcement learning, unlike Dec-POMDP planning approaches that omit such observations.The assumption follows typical MARL practice and is explicitly retained in the formalization.
  • Local non-stationarity: Independent learners face local non-stationarity because teammates’ changing actions can violate stationarity conditions for each agent’s decision process.This motivates designing a single-task decentralized learner specifically for non-stationary Dec-POMDP learning before addressing multi-task distillation.
  • Multi-task formulation: A partially observable MT-MARL task shares an underlying domain while task-specific transition, reward, and observation functions define its variation.This separates common domain structure from task-dependent dynamics and observations.
  • Multi-task formulation: During learning, episodes sample task identities, but execution withholds those identities; the objective is maximizing average empirical execution-time return across episodes.This formalizes the requirement for a policy that generalizes across tasks without explicit task input.

5. Approach

The approach first stabilizes decentralized single-task learning under partial observability, then distills task-specific recurrent Q-networks into a unified policy without task identity. Concurrent experience replay aligns agents’ trajectories while preserving decentralized learning and supports both specialization and multi-task regression.

  • Approach: The two-phase method specializes agents on individual tasks before unifying their DRQNs into one policy across tasks.The unified policy is trained without explicit task identity.
  • 5.1.1. DECENTRALIZED HYSTERETIC DEEP RECURRENT Q-NETWORKS (DEC-HDRQNS): Dec-HDRQNs combine hysteretic learning with recurrent Q-networks to handle teammate exploration, non-stationarity, partial observability, and memory-based decision making.Hysteresis permits eventual degradation of Q-values that were overestimated from unrelated outcomes.
  • 5.1.2. CONCURRENT EXPERIENCE REPLAY TRAJECTORIES (CERTS): CERTs store agents’ experience tuples concurrently by episode, timestep, and agent, while allowing varying trajectory lengths and discarding old episodes FIFO.Concurrent storage keeps sampled experiences synchronized across agents without explicit communication during learning.
  • 5.1.3. TRAINING DEC-HDRQNS USING CERTS: Agents sample concurrent episode minibatches with coincident trace start times, using shared random-number-generator seeds to preserve decentralization.The procedure requires only one-time offline consensus before learning begins.
  • 5.1.3. TRAINING DEC-HDRQNS USING CERTS: Concurrent trace starts are sampled from an expanded timestep interval so every experience has equal probability of appearing in recurrent updates.This addresses underuse of final τ episode timesteps and is especially important for tasks with only terminal rewards.
  • 5.2. Phase II: Dec-POMDP MT-MARL: In Phase II, specialized DRQNs generate regression CERTs for each task, and each agent trains one distilled DRQN by regressing against their action-value vectors.The multi-task phase iteratively alternates data collection with regression and uses a tempered KL-divergence loss.

6. Evaluation

The evaluation tests Dec-HDRQN on increasingly difficult partially observable capture tasks, then evaluates distillation into a single multi-task policy. Hysteresis improves coordination and stability, while distillation achieves near-nominal performance without explicit task identity.

  • Evaluation design: Dec-HDRQN is evaluated on MAST and MAMT capture domains with varying grid sizes, agent-target assignments, noisy transitions, and sparse terminal rewards.The domains use recurrent networks and test coordination under partial observability and target occlusion.
  • Task specialization: Dec-DRQN becomes unstable on the 5 × 5 MAMT task and fails to learn on the 6 × 6 and 7 × 7 tasks, whereas Dec-HDRQN performs well across the tasks.The result is attributed to hysteresis providing stability against alter-exploration and non-stationarity.
  • Task specialization: 0.05±0.16 versus 0.76±0.11: at 8×8 MAST, Dec-DRQN receives lower value than Dec-HDRQN under full observability.The comparison uses two agents and Pf = 0.
  • Task specialization: Dec-HDRQN successfully coordinates three agents in MAMT despite only 0.02% of the 4 × 4 joint state space yielding non-zero reward.Dec-DRQN receives near-zero return after training, while increased Pf = 0.6 produces high variance on the 3 × 3 task.
  • Task specialization: β ∈[0.1, 0.6] consistently enables learning, with β ∈[0.4, 0.5] achieving best performance; lower β corresponds to higher optimism.Anticipated returns Q(o0, a0) overestimate empirical returns because of hysteretic optimism.
  • Multi-task distillation: A single distilled policy achieves near-nominal execution-time performance across MAMT tasks from 3×3 through 6 × 6 without explicit task identity.Specialization runs for 70K epochs, followed by 100K epochs of distillation; simultaneous Multi-HDRQN learning performs poorly even with extended training.

7. Contribution

The paper formulates and addresses multi-task multi-agent reinforcement learning under partial observability. Its approach combines robust decentralized learning with distillation to coordinate agents using one policy across sparse-reward tasks without task identities.

  • Contribution: The paper introduces a formulation and approach for multi-task multi-agent reinforcement learning under partial observability.The evaluated tasks vary grid size, target assignments and dynamics, and sensor failure probabilities.
  • Contribution: The approach combines hysteretic learners, DRQNs, CERTs, and distillation to achieve coordination using a single joint policy across Dec-POMDP tasks.The tasks have sparse rewards and execution does not provide task identities.
  • Contribution: The MAMT capture tasks provide a parametric benchmark family for multi-agent multitask learning.Future work includes skills, heterogeneous agents, and more complex domains with many tasks.

B. Empirical Results: Learning on Multi-agent Single-Target (MAST) Domain

The MAST results compare Dec-DRQN and Dec-HDRQN using empirical returns and anticipated values under full observability. Both plots are evaluated over matched batches of randomly initialized episodes and actions.

  • Experimental setup: MAST results use 2 agents with Pf = 0.0, disabling observation flickering during evaluation.The domain results compare Dec-DRQN and Dec-HDRQN.
  • Experimental setup: Each training epoch is evaluated on a batch of 50 randomly-initialized episodes.This applies to both the empirical-return and anticipated-value plots.
  • Experimental setup: Anticipated-value plots use the exact starting states and actions from the corresponding empirical-return episodes.The anticipated values are plotted alongside the returns for matched trajectories.

C. Empirical Results: Learning on MAMT Domain

The MAMT results compare Dec-DRQN and Dec-HDRQN under observation flickering and examine anticipated values during training. The figures use matched episode batches and report agent convergence trends under shared rewards.

  • Experimental setup: MAMT results compare Dec-DRQN and Dec-HDRQN with 2 agents and Pf = 0.3.The empirical and anticipated-value plots are evaluated during training.
  • Experimental setup: Each training epoch is evaluated on a batch of 50 randomly-initialized episodes, with anticipated values computed for their exact starting states and actions.The left and right plots therefore correspond to the same episode actions and initial conditions.
  • Value convergence: For a given task, agents show similar anticipated-value convergence trends because they share reward.Differences mainly arise from random initial states and independently sampled target occlusion events.
  • Three-agent results: Figure 10 reports MAMT domain results for Dec-DRQN and Dec-HDRQN with 3 agents across 3 × 3 and 4 × 4 tasks.The flickering probabilities are Pf = 0.6 for 3 × 3 and Pf = 0.1 for 4 × 4.

D. Empirical Results: Learning Sensitivity to Dec-HDRQN Negative Learning Rate β

The experiments examine learning sensitivity to β and recurrent training tracelength in a 6 × 6, 2-agent MAMT domain with Pf = 0.25. β spans decentralized to distributed Q-learning settings, and plots track training across epochs.

  • Figure 11 varies β in the 6 × 6, 2-agent MAMT domain with Pf = 0.25.β = 1 corresponds to Decentralized Q-learning, while β = 0 corresponds to Distributed Q-learning without the distributed policy update step.
  • The sensitivity plots use training epoch as the horizontal axis, spanning 0.0 through 40K epochs.
  • Figure 12 varies the recurrent training tracelength parameter under the same 6 × 6, 2-agent MAMT setting.The evaluated tracelengths are 1, 2, 4, and 8.

F. Empirical Results: Multi-tasking Performance Comparison

The comparison evaluates distillation against Multi-HDRQN on multi-task MAMT tasks. Multi-DRQN does not reach specialized-level performance within 500K epochs, whereas the proposed distillation approach reaches nominal performance after 100K epochs.

  • After 500K training epochs, Multi-DRQN failed to achieve specialized-level performance on all trained MAMT tasks.The tasks ranged from 3 × 3 through 6 × 6 MAMT environments.
  • The proposed MT-MARL distillation approach achieved nominal performance after 100K epochs.
  • Figure 13 reports multi-task performance in the MAMT domain with n = 2 agents and Pf = 0.3.
Loading 1703.06182v4…