PaperlayerView exact revision on arXiv

Source-linked AI summary

Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning

Yiqin Yang, Xiaoteng Ma, Chenghao Li, Zewu Zheng, Qiyuan Zhang, Gao Huang, Jun Yang, Qianchuan Zhao

arXiv:2106.03400v2cs.AI

TL;DR

Offline multi-agent RL is challenging because larger action spaces accompany learning from fixed datasets, while current offline algorithms can suffer accumulated extrapolation error. The paper proposes ICQ, which uses dataset pairs for value estimation and decomposes the joint policy under an implicit constraint. It reports controlled extrapolation error across agent counts and performance on complex multi-agent offline tasks.

  • Problem

    Offline multi-agent RL has received little attention despite its larger state and action spaces, and current offline methods are ineffective amid accumulated extrapolation error.

    Source: §1 · p. 1

  • Method

    ICQ estimates values using only state-action pairs in the dataset and extends to multi-agent tasks by decomposing the joint policy under an implicit constraint.

    Source: §1 · p. 2, §4.1.1 · p. 5

  • Results

    ICQ controls extrapolation error within a reasonable range under any number of agents and learns from complex multi-agent datasets, including StarCraft II tasks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper presents the multi-agent ICQ version as the first offline algorithm capable of learning from complex multi-agent datasets.

    Source: §7 · p. 10

Abstract

from arXiv · show

Learning from datasets without interaction with environments (Offline Learning) is an essential step to apply Reinforcement Learning (RL) algorithms in real-world scenarios. However, compared with the single-agent counterpart, offline multi-agent RL introduces more agents with the larger state and action space, which is more challenging but attracts little attention. We demonstrate current offline RL algorithms are ineffective in multi-agent systems due to the accumulated extrapolation error. In this paper, we propose a novel offline RL algorithm, named Implicit Constraint Q-learning (ICQ), which effectively alleviates the extrapolation error by only trusting the state-action pairs given in the dataset for value estimation. Moreover, we extend ICQ to multi-agent tasks by decomposing the joint-policy under the implicit constraint. Experimental results demonstrate that the extrapolation error is successfully controlled within a reasonable range and insensitive to the number of agents. We further show that ICQ achieves the state-of-the-art performance in the challenging multi-agent offline tasks (StarCraft II). Our code is public online at https://github.com/YiqinYang/ICQ.

1 Introduction

Offline RL learns from fixed datasets without environment interaction, but multi-agent settings enlarge the action space and intensify extrapolation error. ICQ addresses this by restricting value estimation to dataset pairs and is evaluated on multi-agent StarCraft II tasks.

1 Introduction

  • Offline RL learns from fixed datasets without interacting with environments, addressing the expense or risk of real-world interaction.The paper identifies autonomous driving as an example of a setting where interaction can be costly or risky.Source: §1 · p. 1
  • In multi-agent offline RL, unseen state-action pairs grow exponentially with agent count, quickly accumulating extrapolation error and impairing performance.The paper reports that current offline algorithms, including methods with value-decomposition structure, are ineffective in these tasks.Source: §1 · pp. 1–2
  • ICQ alleviates extrapolation error by using only dataset state-action pairs for Q-value estimation and decomposing the joint policy under the implicit constraint.Its Q-value evaluation is SARSA-like, and policy learning is converted into supervised regression.Source: §1 · p. 2
  • ICQ controls extrapolation error within a reasonable range under any number of agents and is evaluated on challenging offline StarCraft II tasks.The paper also evaluates single-agent ICQ on D4RL and reports generality across single- and multi-agent, discrete- and continuous-control scenarios.Source: §1 · p. 2

2 Background

The background formalizes cooperative partially observable multi-agent tasks and the offline setting, then outlines centralized training and decentralized execution and BCQ.

2 Background

  • A cooperative multi-agent task is modeled as a Dec-POMDP, with a shared reward and a joint action formed from agents’ individual actions.The tuple includes states, actions, transitions, rewards, observations, agent count, and discount factor.Source: §2 · p. 2
  • Each agent conditions a stochastic policy on its action-observation history, while the dataset contains trajectories generated by the behavior policy.The setting is partially observable; agents draw observations according to an observation function.Source: §2 · p. 2
  • Under centralized training and decentralized execution, training can use the true state and shared information, but execution gives each agent only its own action-observation history.The training stage also has access to every agent’s history.Source: §2 · pp. 2–3
  • BCQ constrains offline policy optimization by selecting the highest-valued action among samples from a generative model intended to produce previously seen actions.A perturbation model adjusts sampled actions within a bounded range to increase their diversity.Source: §2 · p. 3

3 Analysis of Accumulated Extrapolation Error in Multi-Agent RL

The analysis links offline extrapolation error on seen pairs to transitions into unseen pairs, whose impact grows with action-space size. A toy multi-agent example illustrates the resulting divergence in BCQ estimates and the more stable ICQ estimates.

3.1 Extrapolation Error Propagation in Offline RL

  • For a deterministic MDP, error on seen pairs is proportional to the transition probability from seen to unseen state-action pairs.The analysis separates seen-pair error from unseen-pair error and relates propagation to the seen-to-unseen transition block.Source: §3.1 · pp. 3–4

3.2 Toy Example

  • In the toy MMDP, BCQ’s joint action-value estimates diverge as agent count increases, while ICQ maintains a reasonable Q-value.The comparison uses a limited dataset of 32 QMIX-generated trajectories, with the amount of reward-bearing trajectories controlled across datasets.Source: Figure 2 · p. 4, §3.2 · p. 4

3.1 Extrapolation Error Propagation in Offline RL

  • As the action space grows with agent count, limited offline data may be insufficient to reduce seen-to-unseen transitions, allowing estimation error to propagate.With function approximation, unseen-pair values can be arbitrary during training, potentially making Q-values extreme even for seen pairs.Source: §3.1 · p. 4

3.2 Toy Example

  • The toy example and theoretical analysis together show that accumulated extrapolation error can make Q-estimates increasingly prone to divergence as the number of agents rises.The figure caption reports similar true values across agent counts and accurate ICQ estimates in the illustrated task.Source: §3.1 · p. 3, §3.2 · p. 4

4 Implicit Constraint Approach for Offline Multi-Agent RL

ICQ limits offline value estimation to dataset-supported state-action pairs, reweighting target values to alleviate extrapolation error. The paper extends this approach to multi-agent tasks through joint-policy decomposition and λ-return estimation.

4 Implicit Constraint Approach for Offline Multi-Agent RL

  • ICQ estimates values using seen dataset pairs and extends the approach to multi-agent tasks with value decomposition and λ-return estimation.The method is presented as addressing accumulated extrapolation error in offline multi-agent reinforcement learning.Source: §4 · p. 4

4.1.1 Implicit Constraint Q-learning

  • ICQ derives an importance weight from an implicit constraint and reweights target Q-values, yielding a SARSA-like update without unseen pairs.The paper contrasts this with methods that constrain only the policy loss while retaining standard Bellman evaluation.Source: §4.1.1 · p. 5

4.1.2 Theoretical Analysis

  • TICQ is not a contraction, but its values remain within a reasonable range and it converges to TBCQ at an exponential rate in α.The analysis also states that iteratively applying TICQ yields a Q-function not far from the batch-optimal Q-function.Source: §4.1.2 · p. 6

4.1.3 Algorithm

  • Policy training projects the non-parametric optimal policy into the parameterized policy space by minimizing KL distance on the batch distribution.Tuning α from 0 to ∞ interpolates between Q-learning and behavioral cloning.Source: §4.1.3 · p. 6

4.2 Extending ICQ to Multi-Agent Tasks

  • The multi-agent version, ICQ-MA, extends ICQ’s target-Q reweighting framework to multi-agent tasks.The paper names this multi-agent version ICQ-MA.Source: §4.2 · p. 7

4.2.1 Decomposed Multi-Agent Joint-Policy under Implicit Constraint

  • Under CTDE, decentralized policies factorize by agent; a linear value decomposition and mixer-generated weights support the constrained joint policy.The weights reflect agent roles, and a weight approaching zero makes that agent’s policy approximately behavioral cloning.Source: §4.2.1 · p. 7

4.2.2 Multi-Agent Value Estimation with λ-return

  • ICQ extends its operator to n-step estimation using Q(λ), which weights future temporal-difference signals with a decay sequence.The hyperparameter λ balances bias and variance.Source: §4.2.2 · p. 7

5 Related Work

The related work surveys offline RL approaches to limiting policy mismatch and extrapolation error, then contrasts them with ICQ’s value estimation and policy constraints. It also reviews model-based and safe policy improvement methods and their stated challenges.

5 Related Work

  • The review groups prior single-agent offline RL methods into dynamic programming, model-based, and safe policy improvement approaches.Policy-constraint methods are identified as most closely related to ICQ.Source: §5 · p. 8
  • ICQ estimates the current policy’s Q-function using behavior-policy actions, unlike methods that constrain policy mismatch or estimate the behavior policy’s value.The review also notes uncertainty-based approaches, whose performance is limited by the high-fidelity requirements of uncertainty estimates.Source: §5 · p. 8
  • Model-based methods face challenges with high-dimensional image observations and long horizons, while safe policy improvement requires estimating the behavior policy.Accurately estimating that policy is especially difficult when data come from multiple sources.Source: §5 · p. 9

6 Experiments

The experiments compare ICQ-MA and ICQ with baselines on StarCraft II and D4RL, then examine ICQ-MA’s components, constraint strength, data quality, and computational cost. ICQ-MA leads the reported StarCraft II comparisons, while ICQ reaches state-of-the-art performance on many D4RL tasks.

6.1 Multi-Agent Offline Tasks on StarCraft II

  • Each StarCraft II map has 3,000 trajectories, made by evenly mixing data from three behavior-policy levels.The datasets cover ten maps; each algorithm runs with five seeds.Source: §6.1 · p. 9
  • ICQ-MA significantly outperforms all baselines and achieves state-of-the-art performance on every tested StarCraft II map.QMIX, BCQ-MA, and CQL-MA perform poorly; BC-MA performs soundly in near-deterministic StarCraft II.Source: §6.1 · p. 9

6.2 Single-Agent Offline Tasks on D4RL

  • ICQ achieves state-of-the-art performance on many D4RL tasks spanning gym domains, Adroit, and AntMaze.The Adroit tasks involve a 24-DoF robotic hand, while AntMaze requires composing sub-optimal trajectory segments to reach goals.Source: §6.2 · p. 9

6.3 Ablation Study

  • Alternative Q-value estimators and removing the decomposed implicit-constraint module produce poor performance; one-step estimation introduces more bias.The authors report that n-step estimation is not the critical factor behind ICQ-MA’s improvement.Source: §6.3 · p. 10
  • An intermediate constraint coefficient α performs best, between the Q-learning limit at α → 0 and behavior-cloning limit at α → ∞.Smaller α relaxes the constraint and tends to maximize reward.Source: §6.3 · p. 10
  • ICQ-MA is robust to data quality, whereas BC-MA’s performance drops drastically as data quality deteriorates.The comparison varies datasets from Good, Medium, and Poor behavior policies.Source: §6.3 · p. 10
  • BCQ-MA consumes 70% of ICQ-MA’s training time under the same SMAC training steps, while ICQ-MA achieves excellent benchmark performance.The experiments report that ICQ-MA takes somewhat longer than BCQ-MA.Source: §6.3 · p. 10

7 Conclusion

The conclusion presents ICQ as a way to alleviate accumulated extrapolation error by trusting dataset state-action pairs, and identifies its multi-agent version as capable of learning from complex datasets. The appendices provide algorithm procedures and a theorem concerning error propagation in deterministic MDPs.

7 Conclusion

  • ICQ alleviates extrapolation error by trusting dataset state-action pairs, and its multi-agent version learns from complex multi-agent datasets.The authors describe it as the first multi-agent offline algorithm capable of doing so, to their knowledge.Source: §7 · p. 10

A Algorithms

  • The appendix presents single-agent and multi-agent ICQ algorithms, initializing critic and actor networks and, for ICQ-MA, a mixer network.The algorithms take an offline buffer as input and include policy-training steps.Source: A Algorithms · p. 15

B.1 Proof of Theorem 1

  • Theorem 1 concerns propagation of ε_b to ε_s under a deterministic MDP.The provided theorem statement relates that propagation to a transition-matrix expression.Source: §B.1 · p. 16

B.2 Proof of Theorem 2

The appendix analyzes ICQ operator convergence, how agent count affects distribution shift, the constrained policy optimization, and joint-policy decomposition. It also identifies the scope of an additional StarCraft II module-ablation figure.

B.2 Proof of Theorem 2

  • The ICQ operator’s convergence rate is exponential in α, under the proof’s batch-constrained action condition.The proof assumes the behavior policy assigns positive probability to the actions considered.Source: §B.2 · pp. 17, 19

B.3 Proof of Remark 3.2

  • With limited datasets and a fixed number of seen state-action pairs, the concentrability coefficient increases monotonically with agent count in the MMDP analysis.The appendix states C(Π1) < C(Π2) < · · · < C(Πn).Source: §B.3 · p. 19

B.4 Proof of Remark 2

  • The proof frames the constrained policy objective as a convex optimization problem and introduces α as its Lagrangian coefficient.The constraint is expressed using KL divergence between π and µ.Source: §B.4 · p. 20

B.5 Proof of Theorem 3

  • Under a linearly decomposed joint action-value function, Theorem 3 states that the joint policy can be decomposed under the implicit constraint.Source: §B.5 · p. 20

C.1 Additional Ablation Results in StarCraft II

  • Figure 6 covers module ablations in additional StarCraft II environments; compare the ablated modules across those environments.Source: Figure 6 · p. 21

C.2 Additional Results in MMDP

The MMDP appendix reports that BCQ’s estimates diverge as agent count rises while ICQ remains accurate, and includes additional D4RL and MMM-map results.

C.2 Additional Results in MMDP

C.3 Additional Results in D4RL

  • Figure 8 compares D4RL performance across different implementations of ICQ.Source: Figure 8 · p. 22

C.4 Ablation Study

D.1 Implementation details of ICQ

ICQ implementation details describe two ways to calculate ρ, while the baseline appendix outlines the training approaches used for BCQ-MA, CQL-MA, and BC-MA.

D.1 Implementation details of ICQ

  • One option learns the behavior policy with a conditional variational auto-encoder and samples 100 actions to estimate Z(τ) at each value update.Source: §D.1 · p. 23
  • The learned behavior model may introduce unseen pairs; when their extrapolation error is large, the unsafe estimate is truncated and the target Q-value tends to be conservative.Source: §D.1 · p. 23
  • The mini-batch softmax option avoids unseen pairs but ignores state differences, introducing bias that did not harm performance much; it is preferred for multi-agent tasks.Source: §D.1 · p. 23

D.2 Baselines Details

  • BCQ-MA combines weighted per-agent Q-values with a state bias and decomposes its perturbation model by agent.Source: §D.2 · p. 23
  • CQL-MA uses decomposed policy gradients and n-step off-policy targets, while BC-MA trains only the policy by minimizing its stated loss.Source: §D.2 · pp. 23–24

E Multi-Agent Offline Dataset Based on StarCraft II

The StarCraft II dataset organizes maps by difficulty and behavior policies by episode returns; the appendix also points to multi-agent hyperparameters.

E Multi-Agent Offline Dataset Based on StarCraft II

E.1 Hyper-parameters

  • Table 4 presents multi-agent hyperparameters; single-agent hyperparameters are available in the official code.Source: §E.1 · p. 24, Table 4 · p. 24
Loading 2106.03400v2…