Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularization

1 Introduction

• The joint state-action space grows exponentially with agent count, making global regularization difficult and potentially sparse when offline data are limited.This is especially challenging when dataset size and coverage are limited.Source: §1 · p. 2
• Existing local-level regularization may not remain valid globally, can yield over-conservative learning, and may miss coordinated behavior and credit assignment.The cited approaches also do not guarantee that optimized local policies are jointly optimal under a value decomposition.Source: §1 · p. 2
• OMIGA derives implicit local value regularization from global regularization under value decomposition, guarantees joint optimality, and learns in-sample without OOD action queries.The derivation uses offline-dataset samples without requiring knowledge of the behavior policy or approximation of the regularization term.Source: §1 · p. 2
• OMIGA achieves better performance and faster convergence than strong baselines across the evaluated offline multi-agent MuJoCo and SMAC settings.The evaluation uses various types of offline datasets.Source: §1 · p. 2

2 Related Work

• Offline RL approaches include policy constraints, conservative value estimation, uncertainty penalties, and in-sample learning to address distributional shift and OOD value errors.The paper positions OMIGA within in-sample learning, which avoids policy-generated OOD samples during evaluation.Source: §2 · p. 2
• CTDE and value decomposition address the large joint action space by training with global information and decomposing global values into local values.During execution, each agent acts using its individual observation.Source: §2 · p. 2
• Existing offline MARL RL-based methods impose local regularization without ensuring global validity or capturing cooperation, while GCSL methods may be over-conservative and lack stochastic-environment optimality guarantees.OMIGA is an RL-based method using implicit global-to-local value regularization to combine multi-agent modeling and offline learning.Source: §2 · p. 3

3.1 Notations

• The setting is fully cooperative: agents receive individual observations, share a global reward, and learn jointly maximizing policies from a fixed dataset sampled by behavior policies.Offline policy learning uses the dataset without further environment interaction.Source: §3.1 · p. 3

3.2 CTDE Framework and Value Decomposition

• CTDE trains agents with global information but executes decentralized policies, while value decomposition represents the global Q-value as a weighted linear combination of local Q-values.The local Q-values depend on each agent's observation and action.Source: §3.2 · p. 3

3.3 Behavior-Regularized MDP in Offline RL

• A behavior-regularized MDP penalizes divergence between the learned policy and behavior policy to address offline distributional shift.The regularization is added to rewards, with a scale parameter and a function measuring policy-behavior divergence.Source: §3.3 · p. 4

4.1 Multi-Agent POMDP with Global Value Regularization

• Reverse-KL regularization decomposes into local terms when the behavior policy factorizes, and its global policy-evaluation operator is a γ-contraction.Fixed-point iteration converges to the Q-value under the optimal global policy.Source: §4.1 · pp. 4–5

4.2 Global-to-Local Value and Policy Decomposition

• OMIGA decomposes global Q and V with shared global-observation weights, keeping V free of joint actions and embedding those weights in local policies.The shared weights relate credit assignment for Q and V.Source: §4.2 · pp. 5–6

4.3 Equivalent Implicit Local Value Regularizations

• Implicit local V learning avoids estimating typically unknown behavior policies; its objective adds a conservative value penalty on behavioral data.The objective fits local Q while minimizing a value term, resembling conservative optimal value estimation.Source: §4.3 · p. 6

4.4 Algorithm Summary

• OMIGA learns local V, local Q and weight-offset functions, and local policies in three supervised, in-sample steps, avoiding OOD policy-generated actions.The algorithm updates these components from batches sampled from the offline dataset.Source: §4.4 · p. 7

5.1 Offline Multi-Agent Datasets

• The evaluation uses HAPPO-collected offline multi-agent MuJoCo data and MAPPO-derived SMAC datasets at good, medium, and poor quality across four battle maps.Source: §5.1 · p. 8

5.2 Baselines

• OMIGA is compared with BCQ-MA, CQL-MA, ICQ, and OMAR; BCQ-MA and CQL-MA use linear weighted value decomposition.Source: §5.2 · p. 8

5.3 Comparative Evaluation

• OMIGA outperforms all baselines and achieves state-of-the-art performance in most offline multi-agent MuJoCo and SMAC tasks.Each algorithm is evaluated over 32 independent episodes and five training seeds.Source: §5.3 · p. 8
• OMIGA’s implicit global-to-local design brings global regularization into local value learning, while its state- and Q-value functions are learned in-sample without the agent policy.The authors associate this design with better performance and training stability, and report advantages for continuous-action MuJoCo policy learning.Source: §5.3 · pp. 8–9
• On mixed-quality datasets, heterogeneous suboptimal behavior policies make accurate behavior-policy learning difficult for BCQ-MA and OMAR, making OMIGA’s implicit value learning more appealing.Source: §5.3 · p. 9

5.4 Analyses on Policy Learning with Global Information

• On HalfCheetah and 6h_vs_8z, variants without global information perform worse and less stably than OMIGA on difficult cooperative tasks.Source: §5.4 · p. 9

5.5 Analyses on the Regularization Hyperparameter

• The effect of α is dataset-dependent: extreme values degrade HalfCheetah, while higher α improves 6h_vs_8z and very small α performs poorly.The 6h_vs_8z dataset contains enough good transitions, favoring stronger regularization.Source: §5.5 · p. 9

• OMIGA starts from global behavior constraints and derives equivalent implicit local-level value regularization under value decomposition.Source: §6 · p. 10
• The paper says the design captures global information and multi-agent credit assignment, making learned local policies jointly optimal at the global level.Source: §6 · p. 10
• The authors identify developing offline MARL algorithms with other choices of f functions as future work.Source: §6 · p. 10

A.1 Proof of Theorem 4.1

• The A.1 proof invokes two global state-value functions and an optimal global policy under T*, then concludes with a statement about T* optimality.Source: §A.1 · p. 13

A.2 Proof of Proposition 4.2

• For the reverse-KL behavior-regularized objective, the A.2 proof uses Lagrangian and KKT conditions to derive the optimal global policy.The proof identifies u and β as Lagrangian multipliers for equality and inequality constraints.Source: §A.2 · p. 13
• The reverse-KL optimality condition uses a normalization term u(o), with an optimal u* corresponding to the optimal policy π*.The proof states that a continuous, monotonic function has only one solution u*.Source: §A.2 · p. 14

B.1 Multi-Agent MuJoCo

• Multi-agent MuJoCo partitions a robotic system into agents that control specific joints and collaborate toward shared objectives such as walking.Source: §B.1 · p. 14
• The MuJoCo datasets use HAPPO interactions and include expert, early-stopped medium, medium-replay, and mixed medium-expert data.Source: §B.1 · p. 14

B.2 The StarCraft Multi-Agent Challenge

• SMAC pits MARL-controlled units against a built-in heuristic StarCraft II bot, with rewards based on hit-point damage dealt and received.Source: §B.2 · pp. 14–15
• The experiments use four SMAC maps: 5m_vs_6m and 2c_vs_64zg are hard; 6h_vs_8z and corridor are super hard.Source: §B.2 · p. 15
• The SMAC dataset comes from trained MAPPO agents, has good, medium, and poor quality levels, and samples 1,000 episodes per original large dataset.Source: §B.2 · p. 15

C.1 Details of OMIGA

• OMIGA uses 3-layer ReLU MLPs with 256-unit hidden layers for local Q-value, state-value, and policy networks, and a 2-layer, 64-unit weight network.All networks are optimized with Adam.Source: §C.1 · p. 16

C.2 Details of baselines

• BCQ-MA and CQL-MA use linear weighted value decomposition, with policy and value regularization applied to local Q-values, respectively.The comparison also includes ICQ and OMAR.Source: §C.2 · p. 16

C.3 Hyperparameters

• The regularization weight α is 10 on most tasks, with α = 1 for medium-quality HalfCheetah and α = 2 for poor-quality 6h_vs_8z; most SMAC maps use a 5e-4 learning rate, versus 1e-4 on 2c_vs_64zg.Source: §C.3 · p. 16

• The mixed datasets blend pairs of good, medium, and poor SMAC data at 50% each, yielding heterogeneous behavior policies that make behavior-policy learning harder for BCQ-MA and OMAR.Unlike BCQ-MA and OMAR, OMIGA does not need to learn a behavior policy.Source: §D.1 · p. 17
• OMIGA consistently outperforms the other offline MARL baselines across mixed-dataset experiments, with a more leading performance than on the original datasets.The authors interpret this pattern as indicating benefits from OMIGA’s implicit value regularization.Source: §D.1 · p. 17