Source-linked AI summary
Reducing Overestimation Bias in Multi-Agent Domains Using Double Centralized Critics
Johannes Ackermann, Volker Gabler, Takayuki Osa, Masashi Sugiyama
TL;DR
Multi-agent RL must coordinate agents without exponential centralized action spaces or the nonstationarity problems of independent learning, while overestimation bias may hinder learning. The paper proposes MATD3 with double centralized critics and applies it to cooperative-competitive and robotic tasks, where it generally outperforms existing approaches and learns decentralized policies.
Problem
Multi-agent RL faces exponential action-space growth under centralized control and violated Markov assumptions under independent learning; the paper investigates whether overestimation bias also persists in centralized critics.
Method
MATD3 uses decentralized execution with centralized training and addresses overestimation bias through double centralized critics.
Results
MATD3 significantly outperforms MADDPG on most particle-domain tasks and outperforms existing MARL methods while learning decentralized policies for a simulated Ant task.
Takeaways & Limitations
Modeling robotic systems as multi-agent systems can support learning decentralized policies without a centralized controller.
Takeaways & Limitations
The evaluated domains are deterministic, whereas real-world applications are mostly stochastic and stochasticity can increase value-function overestimation.
Abstract
from arXiv · showhide
Many real world tasks require multiple agents to work together. Multi-agent reinforcement learning (RL) methods have been proposed in recent years to solve these tasks, but current methods often fail to efficiently learn policies. We thus investigate the presence of a common weakness in single-agent RL, namely value function overestimation bias, in the multi-agent setting. Based on our findings, we propose an approach that reduces this bias by using double centralized critics. We evaluate it on six mixed cooperative-competitive tasks, showing a significant advantage over current methods. Finally, we investigate the application of multi-agent methods to high-dimensional robotic tasks and show that our approach can be used to learn decentralized policies in this domain.
1 Introduction
Multi-agent tasks are difficult to learn efficiently because centralized formulations create exponentially growing action spaces, while independent learners violate the Markov assumption. The paper addresses these challenges with centralized training, decentralized execution, and a method for reducing overestimation bias.
- Centralizing a multi-agent problem creates an action space that grows exponentially with the number of agents.
- Independent learners treat other agents as part of the environment, but their changing behavior violates the Markov assumption.
- Centralized training with decentralized execution preserves the Markov property during training while allowing individual agents to act separately.
- Learning decentralized robotic policies can avoid a centralized controller and the substantial model knowledge required for decentralized control.
- The proposed MARL method addresses overestimation bias, outperforms previous methods in most evaluated cooperative-competitive tasks, and learns decentralized policies for a simulated robotic task.
2 Background
The background develops multi-agent Markov games, value-based and policy-gradient reinforcement learning, and centralized-critic methods. It then motivates double-critic approaches by describing overestimation in Q-learning and its mitigation in TD3 and related multi-agent methods.
- A Markov game models N agents with joint-action transitions, individual rewards, and agent-specific observations available during decentralized execution.
- Q-learning estimates expected returns recursively, while Double Q-learning reduces positive bias by learning two Q-functions and using one to update the other.
- Policy-gradient methods optimize policy parameters directly, and deterministic policy gradients use deterministic policies that can learn faster in some domains.
- DDPG applies deterministic policy gradients to continuous control with off-policy replay-buffer updates and target networks.
- TD3 addresses DDPG overestimation with two Q-networks, target networks, minimum-Q targets, target-policy smoothing, and delayed policy updates.
- MADDPG extends DDPG to multi-agent settings by combining decentralized execution with centralized critics that access all agents’ policies.
- The multi-agent replay buffer stores joint states, actions, rewards, and next states for centralized training.
3 Overestimation Bias in a Centralized Critic
The paper investigates whether MADDPG exhibits overestimation bias in multi-agent settings and finds that this bias can occur, with worse final performance when it does. The evaluation is limited to deterministic domains, whereas stochasticity can increase overestimation.
- The authors investigate whether overestimation bias found in other RL methods persists in the multi-agent MADDPG setting.
- Under a short-term stationarity approximation, the centralized critic can be reduced to the single-agent setting where overestimation has been shown in DDPG.
- MADDPG tends to overestimate Q-values, especially during earlier episodes, although this does not occur in every individual run.
- When overestimation occurs, it leads to significantly worse final performance.
- The evaluated domains are deterministic, while stochasticity has been shown to increase value-function overestimation by adding noise to function-approximator errors.
4 Multi-Agent TD3
MATD3 extends TD3 to multi-agent centralized training with decentralized execution by learning two centralized critics and using their minimum to reduce overestimation. It also applies target policy smoothing and delayed policy updates, although delayed updates may slow adaptation in adversarial settings.
- MATD3 extends TD3 to multi-agent centralized training with decentralized execution and learns two centralized critics for each agent.
- MATD3 updates critics with the minimum of both estimates to reduce overestimation bias.
- Using the minimum may cause underestimation, but the authors consider it preferable because overestimated actions are selected more often and propagate errors into future targets.
- Target policy smoothing adds clipped Gaussian noise to all agents’ actions during critic updates as regularization based on similar actions having similar values.
- Target networks and policies are updated only after every d critic updates, allowing the critic to become more accurate before policy updates.
- Delayed policy updates are especially important in multi-agent domains but may slow adaptation to an adversary in adversarial settings.
5 Evaluation in Particle Environments
MATD3 is evaluated against MADDPG across cooperative and adversarial particle-environment tasks, with performance depending on whether stable strategies or rapid adaptation is required. The evaluation also examines delayed policy updates, target policy smoothing, and overestimation across agent counts.
- Evaluation setup: The evaluation uses six two-dimensional continuous particle tasks: two cooperative tasks and four adversarial tasks with agent teams facing a single adversary.Agents can exert forces and may communicate through discrete channels.
- Cooperative environments: MATD3 learns a significantly better final policy than MADDPG in Cooperative Communication, while both achieve similar final performance in Cooperative Navigation.On Cooperative Navigation, MATD3 learns the better policy significantly faster.
- Competitive environments: MATD3 outperforms MADDPG in three of four competitive environments using 0-1 normalized final rewards averaged across 20 trials.MATD3 wins in tasks requiring stable winning strategies, whereas MADDPG performs better in Covert Communication.
- Competitive environments: In Covert Communication, MADDPG outperforms MATD3 because rapid adaptation to the adversary is advantageous and MATD3 adapts more slowly.The task requires agents to learn a communication strategy that the adversary must decode.
- Ablations: Less frequent policy updates generally reduce variance and improve final performance, except in Covert Communication, where they help the adversary decrypt the team’s communication.Delayed updates are intended to allow the critic to converge more before policy updates.
- Ablations: Target policy smoothing does not significantly improve performance, and relative overestimation does not differ significantly across the evaluated numbers of agents.Target policy smoothing was tested on Cooperative Navigation across different noise levels.
6 Learning Fully Decentralized Controllers for Robotic Systems
The paper models a high-dimensional robotic system as multiple partially observing agents that learn coordination without a centralized controller. On Ant-v2, MATD3 outperforms MADDPG and independent TD3, while fully observable single-agent TD3 remains an easier comparison.
- Decomposition to Multiple Agents: The robot is partitioned into agents that coordinate through rewards and a centralized critic while executing decentralized policies.This removes the need for centralized control and additional model knowledge for designing a decentralized controller.
- Decomposition to Multiple Agents: Each Ant-v2 agent controls two legs, observes its own positions, velocities, and forces, and receives torso location and velocity plus only the other legs’ positions.The resulting task gives agents partial information about one another’s state.
- Results: MATD3 performs better than MADDPG because it usually recovers from the stationary local optimum and achieves locomotion in the required direction.Independent learner TD3 agents failed to learn a successful policy in all trials.
- Results: Independent learner TD3 failed to learn a successful policy in all trials on the decentralized Ant-v2 task.These agents received the same observations as MATD3 and MADDPG but did not use a centralized critic.
- Results: Single-agent TD3 with full state information eventually outperforms all multi-agent approaches, but only after many time-steps.The single-agent setting is described as significantly less difficult than the decentralized task.
7 Related Work
Related work includes methods that improve MADDPG mainly in adversarial tasks and robotics methods using independent critics or value iteration. The paper positions its approach as addressing cooperative tasks and higher-dimensional robotic settings.
- Multi-Agent Reinforcement Learning: M3DDPG improves MADDPG through adversarial action perturbations, but its reported improvements are limited to adversarial tasks.The paper notes that M3DDPG could potentially be combined with MATD3.
- Decentralized Robotic Control: A robotics approach uses independent critics augmented with other agents’ observations and value iteration, but does not scale to high-dimensional tasks.That work studies comparatively simple tasks rather than the high-dimensional robotic setting considered here.
8 Conclusion and Future Work
The paper concludes that overestimation bias occurs in multi-agent domains and motivates MATD3, which uses double centralized critics with decentralized execution. It also demonstrates decentralized Ant control and identifies a future hybrid with TD3.
- Conclusion: Overestimation occurs in multi-agent domains and significantly hinders convergence.This finding motivates the proposed multi-agent TD3 method.
- Conclusion: MATD3 combines centralized training with decentralized execution and addresses overestimation bias using double centralized critics.The method significantly outperforms MADDPG on most particle-domain tasks.
- Conclusion: The proposed robotic-system formulation learns decentralized policies for the Ant task and outperforms MADDPG in that domain.The robot is treated as a multi-agent system so MARL methods can learn decentralized control.
- Future Work: Future work will investigate a hybrid approach combining MATD3’s initial benefits with TD3’s later performance.The proposed direction targets the complementary learning behavior observed for the two methods.