Source-linked AI summary
Multiagent Cooperation and Competition with Deep Reinforcement Learning
Ardi Tampuu, Tambet Matiisen, Dorian Kodelja, Ilya Kuzovkin, Kristjan Korjus, Juhan Aru, Jaan Aru, Raul Vicente
TL;DR
The paper addresses how decentralized agents can learn and interact in complex multiagent environments. It extends Deep Q-Networks to two independent agents playing Pong under different reward schemes, finding distinct competitive and collaborative strategies. The results support using Deep Q-Networks to study decentralized learning in complex multiagent systems.
Problem
Multiagent reinforcement learning is less understood because joint actions make state transitions, rewards, and action values depend on other learning agents.
Method
The authors control two Pong agents with independent Deep Q-Networks receiving raw screen images and reward signals, while varying the rewarding scheme.
Results
Changing rewards produces competitive scoring behavior, collaborative ball-keeping strategies, and a progression between competitive and collaborative modes.
Takeaways & Limitations
Deep Q-Networks can serve as a practical tool for studying decentralized learning in complex multiagent systems.
Takeaways & Limitations
The study uses the simplest independent-network adaptation, and the authors suggest broader multiagent reinforcement-learning adaptations could improve results.
Abstract
from arXiv · showhide
Multiagent systems appear in most social, economical, and political situations. In the present work we extend the Deep Q-Learning Network architecture proposed by Google DeepMind to multiagent environments and investigate how two agents controlled by independent Deep Q-Networks interact in the classic videogame Pong. By manipulating the classical rewarding scheme of Pong we demonstrate how competitive and collaborative behaviors emerge. Competitive agents learn to play and score efficiently. Agents trained under collaborative rewarding schemes find an optimal strategy to keep the ball in the game as long as possible. We also describe the progression from competitive to collaborative behavior. The present work demonstrates that Deep Q-Networks can become a practical tool for studying the decentralized learning of multiagent systems living in highly complex environments.
Introduction
The paper extends Deep Q-Networks to decentralized multiagent learning in complex environments. In Pong, changing reward schemes produces competitive, collaborative, and intermediate behaviors.
- Reinforcement learning agents adapt behavior through trial-and-error by maximizing rewards collected during environmental interactions.This can support complex long-term strategies in novel or changing environments.
- DeepMind’s results suggest one reinforcement-learning algorithm can generalize across diverse, high-dimensional video-game environments.The cited work applied reinforcement learning using raw game input and reward signals.
- The paper studies how multiple autonomous Deep Q-Networks interact in complex environments using raw screen images and reward signals.Two agents are trained under different rewarding schemes.
- Changing Pong’s reward scheme produces competitive or cooperative behavior, with competitive agents scoring and collaborative agents prolonging play.The study also examines intermediate states between competitive and collaborative modes.
1.1 The Deep Q-Learning Algorithm
Multiagent reinforcement learning is difficult because agents jointly affect state transitions and rewards, creating moving targets and stability challenges. The paper therefore uses one independent Deep Q-Network per agent as a simple decentralized approach.
- Q-learning estimates the quality of actions in particular environmental states to learn a policy maximizing accumulated long-term reward.The challenge increases when agents lack explicit information about environmental dynamics or rewards.
- Deep Q-Networks use convolutional neural networks to approximate Q-value functions from raw sensory input in complex environments.DeepMind reported superhuman performance in several Atari games using this approach.
- Joint actions make each agent’s value depend on other learning agents, producing an ever-moving target and a stability–adaptivity trade-off.This makes multiagent reinforcement learning less understood than single-agent learning.
- Each agent is controlled by an independent Deep Q-Network, using the environment as the sole source of interaction.The authors choose this simple method for its decentralized nature, computational speed, and consistent reported results.
1.2 Adaptation of the Code for the Multiplayer Paradigm
The authors modify DeepMind’s single-player implementation so two independent agents can act and receive separate rewards in multiplayer Pong.
- The original code supported only one player because its agent–emulator communication protocol restricted multiplayer interaction.
- The modification transmits actions from two agents and receives two corresponding reward signals from the environment.
- The modified game screen is used as the shared environmental input for the multiplayer setup.
1.3 Game Selection
Pong was selected as a suitable two-player environment for studying competition and cooperation with Deep Q-Networks. Each agent controls one paddle, while the game’s simple structure supports distinct reward-driven interactions.
- Game Selection: The game-selection criteria required real-time two-player play rather than alternating turns.Alternating games such as Breakout were considered less suitable for multiagent experiments.
- Game Selection: The selected game had to be learnable above human level in single-player mode with Deep Q-learning.Games requiring navigation beyond the algorithm’s demonstrated capability were excluded.
- Game Selection: The environment needed a natural competitive mode and a simple reward change capable of switching between cooperation and competition.
- Game Selection: Pong satisfied the criteria, had existing code support, and could be learned relatively quickly.Its simplicity also made the environment easy to understand.
- Pong Environment: Each of the two agents controls one paddle and independently selects among moving up, moving down, standing still, and firing.
- Scope: Other proposed environments, including Outlaw and Warlords, were outside the scope of the present work.They were suggested for future questions about prisoner’s dilemma and collaboration against adversaries.
1.4 Rewarding Schemes
The study creates different Pong games by changing the rewards assigned when the ball goes out of play. These schemes induce competitive, cooperative, or intermediate behavior.
- Rewarding schemes are adjusted by changing the reward both players receive when the ball leaves play.This creates several distinct games within the same Pong environment.
- Score More than the Opponent (Fully Competitive): The classical zero-sum Pong scheme gives an agent a reward when the ball passes its opponent and a punishment when it misses.
- Loosing the Ball Penalizes Both Players (Fully Cooperative): The cooperative scheme penalizes both players whenever the ball goes out, regardless of which player missed, and gives no positive rewards.The objective is to keep the ball in play as long as possible.
- A further collaborative scheme that rewards both players when the ball goes out was expected not to produce interesting behavior.
- Transition Between Cooperation and Competition: The transition scheme varies ρ from −1 to 1 to explore behavior between competition and cooperation.
1.5 Training Procedure
Agents are trained for repeated epochs while acting only every fourth frame and gradually reducing random exploration. Training progress is monitored through maximal Q-values on held-out game states, with network snapshots retained after each epoch.
- The exploration rate decreases from 1.0 to 0.05 over million time steps and then remains fixed.Random actions are drawn from all possible game commands instead of using the network prediction.
- Training convergence is monitored by recording maximal Q-values for 500 randomly selected game states held out before training.These values estimate how highly each network rates its best action and the quality of each state.
- Snapshots of both Deep Q-Networks are stored after every epoch for later study of the training process.
1.6 Collecting the Game Statistics
The study quantifies Pong behavior by detecting game events and evaluating trained networks after each epoch. Its measures capture exchanges, wall use, and the time needed to restart play.
- Specific Pong events, including ball bounces, are counted using Stella’s integrated debugger and its game-memory locations.The debugger identifies exact memory locations for bounce counts and other events.
- After each training epoch, the current networks play 10 games with different random seeds while exploration is set to 0.01.Measurements are collected for every rewarding scheme to evaluate cooperative and competitive behavior.
- Paddle-bounces per point counts ball exchanges between the two players before one scores.Random agents almost never hit the ball, whereas well-trained agents hit it multiple times in an exchange.
- Wall-bounces per paddle-bounce measures how often the ball hits the top and bottom walls before reaching the other player.The measure distinguishes direct returns from strategies using multiple wall bounces.
- Serving time per point measures the frames required to restart play after the ball is lost.The player who scored must issue the fire command, and rewards may influence whether agents avoid restarting.
- Training is limited to 50 epochs because the predicted Q-values have stabilized.
- A Pong game contains multiple exchanges and ends when one agent reaches 21 points.
2 Results
Independent Deep Q-Network agents develop distinct competitive and collaborative strategies in Pong under different rewarding schemes. Competitive training improves play and rapid serving, whereas collaborative training favors prolonged exchanges, delayed serving, and reduced wall-bounces.
- Emergence of Competitive Agents: Competitive agents become increasingly proficient, eventually playing reasonably well, winning regardless of the server, and sustaining considerable exchanges.Training begins with failures to hit the ball; paddle-bounces increase as both agents improve.
- Emergence of Competitive Agents: Competitive Q-values track game situations and ball speed, dropping when the ball approaches a player or travels faster.Agents predict lower reward when a player may miss, while faster movement produces larger Q-value changes.
- Emergence of Competitive Agents: Competitive agents learn to relaunch the ball rapidly because serving immediately increases their reward expectation.Serving time decreases abruptly during early training.
- Emergence of Collaborative Agents: Collaborative agents are rewarded for avoiding losses, so they learn to keep the ball alive, delay serving, and pass it horizontally without wall contacts.After training, paddle-bounces increase abruptly while wall-bounces decrease gradually.
- Emergence of Collaborative Agents: Collaborative agents’ predicted rewards decrease when the ball moves faster because faster balls increase the risk of mistakes.Their Q-values are much more negative for faster-ball situations than for slower ones.
- Progression from Competition to Collaboration: As the positive reward ρ decreases, behavior shifts toward collaboration: ρ ≤ 0 produces delayed relaunches, longer-lived balls, and fewer wall-bounces.For ρ = 1, ρ = 0.75, and ρ = 0.5, paddle-touches per point remain unchanged, while lowering ρ from 0.5 to 0.25 significantly reduces touches.
3 Discussion
The study shows that autonomous Deep Q-Network agents can learn distinct cooperative and competitive strategies from raw sensory data in Pong. It also identifies Q-value overestimation, limited behavioral metrics, and the two-agent setting as important boundaries.
- Autonomous Deep Q-Networks learned a two-player Pong game from raw sensory data, supporting their use in decentralized multiagent learning.
- Under cooperative rewarding, Pong agents discovered coordinated parallel ball-hitting that kept the ball bouncing between them for a long time.
- In fully competitive play, both networks often predicted rewards near 0.5 although equally skilled agents should expect values around zero.
- More accurate Q-value estimation might produce clear collaboration with higher ρ values and earlier during training.
- The study used the simplest multiagent adaptation, assigning one autonomous Deep Q-Network to control each agent.
- Behavioral characterization was limited to statistics extractable from game memory, excluding measures requiring frame-by-frame screen analysis.
- The system considered two agents in Pong; interaction among tens or hundreds of agents was not feasible with the algorithms used.
4 Supplementary Materials
The supplementary materials document the evolution of cooperative and competitive agents during training and provide code, visualization tools, and result videos.
- Figure 8 tracks Q-value and reward evolution for cooperative and competitive agents over training time.
- The multiplayer-adapted code and visualization tools are available in the authors’ GitHub repository.
- Videos illustrating the study’s core concepts and results are provided through a dedicated YouTube playlist.