PaperlayerView exact revision on arXiv

Source-linked AI summary

Arena: A General Evaluation Platform and Building Toolkit for Multi-Agent Intelligence

Yuhang Song, Andrzej Wojcicki, Thomas Lukasiewicz, Jianyi Wang, Abi Aryan, Zhenghua Xu, Mai Xu, Zihan Ding, Lianlong Wu

arXiv:1905.08085v5cs.MAcs.AI

TL;DR

Multi-agent intelligence lacks a general evaluation platform: existing platforms are either not compatible with multi-agent settings or focus on specific games. Arena combines 35 diverse games with a toolkit for building new multi-agent problems, baseline implementations, and benchmark agents and teams. These resources support population-performance evaluation and comparisons under a stable, uniform standard.

  • Problem

    Existing evaluation platforms are either not compatible with multi-agent settings or are built for specific games, leaving no general platform for multi-agent intelligence.

    Source: Introduction · p. 1

  • Method

    Arena combines 35 diverse multi-agent games with a GUI-configurable social tree, five basic reward schemes, deep multi-agent reinforcement learning baselines, and benchmark agents and teams.

    Source: Introduction · pp. 1–2

  • Results

    Arena provides 100 trained agents and teams as a base for population-performance evaluation and stable, uniform comparisons.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    Researchers can use the toolkit and game set to customize games around new social paradigms and study unexplored multi-agent problems.

    Source: Introduction · pp. 1–2

Abstract

from arXiv · show

Learning agents that are not only capable of taking tests, but also innovating is becoming a hot topic in AI. One of the most promising paths towards this vision is multi-agent learning, where agents act as the environment for each other, and improving each agent means proposing new problems for others. However, existing evaluation platforms are either not compatible with multi-agent settings, or limited to a specific game. That is, there is not yet a general evaluation platform for research on multi-agent intelligence. To this end, we introduce Arena, a general evaluation platform for multi-agent intelligence with 35 games of diverse logics and representations. Furthermore, multi-agent intelligence is still at the stage where many problems remain unexplored. Therefore, we provide a building toolkit for researchers to easily invent and build novel multi-agent problems from the provided game set based on a GUI-configurable social tree and five basic multi-agent reward schemes. Finally, we provide Python implementations of five state-of-the-art deep multi-agent reinforcement learning baselines. Along with the baseline implementations, we release a set of 100 best agents/teams that we can train with different training schemes for each game, as the base for evaluating agents with population performance. As such, the research community can perform comparisons under a stable and uniform standard. All the implementations and accompanied tutorials have been open-sourced for the community at https://sites.google.com/view/arena-unity/.

Introduction

Arena addresses the lack of a general evaluation platform for multi-agent intelligence with 35 diverse games, while also providing tools for creating new multi-agent problems. It additionally offers baseline agents and teams to support stable, uniform evaluation.

Introduction

  • The paper motivates multi-agent learning as a route to innovation because agents act as environments for one another, with improvement proposing new problems for others.The authors contrast this with learning agents that excel at tests but are less innovative.Source: Introduction · p. 1
  • Arena provides 35 multi-agent games with diverse logics and representations, addressing the absence of a general evaluation platform for multi-agent intelligence.Existing platforms either do not support multi-agent settings or are built for specific games.Source: Introduction · p. 1
  • The toolkit can extend basic agent behavior into free-for-all, team-based, or hierarchical scenarios with customized collaboration and competition.The examples retain basic movement and an alive state while changing the social arrangements.Source: Introduction · p. 1
  • A GUI-configurable social tree and five basic multi-agent reward schemes support the creation of new social paradigms and multi-agent problems.Ready-to-use reward functions and verification options support configuring rewards within the schemes.Source: Introduction · pp. 1–2
  • Arena provides Python implementations of state-of-the-art deep multi-agent reinforcement learning baselines and 100 trained agents or teams for population-performance evaluation.The agents and teams are trained with different schemes for each game, supporting comparisons under a stable, uniform standard.Source: Introduction · p. 2

The Platform

Arena combines a Unity-based platform for diverse multi-agent games with tools for building new scenarios. Its game set and configurable social structures support varied multi-agent research.

The Platform

  • Built on Unity, Arena offers realistic rendering and physics, a visual editor for customizing scenarios, and access to a large development community and off-the-shelf assets.The authors say realistic physics introduces stochasticity and can make transferring simulator-learned policies to the real world easier.Source: The Platform · pp. 2–3

Game Sets Towards General Intelligence

  • Arena provides 27 new games, 8 games adapted from prior research, and an interface to StarCraft.The 8 adapted games include realistic rendering, a physics engine, and extensibility to other social paradigms.Source: The Platform · p. 3

Building Toolkit for Multi-Agent Environments

  • The building toolkit combines a GUI-configurable social tree for grouping agents with five basic multi-agent reward schemes applied at each node.It is intended to make social relationships easier to build and verify while handling low-level intelligence such as motor skills.Source: The Platform · p. 3

Other Features

  • Arena also provides broadcast boards at social-tree nodes, optional global-state broadcasts, a default top-down view, and an interface for human players.The human interface allows a person to take the place of any agent in a game.Source: The Platform · p. 3

Social Trees

Arena uses reward schemes to specify social relationships among agents and organizes them in reconfigurable trees. Applying schemes at different levels yields structured problems, from individual skills to team collaboration and competition.

Basic Multi-Agent Reward Schemes

  • Five basic multi-agent reward schemes (BMaRSs) are sets of joint reward functions restricted to produce similar effects on the population.The paper presents the general form as extending reward schemes previously tabulated for a special case of Pong.Source: Social Trees · p. 4
  • Non-learnable schemes (FNL) make an agent’s episode reward unoptimizable through improving its own policy.Isolated schemes (FIS) make each agent’s episode reward independent of the other agents’ policies.Source: Social Trees · p. 4
  • FIS is useful for learning dense internal rewards for robot motor skills before population-level rewards, with examples including energy cost, force penalties, steady velocity, and distance to a target.These rewards address settings where sparse population-level rewards are insufficient to direct basic continuous-control learning.Source: Social Trees · p. 4
  • Competitive schemes (FCP) keep the sum of agents’ episode rewards unchanged when any one agent changes policy, including zero-sum games as a one-step special case.Examples include competing for exhausted resources or rewards based on the order of death.Source: Social Trees · p. 4
  • Collaborative schemes (FCL) have no conflicting interests between agents and include at least one pair sharing an interest; mixed schemes (FCC) cover other cases.Team living time is an FCL example, while FCC can include competitive moments even when total interest can be maximized with particular policies.Source: Social Trees · p. 4

The Social Tree

  • A social tree groups agents and teams, applying reward schemes at each node; the example combines agent-level motion, team-level box-pushing, and global-level racing rewards.The final reward for each agent is a weighted sum of schemes at the three levels, and the GUI permits nodes to be dragged, duplicated, or deleted.Source: Social Trees · p. 5

The Learning Agents

Arena supplies decentralized, self-play, population-based, and centralized-critic baselines, alongside a population-based evaluation metric. The released base population supports rankings for individual agents or groups and includes human rankings as an indication of human-level intelligence.

The Baselines

  • Five baselines cover decentralized PPO, self-play, population-based training, centralized critics, and centralized critics with a counterfactual baseline.The collaborative baselines are CC and CF; SP and PB are based on self-play and population-based training, respectively.Source: The Learning Agents · p. 5

The Evaluation Metric

  • Population performance ranks an agent or group against a base population because evaluation against a single agent or hand-coded bot can be unstable and misleading.Arena releases 100 best agents, trained with different schemes for each game, as the base population.Source: The Learning Agents · p. 5
  • Arena also provides human rankings among the base population as an indication of human-level intelligence in each game.Source: The Learning Agents · p. 5

Experiments

The experiments show that population-performance curves make baseline training progress more analyzable than noisy episode rewards, with clear performance gaps. A Crossroads case study shows different learned strategies under isolated, collaborative, and team-based reward schemes.

Case Study of Social Tree and BMaRSs

  • In Crossroads, isolated FIS rewards produce agents that rush and crash, while shared FCL rewards lead agents to wait for one another.With FCL within teams and FCP between teams, one agent blocks the opposing team until its teammates reach the target, then heads for the target.Source: Experiments · pp. 5–6, Figure 5 · p. 6

Baselines and Evaluation Metric

  • The five baselines are compared on Crossroads and PushBox using settings with collaborative teams competing against one another.SP and PB are investigated at the team level; CC and CF are investigated among collaborating agents within a team.Source: Experiments · p. 6
  • Population-performance curves show clear gaps between methods, while episode-reward curves are extremely noisy in the non-stationary environments.The comparison covers Crossroads and PushBox; SP and PB are evaluated between teams, while CC and CF are evaluated among agents within a team.Source: Experiments · p. 6, Figure 4 · p. 6

Related Work

Related work covers collaborative and competitive multi-agent learning, including centralized training with decentralized execution, self-play, and population-based training. The paper situates Arena against general reinforcement-learning platforms and existing multi-agent platforms.

Multi-Agent Intelligence

  • Prior work surveys multi-agent intelligence and explores both competitive and collaborative settings.Source: Related Work · p. 6

Collaborative Settings

  • Collaborative work ranges from independent learning to centralized systems and centralized training with decentralized execution, alongside joint value functions and communication protocols.The section notes that fully centralized systems can require unavailable global state and may not support extending the number of agents.Source: Related Work · p. 6

Competitive Settings

  • Competitive deep multi-agent learning includes self-play and population-based training, while prior work also addresses agent diversity and opponent-policy overfitting.The passage cites self-play in Leduc Poker and Dota2, and population-based training in StarCraft.Source: Related Work · pp. 6–7

Summary and Outlook

Arena combines a general evaluation platform with a toolkit for building new multi-agent problems, baseline implementations, and a base population for comparisons under a stable, uniform metric.

Summary and Outlook

  • Arena introduces a general multi-agent intelligence platform, a toolkit for building environments, baseline implementations, and a base population for stable, uniform evaluation.Source: Summary and Outlook · p. 7

Experiments: Realistic Rendering, Simulation Speed and Stochasticity

Arena’s stochasticity study reports the most environment branches among the compared platforms. Its rendering matches the best compared scenes, and its parallel implementation maintains ALE-like speed on similarly complex games below 32 concurrent threads.

Stochasticity

Simulation Speed

Realistic Rendering

  • Arena’s realistic rendering effect is at the same level as the best scenes in the compared works.The paper reports an objective comparison of realistic scenes across the platforms.Source: Experiments: Realistic Rendering,… · p. 8

How Verification for BMaRSs is Implemented

The section describes how BMaRS verification is operationalized through sampled tests and episode comparisons, and relates the resulting classes to familiar game types. It also catalogs games, formats, origins, and selected objectives or mechanics across a range of competitive and cooperative settings.

  • A universal statement is judged true when it holds on all N sampled cases and false when it fails on any one.N is the verification sample-count hyperparameter.Source: How Verification for BMaRSs is… · p. 8
  • For FNL and FIS, verification compares episodes under shared sampled conditions while changing the policy being tested, checking whether the corresponding best-response quantity is zero.FNL varies πx; FIS varies πx1 while retaining the other specified variables across episodes.Source: How Verification for BMaRSs is… · p. 8
  • FCP and FCL use paired episodes with shared sampled conditions and a changed πx; FCP tests a zero best-response condition, whereas FCL tests whether BRf_x ≥ 0.For FCL, the procedure samples two agents and compares their episode rewards across the paired runs.Source: How Verification for BMaRSs is… · pp. 8, 10
  • FCC comprises situations outside FNL, FIS, FCP, and FCL, so verification classifies it by ruling out the other four classes.Source: How Verification for BMaRSs is… · p. 10
  • Rock, Paper, and Scissors in normal-form game is a special case of FCP; symmetric zero-sum games are also special cases, so cyclic games are as well.Source: How Verification for BMaRSs is… · pp. 10–11
  • Snake eliminates the shorter player when the two collide, while a track also requires agents to recognize road signs at crossroads under night-rendering conditions.Source: How Verification for BMaRSs is… · p. 14
  • BattleCity-TP uses a third-person view, while BattleCity-FP uses a first-person view; Fighter rewards using bunkers and destroying the opponent.Source: How Verification for BMaRSs is… · pp. 14, 16
  • The RAM-based tasks include destroying a pyramid, transporting bricks to a target, reaching a target first, and collaborating to lift a ball.Source: How Verification for BMaRSs is… · p. 15
  • TowerDefense is discrete, and another RAM-based task has agents control a robot arm to push a ball to the opponent’s base.Source: How Verification for BMaRSs is… · pp. 15–16
  • MiniPUBG-Forest’s grassland map calls for hiding and attacking, while FightForFood ends when food is exhausted and Runner ends at a specified score threshold.Source: How Verification for BMaRSs is… · p. 16

Related papers

Loading 1905.08085v5…