Source-linked AI summary
ROMA: Multi-Agent Reinforcement Learning with Emergent Roles
Tonghan Wang, Heng Dong, Victor Lesser, Chongjie Zhang
TL;DR
Complex cooperative tasks require MARL to balance adaptive learning with efficient specialization, whereas role-based methods commonly predefine roles using domain knowledge. ROMA learns stochastic, emergent roles that condition individual policies and promote identifiable sub-task specialization. On StarCraft II micromanagement, the learned roles are dynamic, versatile, identifiable, and specialized, and the method significantly advances MARL state of the art.
Problem
Existing role-based methods predefine role structures and behaviors using domain knowledge, while MARL is flexible and adaptive but less efficient on complex tasks.
Method
ROMA conditions decentralized policies on stochastic roles inferred from local observations and uses regularizers to make roles identifiable and specialized in sub-tasks.
Results
ROMA learns dynamic, identifiable, versatile, and specialized roles and significantly advances MARL state of the art on StarCraft II micromanagement.
Takeaways & Limitations
Emergent roles provide adaptive policy sharing among agents with similar responsibilities and a mechanism for role-driven sub-task specialization.
Takeaways & Limitations
Existing role-based frameworks depend on prior domain knowledge to predefine responsibilities, which prevents dynamic adaptation to uncertain environments.
Abstract
from arXiv · showhide
The role concept provides a useful tool to design and understand complex multi-agent systems, which allows agents with a similar role to share similar behaviors. However, existing role-based methods use prior domain knowledge and predefine role structures and behaviors. In contrast, multi-agent reinforcement learning (MARL) provides flexibility and adaptability, but less efficiency in complex tasks. In this paper, we synergize these two paradigms and propose a role-oriented MARL framework (ROMA). In this framework, roles are emergent, and agents with similar roles tend to share their learning and to be specialized on certain sub-tasks. To this end, we construct a stochastic role embedding space by introducing two novel regularizers and conditioning individual policies on roles. Experiments show that our method can learn specialized, dynamic, and identifiable roles, which help our method push forward the state of the art on the StarCraft II micromanagement benchmark. Demonstrative videos are available at https://sites.google.com/view/romarl/.
1. Introduction
Cooperative MARL must balance scalable sharing with specialization for complex tasks, while existing role-based systems rely on predefined domain knowledge. ROMA introduces emergent roles that adaptively organize policy sharing and sub-task specialization, improving performance on StarCraft II micromanagement.
- Motivation: Shared decentralized policies improve scalability but struggle to represent many skills in complex multi-agent tasks.Using distinct policies for every agent is unnecessarily complex when agents intermittently perform similar sub-tasks.
- Motivation: Role-based systems decompose tasks by assigning responsibilities to roles, enabling agents with similar roles to share behaviors and experience.These roles are associated with specialized task patterns and can reduce policy-search complexity.
- Research gap: Existing role-based methods rely on prior domain knowledge to predefine role structures and responsibilities, limiting adaptation to uncertain environments.Their predefined decomposition prevents roles from being dynamic and adaptive.
- ROMA: ROMA conditions decentralized policies on stochastic roles inferred from local observations and uses regularizers to make roles identifiable and specialized.The framework connects similar responsibilities with similar policies while encouraging specialization in sub-tasks.
- Results: On StarCraft II micromanagement, ROMA significantly advances MARL state of the art through adaptive policy sharing among agents with similar roles.Visualizations show roles adapting in homogeneous and heterogeneous teams and connecting sub-task specialization with team efficiency.
2. Background
Cooperative MARL is commonly formulated in partially observable settings and trained with centralized training and decentralized execution. CTDE methods often decompose global value estimation into local utilities, but typically retain shared local networks for scalability.
- Dec-POMDP formulation: A cooperative Dec-POMDP models agents selecting individual actions from partial observations, producing joint actions and a shared reward.Each agent acts from its observation history while the environment transitions according to the joint action.
- CTDE: CTDE addresses non-stationarity during training while preserving decentralized policies for execution.Centralized training can use information unavailable to agents at execution time.
- CTDE: Value decomposition methods learn decentralized utility functions and combine them with a mixing network into a global action value.This provides a scalable route to centralized value-based learning with decentralized agent decisions.
- CTDE: Existing CTDE methods typically share local value or policy networks across agents to achieve learning scalability.The background identifies this shared-network mechanism as insufficient for complex cooperative tasks.
3. Method
ROMA introduces emergent roles into CTDE-based MARL by conditioning individual policies on stochastic role embeddings. Two regularizers promote identifiable and specialized roles, while value decomposition trains the framework end to end.
- Framework: ROMA uses CTDE with local utilities combined by a mixing network during centralized training, while execution uses decentralized role-conditioned policies.The mixing network estimates the global action value and is removed during execution.
- Role Representation: Each agent samples a role from a Gaussian distribution whose parameters are generated by a role encoder conditioned on local observations.A role decoder then uses the sampled role to generate individual policy parameters.
- Identifiable Roles: The identifiable-role objective links role embeddings to agents’ long-term behaviors through mutual information and a variational posterior estimator.A GRU trajectory encoder provides the variational estimator used to construct a tractable objective.
- Dynamic Roles: Role representations adapt within episodes to changing demands, supporting offensive formation, protection of injured or dying agents, and alternate fire.Figure 3 reports these behaviors at t=1, t=8, and t=19.
- Specialized Roles: The specialization regularizer encourages agents with similar behaviors to share roles while preserving distinct roles for dissimilar behaviors.A trainable dissimilarity model makes dissimilarity high when mutual information is low, yielding a compact but diverse role set.
- Overall Optimization Objective: The full objective combines the TD loss with identifiable-role and specialization regularizers, weighted by λI and λD.The framework is optimized jointly with gradients from the standard reinforcement-learning TD loss.
4. Related Works
Prior work studies roles in natural and artificial multi-agent systems and develops deep MARL through policy-gradient and value-based methods. ROMA differs from MAVEN by using mutual information for emergent roles rather than committed exploration.
- Role-Based Systems: Role-based multi-agent systems use division of labor so agents with the same role specialize in sub-tasks and reduce design complexity.The role concept is also documented in natural systems such as bees, ants, and humans.
- Deep MARL: Deep MARL includes policy-gradient methods such as COMA, MADDPG, PR2, and MAAC, alongside value-function factorization methods such as VDN, QMIX, and QTRAN.These methods form major lines of deep multi-agent reinforcement-learning research.
- Emergent Behaviors: Emergent fairness and tool usage extend deep-learning perspectives on natural and artificial multi-agent systems.These studies address additional forms of emergent multi-agent behavior.
- Relation to MAVEN: ROMA and MAVEN both optimize mutual information, but ROMA learns diverse and identifiable roles whereas MAVEN targets committed exploration.Their different high-level purposes lead to technical distinctions.
5. Experiments
Experiments evaluate ROMA’s adaptive role learning, sub-task specialization, and performance on homogeneous and heterogeneous StarCraft II teams. Results show dynamic, identifiable roles, improved performance over alternatives and ablations, and stronger advantages on larger populations.
- Experimental Setup: ROMA is evaluated on SMAC maps spanning easy, hard, and super-hard difficulty, including homogeneous and heterogeneous teams.The experiments use five random seeds and report 95% confidence intervals; baselines and role-learning ablations are included.
- Performance on StarCraft II: ROMA substantially outperforms alternative approaches on both homogeneous and heterogeneous maps.The evaluation covers maps with different difficulty levels and uses five random seeds with 95% confidence intervals.
- Ablations: The specialization loss LD contributes more to performance improvements than LI, while LI can make training more stable.Optimizing LI alone only slightly improves performance, whereas omitting both role-learning losses can prevent winning on MMM2.
- Ablations: ROMA’s dynamic sharing links similar responsibilities to similar policies without relying on a larger parameter count.Comparisons with QMIX, QMIX-NPS, and QMIX-LAR support the roles’ contribution to team performance and adaptive learning sharing.
- Performance on StarCraft II: The performance gap between ROMA and ablations is more significant on maps with more than ten agents.This supports the observed benefit of emergent roles for labor efficiency in larger populations.
- Dynamic Roles: Learned roles correspond to automatically discovered sub-tasks and evolve with positions, health, unit types, and training progress.Examples include sacrifice roles for Zealots, distinct Medivac roles, and differentiated Marine and Marauder responsibilities; different unit types have dissimilarity near 0.96 versus around 0.08 for the same type.
- Dynamic Roles: Overall, ROMA learns dynamic, identifiable, versatile, and specialized roles that effectively decompose the task.The experiments associate these emergent roles with improved multi-agent reinforcement learning performance.
6. Closing Remarks
ROMA introduces emergent roles into deep multi-agent reinforcement learning, encouraging specialization on automatically detected sub-tasks and offering a perspective on cooperation connected to division of labor. The paper identifies learning roles through deep reinforcement learning as an initial step toward understanding more complex role emergence and hierarchical interactions.
- ROMA captures emergent roles and encourages agents to specialize on automatically detected sub-tasks.
- The framework offers a perspective for explaining and promoting cooperation within agent teams through an implicit connection to division of labor.
- The paper describes learning roles via deep reinforcement learning as a first attempt, leaving more complex role emergence, division of labor, and hierarchical interactions for future work.
A. Mathematical Derivation
The derivation develops mutual-information-based objectives for identifiable and specialized roles, then converts them into tractable bounds and replay-buffer optimization procedures. It also formalizes how observation-conditioned role distributions support dynamic roles and how trajectory encoders support role differentiation.
- Identifiable Roles: The identifiable-role objective maximizes conditional mutual information between roles and local observation-action histories given current observations.
- Identifiable Roles: A posterior estimator and variational lower bound make the mutual-information objective tractable, with non-negative KL divergence enabling the bound.
- Dynamic Roles: Conditioning the role encoder on local observations makes role distributions independent of local histories given those observations, supporting dynamic roles.
- Specialized Roles: The specialized-role objective is optimized through an upper bound using a replay buffer containing joint trajectories, joint observations, and role vectors.
- Specialized Roles: Role differentiation adds a mutual-information maximization objective because identifiability and observation conditioning alone do not explicitly encourage specialization.
- Derivation Conditions: Role-distribution variances are clipped at 0.1 to keep their entropy non-negative for the derivation’s inequality.
B. Architecture, Hyperparameters, and Infrastructure
ROMA builds on QMIX with per-agent recurrent local utilities and role-conditioned components, using lightweight encoders for stochastic role embeddings and trajectories. Experiments use specified optimization, exploration, replay, parallel-environment, and hardware settings, while role evolution is illustrated on benchmark maps.
- Architecture: ROMA uses QMIX’s mixing architecture, with each agent’s local utility approximated by fully connected, 64-unit GRU, and fully connected layers.
- Role Embedding: Role learning uses simple fully connected encoders, including a 12-dimensional ReLU role encoder that generates Gaussian role means and variances.
- Role Dynamics: Role emergence and evolution are illustrated on the 10m versus 11m map.
- Hyperparameters: Training uses λI = 10^-4, λD = 10^-2, γ = 0.99, RMSprop at 5 × 10^-4, ε-greedy exploration, eight parallel environments, and batches of 32 episodes.
- Infrastructure: Experiments are conducted on an NVIDIA GTX 2080 Ti GPU.
B.2. Baselines and Ablations
ROMA is evaluated against established baselines and ablations, using author-provided implementations for several comparison methods and a parameter-sharing ablation to isolate the effect of shared learning.
- Comparisons: ROMA is compared with baselines and ablations listed in the paper’s Table 1.
- Baselines: COMA, QMIX, and MAVEN use author-provided code with hyperparameters fine-tuned on the SMAC benchmark.
- Ablations: QMIX-NPS retains QMIX’s architecture but removes parameter sharing among agents.
C. Additional Experimental Results
ROMA is evaluated on StarCraft II micromanagement maps with symmetric and asymmetric teams, and its advantage over baselines is especially pronounced as the number of agents increases.
- ROMA is benchmarked on StarCraft II unit micromanagement tasks spanning symmetric and asymmetric teams.The evaluated maps include 6z4b, 6s4z vs 10b30z, and 10z5b vs 2z3s.
- ROMA’s performance advantage is more significant on maps with more agents, including 10z5b vs 2z3s, MMM2, 27 vs 30m, and 10m vs 11m.
- QMIX combines decentralized local utilities into a global action value using a mixing network whose weights and biases are generated by a state-conditioned hyper-network.
C.2. Role Embedding Representations
ROMA learns role embeddings that correspond to specialized sub-tasks and produce differentiated behaviors, including coordinated movement and staged attacks.
- ROMA learns roles that are closely related to sub-tasks in the winning strategy.
- For 27m vs 30m, roles aggregate by agents’ relative positions and produce different movement strategies that form an offensive arc without collisions.
- For 6z4b, role clusters correspond to sub-tasks in which some units exploit Baneling splash damage while others wait before attacking.
- These results indicate that ROMA automatically decomposes tasks and learns versatile roles specialized in particular sub-tasks.
C.3. Additional Results for Role Evolution
ROMA’s roles emerge and evolve during training in homogeneous teams, adapting to agents’ relative positions as they form attack formations.
- On 10m vs 11m, ROMA gradually allocates roles according to agents’ relative positions during training.
- Role differentiation and corresponding individual-policy differences help agents form the offensive arc more efficiently.
- The 10m vs 11m environment contains 10 allied Marines facing 11 enemy Marines.
- The broader MARL literature includes factorized MDPs, coordination graphs, explicit communication, and centralized approaches for cooperative control.