Source-linked AI summary
SIGMA: Structured Noise-Effect-Aware Grouped Multi-Agent Aggregation
Li Mingqian
TL;DR
Cooperative MARL can suffer structured, cooperation-dependent effects from noisy observations, while existing robust methods rarely exploit this structure explicitly. SIGMA uses adaptive grouping with intra-group consensus and inter-group attention to learn robust cooperative representations. Experiments in StarCraft II show stronger robustness under increasing observation noise and smaller degradation than evaluated baselines.
Problem
Noisy observations can disrupt cooperative decision-making through locally correlated and globally heterogeneous effects, but existing robust MARL methods rarely explicitly characterize or exploit these structure-dependent effects.
Method
SIGMA adaptively identifies local cooperation structures, aggregates representations through intra-group consensus, and coordinates groups using inter-group attention.
Results
SIGMA maintains strong cooperative performance under increasing observation noise and exhibits smaller performance degradation than evaluated baselines in StarCraft II Multi-Agent Challenge scenarios.
Takeaways & Limitations
The results suggest that explicitly exploiting cooperation structures is a promising approach to improving cooperative MARL robustness under noisy observations.
Takeaways & Limitations
The evaluation relies on individual training runs and a limited set of SMAC scenarios, so its robustness evidence is preliminary pending multi-seed and broader-task comparisons.
Abstract
from arXiv · showhide
Cooperative multi-agent reinforcement learning (MARL) faces significant challenges in maintaining robust coordination under noisy observations. Although observation disturbances are often introduced independently across agents, their downstream effects on cooperative decision-making can become structured through underlying cooperation structures. We characterize this phenomenon as structured noise effects, where noise-induced decision effects exhibit local correlation among agents with stronger task-related dependencies while remaining globally heterogeneous across different agents and local structures. Existing robust MARL methods, however, rarely explicitly characterize or exploit such structure-dependent noise effects. To address this limitation, we propose SIGMA, a hierarchical collaboration framework that exploits cooperation structures to learn robust representations under noisy observations. SIGMA first organizes agents into adaptive local structures through density-based grouping and performs intra-group consensus aggregation to preserve shared task-relevant information while smoothing agent-specific representation deviations. Inter-group attention then adaptively integrates information across different groups to preserve global coordination while accommodating their heterogeneous contributions. Experiments on noisy-observation tasks in StarCraft II empirically validate the structured noise effects and demonstrate that SIGMA consistently improves robustness under observation noise while maintaining competitive performance in noise-free environments.
Introduction
Noisy observations can disrupt cooperative decision-making beyond directly affected agents, with impacts structured by cooperation dependencies. SIGMA addresses this through adaptive grouping, intra-group consensus, and inter-group attention to improve robustness while preserving coordination.
- Reliable-observation assumptions often fail in real-world MARL because sensing errors, communication limits, and environmental disturbances are inevitable.
- Observation noise can distort local perceptions, hinder cooperative policy learning, and disrupt coordination across teammates.
- Structured noise effects combine locally correlated impacts among task-related agents with globally heterogeneous responses across agents and cooperation structures.
- Existing MARL methods model agent dependencies implicitly through value functions or centralized representations, while later approaches explicitly learn roles, groups, or graph-based relationships.
- SIGMA uses density-based adaptive grouping, intra-group consensus aggregation, and inter-group attention to learn robust representations while preserving local information and global coordination.
- Experiments on noisy-observation SMAC tasks report consistent robustness improvements while maintaining competitive performance in noise-free environments.
Methodology
SIGMA formulates noisy cooperative MARL around structured noise effects: independently introduced disturbances can produce locally correlated but globally heterogeneous decision impacts. It addresses this through adaptive grouping, intra-group consensus, and inter-group attention to learn robust cooperative representations.
- Problem formulation: The problem is modeled as a Dec-POMDP in which agents learn decentralized policies maximizing cumulative team reward under noisy observations.Observation uncertainty arises from sensing errors, communication limitations, and environmental disturbances.
- Structured noise effects: Structured noise effects distinguish independent observation disturbances from their downstream decision impacts, which need not be independent or uniform.The effects are characterized by local correlation and global heterogeneity.
- Structured noise effects: Local correlation links stronger temporal correlations in noise-induced decision impacts to stronger task-related interaction dependencies.This correlation concerns cooperation-dependent responses, not statistically correlated observation disturbances.
- Structured noise effects: Global heterogeneity reflects differing disturbance sensitivities across agents and cooperative groups because of task roles and interaction structures.Thus, structured effects combine locally correlated variations with heterogeneous impact intensities.
- SIGMA framework: SIGMA uses adaptive grouping and hierarchical representation aggregation to exploit cooperation structures under noisy observations.Intra-group consensus preserves shared information while smoothing individual deviations, and inter-group attention restores coordination across groups.
- Adaptive Grouping: SIGMA’s grouping objective favors compact, separated groups while penalizing excessive noise points and undesirable group-count deviations.The penalties discourage both overly broad grouping and excessive fragmentation.
- Hierarchical aggregation: The resulting refined representations are shared within adaptive groups and enriched with cross-group context, allowing integration into existing cooperative MARL algorithms.SIGMA adds a representation-learning module without changing the underlying policy optimization procedures.
Experiments
SIGMA is evaluated on SMAC under independent Gaussian observation noise, with win rate, training dynamics, local Q-effect correlation, and global heterogeneity used to assess robustness and structured noise effects. Across these analyses, SIGMA maintains strong noisy-observation performance, learns more stably, and its ablations show the importance of adaptive grouping and inter-group coordination, although the evaluation remains preliminary and limited in scope.
- Robustness Results: SIGMA maintains consistently high win rates as noise increases, while QMIX and HYGMA degrade more under stronger disturbances.The advantage becomes more evident when local observations are increasingly unreliable.
- Robustness Results: Under strong observation noise, SIGMA improves win rate faster and maintains more stable training performance than baseline methods.The training-curve advantage becomes increasingly evident as training progresses.
- Structured Noise Analysis: Adaptive groups exhibit stronger within-group than between-group Q-effect correlations and greater heterogeneity than matched random groups across SMAC scenarios.The local correlation gap reaches 0.232 on 5m_vs_6m and 0.082 on 8m_vs_9m, while heterogeneity gaps reach 0.251 and 0.290, respectively; positive gaps occur in approximately 92% and 93% of episodes.
- Ablation Study: Ablations show that removing adaptive grouping causes the largest degradation, while removing inter-group attention also substantially reduces performance.Removing intra-group aggregation causes a relatively smaller degradation, indicating an additional robustness benefit from consensus among related agents.
- Limitations: The evaluation is preliminary because each setting uses an individual training run and covers only a limited set of SMAC scenarios.The paper identifies multiple-seed evaluation and broader cooperative scenarios as future work.
Conclusion
The paper characterizes structured noise effects in cooperative MARL and proposes SIGMA to exploit cooperation structures for robust representation learning. StarCraft II experiments show strong performance under increasing observation noise, while broader task, disturbance, and multi-seed evaluation remains future work.
- Structured noise effects can be locally correlated among structurally related agents while remaining globally heterogeneous across cooperation structures.The empirical analysis supports both properties across cooperative scenarios.
- SIGMA adaptively identifies cooperation structures, aggregates information within groups, and uses inter-group attention to coordinate across groups.Its hierarchical design exploits local structural information while preserving dependencies and differences across structures.
- On StarCraft II, SIGMA maintains strong cooperative performance under increasing observation noise and degrades less than evaluated baselines.These results accompany the structured noise analysis.
- Future work will test SIGMA across broader cooperative tasks and disturbance settings, use more comprehensive multi-seed comparisons, and study general structured-noise mechanisms.