Source-linked AI summary
TANDEM: Balancing Centralized and Decentralized Critics to Address the Information Gap in Complex Multi-Agent Environments
Alireza Saleh Abadi, Leen-Kiat Soh, Adam Eck, Prashant Doshi
TL;DR
CTDE trains decentralized policies with centralized information that is removed at execution, creating a gap between training and deployment. TANDEM uses centralized and decentralized reads of one critic, with a gate selecting between them. At execution, TANDEM is best on eight of nine setups and never significantly worse than any of six CTDE baselines.
Problem
CTDE optimizes decentralized policies with a global-state critic that is removed at execution, leaving a gap between training information and the local histories agents use to act.
Source: §1 · p. 1Method
TANDEM reads one critic centrally and through each agent’s execution-time information, using a parameter-free gate to select decentralized advantages when reads agree and centralized ones when they disagree.
Source: §1 · p. 2Results
At execution, TANDEM is best on 8 of 9 setups and never significantly worse than any of the 6 CTDE baselines.
Source: §9 · p. 8Takeaways & Limitations
The gate’s contribution is clearest under openness, where removing it significantly lowers execution return.
Source: §9 · p. 9The evaluated domains each isolate one form of the information gap rather than combining them.
Source: §9 · p. 9
Abstract
from arXiv · showhide
Cooperative multi-agent reinforcement learning under centralized training with decentralized execution (CTDE) shapes decentralized policies with a centralized critic that observes the global state, then removes the critic at execution. The policies are optimized to rely on information they will not have when they act, an information gap between training and execution. It widens under partial observability, observation noise, and openness, where the set of agents and tasks present changes during an episode. Existing methods transfer centralized guidance in one direction only, removing or distilling the critic once, so the policies never rehearse acting without the critic while they can still be corrected. We propose TANDEM, which keeps one centralized critic and reads it two ways at each training step: a centralized read from the global state, and a decentralized read restricted to the information each agent will have at execution. A parameter-free gate compares the two reads for each agent and trains the policy on the decentralized read where they agree, reinstating the centralized read only where the local view is misleading. A shared target pulls the decentralized read toward the centralized one. We formalize the setting as a task- and agent-open decentralized partially observable Markov decision process (TAO-Dec-POMDP) and evaluate TANDEM on three cooperative domains, each isolating one form of the information gap: discovery for partial observability, sampling for observation noise, and openness in wildfire suppression. We compare against six CTDE baselines and ablate the gate and the shared target; the supplementary material holds the full variant sweep, diagnostics, and learning curves. We measure return at execution, once the centralized critic is removed. TANDEM is the best method on eight of the nine setups and never significantly worse than any baseline.
1 Introduction
CTDE trains decentralized policies with a global-state critic that is removed at execution, creating an information gap that can leave policies unable to sustain their training-time coordination. TANDEM addresses this gap by alternating between centralized and execution-matched decentralized critic reads, while the background defines the changing-agent and partial-observation settings it considers.
1 Introduction
- CTDE’s global-state critic is removed at execution, so policies optimized with centralized information may not sustain their coordination when agents must act on local histories.The mismatch is especially relevant under partial observability, observation noise, and openness.Source: §1 · p. 1
2 Related Work
- Teacher–student [18] [19], Kickstarting [34], and Message-Dropout transfer centralized guidance monotonically, so policies do not rehearse execution-time information while remaining correctable.Standard CTDE also retains the critic throughout training and removes it only at execution.Source: §1 · pp. 1–2
1 Introduction
- TANDEM uses one critic for centralized and execution-matched decentralized reads, reinstating the centralized read when local information misleads and sharing a target between reads.A parameter-free gate selects between advantage estimates per agent and step.Source: §1 · p. 2
3.1 Open Agent Systems
- Openness covers changing agent, task, or capability sets; TANDEM addresses agent and task openness while leaving frame openness for future work.Agent openness changes team composition; task openness changes available work.Source: §3.1 · p. 3
3.2 The Open Dec-POMDP
- The open Dec-POMDP models cooperative partial-observation control with a changing active coalition, but not changing tasks; Section 4 extends it to task openness.Its dynamics jointly model next state, observations, and coalition membership.Source: §3.2 · p. 3
3.3 Partial Observability and Observation Noise
- Partial observability restricts each agent to a local view, while observation noise further perturbs the view used at execution.The acting observation can differ from what the agent has learned.Source: §3.3 · p. 3
3.4 CTDE and MAPPO
- CTDE uses global information during training and local histories at execution; MAPPO provides a state-only centralized critic and decentralized policies updated from its advantages.TANDEM requires a state-only critic so it can also be evaluated from an agent’s masked local view.Source: §3.4 · p. 3
4 Problem Formulation
The formulation extends an open Dec-POMDP to represent both changing agents and changing tasks. Tasks are carried in the state, and their presence determines which task-related actions are available.
4 Problem Formulation
- TAO-Dec-POMDP extends the open Dec-POMDP to include task openness alongside its existing agent openness.The model carries the task set and provides an availability map for each agent.Source: §4 · pp. 3–4
- The state pairs remaining world features with the tasks currently present, and each agent’s available actions depend on that task set.Actions addressing a task are available only while that task is present.Source: §4 · p. 4
- Observation noise perturbs the observation component while reporting action availability faithfully; fixing the task set removes task openness and recovers the open Dec-POMDP..Source: §4 · p. 4
- The model excludes voluntary agent departures: an agent exits only when its state forces it to, a scope choice matching the evaluated domains.An example is a firefighter whose suppressant runs out.Source: §4 · p. 4
5 Methodology
TANDEM keeps one centralized critic but reads it globally and through each agent’s masked local state, then gates policy updates between those reads. Its theoretical results characterize when decentralized information suffices, while its attention-based design accommodates changing agent and task sets.
5.1 Two reads of one critic
- One critic supplies a global read and a per-agent masked-state read, both trained toward the same centralized return target.The local read deliberately uses the current masked state rather than the full history, targeting the state–observation gap rather than the memory gap.Source: §5.1 · p. 4
5.2 Gated advantage and policy update
- A parameter-free gate closes when the reads’ advantages agree in sign and reinstates centralized guidance when they disagree.The mixed advantage drives the policy update, and the shared target moves the operating point toward decentralized training as the reads agree.Source: §5.2 · pp. 4–5
5.3 Theoretical properties
- When local information does not determine the state, decentralized policies may fall below the state-conditioned benchmark, while centralized advantages can be high-variance and credit-confounded.The paper attributes these limits to partial observability, observation noise, or openness.Source: §5.3 · p. 5
- Where advantages agree in sign, TANDEM uses the decentralized advantage; centralized guidance enters only on disagreement steps.The decentralized advantage is realizable in expectation from the agent’s information, and the shared target drives its read toward the centralized read subject to the residual information gap.Source: §5.3 · pp. 5–6
- With no sign disagreements, the actor update is exactly decentralized and adds no bias relative to the decentralized objective; the gate is scale-free as its guard tends to zero.The critic still regresses the decentralized read toward the centralized target, so only the actor update becomes decentralized.Source: §5.3 · p. 6
5.4 Computational complexity
- At execution TANDEM costs the same as a decentralized actor, while training adds 1+n critic views per step without adding parameters.Its attention critic has quadratic token-attention cost and is described as scalable as a standard centralized-critic actor-critic.Source: §5.4 · p. 6
5.5 Invariance to the agent and task sets
- The attention critic accepts variable agent and task counts, and the actor applies to whichever agents are active without a fixed roster.Changing sets are handled by token presence and masking, with no additional openness-specific machinery.Source: §5.5 · p. 6
6 Experiments
The experiments compare TANDEM with six CTDE baselines across nine setups in three cooperative domains. They measure execution return after removing the centralized critic, using matched training budgets and held-out evaluation worlds.
6 Experiments
- Discovery tests partial observability, sampling tests observation noise, and wildfire suppression tests openness; each domain includes two gap setups and a control.The gap setups increase difficulty along each domain’s axis, while controls remove the corresponding gap.Source: §6 · pp. 6–7
- Six CTDE baselines spanning policy-gradient, value-factorization, and off-policy actor-critic methods provide the comparison across nine setups.All methods receive the same reward and 1.2M-step training budget.Source: §6 · pp. 6–7
- Performance is execution return on 100 held-out worlds, with the centralized critic removed and paired significance tests corrected across setup comparisons.The final checkpoint is evaluated, and error bars report standard error over the episodes.Source: §6 · p. 7, Figure 3 · p. 7
7 Results
TANDEM performs strongly at execution across the nine setups, and its widest margins occur in discovery, where the critic’s two reads differ most.
7 Results
- At execution, TANDEM is best on 8 of 9 setups and never significantly below any of the six CTDE baselines.Its widest margins are on discovery, where the critic’s two reads have the widest gap.Source: §7 · p. 7
- TANDEM significantly outperforms all six CTDE baselines on five setups, including both discovery gap setups, sampling noise 0.2, the sampling control, and closed wildfire.Source: §7 · p. 7
- TANDEM leads on three of the four remaining setups; only on the more-open setup does a baseline mean exceed its return, without a significant difference.On that setup, MAPPO scores 315.3 versus TANDEM’s 306.6 (corrected p=0.68).Source: §7 · p. 7
- The critic-read gap is widest on discovery, while the gate stays closed on the sampling no-noise and wildfire closed controls, where the reads agree.Source: §7 · p. 8
8 Ablations
The ablations indicate that the gate responds to gap presence, while the shared target improves the decentralized critic read; neither mechanism uniformly raises execution return on every setup.
8 Ablations
- Forcing the gate closed lowers open-wildfire execution return from 290.1 to 272.6, significantly below the full method.On the other five gap setups, the change misses significance; the gate stays closed on the two controls where the reads agree.Source: §8 · p. 8
- The shared target improves decentralized-read fit on gap setups and lowers δ, but its execution-return effect is not significant except on the larger discovery map, where removing it scores higher.Source: §8 · p. 8
- Figure 4 relates sign-disagreement fraction to information-gap size and compares that fraction with and without the shared target on gap setups.The controls have zero disagreement in both conditions.Source: Figure 4 · p. 8
- On the larger discovery map, three variants significantly exceed the full method’s return, so the gate’s selectivity does not maximize return among TANDEM variants on every map.Source: §8 · p. 8
9 Conclusion
The paper concludes that TANDEM addresses the information gap in cooperative MARL under CTDE and performs strongly after the centralized critic is removed.
9 Conclusion
- At execution, TANDEM is best on 8 of 9 setups and never significantly worse than any of the six CTDE baselines.Its margins are widest on discovery, where it significantly outperforms every baseline on the gap setups.Source: §9 · p. 8
- TANDEM leaves frame openness—changes in agents’ goals or capabilities—unexplored.Source: §9 · pp. 8–9
- The evaluation isolates one information-gap form per domain rather than combining forms, and standard cooperative benchmarks remain future work.Because the gate uses sign agreement, it cannot correct cases where both reads agree but are wrong.Source: §9 · p. 9
Supplementary Material
The supplement collects the supporting experimental record, including variants, diagnostics, curves, configurations, and implementation details.
Supplementary Material
- The supplement includes the complete variant sweep, shared-target and gate diagnostics, training returns, learning and validation curves, and gate dynamics.Source: Supplementary Material · p. 11
- Each reported policy is evaluated on 100 unseen execution episodes with the centralized critic removed, using paired seeds and a corrected paired Wilcoxon test.Policies are trained for 1000 episodes under an identical 1.2M-environment-step budget.Source: Supplementary Material · p. 11
- The supplement adds the gate-forced-open and no-decentralized-read variants to the main paper’s two ablations.Source: Supplementary Material · p. 11
S1 All TANDEM variants
Across the nine setups, the full method differs significantly from its variants on only two; its clearest variant advantage is on open wildfire, while several variants score higher on the larger discovery map.
S1 All TANDEM variants
- Only two of nine setups show a significant difference between a TANDEM variant and the full method; sampling no-noise and wildfire closed controls give identical variant returns.Discovery full-observation returns range from 6.23 to 6.72, with no significant difference from TANDEM.Source: S1 All TANDEM variants · p. 11
- On open wildfire, forcing the gate closed scores 272.6, 17.48 below TANDEM’s 290.1 (p<0.05).The gate-closed variant does not differ significantly from TANDEM in any other setup.Source: S1 All TANDEM variants · p. 11
- On the larger discovery map, gate-open (9.10), no-decentralized-read (8.78), and no-shared-target (7.71) variants each significantly exceed TANDEM (6.91).On the smaller discovery map, gate-open and no-read variants score 4.12 versus 3.70, but not significantly.Source: S1 All TANDEM variants · p. 11
S2 The shared target at the critic
The shared target improves the decentralized read on gap setups, while the gate is active where the reads disagree and stays closed on controls. Together, these diagnostics link critic-read alignment to the gate’s selective use of centralized guidance.
S2 The shared target at the critic
- With the shared target, decentralized-read fit rises from 0.63 to 0.77 and sign disagreement falls from 28.7% to 25.3% on sampling noise 0.2.On open wildfire, fit rises from 0.75 to 0.78 and disagreement falls from 25.4% to 18.3%.Source: S2 The shared target at the critic · p. 11, Table S2 · p. 12
- Regressing both reads to one centralized target pulls the decentralized read toward the centralized read, so the gate closes wherever local information is sufficient.The measured shared-target effect is on the critic’s reads; its execution-return effect is reported separately in Table S1.Source: S2 The shared target at the critic · p. 11
S3 The gate against the information gap
- The gap, gate, and sign-disagreement fraction are nonzero on all six gap setups and zero on sampling no-noise and wildfire closed controls.On discovery full-observation, the gap is 0.26, gate mean 0.019, and disagreement 11.2%; disagreement tracks gap presence, not its width.Source: S3 The gate against the information gap · p. 12
S4 Training efficiency
TANDEM leads the CTDE baselines in mean training return across all setups, and validation and execution results remain at or near the top. Its gate remains engaged on gap setups, while training-to-execution return changes little relative to baselines that rely more heavily on centralized signals.
S5 Learning curves
- TANDEM tracks at or above its two ablations and MAPPO throughout training, with the curves separating mainly where the information gap drives the gate open.The separation is most visible on open wildfire.Source: S5 Learning curves · p. 12
S6 Training curves for all CTDE baselines
- Across all setups, TANDEM reaches the highest training return among itself and the six CTDE baselines.On wildfire, TANDEM and MAPPO lead QMIX, VDN, MAVEN, and MADDPG by a wide margin.Source: S6 Training curves for all CTDE… · p. 12
S7 Validation curves during training
- On unseen validation seeds, TANDEM’s return stays at or near the top of the seven methods throughout training on each setup.The validation seeds are disjoint from training and execution seeds and are not used for checkpoint selection.Source: S7 Validation curves during training · p. 12
S4 Training efficiency
- TANDEM has the highest mean training return of the seven methods on every setup under the shared 1.2M-step budget.It exceeds MAPPO from 6.44 versus 4.16 on larger discovery to 259.7 versus 234.1 on open wildfire.Source: S4 Training efficiency · p. 12, Table S4 · p. 13
S8 Gate dynamics over training
- The gate mean and sign-disagreement fraction stay nonzero on six gap setups and flat at zero on sampling no-noise and wildfire closed controls.On discovery full-observation, both rise slowly because the global-state token and LIDAR observation remain different inputs.Source: S8 Gate dynamics over training · p. 13, Table S3 · p. 13
S9 Training versus execution return
- TANDEM is at or near the top in training and holds its return at execution, with a smaller train-to-execution change than baselines leaning most on centralized signals.Figure S5 compares the final 100 training episodes with 100 unseen execution episodes after the critic is removed.Source: S9 Training versus execution return · p. 13, Figure S5 · p. 18
S10 Hyperparameters
The supplement reports a shared-budget tuning protocol, setup definitions, held-out seed pools, and implementation details for TANDEM and its baselines. Comparisons use one trained policy per method at a shared random draw.
S10 Hyperparameters
- All methods were tuned with Optuna under a shared protocol and budget, then trained for 1000 episodes and 1.2M environment steps.Validation seeds informed tuning, while execution used held-out seeds 201–300; value-based methods anneal epsilon over 2000 episodes, leaving epsilon at 0.525 at the final checkpoint.Source: S10 Hyperparameters · p. 19
S11 Setup details
- Each domain has two gap setups and one control; on controls, the critic reads carry the same information or nearly so, leaving the gate closed or nearly closed.Wildfire’s agent and task sets change within an episode.Source: S11 Setup details · p. 19, Table S7
S12 Seeding protocol in full
- Training, validation, and execution use disjoint seed pools, and each comparison tests one trained policy per method at a shared draw.The design favors breadth over environment seeds rather than replication over random-number-generator seeds, so results are conditional on that shared draw.Source: S12 Seeding protocol in full · p. 20
S13 Architecture, optimization, and implementation details
- TANDEM uses a width-64 residual actor and a two-layer entity-token transformer critic with 128-dimensional representations and four attention heads.Its actor uses clipped PPO on the mixed advantage; the two critic reads are trained against one shared normalized centralized-return target.Source: S13 Architecture, optimization, and… · p. 20
S14 Masked-read contents per domain
The section specifies how each domain constructs the decentralized critic read, including a denoised view under sampling noise, and reports final-campaign compute and software details.
S14 Masked-read contents per domain
- Discovery masks retain the global token, the agent’s own token, and targets within LIDAR range while excluding teammates; without observation noise, this matches the agent’s information.Source: S14 Masked-read contents per domain · p. 20
- Sampling masks hide field tokens and critic-only summaries, but retain noise-free entity features, making the decentralized read denoised rather than a strict subset of the actor’s noisy information.The read retains the global token, the agent’s own token, and nearby teammates’ tokens.Source: S14 Masked-read contents per domain · p. 20
- Wildfire masks redact teammates’ water and power, zero selected teammate attributes and the hidden refill rate, and under noise substitute the agent’s stored noisy view for selected readings.The agent’s own token is never overridden.Source: S14 Masked-read contents per domain · p. 20
S15 Compute and software
- Final-campaign training used 126 jobs across nine setups and 14 configurations, with CPU-only Slurm jobs limited to 72 hours; held-out execution used 126 jobs.Training jobs used two CPU cores and 16 GB; execution jobs used two cores and 8 GB. Software included PyTorch ≥2.6 with deterministic algorithms enabled.Source: S15 Compute and software · p. 20