PaperlayerView exact revision on arXiv

Source-linked AI summary

UC-PSRO: Utility-Conditioned Policy-Space Response Oracles with a Communication-Dropout Curriculum for Game-Theoretic Course-of-Action Generation in Adversarial Swarms

Phillip Jiang

arXiv:2608.15372v1cs.AIcs.MA

TL;DR

Generating game-theoretic courses of action for large Blue UAS swarms facing adaptive adversaries requires addressing strategic interaction, commander intent, and communication denial. UC-PSRO combines PSRO self-play, FiLM utility conditioning, and a communication-dropout curriculum, but the curriculum alone is the strongest contributor to mission success while the added mechanisms slow convergence without a demonstrated compensating exploitability benefit.

  • Problem

    Large UAS swarms facing adaptive adversaries require game-theoretic, utility-aware, and communication-aware COA generation rather than single-agent planning.

  • Method

    UC-PSRO combines PSRO self-play, FiLM-based Commander’s-Intent conditioning, and a curriculum that trains decentralized fallback under communication dropout.

  • Results

    The Communication-Dropout Curriculum alone yields the strongest, most denial-robust mission completion, while utility-conditioning and PSRO self-play substantially slow convergence without a clear exploitability reduction.

  • Takeaways & Limitations

    Under a fixed training budget, communication-dropout training is the strongest demonstrated contributor, whereas the combined mechanisms impose convergence costs not yet offset by demonstrated robustness benefits.

  • Takeaways & Limitations

    The utility-conditioned methods were still improving at the available budget, so their ranking and exploitability findings could change with substantially more training.

Abstract

from arXiv · show

We study generating game-theoretically optimized Courses of Action (COAs) for a Blue UAS swarm against an adaptive Red adversary in a communication-degraded environment, motivated by (but not derived from) a public U.S. Air Force SBIR solicitation. We propose UC-PSRO (Utility-Conditioned Policy-Space Response Oracles with a Communication-Dropout Curriculum), combining three mechanisms: (i) PSRO self-play, so Blue and Red policies train as approximate best responses to each other rather than one side against a fixed scripted opponent; (ii) FiLM conditioning of the Blue policy on a Commander's-Intent weight vector, sampled from a Dirichlet distribution during training, so one trained policy is re-steerable at execution time without retraining; and (iii) a curriculum annealing communication-graph edge dropout during training, so the swarm learns decentralized, peer-to-peer fallback instead of depending on full connectivity. We evaluate on a synthetic, unclassified stand-in for the solicitation's maritime scenario, with 5 seeds at N=25 Blue agents and a scalability sweep to N=200. We find a genuine trade-off, not a uniform win: the communication-dropout curriculum alone gives the strongest, most robust mission-completion rates of any learned method, improving counter-intuitively as denial increases (35% to 62% success as dropout rises from 0 to 0.75); adding utility-conditioning and PSRO self-play substantially slows convergence within a fixed budget, and we find no reliable exploitability advantage for self-play over a fixed-opponent policy, both statistically indistinguishable from a small, near-zero gap. We report this honestly as a convergence cost not yet offset by a demonstrated robustness benefit, rather than overstating one method as dominant, and provide a fully vectorized, open environment training at N=200 agents in single-digit milliseconds per step on a single consumer GPU.

1 Introduction

The paper frames large-swarm COA generation as a partially observable, general-sum game and combines PSRO self-play, FiLM utility conditioning, and communication-dropout training in UC-PSRO. Its five-seed study reports a trade-off: dropout training alone is most robust, while the full combination slows convergence without a demonstrated exploitability benefit.

  • 1 Introduction: The study uses a synthetic, unclassified maritime-flavored stand-in rather than real system data or controlled technical specifications.The map, sensing and engagement ranges, and threat models are simulated.
  • 1 Introduction: UC-PSRO combines PSRO self-play, FiLM conditioning on Commander’s-Intent weights, and a Communication-Dropout Curriculum for swarm-versus-adversary COA generation.PSRO trains Blue and Red as approximate best responses to evolving policy populations; FiLM enables execution-time re-steering without retraining; dropout training promotes local or stale-information behavior.
  • 1 Introduction: Across five random seeds, the Communication-Dropout Curriculum alone produces the strongest and most reliable mission-completion behavior among learned methods.Its headline-task success increases counter-intuitively as test-time communication denial increases.
  • 1 Introduction: Adding utility conditioning and PSRO self-play substantially slows convergence at a fixed training budget without a correspondingly clear, statistically distinguishable exploitability reduction.The paper presents this as a convergence cost not yet offset by a demonstrated robustness benefit.
  • 1 Introduction: N=200 agents are supported by a fully vectorized open environment with single-digit milliseconds of environment-step latency per timestep on one consumer GPU.The implementation directly supports the stated 200-agent scale target rather than only extrapolating toward it.

2 Related Work

The paper builds on PSRO, CTDE actor-critic MARL, and multi-objective goal-conditioned RL, while positioning communication-dropout training as a complementary robustness mechanism. Classical swarm-control and pursuit-evasion methods provide reference points, and solicitation citations supply motivating context rather than technical prior work.

  • Game-theoretic and population-based multi-agent RL: PSRO grows policy populations through best-response training and solves a meta-game over empirical payoffs; this work uses the same outer loop with an exact two-player zero-sum linear program.The method specializes PSRO’s meta-solver to the exact linear-programming solution for two-player zero-sum games.
  • CTDE actor-critic MARL: MAPPO supplies the CTDE actor-critic oracle, combining a joint-state critic during training with actors using local, communication-reachable observations at execution.QMIX and MADDPG are identified as alternative CTDE algorithms, while MAPPO is chosen for simplicity and established strong performance.
  • Multi-objective and goal-conditioned RL: Commander’s-Intent conditioning follows multi-objective RL that trains one policy across scalarization-weight distributions, a harder problem than optimizing a single reward.This mechanism is related to Universal Value Function Approximators and envelope multi-objective RL.
  • Robust and decentralized MARL under communication constraints: Communication-Dropout Curriculum complements CommNet and TarMAC by assuming fixed local broadcast and training graceful degradation under channel denial rather than learning an optimal protocol.The distinction is between robustness to degraded communication and learning what and how agents should communicate.
  • Swarm robotics and pursuit-evasion: Potential-field control is the rule-based swarm baseline, while pursuit-evasion game theory is the classical continuous-time analogue of the studied discrete, many-agent adversarial game.The potential-field controller combines attraction to the objective with repulsion from threats and nearby teammates.
  • Motivating context: Three solicitation-cited works motivate the applied problem but are explicitly distinguished from technical prior work for the proposed method.They concern defense game theory, cyber wargaming, and operational analysis.

3 Problem Formulation

The paper models Blue–Red interaction as a partially observable, general-sum stochastic game with synthetic, GPU-simulable dynamics. Blue agents act through local observations over a communication graph, while rewards and termination encode Commander’s Intent and mission success.

  • Game formulation: The environment is a two-team, partially observable, general-sum stochastic game between N homogeneous Blue UAS agents and M heterogeneous Red assets.All ranges, speeds, and probabilities are synthetic placeholders designed for simulation on a single consumer GPU.
  • Blue agents: Each Blue agent observes its state and selects discretized heading-change, speed-level, and binary engage actions while consuming fuel at speed-dependent rates.Fuel exhaustion marks the agent not alive.
  • Red assets: Red combines static defense nodes, mobile interceptors with scripted or trained pursuit policies, and jammers that null communication edges within affected regions.Mobile interceptors use detection, engagement, and per-step kill-probability mechanics shared with static nodes.
  • Communication and observation: Communication edges require proximity and freedom from jamming, while training additionally applies independent Bernoulli dropout; observations include reachable teammates and sensed Red assets.Reachability spans any number of hops within the current communication graph.
  • Reward and termination: The reward is a five-dimensional feature vector scalarized as r = w⊤ϕ, with observations including the Commander’s-Intent weight vector and current dropout probability.Mission progress combines normalized distance reduction, a dense occupancy bonus, and a one-time terminal bonus.
  • Reward and termination: Episodes terminate on success at the survivor threshold, attrition failure below the attrition threshold, or timeout after a fixed number of steps.Success requires enough of the initial Blue roster to be simultaneously alive and within the objective radius.

4 Method: UC-PSRO

UC-PSRO combines utility-conditioned MAPPO, an annealed communication-dropout curriculum, and PSRO self-play to generate decentralized swarm COAs against adaptive adversaries. Its method combines established ingredients for Commander’s-Intent conditioning and communication-denial robustness, while simplifying the game to zero-sum for exact linear-programming meta-strategy solving.

  • Core architecture: UC-PSRO combines a utility-conditioned MAPPO oracle, an annealed communication-dropout curriculum, and a PSRO outer loop growing Blue and Red policy populations.The PSRO loop repeatedly invokes the oracle to move the populations toward an approximate Nash equilibrium.
  • Core architecture: A shared-parameter Blue actor executes decentrally from local, communication-reachable observations, while a centralized pooled critic keeps input size independent of swarm size.Mean- and maxpooling of every agent’s observation supports the same architecture from N=10 to N=200.
  • Communication-Dropout Curriculum: Communication edges are independently dropped during training with pdrop(t) linearly annealed from 0 to pmax, forcing reliance on stale or purely local information.Evaluation instead sweeps fixed pdrop values of 0, 0.25, 0.5, and 0.75.
  • PSRO outer loop: PSRO maintains Blue and Red policy populations initialized with non-learned reference controllers, then uses empirical payoff matrices to update the populations.Blue starts from a potential-field controller, while Red starts from scripted nearest-detected-target pursuit for mobile interceptors.
  • Game formulation: The zero-sum assumption defines Red’s reward as the negation of Blue’s w-weighted scalar reward, enabling exact meta-strategy computation through two-player maximin linear programs.The authors verify the solver against closed-form equilibria and identify the zero-sum treatment as a limitation.
  • Contribution: UC-PSRO’s contribution is combining prior ingredients for swarm-versus-adversary COA generation with formalized Commander’s-Intent utility conditioning and communication-denial robustness.The study also evaluates how each component contributes, including cases where combinations cost more than they provide within a fixed training budget.

5 Experimental Setup

The experiments use a synthetic N=25 Blue-versus-M=6 Red headline scenario, five controller configurations, and a scalability sweep to N=200. Evaluation covers communication-denial robustness, scalability, exploitability, steerability, and defined mission and systems metrics under standardized training budgets.

  • Scenarios: The headline scenario pits N=25 Blue agents against M=6 Red assets on a 100 × 100 synthetic map with a 200-step episode horizon.The scalability sweep uses N ∈{10, 25, 50, 100, 200}, preserves the Blue:Red ratio and threat mix, and grows map area with N.
  • Baselines: Five configurations compare a rule-based potential-field/Voronoi controller, fixed-opponent MAPPO, communication-dropout-curriculum MAPPO, utility-conditioned MAPPO, and full UC-PSRO.All configurations share the same network architecture, optimizer, and applicable training budget.
  • Hardware and training budget: Standalone MAPPO variants train for 600 PPO updates, while UC-PSRO runs 8 PSRO outer-loop iterations with Blue and Red best responses trained for 80 updates each.Experiments use a single NVIDIA GeForce RTX 5070, 12 GB, consumer GPU and 5 random seeds per headline and ablation configuration.
  • Metrics: Evaluation reports mission success rate, swarm survivability, Red-neutralized fraction, episode length, PSRO exploitability gap, and per-step policy inference latency.The PSRO exploitability gap is evaluated only in experiment 3.
  • Five experiments: Five experiments test communication denial at pdrop ∈{0, 0.25, 0.5, 0.75}, scalability, exploitability, and Commander’s-Intent steerability.The scalability experiment reports training wall-clock and inference latency, while exploitability freezes a trained Blue policy and trains a dedicated Red best response.

6 Results

Results show MAPPO+CDC is the strongest learned method, with success increasing under communication dropout, while utility-conditioned methods remain near-zero at the tested budget. UC-PSRO scales to N=200, but its self-play exploitability advantage and intent-conditioned mission benefit are not demonstrated.

  • Mission success: 0–4% success characterizes both MAPPO+Utility and full UC-PSRO across the communication-denial sweep, versus roughly 23–35% for plain MAPPO.The rule-based baseline succeeds essentially every episode but is included as a classical-control reference rather than a learned target.
  • Scalability: N=200 training completes in about 5 hours on a single consumer GPU, with 6.8 ms per environment step and roughly linear scaling after vectorization.The reported sweep is trained to completion through 8 PSRO iterations rather than extrapolated.
  • Exploitability: ∼10× exploitability advantage was an artifact of an earlier reward, while fixed-reward comparisons leave both Blue policies near a small, near-zero gap.The current experiment does not establish a self-play advantage because Red best-response training may be underconverged at 80 updates per PSRO iteration and 80 exploiter updates.
  • Intent steerability: 0 mission successes occur across four Commander’s-Intent conditions, despite FiLM changing the policy’s output distribution without retraining.Episode length saturates at the 200-step timeout in every condition, and behavioral differences remain small at this training budget.
  • Mission success: 71% →89% success recurs for MAPPO+CDC as dropout increases, while plain MAPPO remains comparatively flat at 34–44%.This pattern reproduces across an independent 5-seed campaign, with both utility-conditioned methods converging to exactly 0%.

7 Discussion

The discussion frames communication-dropout training as the strongest observed learned-method contribution while treating utility conditioning, reward design, and zero-sum PSRO as important limitations or unresolved issues. It also reports that the mechanism behind dropout’s benefit remains unconfirmed and that utility-conditioned methods reproduced 0% success in an independent campaign.

  • Communication-dropout curriculum: Communication dropout’s benefit lacks a definitive mechanistic explanation, but increasing denial may discourage strategies that depend on tight coordination over the communication graph.The authors propose measuring within-episode coordination, such as arrival-time variance, as a direct test across pdrop.
  • Utility conditioning: 0% success was reproduced at every dropout level by both utility-conditioned methods, while MAPPO+CDC retained a wide lead in the independent five-seed ablation campaign.The discussion attributes utility-conditioning’s convergence cost to fitting one policy across reward scalarizations sampled from Dirichlet(α1) rather than optimizing a single fixed scalarization.
  • Reward shaping: The headline mission-success metric was initially uniformly zero across learned methods, seeds, and dropout levels, revealing a structural problem rather than merely insufficient training.Diagnosis found that observations lacked any signal indicating the fixed per-episode objective’s location and instead provided each agent’s own absolute position.
  • Game formulation: The PSRO outer loop simplifies the formally general-sum game into a zero-sum Blue/Red payoff to enable exact linear-program solvability.This simplification does not make Blue’s and Red’s true objectives literal negatives, and the discussion identifies Stackelberg modeling as a matching alternative for Blue-led COA generation.

8 Limitations

The study’s conclusions are limited by a synthetic, discretized environment, a single-point training budget, and simplifying assumptions about payoffs and mission success. Utility conditioning changes policy outputs architecturally, but task-relevant steerability and larger-budget robustness remain un demonstrated.

  • Environment and transfer: All ranges, speeds, kill probabilities, and map layouts are synthetic placeholders, not validated real-system specifications.The paper makes no claims about real UAS or air-defense performance.
  • Training budget: 600 PPO updates or 8 PSRO iterations × 80 updates define a single training budget, and utility-conditioned methods remain improving rather than plateaued.Relative rankings and exploitability findings could change with substantially more computation.
  • Game formulation: Zero-sum PSRO enables exact Nash-via-LP solvability but simplifies the general-sum problem; a Stackelberg formulation remains future work.The limitation concerns the payoff formulation used for Blue and Red.
  • Environment and transfer: The study uses discrete heading/speed bins without sim-to-real transfer, hardware-in-the-loop testing, or Tactical Assault Kit-style command-and-control integration.These capabilities are required eventually by the motivating solicitation but are outside this paper’s scope.
  • Mission-success metric: 50%→30% relaxation of the simultaneous-arrival threshold was empirical rather than operationally derived, and this coordination condition made mission success rare.A different cumulative-presence definition could produce different results.
  • Utility conditioning: Commander’s-Intent conditioning changes policy outputs as a function of w, but no clear task-relevant behavioral difference across intents was demonstrated at the available budget.The authors attribute this limitation to UC-PSRO’s slow convergence rather than concluding that the mechanism fails.

9 Conclusion · A Hyperparameters

UC-PSRO combines PSRO self-play, FiLM-based Commander’s-Intent conditioning, and a Communication-Dropout Curriculum for game-theoretic COA generation in a synthetic adversarial-swarm setting. The study uses a fully vectorized environment that scales to the solicitation’s 200-agent target on a single consumer GPU, while reporting multi-seed results beyond only favorable outcomes.

  • 9 Conclusion: UC-PSRO combines PSRO self-play, FiLM-based Commander’s-Intent conditioning, and a Communication-Dropout Curriculum.These mechanisms target game-theoretic Course-of-Action generation in a synthetic adversarial-swarm setting.
  • 9 Conclusion: The evaluation uses a synthetic adversarial-swarm setting motivated by a public SBIR solicitation.The setting is presented as a motivation rather than as the solicitation itself.
  • 9 Conclusion: The fully vectorized environment scales to the solicitation’s 200-agent target on a single consumer GPU.This demonstrates the reported implementation scale for the environment.
  • 9 Conclusion: The study reports results from a rigorous, multi-seed evaluation rather than only outcomes favoring the combined method.The supplied conclusion passage explicitly frames the reporting choice as avoiding selective emphasis on favorable results.
  • A Hyperparameters: Table 6 lists MAPPO / PSRO hyperparameters used as defaults for all reported experiments.The supplied table caption identifies these as the default settings.
  • A Hyperparameters: Table 7 presents the headline scenario parameters.No individual parameter values are provided in the supplied passage.

B Additional results: survivability · C FiLM conditioning sanity check

Survivability is similar across learned methods after the reward fix, while the FiLM sanity check shows that varying only the Commander’s-Intent vector measurably changes policy outputs. The conditioning mechanism therefore meaningfully influences the actor’s output distribution before training.

  • B Additional results: survivability: Survivability is similar across all learned methods after the §7 reward fix.The measure is the fraction of the initial Blue roster alive at episode end, averaged over 5 seeds.
  • B Additional results: survivability: The reward fix removes risk-avoidance as the dominant behavioral separator among methods.This explains why survivability no longer differentiates the learned methods strongly.
  • B Additional results: survivability: Survivability is reported for the headline scenario rather than mission success.The passage explicitly distinguishes this measure from mission success in Table 1.
  • C FiLM conditioning sanity check: Before training, FiLM conditioning was verified by holding the observation fixed and varying only w.The tested Commander’s-Intent vectors were two simplex vertices and the midpoint [0.2]5.
  • C FiLM conditioning sanity check: The actor’s heading-logit outputs differ measurably across the three Commander’s-Intent settings.The compared settings were [1, 0, 0, 0, 0], [0, 0, 0, 0, 1], and [0.2]5.
  • C FiLM conditioning sanity check: The first heading logit varies from −0.036 to −0.059 to −0.003 on an untrained network.This example demonstrates output variation under fixed observations and changing utility weights.

D Convergence diagnostic: reward trend under the original vs. fixed reward · E Environment vectorization

Relaxing the simultaneous-arrival threshold enabled nonzero mission success despite nearly identical reward trajectories, identifying coordination as the binding constraint. Environment vectorization replaced per-pair Python and repeated per-agent graph computations with batched operations and a single connected-components pass.

  • D Convergence diagnostic: reward trend under the original vs. fixed reward: 0.0% mission success persisted throughout training with the original reward and fixed reward at τ=0.5.The fixed reward at τ=0.5 added target-relative observation and a dense at-target bonus while retaining the 50% survivor threshold.
  • D Convergence diagnostic: reward trend under the original vs. fixed reward: 0.9%, 1.0%, 2.6%, and 1.1% mission success occurred across the four 200–600 chunks after relaxing the threshold to τ=0.3.Success became nonzero starting in the 200–300 chunk and peaked at 2.6%.
  • D Convergence diagnostic: reward trend under the original vs. fixed reward: Nearly identical reward trajectories at τ=0.5 and τ=0.3, but success only at τ=0.3, implicated simultaneous-arrival coordination rather than general task difficulty.The passage identifies the coordination requirement as the binding constraint.
  • E Environment vectorization: The environment latency comparison targeted communication-graph construction, engagement rolls, and observation construction on a single consumer GPU host.Environment stepping itself remained CPU-bound regardless of GPU.
  • E Environment vectorization: Per-pair Python loops and repeated individual numpy.linalg.norm calls made call overhead, rather than underlying O(N2) arithmetic, the dominant pre-vectorization cost.Connected-component reachability was also recomputed per agent through breadth-first search.
  • E Environment vectorization: Batched pairwise-distance matrices replaced repeated per-pair computations in the vectorized implementation.This change addressed communication-graph and related pairwise operations.
  • E Environment vectorization: A single connected-components pass per step replaced one breadth-first search per agent.The change reduced repeated graph-reachability computation within each environment step.
Loading 2608.15372v1…