PaperlayerView exact revision on arXiv

Source-linked AI summary

Adversarial Course-of-Action Generation: Game-Theoretic Multi-Agent Algorithms for COA matching & COA generation

Natan Vidra, Alina Kapanova, Arun Kanhai, Spurthi Setty

arXiv:2609.26059v1cs.AIcs.MA

TL;DR

COA generation requires evaluating structured plans against adversarial responses and planning constraints, while existing BattleCOA decision functions are not implemented by this artifact. COA-Bench supplies an offline, inspectable self-play benchmark; sampled response and council evaluations report changes in BLUE advantage and win rate, with results limited by synthetic metrics and templated scenarios.

  • Problem

    COA quality depends on the force package, opponent response, timing, resources, and qualitative planning criteria, while COA-Bench does not implement operational BattleCOA decision functions.

    Source: §1 · p. 1, §2 · p. 2

  • Method

    COA-Bench evaluates typed COA artifacts through adversarial self-play, internal quality and advantage scores, and doctrine-inspired diagnostics.

    Source: §1 · p. 1, §2 · p. 2, §5.2 · p. 3

  • Results

    At k=8, BLUE advantage falls from .516 to .485 and BLUE win rate from .920 to .820; the council yields .509 BLUE advantage and .820 BLUE win rate.

    Source: §6.2 · p. 3

  • Takeaways & Limitations

    COA-Bench is a small, inspectable foundation for stronger BattleCOA-oriented benchmarks and exposes benchmark-design failures such as scenario framing that has no effect on generated content.

    Source: §9 · p. 4

    Synthetic quality, doctrine, and wargame metrics may reward artifacts that would not survive expert review, and 50 templated cases cannot represent operational-planning complexity.

    Source: §8 · p. 4

Abstract

from arXiv · show

Course-of-action (COA) generation is a distributed planning problem: a system must propose structured candidate actions, evaluate them against an adversarial response, and surface options that remain tactically coherent under changing conditions. We present COA-Bench, a small offline benchmark and reproducibility artifact for comparing COA generation policies through self-play. Following the BattleCOA terminology, we reserve COA matching for asset-effect matching and COA generation for course-of-action generation; the present artifact does not implement either DecisionFunction directly. Instead, it represents COAs as typed action chains with conditional branches, assigns a synthetic COA quality score, compares opposing COAs with a BLUE-vs-RED advantage score and Nash-gap distance, and scores doctrinal coherence with an FM 3-0-inspired heuristic rubric. Across 50 synthetic scenarios spanning five operational templates, a sampled best-response policy that draws eight RED candidates reduces BLUE advantage from .516 to .485 and BLUE wargame win rate from .920 to .820; a two-stage multi-agent council with five BLUE proposer agents, RED-team adjudication, and critique-driven revision obtains .509 BLUE advantage and .820 BLUE win rate. We also identify and fix a benchmark-design issue in which scenario framing was stored as metadata but had no effect on generated COA content. COA-Bench is not an operational battle-management system and uses no real, classified, proprietary, or human-subject data. The contribution is an inspectable evaluation harness, preliminary benchmark evidence, and lessons for building auditable agentic planning artifacts.

1 Introduction

COA development must account for adversarial responses, resource and timing constraints, and qualitative planning criteria. COA-Bench offers a modest offline harness for reproducibly evaluating generation policies while keeping assumptions and failures inspectable.

1 Introduction

  • COA quality depends on the force package, adversary, timing, resource constraints, and qualitative criteria including combined-arms balance, sustainment, risk mitigation, and tempo.Source: §1 · p. 1
  • COA-Bench is an offline, synthetic benchmark report designed to expose assumptions and make policy-evaluation failures inspectable.Source: §1 · p. 1
  • Its contributions include a formal adversarial setup, a reproducible harness with typed traces and doctrine-inspired diagnostics, and evidence that sampled responses change evaluation conclusions.The artifact is not an operational planner.Source: §1 · p. 1
  • The artifact compares self-play policies and evaluates synthetic quality, BLUE-vs-RED advantage, Nash-gap distance, doctrinal alignment, diversity, Pareto frontiers, and stylized wargame outcomes.Source: §1 · p. 1
  • The reproducibility package regenerates results without API keys, live systems, or sensitive data.Source: §1 · p. 2

2 BattleCOA Scope

COA-Bench borrows BattleCOA framing to motivate structured candidate plans but treats its scores as internal benchmark measures, not operational decision functions. It is a reproducible research harness, not a full battle-management workflow.

2 BattleCOA Scope

  • BattleCOA motivates representing candidate plans as event structures with assets, effects, paths, and supporting relationships.Source: §2 · p. 2
  • COA-Bench assigns a synthetic quality score to a generated COA and a BLUE-vs-RED advantage score to paired COAs.Source: §2 · p. 2
  • These scores are internal benchmark metrics, not claims about operational BattleCOA decision functions.Source: §2 · p. 2
  • COA-Bench is a reproducible research harness, not an implementation of the full Transformational Model for Decision Advantage or an operational battle-management workflow.Source: §2 · p. 2

3 Problem Formulation

The benchmark frames COA generation as a meta-decision over candidate options and adversarial responses, rather than as one-shot text generation. Its multi-agent design scores, revises, and selects BLUE candidates using quality, doctrine, diversity, and adversarial pressure.

3 Problem Formulation

  • The benchmark asks whether BLUE remains strong when RED samples a response set and selects its highest-scoring adversarial candidate.It computes a utility-like advantage signal, doctrine-inspired diagnostics, and candidate-set statistics for paired COAs.Source: §3 · p. 2
  • COA generation is treated as a meta-decision over options and responses, not a one-shot text-generation task.Source: §3 · p. 2
  • The multi-agent council has BLUE proposers generate candidates and RED teams respond to each candidate before adjudication.Source: §3 · p. 2
  • The adjudicator score combines rescaled synthetic quality, a doctrine proxy, candidate-set diversity, and pressure from the strongest RED response; the top two candidates receive critique-keyed mitigation revisions.Source: §3 · p. 2
  • The reported council uses five proposer roles, four RED responses per candidate, a revision width of two, and weights (.45, .25, .10, .20).It reports a robustness margin and pressure reduction.Source: §3 · p. 2

4 Related Work

The related work situates COA-Bench among adversarial self-play, military decision modeling, agent benchmarks, and structured planning systems. It borrows evaluation ideas while distinguishing its synthetic benchmark from trained policies, validated combat models, and tool-learning benchmarks.

4 Related Work

  • Self-play systems AlphaZero, AlphaStar, and Pluribus motivate adversarial policy evaluation, while CICERO extends strategic reasoning to natural-language negotiation.COA-Bench adopts paired-response evaluation but evaluates simple generation policies rather than training policies.Source: §4 · p. 2
  • Military-modeling precedents include classic attrition equations [7] and Schelling’s strategic-conflict account, but COA-Bench’s attrition check is only a secondary diagnostic, not a validated combat model.Its FM 3-0-inspired rubric is a heuristic feature extractor, not expert judgment.Source: §4 · p. 2
  • Agent benchmarks motivate evaluating planning, action, feedback, and state change; COA-Bench records routes, quality, doctrinal features, diversity, and wargame outcomes to keep behavior inspectable.Related systems include ReAct [17], AgentBench [8], SWE-bench [4], 𝜏-bench [9], ToolSandbox [16], PlanningArena [6], ACPBench, and AgentDiagnose.Source: §4 · p. 2
  • Tool-learning and planning projects [11] [13] [5] [18] motivate structured COA artifacts, but COA-Bench is not a tool-learning benchmark.Its typed actions, conditional branches, and optional tool-call slots allow later LLM-backed policies to be compared with programmatic baselines.Source: §4 · p. 2

5 Benchmark Design

COA-Bench represents plans as typed COAs and evaluates paired BLUE and RED plans with benchmark metrics. It also uses transparent, limited heuristics for doctrinal alignment and wargame outcomes.

5.1 COA Representation

  • COAs are typed objects with objectives, force assignments, domain tags, ordered or chained actions, optional branches, and optional tool calls.Actions include a category, target, priority, and supporting metadata.Source: §5.1 · p. 2
  • Synthetic COA quality combines effectiveness, cost, and risk using default weights of 0.5, 0.3, and 0.2, respectively.The score ranges from −1 to 1 and is an internal benchmark metric, not asset-effect matching.Source: §5.1 · p. 2

5.2 Pairwise Evaluation

  • The BLUE-vs-RED advantage score compares paired COAs, while Nash-gap distance diagnoses imbalance without establishing equilibrium convergence.The advantage is calculated as (q_blue − q_red + 2)/4; quality scores are rescaled to [0, 1] for the Nash-gap diagnostic.Source: §5.2 · pp. 2–3

5.3 Doctrinal and Wargame Diagnostics

  • The doctrinal rubric scores six dimensions, and a separate stylized Lanchester-style check scales raw combat power by synthetic quality and doctrinal alignment.The dimensions are objective clarity, intelligence preparation, combined-arms balance, sustainment, risk mitigation, and tempo or sequencing; both diagnostics are intentionally transparent and limited.Source: §5.3 · p. 3

6 Evaluation

Across 50 synthetic scenarios, sampled RED best responses lower BLUE advantage and win rate, while a multi-agent council changes BLUE generation through critique-driven revision. The evaluation also tests response budgets, exposes implementation choices, and fixes a framing parameter that previously had no effect on generated content.

6.1 Protocol

  • The protocol evaluates 50 scenarios across five operational templates, comparing single-sample, sampled, doctrine-aware, and multi-agent council policies.It records multiple metrics and reports 95% percentile bootstrap intervals over the scenarios.Source: §6.1 · p. 3

6.2 Results

  • Under sampled best-response evaluation, RED becomes tougher: BLUE advantage falls from .516 to .485 as RED’s quality score increases under best-of-eight selection; in the stylized wargame check, BLUE win rate falls from .920 to .820.Source: §6.2 · p. 3
  • BLUE advantage drops from .516 at k=1 to .479 at k=16 as RED samples more candidates.Across eight candidates, mean diversity is .682, mean quality-score spread is .246, and mean Pareto-optimal count is 2.16; the doctrine-aware selector makes a different choice in 56% of scenarios.Source: Table 3 · p. 3, §6.2 · p. 3
  • The council produces .509 BLUE advantage and .820 BLUE win rate; revised candidates are selected in 76% of scenarios.Revised-candidate diversity is .644 versus .571 initially, while pressure reduction is small and its interval overlaps zero.Source: Table 4 · p. 3, §6.2 · pp. 3–4
  • The doctrine-aware selector yields .488 BLUE advantage and .800 BLUE win rate, changing the chosen response in 56% of scenarios.Compared with the quality-greedy response, it incurs .041 quality regret while gaining .106 doctrinal alignment.Source: §6.2 · p. 3

6.3 Implementation Artifact

  • The artifact separates scenario construction, interchangeable response policies, and evaluation outputs, and uses file-based execution to avoid hidden state and API dependence.Its response-budget curve treats RED’s candidate count k as an explicit variable for examining performance against deeper response search.Source: §6.3 · p. 4

6.4 Benchmark-Design Finding

  • A framing parameter that had been metadata-only was patched to affect objectives and, under BLUE-favorable framing, could add an action category; sensitivity delta is .066 [.049, .086].The authors report this as a benchmark-design finding, not a realistic estimate of framing effects.Source: §6.4 · p. 4

7 Industry Lessons

The industry lessons emphasize controlled terminology, adversarial evaluation, useful multi-COA output, and transparent artifacts. COA-Bench’s practical value is framed as inspectability rather than the absolute score of a current policy.

7 Industry Lessons

  • Terminology should distinguish formal COA matching and generation decision functions from synthetic evaluation scores.Reusing those terms for internal metrics created a misleading connection to BattleCOA DecisionFunctions.Source: §7 · p. 4
  • A COA that looks strong against one response may look weaker when the opponent samples several plausible responses.The paper therefore treats adversarial sampling as a factor that can change evaluation conclusions.Source: §7 · p. 4
  • Multi-COA output is more useful than one selected plan when candidates are diverse and nonredundant.Source: §7 · p. 4
  • The benchmark’s value lies in inspecting why a policy looked strong, including candidate diversity, saturated rubric terms, and whether scenario variables affect generated content.The authors connect this emphasis to trajectory diagnosis and reproducible evaluation for distributed AI systems.Source: §7 · p. 4

8 Threats to Validity and Ethics

COA-Bench’s synthetic, offline design limits what its results can establish: its metrics may not match expert judgment, and 50 templated cases cannot represent operational-planning complexity. The artifact evaluates typed COA artifacts rather than a full BattleCOA pipeline, and future live evaluation would require additional safeguards.

8 Threats to Validity and Ethics

  • Synthetic metrics may reward artifacts that would not survive expert review, while 50 templated cases cannot represent operational-planning complexity.The paper identifies these as internal- and external-validity risks, respectively.Source: §8 · p. 4
  • The benchmark evaluates typed COA artifacts, not COA matching or generation as full BattleCOA DecisionFunctions.The paper identifies this as a construct-validity risk.Source: §8 · p. 4
  • Any future live evaluation would require strict data governance, human review, security controls, expert validation, and clear limits on autonomous action.The current artifact performs no external action and is intended only for research on evaluation methodology.Source: §8 · p. 4

9 Conclusion

COA-Bench is a small, inspectable benchmark for evaluating COA-generation policies through adversarial self-play. Its findings include a tougher synthetic opponent under sampled best-response, meaningful candidate diversity, and a benchmark-design failure in which scenario framing had no effect on generated COA content.

9 Conclusion

  • COA-Bench provides a small, inspectable benchmark for evaluating course-of-action generation policies under adversarial self-play.The paper characterizes it as a preliminary foundation for stronger BattleCOA-oriented benchmarks.Source: §9 · p. 4
  • Sampled best-response produces a tougher synthetic opponent, and candidate sets contain meaningful diversity.Source: §9 · p. 4
  • Transparent artifacts exposed a design failure: scenario framing was stored as metadata but did not affect generated COA content.Source: §9 · p. 4

Generative AI Disclosure

Generative AI tools materially assisted with implementation, experiment scripting, terminology review, and manuscript drafting; the authors report responsibility for reviewing and approving the work.

Generative AI Disclosure

  • Generative AI tools materially assisted with software implementation, experiment scripting, terminology review, and manuscript drafting.The authors state that they reviewed the code, verified generated results against released artifacts, checked citations, and approved the final claims and text.Source: Generative AI Disclosure · p. 5
Loading 2609.26059v1…