PaperlayerView exact revision on arXiv

Source-linked AI summary

MA-Bench: Measuring Operational Planning in Language Models

Rodney Lafuente-Mercado, Robert Sincero, Phil Sage, Yuan Xue

arXiv:2610.19138v1cs.LGcs.AIcs.CL

TL;DR

Operational planning lacks a verifiable outcome or unique reference answer, while a fluent brief may still omit required determinations. MA-Bench grades mission-analysis briefs against rubric requirements; across models, missing credit concentrates in inference and omissions, and the best gets 62% of rubric items right.

  • Problem

    Operational planning has no verifiable outcome or unique reference answer, making it difficult to evaluate whether a brief contains the determinations a competent staff would make.

    Source: §1 · pp. 1–2

  • Method

    MA-Bench grades mission-analysis briefs against atomic rubric requirements, distinguishing extraction, synthesis, and inference while scoring omissions and detrimental actions.

    Source: §1 · p. 2

  • Results

    Models’ missing credit concentrates in inference and failures are omissions rather than errors; the best model gets 62% of rubric items right.

    Source: §5.2 · p. 7, §6 · p. 10

  • Takeaways & Limitations

    The benchmark measures whether systems make situation-demanded planning judgments, rather than retrieval or fluency.

    Source: §1 · p. 2

    Planners graded model briefs but wrote none, so the paper has no human baseline.

    Source: §6 · p. 11

Abstract

from arXiv · show

How well can a language model turn a large, incomplete and partly contradictory body of documents into a plan when no single correct answer exists? We introduce Mission Analysis (MA)-Bench, built on mission analysis, the first step of military operational planning, in which a staff reads the full situation and states the problem before anyone proposes a solution. Its 10 expert-built scenarios pair synthetic packets of ten to twelve documents with 1,257 expert-written pass/fail requirements, each containing information on their source, question type, and importance level. A reasoning-model judge grades each brief against the requirements without seeing the documents; it grades like a fifth credentialed planner, leaving a four-planner panel's agreement unchanged (Fleiss $\kappa$ 0.42 to 0.41), and a judge from a second provider reproduces the model ranking (Spearman 0.94). Across twenty frontier and open-weights models the best gets 62% of the rubric items right. Models are strongest on facts stated in the documents (58%) and weakest on judgments that must be inferred (33%), and six failures in ten are omissions rather than errors. MA-Bench grades the determinations, not the prose. We release the packets, rubrics, prompts, scorer and every graded brief with item-level verdicts.

1 Introduction

Operational planning lacks a unique reference answer, and fluent briefs can still omit critical determinations or mishandle evidence. MA-Bench evaluates mission-analysis briefs against expert-authored requirements and tests twenty models under a fixed protocol.

1 Introduction

  • Operational planning has no verifiable outcome or unique reference answer, so evaluating a brief requires checking the determinations it makes and omits.A brief can be plausible yet miss a critical determination, overlook conflicting evidence, or turn uncertain assessment into fact.Source: §1 · pp. 1–2
  • MA-Bench targets judgment under evidence: identifying the determinations a situation demands and whether the system makes them.Mission analysis transforms higher direction and operational information into a defined problem before courses of action are developed.Source: §1 · p. 2
  • MA-Bench grades briefs against 1,257 expert-authored atomic requirements, distinguishing extraction, synthesis, and inference while scoring omissions and detrimental actions.The protocol grades the brief rather than comparing it with a reference text, and can evaluate single-prompt or multi-step systems.Source: §1 · p. 2
  • Twenty models are evaluated under one fixed protocol, with Figure 1 showing weighted scores in the benchmark condition.The figure covers all twenty systems with a 3,000-word cap and medium effort; whiskers show 95% bootstrap intervals over ten scenarios.Source: Figure 1 · p. 2, §1 · p. 3

2 Mission Analysis as an Evaluation Problem

Mission analysis is the planning step in which a staff defines the problem before proposing solutions. The evaluation problem is to assess those judgments from a large, heterogeneous packet without relying on a unique reference brief.

2 Mission Analysis as an Evaluation Problem

  • Mission analysis requires a staff to state the problem before anyone proposes a solution, using direction, intelligence, available forces, and commander guidance.Its determinations include tasks, constraints, facts and assumptions, information needs, risks, and a restated mission.Source: §2 · p. 3
  • For researchers, mission analysis is a long-context reasoning task with expert ground truth but no reference answer across incomplete and partly conflicting documents.For practitioners, it is a measured instrument for a task language models are already being asked to perform.Source: §2 · p. 3
  • Evaluation checks each required determination in an artifact rather than its similarity to a reference, scoring both expected content and detrimental moves.Requirements specify the determination, its evidentiary or doctrinal basis, the reasoning needed, and the consequence of omission or an incorrect conclusion.Source: §2 · pp. 3–4

3 Related Work

MA-Bench connects five literatures but addresses a gap: none grades a doctrinal staff product whose value lies in its defensibility.

3 Related Work

  • MA-Bench intersects agent and occupation-scale benchmarks, planning evaluations, model-based grading, expert-authored rubrics, and language models in military decision-making.The cited literatures include agent and occupation-scale benchmarks [32] [38] [58], planning evaluations [60], model-based grading [11], expert-authored rubrics [1] [3] [17] [52], and military decision-making.Source: §3 · p. 4
  • None of these literatures grades a doctrinal staff product whose value lies in its defensibility.JP 5-0 supplies the standard against which MA-Bench grades.Source: §3 · p. 4
  • The evaluation differs from prior planning work that checks a verifiable end state by grading the defensibility of an operational staff product.The passage distinguishes planning evaluations with checkable end states [60] from MA-Bench's doctrinal standard, JP 5-0.Source: §3 · p. 4

4 Benchmark Design

MA-Bench evaluates mission analysis using expert-validated scenario packets, atomic requirements, and a rubric-based grading protocol. Its design captures extraction, synthesis, inference, omissions, and detrimental actions.

4.1 Scenario packets

  • Each scenario combines an input packet modeled on staff materials, an expert-authored rubric, and grading by a calibrated model judge.Packets are synthetic, expert-validated crises with real geography and actors, spanning ten to twelve documents and 36k to 243k characters.Source: §4 · p. 4, §4.1 · p. 4
  • Packets are deliberately incomplete and may contain unconfirmed reports or figures from different sources that do not reconcile.Documents downstream of completed mission analysis are withheld; recognizing packet limits is itself scored.Source: §4.1 · p. 5

4.2 Rubrics

  • Each scenario has 74 to 170 expert-authored, frozen rubric items, 1,257 total, designed to be atomic, binary, observable, evidence-grounded, self-contained, and non-duplicative.Items are assigned to mission analysis steps and grounding tiers: stated facts, cross-document synthesis, or inferred planning judgments.Source: §4.2 · pp. 5–6
  • Severity labels assign positive weights to desirable determinations and negative weights to detrimental actions, with larger magnitudes for critical items.Critically important and critically detrimental items carry weights of +9 and −9, respectively.Source: §4.2 · pp. 5–6

4.3 Grading and scoring

  • A frozen reasoning-model judge grades each brief against the rubric without seeing the packet, and an unclear requirement is scored as failed.The severity-weighted score rewards passed positive items and charges for wrong moves; those charges use the same units as omissions.Source: §4.3 · p. 6

5 Experiments

Across the benchmark, model performance varies, but no system meets all rubric requirements. Models do better on stated information than inferred judgments, while omissions dominate failures; judge comparisons broadly preserve model rankings.

5.1 Setup

  • The benchmark evaluates sixteen models using a shared prompt, full packet, 3,000-word cap, and at least three briefs per scenario.The 1,020 graded briefs are analyzed by scenario, with seeds averaged within each scenario.Source: §5.1 · p. 7

5.2 Main results

  • 62% of rubric items are correct for the best model, Muse Spark 1.3, whose weighted score is 0.574; the sixteen models span 0.30 to 0.57.No model approaches the ceiling; intervals of about ±0.08 mean adjacent models are not separated, and every brief has three to five flagged wrong moves.Source: §5.2 · p. 7

5.3 Where models fail

  • Positive-item pass rates are 58.1% for Tier 1, 49.0% for Tier 2, and 33.4% for Tier 3, with the gradient holding for every model.Tier 3 accounts for 36% of the rubric and the largest share of missing credit; the tier pass rates are pooled across the benchmark condition.Source: §5.3 · pp. 7–8, Figure 3 · p. 8
  • 60.2% of failed positive items are classified as outright omissions, while 1.6% are incorrect or unsupported content.A flagged mistake appears on 25.9% of detrimental and 28.3% of critically detrimental opportunities.Source: §5.3 · p. 8

5.4 Length, reasoning effort and model scale

  • Raw brief length correlates only 0.26 with score across the twenty systems in the capped condition, and the longest writer ranks fourteenth.More reasoning effort shifts scores by at most 0.08 across five models, with no consistent sign, while stronger family members win in three of four comparisons.Source: §5.4 · p. 9

5.5 Sensitivity to the judge

  • The alternate judge agrees with the production judge on 84.0% of item verdicts, and their model means correlate at Spearman 0.94.Across three judges on a common set of 216 briefs, agreement ranges from 80.0% to 85.1% and model rankings remain stable.Source: §5.5 · pp. 9–10, Table 1 · p. 9

5.6 Agreement with experts

  • The production judge agrees with the four-planner majority on 78.0% of verdicts, with κ = 0.54, and adding it as a fifth rater leaves panel Fleiss κ nearly unchanged.The panel’s Fleiss κ shifts from 0.42 to 0.41 when the judge is included.Source: §5.6 · p. 10, Table 2 · p. 10

6 Discussion and Conclusion

MA-Bench measures whether models make the determinations needed to frame an operational problem, where no verifiable outcome or unique reference answer exists. Its results show substantial inference gaps and omission-heavy failures, while the paper identifies limits to interpreting scores as mission-analysis quality.

6 Discussion and Conclusion

  • The best current model gets 62% of rubric items right; remaining gaps concentrate in inference and forward-looking judgments, with failures more often omissions than errors.The authors note that a higher score still needs holistic expert ratings against packets to establish that it means better mission analysis.Source: §6 · pp. 10–11
  • The judge's agreement with credentialed planners and cross-provider ranking replication support consistent rubric application, but planners wrote no briefs to provide a human baseline.A small set of human-written briefs and holistic expert ratings are proposed to anchor the scale.Source: §6 · p. 11
  • The instrument covers one step of one planning process in one doctrinal tradition, with length caps, lower agreement on detrimental items, and nine content-filter refusals among its limits.Bootstrap intervals of about ±0.08 leave adjacent models indistinguishable, though the table's endpoints are separated.Source: §6 · p. 11

A Extended related work

  • MA-Bench fills a gap across five literatures: no public benchmark grades planning performance on the Joint Planning Process's first step against codified documents using expert-authored atomic rubrics.The paper places it at the intersection of agent benchmarks, planning evaluation, model-based grading, professional-work rubrics, and military decision-making.Source: A Extended related work · p. 17

A.1 Agentic task benchmarks and the fluency bias

Prior benchmarks largely score verifiable task completion, while mission analysis requires a defensible analytical product without one correct answer. This section situates MA-Bench among planning, judge-reliability, rubric-evaluation, military-decision, and benchmark-validity research, and describes the scenario corpus.

A.1 Agentic task benchmarks and the fluency bias

  • Existing agent benchmarks score completed artifacts in stable settings, but MA-Bench targets planning judgments whose value lies in defensibility against doctrine requirements.Related work identifies complementary blind spots in underspecified goals and open-ended professional judgments.Source: §A.1 · p. 17

A.2 Evaluating planning

  • Mission analysis is a structured analytical product with no single correct answer, graded for required determinations that are present, sourced, and doctrinally sound.The section distinguishes this setting from planning benchmarks with verifiable end states.Source: §A.2 · p. 17

A.3 LLM-as-judge: capability and known failure modes

  • MA-Bench's judge is selected by agreement with experts, cites embedded ground truth, and is cross-checked under judges from other providers for self-preference.Malformed or truncated verdicts are rejected and retried because unlisted items count as passes.Source: §A.3 · p. 18

A.4 Rubric-graded evaluation of professional work

  • MA-Bench combines atomic binary criteria with doctrinal citations, extraction/synthesis/inference tiers, and asymmetric severity for omissions and detrimental behavior.The rubric grades what a brief determines, not its prose fluency.Source: §A.4 · p. 18

A.5 Language models in military decision-making

  • Prior military-planning studies examine behavior, simulations, or engagement outcomes, whereas MA-Bench grades mission analysis against codified doctrine.The section notes that mission-analysis requirements are specified in JP 5-0.Source: §A.5 · p. 18

A.6 Benchmark validity

  • Synthetic, expert-validated scenarios keep packet and rubric text off the public internet, while withheld answer documents are quarantined from evaluated systems.The protocol grades the brief alone, allowing systems that produce one to be scored against the same standard.Source: §A.6 · p. 18

B Corpus

  • The corpus spans ten scenarios, with rubric totals of 484 critically important, 536 important, 80 slightly important, 82 detrimental, and 75 critically detrimental items.Across rubric tiers, totals are 400, 406, and 451 for Tiers 1–3.Source: Table 3 · p. 19
  • Every severity category exceeds 5% of the corpus, and each scenario's smallest category exceeds 2%, limiting dependence on any one scenario's mix.Figure 4 displays the five severity categories in clockwise order from critically important to critically detrimental.Source: Figure 4 · p. 20

C Additional rubric items and failure examples

The appendix traces how rubric items distinguish direct extraction, inference, and harmful analytical moves, and explains the scoring scale. These examples show that citations alone do not satisfy a requirement when the brief omits the required determination.

C Additional rubric items and failure examples

  • The rubric also tests whether briefs state that the No-Fly Zone was declared and preserve a forceful-response warning, rather than treating the zone as merely proposed.The example identifies the source as Strategic Context and EXORD paragraph 3.Source: C Additional rubric items and failure… · p. 20
  • A Tier 2 item penalizes treating a moderate-confidence proxy attribution as confirmed fact because the intelligence summary calls it a cross-source assessment, not a finding.The item tests calibrated synthesis rather than copying an uncertain assessment as certainty.Source: C Additional rubric items and failure… · p. 20
  • A commander’s critical-information requirement can name force availability yet fail to connect it to branch readiness or the National Command Authority decision it should inform.The example concerns the lowest-scoring of twelve steps and a Tier 3 requirement.Source: C Additional rubric items and failure… · pp. 20–21
  • A Tier 1 known-facts item can fail when a brief cites the source repeatedly but never states its partner information, even though another brief shows the item is answerable within the word cap.The example makes explicit that source citation is not a substitute for stating the required fact.Source: C Additional rubric items and failure… · p. 21
  • A cited paragraph can still fail the benchmark when a brief never reproduces the required fact; a reader checking citations might pass it, but the rubric does not.This trace illustrates why item-level verdicts evaluate determinations rather than citation presence alone.Source: C Additional rubric items and failure… · p. 21
  • The judge marks the Tier 1 item failed because generic references to partners omit the named partners, host nations, and UNHCR required by the pass condition.The brief mentions partners in intent, tasks, assumptions, and CCIRs, but its known-facts section names none.Source: C Additional rubric items and failure… · p. 21

D Scoring details

  • The signed severity weights normalize a perfect brief to 1 and charge wrong moves in the same units as omissions; weighted and unweighted scores rank systems nearly identically.The two rankings differ only by adjacent-model swaps, with Spearman 0.99.Source: D Scoring details · p. 22
  • Reporting-time clamping or affine transformation leaves the unclamped score and item verdicts available, while detrimental-item counts remain separately reported.Separate counts preserve visibility of a critically detrimental move that may matter more than its numerical weight conveys.Source: D Scoring details · p. 22

E Main results table

Table 4 presents the benchmark-condition model results across weighted and unweighted measures, including a cross-judge score. Table 5 describes brief lengths, truncations, and provider content-filter refusals.

E Main results table

  • Table 4 reports weighted scores with scenario-bootstrap intervals, the same score under a Gemini cross-judge, and unweighted Correct, Passed, and Wrong measures.The benchmark condition uses the production judge, a 3,000-word cap, and medium reasoning effort; the table is sorted by weighted score.Source: Table 4 · p. 22, E Main results table · p. 22

F Generation statistics

  • Table 5 records raw brief word counts, the number truncated at 3,000 words, and outputs ending in provider content filters.It also marks runs using provider-default reasoning effort when the parameter was unavailable or unset.Source: F Generation statistics · p. 23, Table 5 · p. 23

MODEL CONDITION BRIEFS WORDS, MEAN (MIN TO MAX) TRUNCATED REFUSALS

  • The generation statistics distinguish raw output length from text cut at the cap before grading, and count content-filter refusals as outputs.The table's default marker identifies runs at provider-default reasoning effort.Source: Table 5 · p. 23, F Generation statistics · p. 23

G Decomposition by mission analysis step

This block decomposes benchmark results by mission-analysis step and failure mode, then describes the authors’ use of language models in the research process.

G Decomposition by mission analysis step

H Failure modes by model

I Use of large language models

  • An LLM-based coding assistant helped draft analysis and plotting code and edit prose; the authors say they verified every number and claim.The graded briefs and judge are also language models, whose agreement with human experts is evaluated separately.Source: I Use of large language models · p. 26

J Length and effort ablations

Without the word cap, model ranking largely persists while scores generally rise; under the cap, results vary across models and reasoning-effort settings.

J Length and effort ablations

  • Reasoning-effort changes were usually modest: Claude Sonnet 5 loses 0.056 at low effort and gains 0.023 at high effort versus medium.GPT-5.4 gains 0.045 at high effort; three other tested models move by less than 0.03 in either direction.Source: J Length and effort ablations · p. 26
  • Longer output alone does not guarantee higher capped scores: Gemini 3.8 Flash writes the longest capped briefs (5,200 words before truncation) and scores 0.371, while Qwen3.8-Max scores 0.410 without reaching the cap.Source: J Length and effort ablations · p. 26
  • Removing the 3,000-word cap raises GPT-6 Astra from 0.485 to 0.562 (+0.078) and GPT-5.4 from 0.370 to 0.493 (+0.123).The paired difference for Astra has a 95% interval of +0.024 to +0.125; the reported Astra comparison uses three times the length.Source: J Length and effort ablations · p. 26, Figure 7 · p. 27, Table 10 · p. 28
  • Without the cap, the average model gains 0.098, eighteen of twenty score higher, and Claude Opus 5 rises from third to first at 0.699.The uncapped ranking has a Spearman correlation of 0.95 with the benchmark condition.Source: J Length and effort ablations · p. 27, Figure 8 · p. 28

K Scores under the alternate judges

Alternate judges preserve the model ordering despite a leniency offset, with additional checks assessing agreement against expert verdicts. The prompt materials specify both the mission-analysis brief and the rubric-based grading procedure.

K Scores under the alternate judges

  • A repeat run of the frozen judge moved κ by 0.004, and its κ was 0.51 on the 137 expert-study items unchanged by the final rubric revision.The expert study covers 190 items; Table 11 compares candidate judges against expert majority verdicts.Source: K Scores under the alternate judges · p. 29, Table 11 · p. 29
  • Alternate judges are more lenient by a roughly constant offset but preserve model ordering, with Spearman correlations of 0.92 and 0.94 on the common brief set.Figure 9 reports 0.94 for Gemini and 0.94 for Opus on their respective sets.Source: Table 12 · p. 29, Figure 9 · p. 30

L Prompts

  • The task prompt asks for a complete brief spanning twelve mission-analysis areas, from higher-headquarters intent and tasks to risks, forces, and objectives.It requires source attribution and prohibits unsupported information, with gaps stated explicitly or recorded as assumptions.Source: L Prompts · p. 31
  • The task prompt imposes a 3,000-word limit and instructs models to cover every required section before adding depth to any one.Text beyond the limit is discarded before grading.Source: L Prompts · p. 31
  • The judge grades each response against rubric items containing their own ground truth and pass/fail conditions, without receiving the source packet.Items are judged independently, and absent required content fails; the prompt requires failed items only in strict JSON.Source: L Prompts · pp. 31–32

Related papers

Loading 2610.19138v1…