Source-linked AI summary
CivBench: A Long-Horizon Benchmark for Tool-Mediated Agents in Civilization VI
Austin Tudor David Andrews, Liam Wilkinson, Jamie Heagerty, Harry Coppock, Jakob Nicolaus Foerster, Rui Ponte Costa
TL;DR
CivBench addresses limited evidence about how agents sustain monitoring and execute plans during long-horizon, tool-mediated interaction. It provides an MCP-based Civilization VI benchmark with controlled observability and trace-level metrics, revealing under-monitoring of queryable strategic state and frequent gaps between stated commitments and subsequent actions. The authors frame these as pilot, protocol-bound findings rather than a model ranking.
Problem
Existing evaluations provide limited visibility into how reasoning, tool use, and planning interact over long horizons, especially whether agents maintain relevant state awareness and execute stated plans.
Method
CivBench connects language-model agents to Civilization VI through MCP, requires explicit retrieval of non-local state, and measures monitoring and commitment execution from interaction traces.
Results
The study observes two measurable patterns across pilot runs: under-monitoring of strategically relevant queryable state and failure to execute concrete near-term commitments stated in planning reflections.
Takeaways & Limitations
CivBench makes long-horizon monitoring and plan-execution behaviours observable through its MCP interface, narration layer, structured diary, and tool-call logs.
Takeaways & Limitations
The findings are constrained by a shared playbook and a 23-game pilot dataset, so they should not be interpreted as stable model rankings or general deficiencies in strategic reasoning.
Abstract
from arXiv · showhide
We present CivBench, an open-source benchmark for evaluating language model agents in long-horizon, tool-mediated environments through the Model Context Protocol (MCP). A single episode spans 300+ turns and produces thousands of tool calls over a large action space, requiring sustained planning, state monitoring, and execution under partial observability. The environment exposes 76 MCP tools and a narration layer that converts visual game state into structured text. We use CivBench to characterise agent behaviour across four model families in 23 admissible runs. The sample is a pilot, not a model ranking: aggregate outcomes do not reliably discriminate models at this scale. Instead, we introduce two interface-level metrics that the environment makes measurable: Proactive Monitoring Rate (PMR), capturing whether agents actively query latent strategic state, and RAG@10, capturing whether commitments stated in structured planning reflections are executed within ten subsequent turns. Across runs we observe two consistent patterns under a shared playbook protocol. Agents under-monitor strategically relevant state that is available but requires explicit querying: despite playbook guidance to query victory progress every 20 turns, agents do so only every 30 to 75 turns, and in 7 of 20 detectable defeats they failed to query within the 20 turn warning window before game end. Agents also frequently fail to execute near-term commitments stated in their own planning reflections (RAG@10 between 48.2% and 65.8% across models). Both patterns arise despite tool access and explicit guidance, and we interpret them as deviations under instruction rather than absences of capability. We release the environment, scenarios, logs, metrics, and analysis pipeline at https://github.com/lmwilki/civ6-mcp
10 Downing Street
The paper was submitted to the NeurIPS 2026 Evaluations and Datasets Track.
- The paper was submitted to the NeurIPS 2026 Evaluations and Datasets Track.
1 Introduction
CivBench addresses limited visibility into long-horizon agent behaviour by connecting language models to Civilization VI through MCP and measuring monitoring and plan execution from traces. Across 23 pilot runs, it identifies under-monitoring and failures to execute near-term commitments rather than ranking models.
- Long-horizon agents require sustained operation with partial observability, large action spaces, and delayed effects.
- Existing evaluations provide limited visibility into whether agents maintain relevant state awareness or translate stated plans into subsequent actions.
- CivBench connects language-model agents to Civilization VI through MCP, exposing 76 tools and structured narration instead of pixels or custom APIs.
- Its central design separates information availability from retrieval, enabling trace-based measurement of attention allocation and plan execution.
- The benchmark combines fixed scenarios, structured logs, controlled observability, and behavioural metrics including PMR and RAG@K.
- Across 23 runs and four model families, the study measures under-monitoring of latent state and failure to execute near-term commitments as pilot findings, not model rankings.
2 Related Work
CivBench extends prior game-agent and tool-use benchmarks with a production-style MCP interface, controlled observability, and trace-level measures of information retrieval and commitment execution.
- Prior Civilization-based environments use different engines and interfaces, including FreeCiv with Gymnasium-style APIs and alternative game implementations.
- CivBench differs by using MCP, separating state availability from explicit querying, and targeting Civilization VI’s richer mechanics.
- The MCP architecture connects agents to Civilization VI through server-mediated tool calls and structured observations from the narration layer.
- Related work spans interactive environments, Diplomacy, shorter-horizon games, multi-agent frameworks, and component-specific benchmarks.
- Unlike benchmarks focused mainly on task success or final outcomes, CivBench releases full tool-call traces and defines metrics for information retrieval and commitment execution.
3 Environment
The environment uses Civilization VI as a long-horizon, multi-objective testbed and exposes its live state through many MCP tools and a narration layer requiring explicit retrieval. The released artifact supports reproducible trace-level analysis rather than leaderboard ranking.
- Civilization VI: Civilization VI spans 300+ turns, involves seven decision domains, and has six victory conditions requiring competing priorities under resource constraints.
- Civilization VI: CivBench evaluates interaction with live, imperfectly observed game states rather than memorised strategy alone.
- MCP interface: The server exposes 76 MCP tools covering state queries, unit actions, city management, diplomacy, research, governance, religion, trade, and lifecycle operations.
- MCP interface: A typical turn uses 5–15 tool calls, producing thousands over a full game while requiring decisions about which tools to call and when.
- Narration layer: The narration layer returns structured text and only exposes state that the agent explicitly queries.
- Narration layer: This design separates representable information from retrieved information, enabling direct analysis of attention allocation.
- Benchmark artifact: The release includes fixed scenarios, model configurations, playbooks, complete MCP transcripts, recomputable metrics, and a pilot 23-game reference set.
4 Evaluation Framework
CivBench evaluates agents under a shared interaction protocol and uses interface-level metrics to measure monitoring and near-term execution beyond aggregate outcomes. The framework is descriptive, with reproducibility, sampling, and measurement constraints that limit model-comparison claims.
- Experimental design: The shared versioned playbook standardises turn structure, checkpoints, and diary fields while leaving tool selection and prioritisation to each agent.Reported runs use a softer advisory playbook rather than earlier hard triggers.
- Metrics: PMR measures the fraction of non-infrastructure tool calls devoted to proactive strategic monitoring rather than local reaction.Examples include explicit queries for victory progress and diplomacy; PMR is not treated as intrinsically optimal when high.
- Metrics: RAG@10 measures whether concrete commitments stated in planning reflections are executed within ten subsequent turns.It weights executed commitments fully and partial commitments by 0.5, while the fixed horizon limits ambiguity from changing context.
- Metrics: Both PMR and RAG@K are interface-level metrics for monitoring and execution behaviour under tool-mediated interaction.They expose behaviours not visible in aggregate outcomes.
- Experimental design: 23 admissible games across four model families form a descriptive pilot rather than a basis for stable model ranking.The sample is uneven, and between-model differences are intended as descriptive.
- Scope and limitations: Interpretation is constrained by the shared playbook, absent random or scripted baselines, environment dependencies, and protocol-specific metric definitions.The playbook-free baseline was not viable, and results depend on game configuration, seeds, playbook, monitoring-tool definitions, and planning-text interpretation.
5 Results: Descriptive Behavioural Profiles
Across the pilot, aggregate outcomes do not reliably distinguish model families, whereas tool-use traces reveal a consistent allocation pattern: local actions and ordinary state queries dominate, while strategic monitoring remains sparse.
- Aggregate outcomes: Only 3 of 23 admissible runs were victories, and aggregate tests did not support between-model discrimination.Fisher’s exact test gave p = 0.488, while normalised score differences were weakly discriminating with H = 1.90, p = 0.594.
- Aggregate outcomes: Within-model variance dominates between-model differences for most aggregate measures, with only exploration at T100 showing clear exploratory discriminative power.The ICC analysis found ICC=0.717 for exploration at T100, while final score, city counts, and economic yields were near zero or negative.
- Tool-use profiles: Across models, local actions and state queries dominate tool budgets, while strategic monitoring remains a small fraction of activity throughout the game.This composition makes attention allocation observable in long-horizon traces.
6 Analysis
CivBench frames missed strategic queries and incomplete plan execution as trace-level consequences of requiring agents to actively retrieve relevant state. These behaviours persist despite available tools and guidance, but their interpretation remains bounded by the pilot protocol and measurement procedures.
- 6.1 The Sensorium Effect: The sensorium effect arises because agents observe only the state they explicitly query, unlike humans who passively absorb many signals.Each query consumes time and context, making the issue one of attention allocation rather than missing tools.
- 6.1 The Sensorium Effect: 0.96%–2.13% of non-infrastructure calls comprise aggregate PMR, while victory monitoring accounts for only 0.05–0.29%.Agents made 3.7–10.0 get_victory_progress calls per game and queried victory progress every 30–75 turns despite 20-turn guidance.
- 6.1 The Sensorium Effect: 7 of 20 detectable losses lacked a victory-progress query during the 20-turn warning window before game end.A detectable warning means the query would have exposed a rival victory threat at least 20 turns before the end.
- 6.1 The Sensorium Effect: PMR does not increase toward the endgame, suggesting persistent allocation choices rather than a transient context limitation.The interpretation concerns the observed traces under this protocol.
- Broader implications: Because critical state must be actively retrieved, similar attention-allocation dynamics may arise in other tool-mediated settings with non-passive observability.This is a stated transfer possibility beyond Civilization VI, not a demonstrated result elsewhere.
- 6.2 Reflection–Action Gap: RAG@10 ranges from 48.2% to 65.8% across the three model families with sufficient commitments for analysis.A substantial fraction of commitments were not executed within ten turns, with overlapping bootstrap confidence intervals.
- 6.2 Reflection–Action Gap: RAG@10 measures execution fidelity for concrete self-stated near-term commitments, not complete planning quality.Commitments are extracted from diary text and evaluated under a shared LLM-assisted labelling procedure validated at κ = 0.879.
7 Discussion
CivBench extends outcome evaluation with trace-level measures of attention allocation and plan execution, revealing protocol-dependent gaps in monitoring and follow-through.
- Measurement and interpretation: CivBench exposes attention allocation and commitment tracking through long-horizon tool-call traces rather than aggregate outcomes.Separating state availability from state retrieval makes proactive monitoring and plan execution directly observable.
- Structured reflection: The turn diary both mitigates context loss and enables RAG by recording explicit commitments for comparison with subsequent actions.Because the diary elicits planning, RAG measures execution fidelity under the shared protocol rather than spontaneous planning ability.
- Structured reflection: RAG is confounded by prompted planning, while a playbook-free baseline was impractical because only 21% of pre-harness runs reached a natural conclusion.The authors therefore treat RAG as execution fidelity under elicitation, not unprompted planning ability.
- Implications: The study identifies under-monitoring of global state and a reflection-action gap as interface-level failure modes despite available mechanisms and guidance.These observations motivate explicit monitoring schedules and persistent commitment representations.
- Implications: Attention allocation is presented as a distinct capability axis that complements outcome-based evaluation in tool-mediated settings.The proposed direction is to measure what information agents retrieve and how they carry plans forward.
8 Conclusion
CivBench is an open-source MCP benchmark that makes long-horizon agents’ monitoring and plan execution measurable in Civilization VI. Its conclusion emphasizes observable interface-level behaviours rather than model ranking, with reproducible resources released for future evaluation.
- Conclusion: CivBench evaluates language model agents in long-horizon, tool-mediated environments through the Model Context Protocol.The benchmark provides a controlled setting where relevant state must be actively retrieved rather than passively observed.
- Conclusion: The study is not a model ranking; it identifies under-monitoring and reflection-action gaps as measurable interface-level failure modes.Agents often fail to maintain global state or execute near-term commitments even when information and guidance are available.
- Conclusion: CivBench makes these behaviours observable through its MCP interface, narration layer, structured diary, and tool-call logs.This infrastructure links retrieved information and stated plans to subsequent interaction traces.
- Conclusion: The environment, scenarios, logs, metrics, and analysis pipeline are released to support reproducible evaluation.The release is intended to support evaluation and improvement of long-horizon, tool-using agents.
- Environment: Civilization VI combines economic, scientific, cultural, military, diplomatic, spatial, and temporal systems requiring multi-turn decisions.The benchmark’s environment therefore spans diverse state variables and irreversible commitments.
A.4 A Turn in Practice
A representative turn follows an orient–detect–reason–act–reflect loop: the agent queries state, identifies a threat, acts through tools, and records strategic plans and hypotheses. The narration and playbook structure provide actionable context while leaving retrieval and execution decisions to the agent.
- Worked Example: One Complete Turn: The worked example begins with overview and unit queries that expose turn state, rankings, resources, and a nearby barbarian threat.The agent observes Scythia at turn 87, a score deficit to Babylon, and a Saka Horse Archer positioned to respond.
- Worked Example: One Complete Turn: The agent uses a ranged attack against the barbarian warrior, dealing 47 damage and reducing its health from 100 to 53.The action is selected because the horse archer can intercept without taking damage, according to the agent’s reasoning.
- Structured diary: The turn diary records tactical outcomes, strategic standings, concrete next-turn plans, and a hypothesis about Babylon’s diplomacy.The stated plan is to finish the barbarian, move the settler, and found a city on T89.
- Playbook and narration: The interaction loop repeats 5–15 times per turn across 300+ turns, producing a structured trajectory of tool-mediated decisions.Narration selectively flags urgency and presents actionable options such as attack targets and buildable improvements.
- Playbook and narration: The playbook emphasizes explicit querying because agents do not passively receive global game signals.Strategic checkpoints include diplomacy, victory progress, and religion spread every 20 turns.
B Supplementary Results
Supplementary results describe outcome, trajectory, monitoring, tooling, and infrastructure measures across the benchmark. They show weak aggregate model discrimination alongside divergent developmental trajectories and a persistent expansion lag relative to the playbook benchmark.
- City expansion: All models remained below the three-city T50 playbook benchmark: Claude 2.83, GPT 2.83, Gemini 2.67, and Kimi 2.00.The deficit persisted at T100, where the corresponding means were Claude 4.67, GPT 4.67, Gemini 4.00, and Kimi 3.00.
- Supplementary measurements: The supplementary figures cover city count, PMR, tool failure, call latency, harness effects, screenshots, and outcome heatmaps.Their captions define these figures’ measured quantities but do not by themselves establish additional comparative outcomes.
- Trajectories: Score trajectories separate after approximately T50, with early leads persisting and low trajectories failing to recover within the game horizon.Per-turn yield analysis attributes the clearest sustained between-model separation to science output.
E.1 Reproducibility Details
The paper reports open-source code, data, configuration details, and scripts intended to support reproduction, while acknowledging scope, statistical, and environmental limitations. It also states that the reported claims are supported by the experiments and that no theoretical results are claimed.
- Open access and artifacts: The released artifact includes configuration details, raw logs, processed metric tables, plotting scripts, and documentation for reproducing the reported results.It excludes Civilization VI assets, game binaries, and proprietary Firaxis/2K code.
- Reproducibility risks: Reproduction remains sensitive to Civilization VI patches, DLC and operating-system differences, provider-side model updates, and stochastic model sampling.The paper mitigates these risks by pinning scenario seeds, reporting model settings and access dates, logging tool calls, and releasing the analysis pipeline.
- Scope and limitations: The paper states that its abstract and conclusion claims are supported by experimental results, with scope and power limitations discussed separately.The checklist specifically records commercial dependency, cost, playbook confounds, sample size, infrastructure exclusions, non-determinism, and contamination as limitations.
- Theory disclosure: No theoretical results are claimed, so theorem assumptions, formulas, and proofs are not applicable to this work.The paper’s checklist explicitly marks the theory-results question as not applicable.
- Experimental details: Scenario files, model identifiers, endpoints, access dates, harness versions, and full configuration files are reported or released to support experimental reproduction.The evaluation checklist identifies scenario configurations, playbook versions, and reproduction scripts as available.