Source-linked AI summary
FM-Bench: A Benchmark for Long-Horizon Management with Competing Agents
Tianyou Wang, Chongyang Gao, Kezhen Chen, Chen Dong, Yinghao He, Donghan Li, Wangcheng Xu, Hongjiu Zhang, Chi Li
TL;DR
FM-Bench addresses whether language-model agents can sustain effective management over long horizons with hidden information, cumulative consequences, and competing adaptive agents. It evaluates agents running football clubs for 20 years and finds that every frontier model completes the horizon, with claude-fable-5 topping both tracks while managerial behavior—not token spend—best distinguishes performance.
Problem
Evidence remains limited on whether language-model agents can manage organizations over long horizons while facing hidden information, cumulative consequences, and adaptive rivals.
Method
FM-Bench evaluates agents running football clubs for 20 years through hundreds of decisions, deterministic cumulative scoring, competing agents, and six behavioral capability measures.
Results
Every frontier model completes the horizon that blind scripts do not survive, while claude-fable-5 tops both tracks at about 95% of a privileged oracle.
Takeaways & Limitations
Managerial behavior distinguishes models: stronger systems reduce late slow-payoff investment, keep cash invested, and open contract renewals early, while token spend predicts nothing.
Takeaways & Limitations
Because 20-year runs exceed context windows, the benchmark requires a bounded self-authored notebook, making memory curation part of the measured capability.
Abstract
from arXiv · showhide
Language model agents now execute bounded tasks reliably. Whether they can sustain effective decision-making over long horizons, where actions have cumulative consequences and the environment responds to their choices, remains largely unmeasured. FM-Bench (Football Management Benchmark) measures this. An LLM agent runs a football club for 20 in-game years through 26 tools and roughly 340 to 400 decision stops. It drafts a squad on the same budget as every rival, trades players, negotiates contracts, invests in facilities and youth, sets lineups, and answers to a board that can fire it, while a deterministic engine accumulates every year into one final score with no LLM judge or human rater. The solo track plays each of 15 frontier models against a frozen scripted world, and the Arena places the same models plus a scripted anchor in one shared 20-year world; to our knowledge, the first head-to-head evaluation at this scale. We measure six behavioral capabilities behind the score. Across three seeds, all 15 models complete every horizon while the blind scripted baselines die out in most of theirs, and claude-fable-5 tops the solo board on mean score and the Arena, where the title nonetheless rotates among ten models. Neither scale, price, nor vendor predicts the order; the order settles only late in the horizon, and the best first-play human lands only at the bottom of the model board. What separates the models is managerial behavior rather than computation. Higher-scoring models reduce slow-payoff investment near the end, keep cash invested rather than idle, and open renewals well before the deadline, while token spend predicts nothing. No model learns the market's hidden prices from hundreds of rejected bids, and self-managed memory fails in two opposite modes: an archive that only grows or a plan rewritten every season. Code is available at https://github.com/Analogy-AI/fm-bench.
1 Introduction
FM-Bench measures whether language-model agents can manage an organization over long horizons while facing hidden information, cumulative consequences, adaptive rivals, and competing objectives. Its 20-year football environment evaluates models through solo and shared-world tracks, then analyzes the managerial behaviors underlying performance.
- Motivation: FM-Bench targets the gap between short-horizon task benchmarks and management, where agents must sustain an organization against rivals while early decisions reshape later conditions.Management combines hidden information, cumulative consequences, competition, and organizational survival under long-horizon pressure.
- Benchmark design: A run lasts 20 in-game years and roughly 340 to 400 decision stops, ending in one deterministic composite score.The agent uses 26 schema-generated tools to manage a club, including drafting, trading, contracting, investing, lineup selection, and board-facing decisions.
- Evaluation tracks: The solo track evaluates each model against a frozen scripted world, while the Arena places 15 frontier models in one shared 20-year economy to isolate competition.The Arena is presented as the first evaluation of 15 frontier models competing as agents in one persistent world over 20 years.
- Human comparison: Four of six first-play humans failed to complete the track, and the best human finished at the bottom of the model board.Human strengths mirrored model weaknesses, suggesting distinct patterns in human and agent management.
- Behavioral analysis: Six behavioral capabilities explain performance beyond the leaderboard: strong models reduce late slow-payoff investment, keep cash invested, and renew contracts early.The analysis also finds that rejected bids do not reveal true market prices, memory fails as either an ever-growing archive or repeated rewrite, and token spend predicts nothing.
2 Related Work
Existing benchmarks assess bounded tool use, realistic software interaction, or one-shot deliverables, while long-horizon and multi-agent settings introduce extended episodes, economic planning, and competition over time. FM-Bench is positioned against these limitations.
- Bounded tasks: Bounded-task benchmarks measure tool-call selection, realistic software interaction, perceptual difficulty, or one-shot deliverables.Examples include AgentBench, ToolBench, τ-bench, WebArena, OSWorld, SWE-bench, and GAIA.
- Long-horizon evaluation: Long-horizon evaluations stretch episodes to hundreds of steps, but terminal-agent marathons pursue fixed deliverables in stationary worlds while games reward reactive skill.The cited long-horizon settings include terminal-agent marathons, UltraHorizon, and game environments.
- Multi-agent evaluation and arenas: Arena-style rankings and coordination benchmarks study human-preference voting or short-episode cooperation, rather than which agent manages better over time.The passage also identifies Cattle Trade as single-session negotiation ranked by TrueSkill, without an organization to run.
3 FM-Bench
FM-Bench evaluates long-horizon football-club management in a deterministic 20-year world where agents repeatedly inspect and act through a 26-tool interface under hidden information and competing objectives. Its agent-owned memory and season-spanning score make information curation and sustained performance explicit parts of the benchmark.
- Task: Each agent manages one of 16 clubs for 20 in-game years, drafting 25 players under the same fixed budget as every rival before trading, negotiating, investing, and setting matches.Drafts are independent copies, so one club’s selections do not interfere with another’s.
- Interaction loop: The deterministic engine pauses at roughly 340–400 decision stops per completed run, combining 13 scheduled stops with about 6 event-driven stops per season.Agents query state, use action tools, and call advance; ignored offers expire and persistent settings provide only limited autonomy.
- Memory: Every decision stop begins without chat history, so agents must curate future context through auto-context, a queryable archive, an audit log, and an intentional notebook.The notebook is the only memory layer carrying the agent’s plans and reasons, making memory curation a measured capability rather than a harness feature.
- Score: The score counts every season and combines honors, net-worth value added, and squad value through one engine implementation, preventing an endgame sprint from replacing sustained performance.The three channels are log-compressed and uncapped, while calibration constants were tuned on scripted anchors and frozen before scored runs.
4 Experiments and Results
FM-Bench evaluates long-horizon football management over 20-year runs and shows that managerial behavior, rather than scale or token expenditure, separates outcomes. The strongest models manage endgame investment, cash, renewals, market pricing, and persistent plans more effectively, while Arena competition destabilizes solo-track dominance.
- Experimental setup and overall results: 20-year episodes span roughly 340–400 decision stops, with identical clubs, 26 tools, and shared budgets, while the oracle averages 95.54 versus 90.94 for claude-fable-5.The oracle sees hidden state but uses the same interface, budgets, and legality checks as agents.
- Behavioral capabilities: Endgame awareness tracks score most closely, with rs = −0.58, as the winner cuts slow-payoff actions from 2.4 to 0.8 per season.The measure is the late-minus-mid shift in facility and academy actions, comparing years 17–20 with years 2–16.
- Behavioral capabilities: Cash discipline also tracks score, with the winner at 80% idle cash against a field median of 90%, while claude-haiku-4.5 averages 196%.Total facility investment is uncorrelated, indicating that idle cash—not spending volume—drives this signal.
- Behavioral capabilities: The winner opens renewals a median of 18 months before expiry, versus 10 months for claude-opus-4.8 and 11 months for claude-haiku-4.5.Only 4% of the winner’s episodes begin inside the final six months, compared with 20% and 21% for those models.
- Behavioral capabilities: Models fail to discover hidden acceptance prices: the field median needs 30 offers per completed signing, versus 9 for claude-fable-5 and 73 for gemini-3.5-flash.The oracle completes signings at 1.0 offers per completed signing; the winner averages 0.5 offers per season solo and 4.6 in the Arena.
- Arena dynamics: Ten models win at least once in the Arena, where the reigning champion retains the title in only 2 of 19 transitions and claude-fable-5 wins 4 titles.By contrast, the four highest-scoring solo runs hold the league title continuously through year 20.
5 Conclusion · pages 8580–8622, 2025. Appendix
FM-Bench evaluates whether language-model agents can manage a football club over twenty in-game years under four job demands, using deterministic grading without an LLM judge. Every frontier model completes the horizon that blind scripts do not survive, while claude-fable-5 tops both tracks at about 95% of a privileged oracle, indicating the benchmark is not saturated.
- 5 Conclusion: FM-Bench tests whether language-model agents can manage rather than merely execute.Each agent is handed a football club for twenty in-game years.
- 5 Conclusion: Agents manage a football club for twenty in-game years under four demands that define the job.The benchmark evaluates sustained decision-making across the full horizon.
- 5 Conclusion: Every run is graded by a deterministic engine with no LLM judge.This makes the final evaluation independent of model-based judging.
- 5 Conclusion: Every frontier model completes the horizon that blind scripts do not survive.The comparison distinguishes model agents from scripted baselines on horizon completion.
- 5 Conclusion: claude-fable-5 tops both tracks.The model is the strongest performer across the benchmark’s two evaluation tracks.
- 5 Conclusion: About 95% of a privileged oracle is reached by the strongest model.This comparison places claude-fable-5 near, but below, the privileged reference.
- 5 Conclusion: FM-Bench separates models without being saturated.The observed gap from the privileged oracle leaves room for differentiation among systems.
A Limitations
FM-Bench’s results are bounded by thin sampling, a deliberately simplified opponent environment, untested transfer beyond football, and correlational behavioral metrics. Runs remain bit-replayable, but stronger inferential, causal, and cross-domain claims require future work.
- Reproducibility: Every published run remains bit-replayable, so these limitations bound interpretation rather than reproducibility.The reproducibility guarantee holds despite sampling, environment, construct-validity, and metric limitations.
- Sampling: Three solo seeds and one Arena world limit inference: nearby solo models may not separate, and Arena orderings within a few points should be read as ties.Aggregating multiple Arena worlds would require a rank-based aggregate over per-world finishes.
- Environment: The 15 scripted opponents enable comparable absolute scores but do not model adaptive rival managers; the Arena is adversarial complement, not a realism claim.The fixed scripted track is deliberate, while the Arena provides the adversarial setting.
- Construct validity: FM-Bench tests long-horizon management in one domain, so whether its rankings transfer to other long-horizon settings remains untested.Its four demands are domain-general design targets, but cross-domain ranking transfer has not been evaluated.
- Behavioral metrics: Behavioral metrics are correlational: warning-exposure is confounded, coefficients over n = 15 are indicative, and causal probes remain future work.The proposed probes include notebook ablation and plan injection.
B Reproducibility · C The Roster
FM-Bench’s reproducibility rests on deterministic execution, extensive regression and replay testing, and metadata-complete result files. Its roster contains 15 frontier LLM seats and one scripted anchor selected under fixed principles, with serving IDs validated before launch.
- B Reproducibility: The engine uses counter-based RNG keyed by (domain, entity, day), a fixed daily phase order, and no wall clock.These design choices make execution deterministic across runs.
- B Reproducibility: A 284-test suite covers golden-hash guards, full-tool-path replay bit-identity, crash-resume equivalence, AST truth-isolation, observation-audit, and calibration regressions.The suite tests both behavioral outputs and implementation safeguards.
- B Reproducibility: Every result file records the engine commit, params hash, score version, and full knob set.These fields preserve the configuration needed to interpret and reproduce results.
- B Reproducibility: Scripted baselines and the full calibration suite run for free on a laptop, with 20 simulated years completed in 76 s singl.The supplied passage reports this runtime exactly as shown.
- C The Roster: Table 5 lists 15 frontier LLM seats plus one scripted anchor, July 2026.The scripted anchor occupies the 16th seat in the roster description.
- C The Roster: The roster fixes four selection principles: within-family tier curves, closed against open frontier, an agentic and tool-use specialist, and a disciplined scripted anchor.These principles remain fixed even as individual model IDs rotate.
- C The Roster: All 15 serving IDs were validated by tool-calling pings before launch.Same-role substitution is allowed in later rounds, while the selection principles remain fixed.
D What the Human Players Reported · E The Scripted Anchors
Human players experienced FM-Bench as a problem of hidden information, delayed consequences, unclear credit assignment, and simultaneous managerial demands, while also discovering adaptive strategies that models did not articulate. Three deterministic scripts provide reproducible anchors: a disciplined heuristic, an inactive floor, and random activity that performs below inactivity.
- D What the Human Players Reported: Players found hidden prices and renewal terms difficult to infer, making mistaken offers feel unrecoverable and encouraging age-based renewal heuristics.One player experienced repeat-pair markup after angering sellers with low offers.
- D What the Human Players Reported: Delayed consequences made early decisions feel harmless until idle cash and insufficient squad depth produced losses seasons later.The reported experience matches the model-side idle-cash pattern described as the strongest negative correlate among models.
- D What the Human Players Reported: Players struggled to identify which decisions caused losing seasons, and declining board confidence left four of six fired.Unexpected wins after injury-forced lineup changes intensified uncertainty about formation, matchups, or noise.
- D What the Human Players Reported: Players described multitasking valuation, squad maintenance, fatigue management, and wage control, with wages above 85% of revenue leaving little margin for dismissal.The operational pressure combined market pricing, substitutions, and financial risk.
- D What the Human Players Reported: Human players articulated durable five-year signing rules, revised squad-depth policy after failure, and used strategies that models did not state or change mid-run.Renewal lead time is the strongest positive model correlate, but no model states the invariant described by one player.
- D What the Human Players Reported: The six self-reports came from one seed and were used for construct-validity evidence and design input, not capability rankings.Their reports independently identified the study’s four demands and highlighted tracking fatigue, contract clocks, and market depth.
- E The Scripted Anchors: Three deterministic blind scripts used the same 26-tool interface and per-stop budgets as models, fixing a reproducible lower anchor at no rerun cost.The scripts do not read hidden state.
- E The Scripted Anchors: The heuristic manages wages, renewals, transfers, academy promotion, fatigue, tactics, and facilities; idle takes no actions, while random activity scores below idle.Random behavior is reproducible through world-seed and stop-ID seeding, and its lower score shows uninformed activity destroys more value than inactivity.
F Opponent Tiers
The solo track’s 15 scripted opponents are one deterministic manager implementation under easy, medium, and hard settings of five competence levers. Frozen cyclic assignment balances tiers across wealth bands, ensuring every model faces the same opponents without altered information or randomness.
- Tier construction: 15 scripted opponents use one manager implementation with five competence levers set to easy, medium, or hard.Medium is the neutral setting and exactly matches the untiered scripted manager; easy and hard turn all five levers down or up.
- Tier construction: Every lever is a deterministic function of state, so tiers add no randomness and do not change the observation path.An easy board is worse at judging players rather than differently informed.
- Tier assignment: A frozen cyclic hard–medium–easy pattern assigns tiers across club slots ranked by initial cash, placing all three tiers in each wealth band.The pattern is a pure function of parameters, so every tested model meets the same 15 opponents in the same slots.
- Calibration: The current calibration tempers easy’s exploitability and gives hard sharper beliefs, proactive buying, and a more ambitious wage budget.Earlier values made easy net stronger than no tiers because mispriced sellers created free bargains and weak first elevens donated points, while hard was too subtle to compensate.
G Arena Data Quality: the Notebook Audit · H The Oracle Policy
The Arena notebook audit found all 15 models coherent and factually grounded, distinguishing weak play from broken play. The privileged oracle used hidden-state information while obeying the same tools, budgets, and legality checks, applying exact-threshold, score-aware management.
- G Arena Data Quality: the Notebook Audit: 15 models remained self-consistent and factually grounded across the first seven Arena game-years; 10 were clean, 5 had minor style or hygiene issues, and none were broken.No model hallucinated its squad, invented finances, or lost track of its league position.
- G Arena Data Quality: the Notebook Audit: Weak-but-coherent play remained distinguishable from broken play, despite verbose crisis vocabulary, boilerplate, doom loops, or an aging-squad decline.The audit assessed every model’s self-authored notebook writes.
- H The Oracle Policy: Academy sniping promoted only truly high-potential players at the last stop before academy terms lapsed, while early facility funding compounded intake quality and growth.Academy development outpaced bench development.
- H The Oracle Policy: Peak-timed sales countered bids at each buyer’s exact ceiling, while zero-waste scheduling re-solved lineups and styles using true player and opponent information.The oracle refused resales within a year and accounted for the style-switch penalty.
- H The Oracle Policy: The policy was score-aware, recycling excess cash into facilities and squad assets, protecting the final three-season window, preventing expirations, and prioritizing survival when board confidence fell.Trading profit and squad growth fed net worth, while emergency control superseded margin discipline.
I Properties of the Composite Score
The composite score is designed to reward completion, monotonic channel improvements, fair-price discipline, and diminishing returns without permitting early-fire arbitrage or runaway channels. Its loose practical ceiling is approximately 145, while the oracle and best blind model reach 95.54 and 90.94.
- Completion and settlement: Completion is always at least as valuable as early settlement: Sfinal is nondecreasing in t and never exceeds Sraw.For positive early-settled scores, the discount approaches 0.8 just before T, whereas completion pays ρ = 1.
- Channel monotonicity: Sraw increases with honors for H ≥0 and value added for all VA, while squad value M increases it weakly and strictly when M ≥0.The honors term is floored below zero, making it constant for negative H.
- Transfer neutrality: Purchasing at or above fair price reduces net worth and cannot improve the VA channel; only bargains below the discount wedge p < λv raise N.Here λ = 0.8, and wage commitments further strengthen the negative net-worth effect.
- Diminishing returns: The score has diminishing returns: each channel is concave on its gain domain, so no single channel can run away despite the composite being uncapped.The first three +100-honors titles are worth 17.7, 8.7, and 5.9 points, respectively.
- Practical score ceiling: A loose impossibility bound sums to ≈145 under extreme inputs, while the oracle scores 95.54 and the best blind model scores 90.94.The bound assumes 20 titles, VA = 10,000M, and M ≈6,000M, conditions the world cannot jointly provide.
J Behavioral Analysis: Definitions, Construction, and Additional Figures · K The Tool Interface and the Decision-Stop Calendar
The appendices define a deterministic replay-based behavioral measurement pipeline and document the 26-tool interface, decision-stop calendar, memory layers, and scale of a 20-year episode. They distinguish reproducible capability metrics from unstable or dropped measures and separate model-controlled behavior from harness settings.
- J Behavioral Analysis: Definitions, Construction, and Additional Figures: All behavioral metrics use bit-identical replay of solo and Arena action logs, with two passes verifying that replay reproduces each run’s published Sfinal before analysis.All 30 replays matched with zer…
- J Behavioral Analysis: Definitions, Construction, and Additional Figures: Endgame awareness measures late-minus-mid long-horizon actions, comparing years 17–20 with years 2–16, where lower values are better.The actions are invest + promote_youth per season.
- J Behavioral Analysis: Definitions, Construction, and Additional Figures: Proactive control measures median renewal lead and separately tracks episodes opened at six months or less, with higher renewal lead rewarded.Renewal episodes collapse offers to the same player within 60 game-days and include own-club players only.
- J Behavioral Analysis: Definitions, Construction, and Additional Figures: Memory curation reconstructs notebooks each season end under exact append and rewrite semantics, using TF–IDF cosine similarity and scoring distance from the empirically best 0.35 band.TF–IDF is used for determinism and zero external dependencies.
- J Behavioral Analysis: Definitions, Construction, and Additional Figures: Warning exposure and youth harvest have per-seed unstable signs, while tactic reversals are dropped as an axis despite a wide raw spread, including zero for the winner versus 31 on average for claude-haiku-4.5.Tactic reversals count only set_tactics calls that change the active formation or style.
- J Behavioral Analysis: Definitions, Construction, and Additional Figures: Season snapshots reproduce years 5, 10, 15, and 20 using cells for position, honors, net worth, squad value, and completed-now composite score.These snapshots complement the trajectory figures.
- K The Tool Interface and the Decision-Stop Calendar: The agent-world interface contains 26 schema-generated tools, with query, negotiation, and notebook operations governed by distinct per-stop budgets and every human UI button mapped one-to-one.Query tools spend the per-stop query budget, negotiation actions spend the negotiation budget, and notebook tools are free.
- K The Tool Interface and the Decision-Stop Calendar: In a 20-year episode, model turns and state-changing actions are distinct counts: turns may only read or deliberate, while one turn may commit several actions, and observed spreads reflect model choices.The appendix also documents the decision-stop calendar, four memory layers, and run scale.
L One Decision Stop, Concretely
A decision stop exposes the current club state, lets the agent issue multiple queries and actions, and ends only when it calls advance. The engine then resolves the day and carries standing intent into the next stop.
- State observation: At Y12-D181, the agent sees cash, league position, and inbox information before acting.The displayed state includes cash_m 240.8, league position 4, and an 18M rival bid for winger P00294.
- Action execution: The agent can lodge a sealed 45M transfer bid for player P00366, which the engine resolves at day end.The tool call uses make_transfer_offer with amount_m 45, and the engine returns ok: true.
- Stop termination: Within one stop, the agent may continue acting, including offering a five-year contract at 15M annually, before advancing.Advance terminates the stop; query budgets limit how many turns the agent can take.
- Between-stop persistence: After advance, the world simulates day by day, while lineups, tactics, standing orders, and in-flight negotiations under TTL remain active between stops.Standing intent persists until a later decision stop changes it.