Source-linked AI summary
WorldCupArena: Fine-Grained Evaluation of Language Models and Deep-Research Agents on Football Forecasting
Zhaokai Wang, Tianlin Gui, Jiayuan Rao, Shangzhe Di, Yihong Tang, Dingli Liang
TL;DR
Football forecasting benchmarks rarely test timely, evidence-grounded predictions across multiple match details. WorldCupArena evaluates these forecasts before kickoff and finds that detailed scoring reveals differences that winner accuracy misses, while search does not consistently improve match prediction.
Problem
Existing forecasting benchmarks rarely test current evidence use across several related football predictions beyond winners or scores.
Method
WorldCupArena evaluates pre-match forecasts across results, scores, players, events, statistics, and competition outcomes using shared evidence or self-directed search.
Results
Across 104 matches, detailed scoring exposed differences that result accuracy missed, while search did not consistently help and closer misses drove the best model’s larger Scoreline gain.
Takeaways & Limitations
Evaluating football forecasters across detailed outputs distinguishes systems more clearly than winner accuracy alone.
Takeaways & Limitations
The benchmark compares complete systems and cannot identify whether differences arise from the model, search, probability estimates, or output formatting.
Abstract
from arXiv · showhide
Predicting a football match before kickoff requires more than knowing past results: a model must use changing information and make a clear prediction before the answer is available. We present WorldCupArena, a dynamic benchmark for language models and deep-research agents. The 2026 FIFA World Cup is its first evaluation, and the same process can be reused for future leagues and cups. Before each match, a model either receives a common evidence package or searches for information itself. It predicts the result and score, likely players and events, match statistics, and the outcome of the competition. After the match, these predictions are compared with the recorded result. We report result accuracy, exact-score accuracy, and a scoreline score that gives some credit when a predicted score is close but not exact, together with scores for the other prediction tasks. Across 104 matches and 13 systems, models with similar result accuracy differ more clearly on detailed predictions. Compared with betting-market and human-fan baselines, the best system shows only small gains in result and exact-score accuracy, but a clearer gain in Scoreline. New schedules can be added as they begin, allowing the benchmark to evaluate future models without using outcomes that are already known. Code, prompts, predictions, and evaluation scripts are open sourced at https://github.com/wzk1015/WorldCupArena.
1 Introduction
WorldCupArena is a reusable, dynamic football-forecasting benchmark that evaluates models before matches using shared evidence or self-directed search, then scores detailed predictions against official records. Its first evaluation covers 13 systems across 104 World Cup matches and shows that result accuracy alone misses important differences in scorelines, players, events, and statistics.
- Motivation: WorldCupArena addresses the gap between benchmarks with known answers and forecasting tasks requiring useful predictions before events occur.Existing dynamic and forecasting benchmarks often focus on a single answer or coarse event outcome, providing limited evidence about current-information retrieval and detailed prediction quality.
- Motivation: Football supports this evaluation because pre-kickoff information changes and completed matches provide scores, lineups, goals, cards, statistics, and competition paths.Evaluating these outputs together tests whether plausible match predictions are supported by plausible details.
- Benchmark design: Twenty-four hours before each match, models receive a common evidence package or search for sources, submit match and competition predictions, and are scored against the official record afterward.The procedure can be reused for new leagues or cups and future model releases.
- Evaluation and findings: 13 systems were evaluated across all 104 World Cup matches, revealing that similar winner accuracy can conceal differences in scorelines, players, events, and statistics.The evaluation also reports comparisons between shared-evidence models and self-searching agents, while scoring exact answers and close misses separately.
- Evaluation and findings: Adding search produced no reliable gain for match prediction, and several models failed on the same upsets.The benchmark also compares systems with betting-market and human-fan baselines.
2 Related Work
WorldCupArena extends prior sports and forecasting benchmarks by evaluating general-purpose models on unrevealed football matches before kickoff. It combines self-searched versus prepared evidence with predictions spanning match details and whole-competition outcomes.
- LLMs and deep-research agents: Existing deep-research benchmarks test web browsing, tool use, planning, and long investigations, whereas WorldCupArena asks whether additional searching improves predictions for an upcoming match.It compares prepared evidence (S1) with self-searched evidence (S2), scoring both after the match.
- Evaluation scope: WorldCupArena broadens forecasting from result and score to players, events, statistics, tactics, and whole-competition outcomes.Its five evaluation layers increase prediction scope and structural richness from a single match result and score to the outcome of an entire competition.
- Sports understanding and forecasting: Most sports-understanding benchmarks evaluate games after they have been played, while WorldCupArena preserves a before-match setting for general-purpose football forecasting.Prior work covers video actions, broadcast-commentary links, and sports reasoning; football forecasting has often focused on narrower result prediction or tournament simulation.
- Temporal prediction benchmarks: WorldCupArena follows temporal benchmarks that record predictions before outcomes are known and score them afterward, replacing completed competitions with future leagues or cups.This design prevents answers from becoming stale and allows new match sets to be added when one competition ends.
- Statistical sports forecasting: Classical football forecasting uses goal models, team-strength ratings, and uncertainty estimates, while WorldCupArena includes betting odds as a difficult-to-beat market baseline.Prior methods include Poisson, form-adjusted, low-scoring-dependence, Bayesian, and Elo approaches.
3 Benchmark Construction
WorldCupArena uses a fixed pre-match-to-post-match workflow that collects time-matched evidence, records forecasts, and scores them against official records. It standardizes predictions across five layers and supports both shared-evidence and agent-search settings, with configurable procedures for different competitions.
- Benchmark workflow: The benchmark registers each fixture, collects information until the deadline, saves forecasts, and scores them after the official match record is available.Changing competitions alters schedules and rules, but not these four steps or the match-level metrics.
- Prediction targets: Models predict match results and scores, players, events, statistics, and competition outcomes.Competition-level targets depend on the competition’s rules, such as league standings or cup advancement.
- Competition procedure: For the 2026 World Cup, models forecast 72 group matches, derive group tables and eight best third-place teams, then forecast 32 knockout matches.The procedure can be configured for future domestic leagues, continental cups, or World Cups.
- Temporal protocol: Evidence and predictions are saved 24 hours before kickoff, with web-search URLs and timestamps retained for agents.Forecasts containing post-deadline information or revealing the final result are excluded.
- Evidence settings: S1 gives every model the same pre-match package, whereas S2 gives agents only the fixture and requires them to search for evidence.The comparison reflects complete system products rather than a controlled comparison of model architectures.
4 Evaluation Method
WorldCupArena evaluates forecasts with result, scoreline, player, event, statistic, and competition metrics, using weighted aggregation that accounts for missing truth and unequal match coverage. Scores are calibrated for display without changing rankings, while leaderboard eligibility requires sufficient match coverage.
- Result and scoreline metrics: Result accuracy measures correct home/draw/away outcomes, exact-score accuracy requires both goal totals, and Scoreline rewards close misses.All metrics are oriented so larger values are better.
- Player, event, and statistic metrics: Player, event, and statistic predictions are scored using ranked player matches, nearest-event matching with penalties, and sMAPE percentage error for counts.These metrics cover lineups, scorers, goals, cards, substitutions, shots, and corners.
- Availability-aware aggregation: Unavailable truth is excluded from weighted means rather than scored as zero, while confirmed non-events remain valid evidence for predicting that nothing happened.This distinguishes “none happened” from “no data.”
- Leaderboard aggregation: Leaderboard scores average raw match scores before one display transformation, preserve model ordering, and require coverage of at least 52 matches in this paper.Failed calls and invalid responses reduce coverage rather than count as predictions.
- Competition metrics: Competition forecasts receive credit for correct group ordering, advancing teams, exact round matchups, and champion selection, with later rounds weighted more heavily.Round weights increase from 1 for R32 to 16 for the final.
5 Experiments
The experiments evaluate 13 systems across all 104 World Cup matches and compare detailed forecasting metrics with betting-market and human-fan baselines. Results show that fine-grained scoreline and task-specific measures separate systems more clearly than result accuracy, while search does not consistently improve forecasts.
- Baselines: 68.3% result accuracy, 16.3% exact-score accuracy, and 53.35 Scoreline were achieved by the BetVictor baseline across all 104 matches.The human-fan baseline reached 69.7% result accuracy, 15.6% exact-score accuracy, and 53.4 Scoreline over 94 matches; Polymarket reached 65.4% and 8.7% on result and strict exact-score accuracy.
- Metric and task differentiation: 10.3–17.2% exact-score accuracy across systems shows that detailed metrics distinguish models that result accuracy alone cannot separate.Claude Opus 4.7 (Thinking) led T1 and T2, its search version led T3, and Gemini Deep Research led T4; no system led every task.
- Baseline comparison: 70.7% result accuracy was achieved by Claude Opus 4.7 (Thinking + Search), only 1.0 point above the human-fan baseline but 15.14 Scoreline points above BetVictor.Its exact-score accuracy was 17.2%, 0.9 points above BetVictor.
- Search ablation: Search reduced overall score by −0.26 for Claude, −4.26 for GPT-5.4, and −1.03 for Gemini on shared matches.These comparisons involve commercial products rather than identical base models and therefore do not isolate causation.
- Competition-level forecasting: 83.3–88.9 group scores contrasted with 15.3–71.0 pairing scores, showing that models predicted broad group order more consistently than exact knockout routes.The two Claude models led competition T5 with 88.3 and 87.4, aided by predicting the Spain–Argentina final and champion Spain.
6 Discussion
The discussion explains how WorldCupArena should be interpreted, highlighting metric complementarity, subgroup-specific weaknesses, and shared failures. It also describes limitations in attributing system performance and preserving temporal validity, while outlining extensions to inputs and future competitions.
- Interpretation and limitations: Each benchmark score evaluates the complete forecasting system, so differences cannot be attributed to the language model, search tool, probability estimates, or formatting alone.The benchmark compares systems as users receive them and cannot establish which internal component caused a difference.
- Interpretation and limitations: The benchmark reports multiple metrics because result accuracy, Brier score, exact-score accuracy, Scoreline, player, event, and statistic scores capture different forecasting properties.Brier score distinguishes cautious from confident predictions and penalizes misplaced confidence.
- Shared failures: All 13 systems predicted Spain to win Cape Verde, but the match ended 0–0; all 12 available systems also chose France before England won the playoff 6–4.The examples show that systems can follow the same favorite and fail together when matches develop unusually.
- Reproducibility: Prediction times, sources, and model identifiers support later checking, but unreliable webpage dates and silently updated model versions require manual validation for important search-based comparisons.The recorded provenance is useful but not perfect.
- Future development: Future versions may add video, press conferences, tracking data, or formation graphics as defined input settings, while new models will be evaluated on future leagues and cups with predictions saved before kickoff.The completed 2026 World Cup remains a fixed record, and the benchmark grows through new matches rather than retrospective predictions.
7 Conclusion
WorldCupArena evaluates pre-match football forecasting by scoring winners, scores, players, events, statistics, and competition outcomes. Across 104 World Cup matches, detailed scoring exposed differences that result accuracy alone missed, while web search was not consistently helpful and models sometimes selected the same favorite.
- Benchmark scope: WorldCupArena evaluates football predictions made before match results are known, covering winners, scores, players, events, statistics, and competition outcomes.The benchmark is designed for large language models and agents.
- Evaluation findings: 104 World Cup matches showed that detailed scoring reveals differences that result accuracy alone misses.The conclusion emphasizes the value of evaluating prediction details beyond the final result.
- Evaluation findings: Web search does not consistently improve match prediction, and models sometimes fail together by choosing the same favorite.These findings indicate limits in both search-based assistance and model diversity.
A Implementation Details · A.1 Artifact Layout and Lifecycle
The implementation organizes each fixture as a versioned artifact lifecycle with idempotent processing, separated live updates, and post-match truth and grading outputs. Scheduling uses time windows, freezes predictions 24 hours before kickoff, and supports no-op retries and scoring-version refreshes.
- A.1 Artifact Layout and Lifecycle: Fixtures pass through five idempotent phases: ingest, populate, lock-and-predict, completion truth ingest, and grading/site building.Population attaches squads, form, news, statistics, and odds; lock-and-predict freezes the snapshot hash and writes one record per model and setting.
- A.1 Artifact Layout and Lifecycle: Live updates are stored separately and never enter pre-match grading.This preserves the frozen pre-match evidence used for evaluation while allowing match-time state to be polled.
- A.1 Artifact Layout and Lifecycle: Truth ingest creates a normalized truth artifact after completion, while grading writes versioned result files and the site builder emits static language-specific payloads.These outputs complete the post-match artifact lifecycle.
- A.1 Artifact Layout and Lifecycle: Up to seven days before kickoff, the default scheduler ingests fixtures and populates evidence during the day before prediction lock.The scheduler uses windows rather than exact instants.
- A.1 Artifact Layout and Lifecycle: Predictions are frozen 24 hours before kickoff, live state is polled around the match, and truth is graded after completion.The schedule separates pre-match prediction, match-time monitoring, and post-match evaluation.
- A.1 Artifact Layout and Lifecycle: Repeated scheduler ticks are no-ops when current artifacts already exist, and changed scoring versions refresh old result files from saved predictions and truth.This behavior makes reruns idempotent while allowing prior results to be regenerated under updated scoring.
A.2 Full-Tournament Generation
Full-tournament prediction uses two model calls: one for group matches and one for fixed knockout fixtures, with standings and tournament outcomes computed from the saved match list. Evaluation supports leaderboard reporting and independent recomputation against dated tournament truth without hindsight changes.
- Prediction pipeline: Two model calls cover the tournament: group matches first, then matches 73–104 using the model’s standings and fixed official-format knockout fixtures.Code computes group standings and resolves best third-place teams into Round-of-32 slots before the second call.
- Prediction pipeline: Champion, runner-up, third place, and top scorers are derived from the saved match list rather than accepted as unconstrained model outputs.The saved match list provides the basis for these tournament-level outcomes.
- Evaluation and reproducibility: Evaluation records result and exact-score hit counts, denominators, mean scoreline quality, and provisional tournament results while supporting independent recomputation.This redundancy separates easy-to-consume leaderboard reporting from a research artifact that can be recomputed independently.
- Evaluation and reproducibility: Dated tournament truth enables evaluation through semifinal, final, and completed-tournament stages without hindsight changes to the prediction.The truth is kept in a dated configuration file.
A.3 Prediction Record Contract · A.4 Failure States and Recovery · A.5 Current Implementation Boundaries
The implementation preserves both auditable execution metadata and schema-governed forecasts, validates and repairs predictions, and distinguishes recoverable failures from benchmark-integrity violations. Its current truth, leakage, and entity-normalization limitations affect fine-grained layers more than final scores.
- A.3 Prediction Record Contract: Prediction records store execution metadata separately from the parsed, schema-governed football forecast.Metadata includes model identity, timing, token and cost estimates, tool calls, snapshot hash, raw response, validation status, and errors.
- A.3 Prediction Record Contract: Semantic validation checks probability consistency, result-score agreement, 11-player lineups, and home-away statistic completeness after JSON Schema validation.Large probability-normalization errors trigger repair, whose prompts quote failed constraints and request a complete replacement object.
- A.3 Prediction Record Contract: Rejected outputs and repair attempts remain recorded, enabling format reliability to be evaluated as a model capability.
- A.4 Failure States and Recovery: The pipeline labels runs as skipped, failed, invalid, leaked, or successfully graded, excluding only leaked runs for benchmark integrity.Skipped runs reuse current artifacts; failed runs capture transport or provider errors; invalid runs violate the contract after repair.
- A.4 Failure States and Recovery: Recovery is artifact-driven: failed models can be retried independently, truth-ingest failures are deferred, and new scoring versions refresh stored results.Site generation publishes complete payloads only after in-memory construction succeeds, reducing mixed-version metric fields.
- A.5 Current Implementation Boundaries: The truth adapter mainly relies on one football data provider and lacks complete second-source cross-checking for player and event fields.Search publication times require periodic manual leakage review, while aliases and transliterations use pragmatic rather than global normalization.