PaperlayerView exact revision on arXiv

Source-linked AI summary

FutureSim: Replaying World Events to Evaluate Adaptive Agents

Shashwat Goel, Nikhil Chandak, Arvindh Arun, Ameya Prabhu, Steffen Staab, Moritz Hardt, Maksym Andriushchenko, Jonas Geiping

arXiv:2605.15188v1cs.LGcs.AIcs.CL

TL;DR

FutureSim addresses the limited evaluation of long-horizon adaptation in dynamic environments by replaying chronological real-world events for agents to forecast beyond their knowledge cutoff. It provides agents with arriving news and feedback for updating predictions, and reveals substantial room for improvement in frontier-agent adaptation while supporting research on search, memory, reasoning under uncertainty, and harness design.

  • Problem

    Long-horizon adaptation is difficult to evaluate realistically because existing game and simulation benchmarks only provide proxies for environments evolving like the real world.

  • Method

    FutureSim replays timestamped real-world events chronologically, giving agents dated news and feedback on resolved forecasting questions so they can update probability distributions over outcomes.

  • Results

    FutureSim cleanly discriminates frontier-agent capabilities and shows significant room for improvement in test-time adaptation, including in agents’ native harnesses.

  • Takeaways & Limitations

    FutureSim supports studying long-horizon test-time adaptation, search under uncertainty, memory, harness design, and multi-agent dynamics.

  • Takeaways & Limitations

    FutureSim is limited to purely predictive settings because actions that change environment dynamics would require accurately simulating counterfactual worlds.

Abstract

from arXiv · show

AI agents are being increasingly deployed in dynamic, open-ended environments that require adapting to new information as it arrives. To efficiently measure this capability for realistic use-cases, we propose building grounded simulations that replay real-world events in the order they occurred. We build FutureSim, where agents forecast world events beyond their knowledge cutoff while interacting with a chronological replay of the world: real news articles arriving and questions resolving over the simulated period. We evaluate frontier agents in their native harness, testing their ability to predict world events over a three-month period from January to March 2026. FutureSim reveals a clear separation in their capabilities, with the best agent's accuracy being 25%, and many having worse Brier skill score than making no prediction at all. Through careful ablations, we show how FutureSim offers a realistic setting to study emerging research directions like long-horizon test-time adaptation, search, memory, and reasoning about uncertainty. Overall, we hope our benchmark design paves the way to measure AI progress on open-ended adaptation spanning long time-horizons in the real world.

1 Introduction

FutureSim evaluates whether agents can adapt forecasts as real-world information arrives over a long horizon. It replays events chronologically beyond models’ knowledge cutoffs and exposes room for improvement in frontier-agent test-time adaptation.

  • FutureSim asks agents to forecast world events beyond their language model’s knowledge cutoff while adapting to new information.The environment replays real-world events in temporal order, with daily news and feedback on resolved questions.
  • Forecasting tests whether agents’ priors align with world evolution and whether they update those priors appropriately when new information arrives.Because agents’ predictions do not change the underlying environment, evaluation remains grounded in real-world events rather than simulated counterfactuals.
  • The benchmark covers 90 days from January to March 2026 and 330 events, using timestamped source documents to bootstrap the simulation.Agents can search newly available news and receive feedback as questions resolve.
  • FutureSim sandboxs agents against future-information leakage while providing reliably dated offline news snapshots.This design supports evaluation after the agents’ knowledge cutoff.
  • FutureSim is open-ended: agents choose which free-form questions to forecast and when, while reporting probability distributions over multiple possible outcomes.The environment enforces only prediction submission and day advancement, allowing users to vary models, harnesses, and chronological data.
  • Frontier agents show significant room for improvement in test-time adaptation, while FutureSim also supports studies of search, memory, harness design, and multi-agent dynamics.The benchmark is intended to distinguish adaptive capabilities across native agent harnesses.

2 Related Work

FutureSim builds on work using real-world time for continual-learning evaluation and on forecasting benchmarks, while distinguishing itself through chronological, open-ended adaptation to future events. It is presented as a reproducible long-horizon benchmark for adaptive reasoning about real-world events.

  • Prior continual-learning evaluations use real-world time as the underlying distribution shift for language models.
  • FutureSim differs by evaluating forecasts of future events while requiring reasoning under uncertainty and information-seeking actions over a long horizon.
  • FutureSim is described as the only reproducible benchmark in the comparison that tests open-ended adaptive reasoning about real-world events over a long horizon.
  • Related long-horizon benchmarks evaluate agents in games, mobile-use scenarios, or economic simulations whose dynamics are human-designed or model-simulated.The cited examples include BALROG, ARC-AGI 3, GAIA-2, Vending-Bench, and YC-Bench.
  • Earlier language-model forecasting evaluations used static questions with retrieved evidence, while newer benchmarks use live trading on prediction markets.FutureSim instead centers chronological interaction with real-world events.

3 The FutureSim Environment

FutureSim is a daily chronological forecasting environment in which agents receive dated context, submit multi-outcome probability forecasts, update them, and advance time. It evaluates predictions with metrics that capture both correctness and probability calibration.

  • Environment Design: FutureSim advances in daily time-steps corresponding to real-world dates, updating task and context state at each step.
  • Environment Design: Each task row stores a forecasting question’s background, resolution criteria, resolution date, and most recent agent forecast.
  • Environment Design: Agents may assign probabilities to multiple outcomes they generate themselves rather than selecting only one predefined outcome.Once a question’s resolution date passes, its ground-truth outcome is added to the task state.
  • Environment Design: At each time-step, agents can access documents that became available by then, using reliably dated Common Crawl News snapshots.
  • Environment Design: The environment exposes two actions: submitting or revising a forecast and advancing the simulation by one day.Advancing the day runs required evaluations and updates task and context state.
  • Evaluation: Brier Skill Score combines predicted-outcome correctness with the probability distribution, with higher scores better and values ranging from 1 for a fully confident correct answer to −1 for fully wrong guesses.The score is adapted for questions with multiple possible outcomes and is defined as a proper scoring rule.
  • Evaluation: Accuracy measures whether the outcome receiving the most probability is correct, ignoring how probability is distributed across outcomes.Reported accuracy averages across questions, projects unresolved questions using current predictions, and scores unanswered questions as zero.

4 Benchmarking Frontier Agents in FutureSim

FutureSim benchmarks frontier agents as they forecast evolving real-world events using chronological news access, ground-truth resolutions, and different harnesses. Results show substantial differences in adaptation, calibration, updating behavior, and the effects of harness design and memory.

  • Experimental setup: The benchmark creates forecasting questions from news, exposes agents only to date-appropriate articles, and evaluates models over a 90-day simulation.The setup covers 330 events from January to March 2026, with agents operating through harness tools and an offline chronological corpus.
  • Agent performance: GPT 5.5 leads by a large margin in accuracy and Brier skill score, while harness changes substantially affect open-weight model performance.DeepSeek V4 Pro improves in both harnesses; Qwen3.6 Plus and GLM 5.1 improve only in the custom harness, and Qwen3.6 Plus deteriorates in calibration in OpenCode.
  • Agent behaviours: 27.4% of GPT 5.5’s incorrect final predictions assign at least 0.5 probability to the wrong top answer, indicating substantial overconfidence.The corresponding rate for assigning at least 0.75 probability is 9.1%.
  • Agent behaviours: GLM 5.1 has the lowest overconfidence rate at 3.7% above 0.5 among wrong answers but performs poorly with the fewest updates, while Qwen3.6 Plus predicts on only 36.7% of questions.Across models, prior rationales and stored lessons can become treated as hard truths, producing subsequent overconfident mistakes.
  • Human aggregate comparison: GPT 5.5 sometimes leads human prediction-market aggregates but performs worse on other markets, while its Nepal election updates broadly track the aggregate with a lag.The lag is attributed to the less-fresh search corpus, which notably lacks social-media posts.
  • Adaptation and memory: Inference-time memory improves performance, while agents initialized from weak forecasts can remain anchored and fail to reach the zero Brier skill-score baseline.The benchmark also exposes long-horizon context-management challenges and may improve with further harness engineering.

5 Capabilities Tested

FutureSim tests whether agents can focus on relevant questions, search evolving information, remember useful evidence, update forecasts, and learn from resolved predictions. Ablations examine adaptation, search, memory, inference scaling, and multi-agent dynamics.

  • Capabilities Tested: FutureSim tests agents’ ability to focus on questions, search for evidence, remember information, update forecasts, and learn from resolved predictions.These capabilities are presented as requirements for performing well in the simulation.
  • Test-time adaptation: GPT 5.5, Claude Opus 4.6, and DeepSeek V4 Pro improve similarly from shared initial forecasts, while Qwen3.6 Plus barely improves and all finish below Brier skill score 0.The setup fixes initial forecasts to Qwen3.6 Plus’s worst-agent forecasts to isolate adaptation.
  • Memory: All three tested models perform worse without write access to file-based memory, indicating memory supports long-horizon forecasting in the harness.The memory ablation removes agents’ ability to write memories.
  • Search: Fresh daily context and agentic search outperform frozen-corpus or single-retrieval settings, where updates reinforce confidence and can produce negative Brier skill scores.The ablation tests both daily corpus updates and the extent of agentic search.
  • Inference Scaling: Higher GPT 5.5 reasoning efforts improve accuracy and Brier skill score, but xhigh adds no improvement over high despite substantially greater tool use at high effort.The gains are consistent across accuracy and Brier skill score, with diminishing returns at the highest efforts.
  • Multi-agent dynamics: Identical DeepSeek V3.2 agents converge toward aggregate predictions in multi-agent runs, unlike independent runs where predictions diverge over time.Absolute performance is similar between single-agent and multi-agent runs.

6 Conclusion

FutureSim replays post-cutoff world events and updates available context over time so agents maintain probabilistic forecasts. The benchmark reveals suboptimal frontier-agent behavior while supporting experiments that isolate emerging capabilities, within a purely predictive setting.

  • 6 Conclusion: FutureSim replays world events beyond model knowledge cutoffs and updates available context at each timestep while agents maintain probability distributions over outcomes.Forecasting questions are generated from news documents and agents update predictions as the simulation advances.
  • 6 Conclusion: FutureSim’s reported absolute performance is a lower bound because better harnesses, tools, or context corpora can improve results.The conclusion explicitly qualifies the reported capabilities by the available evaluation setup.
  • 6 Conclusion: The benchmark is flexible and extendable, but its purely predictive setting limits applicability where agent actions change environment dynamics or predictions are performative.Decision-making domains requiring accurate counterfactual worlds fall outside this scope.

A.1 Question creation methodology

FutureSim creates free-form forecasting questions from timestamped news articles, then filters and revises them to improve validity and prevent information leakage. The current dataset narrows more than 10,000 articles to 330 questions targeting post-cutoff events.

  • Question creation methodology: The question-generation pipeline prompts an LLM to create free-form short-answer questions whose answers are contained in timestamped source articles.It writes titles, backgrounds, and resolution criteria as if the answer were not yet known.
  • Question creation methodology: The pipeline applies leakage checks, validates formats, revises dates, filters invalid answer types, and removes unsuitable questions.These steps refine question quality and temporal validity.
  • Question creation methodology: The dataset uses Al Jazeera articles from the first quarter of 2026, narrowing 10,000+ source articles to 330 high-quality questions.The retained questions represent 3% of the initial article pool.
  • Question creation methodology: Questions resolve from January 1 through March 28, 2026, targeting events after the evaluated frontier models’ knowledge cutoffs.Resolution dates are chosen to support post-cutoff forecasting.
  • Question creation methodology: Questions are removed when models can answer them using search capped to June 2025 or still fail with full web search as of April 2026.The first filter removes stale or easy questions, while the second addresses possible label noise.

A.2 Example of a Discarded Question

A discarded forecasting question illustrates why FutureSim revises resolution dates: an answer may become inferable before the source article or official announcement. The final environment keeps active questions visible while agents submit and revise probability distributions over time.

  • A.2 Example of a Discarded Question: The discarded PAHO question appeared to concern an April 2026 meeting, but public evidence made its answer inferable by November 11, 2025.The official January 2026 announcement confirmed information that was already strongly determined earlier.
  • A.2 Example of a Discarded Question: Resolution dates are revised to the earliest date when the answer could be inferred with certainty, rather than relying only on the source article date.This prevents questions from leaking information before the evaluation window.
  • Final set of tasks: Each active task accepts forecast revisions as a probability distribution over up to five free-form outcomes until its resolution date.Agents can submit and revise forecasts while questions remain active.
  • Final set of tasks: The final task set contains 330 forecasting questions resolving across 84 dates from January 1 to March 28, 2026.The questions resolve across 122 in January, 92 in February, and 116 in March.
  • Environment and harness: The environment sandboxes agents to dated offline article snapshots and withholds live web search, while its harness manages context, memory, and forecasting workflow over discrete timesteps.These controls prevent future-information leakage during the simulation.

B.3 Sandboxing agents to avoid contamination

FutureSim prevents contamination by isolating agent harnesses and restricting them to date-gated offline news snapshots. An audit showed that external search date filters could reveal future outcomes prematurely, motivating the frozen local corpus.

  • Sandboxing: Sandboxed harnesses use isolated filesystems and controlled network access to prevent unintended post-cutoff information leakage.The setup blocks live web search and limits external access to model-provider endpoints.
  • Date-gated context: Agents access dated Common Crawl News snapshots containing only articles published on or before the current simulation day.The corpus preserves article text, source, URL, and publication date after deduplication.
  • Date-gated context: Local retrieval provides temporal controls equivalent to terminal access, supporting different retrieval and memory strategies within the environment.FutureSim treats the context store as part of the environment rather than a particular agent design.
  • Leakage audit: A Brave Search audit revealed an Olympic medal result on simulated January 30 even though the event resolved on February 7.The returned snippet exposed the medalists and countries before the simulated agent should have accessed the result.

C Metrics

FutureSim evaluates forecasting with probability distributions over multiple outcomes, scoring both correctness and calibration while accounting for when predictions are made and how agents interact. The section also describes diagnostics for action use, abstention, and adaptation comparisons.

  • Brier Skill Score: Brier skill score evaluates a forecast by comparing each reported outcome probability with the resolved answer, including unreported outcomes at probability 0.The answer matcher checks semantic equality between reported outcomes and the ground-truth answer.
  • Brier Skill Score: A fully confident correct forecast scores 1, abstention scores 0, and assigning all probability to wrong outcomes scores −1.The daily avg_brier column averages question-level Brier skill scores, treating questions without forecasts as 0.
  • Worked Example: Listing the correct outcome is insufficient: a forecast assigning 0.55 to Chiefs and only 0.15 to Seattle Seahawks receives a negative Brier skill score.The dominant squared-error penalty comes from assigning low probability to the realized outcome while concentrating probability on the wrong outcome.
  • Properness: The Brier skill score is strictly proper with respect to the projected true belief, so expected score is uniquely maximized by reporting that belief.This result holds under the stated partial one-to-one matching between reported and true outcomes, with missing mass assigned to unmatched outcomes.
  • Time-Weighted Scores: The single-agent time-weighted score rewards positive forecasts made earlier and penalizes negative forecasts earlier because scores accrue each day until update or resolution.For each question, the held forecast supplies the daily Brier skill score, or 0 before any forecast is submitted.
  • Peer and Multi-Agent Scores: Peer-relative scoring subtracts the average score of other agents with active forecasts, while the shared market aggregate averages agents’ current probabilities coordinate-wise.If no other active forecast exists, the peer baseline is 0; missing outcomes in the aggregate receive probability 0.
  • Action Diagnostics: Figure 9 shows that action count does not fully explain performance: Opus 4.6 and DeepSeek V4 Pro act similarly, yet Opus 4.6 performs substantially better.Agents benefiting most from test-time adaptation consistently make more actions, but absolute performance also depends on the agent.
  • Abstention and Adaptation: Allowing abstention assigns accuracy 0 and Brier skill score 0, making question selection part of final performance and potentially skewing adaptation comparisons.Restricting analysis to 46 questions where every model updated preserved rankings but increased absolute scores because those questions likely contained strong new evidence.

D.3 Search Ablations Brier skill score

The Brier skill score ablations show that fresh daily context, iterative agentic search, and additional reasoning effort improve calibrated forecasting, while some settings produce overconfidence or negative scores. Multi-agent runs also show slightly higher accuracy than independent single-agent runs, though deeper analysis is deferred.

  • Fresh daily news context substantially improves Brier skill score over freezing the corpus at day 0.With no new articles, updates reinforce confidence without evidence, causing overconfidence and an eventual negative Brier skill score.
  • Agentic search outperforms a single retrieval query for Brier skill score, indicating that iterative search supports calibrated beliefs.A single retrieval query also produces a highly negative Brier skill score.
  • Restricting evaluation to 46 questions where all models update preserves the main test-time adaptation trends across accuracy and Brier skill score.Question selection does not significantly change the observed trends.
  • Higher GPT 5.5 reasoning effort improves Brier skill score, with the largest gains away from no reasoning and diminishing returns at the highest efforts.Reasoning effort “none” yields a highly negative Brier skill score, while the benefit plateaus after high effort.
  • Multi-agent DeepSeek v3.2 runs maintain higher accuracy than independent single-agent runs, although the phenomenon receives only anecdotal analysis.The authors leave deeper exploration to future work because of cost reasons.

E Prompts

The prompts define FutureSim’s native and custom forecasting harnesses, including daily updates, news access, persistent memory, prediction rules, and Brier-based evaluation. The custom harness adds update guidance, memory management, and retained performance context across sessions.

  • E.1 Native Harness: The native harness presents the current date, next update date, available questions, prediction state, and daily news-access tools.Agents can search news, submit one forecast for one question, and advance to the next day.
  • E.1 Native Harness: Native scoring combines calibrated accuracy with time weighting, rewards updates after new evidence, and assigns zero contribution to unanswered active questions.Brier skill score is 1.0 for perfect prediction, 0.0 for abstention, and negative when worse than abstaining.
  • E.1 Native Harness: The native workspace provides read-only market.csv, dated article files, and writable memory for forecasting notes and supporting scripts.The prompt permits agents to create files or directories for forecasting strategies, calibration notes, scripts, and per-question research.
  • E.1 Native Harness: Native submission rules require active unresolved questions, real outcomes, at most the allowed number of outcomes, and probabilities summing to no more than 1.0.Placeholder outcomes such as “Unknown,” “TBD,” “Other,” and “N/A” are prohibited.
  • E.2 Our Harness: The custom harness adds daily feedback, update priorities, action and context-budget guidance, and a rule against changing predictions without specific new evidence.Questions resolving the next day receive highest update priority, followed by unanswered questions and questions with newly discovered information.
  • E.2 Our Harness: The custom harness retains predictions, resolved-question results, cumulative performance, and newly arrived articles across sessions.The prompt states that context is cleared after each session while memory and past predictions persist.
  • E.2 Our Harness: Custom-harness memory is organized into per-question notes and indexed meta-insights managed through add, update, retrieve, and delete operations.Per-question memory is capped at 1000 characters per row, while meta-insights are capped at 500 entries.
  • E.2 Our Harness: The custom harness preserves the same market schema, news-based evidence access, Brier skill score, and calibrated probability requirements as the native setup.The market includes question metadata, resolution fields, prediction history, options, and ground truth.
Loading 2605.15188v1…