PaperlayerView exact revision on arXiv

Source-linked AI summary

Forecast-Dojo: Replayable Environments for Benchmarking and Training LLM Forecasting Agents

Liqin Ye, Haorui Wang, Fardin Ahmed, Rongzhi Zhang, Yuan He, Ziyuan Lin, Yanbin Yin, Jing Peng, Michael Galarnyk, Sudheer Chava, Chao Zhang

arXiv:2609.28876v1cs.AIcs.LG

TL;DR

Forecasting benchmarks often capture either evolving live questions or replayable historical cutoffs, but not both in one reusable setting. Forecast-Dojo replays resolved events as dated forecasting steps with research tools and outcome feedback. Research tools improve forecasts across models, though all evaluated models remain behind historical market forecasts.

  • Problem

    Live benchmarks evolve with new evidence but depend on wall-clock time, while historical benchmarks are replayable but typically assess each question at only one past point.

    Source: §1 · p. 1

  • Method

    Forecast-Dojo replays resolved events as sequences of dated forecast steps, pairing temporally restricted research with optional persistent belief notebooks and outcome-based feedback.

    Source: §1 · p. 1, §3 · p. 4

  • Results

    Research tools improve Brier score and accuracy for all 12 models, yet every evaluated model remains behind historical market forecasts on both metrics.

    Source: §1 · p. 2, §6 · p. 11

  • Takeaways & Limitations

    Forecast-Dojo produces outcome-grounded interaction trajectories for learning, with supervised fine-tuning demonstrating improved held-out forecasting performance over a base model.

    Source: §1 · p. 2, §5.5 · p. 10

    Persistent belief notebooks reduce research costs but do not consistently improve forecast quality.

    Source: §1 · p. 2, §5.2 · pp. 8–9

Abstract

from arXiv · show

We introduce Forecast-Dojo, a replayable environment for benchmarking and training LLM forecasting agents. It combines resolved prediction-market questions with dated news, allowing agents to research an event and revisit their predictions at successive historical dates. The same tasks and tools support repeated evaluation, collection of training interactions, and feedback from recorded outcomes without waiting for new events to resolve. Forecast-Dojo contains 1,568 Polymarket events, split by time into training and evaluation periods, and 18.8M dated news articles. In an evaluation of 12 models, research tools lower Brier score for all 12. Forecasts also improve as events unfold, with the largest gains at steps where more newly dated evidence is recorded. Every model still trails historical market forecasts in both Brier score and accuracy. A belief notebook carried between dates lowers research cost but does not consistently improve forecast quality. Beyond evaluation, Forecast-Dojo provides interaction trajectories and outcome feedback for agent learning, with supervised fine-tuning as a proof of concept.

1 INTRODUCTION

Forecasting benchmarks must capture how evidence and predictions change over time while preserving replayable, comparable conditions. Forecast-Dojo provides that setting, and research tools improve forecasts across all 12 evaluated models, though every model remains behind historical market forecasts.

Motivation

  • Live benchmarks evolve with the world but depend on wall-clock time, while historical benchmarks are reusable but usually evaluate a question at only one past point.The paper argues for replayable historical steps that evolve with evidence, can be scored immediately, and support training.Source: §1 · p. 1

Contribution

  • Forecast-Dojo replays resolved events as dated forecast steps, letting agents use only information available by each date while outcomes remain hidden until scoring.Steps can be reset and rerun, or traversed sequentially with persistent agent memory.Source: §1 · pp. 1–2

Empirical results

  • Research tools lower Brier scores for all 12 models, but every model still trails historical market forecasts in Brier score and accuracy.The best model Brier score is 0.546, compared with 0.498 for the market; notebook effects on forecast quality are mixed.Source: §1 · p. 2

Contribution

  • The dataset covers 1,568 Polymarket events and 18.8M CC-News articles, with time-based training and evaluation splits for benchmarking and trajectory-based learning.It includes 6,122 forecast steps: 1,338 training events and 230 evaluation events.Source: §1 · p. 2

2 RELATED WORK

Prior forecasting work includes dated-news benchmarks, live unresolved-event benchmarks, and interactive research environments. Forecast-Dojo distinguishes itself through fixed shared historical checkpoints that support matched comparisons and reusable learning trajectories.

Forecasting with dated evidence

  • ForecastQA and Autocast pair forecasting questions with dated evidence, while ForecastBench and FutureX collect predictions on unresolved events.These settings differ in whether they use historical or unresolved questions.Source: §2 · pp. 2–3
  • Unlike live forecasting, Forecast-Dojo replays resolved events for repeatable research and immediate outcome feedback while keeping market probabilities outside the forecasting prompt.Figure 1 depicts dated research steps and evaluator-only access to outcomes.Source: §2 · p. 3, Figure 1 · p. 3

Interactive research and forecast revision

Environments for training agents

  • Forecast-Dojo supplies a shared task and tool interface for collecting research interactions and evaluating agents on separate events, while leaving learning algorithms separate.Supervised fine-tuning demonstrates one use of the interface.Source: §2 · pp. 3–4

3 FORECASTING AS AN INTERACTIVE TASK

Forecast-Dojo treats forecasting as a sequence of dated steps for the same event, with the evidence boundary advancing over time. Agents may carry a belief notebook between fresh interactions, while outcome-based feedback is kept outside the forecasting context.

Events, forecast steps, and episodes

  • A forecast episode is an ordered sequence of forecasts for one question and eventual outcome, made at dates with differing historical evidence.At each step, the agent reports a probability distribution over the question’s mutually exclusive outcomes.Source: §3 · p. 4

Within-step interaction

  • Within each step, agents can search dated articles, inspect retrieved documents, and use computation before submitting a forecast conditioned on their interaction history.Different rollouts can follow different research paths even at the same date.Source: §3 · p. 4

Progression across forecast steps

  • Each step starts a fresh interaction; memory-on mode supplies a belief notebook summarizing the prior assessment, evidence, and open questions alongside the new date.Memory-free mode carries no information from earlier steps.Source: §3 · p. 4

Outcome feedback

  • After the realized outcome becomes available, the environment scores each forecast without exposing the outcome or feedback in the agent’s forecasting context.This separation lets the same task interface support different evaluation or learning procedures.Source: §3 · p. 4

4 FORECAST-DOJO

Forecast-Dojo instantiates a replayable forecasting task with selected resolved events, a dated news corpus, scheduled forecast checkpoints, and a shared agent interface. The same interaction records support evaluation and learning, with outcomes used for scoring only after forecasts are made.

4 FORECAST-DOJO

  • The section introduces Forecast-Dojo’s events, information corpus, forecast episodes, runtime, and evaluation records as its concrete task components.These components instantiate the abstract interactive forecasting task described earlier.Source: §4 · p. 4

4.1 FORECASTING EVENTS

  • Resolved Polymarket events are selected for sustained forecast horizons, market activity, nontrivial price histories, and sufficient pre-resolution evidence.The event pool spans binary and mutually exclusive multi-option questions.Source: §4.1 · p. 5

4.2 INFORMATION CORPUS

  • The CC-News corpus is cleaned, filtered for quality and near-duplicates, dated using publication metadata or crawl timestamps, and indexed for historical retrieval.After filtering, it contains approximately 18.8M articles; retrieval filters by the UTC day of the assigned timestamp.Source: §4.2 · p. 5

4.3 TEMPORAL TASK CONSTRUCTION

  • Each event receives a sublinear number of forecast steps across its lifetime, selected to favor market-belief movement and periods with more news activity.Events are split by complete temporal windows, keeping each episode wholly in training or evaluation.Source: §4.3 · pp. 5–6

4.4 AGENT INTERACTION

  • Agents can search, read, and compute using evidence dated by the forecast step; memory-on mode transfers the prior belief notebook, not prior conversation or tool history.Memory-free mode transfers no information from earlier steps.Source: §4.4 · p. 6

4.5 EVALUATION AND LEARNING

  • Forecasts support step- and trajectory-level evaluation, while logged interactions and outcome feedback from training events can be used by diverse learning methods.Table 3 reports recorded-forecast averages and standard deviations across four rollouts, with unusable forecasts replaced by uniform distributions.Source: §4.5 · p. 6, Table 3 · p. 7

5 EXPERIMENTS

Across the evaluated models, research tools improve forecasts, but historical market forecasts remain ahead. Forecasts improve as events unfold and more evidence is recorded; belief notebooks reduce research cost inconsistently affecting forecast quality, while trajectories also support a fine-tuning proof of concept.

5.1 EXPERIMENTAL SETUP

  • Twelve models face 230 held-out events (797 event-date pairs) in three settings; evaluation reports Brier, top-label accuracy, and Information-α.The settings are no tools, memory-free research, and research with a notebook carried between dates.Source: §5.1 · p. 7

5.3 FORECASTING OVER TIME

  • Mean Brier falls from 0.670 to 0.606 with memory-free research and 0.671 to 0.612 with memory-on research, while no-tools forecasts remain near 0.769.The market improves from 0.597 to 0.438; all 12 models improve from first to last stage with memory-free research.Source: Figure 2 · p. 8, §5.3 · p. 9

5.2 OVERALL FORECASTING PERFORMANCE

  • Memory-free research lowers Brier and improves accuracy for all 12 models; GPT-5.5 improves from 0.698 to 0.564 in Brier and 43.85% to 57.69% in accuracy.DeepSeek-V3.2 also improves from 0.805 to 0.655 in Brier and 34.34% to 47.73% in accuracy.Source: §5.2 · p. 8
  • A belief notebook lowers mean Brier for six models and raises it for six; GPT-5.6 Sol’s overall gain coincides with fewer unusable reports, while usable-only Brier changes by +0.0010.The comparison distinguishes forecast quality from output reliability; notebook effects on quality are model-dependent.Source: §5.2 · pp. 8–9
  • The market scores Brier 0.498 and accuracy 64.55%, ahead of the best model scores of 0.546 and 57.69%, respectively.Information-α is negative for every model in all three settings; market probabilities may use information outside the agents’ news archive.Source: §5.2 · p. 9

5.3 FORECASTING OVER TIME

  • Memory-free stepwise Brier change ranges from −0.007 in the lowest new-evidence quartile to −0.071 in the highest; memory-on evidence capture correlates with improvement (Spearman ρ = 0.74).Evidence capture is measured relative to other models’ notebooks across 12 models.Source: §5.3 · pp. 9–10

5.4 COST ANALYSIS

  • Notebook reuse cuts research cost after the first forecast; all five proprietary models save 8%–33% (median 24%), with the largest savings when little new evidence arrives.The first forecast costs about the same in both settings; memory-on execution is cheaper from the second forecast onward.Source: Figure 3 · p. 9, §5.4 · p. 10

5.5 TRAINING PROOF OF CONCEPT

  • On 230 held-out events, fine-tuning lowers Brier 0.924 to 0.749 (paired improvement −0.176) and raises accuracy 34.8% to 42.8% (+8.1 points), alongside 1.4 to 3.7 tool calls.The results are a proof of concept that Forecast-Dojo trajectories can support training agents.Source: Table 4 · p. 10, §5.5 · p. 10

6 CONCLUSION

Forecast-Dojo replays resolved prediction-market events across historical forecast states, supporting controlled comparisons as events evolve. Research tools improve forecasts, but agents remain behind historical markets, and persistent memory more consistently reduces research cost than improves forecast quality.

6 CONCLUSION

  • Forecast-Dojo replays resolved prediction-market events as fixed sequences of historical forecast states, enabling comparisons under the same information conditions throughout an event.The environment also produces outcome-grounded interaction trajectories that can support learning algorithms.Source: §6 · p. 11
  • Research tools improve Brier score and accuracy for all 12 models, with larger step-wise gains when more newly dated evidence becomes available.Models recording more newly dated evidence also tend to improve more over an episode.Source: §6 · p. 11
  • All evaluated agents remain behind historical market forecasts, while persistent memory reduces research cost more consistently than it improves forecast quality.The interaction trajectories and outcome feedback can support learning methods ranging from supervised fine-tuning to trajectory-level and reward-based optimization.Source: §6 · p. 11

A DATASET AND BENCHMARK DETAILS

This appendix describes the construction of the information corpus, the collection and filtering of forecasting events, and the selection of forecast dates.

A DATASET AND BENCHMARK DETAILS

A.1 INFORMATION CORPUS

The information corpus combines dated CC-News articles with text and quality filtering, timestamp assignment, duplicate suppression, and retrieval constrained by forecast date.

A.1 INFORMATION CORPUS

  • The corpus uses 15 monthly CC-News releases from March 2025 through May 2026, comprising 7,163 WARC files, with article text and URL, crawl, publication, and modification metadata extracted.Eligible HTML responses contribute titles, summaries, body text, and timestamps.Source: §A.1 · p. 15
  • The filtering pipeline normalizes text, retains English-language documents, and applies quality filters adapted from Gopher, C4, and FineWeb.Table 5 identifies the filtering pipeline; aggregate post-quality counts are reported in Table 8a.Source: §A.1 · p. 15, Table 5 · p. 16
  • Article dates use a priority-ordered publication or modification timestamp, with crawl time as fallback, and serve as the primary temporal boundary for retrieval.In the final corpus, 58.6% of articles use publication timestamps, 28.3% modification timestamps, and 13.1% crawl timestamps; a content-level screen also checks for temporal inconsistency.Source: §A.1 · p. 15
  • Duplicate suppression combines canonical-URL matching, same-title suppression within seven days, and MinHash-LSH over lower-cased word 5-grams.The procedure retains the earliest-published representative of each connected body-level duplicate component and is distinguished from semantic deduplication.Source: §A.1 · pp. 15–16
  • Articles are embedded using their title, summary, and body, then indexed by publication month with rows sorted by date.At retrieval, the date restriction is applied before similarity ranking, and full-article access independently rechecks the temporal constraint.Source: §A.1 · p. 16

A.2 EVENT COLLECTION AND FILTERING

The benchmark filters resolved Polymarket events for usable market trajectories and sufficient, non-leaking pre-resolution evidence, then selects temporally distributed forecast dates using market movement and news activity.

A.2 EVENT COLLECTION AND FILTERING

  • Eligible events are binary single-market events or mutually exclusive multi-option negative-risk bundles; other market layouts are excluded.Terminal YES prices determine labels, and daily market beliefs are reconstructed from UTC-day mean CLOB YES-price observations.Source: §A.2 · p. 16
  • Market-side filtering requires an event span of at least eight UTC calendar days and sufficient trading activity on the market leg corresponding to the realized outcome.The process also measures learning signal from changes in the market probability assigned to the realized outcome and removes events that are nearly certain or show little temporal variation.Source: §A.2 · p. 17
  • A complementary corpus-side screen checks whether pre-resolution evidence is sufficient for informed forecasting and whether retrieved content may leak outcome information.Targeted predictive-evidence and background queries are generated before resolution, without searches for the realized outcome or post-resolution reports.Source: §A.2 · p. 17
  • The retrieved evidence is classified as sufficient, partial, or insufficient, and only events judged sufficient are eligible for the final benchmark.The evidence set consists of the 15 highest-ranked unique documents retrieved using dense and BM25 top-10 results combined by reciprocal-rank fusion.Source: §A.2 · p. 17
  • A content-level leakage screen complements timestamp-based retrieval restrictions, and retained events must have no leakage flag.The judge does not receive the structured realized outcome, final market prices, or crowd trajectory.Source: §A.2 · pp. 17–18

A.3 FORECAST-STEP CONSTRUCTION

  • Training and evaluation are split by each event’s complete tradeable lifetime, with events crossing the March 1 boundary excluded from both splits.Training events start on or after June 1, 2025 and close before March 1, 2026; evaluation events start on or after March 1, 2026 and close before June 1, 2026.Source: §A.3 · p. 18
  • Eligible forecast dates are selected from a daily UTC grid after a two-day buffer before the recorded close date.The selection score combines max-normalized market movement and global news volume, with weights of 0.7 and 0.3, respectively, plus a small term favoring later dates.Source: §A.3 · p. 18
  • The eligible sequence is divided into contiguous equal-count bins, and the highest-scoring date in each bin is chosen to preserve temporal coverage while favoring market movement or news activity.Corpus activity is a global news-volume signal, not an event-specific relevance score.Source: §A.3 · p. 18

B EXPERIMENT DETAILS

The appendix records the evaluated models’ knowledge or checkpoint dates and specifies execution settings for evaluation and training.

B.1 KNOWLEDGE CUTOFFS AND CHECKPOINT DATES

  • All evaluated models’ reported knowledge cutoffs or checkpoint release dates precede the evaluation period, which begins March 1, 2026.A public checkpoint release date is used when no official knowledge cutoff is reported.Source: §B.1 · pp. 18–19, Table 9 · p. 19
  • When no effort parameter is sent, the model’s default reasoning setting applies.The request carries no effort parameter.Source: §B.1 · p. 19

B.2 MODELS AND EXECUTION PROTOCOLS

  • Evaluation runs use the same 230 held-out events and four rollouts per date, with shared tool budgets for tool-enabled settings.Tool runs allow 120 iterations and 400 calls per forecast step; no-tool runs remove tools and the corpus.Source: §B.2 · p. 19
  • Training-use-case runs use memory-free execution, Python, and identical budgets of 80 tool iterations and 200 calls per forecast for both models.Both models use temperature 0.6, a 32,768-token output cap, and top-p = 1.0.Source: §B.2 · p. 19

B.3 EVALUATION METRICS

The appendix defines the evaluation metrics and explains how unusable forecasts, inference costs, and tool use are handled and summarized.

B.3 EVALUATION METRICS

  • The multiclass Brier score uses the union of reported labels and the truth label, with unreported outcomes assigned zero probability and unsupported reported labels retaining their mass.Scores use the standard [0, 2] scale without normalization by outcome count or an extra binary-event factor.Source: §B.3 · p. 20
  • Accuracy uses fractional credit for tied top predictions, while Information-α measures improvement over the contemporaneous market belief when a scalar market probability is available.Positive Information-α favors the agent; neither realized outcomes nor market probabilities are provided to the forecasting agent.Source: §B.3 · p. 20

B.4 AGGREGATION AND FAILURES

  • Unusable recorded outputs are scored as uniform forecasts, while missing records are excluded from score means and counted as failures separately.An output is unusable if parsing fails or it contains no positive finite probability mass.Source: §B.4 · p. 20

B.5 INFERENCE COST AND TOOL USAGE

  • Inference-cost and tool-use statistics are descriptive resource estimates, not hardware-normalized efficiency comparisons.Research calls include search, scrape, and Python calls, including errors; memory-on forecasts begin each episode without prior memory.Source: §B.5 · p. 20

B.6 MEASURING NEWLY AVAILABLE EVIDENCE

The appendix defines how newly dated evidence and forecast changes are indexed across event steps, and notes that evidence availability is only a proxy.

B.6 MEASURING NEWLY AVAILABLE EVIDENCE

  • Longitudinal analyses use relative step position and retain adjacent recorded forecasts for evidence-quartile comparisons.The reported transitions are grouped by evidence-index quartile and averaged within events, then equally across events.Source: §B.6 · pp. 20, 22
  • Table 11 reports scheduled-forecast failure rates by condition for 12 model configurations, with each cell covering 3,188 scheduled forecasts.Failures include missing outputs, parser rejection, or reports without positive finite probability mass.Source: Table 11 · p. 21
  • Table 12 summarizes average provider-estimated cost and research calls per recorded forecast; open-weight models lack comparable per-forecast prices.Research calls include search, scrape, and Python calls, including errors.Source: Table 12 · p. 21, §B.6 · p. 21
  • New evidence counts ledger entries whose recorded evidence dates fall between consecutive forecast dates; late-discovered earlier evidence and carried entries are not counted again.Both active and superseded entries count, while invalid or future-dated entries are excluded.Source: §B.6 · p. 21
  • The leave-one-model-out availability index averages other models’ notebook counts, avoiding direct coupling to the evaluated model’s own recording behavior.The index is also used when analyzing memory-free forecasts.Source: §B.6 · p. 22
  • The evidence index is a proxy, not an exhaustive corpus count, and can miss relevant evidence that no model records.It depends on evidence recorded in other models’ notebooks.Source: §B.6 · p. 22

B.7 EVIDENCE CAPTURE AND FORECAST IMPROVEMENT

This section describes how agents capture and use dated evidence when updating forecasts, including prompts for research, reasoning, and preserving belief updates. It presents evidence capture as diagnostically associated with episode-level forecast improvement and specifies recency requirements that vary by forecast type.

B.7 EVIDENCE CAPTURE AND FORECAST IMPROVEMENT

  • Spearman ρ = 0.74 links evidence capture with episode improvement, exceeding search (0.31) and scrape counts (0.49); label normalization yields ρ = 0.60.The authors caution that these relationships are correlational and that notebook entries and recorded dates may be imperfect; evidence missed by all models is invisible to the measure.Source: §B.7 · p. 22

C PROMPTS

  • The no-tools prompt directs agents to read resolution criteria precisely, anchor forecasts in relevant base rates, and treat the forecast date as the information cutoff.It asks agents to express uncertainty as probabilities and commit to a forecast under a proper scoring rule.Source: C PROMPTS · p. 23

3. Weigh the evidence systematically

  • The memory-on prompt adds dated-news search, article scraping, and code for calculations while directing agents to decompose questions and weigh recent, direct evidence.It also calls for corroborating decisive facts, testing what could make a forecast wrong, and avoiding calculations based on invented inputs.Source: §3 · pp. 25–26

assessment — your current view

  • The belief notebook records the current probabilities, unresolved questions, and an evidence ledger with claims, directional implications, observed dates, status, and provenance.The notebook is intended to preserve enough accumulated research to reconstruct a forecast at a later date.Source: §3 · p. 26, assessment — your current view · p. 26

Maintaining the notebook

  • Notebook updates append new facts, mark superseded evidence rather than deleting it, preserve interpretation in notes, and keep probabilities consistent with active evidence.The query-decomposition prompt also restricts searches to legitimate pre-resolution signals and context, not documents aimed at revealing the outcome.Source: Maintaining the notebook · pp. 26–27
  • The sufficiency judge assesses whether retrieved pre-resolution articles let a competent forecaster form a calibrated belief without outside-corpus knowledge.Its review window ends two days before the recorded close date.Source: Maintaining the notebook · p. 28

HOW TO DECIDE EACH FIELD verdict — three buckets

  • The judge maps informativeness scores of 0, 1, and 2–3 to INSUFFICIENT, PARTIAL, and SUFFICIENT, respectively, after considering the question’s predictive horizon.Scores represent no, small, moderate, or large movement from a 50/50 prior.Source: HOW TO DECIDE EACH FIELD verdict —… · p. 28
  • Evidence freshness varies by forecast type: fixed-window snapshots need evidence from days to two weeks, evolving trajectories need roughly the last month, and one-shot events with persistent preconditions may use older actor or mechanism coverage.Figure 8 presents the evidence-sufficiency and leakage judge prompt used for corpus-side filtering.Source: HOW TO DECIDE EACH FIELD verdict —… · p. 28, Figure 8 · p. 29

Related papers

Loading 2609.28876v1…