Source-linked AI summary
ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilities
Ezra Karger, Houtan Bastani, Chen Yueh-Han, Zachary Jacobs, Danny Halawi, Fred Zhang, Philip E. Tetlock
TL;DR
ForecastBench addresses the lack of a standardized, leakage-resistant framework for evaluating ML forecasts of future events. It dynamically generates and updates 1,000 unresolved questions, collects forecasts from LLMs and human groups, and compares their accuracy on a random 200-question subset. Expert forecasters outperform the top-performing LLM, leaving substantial room for improved real-time AI forecasting.
Problem
There is no standardized framework for evaluating ML forecasting accuracy, while static benchmarks become obsolete and risk knowledge contamination.
Method
ForecastBench continuously gathers unresolved future-event questions, elicits forecasts from LLMs and human groups, and updates a public leaderboard as questions resolve.
Results
Superforecasters achieved a mean Brier score of 0.096, outperforming the top LLM, Claude 3.5 Sonnet at 0.122, with p < 0.001.
Takeaways & Limitations
Current LLMs underperform superforecasters, leaving room for systems that continuously incorporate current events and reason over longer time frames.
Takeaways & Limitations
Human survey participants consented to public release and potential AI training use of their anonymized forecast and rationale data.
Abstract
from arXiv · showhide
Forecasts of future events are essential inputs into informed decision-making. Machine learning (ML) systems have the potential to deliver forecasts at scale, but there is no framework for evaluating the accuracy of ML systems on a standardized set of forecasting questions. To address this gap, we introduce ForecastBench: a dynamic benchmark that evaluates the accuracy of ML systems on an automatically generated and regularly updated set of 1,000 forecasting questions. To avoid any possibility of data leakage, ForecastBench is comprised solely of questions about future events that have no known answer at the time of submission. We quantify the capabilities of current ML systems by collecting forecasts from expert (human) forecasters, the general public, and LLMs on a random subset of questions from the benchmark ($N=200$). While LLMs have achieved super-human performance on many benchmarks, they perform less well here: expert forecasters outperform the top-performing LLM ($p$-value $<0.001$). We display system and human scores in a public leaderboard at www.forecastbench.org.
1 INTRODUCTION
ForecastBench addresses weaknesses in static LLM forecasting evaluations with a continuously updated benchmark of unresolved future-event questions. Initial results show current LLMs perform about as well as minimally experienced human aggregates but significantly worse than superforecasters.
- Motivation: The benchmark responds to the expense, time demands, domain specificity, and bias concerns associated with human forecasting by evaluating automated systems at scale.The paper positions ML and LLM forecasting as a response to these limitations.
- Motivation: Static benchmarks become obsolete as model knowledge cutoffs change, risk contamination from inaccurate cutoff estimates and post-training knowledge, and can be manipulated or overfit.These drawbacks make continuously tracking and comparing frontier models difficult.
- Contribution: ForecastBench continuously gathers unresolved questions, elicits LLM and human forecasts, and updates a public leaderboard as outcomes resolve.Its automated system sources questions daily from prediction markets, forecasting platforms, and real-world time series.
- Initial results: State-of-the-art LLMs performed roughly as well as a simple median of humans with no or minimal forecasting experience, despite retrieval, prompt engineering, and crowd-forecast access.The models performed significantly worse than the median forecast of superforecasters.
- Future directions: The released forecast-and-rationale dataset is intended to support future fine-tuning, testing, model updating, and longer-horizon reasoning research.The authors identify continuous current-event updating and extended-time-frame reasoning as improvement directions.
- Related work: ForecastBench avoids leakage because each question’s final resolution is determined in the future, while covering questions with and without underlying time series.This makes the benchmark broader than standard statistical time-series forecasting evaluations.
2 PRELIMINARIES
The paper defines forecasting as probabilistic prediction of future events, evaluates binary forecasts with Brier score, and tests a broad set of language models. These preliminaries establish the task, metric, and model pool used in ForecastBench.
- Forecasting: A forecasting question asks for probabilities assigned to potential outcomes of a future event.Forecasting platforms and prediction markets host many such questions, including public-interest events.
- Metrics: For binary questions, the Brier score is (f − o)^2, where f is the probabilistic forecast and o is the outcome.Lower scores are better; 0.25 corresponds to an uninformed 0.5 forecast, and the metric incentivizes truthful reporting.
- Models: The initial benchmark evaluates 17 LLMs, including GPT-4, GPT-4o, Claude-3.5-Sonnet, Gemini 1.5 Pro, Llama, Mistral, Mixtral, and Qwen models.The model set spans several major model families and sizes.
3 BENCHMARK, LEADERBOARD, AND DATASETS
ForecastBench maintains a continuously updated question bank and samples standardized sets spanning market and dataset sources, varied horizons, and Boolean combinations. It resolves forecasts over time and publishes datasets and leaderboard comparisons.
- Questions and Resolution Values: The benchmark continuously adds and filters questions, updates resolution values nightly, releases sets every two weeks, and resolves submitted forecasts with daily leaderboard updates.
- Questions and Resolution Values: Questions come from prediction markets, forecast aggregation sites, and established real-world event datasets, with liquid market questions selected as higher-quality inputs.
- Question Sets: 1,000 LLM questions are released biweekly with balanced sampling across sources and categories, while the 200-question human set is sampled from them.
- Question Sets: Dataset questions use eight forecast horizons from 7 to 3,650 days, whereas market questions receive one forecast, enabling short-, medium-, and long-term evaluation.
- Question Sets: Half of each question set comprises Boolean combinations of two standard questions, producing forecasts for all four outcome combinations.This design supports millions of possible sampled questions and requires forecasters to account for relationships between events.
- Datasets: ForecastBench generates four datasets that grow over time and makes forecast questions, submissions, historical leaderboard updates, and biweekly question sets available to researchers.
4 HUMAN FORECASTER BASELINE
The human baseline compares median forecasts from the general public and superforecasters on the same 200-question set. Public participants provide broader coverage, while superforecasters contribute fewer but more specialized responses.
- Human Forecasts: Median forecasts from the general public and superforecasters serve as group-level performance estimates on the shared 200-question set.
- General Public: 500 public participants each forecast 20 random questions in a one-hour survey, supplying forecasts and rationales.
- Response Coverage: Public questions received at least 40 responses each, averaging 49 responses, whereas superforecaster questions had at least 3 and averaged 8 responses.
- Superforecasters: 39 superforecasters forecast 20 random questions during a 9-day experiment that later allowed participants to see one another’s forecasts.
5 LLM BASELINE
ForecastBench evaluates LLM forecasting with multiple prompting, retrieval, crowd-forecast, and ensemble baselines. Superforecasters outperform the best LLMs, while model accuracy correlates with Arena scores and estimated training compute.
- Baselines: The seven baselines combine zero-shot or scratchpad prompting with retrieved news, crowd forecasts, freeze values, and multi-model aggregation.The final ensemble averages nine forecasts from three models and three superforecaster-crafted prompts.
- Comparing humans and LLMs: Superforecasters achieve a mean Brier score of 0.096, outperforming the general public at 0.121 and Claude 3.5 Sonnet at 0.122, both p < 0.001.These comparisons use a random 200-question subset, with forecasts spanning 7-, 30-, 90-, and 180-day horizons.
- Failure modes: LLMs perform significantly worse on combination questions, widening the gap between superforecasters and GPT-4o to 0.054 Brier points.The human comparison assumes independence among component events, which underestimates human accuracy when dependencies could be modeled.
- Comparing LLMs: Across 1,000 questions, Claude 3.5 Sonnet slightly outperforms GPT-4 Turbo, while recent topical news does not improve performance.Most top models use scratchpad prompting and crowd forecasts for market questions.
- LLM performance and forecasting accuracy: LLM Arena scores correlate with forecasting accuracy, with r = −0.68 and p = 0.003, while the fitted relationship projects superforecaster-level performance near Arena score 1406.The projected Arena score has a bootstrapped 95% CI of 1346–1633.
- LLM performance and forecasting accuracy: Estimated training compute has a marginally significant relationship with Brier score, projecting superforecaster-level performance near 6.49 × 10^26 compute.The bootstrapped 95% CI is 9.69 × 10^25–8.65 × 10^28, with r = −0.67 and p = 0.046.
6 DISCUSSION
ForecastBench provides a continuously updated evaluation of real-time LLM forecasting without relying on questions whose answers were already known. Its initial results show that current LLMs remain below superforecasters.
- Discussion: ForecastBench continuously updates forecasting questions and evaluates LLMs on events unresolved at submission, reducing data-leakage risks.The benchmark is designed as a robust real-time evaluation environment.
- Discussion: State-of-the-art LLMs show promising potential but underperform superforecasters, revealing persistent challenges for accurate real-time forecasting.The paper presents this performance gap as evidence that current systems are not yet reliable substitutes for expert forecasters.
- Discussion: The public leaderboard and standardized forecast-and-rationale dataset support future fine-tuning, architecture research, and adaptive forecasting systems.The released resources include real-time scores for LLMs and humans.
8 ETHICS STATEMENT
The study obtains informed consent for public release and AI-training use of human forecast data, anonymizes released text, and makes its benchmark resources publicly available.
- Ethics statement: Human participants consented to public release and potential AI-training use of their anonymized forecasts and rationales.The consent process was approved by the study’s IRB under number 855431.
- Ethics statement: Researchers manually review participant text for personally identifiable information before release and plan equivalent reviews for future forecasting rounds.This review process is required by the stated IRB requirements.
- Ethics statement: The project publishes its datasets, code, documentation, and bi-weekly question sets through public repositories and websites.Datasets use the CC BY-SA 4.0 license, while the codebase uses the MIT license.
- Ethics statement: Benchmark maintenance is funded through mid-2027, with datasets intended to remain available on GitHub if further funding is not provided.
B DATASETS
ForecastBench releases openly usable question, resolution, and forecast datasets in JSON, built from public sources and regularly updated. The datasets define standardized fields for questions, resolutions, forecasts, sources, metadata, and participant groups.
- B DATASETS: The datasets are released as JSON files with no restrictions on use.Question, resolution, and forecast sets are published for research and other applicable purposes.
- B.1 QUESTION AND RESOLUTION SETS: Questions include their source, forecast wording, background, resolution criteria, market metadata, due dates, and combination-question structure.Combination questions store component questions and direction vectors for negated or non-negated components.
- B.1.1 EXAMPLES: Resolution records specify dates, resolved values, resolution status, and URLs or criteria used to evaluate forecasts.The question bank is updated with new questions and resolution values, while invalid, low-quality, and resolved questions are removed from active sampling.
- B.2 FORECAST SETS: Forecast sets contain question-set headers and forecast arrays, with separate formats for public, superforecaster, and language-model predictions.Public and superforecaster data are anonymized, while aggregated forecasts omit user identifiers and reasoning fields.
E.2 AGGREGATION METHODS
The benchmark compares several forecast-aggregation methods and finds the geometric mean and geometric mean of log odds achieve the best performance among the evaluated methods.
- E.2 AGGREGATION METHODS: Model inference uses temperature 0 and a maximum output length of 2,000 tokens for reasoning and probabilistic forecasts.
- E.2 AGGREGATION METHODS: The geometric mean and geometric mean of log odds achieve the best performance among the evaluated aggregation methods.The comparison covers Median, Trimmed Mean, Geometric Mean, and Geometric Mean of Log Odds.
G STATISTICAL DETAILS
ForecastBench reports leaderboards for human and LLM question sets and uses statistical comparisons that account for paired forecasts, while noting dependence between questions as a limitation.
- G STATISTICAL DETAILS: Pairwise statistical tests may overstate confidence because they assume questions are independent despite possible within-topic or cross-event correlations.The authors argue that diverse domains and sources may reduce the practical impact of this dependence.
- H LEADERBOARDS: TOP 50: The study publishes top-50 leaderboards for human questions, LLM questions, and human forecasts extended with combination questions.The human question set contains 200 standard questions, while the full LLM set contains 1,000 questions.
- H LEADERBOARDS: TOP 50: The benchmark aggregates forecasts from 500 general-public participants and 39 superforecasters, with public participants recruited through Prolific and Facebook.Superforecasters could inspect and revise forecasts using other superforecasters’ forecasts and rationales.
- G STATISTICAL DETAILS: The evaluation covers 17 LLMs and compares zero-shot, scratchpad, retrieval-augmented, and human-forecast-augmented prediction methods.
J.2 SCRATCHPAD WITH INFORMATION RETRIEVAL BASELINE
The retrieval-augmented scratchpad baseline adds retrieved news to forecasting prompts using a multi-stage information-retrieval pipeline and fixed search, ranking, and summarization settings.
- J.2 SCRATCHPAD WITH INFORMATION RETRIEVAL BASELINE: The scratchpad prompt adds retrieved information before the forecasting instructions, while combination questions include information about both component questions.
- J.2 SCRATCHPAD WITH INFORMATION RETRIEVAL BASELINE: The retrieval pipeline generates search queries, retrieves news, filters and reranks articles, and summarizes the selected text before forecasting.The implementation uses the information-retrieval system from Halawi et al. (2024).
- J.2 SCRATCHPAD WITH INFORMATION RETRIEVAL BASELINE: The system uses six search-query keywords, query-length limits of five words for NewsCatcher and Google News, and up to 20 summaries for final output.
- J.2 SCRATCHPAD WITH INFORMATION RETRIEVAL BASELINE: Articles are prefiltered with embeddings, ranked by an LLM using titles and the first 250 tokens, and sorted by date.The cosine-similarity threshold is 0.5, and the ranking method is llm-rating.
- J.2 SCRATCHPAD WITH INFORMATION RETRIEVAL BASELINE: Inference uses temperature 0 and a 2,000-token maximum output length to accommodate reasoning and probabilistic forecasts.
L GENERAL PUBLIC SURVEY DEMOGRAPHICS
The general-public survey collected demographic information on age, gender, ethnicity, and country of residence, while comparable demographic data were not collected for superforecasters.
- 500 general-public forecasters provided demographic information covering age, gender, ethnicity, and country of residence.These characteristics are summarized in the survey’s demographic tables.
- The study did not collect similar demographic information from participating superforecasters.The authors nevertheless state that they are reasonably certain this group is roughly representative of superforecasters overall.
- Prior Good Judgment Project data described superforecasters as predominantly men and U.S. citizens, with an average age of 40 years.The cited prior sample was reported as 83% men and 74% U.S. citizens.
M PERFORMANCE BREAKDOWN
Performance is broken down by forecast category and horizon, with superforecasters generally outperforming Claude 3.5 Sonnet except in specified cases.
- Tables report Brier scores separately by forecast category and forecast horizon for Claude 3.5 Sonnet and superforecasters.The category table and horizon table define the two dimensions of the performance breakdown.
- Claude 3.5 Sonnet slightly outperformed superforecasters on Environment & Energy questions.The comparison uses the top LLM configuration described in the accompanying table.
- Superforecasters outperformed Claude 3.5 Sonnet at the 7-day, 30-day, and 180-day horizons, while Claude performed better at 90 days.The breakdown includes only horizons resolved by publication: 7, 30, 90, and 180 days.