PaperlayerView exact revision on arXiv

Source-linked AI summary

Approaching Human-Level Forecasting with Language Models

Danny Halawi, Fred Zhang, Chen Yueh-Han, Jacob Steinhardt

arXiv:2402.18563v1cs.LGcs.AIcs.CLcs.IR

TL;DR

Forecasting matters for policy and decision making, but human forecasts can be expensive, delayed, or domain-limited. The paper develops and evaluates a retrieval-augmented language-model forecasting system, finding that it approaches the human crowd on average and surpasses it in a selective setting.

  • Problem

    Forecasting informs policy and decision making, while human forecasting can be expensive, delayed, domain-limited, and often lacks explanatory reasoning.

    Source: §1 · p. 1

  • Method

    The paper develops a retrieval-augmented language-model system and self-supervised fine-tuning approach to generate forecasts and explanatory reasoning.

    Source: §4.2 · p. 6, §8 · p. 13

  • Results

    The system approaches aggregated human forecasts on the test set and outperforms the human crowd in a selective setting.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results suggest language-model systems may generate forecasts at competitive-human levels and support automated, scalable forecasting for institutional decision making.

    Source: §8 · p. 13

    Reported standard errors assume i.i.d. data, although the time-series data likely make them too small.

    Source: §3.2 · p. 5

Abstract

from arXiv · show

Forecasting future events is important for policy and decision making. In this work, we study whether language models (LMs) can forecast at the level of competitive human forecasters. Towards this goal, we develop a retrieval-augmented LM system designed to automatically search for relevant information, generate forecasts, and aggregate predictions. To facilitate our study, we collect a large dataset of questions from competitive forecasting platforms. Under a test set published after the knowledge cut-offs of our LMs, we evaluate the end-to-end performance of our system against the aggregates of human forecasts. On average, the system nears the crowd aggregate of competitive forecasters, and in some settings surpasses it. Our work suggests that using LMs to forecast the future could provide accurate predictions at scale and help to inform institutional decision making.

1 Introduction

The paper motivates language models as a way to automate judgmental forecasting and presents a retrieval-augmented system that retrieves information, reasons, and aggregates predictions. It evaluates the system on forecasting-platform questions and reports performance approaching the human crowd, with stronger performance in a selective setting.

1 Introduction

  • Judgmental forecasting draws on human judgment and contextual knowledge, while human effort can make forecasts expensive, delayed, or domain-limited.The paper motivates language models as a potentially cheap and timely way to produce forecasts and explanatory reasoning.Source: §1 · p. 1
  • The system automates retrieval of relevant news, reasoning and forecasting, and aggregation of individual forecasts into a final prediction.Each step uses one or more prompted or fine-tuned language models.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • The evaluation uses binary questions from five forecasting platforms, with test questions published after June 1, 2023, to avoid pretraining leakage.Earlier questions are used for hyperparameter search and fine-tuning.Source: §1 · p. 2
  • Self-supervised fine-tuning trains a language model on its training-set forecasts that outperformed the crowd, aiming to improve its reasoning and predictions.The system also searches retrieval and prompting configurations for end-to-end performance.Source: §1 · p. 2
  • The optimized system approaches aggregated human forecasts on the test set and outperforms the crowd when heuristics select which questions to answer.Performance is measured by Brier score.Source: §1 · p. 2

2 Related Work

Prior work includes datasets and tournament evaluations for event forecasting, retrieval-augmented forecasting, and statistical time-series prediction. This paper situates its focus in judgmental forecasting and contrasts its probability forecasts and Brier-score evaluation with earlier approaches.

Event forecasting

  • ForecastQA uses crowdworker questions based on news, while Autocast compiles competition questions through 2022; this paper sources 2023–2024 questions for recent language models.The passage notes that no machine-learning system approached human forecasters on Autocast in a competition with a large prize pool.Source: §2 · pp. 2–3
  • Unlike Yan et al. (2024), who report accuracy for a Fusion-in-Decoder model predicting binary resolutions, this work elicits explanations and probabilities and evaluates Brier score.Yan et al. (2024) report improved accuracy on Autocast using retrieval.Source: §2 · p. 3
  • Earlier tournament evaluations found GPT-4 and other language models underperformed the human crowd; the authors say this aligns with their own Section 3.4 findings.They distinguish their work by efforts to improve language models for forecasting.Source: §2 · p. 3
  • Transformer and language-model work on statistical time-series forecasting is distinct from this paper’s focus on judgmental forecasting.Source: §2 · p. 3

Information retrieval

  • For event forecasting, the paper describes retrieval of news articles using language models for query expansion, relevance ranking, and summarization.It frames access to diverse, up-to-date information as crucial for forecasting.Source: §2 · p. 3

Calibration

  • Forecasting tournaments use proper scoring rules such as Brier score, which incentivize calibration; deep-learning calibration is also covered by cited surveys.The passage cites Brier (1950), Gneiting and Raftery (2007), Gawlikowski et al. (2021), and Wang (2023).Source: §2 · p. 3

3 Preliminaries: Data, Models and Baseline

The paper builds a curated forecasting dataset from five platforms and evaluates models using simulated forecast dates and Brier score. In zero-shot tests, even the strongest model trails the human crowd aggregate.

3.4 Models are not naturally good at forecasting

  • GPT-4-1106-Preview achieved the lowest Brier score (.208), trailing the human crowd (.149); most models scored around or worse than random guessing (.25), while only GPT-4 and Claude-2 series beat baseline by >.02.Source: Table 2 · p. 4, §3.4 · p. 5

3.1 Dataset

  • The raw dataset contains 48,754 questions from five platforms; curation filters ill-defined or low-activity questions and yields 5,516 binary questions across train, validation, and test.The split contains 3,762 training, 840 validation, and 914 test questions; every test question was opened after June 1, 2023, after the base models’ training cutoff.Source: §3.1 · p. 4

3.2 Evaluation

  • The evaluation simulates forecasting by retrieving historical news between each question’s opening date and a simulated forecast date, using five geometrically spaced dates per question.Dates after a question resolves are excluded; the schedule retains 86% of retrieval dates on average.Source: §3.2 · pp. 4–5
  • Brier score evaluates binary probability forecasts as a strictly proper scoring rule, with .25 as the unskilled baseline; reported standard errors likely understate uncertainty because observations are time-series, not i.i.d.Scores are averaged across retrieval dates within each question, then across questions.Source: §3.2 · p. 5

3.3 Models

  • The baseline comparison evaluates 14 instruction-tuned language models without retrieval, using zero-shot and scratchpad prompting strategies.For each strategy, candidate prompts are selected on validation data and evaluated on the test set.Source: §3.3 · p. 5, §3.4 · p. 5

4 Our System

The system combines news retrieval with prompted forecasting, reasoning, and aggregation. Table 3 reports results across full-test and selective settings, where averaging system and crowd predictions is best in every setting.

4.1 Retrieval

  • Retrieval generates queries, fetches news, filters and reranks articles by relevance, then summarizes the top-ranked articles for forecasting.The system combines direct question queries with queries generated from decomposed sub-questions; it uses NewsCatcher and Google News and filters articles using an LM relevance rating.Source: §4.1 · pp. 5–6

4 Our System

  • In every Table 3 setting, the average of system and crowd predictions is the best-performing entry.The table covers all test questions and selective evaluations for uncertain crowd forecasts, early retrieval, at least five relevant articles, and all three criteria jointly.Source: Table 3 · p. 6

4.2 Reasoning

  • Reasoning prompts ask the model to rephrase the question, weigh arguments for and against the outcome, form an initial forecast, and check confidence against historical base rates.The paper motivates explicit reasoning as a way to handle forecasting beyond a single forward pass and to inspect explanations.Source: §4.2 · pp. 6–7

4.3 Ensembling

  • The ensemble combines six forecasts—three from the base model and three from the fine-tuned model—using a trimmed mean selected on validation performance.The base model uses the optimal and two next-best scratchpad prompts; the fine-tuned model is sampled three times.Source: §4.3 · p. 7

5 Optimizing the System

The paper fine-tunes a reasoning model on strong training forecasts and tunes retrieval and prompting choices on validation performance. Its optimization selects GPT-4-1106-Preview and a trimmed-mean ensemble, while the system is reported to be naturally well calibrated.

5.1 Fine-tuning a Reasoning Model

  • Fine-tuning selects forecasts that beat the crowd, limits deviations to 0.15, and averages retained predictions with crowd forecasts to mitigate overconfidence.From 73,632 generated reasonings, 13,253 meet the selection criteria; the model is fine-tuned on the 6,000 most recent points for two epochs.Source: §5.1 · pp. 7–8

5.2 Hyperparameter Sweep

  • GPT-4-1106-Preview has two years of more recent knowledge than the fine-tuned-eligible model but was unavailable for fine-tuning.Source: §5.2 · p. 8
  • Relevance ranking and presenting k = 15 article summaries achieve the lowest average Brier score in the first hyperparameter-optimization stage.Source: §5.2 · p. 9
  • The sweep retains three top reasoning prompts from 15 candidates, and the trimmed mean performs best among five evaluated ensemble methods.Source: §5.2 · p. 9
  • GPT-4-1106-Preview yields a 0.01-0.03 lower Brier score than Claude-2.1, so the final system uses GPT-4-1106-Preview and fine-tuned GPT-4-0613.Source: §5.2 · p. 9
  • The system is naturally well calibrated, and standard calibration methods such as binning and isotonic regression do not improve performance.Source: §5.2 · p. 9, Figure 3 · p. 9

6 Evaluations

The full system approaches the human crowd on the test set, while validation analyses identify settings where it is stronger or weaker. Selective forecasting and crowd-system ensembles improve on the crowd aggregate in reported settings.

6.1 System Nears Human Performance

  • The system’s test-set Brier score is .179 versus the crowd’s .149, and its accuracy is .715 versus the crowd’s .770.Its Brier score also significantly outperforms the best baseline model, GPT-4-1106-Preview, at .208.Source: §6.1 · pp. 9–10, Table 4 · p. 10
  • Across categories, the system nearly matches the crowd on Sports but falls behind on Environment & Energy; test-set calibration error is .42 versus .38 for the crowd.The authors caution that subcategory comparisons may be noisy because sample sizes are smaller.Source: §6.1 · p. 10

6.2 System Strengths and Weaknesses

  • On validation, the system’s Brier score is .199 versus .246 when crowd predictions are .3–.7; it also performs relatively better earlier and with more retrieved articles.The crowd outperforms the system on highly certain questions, and the system’s validation forecasts are described as well-calibrated.Source: §6.2 · pp. 10–11, Figure 4 · p. 11

6.3 System Beats Crowd in the Selective Setting

  • With all three selective conditions applied, the system’s Brier score is .240 versus .247 for the crowd, covering 22% of forecasts and 43% of questions.The authors report that the system beats the crowd by more than 1.5 standard errors in both Brier score and accuracy.Source: §6.3 · pp. 11–12

6.4 System Complements the Crowd

  • A weighted average giving the crowd four times the system’s weight improves full-test Brier score from .149 to .146.Using an unweighted average also outperforms the crowd when its prediction is between .3 and .7: .237 versus .24 Brier score.Source: §6.4 · p. 12

7 Ablations

Ablations show that fine-tuning and retrieval contribute to the system’s performance, while substituting a less capable fine-tuned model changes the score only slightly.

Fine-tuning a less capable model

  • Replacing fine-tuned GPT-4 with fine-tuned GPT-3.5 yields a Brier score of .182, compared with .179 for the original system.The GPT-3.5 model was fine-tuned on 13,253 samples.Source: §7 · p. 12

No fine-tuning

  • Without fine-tuning, the system’s Brier score rises to .186, an increase of .007 over the original score.The authors conclude that fine-tuning the reasoning model significantly boosts system performance.Source: §7 · p. 12

No fine-tuning and no retrieval

  • Without retrieval and fine-tuning, the system scores .206 Brier, near the .208 best baseline; reasoning prompts alone provide only a minor improvement.The ablated setup uses base GPT-4-1106-Preview without news retrieval.Source: §7 · p. 13

8 Conclusion

The authors present a retrieval-based forecasting system and self-supervised fine-tuning approach, alongside a dataset compiled from five forecasting competitions. They identify directions for expanding and improving forecasting systems and suggest that LM-based forecasting may reach competitive human levels.

8 Conclusion

  • The system combines a novel LM-based retrieval mechanism with self-supervised fine-tuning to generate reasonings with accurate predictions.The retrieval mechanism uses an LM to determine what information to source and how to evaluate its relevance.Source: §8 · p. 13
  • The released dataset is described as the largest and most recent forecasting dataset compiled from five real-world competitions.The release is intended to facilitate further research.Source: §8 · p. 13
  • With a larger training corpus, the proposed self-supervision can be iterated by generating more training examples from a model’s prior optimal predictions and reasoning.The process can be repeated until the training data is exhausted.Source: §8 · p. 13
  • The authors propose expanding training data with postdiction questions from later LMs or by reframing historical events as forecasting questions.They also suggest specializing models through fine-tuning on domain knowledge, based on observed category-level relationships with pretraining knowledge.Source: §8 · p. 13
  • Newer LM generations forecast better in the reported comparison: GPT-4-1106 outperforms GPT-4-0613 by .02 Brier score.The authors expect fine-tuning the newer model would improve performance, but describe this as an expectation.Source: §8 · p. 13
  • The authors suggest LM-based systems may soon produce accurate forecasts at competitive-human levels and help inform institutional decision making.Source: §8 · p. 13

A.1 Models

  • The model overview lists sources, weight availability, knowledge cutoffs, and evaluation costs; exact Mistral-series cutoffs are not publicly reported.Source: Table 6 · p. 17

A.2 Testing Potential Leakage from Post-training

The appendices report checks for post-training leakage, describe platform crowd aggregates, and document baseline prompt evaluation. Baseline models perform far below the human aggregate, and prompt selection uses validation-set results.

A.2 Testing Potential Leakage from Post-training

  • The authors report no evidence of post-training leakage in two tested models: both failed to answer or hallucinated on 20 post-cutoff events, while GPT-4 answered 20 pre-cutoff event questions correctly.Source: §A.2 · p. 17

A.3 Crowd Predictions

  • Crowd predictions are platform-specific aggregates, including Metaculus’s recency-weighted median, GJOpen’s mean of recent forecasts, and prediction-market prices on Manifold and Polymarket.INFER reweights individual forecasts over time, with exact aggregation details not found on its website.Source: §A.3 · pp. 17–18

B.1 Evaluation Method

  • Prompt selection compares candidate prompts on validation Brier scores; zero-shot uses the lowest average score, while scratchpad selection uses refusal counts when scores are similar.The winning zero-shot prompt averages 0.246 Brier; the winning scratchpad prompt averages 88 refusals, versus 106, 93, and 94 for the alternatives.Source: §B.1 · p. 18

B.2 Baseline Evaluation Results

  • All evaluated zero-shot and scratchpad models fall significantly far from the human aggregate on test-set Brier score.The caption reports a random baseline of 0.250 and human crowd score of 0.149.Source: Table 7 · p. 20

B.3 Knowledge Evaluation by Category

The section compares models’ knowledge of resolved questions across categories with their zero-shot forecasting performance, finding that the measures are correlated in several domains.

B.3 Knowledge Evaluation by Category

  • The observed knowledge-performance correlation suggests domain-adaptive training could improve forecasting in categories where models’ existing knowledge is limited.The authors present domain-adaptive training as a direction for future research.Source: §B.3 · pp. 20–21
  • The analysis measures pre-trained models’ accuracy on resolved training and validation questions, then compares it with category-level zero-shot Brier scores on the test set.The knowledge-accuracy comparison is in Table 8; test-set Brier scores are in Table 9.Source: §B.3 · p. 20, Table 8 · p. 20, Table 9 · p. 21
  • Knowledge accuracy on training and validation questions correlates with zero-shot test Brier scores across categories, including strong negative correlations in Politics & Governance, Arts & Recreation, and Education & Research.Table 10 marks absolute correlations of 0.3 or greater as high.Source: §B.3 · pp. 20–21, Table 10 · p. 21

C.1 Data Collection and Curation

The paper curates forecasting questions from multiple platforms, reports supplementary dataset statistics, and details a four-step system for retrieving relevant news articles.

C.1 Data Collection and Curation

  • Questions are gathered from forecasting platforms, categorized with GPT-3.5-Turbo, screened for ill-defined wording, and manually filtered for low quality or insufficient engagement.Manual review also removes ill-defined questions missed by the initial screening.Source: §C.1 · pp. 22–23

C.2 Further Statistics and Samples

  • Supplementary statistics map location mentions and describe platform distributions, sample records, community-prediction changes, question dates, and retrieval-date indices.The materials include a world map, platform statistics, sample data, prediction trajectories, and question and retrieval-date figures.Source: §C.2 · p. 23, Figure 10 · p. 24, Table 11 · p. 24

D Details about Our System

  • The system appendix specifies optimized settings, including hyperparameters identified through the Section 5.2 sweep.The appendix introduces further system details and their relation to the sweep.Source: D Details about Our System · p. 27

D.1 Retrieval System

  • For retrieval, GPT-4 generates search queries; news APIs return recent English-language articles, with linked background pages also scraped under date and source restrictions.The system combines generated queries with the question itself and restricts linked-page retrieval to whitelisted, timestamped sources within the retrieval range.Source: §D.1 · p. 27
  • GPT-3.5 rates article relevance from titles and the first 250 words, discards ratings of 3 or below, and summarizes retained articles for the forecasting question.The summary prompt includes the question and its background; long articles are truncated to fit the context window.Source: §D.1 · p. 27

D.2 Reasoning System

The system generates forecasts with prompted GPT-4 and a fine-tuned GPT-4 model, using retrieved article summaries as context.

D.2 Reasoning System

  • The system prompts GPT-4-1106-Preview with selected reasoning prompts and also uses a fine-tuned GPT-4 model, which needs no detailed scratchpad instructions.The fine-tuned model’s prompt consists of the question’s basic information followed by retrieval context.Source: §D.2 · p. 27, Figure 16 · p. 27
  • Claude-2.1 generated reasoning-prediction pairs for fine-tuning but is not directly used to generate forecasts in the system.Source: §D.2 · p. 27
  • Prompted-model scratchpads guide the reasoning path after providing the question information and retrieved article summaries.The illustrated scratchpad prompt is the one with the lowest validation Brier score.Source: §D.2 · p. 28, Figure 15 · p. 30

E.1 Hyperparameter Sweep

The sweep selects system settings and compares news retrieval and relevance-rating alternatives; example outputs illustrate both forecasts closer to the crowd and cases of overconfidence.

E.1 Hyperparameter Sweep

  • The top two search-query prompts produced average relevance ratings of 3.08 and 3.09; among articles rated at least 4, their average ratings were 4.37 and 4.38.Source: §E.1 · p. 31
  • On validation data, the best summarization prompt scored 0.193 versus 0.201 for the second-best; 15 article summaries ordered by relevance scored 0.177, and the best reasoning prompt scored 0.167.The article-count and ordering comparison used GPT-4-1106-Preview; two other top reasoning prompts scored 0.170 and 0.174.Source: §E.1 · p. 31
  • Trimmed mean achieved the lowest validation Brier score among the tested ensembling methods, while universal self-consistency did not improve on the baseline.Table 14 reports validation-set Brier scores across the methods; the baseline is the average Brier score of the input forecasts.Source: §E.1 · p. 31, Table 14 · p. 31

E.2 News API Evaluations

  • Google News and NewsCatcher had the highest aggregate article-relevance scores among five evaluated APIs, at 39 and 35, respectively.The evaluation used generated queries for 24 unresolved questions and restricted retrieval to the preceding 24 hours.Source: §E.2 · p. 32

E.3 Relevance Rating Approximations

  • The relevance-rating approximations compare article excerpts or embedding similarity with full-text GPT-4 ratings treated as gold labels.The excerpt method gives the model an article title and its first 250 words.Source: §E.3 · p. 32
  • Title-only relevance ratings can mislead when scraping leaves article text incomplete or returns an error message, so the authors regard that approach as unviable.Source: §E.3 · p. 32
  • At a threshold above 70% recall, title-plus-first-250-words ratings by GPT-3.5-Turbo had the highest precision and saved about 70% of article-processing cost.At threshold 4, this approach had recall 0.73 and precision 0.65; the comparison also tested Mixtral ratings and cosine similarity.Source: §E.3 · p. 32

F Example Outputs

  • In two NO-resolved examples, the model’s Reddit forecast of 20% was closer than the crowd’s 40%, and its Trump forecast of 55% was close to the crowd’s 60%.Source: Figure 18 · p. 34, Figure 20 · p. 36

Related papers

Loading 2402.18563v1…