Source-linked AI summary
Forecasting Future World Events with Neural Networks
Andy Zou, Tristan Xiao, Ryan Jia, Joe Kwon, Mantas Mazeika, Richard Li, Dawn Song, Jacob Steinhardt, Owain Evans, Dan Hendrycks
TL;DR
Forecasting matters for decisions, but automating expert judgment across real-world events remains an open challenge. Autocast pairs diverse tournament questions with date-organized news for leakage-free historical evaluation. Language models improve with scale and retrieval but remain well below human forecasters.
Problem
The paper asks whether language models can automate expert judgment in forecasting, a task whose forecasts inform policy and decision making.
Source: §1 · pp. 1–2Method
Autocast combines diverse forecasting-tournament questions with date-organized news to evaluate models using information available before historical outcomes.
Source: §1 · pp. 2–3Results
Language models improve with model size and information retrieval but remain far below human forecasts in accuracy and calibration; the best ML model gets 65% accuracy on binary outcomes versus 92% for humans.
Source: §1 · pp. 2–3Takeaways & Limitations
Autocast provides a challenging real-world test for ML forecasting, a capability that could support more informed decision making.
Source: §1 · pp. 1–2
Abstract
from arXiv · showhide
Forecasting future world events is a challenging but valuable task. Forecasts of climate, geopolitical conflict, pandemics and economic indicators help shape policy and decision making. In these domains, the judgment of expert humans contributes to the best forecasts. Given advances in language modeling, can these forecasts be automated? To this end, we introduce Autocast, a dataset containing thousands of forecasting questions and an accompanying news corpus. Questions are taken from forecasting tournaments, ensuring high quality, real-world importance, and diversity. The news corpus is organized by date, allowing us to precisely simulate the conditions under which humans made past forecasts (avoiding leakage from the future). Motivated by the difficulty of forecasting numbers across orders of magnitude (e.g. global cases of COVID-19 in 2022), we also curate IntervalQA, a dataset of numerical questions and metrics for calibration. We test language models on our forecasting task and find that performance is far below a human expert baseline. However, performance improves with increased model size and incorporation of relevant information from the news corpus. In sum, Autocast poses a novel challenge for large language models and improved performance could bring large practical benefits.
1 Introduction
Autocast brings together diverse tournament questions and date-organized news to test whether language models can reproduce forecasting under historically available information. The reported models remain far below strong human forecasts, although performance improves with model size and information retrieval.
1 Introduction
- Forecasting could support more informed decisions, but relying on scarce human expertise limits how many questions receive forecasts.The paper identifies information retrieval, reasoning, and quantitative modeling as tasks ML could help automate.Source: §1 · pp. 1–2
- Autocast collects thousands of public-interest questions from forecasting tournaments across topics, answer formats, and horizons ranging from days to decades.Its questions have a strong human baseline from the crowd aggregate of competitive forecasters.Source: §1 · p. 2
- A date-organized news corpus enables historical retrodiction using only information available before the forecast outcomes, avoiding future-information leakage.The corpus is curated from Common Crawl news items.Source: §1 · pp. 2–3
- 65% accuracy for the best ML model on binary outcomes versus 92% for humans shows a substantial gap from aggregate human forecasts.The random baseline is 50%.Source: §1 · p. 2
2 Related Work
Prior work motivates forecasting through information retrieval and calibrated probabilities, while identifying gaps in question quality and evaluation. The paper positions Autocast against these efforts and links forecasting to truthful answers on questions that eventually become verifiable.
2 Related Work
- ForecastQA uses crowdworker-written questions that the authors describe as often ambiguous or nonsensical, unlike Autocast’s experienced-forecaster questions and real expert forecasts.The passage also contrasts Autocast’s real forecasting baselines with ForecastQA’s retrospective human baseline.Source: §2 · p. 3
- Forecasting benefits from current, specialized information across sources, motivating an architecture for comparing multiple news articles daily, though article length and time horizon are limited.The related work connects this design to Wang and McAllester (2020).Source: §2 · pp. 3–4
- Prior calibration work mostly addresses classification with fixed classes, while Autocast also requires predictions of continuous quantities spanning multiple orders of magnitude.The passage says this numerical forecasting setting had not been explored in prior work.Source: §2 · p. 4
- Forecasting questions are challenging but eventually verifiable, unlike answers to some open problems in philosophy or science that humans may struggle to evaluate.The passage frames forecasting as relevant to truthful answers on questions that people find difficult or costly to answer.Source: §2 · p. 4
3 The Autocast Dataset
Autocast assembles tournament forecasting questions with crowd-forecast histories and date-organized news to support evaluation under historical information conditions. Its questions span topics and horizons; analyses show changing category mixes and improving crowd forecasts as resolution approaches.
3 The Autocast Dataset
- Autocast contains 6,707 tournament questions with clear resolution criteria, broad public interest, and true/false, multiple-choice, numerical, or date answers.Participants update forecasts from each question’s start date until its close date.Source: §3 · p. 4
- Questions include start and close dates, answers, crowd-forecast time series, and metadata; half are unresolved, while the test set contains resolved questions.Unresolved questions can support training through crowd forecasts as proxies for ground truth; Figure 4 depicts daily GPT-2 forecasts using retrieved news and targets combining crowd predictions with resolved outcomes.Source: §3 · p. 5, Figure 4 · p. 5
- A mid-2021 date cutoff places test questions’ resolutions between mid-2021 and mid-2022, avoiding random splits that fail to simulate forecasting chronology.Pretraining on data after mid-2021 would also fail to simulate forecasting faithfully.Source: §3 · p. 5
- The accompanying CommonCrawl news corpus is organized by publication date and contains more than 200GB of articles from 2016 to mid-2022.Date organization supports historical evaluation using information available at the time of a forecast.Source: §3 · p. 5
3.1 Dataset Analysis
- Questions cover five main categories and 44 subcategories across a wide geographical range, testing both subject breadth and operationalized quantitative depth.Categories include Economy, Politics, Science, Social, and Other.Source: §3.1 · p. 6
- Autocast is updated periodically as tournament questions grow, adding training data and fresh test questions for assessing overfitting.If current trends continue, twice as many questions will be available in two years.Source: §3.1 · p. 6
- Human crowd forecasts become more accurate from question start to close, and supplementary results show they are well-calibrated.Probabilistic crowd judgments enable calibration evaluation.Source: §3.1 · p. 6
- Question-category distributions shift over time: Social questions increase from 12.6% of training questions to 28.2% of test questions.The authors suggest the increase may be due to the COVID-19 pandemic.Source: §3.1 · p. 6
4 Experiments
Experiments compare forecasting baselines and retrieval-augmented models using question-type metrics and a combined score. Retrieval substantially improves performance over models without retrieval, though all remain below human forecasts; analysis examines retrieved evidence and performance over time.
4.1 Baselines
- The Crowd baseline uses the final aggregate human forecast, while Random uses analytic accuracy for categorical questions and uniform sampling for bounded numerical answers.UnifiedQA-v2 is evaluated zero-shot on classification; T5 is fine-tuned, with an additional linear head for numerical answers.Source: §4.1 · pp. 6–7
- Retrieval methods encode BM25-retrieved articles with FiD and cross-encoder reranking, excluding articles published after question close or resolution.FiD Static uses the top 10 reranked articles; FiD Temporal uses daily top-article embeddings with GPT-2 and auxiliary crowd-prediction supervision.Source: §4.1 · p. 7
4.2 Metrics
- The combined Score is (T/F + MCQ − Numerical)/2, with categorical accuracy and numerical ℓ1 distance bounded between 0% and 100%.A perfect combined score is 100%, and its lower bound is −50%.Source: §4.2 · p. 7
4.3 Forecasting Evaluation
- Retrieval-based methods raise Average score by 93% over UnifiedQA-v2 and 15% over T5, and become more effective with larger models.UnifiedQA-v2 is close to random zero-shot performance, while fine-tuned T5 performs better but remains near chance on multiple-choice questions.Source: §4.3 · p. 8
- FiD Temporal has a slightly higher Average score than FiD Static, but the largest FiD Static model has the highest individual score.The authors conclude that temporal auxiliary crowd-prediction training neither harms nor helps relative to static retrieval.Source: §4.3 · p. 8
4.4 Model Analysis
- Retrieved articles are often relevant and provide information unavailable from the baselines’ pre-2020 T5 pretraining, though more sophisticated retrieval may raise efficiency concerns.The paper suggests better integration of retrieved articles could yield large improvements.Source: Figure 5 · p. 7, §4.4 · p. 8
- Prediction scores rise toward resolution for both the crowd and an ensemble of the two largest FiD Temporal models; Science & Technology is the most challenging category for models.The authors suggest that more relevant information becomes available nearer resolution; the ensemble has slightly higher final performance than individual models.Source: Figure 6 · p. 8, §4.4 · p. 8
5 Calibrated Prediction of Numerical Quantities
The section introduces IntervalQA for calibrated numerical prediction across a wide range of values, then evaluates confidence-interval prediction with DeBERTa-v3 models. All three reported metrics decrease with model size.
5.2 Experiments
- All three metrics decrease with model size; Table 3 reports lower RMS Calibration Error and median interval size for larger DeBERTa-v3 models.The three metrics are RMS Calibration Error, Point Estimate Distance, and Conf. Interval Length.Source: Table 3 · p. 9, §5.2 · p. 9, §5.1 · p. 9
5 Calibrated Prediction of Numerical Quantities
- IntervalQA contains about 30,000 numerical-answer questions drawn from NLP datasets, with answers spanning orders of magnitude; its questions are estimation tasks, not forecasts.The dataset addresses calibration for numerical quantities, a setting with fewer than 1,000 numerical training questions in Autocast.Source: §5 · pp. 8–9
5.1 Metrics
- RMS Calibration Error assesses whether confidence intervals contain targets at their stated rates across the output range; the section also reports point-estimate error and interval length.The metric uses fixed confidence levels from 50% to 95%.Source: §5.1 · p. 9
5.2 Experiments
- The experiments fine-tune DeBERTa-v3 to predict a point estimate and confidence intervals using point-estimate, boundary, and interval-length losses.Models are trained for five epochs with batch size 100.Source: §5.2 · p. 9
6 Conclusion
The paper introduces Autocast and IntervalQA and evaluates language-model baselines for forecasting and numerical calibration. Model size and information retrieval improve forecasting performance, while the results leave substantial room for improvement.
6 Conclusion
- Autocast pairs thousands of tournament forecasting questions and aggregated human predictions with dated Common Crawl news for evaluations that avoid future-information leakage.The dataset measures neural networks’ ability to forecast future world events.Source: §6 · p. 9
- Larger models and information retrieval improve forecasting performance, and the paper concludes that substantial room for future improvement remains.The conclusion also reports evaluating state-of-the-art language models on IntervalQA for numerical calibration.Source: §6 · p. 9
- IntervalQA is a collection of numerical prediction questions with targets spanning a wide dynamic range, intended to support evaluation of calibration.Source: §6 · p. 9
A.1 Autocast Experiments
The appendix compares calibration of crowd forecasts and model variants, and describes the FiD Temporal architecture and news retrieval setup. Crowd-informed FiD Temporal forecasts have lower calibration error than FiD Static forecasts.
A.1 Autocast Experiments
- Figure 8’s crowd calibration curve plots the fraction of positive outcomes against predicted probability for resolved true/false questions.Source: §A.1 · p. 14
- 8% calibration error for the human crowd compares with 40% for the largest FiD Static model, while crowd predictions reduce FiD Temporal calibration error to 17%.Source: §A.1 · p. 14
- FiD Temporal uses separate output heads for true/false, multiple-choice, and numerical questions, and normalizes each task’s loss by its uniform-random baseline before summing.The multiple-choice head has 12 outputs, and averaging loss over each question’s prediction sequence gives questions equal weight.Source: §A.1 · p. 14
- For FiD Temporal, the system retrieves the top article from each active day, then aggregates and ranks those articles; FiD Static uses the top 10 ranked articles.Source: §A.1 · p. 14
A.2 Confidence Intervals
The appendix describes nested confidence-interval construction, a three-part training loss, and a local calibration metric designed for targets spanning a large dynamic range.
A.2 Confidence Intervals
- Intervals start from a point estimate and add non-negative, asymmetric deltas, making them nested and wider at higher confidence levels.Source: §A.2 · p. 14
- The baseline loss combines point-estimate MSE, boundary MSE for interval bounds on the wrong side of targets, and an interval-length penalty.Labels are log-normalized, and the boundary or length penalty is activated according to whether empirical containment is below or above the target confidence level.Source: §A.2 · p. 14, Figure 7 · p. 15
- Adaptive RMS calibration sorts examples by target value into fixed-size bins, computes local calibration error across confidence levels, and averages it.The metric is intended to measure calibration across a large dynamic range of output values.Source: §A.2 · pp. 14–15
B.1 Dataset Details
Autocast’s appendix describes a broad forecasting dataset and reports category-level gaps between models and crowd forecasts. Its risk discussion presents forecasting as potentially useful for institutional decision-making while acknowledging uncertainty about its relationship to general AI capabilities.
B.1 Dataset Details
- 6,707 unique questions expand to 9,757 after true/false balancing; the train/test split separates questions resolved before and after 5-11-2021.The training set contains 4,411 examples and the test set 1,292; the passage says the baselines use no unresolved questions, avoiding leakage.Source: §B.1 · pp. 16–17, Table 6 · p. 17
- Science & Technology questions are hardest for FiD Static and FiD Temporal; crowd predictions perform similarly across categories but remain below a perfect score.The passage also reports a substantial gap between model and crowd predictions.Source: §B.1 · p. 17
C X-Risk Sheet
- Individual responses to the x-risk checklist do not decisively establish whether this work is relevant to existential-risk reduction.Source: C X-Risk Sheet · p. 18
C.1 Long-Term Impact on Advanced AI Systems
- The checklist frames forecasting as potentially improving institutional decision-making and epistemics, while countering eroded epistemics and potentially lowering conflict risk.It describes the conflict-risk effect as diffuse and says automated tools could assist institutions at several levels.Source: §C.1 · p. 18
C.2 Safety-Capabilities Balance
- The relationship to general capabilities is unclear; the authors suggest focusing on forecasting systems and stopping if capability externalities prove difficult to avoid.They also note that better forecasting could improve general capabilities or raw AI power.Source: §C.2 · p. 19
C.3 Elaborations and Other Considerations
The discussion distinguishes forecasting known probabilities from discovering overlooked risks, and cautions that forecasting tools may be poorly suited to tail risks. It also describes ways forecasts could encourage risky behavior or delay decisions.
C.3 Elaborations and Other Considerations
- The authors say human forecasts are unnecessary for training if resolutions are available, while human forecasters provide richly annotated training data.They describe the intended endpoint as automated systems that do not depend on human reliability.Source: §C.3 · p. 19
- Forecasting competitions refine estimates of known unknowns, while the authors argue that surfacing ignored considerations may offer greater risk-reduction value.They describe brainstorming tools as a complementary way to bring unknown unknowns onto the radar.Source: §C.3 · p. 20
- Forecasting metrics may not sufficiently penalize ignored tail risks or reward prescience about Black Swans, limiting their fit for tail-risk settings.The passage distinguishes short-run probability judgments from contingency planning for systemic tail risks.Source: §C.3 · p. 20
- Uncertain or low-risk forecasts could encourage leaders to delay action or treat risk as absent, potentially increasing risky behavior.The passage cites automation bias and risk compensation as possible mechanisms.Source: §C.3 · p. 20