PaperlayerView exact revision on arXiv

Source-linked AI summary

Future Is Unevenly Distributed: Forecasting Ability of LLMs Depends on What We're Asking

Chinmay Karkar, Paras Chopra

arXiv:2511.18394v1cs.LG

TL;DR

The paper addresses limited understanding of which forecasting questions LLMs answer confidently and where they fail. It evaluates multiple model families on real-world questions with and without time-bounded news context, finding that forecasting performance varies by domain and prompt framing.

Abstract

from arXiv · show

Large Language Models (LLMs) demonstrate partial forecasting competence across social, political, and economic events. Yet, their predictive ability varies sharply with domain structure and prompt framing. We investigate how forecasting performance varies with different model families on real-world questions about events that happened beyond the model cutoff date. We analyze how context, question type, and external knowledge affect accuracy and calibration, and how adding factual news context modifies belief formation and failure modes. Our results show that forecasting ability is highly variable as it depends on what, and how, we ask.

Introduction

LLM forecasting remains less explored than their performance on other benchmarks, and prior work leaves unclear which questions models answer confidently or where they fail. This paper evaluates multiple model families across domains, with and without news, and categorizes recurrent failure modes.

Introduction

  • Prior forecasting studies leave unclear which questions LLMs answer confidently and where they fail, despite the growing use of AI in high-stakes decisions.The paper frames this as a gap in understanding how to interpret apparent forecasting success.Source: Introduction · p. 1
  • The evaluation compares reasoning-optimized and non-reasoning LLMs using qualitative analysis, accuracy, and Brier score.It also examines behavior when factual news context is added before prediction.Source: Introduction · p. 1
  • The study analyzes forecasting across multiple domains with and without contextual news inputs.Source: Contributions · p. 1
  • The authors categorize recurrent forecasting failure modes, particularly those emerging when contextual information is introduced.They focus on cases where reasoning and calibration diverge.Source: Contributions · p. 2

Related Work

Related work spans forecasting tournaments, formal benchmarks, temporal-reasoning datasets, and market-facing tools. This paper distinguishes its analysis by examining news-related failure modes and performance differences across question categories.

Related Work

  • Early forecasting tournaments found unassisted GPT-4 underperformed aggregate human forecasters, while later fine-tuning and reinforcement-learning efforts reported measurable gains.Related studies include human-comparable accuracy and direct comparisons with expert forecasters (Halawi et al. 2024; Lee et al. 2025; Lu 2025).Source: Related Work · p. 2
  • Metaculus and ForecastBench use real prediction-market questions, while related work also examines calibration, uncertainty, logical leakage, retrieval reliability, and cutoff-related contamination.The cited benchmarks include Metaculus 2025a and ForecastBench (Karger et al. 2025).Source: Related Work · p. 2
  • ForecastQA (Jin et al. 2021), Autocast (Zou et al. 2022), ExpTime (Yuan, Ding, and Vlachos 2024), FOReCAst (Yuan, Ding, and Vlachos 2025), and FutureX (Zeng et al. 2025) assess long-horizon prediction and confidence under streaming updates.Source: Related Work · p. 2
  • Metaculus Forecasting Tools (Metaculus 2024) and ManifoldBot (Microprediction 2024) connect LLM agents to market-style systems; this study instead analyzes news-related failures and category differences.Source: Related Work · p. 2

Methodology

The methodology builds a filtered, categorized forecasting benchmark and evaluates several LLMs on balanced questions, with and without time-bounded news context. Predictions are assessed for correctness and probabilistic calibration.

Data Processing

  • The authors collected about 10,000 prediction-market questions from January to July 2025, selecting questions beyond the models’ cutoff dates and noting that many were noisy or hyper-localized.Source: Data Processing · p. 2
  • After low-liquidity filtering, Gemini 2.5 Flash classified questions into six categories with five subcategories each, and a second judging pass excluded localized or non-forecasting items.The final curated dataset contained 392 questions, with metadata including creation time, resolution time, and final resolution probability.Source: Data Processing · pp. 2–3

Evaluation Methodology

  • The evaluation uses a uniform, seed-42 sample of 150 questions with equal numbers per category for cross-category comparison.Source: Evaluation Methodology · p. 3
  • GPT-5, GPT-4.1, DeepSeek-R1, and Claude 3.7 Sonnet receive the question and creation date without external tools, retrieval, or web search.Sampling used temperature 0.0 and a 4,500-token maximum.Source: Evaluation Methodology · p. 3
  • Accuracy measures resolution matches, while Brier score and ECE assess probabilistic calibration, with ECE comparing accuracy and confidence across bins.Source: Evaluation Methodology · p. 3

Evaluation with News Context

  • For the news condition, up to ten snippets retrieved by question text are bounded by the question’s creation date, and post-date leakage was removed.Models are then scored with the same accuracy, Brier score, and ECE metrics.Source: Evaluation with News Context · p. 3

Results by Category

Without news context, performance varies by both model and domain: GPT-5 and Claude 3.7 show strong calibration in structured domains, while DeepSeek-R1 and GPT-4.1 have higher ECE in noisier domains.

Results by Category

  • GPT-5 and Claude-3.7 achieve strong calibration in Geopolitics and Politics, while DeepSeek-R1 and GPT-4.1 show higher ECE in noisier Entertainment and Technology domains.Table 1 reports category-wise accuracy, Brier Score, and ECE without news context.Source: Results by Category · p. 3

News-Augmented Forecasting

Adding up to ten time-bounded news snippets improves some category-level metrics but worsens others. The reported news-context results contrast gains in Finance and Sports with declines in Entertainment and Technology.

News-Augmented Forecasting

  • Finance and Sports benefit from news context, with lower Brier scores and improved ECE.The supplied passage does not report the corresponding numeric values.Source: News-Augmented Forecasting · p. 3
  • Entertainment and Technology show declines with news context, consistent with recency bias and noise amplification.Source: News-Augmented Forecasting · p. 3
  • The no-news baseline reports category-wise Accuracy, Brier Score, and ECE, averaged over 25 questions per category.Source: Table 1 · p. 4

Analysis and Failure Modes with Context

News context can clarify question scope and surface useful signals, but it also introduces retrieval leakage and several ways for models to shift toward misleading information. The reported failure modes include definition drift, recency bias, and rumour overweighting.

Analysis and Failure Modes with Context

  • News retrieval sometimes includes post-cutoff articles that effectively reveal the question’s resolution, despite temporal bounds.Source: Analysis and Failure Modes with Context · p. 4
  • News context can clarify time scope and surface relevant signals, but is also prone to multiple failure modes.Source: Analysis and Failure Modes with Context · p. 4
  • Recent headlines can outweigh historical trends, sometimes turning a correct forecast into an incorrect one.In the S&P 500 example, news shifted a correct NO forecast to an incorrect YES; headlines dominated the model’s prior and produced an overconfident breakout bet.Source: Analysis and Failure Modes with Context · p. 4
  • Models may anchor on unverified speculation in retrieved news and switch from a correct to an incorrect resolution.The tariff example describes a high-confidence NO changing to a high-confidence YES based on headlines about a possible increase.Source: Analysis and Failure Modes with Context · p. 4
  • News can shift a model’s interpretation of an acronym, changing its semantic grounding and producing an incorrect prediction.For MATS, snippets about a trucking expo displaced the academic-program interpretation and led to a misplaced NO.Source: Analysis and Failure Modes with Context · pp. 4–5
  • DeepSeek-R1 returns answer and confidence tags but does not verbalize reasoning traces, even when prompted to do so.Source: Analysis and Failure Modes with Context · p. 5

Conclusion

The conclusion characterizes LLM forecasting as uneven and dependent on data coverage and prompt framing. The accompanying appendix passages specify the forecasting prompts and the question-classification and filtering stages.

Conclusion

  • Forecasting competence is uneven, and news can improve or worsen accuracy through definition drift, rumour anchoring, and recency bias.The conclusion frames future-reasoning ability as conditional and calls for benchmarks that distinguish knowledge recall from probabilistic inference.Source: Conclusion · p. 5

A.1 Core Forecasting Prompt (No News)

  • The no-news prompt asks models to rephrase the question, weigh reasons for both YES and NO, aggregate considerations, and output a probability and confidence.Source: §A.1 · pp. 5–6

A.2 News-Augmented Forecasting Prompt

  • The news-augmented prompt keeps the base forecasting prompt and appends up to ten time-bounded news snippets before inference.Source: §A.2 · p. 6

A.3 Category Classification Judge Prompt

  • The category-classification prompt assigns questions to specified topical categories and subcategories, or marks them irrelevant.Source: §A.3 · pp. 6–7

A.4 Refined Filtering Judge Prompt

  • The refined filtering prompt classifies prediction-market questions by whether they concern public, verifiable outcomes, excluding personal or unverifiable matters.Source: §A.4 · p. 7

B.1 Recency Bias Example: “S&P 500 above $6050 on June 13?”

For the S&P 500 question, two forecasts made different calls: one estimated a 0.34 probability of YES and answered NO, while the other estimated 0.55 before adjustment and answered YES with 0.54 confidence.

B.1 Recency Bias Example: “S&P 500 above $6050 on June 13?”

  • 0.34 was the first response’s probability for YES; it answered NO with 0.66 confidence.Its scenario analysis placed odds near one-third for YES.Source: §B.1 · p. 8
  • The first response estimated scenario-dependent YES odds of 10%, 25–40%, or 50–65% as the starting level rose from 5,800 to at least 6,000.Source: §B.1 · p. 8
  • 0.55 was the second response’s initial YES probability; it adjusted to 0.54 confidence and answered YES.Its reasoning cited the index near 6,000 and a possible 0.8% move above 6,050 over two sessions.Source: §B.1 · p. 8

B.2 Rumour Overweighting: “Tariffs on China above 150% by end of June?”

Two forecasts of whether tariffs on China would exceed 150% by June 30 both answered no, but used different interpretations and confidence levels. One treated broad headline tariffs as the main question; the other emphasized definitional ambiguity, including product-specific duties.

B.2 Rumour Overweighting: “Tariffs on China above 150% by end of June?”

  • Both forecasts answered NO, but confidence differed: 0.88 for response 1 and 0.58 for response 2.Both predictions were marked correct in the example.Source: §B.2 · p. 8
  • Response 1 treated the question as ambiguous between broad tariffs and product-specific duties, noting that some anti-dumping or countervailing rates could exceed 150%.It judged broad headline escalation unlikely in the short timeframe absent a major trigger or imminent trade case.Source: §B.2 · pp. 8–9
  • Response 2 focused on a U.S. headline tariff above 150%, weighing inflation, legal risks, and the short window against the administration’s ability to act quickly.It noted that exactly 150% would not meet the question’s greater-than-150% threshold.Source: §B.2 · pp. 9–10

B.3 Definition Drift: “Will MATS applications open in March?”

The two forecasts for whether MATS applications would open in March diverged over what “MATS” and “applications” referred to. One forecast treated it as a recurring academic program; the other focused on the Mid-America Trucking Show and its registration timing.

B.3 Definition Drift: “Will MATS applications open in March?”

  • The forecasts disagreed: response 1 answered YES with 0.58 confidence and was correct, while response 2 answered NO with 0.35 confidence and was incorrect.The example records different interpretations of the acronym and application timeline.Source: §B.3 · p. 10
  • Response 1 treated MATS as a recurring program whose summer-cohort applications had typically opened around March, while allowing for schedule changes and acronym ambiguity.Its considerations favored March based on prior cycles but reduced confidence because the program was unclear.Source: §B.3 · p. 10
  • Response 2 considered the Mid-America Trucking Show, where exhibitor and attendee processes generally open months before the March event, making a March opening atypical.It also noted that “applications” might not accurately describe attendee registration.Source: §B.3 · p. 11

B.4 Missing Reasoning Traces in DeepSeek-R1

For the Washington Commanders–Atlanta Falcons question, both displayed DeepSeek-R1 predictions answered YES at 0.7 confidence, matching the recorded YES resolution.

B.4 Missing Reasoning Traces in DeepSeek-R1

  • Both displayed predictions answered YES at 0.7 confidence, and the recorded resolution was YES.The example marks both predictions correct.Source: §B.4 · p. 11
  • The recorded outcome for the Week 17 Washington Commanders home game against the Atlanta Falcons was YES.The example identifies the match as December 28, 2024.Source: §B.4 · p. 11
  • The displayed response text contains the final YES answer and 0.7 confidence for each prediction.Source: §B.4 · p. 11
Loading 2511.18394v1…