PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Open-Ended Reasoning to Predict the Future

Nikhil Chandak, Shashwat Goel, Ameya Prabhu, Moritz Hardt, Jonas Geiping

arXiv:2512.25070v2cs.LGcs.CL

TL;DR

The paper addresses scalable forecasting under uncertainty by creating open-ended questions from global news and training language models on them. OpenForecaster 8B becomes competitive with much larger proprietary models, while forecasting training improves accuracy, calibration, and consistency.

  • Problem

    High-stakes decisions require forecasting under uncertainty, but scalable training data for open-ended forecasting is difficult because resolved outcomes provide slow feedback and binary questions yield noisy rewards.

  • Method

    The authors automatically construct OpenForesight from global news, use offline retrieval to prevent leakage, and train Qwen3 thinking models with reinforcement learning using accuracy and Brier-score rewards.

  • Results

    OpenForecaster 8B becomes competitive with much larger proprietary models, improving accuracy, calibration, and long-term consistency across held-out and external forecasting evaluations.

  • Takeaways & Limitations

    Forecasting-specific training can improve language-model prediction quality and calibration beyond the training distribution while enabling scalable open-ended forecasting research.

  • Takeaways & Limitations

    The evaluation uses model weights frozen in April 2025 and tests forecasts resolved between May and August 2025.

Abstract

from arXiv · show

High-stakes decision making involves reasoning under uncertainty about the future. In this work, we train language models to make predictions on open-ended forecasting questions. To scale up training data, we synthesize novel forecasting questions from global events reported in daily news, using a fully automated, careful curation recipe. We train the Qwen3 thinking models on our dataset, OpenForesight. To prevent leakage of future information during training and evaluation, we use an offline news corpus, both for data generation and retrieval in our forecasting system. Guided by a small validation set, we show the benefits of retrieval, and an improved reward function for reinforcement learning (RL). Once we obtain our final forecasting system, we perform held-out testing between May to August 2025. Our specialized model, OpenForecaster 8B, matches much larger proprietary models, with our training improving the accuracy, calibration, and consistency of predictions. We find calibration improvements from forecasting training generalize across popular benchmarks. We open-source all our models, code, and data to make research on language model forecasting broadly accessible.

1 Introduction

The paper asks how language models can improve open-ended forecasting, addressing limited scalable training data and noisy binary-question rewards. It introduces automated news-based data creation, leakage controls, retrieval, and RL, reporting competitive accuracy and calibration from an 8B forecaster.

  • The paper studies how to train language models to better forecast open-ended questions under uncertainty.
  • Open-ended forecasting data is difficult to scale because detailed reasoning traces are scarce, event resolution is slow, and binary questions produce noisy rewards.
  • An automated pipeline synthesizes open-ended questions from global news, using one model for generation and another for filtering and leakage-resistant rewriting.
  • The system uses static monthly news snapshots, trains only on events through April 2025, and withholds the diverse May–August 2025 test set until final evaluation.
  • Retrieval improves forecasting despite stopping retrieved articles one month before resolution, while combining accuracy with an adapted Brier reward avoids the calibration or exploration weaknesses of single objectives.
  • On held-out and external benchmarks, the specialized 8B model is competitive with much larger proprietary models in accuracy and calibration, with improved long-term consistency and transferable calibration.
  • The authors release the dataset, code, and models to support accessible research on language-model forecasting.

2 Related Work

Prior work mainly studies numeric or time-series forecasting, while language-model forecasting often uses retrieved web information and prediction-market questions. This paper instead emphasizes natural-language world-event forecasting with static news data to reduce leakage and distributional concerns.

  • The paper focuses on discrete world events expressed as natural-language questions and answers, rather than numeric or time-series forecasting.
  • Prior language-model forecasting studies commonly source questions from prediction markets, whose questions are mostly binary yes-or-no outcomes.
  • Forecasting systems benefit from information available before events resolve, so prior work often retrieves articles through web-search APIs.
  • The paper uses static monthly global-news snapshots instead, addressing concerns that online search can leak outcomes and prediction-market questions can carry distributional biases.
  • Prior reinforcement-learning and supervised-finetuning work supports language-model reasoning and forecasting, but this paper targets uncertainty-aware forecasting rewards.

3 Open-Ended Forecasting

Open-ended forecasting permits arbitrary natural-language questions and unstructured answers, including possibilities that are difficult to enumerate. The paper defines semantic accuracy and an adapted Brier score for evaluating both correctness and probability calibration.

  • Open-ended forecasting allows arbitrary natural-language questions and does not require a predefined structured outcome set.
  • This format can capture unexpected possibilities that binary or multiple-choice prediction markets may exclude, including scientific, geopolitical, and technological events.
  • The setup asks a model to produce a short textual answer y and probability q for a resolved question with known outcome y⋆.
  • Accuracy is measured by an evaluator model that checks semantic equivalence between the attempted answer and the ground-truth outcome.
  • Measuring Calibration: The adapted Brier score gives positive scores to correct predictions and negative scores to incorrect predictions, while rewarding both accuracy and truthful probability reporting.
  • Training Algorithm: Training uses outcome-based GRPO reinforcement learning with sampled completions and rewards based on answers, probabilities, and resolved outcomes.
  • Initial Policy: The initial policy is Qwen3 Thinking 8B, trained through April 2025 and evaluated on questions resolving from May through August 2025.

4 Generating Open-Ended Forecasting Questions from News

OpenForesight converts daily news into filtered, leakage-checked open-ended forecasting questions through an automated generation and selection pipeline. Filtering removes ambiguous or answer-leaking samples, while scaling the resulting dataset improves forecasting performance.

  • Question Generation: The pipeline generates up to three short-answer forecasting samples per article, each containing a question, background, resolution criteria, answer, and source link.
  • Filtering: A sample selector keeps questions that are article-grounded, forward-looking, definite, unambiguous, and resolvable by the publication date.
  • Leakage Control: The pipeline rewrites or rejects samples whose titles, backgrounds, or resolution criteria reveal the answer, then applies string matching to remove remaining direct leakage.
  • Dataset Construction: The same methodology creates train, validation, and test splits from different news sources to assess generalizable forecasting skills.
  • Filtering Results: 60% of question-answer candidates are marked invalid, while 39% of articles with multiple valid questions are reduced by selecting one question.
  • Leakage Results: Over 40% of selected questions directly contain the answer string, and rewriting plus rejection removes approximately 90% of these cases before final string filtering.
  • Filtering Results: Applying all filtering stages yields higher accuracy and Brier score than leakage removal alone, using one-third the data and training steps.
  • OpenForesight: The final dataset contains 52K samples from unique articles, and Qwen3-32B achieves 95% accuracy when validating generated questions against their source articles.

5 Prediction System

The prediction system combines leakage-aware retrieval with reinforcement-learning reward design to improve forecasting accuracy, calibration, and exploration. Validation results favor an accuracy-plus-Brier reward and five retrieved news chunks.

  • Reward Design: Adding an accuracy term improves exploration on hard, low-confidence questions where Brier score provides little correctness signal.The proposed reward produces “Unknown” in about 4% of samples, versus about 40% for Brier-only training.
  • Reward Design: Accuracy + Brier score achieves the best validation performance on both accuracy and Brier score.It improves accuracy beyond Brier-only training while maintaining equal Brier score.
  • Retrieval: The system uses offline news retrieval restricted to articles published at least one month before each question’s resolution date.This design provides relevant evidence while reducing future-information leakage.
  • Retrieval: Retrieval improves accuracy by 9–18% across model families and sizes, with gains plateauing after five chunks.The system therefore fixes k = 5 retrieved chunks for subsequent evaluations.

6 Final Results

Held-out evaluations show that OpenForesight training improves forecasting accuracy, calibration, and long-term consistency. OpenForecaster8B competes with much larger models on open-ended and FutureX benchmarks, while calibration gains transfer to other domains.

  • Evaluation Setup: The evaluation uses held-out questions from May to August 2025 and the external FutureX benchmark, with models evaluated using retrieval where applicable.The paper describes these tests as unseen during development to avoid decisions based on future information.
  • Open-ended Test Set: OpenForecaster8B outperforms GPT OSS 120B on Brier score and beats Qwen3 235B on accuracy in the held-out open-ended test set.Training also improves other model families, including a 25% absolute accuracy gain for Llama 3.1 8B Instruct.
  • External Benchmarks: On FutureX, the trained model has the strongest accuracy by a large margin and is close to the best on Brier score.Training also improves long-term consistency by 44% on arbitrage metrics and 19% on frequentist metrics.
  • Calibration Generalization: Forecasting training improves calibration on both the held-out test set and out-of-distribution benchmarks.The reported benchmarks include SimpleQA, GPQA-Diamond, and MMLU-Pro.

7 Conclusion

The paper presents scalable training for open-ended forecasting and adapts the Brier score to free-form answers. Its scoring setup rewards correct, calibrated predictions but assumes probability is concentrated on one reported answer, a limitation the authors acknowledge.

  • Scalable open-ended forecasting is the paper’s central contribution, supported by a free-form Brier-score framework for training and evaluation.The framework evaluates a predicted answer and its assigned probability rather than restricting outcomes to a predefined binary or multiclass set.
  • Correct predictions receive positive scores, incorrect predictions negative scores, and score magnitude grows quadratically with confidence.Assigning q = 0 yields a baseline score of 0 regardless of the predicted answer, with the reported score ranging from −1 to 1.
  • The adapted score is equivalent to a proper reward that encourages both accuracy and truthful probability reporting for the most likely answer.The paper distinguishes this free-form score from the traditional binary Brier score while relating it to prior forecasting work.
  • The free-form scoring simplification assumes zero probability for semantically different answers, although forecasters may assign probability to multiple guesses.The authors leave reporting all nonzero-probability guesses for future work.

B.1 Ablation: Using Prediction Market Binary Data

The ablation compares binary-only, free-form-only, and mixed supervision under matched data budgets. Free-form supervision is most useful for open-ended evaluation, while mixing formats provides the strongest cross-format trade-off.

  • 22.4% accuracy and 0.086 free-form Brier score follow free-form-only RL, improving from 19.3% and −0.009.The free-form-only condition produces the largest gains on the free-form test set.
  • Binary-only supervision yields minimal gains on free-form evaluation, with free-form Brier score improving only to 0.004.This comparison isolates the learning signal under matched example counts and compute budgets.
  • The binary+free-form mixture offers the best overall trade-off across free-form and binary testing formats.Both binary-only and mixed supervision improve binary evaluation, but the mixed setting is strongest across formats.
  • The results indicate that free-form data is essential for open-ended forecasting, whereas mixed supervision is more robust across task types.The authors do not identify one universally optimal recipe because the preferred mixture depends on the evaluation format.

B.2 Varying models and evaluation months

Retrieval gains plateau after roughly five article chunks, and OpenForesight training improves performance across model families and evaluation months. The trained models remain stronger over time and lead other models in Brier score.

  • Varying retrieval: Retrieval improvements plateau at around five articles for both Qwen3-8B and GPT-OSS-120B, motivating five-article retrieval in later experiments.The largest gains occur among the first few retrieved chunks.
  • Varying models: OpenForesight training substantially improves Llama and Gemma-family models in accuracy and Brier score, with Llama performance exceeding larger comparison models.The reported cross-family evaluation includes Llama-3.1-8B-Instruct, Llama-3.2-3B-Instruct, and Gemma-3-4B-Instruct.
  • Evaluation months: Model accuracy and Brier score decline gradually by resolution month, consistent with later forecasts being more difficult.The test set contains 94 May, 85 June, 76 July, and 47 August questions after filtering.
  • Evaluation months: Trained models consistently outperform their original versions and all other models in Brier score across the monthly evaluation.This result accompanies the month-by-month degradation in overall forecasting performance.

B.3 Ablation with Supervised Finetuning

The paper adds supervised finetuning using forecasting reasoning traces before reinforcement learning. The final SFT-plus-RL model achieves higher accuracy and substantially better calibration, despite a slightly lower Brier score.

  • The SFT stage uses Grok-3-Mini to generate forecasting reasoning traces before GRPO training on the same data mixture and reward design.The resulting checkpoints include Qwen3-8B-sft and Qwen3-8B-sft-rl.
  • The Qwen-8B-sft-rl model’s calibration is described as near perfect in the reported RL comparison.Figure 14 highlights calibration alongside accuracy for the SFT and RL variants.
  • The final SFT-plus-RL model achieves higher accuracy and much better calibration than the distilled SFT model, with a slightly lower Brier score.The comparison uses the curated May–August 2025 test set and Figure 14a.

B.4 Consistency Evaluation

The RL-trained model improves consistency across most checks, with especially large gains for Boolean logic and consequence relations but regressions in negation consistency.

  • B.4 Consistency Evaluation: 15% fewer arbitrage violations and 4% fewer frequentist violations overall after RL training.The reported consistency table summarizes average violation scores and relative changes before and after RL.
  • B.4 Consistency Evaluation: Boolean logic checks improve by 75% for arbitrage violations and 42% for frequentist violations.
  • B.4 Consistency Evaluation: Consequence checks improve by 67% for arbitrage violations and 39% for frequentist violations.
  • B.4 Consistency Evaluation: Negation consistency shows notable regressions despite improvements across most other checks.

B.5 Evaluation on Metaculus Questions

The Metaculus evaluation uses 449 filtered binary questions and measures both accuracy and Brier score. OpenForecaster 8B outperforms several larger models, while GPT-OSS-120B performs significantly better than the remaining models.

  • B.5 Evaluation on Metaculus Questions: The benchmark contains 449 filtered, high-interest binary questions from Metaculus and reports accuracy and Brier score.Questions about meta-predictions, stock prices, or low trading volume were excluded.
  • B.5 Evaluation on Metaculus Questions: OpenForecaster 8B outperforms larger models including DeepSeek-R1 and Llama-4-Maverick on Metaculus questions.
  • B.5 Evaluation on Metaculus Questions: GPT-OSS-120B performs significantly better than the other evaluated models.
  • B.5 Evaluation on Metaculus Questions: The reported prediction-market Brier score ranges from −1 to 0, with −0.25 corresponding to a constant 50% prediction.This score differs from the Brier score used earlier for freeform questions.

C.1 Training Details

Training primarily uses Qwen3-8B with GRPO, retrieval, structured batching, and outcome-based rewards for freeform and binary forecasting tasks.

  • C.1 Training Details: Training uses Qwen3-8B with GRPO, retrieving up to five news chunks and capping prompts at 4,096 tokens and responses at 8,192 tokens.The implementation uses VeRL and a vLLM-based sampler with eight samples per prompt during training.
  • C.1 Training Details: The 54K-sample training set combines 52K OpenForesight freeform questions with 2K resolved Metaculus binary questions.Freeform questions are ordered first and binary questions last rather than randomly mixing all samples.
  • C.1 Training Details: Separate freeform and binary batches are crucial for strong performance on binary prediction-market questions.
  • C.1 Training Details: Freeform rewards sum accuracy and Brier score, while binary rewards use negated binary Brier score plus a format penalty for unparsable outputs.The total reward ranges are [−2, 2] for freeform questions and [−2, 0] for binary questions.
  • C.1 Training Details: The final RL run took about 1,000 H100 GPU-hours, while all ablations totaled approximately 20,000 H100 GPU-hours.

D.1 Analysis of Best Question Selection

The question-selection pipeline uses automated generation and filtering, then selects among multiple candidates using relevance, specificity, and unambiguity; earlier candidates are chosen more often.

  • D.1 Analysis of Best Question Selection: The pipeline retains one valid question from single-question articles and uses an LLM to choose among candidates from 61% of articles producing multiple questions.Thirty-nine percent of source articles failed to produce any valid questions.
  • D.1 Analysis of Best Question Selection: Question 1 is selected 43.6% of the time, compared with 31% for Question 2 and 26.7% for Question 3.The selection frequencies indicate that the first generated candidate is chosen most often.
  • D.1 Analysis of Best Question Selection: The authors interpret the selection pattern as evidence that earlier generation attempts frequently produce higher-quality questions.
  • D.1 Analysis of Best Question Selection: The curated dataset is dominated by People and Places answers at 65%, while miscellaneous entities account for 35%.People include individuals at nearly 45% and locations at 20%.
  • D.1 Analysis of Best Question Selection: The final manually filtered test set contains 302 questions after checks for ambiguity, future resolution, relevance, and established answers.The guidelines reject questions with multiple possible answers, late resolution, narrow relevance, or already-known outcomes.

E.1 Qualitative Analysis of Final Answers

The qualitative analysis examines where forecasting training improves or worsens predictions, including domain and question-form patterns and recurring reasoning failures. Improvements are substantial in several domains and question types, but the model still guesses from general knowledge or selects prominent entities when retrieved evidence is insufficient.

  • Domain and question-form analysis: Significant improvements occur in World, Australian, and US news, but not sports, suggesting weaker performance on sports-heavy markets.Table 6 reports no significant change for sports and explicitly connects this pattern to prediction markets such as Kalshi.
  • Domain and question-form analysis: Significant gains appear on “what,” “which,” and “who” questions, alongside a slight regression on “where” questions.This comparison is reported in Table 7’s question-form analysis.
  • Failure modes: The most common failure occurs when retrieved articles lack the needed information, prompting guesses from general knowledge despite acknowledged uncertainty.One example infers a certification date from a broad time range, while another applies naming conventions from an unrelated example.
  • Failure modes: When several entities appear in context, the model often chooses the most prominent or frequently mentioned one rather than the correct answer.In the football example, it selects Bayern Munich based on prominence and team strength even though the required group-stage results are unavailable.
Loading 2512.25070v2…