Source-linked AI summary
How Proper Scoring Rules Shape LLM Forecasting
Benjamin Turtel, Paul Wilczewski, Kris Skotheim, Ville A. Satopää, Philip E. Tetlock
TL;DR
LLM forecasting provides an objectively evaluable setting for studying how reward choices shape probabilistic behavior. The paper compares five proper scoring rules as terminal rewards for binary forecasts of resolved news events under controlled training. Although all rules are theoretically proper, the resulting forecasters differ in performance profiles and error composition, while the single-seed design limits how broadly the differences can be interpreted.
Problem
The paper asks whether proper scoring rules with the same truthfulness incentive nevertheless produce different LLM forecasting behaviors under finite policy optimization.
Method
The study trains models on binary forecasts of resolved news events, varying only the proper scoring rule while holding the shared training setup fixed, and evaluates scores, calibration, discrimination, probability use, and BIN profiles.
Results
The reward-trained models have different performance profiles and estimated bias–information–noise compositions; Brier training gives the best Brier score and AUC-ROC, while log training gives the best log score and lowest expected calibration error.
Takeaways & Limitations
Proper scoring rules need not behave interchangeably as LLM training objectives, and reward choice may shape both forecasting quality and error structure.
Takeaways & Limitations
The analysis uses one base model, one binary news-event dataset, one shared hyperparameter configuration, and a single training seed per reward condition.
Abstract
from arXiv · showhide
This paper evaluates how reward function choice shapes the performance and behavior of LLM forecasters. We compare five proper scoring rules as training objectives for binary forecasts of resolved real-world events. Although the rules share the same theoretical incentive for truthful probability reporting, the resulting models differ in calibration, probability use, and estimated profiles of bias, information, and noise, with smaller differences in aggregate accuracy and discrimination. The Brier-trained model has the lowest observed Brier score and highest AUC-ROC, while the log-trained model has the highest observed log score and lowest calibration error. Models with similar aggregate performance also reach that performance through different combinations of bias, information, and noise. Proper scoring rules therefore need not behave interchangeably as training objectives. Reward choice may shape not only how well an LLM forecasts, but how its forecasting errors are structured. Each condition uses a single seed, so some differences may reflect training stochasticity.
1 Introduction
This section motivates forecasting as an objectively evaluable test of LLM reasoning and examines how five proper scoring rules shape learned forecasters under controlled finite training. Despite shared truthfulness incentives, reward choices are associated with distinct performance, calibration, probability-use, and bias–information–noise profiles.
- Forecasting tests whether language models can combine conflicting evidence, express uncertainty probabilistically, and predict under limited information with objective post-resolution scoring.
- The study trains binary-event forecasters on resolved news questions while holding the model, data, optimizer, training steps, and rollout budget fixed across reward conditions.Forecast horizons range from 7 to 90 days across politics, geopolitics, economics, business, science, and sports.
- The comparison covers five strictly proper scoring rules whose curvature, boundedness, tail behavior, and emphasized probability regions differ.The rules are logarithmic, Brier, spherical, Beta(2, 8), and Beta(8, 8).
- The evaluation measures aggregate score, calibration, discrimination, probability-scale use, and estimated contributions from bias, information, and noise.
- The Brier-trained model achieves the best Brier score and AUC-ROC, whereas the log-trained model achieves the best log score and lowest expected calibration error.
- BIN analysis reveals distinctions among learned forecasters that aggregate forecasting metrics alone do not show.
- The experiments provide a controlled finite-training comparison showing that proper scoring rules with the same population-level incentive can be associated with different learned forecasters.
2 Related Literature
Related work positions this study within language-model forecasting and proper-scoring-rule research. Unlike prior work centered mainly on forecasting systems, supervision, or evaluation, this paper holds the training setup fixed to study reward-function effects.
- Prior systems evaluate language models on temporally grounded real-world forecasts using news evidence, retrieval, structured reasoning, and forecast aggregation.
- ForecastBench compares model forecasts with human forecasts on unresolved questions to reduce temporal leakage.
- Future-as-Label uses outcomes observed after an information cutoff as supervision for forecasts based only on pre-cutoff evidence.
- Proper scoring rules incentivize truthful probabilistic reports in expectation but differ in curvature, boundedness, tail behavior, and emphasized probability regions.
- This study examines those scoring-rule differences under Group Relative Policy Optimization, where advantages compare rewards across responses sampled for the same input.
- Forecast evaluation considers both aggregate accuracy and probabilistic-error structure through reliability, resolution, uncertainty, and bias–information–noise decompositions.
3 Method
The paper trains LLMs to forecast binary real-world events under temporally constrained information, varying only the proper scoring rule used as terminal reward. It evaluates reward functions through performance, calibration, discrimination, forecast structure, and relative-policy optimization behavior.
- Forecasting setup: Forecasts answer whether an event will occur by a specified resolution time, using information available before the prediction cutoff.Questions resolve to Yes or No, and the model outputs a probability p ∈(0, 1) for the event occurring.
- Forecasting setup: The predictor generates reasoning and a final probability, while a fixed external resolver assigns the binary outcome from post-cutoff evidence.The resolver does not observe the forecast, evaluate the reasoning trajectory, or provide preference judgments.
- Reward functions: Five proper scoring rules—log, Brier, spherical, Beta(2, 8), and Beta(8, 8)—serve as terminal rewards based only on the parsed forecast and resolved outcome.Beta(2, 8) emphasizes lower-probability regions, whereas Beta(8, 8) concentrates weight near 0.5; beta-family forecasts are clipped to [10^-3, 1 −10^-3].
- Reward functions: Reward geometry differs across probability regions: log and Beta(2, 8) are more sensitive to low probabilities, while Beta(8, 8) varies most near p = 0.5.Brier and spherical rewards vary more smoothly, and these differences can produce distinct learning signals under finite policy optimization.
- Evaluation metrics: Evaluation combines calibration, discrimination, aggregate scores, and a bias–information–noise decomposition of expected Brier score.ECE uses 10 probability bins, AUC-ROC evaluates ranking of positive above negative outcomes, and the decomposition separates systematic bias, valid information, and nonsystematic noise.
- Training details: Dr. GRPO compares each trajectory’s reward with other trajectories from the same forecasting state, updating the policy toward above-average forecasts.The group mean acts as a state-specific baseline; training holds fixed the model, data, optimizer, steps, rollout budget, and hyperparameters while varying only reward function, with one seed per condition.
4 Results
Across held-out forecasting evaluation, reward-trained variants generally improve calibration and Brier performance over the base model, but differ in aggregate metrics and in the bias–information–noise profiles underlying those outcomes.
- Aggregate Performance: All reward-trained variants have lower observed Brier score and ECE than the base model, while missing forecasts remain rare at ≤0.27%.Relative to base, four variants improve Brier score at the 1% significance level and Beta(8, 8) at the 5% level.
- Aggregate Performance: The Brier-trained model achieves the best Brier score and AUC-ROC, whereas the log-trained model achieves the best log score and lowest ECE.
- Calibration and Probability Use: Reward-trained variants generally track the perfect-calibration diagonal more closely than the base model and place more forecast probability mass at lower values.Calibration curves bin predictions by equally spaced probability intervals and compare observed frequencies with predicted probabilities.
- Bias–Information–Noise Decomposition: BIN decomposes Brier-score differences into systematic bias, outcome-relevant information, and nonsystematic forecast noise, with components summing to model-implied rather than observed improvement.The model-implied and observed differences differ by roughly one to two percentage points of base Brier in these results.
- Pairwise BIN Contrasts: Pairwise contrasts show log and Beta(2, 8) strongest on bias, Brier and Beta(8, 8) strongest on information, and log strongest on noise.The log variant has near-unit posterior probability of lower noise than every other reward-trained variant, while Beta(8, 8) has the weakest noise profile.
5 Discussion and Limitations
The study finds that proper scoring rules produce different learned forecasting profiles despite sharing theoretical propriety. These differences motivate potential uses for error-profile diversity, while the findings remain bounded by the study’s controlled setup and limited generalization evidence.
- Discussion: Reward-trained models generally outperform the base model, but variants with similar aggregate scores differ in calibration, discrimination, probability-scale use, and BIN profiles.The BIN decomposition reveals distinctions that aggregate forecasting metrics alone do not show.
- Discussion: Proper scoring rules can differ as LLM training objectives because finite policy optimization produces different reward distributions and group-relative advantages.The paper frames this as an empirical distinction among theoretically proper objectives, not a challenge to propriety.
- Implications: The composition of forecast error can matter when forecasts are aggregated, acted on directly, or used for ranking and prioritization.Lower noise may favor direct action, while information gains may matter for ranking; nonsystematic errors may be reduced through aggregation.
- Implications: Different BIN profiles may support complementary forecast ensembles, even when individually trained models have similar standalone scores.The paper proposes testing deliberate combinations and jointly optimizing ensemble members with their aggregator.
- Limitations: The analysis uses one base model, one binary news-event dataset, one shared hyperparameter configuration, and a single seed per reward condition.Observed differences may partly reflect training stochasticity, and fixed hyperparameters do not compare each reward at its individually optimized configuration.
- Limitations: The results are a controlled analysis of reward-function behavior in this setting, not evidence that one scoring rule is universally preferred.Generalization may depend on models, datasets, base rates, optimization procedures, training budgets, and random seeds.
A.2 Example 1: Russia–Ukraine Ceasefire
This example asks whether Russia and Ukraine will sign a formal ceasefire by the specified deadline. The supplied context describes ongoing negotiations, optimism about progress, and unresolved disagreements over territory and other terms.
- Question and resolution: The question resolves Yes only if authorized Russian and Ukrainian representatives sign a written ceasefire agreement by February 28, 2026.Informal understandings and unilateral pauses do not qualify; official state sources or major international news outlets must confirm the document.
- Context: Negotiations were described as making progress, but key issues remained unresolved, especially whether Ukraine would surrender additional territory to Russia.The context also reports disagreement over broader terms and Russian demands concerning territory, military limits, and NATO.
- Context: Ukraine reportedly backed proposals or agreed to core terms, while Russia had not confirmed agreement details and ruled out major concessions.Russian statements characterized proposals as a basis or issue for discussion rather than a draft agreement.
- Context: The context includes continuing military attacks alongside diplomatic meetings and planned talks involving U.S., Ukrainian, and Russian officials.Reports describe strikes and bombardments while negotiations continued in Moscow- and Abu Dhabi-related channels.
A.3 Example 2: India–Russia Payment Mechanism
This example asks whether an India–Russia summit statement will confirm a deployed rupee–ruble alternative payment mechanism by January 15, 2026. The context describes expected payment-system cooperation but does not establish implementation.
- Question and resolution: The question resolves Yes only if an official summit statement or bilateral agreement explicitly announces an alternative national-currency payment system outside SWIFT.The system must be ready for use or actively deployed, not merely under discussion.
- Context: The summit was expected to produce a joint statement and agreements involving rupee–ruble settlement and mechanisms using Special Rupee Vostro Accounts.The reported expectations also included defense, trade, energy, and oil-related outcomes.
- Context: Officials discussed linking Russia’s SBP or Faster Payments System with India’s UPI to enable Mir–RuPay transactions and mutual settlements.These discussions were framed as plans or hopes for cooperation to ease payment issues after sanctions.
- Context: The context presents broader local-currency settlement and payment cooperation, while noting that a unified BRICS currency challenge to the dollar remains distant.India was described as having no plans for a BRICS currency and emphasizing stability.
B Bootstrap Confidence Intervals
Table 3 reports paired bootstrap differences in Brier score between reward-trained variants and the base model. Resampling is conducted at the held-out-question level while retaining all five sampled forecasts for each selected question.
- Bootstrap procedure: Each bootstrap resample draws held-out questions with replacement and retains all five sampled forecasts associated with each selected question.The intervals are therefore paired at the question level rather than resampling individual forecasts independently.
- Uncertainty: Confidence intervals are shown at the 1%, 5%, and 10% significance levels.The table uses these levels to characterize uncertainty around the paired score differences.
- Table interpretation: Table 3 reports paired bootstrap differences in Brier score relative to the base model.Differences are computed as reward-trained variant minus base model, so negative values indicate lower Brier score.
C Detailed Results by Evaluation Method
The appendix tests whether forecasting and bias–information–noise results depend on how stochastic model outputs are evaluated. It compares single forecasts, median-of-five aggregation, and pooled five-rollout evaluation.
- Three evaluation specifications are compared: single-sample inference, median-of-five aggregation, and pooled five-rollout evaluation.These analyses assess sensitivity of the main conclusions to the treatment of stochastic model outputs.
- For median-of-five aggregation, each question’s final forecast is the median of five independently sampled probabilities.
- For pooled five-rollout evaluation, all five sampled forecast–outcome pairs are retained for forecast-level metrics, while AUC-ROC is averaged across five rollout sets.
- Single-sample and median-of-five evaluations contain one forecast per question (N = 965), whereas pooled evaluation contains five forecasts per question (N = 4,825).
C.1 Sensitivity to Evaluation Method
Forecasting metrics are compared across single-sample, median-of-five, and pooled five-rollout evaluations. Overall patterns are broadly similar, but median aggregation generally improves performance and changes some reward-variant rankings.
- Tables 4–6 compare forecasting metrics under single-sample, median-of-five, and pooled five-rollout evaluation methods.
- Median-of-five aggregation generally improves forecasting performance but changes some relative rankings among reward variants.
C.2 BIN Contributions Across Evaluation Methods
BIN contributions show both stable and evaluation-sensitive patterns across single-sample, median-of-five, and pooled five-rollout analyses.
- The Brier variant has the largest information contribution across evaluation methods.
- The log variant consistently shows a positive noise contribution across evaluation methods.
- Notable evaluation-dependent changes occur for the Beta(2, 8) and Beta(8, 8) variants.