PaperlayerView exact revision on arXiv

Source-linked AI summary

LEAP: Likelihood Elicitation and Aggregation for LLM-based Probabilistic Forecasting

Yufei Chen, Yiran Zhao, Xiaogang Xu, Qipeng Xie, Jiafei Wu, Zhe Liu

arXiv:2609.01337v1cs.AI

TL;DR

LLM forecasters often read all collected evidence together, making individual evidence effects hard to isolate and uncertainty difficult to preserve. LEAP elicits prior and per-item likelihood parameters, combines them with a deterministic probabilistic model, and improves prediction and calibration across most evaluated settings while enabling reproducible audits.

  • Problem

    Many LLM forecasting systems still ask an LLM to read the full evidence bundle and produce the final forecast, obscuring individual evidence effects and collapsing uncertainty over competing outcomes.

  • Method

    LEAP has the LLM interpret each evidence item separately, estimate an explicit prior and likelihood parameters, and combine them through deterministic probabilistic aggregation.

  • Results

    LEAP improves prediction accuracy and probabilistic quality in most settings and metrics across five base models and four external agent CLI frameworks, while roughly halving expected calibration error and overconfidence.

  • Takeaways & Limitations

    Closed-form aggregation makes evidence contributions reproducible through leave-one-out updates, providing an audit signal unavailable from a free-text LLM rationale.

  • Takeaways & Limitations

    LEAP operates only after evidence collection and cannot recover information that was not retrieved; it also uses more inference calls than Monolithic.

Abstract

from arXiv · show

LLM-based forecasting systems have improved on real-world tasks such as financial markets and sports outcomes, largely through stronger search and tool use. Many systems still ask an LLM to read all collected evidence together and produce the final forecast. We call this design Monolithic Prediction. It can obscure how individual evidence items affect the result and collapse uncertainty across competing outcomes. We propose LEAP (Likelihood Elicitation and Aggregation for Probabilistic forecasting), which reorganizes how collected evidence is used in the prediction stage. LEAP examines each evidence item separately and elicits likelihood parameters that describe its implications for the target. An explicit prior and a deterministic probabilistic model then combine these likelihoods into a posterior distribution. This procedure supports continuous, single-choice, and multi-choice forecasts while preserving reproducible evidence contributions. We build a benchmark covering forecasting, information-seeking, and browsing tasks, and evaluate LEAP on our own agent loop and several agent CLI frameworks. Given the same evidence, LEAP improves most prediction and calibration metrics across models and remains stronger under controlled comparisons of prior access, inference budget, and aggregation.

1 Introduction

LLM forecasting systems commonly collect evidence through increasingly capable agent loops but still make final predictions by reading the full evidence bundle holistically. LEAP instead interprets evidence items separately, aggregates them probabilistically, and improves reported forecasting performance while enabling item-level audits.

  • Motivation: Holistic LLM prediction reads the gathered evidence as a whole after search and source integration, leaving the final forecasting stage opaque.The approach uses the same broad evidence bundle to produce the answer in one or more forecasting calls.
  • Motivation: This design cannot isolate any specific evidence item's influence and can collapse uncertainty over competing outcomes into a single answer.Long-context aggregation also makes the final prediction more vulnerable to forgetting and hallucination.
  • LEAP: LEAP examines each evidence item in isolation, elicits likelihood parameters alongside an explicit prior, and combines them into a posterior distribution with a deterministic probabilistic model.The design reorganizes only the final prediction step rather than evidence collection.
  • LEAP: LEAP exposes which evidence items contributed to the posterior and supports auditing by removing an item and recomputing the prediction.This makes evidence contributions inspectable rather than leaving them implicit in a free-text rationale.
  • Evaluation: Across forecasting, information-seeking, and browsing tasks, LEAP improves over Monolithic on most settings and metrics across models and agent frameworks.The evaluation uses a shared evidence set for both methods under strict temporal isolation and knowledge-cutoff constraints.
  • Contributions: The paper contributes a fixed-evidence prediction formulation, the LEAP method, a cross-task benchmark, and diagnostic analyses of gains and evidence-level auditability.The benchmark spans multiple base models and several agent CLI frameworks.

2 Related Work

Prior work strengthens evidence collection, probabilistic inference, calibration, and explanation faithfulness, but many forecasting systems still consume gathered evidence holistically. LEAP focuses specifically on preserving item-level evidence assessments until deterministic aggregation.

  • LLM-based forecasting systems: Recent LLM forecasting systems combine search, retrieval, decomposition, tool use, ensembling, and calibration, while final forecasts often still read gathered material as a whole.This motivates separating evidence collection from final prediction.
  • LLM-assisted Bayesian inference: LLM-assisted Bayesian methods use generated factors, coarse judgments, or conditional probabilities to parameterize probabilistic models and networks.These approaches establish a related route for incorporating LLM judgments into probabilistic inference.
  • Evidence collection and agent evaluation: Retrieval-augmented generation, passage readers, and browser-based systems ground answers in external documents and organize evidence before prediction.These methods primarily address the evidence-collection stage relevant to LEAP's setting.
  • Auditability and faithful explanations: Faithful-explanation work links useful reasoning traces to the computation that produces the answer, including deterministic final execution.LEAP's evidence-level auditability follows this connection between explanations and computation.
  • Scope of LEAP: LEAP starts from retrieved evidence for one forecasting task, elicits one likelihood per item, and keeps these local assessments separate until aggregation.This distinguishes its prediction-stage focus from related Bayesian and iterative-search systems.

3 Problem Formulation

The paper formalizes evidence-grounded forecasting as a two-stage process: an agent collects timestamp-valid evidence, then a prediction method maps the fixed task and evidence set to an output appropriate to the target type.

  • 3.1 Task and Two-Stage Decomposition: A forecasting task is represented by question q, cutoff time tfreeze, output type τ, and optional candidate set O.Usable evidence must be timestamped no later than tfreeze.
  • 3.1 Task and Two-Stage Decomposition: Evidence-grounded forecasting separates collection from prediction: an agent gathers timestamped passages, then the prediction stage maps (T, E) to forecast f without additional retrieval.The same collected evidence can therefore be consumed by different prediction methods.
  • 3.2 Forecast Targets: The output type τ determines the latent quantity and forecast representation across continuous, single-choice, and multi-choice targets.These types cover numeric quantities, exactly one correct option, and independently evaluated option labels.
  • 3.2 Forecast Targets: Continuous targets represent real-valued quantities and report fixed quantiles to capture central tendency and uncertainty.Examples include closing prices and published economic indicators.
  • 3.2 Forecast Targets: Single-choice forecasts assign probabilities summing to one across K candidate options, with each entry representing an option's correctness probability.Exactly one candidate is correct in this target type.
  • 3.2 Forecast Targets: Multi-choice forecasts assign one probability to each option independently, so the K entries need not sum to one.The latent target is a vector of yes/no labels rather than one selected option.
  • 3.3 Monolithic Baseline and Evaluation Protocol: Monolithic Prediction uses the same LLM and evidence set as LEAP, reads (T, E) in one prompt, and emits the τ-specified forecast in one pass.For continuous targets it is additionally asked for an uncertainty range for comparable scoring.
  • 3.3 Monolithic Baseline and Evaluation Protocol: Both methods use the same collected (T, E), so score differences reflect evidence consumption during prediction rather than retrieval differences or search budget.The collection stage runs once per task.

4 Method

LEAP replaces direct LLM forecasting with local parameter elicitation and deterministic probabilistic aggregation. It estimates priors and per-item likelihoods, applies safeguards and closed-form updates, and reports reproducible evidence contributions.

  • Core design: LEAP consumes (T, E) and divides prediction into local parameter elicitation followed by deterministic aggregation with a closed-form update.The LLM supplies local evidence interpretations, while the probabilistic model produces the final forecast.
  • Probabilistic model: The Bayesian model combines a prior P0 with evidence likelihoods Pi under a conditional independence assumption, while dependent evidence is clustered before aggregation.The retained subset R is used for the posterior update.
  • Probabilistic model: Conjugate prior-likelihood pairs make the posterior update closed form across Gaussian, categorical-multinomial, and independent Bernoulli output models.A tempered version adds role weights and reduces to the unweighted update at unit settings.
  • 4.2 Parameter Elicitation: The LLM estimates prior and likelihood parameters rather than a forecast, with each evidence-level call seeing only T and one evidence item.It cannot see the full evidence set, other items, accumulated evidence, or a partial posterior.
  • 4.2 Parameter Elicitation: Prior parameters come from reliable numerical history when available, otherwise from an evidence-free LLM call or a conservative unbiased prior for discrete tasks.The source depends on the informative base-rate information available for the target.
  • 4.2 Parameter Elicitation: Each isolated evidence item yields structured likelihood information about candidate outcomes plus a dependency key identifying its source family.Continuous targets use target-scale observations and evidence-strength-dependent deviations; discrete targets use option support or opposition labels.
  • 4.3 Deterministic Aggregation: The deterministic aggregation step produces quantiles, normalized option probabilities, or Bernoulli marginals according to the target type.It uses no additional LLM call after elicitation.
  • 4.3 Deterministic Aggregation: For each retained item j, LEAP recomputes the update without j and reports the resulting forecast change as reproducible leave-one-out contribution ∆j.This decomposition provides an evidence-level audit signal.

5 Experiments

Experiments compare LEAP with Monolithic under shared, temporally isolated evidence across benchmark tasks, models, and agent frameworks. LEAP generally improves predictive and calibration outcomes, with benefits persisting under component, horizon, budget, and aggregation analyses.

  • Evaluation setup: The benchmark covers forecasting, information-seeking, and browsing tasks, evaluated with shared evidence and temporal isolation.The evaluation uses evidence available by each task timestamp and base models whose knowledge cutoffs precede evaluated timestamps.
  • Main results: LEAP improves FutureX, Spherical score, and accuracy for every tested base model, with FutureX gains of 3.6 to 18.1 points.NCRPS also improves for every base model, while Brier improves on three of five models and is close or essentially tied on the other two.
  • Main results: Across four external agent CLI frameworks, LEAP improves most reported metrics, with macro-average gains of 9.8 FutureX points, 4.7 accuracy points, 16.5 Brier points, 14.1 Spherical points, and 12.9 NCRPS points.The authors present this as evidence that LEAP can operate as a downstream probability skill without modifying evidence-collection pipelines.
  • Calibration: LEAP roughly halves ECE from 0.184 to 0.088 and Adaptive ECE from 0.177 to 0.091, while reducing overconfidence from 0.317 to 0.150.The calibration analysis attributes the difference to Monolithic concentrating probability on one option while LEAP distributes probability more conservatively.
  • Robustness across forecast horizons: As forecast horizons increase from 7 to 60 days, FutureX improvement rises from 6.8 to 11.9 points and the overconfidence gap widens from 13.3 to 23.3 points.The reported interpretation is that LEAP produces wider posteriors when evidence becomes more indirect.
  • Component ablation: Removing the prior drops FutureX below Monolithic from 0.651 to 0.643 and raises ECE to 0.213, while removing dependency clustering raises ECE from 0.088 to 0.158.Reliability sampling has a smaller consistent effect, lowering FutureX to 0.714 and raising ECE to 0.122; LEAP otherwise retains its advantage over Monolithic.
  • Controlled comparisons and efficiency: LEAP retains its advantage with matched prior access or comparable token budgets, performs better than linear opinion pooling, and changes little across tested likelihood strengths.Median latency is close to the two-sample ensemble, whereas p95 latency is higher because evidence-level calls complete at different speeds.

6 Conclusion

LEAP separates evidence collection from prediction, interprets each item locally, and combines likelihoods with a prior through deterministic aggregation. Under fixed-evidence evaluation, it improves prediction accuracy and probabilistic quality across diverse models and agent frameworks.

  • LEAP leaves evidence collection unchanged, interprets each evidence item locally, and combines likelihoods with a prior through deterministic probabilistic aggregation.
  • Under fixed-evidence evaluation, LEAP improves prediction accuracy and probabilistic quality in most settings and metrics across five base models and four external agent CLI frameworks.Calibration diagnostics roughly halve expected calibration error and overconfidence, with gains widening at longer forecast horizons.
  • Closed-form inference makes evidence contributions reproducible by removing one item and rerunning the posterior update.This produces an audit signal unavailable from a free-text LLM rationale.

Limitations

LEAP is limited to converting an already collected fixed evidence set into a forecast, rather than improving evidence collection. Its evaluation scope is also bounded by the studied benchmark, languages, domains, horizons, and agent designs.

  • LEAP changes the prediction step after evidence collection but does not improve the upstream collection stage itself.Sparse, outdated, or weakly relevant evidence is expressed as a wider posterior rather than information recovered from retrieval.
  • The evaluation covers English forecasting and agentic information-seeking tasks using a fixed collection protocol and a limited set of base models.
  • Evaluation on other languages, specialised scientific domains, longer forecast horizons, and substantially different agent designs remains future work.LEAP also uses more inference calls than Monolithic, creating a practical cost tradeoff for calibration and auditability.

Ethical Considerations

The paper frames LLM probabilistic forecasts as decision support rather than authoritative predictions. It emphasizes human oversight, source verification, expert judgment, and responsible handling of data-access and misuse risks.

  • LLM probabilistic forecasts should be treated as decision support rather than authoritative predictions, especially for consequential events.
  • Human oversight, clear uncertainty communication, and domain-expert review are recommended where incorrect forecasts could cause material harm.
  • LEAP’s audit trail does not eliminate risks from incomplete, biased, outdated, or unrepresentative sources or factual errors and biases in local LLM interpretations.The method does not remove the need for source verification or independent expert judgment.
  • Deployments collecting external evidence should respect access policies, copyright restrictions, and source privacy expectations, while automated forecasting may also support manipulation or strategic targeting.

C.1 Dependency Key Canonicalisation

Dependency-key canonicalisation makes clustering insensitive to minor surface variations in free-text keys. Raw keys are normalized into canonical token forms so semantically equivalent keys can collapse together.

  • C.1 Dependency Key Canonicalisation: Free-text dependency keys may differ through word order, hyphenation, or stopwords while referring to the same dependency.
  • C.1 Dependency Key Canonicalisation: Canonicalisation lowercases keys, splits common separators, removes fixed stopwords, sorts remaining tokens, and joins them with one delimiter.
  • C.1 Dependency Key Canonicalisation: Keys with matching token sets collapse to the same canonical form, including reordered BAFTA longlist examples.

D.1 Benchmark Construction

The benchmark converts forecasting, information-seeking, and browsing tasks into structured probabilistic questions through source selection, rewriting, review, and assembly. The final evaluation manifest contains 347 reviewed tasks, with temporal leakage audited against task-specific freeze times.

  • Quality control: Manual review checks answer-type validity, distractor plausibility, label conflicts, temporal leakage, and whether answers fit supported output types.Items are dropped or quarantined when their answers cannot be represented cleanly or conflicts are verified.
  • Temporal design: For GAIA and BrowseComp, tfreeze is a benchmark-snapshot retrieval cutoff rather than a forecast resolution date.For FutureX, tfreeze retains the original question time associated with the forecasting item.
  • Evaluation protocol: The benchmark is evaluation-only: all 347 retained tasks are used for reported evaluation, with a separate 60-task diagnostic subset reserved for Section 5.3 analyses.No train/dev/test splits are defined because models are not trained, fine-tuned, or selected on this benchmark.
  • Source tasks: FutureX, GAIA, and BrowseComp provide forecasting, information-seeking, and browsing-oriented question-answering tasks, respectively.GAIA and BrowseComp items are rewritten into structured probabilistic tasks, while ranking resolutions are excluded from FutureX.
  • Leakage audit: A temporal leakage audit checks every retrieved page against its task’s tfreeze and found no cutoff violations affecting reported results.The audit also searches heuristically for post-resolution language or answer-revealing phrasing.
  • Benchmark assembly: 347 reviewed tasks form the final benchmark manifest after removing unsuitable ranking, answer-type, duplicate, and label-conflict items.The initial pool included 160 FutureX tasks, 103 GAIA candidates, and 100 BrowseComp candidates.

D.4 Diagnostic Subset for Analysis

The diagnostic analyses use a fixed 60-task subset and controlled inference settings to examine calibration, robustness, ablations, evidence grouping, and source-wise performance. Results indicate that repeated-source handling affects calibration and that aggregate improvement appears across benchmark sources.

  • Diagnostic subset: The diagnostic subset contains 60 stratified tasks spanning the three output types and is held constant across calibration, lead-time robustness, and component-ablation analyses.GPT-5.4-mini is the base model, and no model selection is performed on the subset.
  • Inference controls: Main evaluations use five independent seeded passes per model-method cell under fixed budgets, with median score variability σ ≈0.010.The best-to-worst seed spread within a cell never exceeds 0.035.
  • Controlled analyses: The fixed setup allows comparisons of prediction-time budget, latency, aggregation, likelihood strength, evidence grouping, and source-wise performance.Evidence-grouping and source-wise comparisons hold collected evidence fixed across methods.
  • Evidence grouping: Source-key clustering gives the best FX and Brier scores, whereas domain clustering gives the lowest ECE after retaining fewer evidence items.Without clustering, repeated reports can be counted more than once and calibration degrades.
  • Source-wise results: Aggregate improvement is present in each benchmark source under the diagnostic setup.The source-wise results are reported in Table 14.

D.12 Case Studies

The case studies illustrate how LEAP handles wrong confident predictions, noisy evidence, and missing evidence while exposing evidence-level contributions. An expanded continuous example further traces prior construction, local likelihoods, safeguards, posterior parameters, and leave-one-out effects.

  • Case 1: In the cathedral case, LEAP selects the correct answer at p=0.54 while Monolithic assigns p=0.85 to an incorrect option.LEAP exposes the evidence items that drove the decision, including grave-slab identification and a Christ Church search.
  • Case 2: In the rugby case, LEAP assigns p=0.80 to the correct Georgia option while Monolithic incorrectly assigns p=0.33 to Portugal.One official tournament page strongly supports Georgia, while a semifinalist bracket pulls in the opposite direction.
  • Case 3: With no retained evidence, LEAP returns a uniform p = 0.25 distribution over four options instead of Monolithic’s confident incorrect prediction at p=0.63.The case presents the uniform forecast as poorly resolved but well calibrated, contrasting with a sharper wrong forecast.
  • Expanded audit trace: The expanded AAPL case asks for the January 23, 2026 daily high, with ground truth 249.41 USD and a Monolithic prediction of 249.0.LEAP begins by constructing a data-derived prior from the latest seven daily highs.
  • Posterior contributions: In the expanded trace, the prior, E1, and E5 contribute approximately 57.5%, 20.7%, and 21.2% of posterior precision, respectively.All other retained items together contribute less than 1%.
  • Leave-one-out audit: Leave-one-out analysis identifies E5 as shifting the final mean by ∆= −0.6727 and E1 by ∆= +0.1520.The trace makes these evidence-level effects reproducible alongside local interpretations and posterior calculations.
Loading 2609.01337v1…