From Narrative to Auditable Forecasts: A Structured Scaffold for Agentic Forecasting

1 Introduction

• Existing agentic forecasts often balance evidence narratively and assign probabilities without an explicit computational path, making calibration and post-event inspection difficult.The passage notes that qualitative balancing can produce suboptimally calibrated estimates, especially when a natural quantitative anchor exists.Source: §1 · p. 2
• AUDITFORECAST derives a base probability from a suitable quantitative model and model-guided retrieval, then mechanically combines out-of-scope situational updates in odds space.This makes forecast construction a structured process with explicit intermediate objects.Source: §1 · p. 2
• AUDITFORECAST decomposes forecasts into a model-derived base probability and situational factor updates, and its report supports systematic post hoc analysis.The paper frames this as addressing the structural limitation of narrative aggregation.Source: §1 · p. 2
• AUDITFORECAST improves accuracy over strong agentic baselines, surpasses market-implied references, and is Pareto-dominant in cost–accuracy against substantially more expensive deep-research agents.The introduction also describes the scaffold as producing an auditable report for post hoc analysis.Source: §1 · p. 2

2 Related Work

• Prior LLM forecasting work explores forecast aggregation, retrieval-augmented pipelines, prompting, and training-based approaches including fine-tuning, self-play, and reinforcement learning.The cited work is presented as establishing a promising foundation for LLM-based forecasting.Source: §2 · p. 2
• ForecastBench, ProphetArena, and FutureX provide live evaluation settings for probabilistic forecasts on unresolved real-world events, with FutureX also assessing search, planning, and tool use.The passage describes these benchmarks as an evaluation substrate for forecasting agents.Source: §2 · p. 2
• Unlike benchmark work centered on evaluation, this paper focuses on a forecasting scaffold that makes the evidence-to-probability mapping explicit, auditable, and better calibrated.Figure 2 depicts the AUDITFORECAST workflow alongside its forecasting report.Source: §2 · p. 2, Figure 2 · p. 3

3.1 Overview

• A quantitative baseline models the expected trajectory under ordinary conditions, while case-specific developments outside its scope provide complementary updates.The paper gives a sudden geopolitical event shifting market sentiment as an example of an external factor.Source: §3.1 · p. 3

3.2 Structured Forecast Decomposition

• The decomposition treats p0 as prior odds and each situational factor as a likelihood-ratio update; factor-level multiplication follows when evidence variables are conditionally independent given the outcome.The resulting posterior odds are converted to the final forecast probability.Source: §3.2 · p. 4

3.3 The AUDITFORECAST Scaffold

• Stage 1 selects a suitable quantitative model, records its inputs and assumptions, retrieves model-required data, and computes the base probability p0.Model-guided retrieval steers search toward inputs needed to instantiate the chosen model.Source: §3.3 · p. 4
• Factor identification starts from the constructed baseline, seeking relevant evidence it does not encode and checking that proposed factors are outside its scope and sufficiently distinct.Candidates sharing an underlying causal mechanism are revised, merged, or discarded to address the conditional-independence assumption.Source: §3.3 · pp. 4–5
• For each validated factor, the agent assesses its evidence-based direction and assigns an odds multiplier LRi; the multipliers and p0 determine the final probability.LRi > 1 favors YES, LRi < 1 favors NO, and LRi = 1 is neutral.Source: §3.3 · p. 5
• The structured report records the baseline derivation, supporting evidence and scope for situational factors, and the final aggregation, enabling inspection and post hoc analysis.The decomposition records p0 and each factor’s quantitative update term.Source: §3.2 · p. 4, §3.3 · p. 5

4.1 Experimental Settings

• The evaluation covers Polymarket, Kalshi, and FutureX with GPT-5.4, GPT-5.4-mini, and Gemini-3-Flash; the reported metric is Brier score, where smaller is better.The live evaluation includes questions resolving in April–May 2026, with 831 questions for GPT-5.4 and GPT-5.4-mini and 709 for Gemini-3-Flash.Source: §4.1 · pp. 5–6

4.2 Main Results

• Across backbones and benchmarks, AuditForecast consistently improves over Think&Search; all tested Polymarket backbones and GPT-5.4 on Kalshi also beat market-implied references.Adding factor updates further improves performance in all reported settings.Source: §4.2 · p. 6, Figure 3 · p. 6

4.3 Effect of Varying Reasoning Effort

• At low, medium, and high reasoning effort, AuditForecast outperforms Think&Search and has lower Brier scores than the market-implied reference; it remains comparatively stable at low effort.At medium and high effort, it uses substantially fewer searches than Think&Search while achieving better accuracy.Source: §4.3 · p. 6, Figure 4 · p. 7

4.4 Comparison with Deep Research Agents

• On 142 Polymarket questions, AuditForecast is Pareto-dominant in cost and accuracy, beating three deep-research agents and using substantially fewer web searches.It also improves over the Market-Implied reference; the comparison uses GPT-5.4 for AuditForecast and smolagents-deep-research.Source: §4.4 · p. 7, Figure 5 · p. 7

4.5 Comparison with Time Series Forecasting

• On value-threshold forecasts, AuditForecast outperforms Chronos-2 and TimesFM, and its base probability alone already surpasses both time-series models.The comparison uses one-year historical series for 369 questions; the paper notes that time-series-only models cannot capture real-time developments and news.Source: §4.5 · p. 7, Table 3 · p. 7

4.6 Component-Level Ablation Study

• In component ablations, the full pipeline achieves the best Brier score, while removing or replacing any component degrades performance.Model-guided retrieval uses fewer searches and yields a more accurate base probability than retrieval-then-modeling; odds-space updates also preserve valid probabilities, unlike direct probability-space addition.Source: §4.6 · p. 8, Table 4 · p. 8

5.1 Auditing Base Probability Derivation

• For stock-range forecasts, 91.8% use a Parametric CDF, typically assuming a zero-drift lognormal price distribution and evaluating its CDF at the question threshold.The audit finds that agents tend to converge on a modeling method for questions of the same type but diverge on the data source used to instantiate it.Source: §5.1 · pp. 8–9
• Each dominant volatility-input subtype—implied, realized, or combined—has a lower Brier score than the Market-Implied reference on the matched subset.The result supports the Parametric CDF template as competitive with any of these reasonable volatility inputs.Source: §5.1 · pp. 8–9, Figure 6 · p. 9

5.2 Auditing Factor Updates

• For GPT-5.4 stock-range forecasts, 73% of factor directions are correct and 66% of factor magnitudes are well calibrated.The audit describes magnitudes as asymmetric: the agent rarely under-reacts, while overstated factors are more common than understated ones.Source: §5.2 · p. 9, Figure 7 · p. 9
• The agent captures 74% of obtainable factors, but misses some signals and cross-source patterns; retrieved signals are sometimes not made explicit as factors.The audit limits missing-factor analysis to information available at forecast time and reports an Iran-ceasefire factor appearing in only about half of forecasts resolving on 2026-04-22.Source: §5.2 · p. 9

• Across multiple live benchmarks, AuditForecast improves accuracy over strong agentic baselines and market-implied references while supporting systematic post hoc analysis through auditable reports.Source: §6 · p. 9

Limitations

• Explicit forecast construction and auditable reports support rich post hoc analysis.The authors describe systematic, comprehensive analysis and reusable experience as future work toward a more self-evolving system.Source: Limitations · p. 10
• Post hoc analysis exposes accuracy differences across data sources for the same base-probability model and signal classes agents may overlook when identifying situational factors.The authors call for more systematic analysis of recurring failures in model selection, data retrieval, and factor updating, then feeding patterns back into the scaffold.Source: Limitations · p. 10
• The current scaffold makes forecasting behaviors visible but does not yet systematically turn those observations into reusable improvements to the forecasting system.Systematic feedback into the scaffold is identified as an important direction for future work.Source: Limitations · p. 10

Ethical Considerations

• AuditForecast does not guarantee correctness or eliminate bias in underlying models, retrieved information, or probabilistic judgments.The authors note that forecasting agents may influence real-world decisions in finance, politics, and public affairs.Source: Ethical Considerations · p. 10
• In high-stakes settings, AuditForecast is a decision-support tool rather than a substitute for expert judgment.Its structured report may help users inspect forecast construction, but should not be interpreted as a guarantee of reliability.Source: Ethical Considerations · p. 10
• Explicit, auditable outputs support closer scrutiny of agent behavior and post hoc analysis when failures occur.The authors describe this transparency as a step toward safer deployment, not as a guarantee of reliability.Source: Ethical Considerations · p. 10

A Bayesian Derivation

• The derivation defines Y as the YES/NO event outcome and E as evidence associated with validated factors.It applies Bayes’ rule to relate the event outcome and evidence.Source: A Bayesian Derivation · p. 12
• Posterior odds equal prior odds multiplied by a likelihood-ratio term.The derivation obtains the odds form by taking the ratio of the YES and NO probability expressions.Source: A Bayesian Derivation · p. 12
• If evidence variables are conditionally independent given the event outcome, the joint likelihood ratio factorizes across factors.This conditional-independence assumption supports the odds-space aggregation across validated factors.Source: A Bayesian Derivation · p. 12
• AuditForecast instantiates prior odds from the quantitative-model-derived base probability p0 and represents the final forecast with final odds and probability.The derivation defines a factor-level likelihood-ratio term before expressing posterior odds through the factor updates.Source: A Bayesian Derivation · p. 12

B.1 Evaluation Datasets

• Evaluation questions were unresolved at test time and drawn from Polymarket, Kalshi, and FutureX during an April–May 2026 window.FutureX Levels 1 and 2 multiple-choice questions were converted into independent binary problems; overlaps were removed, and resolution was within one week of forecast run time. Evaluation sets were not perfectly identical across foundation models.Source: §B.1 · p. 12

B.2 Agentic Scaffold Implementation

• The scaffold uses a single agent, hosted web-search tools, and a structured output schema, with no explicit limit on search calls.It is implemented with the OpenAI Agents SDK and Google’s Agent Development Kit.Source: §B.2 · pp. 12–13

AUDITFORECAST System Prompt

• The system prompt directs the agent to derive a base probability, identify independent factors outside its scope, and combine their odds multipliers mechanically.It defines p0 as the baseline YES probability under the default expected trajectory before additional factors; quantitative models are preferred when naturally applicable, otherwise qualitative assessment is allowed.Source: AUDITFORECAST System Prompt · p. 13

AUDITFORECAST User Prompt

• The user prompt supplies the forecasting question, resolution criteria, and available dates, then requests a structured JSON report of evidence, assumptions, factors, and the final probability.The output schema records the base probability and odds, factor evidence and assessments, and the Bayesian update; arithmetic is assigned to code execution.Source: AUDITFORECAST User Prompt · p. 13

C Qualitative Case Comparison: Deep Research Agents vs. AUDITFORECAST

• Deep-research agents collected relevant evidence but left it unquantified and mechanically uncombined, incurring substantial search cost while producing suboptimal accuracy.AUDITFORECAST instead decomposes the base probability and factor-level updates to show how evidence maps to a forecast.Source: C Qualitative Case Comparison: Deep… · p. 14

FINAL_PROBABILITY: 0.75

• In the Apple threshold case, the three deep-research reports assigned probabilities of 0.75, 0.45, and 0.43 despite the No ground truth.The 0.75 report argued that bullish momentum made a close above $300 likely; the other two leaned below an even chance.Source: C Qualitative Case Comparison: Deep… · p. 14, FINAL_PROBABILITY: 0.75 · p. 14, FINAL_PROBABILITY: 0.45 · p. 15

FINAL_PROBABILITY: 0.43

• AUDITFORECAST assigned 0.19 after deriving a zero-drift, one-session lognormal base probability and applying factor likelihood ratios in odds space.Its report shows the base probability calculation and the resulting product of factor multipliers.Source: FINAL_PROBABILITY: 0.43 · p. 15
• The reports illustrate the same auditable workflow beyond financial questions, with examples for Olivia Dean’s streaming total and a Mariners–Cardinals game.The examples use different underlying anchors while retaining an explicit path from evidence to forecast probability.Source: FINAL_PROBABILITY: 0.43 · pp. 15–16

D.1 Base Probability Hierarchy Extraction

• Base-probability hierarchy extraction groups reports of the same question type by quantitative method and data source, using GPT-5.4 as judge with human validation.The hierarchy is initialized from 100 reports and refined incrementally in batches.Source: §D.1 · p. 16
• The extraction procedure preserves existing codes, adds or splits uncovered categories, and distinguishes qualitative no-formula cases (E1) from numeric heuristic p0 computations (E2).A judge independently classifies each computation within the resulting method-by-data-source hierarchy.Source: §D.1 · p. 17

D.2 Auditing Factor Updates

• A web-search-based post hoc audit reconstructs events from forecast run time to resolution, then assesses factor direction and magnitude and identifies missing drivers.The forecast pipeline starts from base probability p0, multiplies odds by each factor’s likelihood ratio, and converts final odds back to probability.Source: §D.2 · p. 17

D.3 Effectiveness of Independence Checking

• For events with merged correlated factors (43% of evaluated events), independence checking reduces redundant factor retention and excessive odds-space updates, improving Brier score.Without the check, correlated signals can contribute separate multiplicative updates, producing larger adjustments from the base probability and more overconfident forecasts.Source: Table 7 · p. 18, §D.3 · p. 18

D.4 Generic Structured Scaffold

• AUDITFORECAST consistently outperforms the free-form structured scaffold across Polymarket and Kalshi and across model backbones.The free-form method often starts from an unjustified 0.5 baseline and applies linear evidence weights. AUDITFORECAST instead justifies p0, limits updates to developments outside the baseline model’s scope, and aggregates them explicitly.Source: §D.4 · p. 18, Table 9 · p. 19

D.5 Stability Analysis

• 86% of GPT-5.4 and 88% of GPT-5.4-mini factor clusters had consistent directions across paraphrases, with small likelihood-ratio variability.Table 10 reports stability of likelihood ratios across naturally occurring factor paraphrases.Source: §D.5 · p. 19, Table 10 · p. 19
• Average per-question standard deviation was 0.041 for GPT-5.4 and 0.039 for GPT-5.4-mini across independent runs.For both models, 94–96% of questions had a standard deviation below 0.10.Source: §D.5 · p. 19
• The stability evaluation used five independently seeded runs on 50 Polymarket events resolved before July 10.It assessed both end-to-end forecast variability and consistency of factor-level likelihood ratios across naturally occurring paraphrases.Source: §D.5 · p. 19