PaperlayerView exact revision on arXiv

Source-linked AI summary

When Should Forecasting Agents Reason? Behavioral Stress Tests for Reliability Routing

Yufeng Wang

arXiv:2609.28475v1cs.AIcs.CL

TL;DR

Forecasting agents combine reasoning, retrieval, and other behaviors, but it is unclear when each should be trusted across heterogeneous evidence sources. The paper introduces feature-based reliability routing and finds that walk-forward self-adjustment achieves the best mean Brier score among its deterministic systems across 16 LLM vintages, although gains are narrow and historical and search baselines remain competitive.

  • Problem

    It remains unclear whether forecasting questions should trigger uniform reasoning or instead route among reasoning, retrieval, market priors, and historical analogs as evidence sources differ.

    Source: §1 · p. 1

  • Method

    The paper builds a behavioral stress-test harness and RELIABILITYROUTE, which selects forecasting mechanisms using non-label reliability features rather than source-name decisions.

    Source: §1 · pp. 1–2

  • Results

    Source-dependent error profiles favor different mechanisms; across 16 LLM vintages, walk-forward self-adjustment achieved 0.1839 mean Brier versus 0.1867 for the frozen router.

    Source: §1 · p. 2, §7 · p. 8

  • Takeaways & Limitations

    Before adding more reasoning, a forecasting agent should estimate which evidence source deserves control and update that estimate only from resolved evidence.

    Source: §7 · p. 8

    The single-vintage gain is modest, its paired bootstrap interval against the strongest single baseline slightly overlaps zero, and HISTORICALANALOG remains close in multi-vintage mean Brier.

    Source: §6 · p. 8

Abstract

from arXiv · show

Forecasting agents increasingly combine language-model reasoning, retrieval, ensembling, and calibration, but it remains unclear when each behavior should be trusted. We study this question on ForecastBench-style binary forecasting tasks, treating the choice to retrieve, reason, defer to a market prior, or use a historical analog as an observable agent behavior rather than a hidden implementation detail. Our central finding is that mechanism choice is source-dependent: structured analogs dominate for some data-generating processes, while market/crowd-style and conservative baselines are better for others. We introduce ReliabilityRoute, a structural intervention that steers forecasting-agent behavior using reliability features such as historical coverage, market-prior availability, source-prior sharpness, evidence strength, evidence disagreement, and horizon. A fixed 2024-fitted rule closely matches a hand taxonomy without hard-coded source-name decisions, while a walk-forward self-adjusting rule refits thresholds from previously resolved vintages and obtains the best mean Brier score among our deterministic systems across 16 later LLM vintages. The gain is modest and historical/search baselines remain highly competitive. The main contribution is therefore a behavioral stress test showing that more reasoning is not always better; forecasting agents should first estimate which evidence source deserves control, routing policies should themselves adapt under auditable constraints, and reproducibility artifacts are available at https://github.com/louiswang524/forcastagent

1 Introduction

The paper frames forecasting as a source-dependent choice among reasoning, retrieval, priors, and historical analogs, and introduces a reliability-feature router rather than a monolithic agent. Walk-forward self-adjustment achieves the best mean Brier score among the reported deterministic systems across 16 LLM vintages, while historical and search baselines remain competitive.

  • Forecasting evidence varies by source, so uniform reasoning can invoke expensive or brittle behavior when simpler evidence is more reliable.Fitting component winners on calibration data can also overfit the source and horizon mixture of a vintage.Source: §1 · p. 1
  • Across 16 LLM vintages, walk-forward self-adjustment reduced mean Brier from 0.1867 for the frozen reliability router to 0.1839, beating the frozen rule on 10 of 16 vintages.Source: §1 · p. 2
  • 0.1846 Brier for RELIABILITYROUTE improved on 0.1876 for HISTORICALANALOG on the 2025-10-26 evaluation set.Source: §1 · p. 2
  • The reproducible ForecastBench-style harness treats mechanism choice as an observable behavior and compares multi-agent, evidence-graph search, historical-analog, and feature-routing mechanisms.Source: §1 · pp. 1–2
  • RELIABILITYROUTE replaces hard-coded source-name choices with auditable thresholds over coverage, market-prior availability, prior sharpness, evidence strength, disagreement, and horizon.Source: §1 · p. 2
  • The study supplies route and source-slice diagnostics, threshold-stability analysis, multi-vintage stress tests, oracle-routing headroom, and calibration ablations.These analyses examine when structured analogs, market/crowd priors, and agentic reasoning are useful.Source: §1 · p. 2

2 Method

The method evaluates binary-event forecasts and decomposes agents into behavioral mechanisms, including historical analogs, evidence-graph search, and a reproduced multi-agent baseline. RELIABILITYROUTE selects among mechanisms using non-label reliability features, with fixed and walk-forward threshold-fitting variants.

2.1 Problem Setting

  • Binary questions receive event probabilities, evaluated using Brier score, log score, and calibration error; lower Brier and log scores are better.Source: §2.1 · p. 2

2.2 Behavioral Mechanisms

  • For structured sources, the historical analog forecaster uses pre-cutoff observations over matching horizons and requires at least 20 comparable historical outcomes.When no historical analog is available, it falls back to the source prior in the question metadata.Source: §2.2 · pp. 2–3
  • The evidence-graph search loop plans queries for base rates, recent evidence, resolution criteria, and counterevidence, then updates a logit prior using retrieved evidence and aggregates role forecasts.Reported deterministic experiments use frozen benchmark evidence and leakage-checked analog documents, not live web search.Source: §2.2 · p. 3
  • The reproduced AIA-style baseline uses five focused agents, median aggregation by a supervisor, and a fixed 1.15 extremization slope in logit space.It is a reproducible comparator, not a claim about the original system’s exact performance.Source: §2.2 · p. 3

2.3 Reliability Routing

  • RELIABILITYROUTE uses a small rule over non-label reliability features, while the comparison policies include source-taxonomy and calibration-label-fitted routing.Its available features include historical coverage, market/crowd prior availability and sharpness, evidence strength and disagreement, and horizon.Source: §2.3 · p. 3
  • The fixed router freezes thresholds selected on 2024 calibration data; the self-adjusting router refits within the same pre-specified rule family using earlier resolved vintages.The feature set, rule order, threshold construction, and component forecasters are fixed before walk-forward scoring.Source: §2.3 · p. 4, Table 1 · p. 4

3 Experimental Setup

The experiments test transfer from a 2024 calibration vintage to later ForecastBench vintages, then assess chronological adaptation using only previously resolved vintages. Comparisons span forecasting mechanisms, scoring and routing metrics, and post-hoc calibration.

Benchmark

  • The primary transfer protocol calibrates design choices on the 2024-07-21 human-written vintage and evaluates on the 2025-10-26 LLM-written vintage without retuning.The self-adjusting protocol refits thresholds chronologically using only previously resolved vintages; combination questions are excluded.Source: §3 · p. 4, Table 2 · p. 5

Baselines

  • Comparators include historical analogs, search, market-only forecasting, diagnostic routers, an evidence-graph baseline, and a deterministic AIA-style reproduction.The oracle source and question routers use 2025 labels and serve only as diagnostic upper bounds.Source: §3 · p. 4

Behavioral and outcome metrics

  • Outcomes include Brier score, log score, calibration error, and selected mechanism, with paired question-level bootstrap intervals and threshold-perturbation diagnostics.Paired comparisons use 2,000 bootstrap samples; perturbations track both route changes and Brier-score changes.Source: §3 · p. 5

Calibration ablation

  • The calibration ablation freezes global or source-conditional Platt-style transformations fitted on the 2024 vintage before 2025 evaluation.Source-conditional fits require at least 20 calibration targets per source and otherwise fall back to the global fit.Source: §3 · p. 5

4 Results

The results distinguish fixed transfer on one vintage from walk-forward adaptation across 16 later vintages. They also examine route behavior, source-level tradeoffs, and routing and calibration ablations.

4.1 No-Retune Transfer

  • On the 2025-10-26 vintage, fixed RELIABILITYROUTE has the best deterministic non-oracle Brier score, improving by 0.0030 over HISTORICALANALOG.The self-adjusting rule is not better on this individual vintage; the paired-bootstrap interval against HISTORICALANALOG is [−0.0062, 0.0005].Source: §4.1 · p. 5

4.2 Behavioral Route Diagnostics

  • Fixed RELIABILITYROUTE routes 589 of 1,077 targets to historical analogs and 488 to the conservative baseline, while the fitted router sends 870 to graph/search behavior.The fixed rule nearly matches the hand taxonomy; the self-adjusting rule routes 1,004 targets to historical analogs and 73 to the conservative baseline.Source: Table 3 · p. 6, §4.2 · p. 6

4.3 Multi-Vintage Stress Test

  • Mean Brier falls from 0.1867 for the frozen reliability router to 0.1839 for the self-adjusting router across 16 vintages, the best mean among deterministic systems.It wins 7 of 16 vintages and beats the frozen rule on 10 of 16; the gain over HISTORICALANALOG at 0.1841 is modest.Source: Table 4 · p. 6, §4.3 · p. 6
  • Across vintages, single-threshold perturbations change 4.3% of routes and Brier by 0.0007 on average, while the rule leaves an audit trail of behavioral drift.The most disruptive perturbations change up to 21% of routes on some vintages.Source: §4.3 · pp. 6–7

4.4 Source-Slice Diagnostics

  • Historical analogs lead on ACLED, FRED, and Wikipedia, while the reproduced multi-agent baseline is stronger for DBnomics, YFinance, and several market-family sources.Self-adjustment helps some small market-like slices but over-shifts toward analogs on DBnomics and YFinance.Source: Table 5 · p. 7, §4.4 · p. 7

4.5 Ablations and Negative Results

  • The oracle source router reaches 0.1844, only 0.0002 Brier better than fixed RELIABILITYROUTE, while the question-level oracle remains an upper bound using evaluation labels.The source-level comparison suggests reliability routing captures much of the single-vintage headroom.Source: Table 6 · p. 7, §4.5 · p. 8
  • FITTEDROUTE’s source-specific choices from the 2024 calibration vintage fail on the 2025 vintage, yielding Brier 0.1949, worse than HISTORICALANALOG and SEARCHLOOP.The result supports constraining routing by reliability structure or validating it across vintages.Source: §4.5 · p. 7
  • Global calibration worsens Brier from 0.1846 to 0.1882, and source-conditional calibration worsens it to 0.2004.The authors suggest calibration mismatch is not a single global slope/intercept problem and that source-level calibration overfits small slices.Source: §4.5 · p. 7

5 Related Work

The related work places the study among probabilistic forecasting, human forecasting, prediction markets, and LLM forecasting systems. Its focus is mechanism choice as observable behavior.

Forecast evaluation

  • Probabilistic forecasting is commonly evaluated with proper scoring rules.ForecastBench provides a dynamic benchmark for AI, public, and expert forecasts on future-resolving questions (Karger et al., 2024).Source: §5 · p. 8

Human forecasts and markets

  • Human forecasting research studies the training and evaluation of expert forecasters, while prediction markets aggregate dispersed information through prices.Source: §5 · p. 8

LLM forecasting systems

  • AIA Forecaster combines agentic search, supervisor aggregation, and calibration (Alur et al., 2025); this study instead isolates mechanism choice as observable agent behavior.Source: §5 · p. 8

6 Limitations

The study is component-level, and its gains are modest or narrow across the reported settings. Its routing features and calibration behavior leave important scope boundaries for interpretation and use.

Component-level comparison

  • The reproduced AIA-style baseline differs from the reported system in search stack, prompts, models, and calibration, so it is a comparison baseline rather than a reproduction claim.Source: §6 · p. 8

Performance scope

  • The multi-vintage gain is narrow: HISTORICALANALOG is close, and several vintages are won by simple baselines or the hand taxonomy.The single-vintage gain is modest, and its paired-bootstrap interval against the strongest single baseline slightly overlaps zero.Source: §6 · p. 8

Routing-policy validation

  • Reliability thresholds come from a human-designed rule family; the authors call for validating adaptation with nested development splits and external forecasting datasets.Source: §6 · p. 8

Behavioral scope

  • The behavioral analysis observes mechanism selection and route-level features, not search-query refinement, tool-use trajectories, or multi-agent disagreement dynamics.Source: §6 · p. 8

Calibration

  • Brier and log score improve on some vintages while calibration error often worsens, and the tested global and source-conditional calibration attempts fail to transfer.The authors identify conditional calibration as unsolved for downstream users requiring calibrated probabilities.Source: §6 · p. 8

7 Conclusion

Forecasting agents should choose behaviors according to source-dependent error profiles rather than reason uniformly. The paper recommends estimating which evidence source deserves control and updating that estimate only from resolved evidence.

  • Historical analogs, market/crowd-style baselines, and search/graph reasoning have different source-dependent error profiles.Source: §7 · p. 8
  • A feature-based reliability router can improve performance on individual vintages, while walk-forward self-adjustment improves mean multi-vintage Brier.Source: §7 · p. 8
  • The paper’s practical lesson is to estimate which evidence source deserves control before adding more reasoning, and update that estimate only from resolved evidence.It identifies within-source forecasting, richer behavioral traces, and robust conditional calibration as areas for future complexity.Source: §7 · p. 8

Reproducibility Statement

The reproducibility artifact provides the deterministic forecasting-agent harness and materials for rerunning the reported experiments. Deterministic results require no API key.

Ethics Statement

Probabilistic forecasting systems can support better decision-making, but overconfident or poorly calibrated forecasts may cause harm without human oversight. The paper reports calibration error, negative transfer results, and limitations alongside accuracy metrics.

  • Overconfident or poorly calibrated probabilistic forecasts may cause harm if used without human oversight.Source: Ethics Statement · p. 9
  • Probabilistic forecasting systems can support better decision-making.Source: Ethics Statement · p. 9
  • The paper reports calibration error, negative transfer results, and limitations alongside accuracy metrics.Source: Ethics Statement · p. 9
Loading 2609.28475v1…