PaperlayerView exact revision on arXiv

Source-linked AI summary

Agentic Forecasting with Structured Linguistic Beliefs

Kevin Murphy

arXiv:2604.18576v5cs.AI

TL;DR

Search-enabled forecasting agents accumulate results in context, raising concerns about context rot. BLF adds a structured linguistic belief state to a ReAct-style forecasting loop. On 400 historical ForecastBench question-date instances, it had the highest overall Brier Index point estimate among the compared external agents.

  • Problem

    Search-enabled forecasting agents accumulate results in context, which can create context-rot problems.

    Source: §1 · p. 1

  • Method

    BLF augments a search-enabled ReAct loop with an LLM-updated belief state containing a probability estimate and structured natural-language evidence summaries.

    Source: §1 · p. 1, §3 · p. 3

  • Results

    On 400 ForecastBench question-date instances, BLF had the highest overall Brier Index point estimate among the compared external agents.

    Source: §8 · p. 9

  • Takeaways & Limitations

    The paper studies structured linguistic beliefs as a form of context engineering for search-enabled forecasting agents.

    Source: §8 · p. 9

    The evaluation is restricted to binary outcomes and backtesting and does not establish live forecasting performance or complete cost-normalized superiority.

    Source: §8 · p. 9

Abstract

from arXiv · show

We propose the Bayesian Linguistic Forecaster (BLF), which is a search-based ReAct style agent for (binary) forecasting that augments its context with a linguistic 'belief state', consisting of a probability estimate and a \emph{structured} natural-language evidence summary. (The belief update is 'Bayesian-inspired' rather than exact Bayesian inference, although we do include a comparison to a more explicit Bayesian approach using LLM-elicited likelihood surrogates.) On 400 historical ForecastBench question-date instances, BLF has the highest overall Brier Index among the compared external agents, and is the only compared method to significantly outperform a crowd+empirical-prior baseline in terms of overall performance. Our backtesting framework combines date-aware search tools, leakage auditing, and paired comparisons for variance control to provide a rigorous empirical framework for comparing agentic forecasting systems.

1 INTRODUCTION

Forecasting agents can accumulate search results in context, creating context-rot concerns. BLF adds a probability-and-evidence belief state, and the paper reports strong ForecastBench comparisons while qualifying the Bayesian analogy and the historical evaluation.

1 INTRODUCTION

  • BLF pairs a probability estimate with natural-language evidence summaries in a belief state updated by the LLM at each step.The update is designed to resemble a sequential Bayesian update, but is an LLM forward pass rather than exact Bayesian inference.Source: §1 · p. 1
  • The evaluation combines a four-layer date-leakage defense with paired bootstrap comparisons to control for question difficulty, which accounts for 62% of performance variance.The post-hoc audit estimated residual leakage at approximately 1.5% of retained search results.Source: §1 · p. 1
  • BLF achieved the highest Brier Index on every question type against the top methods in 400 ForecastBench backtesting questions.The comparison covered Overall, Market, and Dataset questions.Source: §1 · p. 1
  • The LLM-generated probability updates may not meet the consistency criteria for proper Bayesian inference.The paper compares this heuristic with an explicit Bayesian procedure using LLM-estimated likelihoods, which had lower point-estimate accuracy.Source: §1 · p. 1
  • The reported leaderboard comparison uses results frozen when the work was done on 2026-04-20, although the LLMs and leaderboard have since been updated.The authors consider the frozen results useful for comparing methods.Source: §1 · p. 2

2 EXPERIMENTAL SETUP

The evaluation uses historical binary ForecastBench questions, with market and dataset problems represented across two tranches. It compares forecasting agents under date-aware backtesting controls and reports Brier Index scores.

2 EXPERIMENTAL SETUP

  • ForecastBench combines judgmental market questions with binarized time-series questions from multiple data sources.The setup defines the task as estimating the probability of a binary outcome at a later resolution date given data available by the forecast date.Source: §2 · p. 2
  • The evaluation contains 400 question-date instances across two ForecastBench tranches, yielding 791 binary resolution events because dataset questions can resolve at multiple horizons.Each tranche contains 100 market and 100 dataset questions; the 400 instances represent 355 distinct underlying questions.Source: §2 · p. 2
  • The Brier Index is 100 × (1 − Brier Score), with 100% best, 0% worst, and 50% for chance; overall ForecastBench scores average market and dataset means equally.The mean Brier Score is based on squared prediction errors for binary outcomes.Source: §2 · p. 2
  • Backtesting uses models with knowledge cutoffs before the tranche dates and four leakage defenses, while a second-model audit flags post-cutoff information in 1.5% of retained search results.The defenses include search date filtering, an LLM leak classifier, date-clamped data tools, and URL blocking for resolution sources; the authors caution that these controls cannot guarantee zero leakage.Source: §2 · p. 2
  • External comparisons include Cassi, Grok 4.20, Foresight-32B, and a crowd-plus-empirical-prior baseline; NoBel retains the search loop but omits structured beliefs.The experimental setup also specifies three LLMs for the paper's own runs and an algorithm that applies date cutoffs and leak filtering to search results.Source: §2 · pp. 2–3

3 METHOD

BLF modifies a ReAct forecasting loop by carrying a structured linguistic belief state alongside actions and observations. The method also specifies information sources and ways to combine independent forecast trajectories.

3 METHOD

  • At each step, BLF produces a belief state and an action, receives tool results, and appends the resulting information to either the full history or the initial context.The loop can run for up to 10 steps, but the agent may submit early.Source: §3 · p. 3
  • The belief state is a semi-structured JSON object containing a probability, confidence level, evidence for and against, and open questions.NoBel retains the full history without requesting or maintaining this structured belief.Source: §3 · p. 3
  • BLF can search and summarize web pages, fetch URLs, or use source-specific data tools before submitting a forecast.A meta-controller selects available tools by question type; the submit and timeout paths use different probability clipping ranges.Source: §3 · p. 3
  • Market questions receive a crowd-price anchor, while dataset questions receive an empirical source-and-subtype prior that has negligible effect on BLF but helps the no-LLM baseline.Source: §3 · pp. 3–4
  • Independent trajectories can be combined by arithmetic or logit averaging, or by shrinkage toward 0.5; preliminary LLM-based aggregation performed worse than simple mean aggregation on ForecastBench.A calibration comparison found no statistically resolved overall Brier Index gain for Pro in the historical comparison, without ruling out gains on other backbones or subsets.Source: §3 · p. 4

4 RESULTS

BLF has the highest reported Brier Index point estimate across Overall, Market, and Dataset, and is the only compared method to significantly outperform the crowd+empirical-prior baseline on Overall BI.

Comparison to SOTA

  • BLF has the highest reported BI point estimate for Overall, Market, and Dataset, with significant Overall and Dataset gains over Cassi and Foresight.Grok is not statistically distinguishable from BLF on its tranche-A coverage; only the market gap against Foresight reaches significance.Source: §4 · p. 4, Table 1 · p. 5

Post-hoc calibration

  • Calibration gains were not statistically resolved overall for Pro in the historical comparison.Platt and hierarchical calibration showed BI gains on questions with and without the crowd baseline, but no statistically resolved overall gain.Source: §4 · p. 4

Comparison to LLM-free baseline

  • Among methods compared, BLF alone significantly improves Overall BI over the LLM-free crowd+empirical-prior baseline on its full coverage; Grok and Cassi trend positive but are not significantly above it.Source: §4 · p. 5, Table 2 · p. 5

Benefits of the belief state

  • Adding the structured belief state improves overall BI for PRO and FLASH, mainly through Market questions, but yields no resolved overall gain for GPT.Market cross-trial standard deviation fell from 0.094 to 0.030 for PRO and 0.105 to 0.032 for FLASH; it rose slightly for GPT.Source: §4 · p. 5, Table 3 · p. 6

Benefits of averaging multiple trials

  • Increasing the number of independent trials improves BI with arithmetic averaging, while logit averaging gives a small additional boost.Leave-one-question-out tuning selects αq = 1, making shrinkage equivalent to standard logit averaging.Source: §4 · p. 5, Figure 2 · p. 6

Benefits of the belief state

  • Earlier Kimi-K2.5 gains were omitted because they confounded forecast quality with recovery from submission failures.NoBel market submission rate was 6.2%, versus 82.4% with belief tracking.Source: §4 · p. 5

Full-history versus Markov memory

  • In the 100-question memory study, Markov memory has a higher Overall BI point estimate than full-history memory, but every interval includes zero.The difference is +2.40 BI (95% CI [−0.17, 4.36]) and favors Markov mainly on markets.Source: §4 · p. 6, Figure 2 · p. 6

5 EXAMPLE OF A BELIEF STATE TRACE

Across five trials on one question, different search paths produce divergent probability trajectories, motivating aggregation; Trial 2 identifies a static-map uncertainty missed by the other four.

5 EXAMPLE OF A BELIEF STATE TRACE

  • Across five trials, probabilities begin at 0.5 and diverge after early skeptical updates as different search paths produce different conclusions.The inter-trial standard deviation is 0.20; the mean forecast is 0.57.Source: §5 · pp. 6–7, Figure 3 · p. 7
  • In Trial 2, probability initially fell to 0.05, rose to 0.65 after finding the executive order, stayed there for 7 search steps, then incorporated static-map uncertainty at step-10 submission and fell to 0.28.Source: Figure 4 · p. 7
  • Trial 2’s final reasoning synthesizes latent static-map concerns, lowering its probability from 0.65 to 0.28 and identifying the uncertainty the other four trials miss.The change is not prompted by a new search result, and the mean forecast does not beat the best individual trial on this example.Source: §5 · p. 7

6 EXPLICIT BAYESIAN UPDATING

The paper tests explicit Bayesian updates using LLM-elicited likelihood surrogates, alternative evidence conditioning, and tempering. On the reported replay, these likelihood-based approaches score below BLF.

6 EXPLICIT BAYESIAN UPDATING

  • Likelihood estimates can condition on the query, raw history, or structured belief evidence, but exact chain-rule updating is not assured without full-history conditioning or correctly specified likelihoods.The explicit update also uses a tempering coefficient.Source: §6 · p. 8
  • The authors elicit typicality ratings from 0–10 for each possible outcome and use them as likelihood surrogates, with a floor preventing infinite weights.The bounded scale also caps each log-likelihood term; the scores are not normalized text likelihoods.Source: §6 · p. 8
  • Task-aware summaries average 798 characters, about 11% of the 7,571-character raw observations, and are conditioned on the question and query.The likelihood is evaluated on the summary instead of the raw observation.Source: §6 · p. 8
  • At shared α=0.35, removing the belief summary increases BS by 0.0057, but its incremental benefit is unresolved.The 95% paired question-bootstrap CI is [−0.0122, 0.0290]; α was selected on this sample, and likelihood scores were elicited on different dates.Source: §6 · p. 8
  • BLF scores BI=70.38 and ECE=0.083, while the best likelihood-based approach scores BI=65.73 on the reported replay.The best likelihood approach conditions on the belief state plus summarized past evidence; the table replays methods on the same evidence sequence.Source: §6 · p. 9, Table 4 · p. 9

7 RELATED WORK6

The paper situates BLF among retrieval-augmented forecasting systems, emphasizing its structured running belief summary as a context-engineering change rather than an exact Bayesian algorithm.

7 RELATED WORK6

  • Retrieval-augmented forecasting systems combine search, reasoning, aggregation, and calibration.The cited examples include Halawi et al. (2024), Hsieh et al. (2024), and Alur et al. (2025).Source: §7 · p. 9
  • BLF follows ReAct’s sequential reasoning-and-action pattern but uses a structured running probability-and-evidence summary, not a new exact Bayesian inference algorithm.The paper frames its contribution as an empirical study of this representation.Source: §7 · p. 9
  • BayesAgent constructs explicit graphical models, while forecasting-model training is presented as complementary to changes in the agent harness.Likelihood-based proposals motivate the paper’s negative results in Section 6.Source: §7 · p. 9

8 CONCLUSION

The paper reports BLF’s highest overall Brier Index point estimate among compared external agents on 400 ForecastBench question-date instances, while limiting its claims to binary-outcome backtesting.

8 CONCLUSION

  • BLF has the highest overall Brier Index point estimate among compared external agents on 400 ForecastBench question-date instances.The evaluation does not establish live forecasting performance or complete cost-normalized superiority.Source: §8 · p. 9
  • The evaluation is restricted to binary outcomes and backtesting; cutoff filters reduce but cannot eliminate leakage risk.The paper also notes that its memory comparison is exploratory and excludes incomplete paired submissions.Source: §8 · p. 9

AI USE STATEMENT

  • The paper used generative AI tools to help write code, launch and analyze experiments, and prepare writing and figures.Source: AI USE STATEMENT · p. 10

ETHICS

  • Because forecasting systems can have sensitive uses, including geopolitical forecasting, the paper recommends caution and human oversight.Source: ETHICS · p. 10

REPRODUCIBILITY STATEMENT

A.1 DATASETS

  • The evaluation uses AIBQ2 and ForecastBench, including 113 AIBQ2 questions and roughly 500 ForecastBench questions per fortnightly round.Source: §A.1 · p. 15

A.2 METRICS

  • Brier Score is lower-is-better, while Brier Index is higher-is-better; within a common evaluation split, BI and mean BS give equivalent rankings.The paper also describes adjusted Brier Score as controlling for question difficulty.Source: §A.2 · p. 15

A.3 EXTERNAL METHODS WE COMPARE TO

The section identifies search-based ForecastBench comparators and describes the safeguards used to make historical backtests more credible. It also reports the models and compute setup used for the experiments.

A.3 EXTERNAL METHODS WE COMPARE TO

  • The leaderboard snapshot lists Cassi at ABI 67.9, Grok 4.20 at ABI 67.8, and Foresight-32B at ABI 67.2, but their scores cover different resolved questions.The authors therefore characterize leaderboard comparisons as approximate and point to paired analysis on shared questions for rigorous comparisons.Source: Figure 5 · p. 16, §A.3 · p. 16

A.4 LLMS USED

  • The experiments use Gemini-3.1-Pro for most agent runs and Gemini-3-Flash for search filtering and summarization, with a 10-step and 10-minute cap.A full evaluation of five trials on 400 questions takes 1–3 hours using 50 parallel workers; one method on the full A ∪ B dataset costs about $250 at stated Gemini pricing.Source: §A.5 · pp. 16–17

B.1 MITIGATING SEARCH LEAKAGE.

  • Brave Search applies a freshness cutoff, then an LLM classifier checks page content because a pre-cutoff timestamp can coexist with post-cutoff updates.The authors report other evaluated engines lacked reliable date filtering or full page text, which harmed downstream performance.Source: §B.1 · p. 18
  • Data tools enforce observation-date cutoffs, but this does not ensure historical data-vintage correctness for sources that revise past values.Search results and URL lookups are also blocked from known resolution-source URLs that could reveal outcomes.Source: §B.1 · p. 18

B.2 MITIGATING PARAMETRIC LEAKAGE.

  • To reduce parametric leakage, the authors use models whose advertised knowledge cutoffs precede the ForecastBench forecast dates.They note that larger frontier models may carry greater parametric-leakage risk because memorized world knowledge scales roughly log-linearly with effective parameter count (Li, 2026).Source: §B.2 · p. 18

B.3 POST-HOC LEAKAGE AUDIT.

The section audits residual search leakage and describes the forecasting agent’s belief representation, tool loop, and question-specific actions. It also details the retrieval and time-series tools and the policy choices used to select them.

B.3 POST-HOC LEAKAGE AUDIT.

  • Among 1,377 search results retained by the runtime filter, the post-hoc audit flags 21 leaks (1.5%); the runtime filter has 93.8% recall and a 29.9% false-positive rate.These are second-LLM verdicts rather than human-verified ground truth, and the flagged-exposure rate does not measure how much evidence affected forecasts.Source: §B.3 · p. 19, Table 6 · p. 19

C.1 LINGUISTIC BELIEF STATE

  • The linguistic belief state is a semi-structured running representation that pairs a probability estimate with natural-language evidence summaries.It also tracks confidence, evidence for and against the outcome, and open questions for further investigation.Source: §C.1 · p. 20

C.2 AGENT LOOP

  • At each step, the LLM generates an action and updated belief; the environment returns a cutoff-restricted observation that is added to the agent’s message history.Full-BLF retains the history, while Markov-BLF retains the fixed prompt and newest action, observation, and belief; the update is a working-memory scaffold, not a formal Bayesian posterior.Source: §C.2 · pp. 20–21

C.3 WEB SEARCH TOOLS

  • The agent receives search snippets first and can selectively load saved page content into a sub-LLM for question-specific summaries, avoiding flooding the main context with irrelevant results.Brave supports date filtering; the authors note that BLF can work with any search engine.Source: §C.3 · p. 22

C.4 TIME-SERIES TOOLS

  • Time-series tools retrieve historical values only through the forecast date, and can return a model-based estimate alongside the raw history.DBnomics uses the estimate directly because the LLM struggled to interpret raw data; yfinance and FRED use a combined history-and-estimate tool for efficiency.Source: §C.4 · p. 22

C.5 OTHER TOOLS

  • Market-price history tools were not found useful beyond web search so far, possibly because the LLM does not know how to interpret the data.Specific-URL lookup is constrained by a blacklist of sites that may contain the outcome; Wikipedia retrieval can use a snapshot at or before the forecast date.Source: §C.5 · p. 23

C.6 POLICIES

  • A policy selects among question-specific tools and may stop at any time, subject to a 10-step maximum.The current meta-controller forces DBnomics questions to call a modeling tool before submitting; for other types, the LLM may choose among applicable tools in any order.Source: §C.6 · p. 24

C.7 CROWD ESTIMATE AND EMPIRICAL PRIORS

This section describes how BLF uses crowd estimates or empirical priors and how it aggregates and calibrates forecasts. It distinguishes empirical performance choices from formal guarantees and notes a limitation of the calibration evaluation.

C.7 CROWD ESTIMATE AND EMPIRICAL PRIORS

  • On dataset questions, submitting empirical priors achieves BI 58.3, while the full system achieves BI 62.7.For market questions, simply predicting the crowd estimate achieves BI ≈81.5; crowd signals do not apply to dataset questions, and empirical priors do not apply to market questions.Source: §C.7 · pp. 24–25

C.8 MULTI-TRIAL AGGREGATION

  • Arithmetic averaging improves expected Brier score as the number of trials increases, and its BI improvement is confirmed empirically.Logit averaging does not consistently outperform arithmetic averaging; the paper’s preference for it on ForecastBench is empirical rather than formal.Source: §C.8 · p. 25

C.9 SHRINKAGE AGGREGATION

  • The heuristic shrinkage rule contracts logit-averaged forecasts more toward the prior when trial forecasts disagree more.Its floor and contraction rate are selected by cross-validation; a floor of 1 disables shrinkage, while a floor of 0 permits full shrinkage toward the prior.Source: §C.9 · p. 26

C.10 CALIBRATION

  • Historical Platt-scaling results use leave-one-record-out calibration, so other horizons or dates from the same question can remain in training and estimates may be optimistic.This record-level protocol does not establish performance on unseen questions and differs from the question-held-out aggregation analysis.Source: §C.10 · p. 27

D FB: FURTHER RESULTS

BLF’s overall advantage is qualified by source-level variation and by the fact that question difficulty explains much more score variance than method effects. The further results also describe tool use and show that the selected shrinkage setting can collapse to ordinary logit averaging.

D FB: FURTHER RESULTS

  • At c=1, leave-one-out tuning selects f=1 and c=0, so shrinkage is a no-op and aggregation reduces to plain logit averaging.The regenerated common-pool trial-count analysis selects no shrinkage under question-held-out tuning on FB without anchors.Source: D FB: FURTHER RESULTS · p. 28

D.1 FB: COMPARISON TO SOTA

  • BLF has the highest point estimates for BI and ABI and the lowest BS, with market BI 83.8 versus Cassi 82.0 and crowd baseline 81.5.Paired comparisons are needed because method coverage differs.Source: §D.1 · p. 28, Table 12 · p. 28
  • +3.4∗∗ overall BI is the only significant BLF-versus-crowd+emp gain; the dataset gap is +4.5∗∗∗, while the market gap is +2.3ns.The market difference is in the right direction but within noise; none of the three external methods beats the baseline on overall BI.Source: §D.1 · pp. 28–29, Table 12 · p. 28

D.2 FB: PER-SOURCE ANALYSIS

  • Across sources, all methods are at chance on yfinance and struggle with FRED, while BLF’s tools help on DBnomics and all methods do well on ACLED and Wikipedia.The DBnomics KNN bypasses the LLM; the FRED statistical estimate supplements raw data, while extreme base rates make ACLED and Wikipedia questions easy.Source: §D.2 · p. 29, Figure 7 · p. 29
  • On market questions, BLF outperforms other methods by a large margin, which struggle to beat the prior.The paper studies the sources of these gains in subsequent analysis.Source: §D.2 · p. 30

D.3 TOOL USAGE

  • Web search dominates tool calls across sources, while source-specific tools are selective and DBnomics uses its KNN model without LLM tool calls.Polymarket averages about 5.5 calls per question, including market-info fetches in addition to web search.Source: §D.3 · p. 30, Figure 8 · p. 30

E.1 VARIANCE DECOMPOSITION

  • Question difficulty accounts for 62.2% of total variance, while method effects account for 2.7%, despite a significant omnibus effect (F = 164.5, p ≪0.001).The 35.1% residual includes inter-trial stochasticity and unmodeled method–question interactions, not solely LLM sampling variability.Source: §E.1 · p. 31, Table 14 · p. 31

E.2 PAIRED ANALYSIS

The paired-analysis section describes methods for comparing forecasts on matched events and notes remaining dependencies and uncertainty in inference. The accompanying benchmark context spans AIBQ2 and ForecastBench, with date-specific instances that can be dependent.

E.2 PAIRED ANALYSIS

  • The mixed-effects estimator treats question effects as random and supports inference on new questions, while fixed- and mixed-effects method estimates coincide under exact balance.With approximate balance, their difference is on the order of the imbalance fraction; the authors report this fraction is very small in their setting.Source: §E.2 · pp. 31–32
  • Paired BI contrasts compare methods on common resolution events, but pairing alone does not remove confounding from model, tools, or compute budgets.Event resampling does not capture dependence across horizons within a question; question–date resampling leaves cross-date dependence unmodeled.Source: §E.2 · p. 32
  • Bootstrap tests are exploratory and unadjusted for multiple comparisons, and fitted post-processing intervals are conditional on fits unless otherwise stated.They do not include a full nested refitting analysis.Source: §E.2 · p. 32

F.1 AIBQ2

  • AIBQ2 has 113 binary Metaculus tournament questions, a 22% base rate, and mostly short 1–8 week horizons, with many resolving at the tournament deadline.Questions span 10 manually clustered topics, led by Domestic Politics (35/113) and Geopolitics & Conflict (29/113).Source: §F.1 · p. 34, Figure 10 · p. 34, Figure 11 · p. 34

F.2 FORECASTBENCH

  • ForecastBench combines judgemental market questions with binary time-series questions evaluated against forecast-date values at multiple future horizons.The dataset questions include horizons from 7 through 3650 days, while market questions concern future events across sources.Source: §F.2 · p. 35
  • The 400 ForecastBench question-date instances represent 355 underlying questions, so date-specific instances may be dependent even when analyzed separately.Source: §F.2 · p. 35

F.2.1 TRANCHES

The appendix details how ForecastBench tranches were selected and describes substantial variation in source predictability. The reported source patterns range from near-random stock movements to seasonal temperature data, while several question sets have extreme base rates.

F.2.1 TRANCHES

  • Two ForecastBench tranches each contain 100 market and 100 dataset questions selected after model knowledge cutoffs, yielding 400 question-date instances and 791 binary resolution events.Questions were selected for available resolved values and overlap with leading-method submissions; comparisons are paired within tranche.Source: §F.2.1 · pp. 35–36, Figure 12 · p. 36

F.2.2 STATISTICAL PROPERTIES OF THE TRANCHES

  • Source-level BI varies from 90.7 on Manifold and 83.0 on Wikipedia to 49.6 on yfinance and 53.2 on FRED for the best system.Market horizons range from one week to five months; dataset horizons are 7, 30, and 90 days.Source: Table 16 · p. 36, §F.2.2 · p. 37, Figure 14 · p. 37

F.2.3 REPRESENTATIVENESS OF THE TRANCHES

  • The tranche window is stable but its overall BI is about 3 points above the wider-corpus mean, mainly because tranche market questions are easier by 6.5 BI.Dataset questions are 1.2 BI harder; the authors expect the resulting upward bias to affect other methods similarly.Source: §F.2.3 · p. 38, Figure 16 · p. 38

F.3 FB: YAHOO FINANCE (STOCK PRICES)

  • Yfinance stock prices approximate a random walk, making higher-versus-lower predictions nearly impossible and favoring p ≈ 0.5; BLF’s yfinance BI is approximately 0.5.The source questions compare future closing prices with the forecast-date price, which the system retrieves rather than using the stale freeze value.Source: §F.3 · pp. 38–39, Figure 17 · p. 39

F.4 FB: FRED (ECONOMIC INDICATORS)

  • FRED series mix trends and near-random walks: roughly 8 of 36 are strongly autocorrelated, 18 are effectively random walks, and the rest are intermediate.A series-pattern classifier and corresponding statistical models are left for future work; the tranche FRED base rate is 57%.Source: §F.4 · p. 40, Figure 18 · p. 40, Figure 19 · p. 41

F.5 FB: DBNOMICS (TEMPERATURE)

  • DBnomics temperature has strong seasonality, and BLF’s empirical exceedance model yields BI 60.8 versus Cassi 51.0 while bypassing the LLM.Pooling nearby-station data gives negligible BI gain; near-horizon weather forecasts are identified as the main bottleneck.Source: §F.5 · pp. 41–42, Figure 20 · p. 41, Figure 21 · p. 42

F.6 FB: ACLED (ARMED CONFLICT)

  • ACLED’s overall base rate is 12.5% (5 True out of 40), making an all-No forecast a strong baseline; none of the 10x-spike questions resolved True.The agent relies on web search because no source-specific ACLED data tool is available.Source: §F.6 · p. 43

F.7 FB: WIKIPEDIA

  • Wikipedia’s 40 questions have a 22.5% overall base rate but vary sharply by article, and BLF reaches BI 83.0 on questions often answerable from their structure.Vaccine questions all resolved False, world-record questions all True, and FIDE rank questions were more often True than Elo-gain questions.Source: §F.7 · pp. 43–44, Figure 23 · p. 44, Figure 24 · p. 44

G.1 KNN MODEL FOR DBNOMICS

This section describes source-specific time-series forecasting methods and their implementation prompts, alongside related work on forecasting benchmarks, agentic systems, and Bayesian approaches. It also explains how search-result snippets are distilled into evidence focused on resolution criteria, with a shorter fallback when those criteria are unavailable.

G.1 KNN MODEL FOR DBNOMICS

  • KNN achieves BI 59.1 on DBnomics, outperforming Cassi (51.0) and the tested standalone alternatives.Table 17 compares standalone models on A∪B, covering 40 questions and 119 resolution dates.Source: Table 17 · p. 45, §G.1 · p. 45
  • KNN selects historical observations within a day-of-year window and estimates exceedance frequency with Laplace smoothing, using w = 10 and no age restriction for DBnomics.The method assumes a stationary periodic distribution and typically uses about 270 neighbors per resolution date.Source: §G.1 · p. 45

G.2 LINEAR TREND MODELS FOR YFINANCE AND FRED

  • For non-seasonal yfinance and FRED series, the method shrinks a local linear extrapolation toward 0.5, with source-specific shrinkage and fitting windows.The shrinkage factors are 0.1 for yfinance and 0.5 for FRED; OLS uses the most recent 60 and 30 observations, respectively.Source: §G.2 · pp. 45–46

G.3 HYBRID KNN AND LINEAR TREND MODEL FOR YFINANCE AND FRED

  • The yfinance/FRED hybrid combines linear, prior-year KNN, and 0.5 forecasts; it uses a seven-day window over five years and falls back to linear alone without prior-year data.The KNN component captures annual periodicity; β is 0.5 when prior-year data is available and 1 otherwise.Source: §G.3 · p. 46

H EXTENDED RELATED WORK

  • The related-work review covers benchmark and evaluation limitations, tool-augmented forecasting, and Bayesian agents; it contrasts BLF’s sequential linguistic belief state with BayesAgent’s explicit graphical model.It also frames RL approaches as complementary: they improve base models, while this work improves the agent harness.Source: H EXTENDED RELATED WORK · pp. 47–48

I PROMPTS

  • The BLF prompt requires accumulating sourced evidence and updating a belief state, while NoBel omits those instructions and accumulates raw search results in message history.Question prompts supply resolution dates and the knowledge cutoff, with crowd and prior estimates included only in applicable configurations.Source: §I.1 · p. 49, §I.2 · p. 49, §I.3 · p. 50

I.4 ZERO-SHOT PROMPT

  • 65.3 BI with the Halawi zero-shot template compares with 70.6 BI using the paper’s standard prompt for ZS+crowd+emp on FB A∪B.The zero-shot setup uses the standard BLF system prompt but exposes only the submit tool.Source: §I.4 · p. 50

I.6 SEARCH-RESULT SUMMARIZATION PROMPT

  • A separate summarizer LLM compresses selected search snippets using a question-aware prompt that extracts resolution-relevant facts and reduces irrelevant evidence in the belief state.The prompt asks for concrete facts, data, dates, and named-source opinions; it distinguishes facts from speculation, omits irrelevant information, and forbids adding a forecast. Gemini-3-Flash is the default summarizer; without resolution criteria, the prompt instead asks for a clear, concise summary of facts relevant to the question.Source: §I.6 · pp. 51–52

I.7 LIKELIHOOD ESTIMATION PROMPTS; I.5 SEARCH-RESULT DATE-FILTER PROMPT

  • The likelihood prompt elicits 0–10 evidence-typicality scores separately for YES and NO, while the search-date filter drops results whose text describes post-cutoff events or updates.The date filter rejudges result content without relying on publication-date metadata; results not explicitly marked DROP are retained.Source: §I.5 · p. 51, §I.7 · p. 52

Related papers

Loading 2604.18576v5…