Source-linked AI summary
LLM-based Agents for Forecasting and Prediction: Methods, Training, Evaluation, and Applications
Xiaogang Xu, Jiaqi Tang, Jianmin Chen, Yingying Yan, Zhenchao Tang, Xiangxin Zhou, Xiaobin Hu, Wei Wei, Jinfeng Wu, Qifeng Chen, Lu Zhou, Jiafei Wu, Zhe Liu, Jianwei Yin, Weimin Zheng
TL;DR
Forecasting research increasingly combines LLMs with temporal data, retrieved evidence, tools, and iterative prediction, but the central question is whether language-model participation adds reproducible value. This review organizes these systems, evaluates their training and measurement practices, and concludes that gains remain mixed and that credible progress depends on stronger leakage-resistant, cost-aware evaluation.
Problem
Current evidence does not establish that LLMs extrapolate numerical sequences more accurately than specialized forecasting models, making the value of language-model participation an open question.
Method
The paper reviews LLM-based forecasting agents across standalone, tool- and retrieval-augmented, and hybrid architectures, covering training, evaluation, applications, and deployment risks.
Results
Evidence remains mixed: language can provide useful predictive information and complement structured models, but LLMs are not established as general replacements for statistical, numerical, or time-series foundation models.
Takeaways & Limitations
Credible systems should use controlled ablations, strong non-LLM baselines, live or time-stamped evidence, calibration-aware evaluation, auditable provenance, and joint reporting of predictive scores and costs.
Takeaways & Limitations
Representation studies may optimize an interface to a non-contributing module, and isolating encoding quality from LLM presence on text-rich tasks remains unreported.
Abstract
from arXiv · showhide
Large language models (LLMs) now support forecasting systems that combine language-based reasoning with temporal data, evidence retrieval, external tools, and iterative prediction. We investigate LLM-based forecasting agents, meaning systems in which a language model contributes to a scored prediction about a future or currently unobserved target. We organize architectures into three groups. Standalone LLM workflows operate on encoded time series or event context. Tool- and retrieval-augmented agents incorporate external evidence. Hybrid systems pair LLMs with statistical or foundation models. We then review training methods and evaluation protocols. We examine negative as well as positive evidence, including sensitivity to small input perturbations, ablations in which the LLM component does not improve accuracy, and benchmark gains that may reflect contamination instead of temporal reasoning. We cover applications in finance, weather, health, energy, and operations, and we summarize the benchmarks and datasets used for evaluation. The evidence indicates that measurement is a central limitation. Future work requires calibration under distribution shift, contamination-resistant live evaluation, explicit reporting of cost and accuracy together, and methods for handling feedback between deployed forecasts and the outcomes being forecast.
1 Introduction
The review examines LLM-based forecasting agents as systems that combine language models with temporal data, evidence, tools, or iterative control, while asking when their added complexity produces credible gains. It organizes architectures, evaluation challenges, applications, and open measurement problems across forecasting domains.
- Scope: LLM-based forecasting agents contribute reasoning, representation, or control to scored predictions about future or currently unobserved targets.The review includes prompt-only workflows, retrieval and tool use, and hybrid systems paired with statistical or time-series foundation models.
- Architectures: Standalone workflows operate on serialized series or event context, whereas other agents retrieve evidence, invoke tools, iterate through reason-and-act loops, or coordinate specialized models.These designs span prompt-only, retrieval-augmented, multi-agent, and hybrid architectures.
- Evidence and tools: Language evidence can add signals absent from historical numerical observations, but retrieval and tool use introduce source-validity, selection, stopping, and inference-cost decisions.Policy announcements, filings, news, and outbreak reports may precede their effects in numerical series.
- Applications: The review covers finance, weather, health, energy, and operations, where specialist baselines, financial returns, latency, integration, and differing failure costs shape evaluation.Point error alone cannot compare poorly calibrated probabilities, delayed warnings, and unprofitable trading actions.
- Contribution: Existing reviews emphasize architectures or general agents, while this study integrates tool-mediated control, live evidence, outcome-based evaluation, and forecasting-specific measurement.Its contributions include a unified agency continuum, taxonomy, evaluation analysis, domain constraints, and a research agenda.
- Evaluation: The paper treats contamination, leakage, calibration, cost, aggregation, and human comparisons as central controls for judging whether reported forecasting gains are credible.It explicitly includes positive and negative evidence and examines both deployment risks and open problems.
2 Scope and Formulation
The paper defines forecasting agents by their contribution to explicitly scored predictions made from information available before the target is observed. It unifies numerical and event forecasting, formalizes agentic interaction and graded agency, and highlights disclosure and leakage limits in the reviewed evidence.
- 2.1 Scope: The scope includes LLM systems that contribute reasoning, representation, or control to verifiable scored predictions, while excluding nonscorable commentary and purely numerical models.Eligible outputs include points, distributions, categories, and event probabilities.
- 2.1 Scope: The review covers numerical targets and discrete events under a common outcome-based forecasting boundary.Numerical forecasts are commonly compared with statistical or foundation models, while event forecasts use human or market references.
- 2.2 Corpus Construction: The corpus was built from scholarly and preprint searches using model, task, and mechanism terms, then expanded through backward citation search and full-text screening.The corpus is maintained as a living snapshot of the literature.
- 2.2 Corpus Construction: The corpus artifact reports venue, publication-year, and application-domain distributions, but these quantities can change as the living paper is updated.Contested claims relying solely on preprints are explicitly identified.
- 2.2 Corpus Construction: A pilot audit of 12 benchmark papers found disclosure gaps, including omitted agent scaffolds and absent inference-cost reporting among eight agent papers, so the review avoids cross-paper leaderboards.Retrospective benchmarks also risk pre-training contamination and lookahead feature leakage.
- 2.3 A Unified Formulation: A forecasting task predicts an unobserved target from information strictly available before decision time, with numerical and event formulations receiving forecast-form-appropriate scoring rules.Event probabilities commonly use Brier or logarithmic scores, while continuous distributions can use CRPS.
- 2.4 What Makes a Forecaster an Agent: The agent framework represents systems with a policy, action space, tools, memory, and predictive readout, updating state through actions, observations, and memory before emitting a forecast.Actions may include retrieval, tool calls, memory writes, delegation, and termination.
- 2.4 What Makes a Forecaster an Agent: Agency is graded from prompt-only emission to sequential controller systems, with retrieval and tool use adding selected evidence or task-specific invocations at inference time.Observable traces support process-fidelity evaluation, but coherent traces do not guarantee faithful reasoning.
3 Agent Architectures
The review organizes forecasting agents by how the LLM represents temporal information and interfaces with tools, retrieval, other agents, or numerical forecasting models. It also highlights unresolved robustness, evaluation, cost, and attribution problems across these architectures.
- Architecture taxonomy: Forecasting architectures fall into standalone LLMs, tool-augmented systems, and hybrids pairing LLMs with statistical, deep-learning, or time-series foundation backbones.These categories describe dominant coupling patterns rather than mutually exclusive classes.
- Serialization, tokenization, and prompt content: Serialization designs trade numerical granularity and contextual information against token efficiency and attention cost.Digit-level tokenization can lengthen sequences, while patching reduces attention cost and expands the effective lookback window; encoding granularity can limit distributional resolution.
- Control flow and tool use: Control flow ranges from fixed refinement and adaptive routing to retrieval, open-web research, memory, executable tools, and multi-agent orchestration.These systems differ in model-call budgets, evidence access, routing modes, and whether intermediate state or specialist tools are used.
- Evaluation limitations: Robustness and attribution remain weak: small perturbations have inconsistent effects, and removing the LLM can leave numerical-benchmark accuracy unchanged or improved.These findings complicate claims that gains arise from language reasoning rather than encoding choices, text content, or non-contributing modules.
- Evidence and memory: Retrieved evidence and forecasting-specific memory can improve selected results, but live deployment, query-level guarantees, distribution-shift tracking, and security remain unresolved.Retrospective evidence and untested memory benefits limit conclusions about deployed streaming systems and long-term adaptation.
- Hybrid pairing with numerical models: Hybrid evaluations are confounded because comparisons often omit numerical foundation-model baselines and do not isolate the contributions of text, parameter scale, regimes, or language pre-training.The review therefore treats reported LLM gains as insufficient to identify their source or establish temporal generalization.
4 Training and Adaptation
Training and adaptation span lightweight interfaces, inference-time computation, supervised and reinforcement learning, and reward design. Evidence remains mixed on whether LLM components add forecasting accuracy, while fair ablations and robustness under regime shifts remain unresolved.
- Adaptation: Lightweight interfaces adapt numerical inputs to frozen or partially updated LLMs, creating different transfer and compute trade-offs.GPT4TS updates selected normalization and positional-embedding parameters, whereas Time-LLM trains a reprogramming layer around frozen LLM weights.
- Inference-Time Adaptation: Inference-time reasoning has mixed effects: verified traces can help, but excessive deliberation may reduce accuracy and worsen calibration.Input perturbation and contextual gating improve selected forecasts, whereas repeated sampling offers little numerical diversity.
- Reinforcement Learning: Outcome rewards provide authoritative labels for event forecasts but are sparse and reveal little about which search or reasoning actions deserve credit.Market trajectories offer denser targets but may reflect liquidity, herding, or participant error.
- Reward Design: Proper-score rewards address calibration because correctness alone does not penalize overconfidence, and adding Brier score can improve both accuracy and calibration.Outcome-conditioned gradients may instead encourage rationales that explain realized labels rather than decision-time beliefs.
- Ablation Evidence: Ablation evidence is mixed: removing the LLM sometimes leaves accuracy unchanged or improves it, while larger-scale reevaluation reports positive language-model contributions.Decisive comparisons require fixed datasets, horizons, splits, preprocessing, tuning budgets, random seeds, and paired uncertainty intervals.
5 Evaluation
Reliable evaluation requires point-in-time, contamination-aware benchmarks that report suitable scores, uncertainty quality, decision relevance, and inference costs. The review also finds unresolved calibration, leakage, aggregation, and deployment-realism challenges across numerical, event, and market settings.
- Evaluation Principles: Reliable evaluation requires predictions to use only pre-decision information while reporting forecast scores, uncertainty quality, and inference costs.Numerical and event targets require different baselines, and benchmark construction must audit pre-training contamination.
- Contamination Controls: Rolling evaluation and post-training data reduce different contamination risks, but static archives cannot guarantee absence from pre-training data.FEV-Bench uses rolling evaluation across 100 real tasks, while TIME reports 50 recently collected datasets and 98 operationally aligned tasks.
- Live and Market Evaluation: Benchmarks differ in realism: live-capital settings expose decisions to risk, simulated replay uses historical order books, and proposed designs may lack prospective results.Pooling market questions across lifecycle stages can also confound sampling strategy with forecasting ability.
- Contamination and Leakage: Feature leakage, dataset reuse, temporal overlap, and parametric leakage can produce optimistic evaluations that do not generalize.Existing loss detectors cannot establish clean zero-shot guarantees for time-series models, making fresh post-cutoff and prospective tasks stronger controls.
- Benchmark Gaps: Strong temporal controls do not guarantee deployment realism, and market settlement does not provide a numerical baseline, leaving a decision-critical numerical region empty.A benchmark would need point-in-time numerical inputs, realistic downstream decisions, and a specialist comparator under the same protocol.
- Scoring and Uncertainty: Reasoning-enhanced variants can become overconfident, with probabilities exceeding observed frequencies and higher expected calibration error after extended deliberation.Linear probing can yield better-calibrated probabilities than verbalized forecasts, while post-cutoff updating remains conservative and inconsistent.
- Aggregation: Aggregation can approach or exceed human-crowd performance when forecast errors are diverse, and learned aggregators can exploit disagreement signals.Cross-model deliberation helps heterogeneous groups but not homogeneous same-model groups, while unique evidence subsets can reduce herding.
- Human–Agent Hybrids: Human–AI evaluation should include both general crowds and expert superforecasters, because interaction design and collaborative traits are not universal explanations of performance.AI assistance improves human forecast accuracy in a preregistered study, but the mechanism was not isolated.
6 Applications
Applications use LLMs mainly for unstructured context, workflow control, and explanation, while domain-specific models remain important numerical and operational baselines. Evaluation must separate predictive skill from downstream decisions and account for domain-specific constraints.
- Cross-domain constraints: Application objectives differ: finance separates forecast quality from portfolio returns, weather compares against specialist numerical models, and energy and operations impose real-time integration constraints.These constraints are summarized in Table 5 and reflect different decision settings across domains.
- Finance and prediction markets: Financial evaluations should report forecast quality separately from position-dependent returns because identical probabilities can yield different returns under sizing, turnover, and execution choices.Directional predictions do not directly determine trading positions; policy-level evaluation captures continuous position management and multi-timescale risk.
- Finance and prediction markets: Hybrid financial models assign LLMs narrower roles such as extracting sentiment, confidence, or asset-dependence features, but matched-cost accuracy comparisons remain unsettled.One regime-conditioned volatility study reports outperforming classical baselines without isolating which component produced the gain.
- Weather, energy, and other domains: Weather agents support downstream response workflows rather than matching specialist models in numerical prediction, so workflow gains should not be reported as improved meteorological skill.The relevant comparison holds the numerical forecast fixed and assesses warning interpretation, tool use, jurisdiction selection, or response timing.
- Weather, energy, and other domains: Energy and demand applications combine time-aligned news, calibration, event databases, and business context with numerical baselines, but some proposed routing benefits remain unproven.Examples include renewable-energy forecasting, e-commerce demand, and air-ticket forecasting in retrospective case studies.
7 Risks and Open Problems
LLM-based forecasting faces risks from miscalibration, leakage, weak reporting, broad tool access, and feedback between forecasts and actions. Open problems therefore center on evaluation protocols, calibration, training objectives, and comparable cross-domain measurement.
- Forecasting risks: Fluent explanations can mask miscalibration because verbal confidence is not reliably aligned with accuracy, making proper scores and coverage tests necessary.Rationale length is not evidence of forecast quality or calibration.
- Evaluation pathologies: Benchmark results can become stale, decision-time leakage can inflate apparent performance, and incomplete scaffold or cost disclosure can obscure evaluation validity.Disclosure quality alone does not establish that a forecast is valid.
- Tool access and reflexivity: Broad tool permissions increase possible downstream actions, while open-web retrieval can expose live forecasters to fabricated or low-quality evidence.Vetted corpora reduce retrieval risk but may reduce timeliness.
- Measurement priorities: Unresolved questions, simulated environments, replay evaluation, consistency checks, and point-in-time covariates address different leakage and instability pathways rather than providing one universal safeguard.Point-in-time covariates do not by themselves prevent retrieval, benchmark-construction, or revised-label leakage.
- Training and adaptation: Training comparisons should include post-hoc calibration and state-conditioned, gradient-masked rewards because naive single-outcome rewards can corrupt reasoning and reward luck.Under regime shifts, similarity retrieval degrades and regime-conditioned methods add state-dependent structure.
- Shared protocols and collaboration: Shared protocols should require proper scores while preserving target-specific horizons, distributional metrics, foundation-model baselines, and live evaluation requirements.Event and numerical forecasting protocols emphasize different evaluation dimensions despite both estimating predictive distributions.
8 Conclusion
The review finds mixed evidence for LLM-based forecasting agents and places their strongest supported role in language evidence, event semantics, tool interaction, and auditable explanation. It concludes that progress depends on contamination-resistant, cost-aware evaluation and component-level evidence of decision value.
- Conclusion: Evidence remains mixed, and LLMs are not established as general replacements for statistical, numerical, or time-series foundation models.Controlled ablations are needed because gains may come from retrieval, prompts, interfaces, aggregation, or numerical backbones rather than the LLM.
- Conclusion: Current conclusions are limited by decision-time leakage, pre-training contamination, benchmark staleness, incomplete calibration and cost reporting, and retrospective evaluation.ForecastBench finds experts ahead of evaluated models, while PolyBench reports simulated execution losses rather than live-capital evidence.
- Future evaluation: Studies should use live or time-stamped evidence, proper scores, calibration-aware objectives, component-level ablations, strong non-LLM baselines, and explicit reporting of model calls, tokens, latency, and development costs.Retrieved evidence and tool actions should have auditable provenance, while deployment monitoring should track distribution shift, correlated errors, and forecast-induced feedback.