Source-linked AI summary
From News to Forecast: Integrating Event Analysis in LLM-Based Time Series Forecasting with Reflection
Xinlei Wang, Maike Feng, Jing Qiu, Jinjin Gu, Junhua Zhao
TL;DR
The paper addresses the challenge of connecting unstructured social-event information with numerical time series forecasting. It integrates news through LLM-based selection and evaluation agents, then fine-tunes an LLM forecasting model; experiments report improved accuracy across diverse domains. The approach is most applicable where human or market activity influences trends and is constrained by pretrained LLM token limits.
Problem
Time series models have limited ability to capture sudden disruptions and social or economic events that numerical data alone may not represent.
Method
The method combines news and time series data, uses LLM agents to filter and iteratively refine relevant events, and fine-tunes an LLM for forecasting.
Results
Experiments report superior prediction accuracy across finance, energy, traffic, and bitcoin domains, especially for energy-demand patterns.
Takeaways & Limitations
Integrating news with LLM-based forecasting enriches predictions with domain knowledge and supports more context-aware forecasting aligned with real-world dynamics.
Takeaways & Limitations
The approach is less suitable for meteorological or predominantly physical data, and pretrained LLM token limits can truncate large inputs and affect long-term predictions.
Abstract
from arXiv · showhide
This paper introduces a novel approach that leverages Large Language Models (LLMs) and Generative Agents to enhance time series forecasting by reasoning across both text and time series data. With language as a medium, our method adaptively integrates social events into forecasting models, aligning news content with time series fluctuations to provide richer insights. Specifically, we utilize LLM-based agents to iteratively filter out irrelevant news and employ human-like reasoning to evaluate predictions. This enables the model to analyze complex events, such as unexpected incidents and shifts in social behavior, and continuously refine the selection logic of news and the robustness of the agent's output. By integrating selected news events with time series data, we fine-tune a pre-trained LLM to predict sequences of digits in time series. The results demonstrate significant improvements in forecasting accuracy, suggesting a potential paradigm shift in time series forecasting through the effective utilization of unstructured news data.
1 Introduction
The paper addresses the difficulty of incorporating social events into time series forecasting by combining unstructured news with numerical inputs. LLM-based agents select and refine relevant news, while the resulting context is used to fine-tune forecasting models.
- Traditional forecasting methods struggle with sudden external disruptions and do not systematically connect complex social events with time series fluctuations.
- News supplies contextual information about unexpected incidents, policy changes, technological developments, and public sentiment shifts that numerical data may miss.
- The framework embeds news and supplementary information into time series data using textual prompts and fine-tunes large language models for forecasting.
- LLM agents dynamically filter news, pair selected items with time series data, and iteratively refine selection logic using forecasting results.
- Experiments report superior prediction accuracy across finance, energy, traffic, and bitcoin domains, with particularly strong usefulness for energy-demand patterns.
2 Related Work
Prior work spans traditional, deep-learning, text-enhanced, and LLM-based time series forecasting, while reasoning-agent research provides tools for iterative language-model decision making.
- Time series forecasting: Traditional forecasting assumes historical patterns persist, whereas deep learning captures nonlinearities and dependencies in larger, more complex datasets.
- Textual information: Text-enhanced forecasting has used social media, news, and public reports across finance, energy, entertainment, pandemics, and tourism.
- Textual information: Earlier text-based methods often relied on keyword frequencies or dummy variables, while later approaches extracted word-frequency and sentiment features.
- Language models for time series forecasting: LLMs such as GPT and LLaMa have been applied to time series processing, including temporal representation learning and language-model-based forecasting.
- Reasoning with language models: Chain-of-Thought and Tree-of-Thought approaches support stepwise reasoning, checking, memory, and multi-round dialogue in language-model agents.
3 Method
The method treats forecasting as conditional sequence generation enriched with contextual events, using reasoning and evaluation agents to select news, construct training data, and iteratively update the pipeline.
- Method motivation: The system must flexibly process unstructured news, connect it to time series data, and use societal reasoning to identify relevant information.
- Forecast formulation: Time series forecasting is formulated as conditional generation, allowing an LLM to predict digit-token sequences from prior numerical values.
- Event conditioning: News events provide context for spikes and drops by representing social causes that numerical time series alone cannot indicate.
- Event conditioning: News and supplementary information are incorporated as textual conditions that influence the conditional probability of future time series values.
- Data construction: Candidate news is initially aligned with time series by matching frequencies, forecast horizons, and geographic areas before detailed filtering.
- Reasoning agent: A reasoning agent screens and categorizes news by relevance, impact duration, direction, affected region, reporting time, and rationale.
- Evaluation agent: An evaluation agent compares predictions with ground truth and historical news to identify overlooked events and improve filtering logic.
- Overall pipeline: The overall pipeline alternates news selection, data pairing, LLM fine-tuning, validation, and reasoning updates across iterations.
4 Experiments
The experiments evaluate news-enhanced forecasting across human-influenced domains, prompt designs, iterative filtering, and baseline comparisons. Results indicate that carefully selected news improves forecasting, while irrelevant or overly abundant news can impair it.
- Experimental setup: The evaluation uses traffic, exchange, Bitcoin, and Australian electricity-demand series, supplemented with weather, calendar, and economic information.These datasets target forecasting settings influenced by human activities and social events.
- Prompt designs: Four prompt designs compare pure numerical tokens, descriptive sentences, unfiltered news, and agent-filtered news.The final design tests whether relevance-filtered news improves over numerical or textual alternatives.
- News integration: Proper news and supplementary information significantly improve performance across all four domains, whereas poorly selected news can severely impair results.Excess tokens can reduce LLM performance, and irrelevant news can add noise or incorrect causal information.
- Iterative analysis: Iterative evaluation generally improves news filtering, with two iterations often sufficient and multiple iterations outperforming a single iteration.The evaluation agent uses prediction outcomes and reflection to refine the selected news.
- Baseline comparison: The method significantly outperforms historical-only methods for electricity, exchange rates, and Bitcoin, but gains are modest for traffic because available news lacks local road-level detail.Traffic performance is also affected by MSE sensitivity to unreported traffic spikes, while MAE indicates reliable average accuracy.
- Case study: With-news electricity predictions are closer to actual demand than no-news predictions, especially at timestamps affected by abrupt events.The comparison is shown for lockdown, residential-consumption, and anticipated-outage news effects.
5 Conclusion and Discussion
The paper concludes that LLM forecasting and LLM agents can use news to support more adaptive forecasting aligned with real-world dynamics. The approach remains constrained by domain applicability and pretrained-LLM token limits.
- Conclusions: LLM-based forecasting and agents identify missed news, refine news-selection logic, and assess event impacts on predictions.The conclusion presents this process as supporting more nuanced and context-aware forecasting.
- Scope: News integration is most effective where human and market activities substantially influence trends.The approach is less suitable for meteorological or physical data with minimal human influence.
- Limitations: Pretrained LLM token limits complicate processing large time-series inputs or multiple sequences and can cause truncation affecting long-term predictions.The stated limitation concerns simultaneous input length rather than the general usefulness of news integration.
- Future work: Future work includes attributing news factors that affect accuracy and equipping reasoning agents with more advanced analytical and real-time tools.These directions aim to optimize news integration and extend the model’s analytical capabilities.
- Broader impact: News use requires checks for accuracy, balance, credible sourcing, and misinformation risks because it may influence public opinion.The paper identifies potential misuse of news, including the spread of fake news.
A.1 Experimental Setting / Details
The experiments cover multiple forecasting datasets and model families, using parameter-efficient LLM fine-tuning and a five-metric evaluation framework.
- Datasets and baselines: The benchmark includes electricity demand, exchange rates, traffic volume, and Bitcoin price forecasting tasks.These datasets span half-hourly, hourly, and daily frequencies across four domains.
- Datasets and baselines: Baselines include transformer-based models, the CNN-based TimesNet, and the MLP-based DLinear.The transformer baselines include Informer, Autoformer, FEDformer, Pyraformer, PatchTST, iTransformer, and FiLM.
- Fine-tuning: The method fine-tunes Llama 2 with LoRA, using rank 8 or 16, alpha 16, and a learning rate of 0.0001.Numeric tokens are formatted with three significant figures to limit token usage.
- Evaluation: Evaluation uses MSE, RMSE, MAE, MAPE, and accuracy rate to compare prediction performance.The metrics respectively address squared errors, error magnitude, absolute errors, percentage error, and the proportion of correct results.
- Evaluation: For the metric formulas, y_i denotes the unnormalized true load, ŷ_i the unnormalized prediction, and n the number of prediction timestamps.These definitions specify the quantities used in the evaluation calculations.
A.2 Example of Numerical Input for Fine-tuning LLM
The numerical fine-tuning format presents historical values and contextual inputs separately, then produces a numerical forecast sequence.
- Input-output structure: The instruction contains a truncated historical numerical sequence such as 7015.7, 6875.1, and 6634.6.This sequence represents the numerical history supplied for forecasting.
- Input-output structure: The input combines region, date, weekend and holiday status, weather variables, and historical numerical values.The example specifies NSW, dates, calendar status, temperatures, humidity, pressure, and a half-hourly load context.
- Input-output structure: The output is a predicted numerical sequence, including values such as 6592.6, 6467.0, and 6312.3.The forecast is formatted as a continuation-like sequence of numeric tokens.
A.3 Example of Textual Input for Fine-tuning LLM
The textual fine-tuning format converts numerical forecasting context into natural-language instructions containing history, task conditions, and the target forecast.
- Textual prompt structure: The instruction introduces the historical load data in a natural-language prompt.It embeds the numerical history within text rather than presenting only a bare number sequence.
- Textual prompt structure: The input asks the model to predict next-day NSW load using dates, weekend and holiday status, frequency, history length, and weather conditions.The example specifies 30-minute sampling, one day of historical data, and start and prediction dates.
A.4 News Sources and Details
The study constructs domain-specific news inputs from historical periods, analyzes selected events and keywords, and tests how supplementary information and language-model choice affect the setup.
- News collection: News is gathered for traffic, Bitcoin, and other forecasting periods by filtering GDELT with domain-specific keywords and parsing associated web pages.The process aligns news coverage with the time periods represented in the time-series datasets.
- Event characterization: The event distribution is illustrated through a news-category word cloud, daily event distribution, and random-versus-non-random event proportion chart.These figures provide complementary views of the selected event corpus.
- News characterization: The selected-news analysis records keywords, their frequencies, and the total number of selected articles for each domain.These statistics are summarized in the event-selection table.
- Event characterization: Random events include disasters, accidents, health crises, and crimes, while normal events include political, sports, cultural, economic, and holiday events.An LLM agent categorized events from January 1 to August 5, 2019, finding that random events averaged 27.7% of all events.
- Input ablations: Using filtered news with partial supplementary information performed better than using filtered news alone or only supplementary information.Removing all supplementary information and retaining only news produced slight improvements over supplementary information alone but did not match the partial-information setting.
- Language-model comparison: Mistral v0.1 produced results similar to Llama 2, whereas Gemma 2B performed slightly worse.The paper attributes the smaller model’s weaker results potentially to its limited number of parameters and notes that additional training adjustment may be needed.
A.5 More Results
The method performs well on sudden events and distribution changes, while fine-tuned LLMs remain comparable to baseline methods on a traditional long-term forecasting dataset.
- The method better predicts some sudden events and cases involving distribution changes.
- On the Ill dataset, fine-tuned LLMs achieve performance comparable to baseline methods for univariate forecasts with prediction lengths of 24, 36, 48, and 60.
- The experiments also compare different data-incorporation strategies and different pre-trained language models.
A.6 Full Prompt Design
The appendix details prompts that guide news selection by effect horizon and domain-specific factors, evaluate predictions for missed news, and refine forecasting logic from those analyses.
- A.6.1 Prompt Example of Reselecting News through the Reasoning Agent:: The reasoning agent classifies electricity-demand news by positive or negative effects and by long-term, short-term, or real-time impact.
- A.6.1 Prompt Example of Reselecting News through the Reasoning Agent:: Electricity prompts associate demand changes with weather, public events, population, industrial development, urbanization, energy transition, downturns, efficiency, demographics, and policy.
- A.6.1 Prompt Example of Reselecting News through the Reasoning Agent:: Selected electricity news is requested in JSON with the relevant region, time, effect horizon, and rationale, including examples involving renewable projects, extreme heat, cold snaps, outages, and network disruption.
- A.6.2 Prompt Example of Evaluating Predictions through the Evaluation Agent:: The evaluation agent receives task and background information, predictions, actual values, errors, selected news, and all news to identify overlooked events.
- A.6.2 Prompt Example of Evaluating Predictions through the Evaluation Agent:: An example identifies a Brisbane lockdown as missed news because it could affect domestic consumption, business conditions, investor sentiment, and AUD risk perception.
- A.6.2 Prompt Example of Evaluating Predictions through the Evaluation Agent:: The evaluation output can generate new prediction logic, including risk aversion and domestic-consumption effects associated with a lockdown.
- A.6.3 Prompt Example of Refining the Final Logic: The refinement prompts polish accumulated logic and rephrase the current selection logic into an adjusted version.
A.7 Missed News Examples
The missed-news examples show how agents connect prediction errors with indirect or delayed event effects and use those findings to refine domain-specific news reasoning.
- The agent analyzes prediction errors and news to identify overlooked events and adjust expectations about how events affect time series.
- Geopolitical tension involving Australia and China was judged potentially relevant to the AUD through trade and investor sentiment, but its immediate effect appeared minimal or offset.
- A Saudi net-zero announcement was considered potentially relevant to the AUD through global oil prices and economic stability, although zero deviation suggested accurate or delayed effects.
- Other examples connect US political announcements, Australian housing activity, and vaccine concerns with currency sentiment or economic recovery expectations.
- The agent filters news according to each forecasting task and provides domain-specific selected-news examples in Table 8.
- The reasoning logic covers domain factors including Australian electricity demand, AUD/USD movements, Bitcoin price changes, and California traffic volume.
- Traffic reasoning incorporates weather, infrastructure changes, events, road closures, travel speeds, alternative routes, and traffic-flow variations.