PaperlayerView exact revision on arXiv

Source-linked AI summary

From News to Forecast: Integrating Event Analysis in LLM-Based Time Series Forecasting with Reflection

Xinlei Wang, Maike Feng, Jing Qiu, Jinjin Gu, Junhua Zhao

arXiv:2409.17515v3cs.AI

TL;DR

Forecasting methods can struggle with external disruptions and do not systematically connect social events to time-series fluctuations. This paper uses LLM agents to select relevant news and integrate it with time-series data, reporting superior prediction accuracy across finance, energy, traffic, and bitcoin domains.

  • Problem

    Traditional forecasting methods have limitations with external disruptions and do not systematically connect complex social events to time-series fluctuations.

    Source: §1 · p. 1

  • Method

    The approach uses reasoning LLM agents to iteratively select relevant news, pair it with time-series data, and fine-tune an LLM forecaster.

    Source: §1 · pp. 2–3

  • Results

    Experiments report superior prediction accuracy across finance, energy, traffic, and bitcoin domains, with news integration particularly adept at navigating energy-demand complexities.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The findings support incorporating news and domain knowledge for more context-aware forecasting aligned with real-world dynamics.

    Source: §5 · p. 9

    The framework is less suitable for meteorological or physical data where human activities have minimal impact on trends.

    Source: §5 · p. 10

Abstract

from arXiv · show

This paper introduces a novel approach that leverages Large Language Models (LLMs) and Generative Agents to enhance time series forecasting by reasoning across both text and time series data. With language as a medium, our method adaptively integrates social events into forecasting models, aligning news content with time series fluctuations to provide richer insights. Specifically, we utilize LLM-based agents to iteratively filter out irrelevant news and employ human-like reasoning to evaluate predictions. This enables the model to analyze complex events, such as unexpected incidents and shifts in social behavior, and continuously refine the selection logic of news and the robustness of the agent's output. By integrating selected news events with time series data, we fine-tune a pre-trained LLM to predict sequences of digits in time series. The results demonstrate significant improvements in forecasting accuracy, suggesting a potential paradigm shift in time series forecasting through the effective utilization of unstructured news data.

1 Introduction

Traditional forecasting can miss sudden disruptions and does not systematically link social events to time-series fluctuations. The paper proposes integrating news context and numerical data through LLM-based forecasting and agents that select and evaluate relevant news.

Introduction

  • The method uses textual prompts to combine news and supplementary information with time series, fine-tuning LLMs to forecast by predicting the next token.The resulting forecasts take the textual context in the input prompts into account.Source: §1 · pp. 1–2
  • LLM agents dynamically filter news using human-like reasoning, then compare forecasts with relevant events to refine selection and recover overlooked information.The agents use few-shot learning and chain-of-thought prompting to link news events to forecasting discrepancies.Source: §1 · pp. 2–3
  • The framework combines unstructured news with numerical time-series inputs to provide context and improve responsiveness to social events and real-world dynamics.News can capture real-time events and qualitative, non-numeric influences that numerical data alone may miss.Source: §1 · pp. 1–2
  • The paper builds a multi-domain dataset pairing task-specific time series with verified public news reports and reports superior prediction accuracy across finance, energy, traffic, and bitcoin domains.The introduction highlights energy demand patterns as a domain where news helps navigate complexity.Source: §1 · p. 3

2 Related Work

Prior forecasting work has advanced from statistical models to deep learning and LLM-based methods, but often omits external context or uses LLMs without textual inputs and language reasoning. Earlier text-integration methods have limited contextual understanding, while agent research offers reasoning and reflection techniques relevant to more capable systems.

Time series forecasting

  • Deep-learning and pretrained forecasting models handle larger, more complex time series than traditional statistical methods, but often neglect external and contextual factors.Traditional approaches assume past patterns persist and were limited to small-scale datasets; deep learning captures nonlinearities and dependencies from historical data, while pretraining enables fine-tuning with fewer data and resources [33] [40] [59] [6] [24] [58] [63].Source: §2 · p. 3
  • Text-enhanced forecasting has used keyword counts, dummy variables, and engineered NLP features, but these approaches miss nuanced meaning and long-text dependencies; LLM use remains underexploited.Applications span finance, energy [3] [41], entertainment, pandemics, and tourism [47]; traditional NLP and machine learning feature extraction, including Bai et al. [3], require labor-intensive engineering and lack deep contextual understanding.Source: §2 · p. 3

Language models for time series forecasting

  • Existing LLM forecasting methods mainly map numerical time-series inputs for regression, without external textual inputs or LLM language reasoning.Examples include TEMPO [6], TIME-LLM [23], FPT, Lag-LLaMa [46], and next-token forecasting by Gruver et al. [17].Source: §2 · p. 3

Reasoning with language models

  • Language-model reasoning work includes Chain of Thought [57], Tree of Thoughts [61], feedback-reflecting agents with memory [50], tool-making [5], and agent debate [11].These approaches use intermediate steps, trial-and-error, feedback, or interleaved reasoning and actions to support task completion [28] [32] [62].Source: §2 · p. 3

3 Method

The method combines an LLM-based forecaster with agents that filter news and evaluate predictions, refining news selection through feedback from forecast errors.

3.1 Rethinking Time Series Forecasting Problem and Elements.

  • The forecaster treats time series forecasting as conditional generation of digit tokens, with news and supplementary information supplied as textual conditions on predictions.Event context can explain spikes or drops that historical series alone cannot causally explain; the paper also describes weather conditions as textual input. Pre-trained LLMs have shown few-shot forecasting capability [17].Source: §3.1 · pp. 4–5
  • Supervised instruction tuning trains the LLM to forecast from historical series paired with news and supplementary information, using LoRA to reduce computational demands.Training examples are formatted as text input-output pairs and use the pre-training loss function; LoRA updates only a small subset of parameters.Source: §3.1 · p. 5

3.2 Analytical Agent for Aggregation and Reasoning of Contextual News Information.

  • The reasoning agent filters news using impact logic that can be generated automatically or supplied by users, after news is roughly paired by time, horizon, and geography.Few-shot prompting and CoT [57] support its multi-step process: identify influencers and their impact and duration, categorize relevant news with rationale, and return selected items in JSON.Source: §3.2 · pp. 5–6
  • After model training, the evaluation agent examines prediction errors against ground truth and selected or historical news to identify overlooked events and update future filtering logic.It analyzes error distributions over time, then consolidates updates across validation predictions into a revised strategy.Source: §3.2 · pp. 6–7

3.3 Overall Pipeline

  • The pipeline iteratively pairs filtered news with time series, fine-tunes and validates the forecaster, then uses prediction-error feedback to refine the final news filter.The evaluation agent checks for potentially influential missing news at each iteration; the reasoning agent consolidates updates into the definitive filter for final-model training. The agents use GPT-4 Turbo.Source: §3.3 · p. 7

4 Experiments

The experiments test news-enhanced forecasting across human-influenced time series and show that relevant news and iterative reflection can improve predictions. Results also reveal limits when news is irrelevant or too coarse for the target data.

4.1 Data preparation

  • Experiments pair news from GDELT and real-time sources with traffic, exchange, Bitcoin, and Australian electricity series spanning daily to half-hourly frequencies.Weather, calendar dates, and economic indicators provide supplementary information, including temperature, pressure, wind, humidity, seasonal effects, GDP, inflation, and employment data.Source: §4.1 · pp. 7–8

4.2 Results

  • Properly selected news and supplementary information improve forecasting across all four domains, but excessive or irrelevant news can hurt through token overload and noisy causal signals.In the electricity example, predictions with news are closer to actual values, especially at timestamps when abrupt events affect demand.Source: §4.2 · pp. 8–9
  • In most cases, two evaluation-agent iterations suffice for significant improvement, and multiple iterations consistently outperform a single iteration through reflection.The iterative process refines news filtering based on prediction outcomes.Source: §4.2 · p. 8, Table 2 · p. 9
  • The method outperforms historical-series-only methods for electricity demand, exchange rates, and Bitcoin, while traffic gains are modest.Traffic results are limited by region- or globally sourced news that lacks local road detail; MSE is sensitive to traffic spikes, while MAE indicates reliable average accuracy. The model operates without normalization to retain the physical meaning of outputs.Source: §4.2 · pp. 8–9, Table 3 · p. 9

5 Conclusion and Discussion

The authors argue that integrating news through LLM-based forecasting and agents can enrich time series forecasting with context-aware reasoning. Agents identify missed news, refine their logic, and assess event impacts on predictions.

Conclusion

  • The approach integrates news with LLM-based forecasting and agents that identify missed news, refine their logic, and assess event impacts on predictions.The authors advocate incorporating extensive domain knowledge for adaptive forecasting aligned with real-world dynamics.Source: §5 · p. 9

Limitations

  • The approach is less suitable for meteorological or physical data where human activity has little influence, and pretrained LLM token limits can truncate inputs and affect long-term prediction accuracy.The authors position the framework as enhancing, not replacing, traditional tasks such as classification and interpolation.Source: §5 · pp. 9–10

Future work

  • Future work will attribute news factors’ effects on accuracy and equip reasoning agents with advanced analytical tools for sophisticated processing and real-time techniques.These enhancements are intended to improve prediction precision and relevance while expanding applicability.Source: §5 · p. 10

Broader impact

  • Responsible use requires checking news for accuracy, balance, credibility, and misinformation risks, while potential applications include forecasting GDP, carbon emissions, and public health outcomes.The authors emphasize that these uses could affect policymaking and public welfare.Source: §5 · p. 10

A Appendix / Supplemental Material · A.1 Experimental Setting / Details

The appendix details the forecasting datasets, baseline and fine-tuning setups, and evaluation metrics used in the experiments.

A.1 Experimental Setting / Details

  • Table 4 describes half-hourly state-level electricity demand, daily Australian-dollar exchange rates, hourly California traffic volume, and daily Bitcoin prices.These are the forecasting-task datasets described in the table.Source: Table 4 · p. 14
  • Baselines include Informer, Autoformer [59], FEDformer, Pyraformer, PatchTST [40], iTransformer [38], FiLM, Timesnet [58], Dlinear, and GPT4TS[70].The set spans transformer-based models, a CNN-based model, an MLP-based model, and an LLM-based model.Source: §A.1 · p. 14
  • Comparative baselines receive the same information as the proposed method except news, use normalized variables and dummy-coded non-numeric variables, and are tuned across four learning rates with three initializations.The best-performing configuration is selected; official architecture settings are used.Source: §A.1 · p. 14
  • The proposed method fine-tunes Llama 2 [55] with LoRA [19] at rank 8 or 16, alpha 16, learning rate 0.0001, and three-significant-figure numeric tokens.Training uses one NVIDIA A100 40G GPU for several hundred to 1,000 iterations on curated data, taking several hours.Source: §A.1 · p. 14
  • Evaluation uses MSE, RMSE, MAE, MAPE, and accuracy rate to assess prediction performance.The appendix describes MSE as emphasizing larger errors, RMSE as expressing error magnitude on the original data scale, MAE as balancing errors of different magnitudes, and MAPE as percentage error relative to actual values.Source: §A.1 · p. 14
  • The evaluation definitions use unnormalized true load values yi, unnormalized predictions ŷi, and n prediction timestamps.Accuracy rate is defined as the proportion of true results among all examined cases.Source: §A.1 · p. 14

A.2 Example of Numerical Input for Fine-tuning LLM

The example presents a fine-tuning instance with an instruction sequence, a dated NSW input containing calendar labels and numeric values, and a numeric output sequence.

A.2 Example of Numerical Input for Fine-tuning LLM

  • The input lists NSW dates, weekend and public-holiday labels, and numeric values including 286.5, 297.96, 34.0, and 1012.0.Source: §A.2 · p. 15

A.3 Example of Textual Input for Fine-tuning LLM

The example formats load forecasting as a prompt with historical numeric data, prediction context including weather and relevant news, and a numeric sequence as output.

A.3 Example of Textual Input for Fine-tuning LLM

  • The instruction supplies historical load values as the time-series input for fine-tuning.The example shows a sequence of load measurements before and after an ellipsis.Source: §A.3 · p. 15
  • The prompt specifies the NSW forecast date and interval, historical coverage, weekend and holiday status, weather conditions, and two bushfire news items linked to load fluctuations.The news items describe possible short-term demand effects from firefighting, emergency services, evacuations, and communications.Source: §A.3 · p. 15
  • The illustrated output is a sequence of values, shown as comma-separated numbers with ellipses before and after the listed values.Source: §A.3 · p. 15

A.4 News Sources and Details

The study assembled domain-specific news for traffic and Bitcoin forecasting, then used LLM agents to classify and select relevant events. Its analysis also reports the event mix and the value of retaining supplementary information with filtered news.

A.4 News Sources and Details

  • The corpus retained 5,867 traffic articles and 5,906 Bitcoin articles after invalid and redirected links were removed from GDELT results.The source periods were 2015–2016 for traffic and 2019–2021 for Bitcoin; traffic data came from Yahoo, and Bitcoin data from various websites.Source: §A.4 · p. 15
  • Across Jan 1–Aug 5, 2019, agents labeled 27.7% of events random (unplanned crises, accidents, disasters, or crimes), versus normal planned or pattern-based events.Figures 7 and 8 show the daily distribution and proportions of random and non-random events.Source: §A.4 · p. 16, Figure 7 · p. 16, Figure 8 · p. 16
  • Selected news mainly covered economic or political events, health crises, natural disasters, technology development, and social sentiment; reflection helped surface unexpected or counterintuitive missed news.The reflection agent analyzed training prediction errors and missed news.Source: §A.4 · p. 16
  • Case 6, combining filtered news with all supplementary information, yielded the best results; news alone did not match Case 4, and supplementary information remained essential.Case 3 used only filtered news, while Case 4 used filtered news and partial supplementary information.Source: Table 6 · p. 16

A.5 More Results

The method performs comparably to traditional baselines on the Ill forecasting dataset and shows stronger predictions in some sudden-event and distribution-change cases. Additional experiments compare data-incorporation approaches and pretrained language models.

A.5 More Results

  • On the Ill dataset, fine-tuned LLMs achieve performance comparable to baseline methods for univariate forecasts with prediction lengths of 24, 36, 48, and 60.The comparison is reported in Table 9; the passage does not provide numerical scores.Source: §A.5 · p. 17
  • The showcased forecasting tasks indicate better predictions in some sudden events and cases with distribution changes.The showcases appear in Figures 13–16.Source: §A.5 · p. 17
  • Additional experiments compare different data-incorporation types and different pretrained language models.These comparisons are reported in Tables 6 and 7, respectively.Source: §A.5 · p. 17

A.6 Full Prompt Design … A.9 Reasoning Logic Examples

The appendix illustrates how agents select news by its anticipated relationship to forecasting targets, evaluate errors for overlooked events, and revise prediction logic; it also presents selected-news and reasoning examples across domains.

A.6.1 Prompt Example of Reselecting News through the Reasoning Agent:

  • The electricity prompt organizes potential load increases into short- and long-term drivers, including weather, social events, population growth, industrial development, urbanization, and energy transition.It contrasts these with possible decreases such as downturns, efficiency improvements, and mild weather.Source: §A.6.1 · p. 17
  • The news-selection prompt asks the agent to classify relevant items as long-term, short-term, or real-time effects, include region and time, and return valid JSON or no news.The accompanying logic also lists load-decreasing factors such as economic downturns, efficiency improvements, and mild weather.Source: §A.6.1 · pp. 17–18

A.6.2 Prompt Example of Evaluating Predictions through the Evaluation Agent:

  • The evaluation agent compares predictions with actual values and news to identify overlooked events; an example flags Brisbane’s COVID-19 lockdown as potentially relevant to the AUD/USD forecast.The example’s proposed updated logic links the lockdown to risk aversion, consumption, market sentiment, commodity prices, and perceptions of economic management.Source: §A.6.2 · pp. 19–20

A.6.3 Prompt Example of Refining the Final Logic

  • The refinement prompts ask the agent to condense updated news-selection logic and rephrase the current prediction logic accordingly.Source: §A.6.3 · p. 20

A.7 Missed News Examples

  • Missed-news examples span geopolitical tensions with China, an oil producer’s net-zero target, and a US presidential candidacy, with possible trade, commodity-market, and investor-sentiment links to the AUD or forex.The examples also note minimal or zero prediction deviations in the first two cases, suggesting immediate effects may have been small, offset, or delayed.Source: §A.7 · p. 21
  • Other missed-news examples propose that strong Australian housing activity could support AUD sentiment, while vaccine concerns could temper expectations for recovery and reopening.Source: §A.7 · p. 21

A.8 Selected News by Agents

  • Table 8 presents examples of agent-selected news for different forecasting domains, with selection based on the prediction task and accompanied by rationale.Source: §A.8 · p. 21, Table 8 · p. 22

A.9 Reasoning Logic Examples

Related papers

Loading 2409.17515v3…