PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM-Integrated Bayesian State Space Models for Multimodal Time-Series Forecasting

Sungjun Cho, Changho Shin, Suenggwan Jo, Xinya Yan, Shourjo Aditya Chaudhuri, Frederic Sala

arXiv:2510.20952v1cs.LGcs.AI

TL;DR

Multimodal forecasting must combine numerical histories with textual context, while existing forecasters can be limited by fixed windows and lack explicit uncertainty modeling. LBS integrates a probabilistic state space model with a pretrained LLM to estimate states from text and generate text from latent trajectories. On TIMETEXTCORPUS, it improves numeric forecasting accuracy by an average of 13.20% over unimodal and multimodal baselines.

  • Problem

    Existing multimodal forecasters are commonly trained with fixed input windows and prediction horizons and no explicit uncertainty modeling, despite the value of textual context for forecasting and interpretability.

    Source: §1 · p. 1, §3.1 · p. 3

  • Method

    LBS combines a Bayesian SSM with a pretrained LLM, using textual inputs for posterior state estimation and latent trajectories to generate textual forecasts.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    13.20% average improvement in numeric forecasting accuracy over unimodal and multimodal baselines on TIMETEXTCORPUS, alongside coherent and contextually relevant textual predictions.

    Source: §1 · p. 2

  • Takeaways & Limitations

    LBS grounds numerical and textual forecasts in a shared latent dynamical system, with uncertainty estimation and flexible prediction horizons.

    Source: §6 · p. 9

    LBS is trained with single-step batches; multi-step training could improve performance but poses computational and compatibility challenges with LLM optimization frameworks.

    Source: §6 · p. 9

Abstract

from arXiv · show

Forecasting in the real world requires integrating structured time-series data with unstructured textual information, but existing methods are architecturally limited by fixed input/output horizons and are unable to model or quantify uncertainty. We address this challenge by introducing LLM-integrated Bayesian State space models (LBS), a novel probabilistic framework for multimodal temporal forecasting. At a high level, LBS consists of two components: (1) a state space model (SSM) backbone that captures the temporal dynamics of latent states from which both numerical and textual observations are generated and (2) a pretrained large language model (LLM) that is adapted to encode textual inputs for posterior state estimation and decode textual forecasts consistent with the latent trajectory. This design enables flexible lookback and forecast windows, principled uncertainty quantification, and improved temporal generalization thanks to the well-suited inductive bias of SSMs toward modeling dynamical systems. Experiments on the TextTimeCorpus benchmark demonstrate that LBS improves the previous state-of-the-art by 13.20% while providing human-readable summaries of each forecast. Our work is the first to unify LLMs and SSMs for joint numerical and textual prediction, offering a novel foundation for multimodal temporal reasoning.

1 Introduction

Forecasting can benefit from textual context as well as numerical histories, but integrating both modalities raises challenges in estimating latent states from text and generating text from states. LBS combines a probabilistic state space model with a pretrained language model for joint numerical and textual forecasting.

1 Introduction

  • Textual context can provide forecasting signals that numerical data alone cannot fully capture, while textual forecasts can improve interpretability in high-stakes decisions.Examples include clinical notes, financial reports, and weather descriptions.Source: §1 · p. 1
  • Integrating a probabilistic SSM with a pretrained LLM raises two challenges: estimating posterior states using textual observations and generating temporally grounded text from latent states.The SSM offers uncertainty quantification and flexible, variable-length prediction horizons.Source: §1 · p. 1
  • Figure 1 illustrates a single-step setup in which context-summary tokens and a target value inform posterior state estimation, while a forecast state conditions text generation.Prompt templates accompany both steps when instruction-tuned LLMs are used.Source: Figure 1 · p. 2
  • LBS summarizes textual inputs into tokens for text-conditioned Bayesian state estimation and generates temporally coherent forecasts by conditioning an LLM on latent state trajectories.The authors describe LBS as the first framework to tightly couple probabilistic SSMs and LLMs for forecasts across both modalities.Source: §1 · p. 2
  • 13.20% average improvement in numeric forecasting accuracy over unimodal and multimodal baselines, with coherent and contextually relevant textual predictions.The evaluation uses the multimodal TIMETEXTCORPUS benchmark.Source: §1 · p. 2
  • The paper identifies LBS as a multimodal forecasting architecture that combines a Bayesian SSM with pretrained LLMs and evaluates it on climate and medical domains.The benchmark is TIMETEXTCORPUS.Source: §1 · p. 2

2 Related Work

Prior work has explored language models for multimodal time-series forecasting and Bayesian state-space models for temporal modeling, but the paper identifies their integration as an unexplored combination. LBS applies that combination to joint numerical and textual forecasting.

2 Related Work

  • Prior multimodal forecasting work uses LLMs directly, auxiliary modules, cross-attention, shared embeddings, foundation models, or concurrent text inputs.The cited examples include [27] [8], [37], [20], [34], and [24].Source: §2 · p. 3
  • Bayesian state-space research spans Kalman filtering and nonlinear extensions, as well as neural latent dynamics for time series and partially observable control.Examples include the Kalman filter, Unscented Kalman Filter, KVAE [5], PlaNet [11], Dreamer [12], and KalmanNet [31].Source: §2 · p. 3
  • The paper identifies a gap: prior work largely focuses on unimodal data, and it presents LBS as the first integration of pretrained LLMs with probabilistic SSMs for joint numerical and textual forecasting.The authors describe this as extending Bayesian state estimation into multimodal generative modeling.Source: §2 · p. 3

3 Preliminaries

Multimodal forecasting combines numerical series with aligned text to predict future values and text, while common multimodal forecasters use fixed windows and do not explicitly model uncertainty. LBS's probabilistic SSM represents shared latent dynamics underlying both modalities, with an LLM modeling text from those states.

3.1 Multimodal Forecasting

  • In multimodal forecasting, aligned numerical observations and text are used to predict future values and corresponding text.Source: §3.1 · p. 3
  • Common multimodal forecasters map fixed-length input windows to fixed forecast horizons without explicitly modeling uncertainty, limiting their adaptability to real-world scenarios.Source: §3.1 · p. 3

3.2 Bayesian State Space Models

  • The SSM provides a flexible temporal backbone for latent dynamics, while a pretrained, fine-tuned LLM models text conditioned on each latent state.The numeric observation model is Gaussian with fixed variance and a mean computed from the latent state by an MLP.Source: §3.2 · p. 4
  • LBS assumes a shared latent state evolves stochastically over time and generates numerical and textual observations as conditionally independent emissions.The state transition, numeric emission, and textual emission distributions define the model's three components.Source: §3.2 · p. 4
  • Training maximizes a multimodal ELBO using a variational posterior conditioned on numerical and textual observations, balancing data reconstruction, temporal regularization, and prediction under latent-state uncertainty.The variational posterior is parameterized via a deep Kalman filter.Source: §3.2 · p. 4

4 LBS: LLM-Integrated Bayesian State Space Model

LBS adapts a pretrained LLM to encode text for posterior state estimation and generate text conditioned on latent state trajectories. Its components compress textual observations into latent summaries, infer posterior states, and use sampled states to condition text generation.

4 LBS: LLM-Integrated Bayesian State Space Model

  • LBS addresses text-conditioned posterior estimation and latent-state-conditioned text generation by adapting a pretrained LLM within a probabilistic state-space framework.Source: §4 · p. 5

4.1 Text-conditioned Posterior State Estimation

  • Learnable summary tokens compress each textual observation into a vector projected to the latent-state dimension for posterior inference.The LLM processes the observation with appended summary tokens; their final hidden states are concatenated and projected by an MLP.Source: §4.1 · p. 5
  • A neural Kalman filter estimates a Gaussian posterior's mean and diagonal covariance from the text summary, numeric target, and prior state.The end-to-end process fine-tunes the LLM with LoRA to encode forecasting-relevant information in the summary tokens.Source: §4.1 · p. 5

4.2 State-conditioned Text Generation

  • A sampled posterior state is projected into prefix tokens for the LLM, conditioning text generation on dynamics captured by the temporal SSM backbone.This design assumes the backbone encodes time-specific information such as trends, event structure, or contextual shifts in a compact latent space.Source: §4.2 · p. 6
  • LBS shares LLM weights between text encoding and decoding, and trains on temporally ordered single-step batches with hidden states passed to the next iteration.Stateful training addresses the computational expense of passing each text observation through the LLM twice during long-sequence training.Source: §4.2 · p. 6

5 Experimental Results

The experiments assess LBS’s forecasting performance, uncertainty, latent-state structure, and sensitivity to LLM size. Across the reported analyses, LBS improves forecasting, while textual gains tend to grow at longer horizons.

5 Experimental Results

  • The evaluation uses TTC-CLIMATE weather descriptions and TTC-MEDICAL nursing notes paired with daily measurements, and compares multimodal methods using LLaMA3.1-8B.The LBS backbone is a single-layer GRU with latent dimension 16; one LBS model is tested across forecast horizons.Source: §5 · pp. 6–7, §5.1 · p. 7

5.1 Time-Series Forecasting Performance

  • 5.13% on TTC-CLIMATE and 21.28% on TTC-MEDICAL: LBS improves the best baseline performances on average across most prediction horizons.A single LBS model is evaluated across all horizons, unlike baselines trained separately for each horizon; Table 1 also reports horizon-averaged RMSE and relative improvements from incorporating text.Source: §5.1 · p. 7, Table 1 · p. 7
  • LBS generates temporally coherent, context-aware textual forecasts from a prior-state embedding and a simple prompt, without direct access to previous text.Figure 5 compares an LBS-generated TTC-CLIMATE forecast with ground-truth text.Source: §5.1 · p. 7, Figure 5 · p. 7

5.2 Uncertainty in Forecasts

  • LBS provides uncertainty intervals, with predicted variance higher during high-fluctuation periods such as early winter and lower during calmer periods such as summer.The single-step comparison places HybridMMF above LBS and shows the LBS prediction variance alongside the true values.Source: §5.2 · p. 8, Figure 6 · p. 8

5.3 Analysis on Latent State Trajectory

  • LBS’s posterior latent states exhibit seasonal periodicity aligned with the ground-truth signal, supporting transparency and diagnosis of potential errors.The visualization shows ground-truth signals and three t-SNE components, with dashed lines marking yearly intervals.Source: §5.3 · p. 8, Figure 7 · p. 8

5.4 Effect of LLM Scaling

  • Larger LLMs do not consistently improve forecasting; Qwen2.5-7B is outperformed by its 1.5B variant, while textual gains tend to grow with prediction horizon.The scaling analysis is limited to TTC-CLIMATE; the authors suggest SSM capacity and text compression as possible explanations for the scaling result.Source: §5.4 · pp. 8–9, Figure 8 · p. 9

6 Concluding Remarks

LBS combines a Bayesian SSM with pretrained LLMs for multimodal forecasting, and the TTC experiments report baseline improvements and larger textual gains at longer horizons. The authors identify multi-step training and newer state-space architectures as future directions constrained by current computational and implementation challenges.

6 Concluding Remarks

  • LBS integrates a Bayesian SSM with pretrained LLMs for multimodal forecasting, grounding numerical and textual observations in a shared latent dynamical system.The conclusion describes coherent forecasts, uncertainty estimation, and flexible prediction horizons.Source: §6 · p. 9
  • On TTC, LBS outperforms existing baselines, with textual data providing greater gains at longer forecasting horizons.Source: §6 · p. 9
  • Multi-step training may improve long-horizon performance, but computational and LLM-optimization compatibility challenges remain; integration with Mamba is left to future work.The authors note that optimized implementations for stateful training are still in development.Source: §6 · p. 9

Broader Impact Statement

The authors warn that coherent LLM-generated forecasts may be misused or mistaken for authoritative facts, particularly in high-stakes domains. They recommend caution, clear communication of limitations, and human oversight and fact verification.

Broader Impact Statement

  • LLM-generated text conditioned on structured signals can sound plausible and be misinterpreted as authoritative or factual.Source: Broader Impact Statement · p. 10
  • The authors identify healthcare and climate science as high-stakes settings where uncritical sharing of textual forecasts may contribute to misinformation.Source: Broader Impact Statement · p. 10
  • They recommend caution, clear communication of model limitations, human oversight, and fact verification for text outputs.Source: Broader Impact Statement · p. 10

A Extended Related Work

Prior work combines language models and time-series methods in several ways, while Bayesian state-space research has largely focused on unimodal data. LBS is presented as the first integration of pretrained LLMs with probabilistic state-space models for joint numeric and textual forecasting.

Multimodal Time-Series Forecasting

  • Earlier multimodal forecasting work uses textual context, cross-attention, shared embeddings, foundation models, or text-derived auxiliary variables, while TTC provides a benchmark and dataset [27] [37] [26] [35] [20] [34] [24].The approaches span news selection and analysis, joint text and time-series prediction, and conversion of text into auxiliary variables.Source: A Extended Related Work · p. 13

Research Gap

  • Prior LLM forecasting work had not integrated LLMs into state-space models; the paper presents this integration as enabling forecasting performance gains and principled uncertainty quantification.The broader gap is that existing work largely targets unimodal data, whereas LBS combines pretrained LLMs with probabilistic state-space models for joint numeric and textual forecasting.Source: A Extended Related Work · p. 13

Bayesian State Space Models

  • Bayesian filtering spans Kalman, extended and unscented Kalman, ensemble, and sequential Monte Carlo methods for linear or nonlinear, Gaussian or non-Gaussian settings.Deep neural parameterizations have also been combined with probabilistic latent dynamics in KVAE, retail forecasting, and model-based control [5] [11] [12] [31].Source: A Extended Related Work · p. 13

B Derivation of Training Objective

The training objective is derived from LBS's autoregressive structure using Jensen’s inequality. Its terms include observation likelihoods and a transition-related contribution conditioned on the inferred state distribution.

Derivation of Training Objective

  • The training objective is derived from LBS’s autoregressive structure and Jensen’s inequality.The displayed likelihood concerns the joint numerical and textual observations, with latent states integrated out.Source: B Derivation of Training Objective · p. 14
  • The objective includes value and text likelihood terms together with a transition term evaluated under q(xt | y1:t, D1:t).The passage labels the observation terms as value likelihood and text likelihood.Source: B Derivation of Training Objective · p. 14

C Additional Experiment Details

The additional experiments use climate and medical data with chronological train, validation, and test splits, and specify model, prompt, optimization, and loss-weighting choices. The appendix identifies the benchmark’s full test-RMSE results without providing their values here.

Datasets

  • TTC-CLIMATE contains daily Washington, DC temperatures with weather descriptions, while TTC-MEDICAL contains daily heart rates and nursing notes for 73 patients averaging 104 days of data.Both datasets use the first 80% of timestamps for training, the next 10% for validation, and the final 10% for testing.Source: C Additional Experiment Details · p. 14

Models and Prompts

  • The model uses a one-layer GRU with 16-dimensional state and hidden dimensions, LLaMA3.1-8B [7] adapted with LoRA, and eight summary and prefix tokens.The prompts ask the LLM to compress text into summary vectors and to generate textual forecasts from a belief state and date.Source: C Additional Experiment Details · p. 14

Optimization

  • Training uses AdamW, cosine learning-rate decay from 5e-4 to 5e-5, up to 20 epochs, and early stopping after five validation epochs without improvement.A free-nats threshold of 2.5 is annealed to zero during training.Source: C Additional Experiment Details · p. 15

Loss Weighting

  • Uniform loss weighting tends to favor text likelihood and underfit numerical predictions; the chosen weights are αval = αKL = 1.0 and αtext = 0.1.The authors describe this weighting as a good trade-off without additional tuning.Source: C Additional Experiment Details · p. 15

Full Results

  • Table 2 reports test RMSE on TTC-CLIMATE and TTC-MEDICAL across prediction horizons, with the best result for each horizon highlighted.The supplied caption does not state any table values or identify a winning method.Source: Table 2 · p. 15, C Additional Experiment Details · p. 15

Related papers

Loading 2510.20952v1…