PaperlayerView exact revision on arXiv

Source-linked AI summary

LLMODE: Aligning ODEs with LLMs via Gated Token Injection for Irregular Spatio-Temporal Forecasting

Di Zhang, Jingyang Zhang, Ziqian Wang, Chi Zhang, Yikun Ban, Ziwei Zhang, Ruijie Wang

arXiv:2608.29640v1cs.LGcs.AI

TL;DR

Irregular spatio-temporal forecasting is difficult because observations are asynchronous and incomplete, while LLM conditioning faces representation-space and context-length constraints. LLMODE reconstructs continuous-time graph dynamics, compresses dynamic and statistical evidence into fixed-budget memories, and injects them into a frozen LLM through dual-source gated cross-attention. Across urban and physical benchmarks, it achieves competitive performance, with stronger advantages under sparse or dynamically complex observations and strong zero-shot generalization to unseen regions.

  • Problem

    Irregular spatio-temporal forecasting must handle asynchronous, incomplete graph observations while LLM-based conditioning faces representation-space alignment and context-length constraints.

  • Method

    LLMODE uses a graph-aware ODE encoder, fixed-budget resampling of dynamic trajectories, statistical context memory, and dual-source gated cross-attention to condition a frozen LLM.

  • Results

    LLMODE achieves competitive or superior performance across five urban and physical benchmarks, with stronger gains under sparse or dynamically complex observations and strong zero-shot transfer to unseen regions.

  • Takeaways & Limitations

    The framework supports token-efficient forecasting with improved robustness under sparse observations and generalization to unseen regions.

Abstract

from arXiv · show

Large language models (LLMs) have shown promise for spatio-temporal forecasting, but existing approaches often rely on regularly sampled token sequences and struggle with irregular observations because of temporal asynchrony, representation-space misalignment, and limited context windows. We propose LLMODE, a token-efficient framework for irregular spatio-temporal forecasting with a frozen LLM backbone. LLMODE first uses a graph-aware ODE encoder to reconstruct irregular graph observations as a continuous-time latent trajectory. A Fixed-Budget Perceiver Resampler then compresses this variable-length trajectory into a fixed number of dynamic memory tokens. In parallel, compact statistical descriptors are encoded and resampled into context memory tokens. A dual-source gated cross-attention module injects both memories into the frozen LLM, enabling controlled utilization of external spatio-temporal evidence. Experiments on three real-world urban datasets and two physical-dynamics benchmarks show competitive overall performance, with clearer advantages under sparse or dynamically complex irregular sampling. Additional evaluations on unseen urban regions further demonstrate strong zero-shot generalization without adaptation.

1 Introduction

Irregular spatio-temporal forecasting requires modeling asynchronous, incomplete graph observations while addressing token-space alignment and context-length constraints in frozen-LLM conditioning. LLMODE combines continuous-time graph dynamics, fixed-budget memory compression, and gated evidence injection to achieve competitive forecasting and stronger performance in sparse or complex settings.

  • Motivation: Irregular urban and physical-system observations have non-uniform timestamps, missing values, and asynchronous updates across graph locations or entities.Accurate forecasting must jointly model temporal irregularity, incomplete measurements, and graph dependencies.
  • Challenges: Most LLM-based forecasters assume regular sampling and fixed-step tokenization, obscuring actual time gaps, missing intervals, and asynchronous node updates.Existing irregular-aware work remains token-discrete and primarily targets individual sequences, leaving spatial coupling and temporal irregularity to be handled together.
  • Challenges: Neural ODE encoders reconstruct latent trajectories over actual elapsed time, but exposing all states creates redundant tokens and compressed numerical representations remain misaligned with language-pretrained spaces.Effective integration therefore requires compact memory construction and prediction-aware evidence injection.
  • Approach: LLMODE combines a graph-aware ODE encoder, Fixed-Budget Perceiver Resampler, statistical context memory, and dual-source gated cross-attention within a frozen LLM.The framework compresses continuous-time dynamics into dynamic memory tokens and injects dynamic and context evidence through selected LLM layers.
  • Findings: Experiments across three urban datasets and two physical-dynamics benchmarks show competitive performance, with clearer advantages under sparse or dynamically complex observations.The framework also reports strong zero-shot generalization to unseen regions.

2 Problem Formulation

The problem is irregular spatio-temporal graph forecasting: observations arrive at node-specific, non-uniform timestamps, and the model must predict multivariate targets at arbitrary query times.

  • Graph and observations: The system is represented by a fixed graph G = (V, E), with each node carrying a D-dimensional observation vector over continuous time.V and E denote the node and edge sets, while N is the number of nodes.
  • Irregular observations: Observations are collected irregularly rather than on a shared regular grid, with different nodes potentially having different timestamp sets.The intervals between successive observations are non-uniform.
  • Forecasting objective: The forecasting objective jointly predicts a multivariate target vector for each node at each arbitrary query time.The target contains D jointly predicted variables for every requested timestamp.

3 Method

LLMODE reconstructs graph dynamics continuously, compresses dynamic and statistical evidence into fixed-size memories, and injects both streams into a frozen LLM through gated cross-attention. The design emphasizes informative temporal changes while limiting token and attention costs.

  • 3.1 Graph-aware ODE Encoder: A graph-aware ODE encoder propagates latent node states over actual elapsed intervals and applies observation corrections when new measurements arrive.Corrected states receive a gated graph-based residual update before continuous propagation continues.
  • 3.1 Graph-aware ODE Encoder: The encoder collects propagated states across time into a continuous-time latent trajectory that captures temporal evolution and structural interactions.The trajectory is produced from graph-aware latent states and serves as input to fixed-budget resampling.
  • 3.2 Fixed-Budget Perceiver Resampler: A Perceiver Resampler distills the variable-length trajectory into K dynamic memory tokens instead of exposing all reconstructed states to the LLM.This reduces computational cost and avoids dilution from highly redundant adjacent states.
  • 3.2 Fixed-Budget Perceiver Resampler: A dynamics-aware attention bias favors latent states from intervals with larger instantaneous changes while retaining content-based attention.The bias uses normalized latent dynamics and a strength coefficient β to encourage preservation of more pronounced changes.
  • 3.2 Fixed-Budget Perceiver Resampler: Resampled dynamic vectors receive explicit time labels derived from the same attention weights, aligning memory content with requested forecast timestamps.The labeled vectors form the dynamic memory used downstream.
  • 3.3 Dual-Source Gated Cross-Attention: Compact statistical descriptors are encoded and resampled into K context tokens that complement the fine-grained dynamic memory with global information.The two memory streams are injected into selected frozen-LLM layers through dual-source gated cross-attention, with zero-initialized gates providing an identity start.
  • 3.3 Dual-Source Gated Cross-Attention: Each query-slot token corresponds to a requested prediction timestamp, and its final hidden state is projected by a lightweight regression head into the corresponding forecast.For Vicuna-7B, the injection modules are inserted after Transformer Layers 1 and 16.

4 Experiments

Experiments evaluate LLMODE across urban and physical benchmarks, unseen regions, sparse observations, token budgets, evidence selection, and frozen-LLM utilization. Results show competitive forecasting, stronger robustness under sparse or complex dynamics, efficient memory compression, and improved zero-shot transfer.

  • Q1: Overall effectiveness: LLMODE achieves competitive or superior performance across three urban datasets and two physical-dynamics benchmarks, with strongest gains on sparse or interaction-heavy settings.Smaller gains occur on denser mobility datasets.
  • Q2: Unseen-region generalization: LLMODE achieves the best performance on five of six zero-shot metrics and an average relative improvement of 11.4% across all six metrics.The zero-shot evaluation trains on 80 regions and tests on 80 unseen regions without fine-tuning.
  • Q3: Necessity of continuous-time modeling: As missing rates increase, LLMODE maintains the lowest errors while ST-Encoder degrades more rapidly, with a widening advantage under sparser observations.The results support continuous-time modeling for reconstructing dynamics over long unobserved intervals.
  • Q4: Effectiveness of fixed-budget dynamic evidence: Compared with UrbanGPT, LLMODE reduces token budget by nearly 90% and inference latency by approximately 87% while achieving better zero-shot accuracy.K = 5 and K = 8 provide the strongest accuracy–efficiency trade-off; larger budgets and full trajectories yield no consistent gains.
  • Q4: Effectiveness of fixed-budget dynamic evidence: Learned resampling outperforms random selection and linear projection, while dynamics-aware bias and explicit time labels preserve salient changes and their temporal locations.Removing either design component degrades performance.
  • Q5: Frozen-LLM contribution and evidence utilization: Replacing the frozen LLM with trainable alternatives preserves comparable supervised performance but substantially degrades zero-shot performance on unseen regions.Three frozen LLM backbones achieve comparable performance on seen and unseen regions, while gated cross-attention outperforms prompt concatenation.
  • Q5: Frozen-LLM contribution and evidence utilization: Dynamic memory alone is insufficient, shuffled context degrades performance, and gated injection improves integration of complementary dynamic and sample-specific context memories.These ablations indicate that controlled access to both evidence sources matters.
  • Qualitative case study: In a supervised NYCtaxi case, LLMODE follows the ground-truth trajectory more closely and better captures peak magnitude and timing than tPatchGNN.The comparison is presented as consistent with continuous-time modeling, fixed-budget dynamic memory, and gated injection.

5 Related Work

Related work spans graph-based spatio-temporal forecasting, numerical-to-text conversion, and methods that align numerical tokens with LLM representations. LLMODE is situated among approaches using prompting, projections, adapters, fine-tuning, and cross-attention.

  • Spatio-temporal forecasting: Spatio-temporal forecasting methods combine temporal modeling with spatial dependency encoding, using grid-based convolutional operators or graph-based GNN architectures.Graph-based approaches pair GNNs with recurrent or convolutional temporal modules.
  • LLM-based forecasting: Related time-series methods either convert numerical sequences into text or align compact numerical tokens with LLM representations through prompting, projections, adapters, or partial fine-tuning.Some multimodal architectures use cross-attention for integration.

6 Conclusion

LLMODE reconstructs continuous-time graph dynamics from irregular observations, compresses them into fixed-budget memories, and injects dynamic and context evidence into a frozen LLM. Across urban and physical benchmarks, it delivers competitive performance, sparse-observation robustness, unseen-region generalization, and lower token usage and latency than existing LLM forecasters.

  • Conclusion: LLMODE combines graph-aware ODE reconstruction, fixed-budget dynamic memory, and dual-source gated cross-attention into a frozen-LLM forecasting framework.The framework handles asynchronous and partially observed graph data.
  • Conclusion: Experiments validate competitive forecasting, improved sparse-observation robustness, strong unseen-region generalization, and substantially reduced token usage and inference latency.The conclusion covers both urban and physical-dynamics benchmarks.

A.1 Dataset Details

The benchmarks combine event-level urban records and simulated particle dynamics, preserving spatial structure and irregular, asynchronous observations. Evaluation includes supervised and zero-shot forecasting with dataset-specific horizons and training-only normalization.

  • Urban datasets: NYCtaxi, NYCbike, and NYCcrime represent taxi flows, bike-share flows, and crime incidents over grid-like urban regions.NYCtaxi has 263 regions at approximately 3 km × 3 km resolution; NYCbike and NYCcrime have 2162 regions at approximately 1 km × 1 km resolution.
  • Urban dataset construction: Event-level records are assigned to spatial regions and aggregated into graph-structured node observations, including inflow/outflow counts and regional crime-category counts.Each graph node corresponds to an urban region.
  • Irregular sampling: Urban inputs retain non-uniform intervals and asynchronous or incomplete node histories, with NYCtaxi and NYCbike intervals ranging from 10 to 360 minutes.This preserves event-driven irregularity for forecasting evaluation.
  • Task setup: Urban forecasting uses 24-hour histories to predict 12 hours for taxi and bike data, and 96-day histories to predict 48 days for crime data.Taxi and bike observations are cumulative daily flows; crime observations use sliding-window cumulative counts.
  • Training and evaluation: The supervised setting evaluates training regions, while zero-shot prediction tests 80 additional unseen NYC regions without adaptation.Testing uses held-out time periods for the urban datasets.
  • Physics-based datasets: Springs and Charged simulate five interacting particles in a 2D box, with trajectories generated for 60,000 and 12,000 integration steps, respectively.Particles may collide with boundaries but experience no friction or damping.
  • Physics irregular sampling: Particle observations are asynchronously subsampled by independently selecting 20–28 timestamps per particle from reduced-resolution trajectories.Downsampling factors are 500 for Springs and 100 for Charged.
  • Physics task setup: Physics forecasting uses 60 sampled history steps and the subsequent 60 sampled steps as prediction targets, corresponding to 30,000 Springs or 6,000 Charged original integration steps.The task is based on asynchronous partial observations.

A.2 Training Configuration

LLMODE uses two training stages: first, the graph-aware ODE encoder learns continuous dynamic representations; second, the forecasting framework is optimized with the pretrained LLM and ODE encoder frozen.

  • Stage 1: Training the Graph-Aware ODE Encoder: Stage 1 pretrains the graph-aware ODE encoder with GRU-ODE-Bayes using reconstruction likelihood and Bayesian posterior consistency regularization.Both losses are computed only on observed dimensions to accommodate feature-wise missing observations.
  • Stage 2: Training the Frozen-LLM Forecasting Framework: The pretrained encoder generates continuous-time latent trajectories from irregular observations and distills them into dynamic memory tokens.
  • Stage 2: Training the Frozen-LLM Forecasting Framework: Stage 2 optimizes the forecasting framework while keeping the pretrained LLM backbone and graph-aware ODE encoder frozen.
  • Stage 2: Training the Frozen-LLM Forecasting Framework: The final objective minimizes prediction error between forecasts and ground-truth states at all query timestamps.The predicted and ground-truth targets are multivariate vectors for each node and query time.

A.3 Implementation Configuration

The implementation uses compact latent and memory dimensions, fixed token budgets for both memory streams, and standard AdamW optimization with repeated seeded experiments.

  • Model configuration: The graph-aware ODE latent dimension is d_ode = 50, while ODE states and text-encoder outputs are projected into a shared memory space before resampling.Vicuna-7B uses hidden dimension d_LLM = 4096.
  • Model configuration: Both memory streams use a two-layer Perceiver Resampler with eight attention heads and per-head dimension 64.
  • Model configuration: The dynamic and context memory streams each use a fixed token budget of K = 5.
  • Optimization and hardware: Trainable modules use AdamW with learning rate 2 × 10^-4, cosine scheduling, and a 0.03 warmup ratio.
  • Optimization and hardware: Experiments use per-device batch size 32, validation-loss early stopping with patience 10, and averages over five random seeds.The seeds are 42, 43, 44, 45, and 46, with results taken from the best validation checkpoint.

A.4 GRU-ODE Parameterization and Continuous Propagation

The GRU-ODE parameterization gives the continuous latent dynamics a gated update structure, while numerical integration propagates states across each node’s actual nonuniform observation interval.

  • GRU-ODE parameterization: The vector field uses a GRU-like gated structure in which the update gate and candidate state are computed from the current latent state.Element-wise multiplication combines the gated quantities while preserving continuous-time evolution.
  • Continuous propagation: Numerical integration computes each node’s pre-update state by applying the vector field between consecutive observation times.
  • Continuous propagation: The default Euler solver integrates over the actual interval ti − ti−1, allowing nonuniform sampling gaps and arbitrary intermediate-time queries.The graph-corrected state becomes the initial condition for the next interval.

A.5 Observation Encoding and State Correction

LLMODE encodes irregular observations by predicting their expected values and uncertainty, measuring discrepancies, and using these descriptors for state correction. Its fixed-budget resampling and gated evidence injection reduce external-memory costs while preserving forecasting quality under several tested conditions.

  • Observation Encoding and State Correction: The observation head predicts each feature’s expected mean and scale from the propagated latent state.The mean represents the expected observation, while the scale represents feature-wise uncertainty.
  • Observation Encoding and State Correction: Normalized discrepancies quantify how substantially new observations differ from values expected by the propagated latent state.The discrepancy is stabilized with a small constant before being incorporated into the feature descriptor.
  • Observation Encoding and State Correction: Feature descriptors combine predicted means, predicted scales, actual observations, and discrepancies before correction updates the latent state.The correction mechanism uses this observation information together with the pre-correction state.
  • Fixed-Budget Resampling: Fixed-budget resampling converts variable-length trajectories into a fixed number of dynamic memory tokens for interaction with the frozen LLM.The resulting dynamic memory is used for subsequent alignment and interaction with query-slot hidden states.
  • Context Memory: The context-memory resampler similarly compresses variable-length textual representations into a fixed number of context memory tokens.It reuses latent-query resampling without dynamics-aware bias or explicit time labeling.
  • Efficiency and Evaluation: LLMODE reduces UrbanGPT inference latency from 646.73 ms to approximately 84 ms while substantially lowering the token budget.Fixed-budget compression also reduces the reported token budget by approximately 75% versus full-trajectory exposure while maintaining or improving forecasting accuracy.
Loading 2608.29640v1…