PaperlayerView exact revision on arXiv

Source-linked AI summary

SCENARIODIFF: A Scenario-level Guidance Framework for Multimodal Time Series Forecasting--Extended Version

Tuan-Binh Tran, Dat Nguyen Cong, Duc-Trong Le, Thanh Trung Huynh, Tung Kieu

arXiv:2608.17164v1cs.LG

TL;DR

Multimodal forecasters must handle event-driven deviations in noisy, weakly aligned text without asking LLMs to predict numbers directly. SCENARIODIFF hierarchically converts documents into historical evidence, future scenarios, and local anchors, yielding its clearest gains in event-driven domains on Time-MMD.

  • Problem

    Event-driven textual deviations require future-oriented hypotheses and localized constraints that guide numerical forecasters without direct LLM numerical prediction.

  • Method

    SCENARIODIFF hierarchically transforms documents into historical evidence, a forecast-horizon scenario, and localized anchor guidance for a Multimodal Diffusion Transformer.

  • Results

    SCENARIODIFF achieves the strongest horizon-level performance on Time-MMD, with its clearest gains in event-driven domains, especially Economy and Security.

  • Takeaways & Limitations

    Hierarchical scenario guidance is most beneficial when textual evidence contains actionable signals about externally driven future dynamics.

Abstract

from arXiv · show

Textual context such as news, reports, and logs can provide valuable signals for time series forecasting, especially when future dynamics are driven by external events that are not yet visible in historical values. Existing multimodal forecasting methods often either ask large language models (LLMs) to predict numerical values directly or fuse text and time series implicitly, making contextual influence difficult to interpret and control. We propose SCENARIODIFF, a hierarchical contextual reasoning framework for multimodal time series forecasting under noisy and weakly aligned documents. SCENARIODIFF organizes contextual information into three levels: a Historical Context Agent extracts stepwise evidence from raw documents, a Scenario Agent produces a qualitative scenario description for the forecast horizon, and an Anchor Guidance Agent generates sparse anchor points for event-relevant future regions. These structured signals condition a Multimodal Diffusion Transformer, while Anchor Blended Sampling locally refines generated trajectories without retraining. Experiments on the Time-MMD benchmark show that SCENARIODIFF is especially effective in event-driven domains, demonstrating the value of explicit hierarchical scenario guidance for multimodal time series forecasting. Our full implementation is available at https://anonymous.4open.science/r/ScenarioDiff_ICDM-2C4C

I. INTRODUCTION · II. RELATED WORK

SCENARIODIFF addresses the difficulty of using noisy, weakly aligned textual context in multimodal time series forecasting by organizing evidence into hierarchical, interpretable guidance. It combines structured scenario conditioning and inference-time anchor guidance, with strongest gains reported for event-driven domains.

  • I. INTRODUCTION: Multimodal forecasting pairs numerical histories with news, reports, and logs, but noisy, redundant, weakly aligned text makes contextual influence difficult to identify and control.Experts commonly interpret temporal signals alongside external context rather than numerical histories alone.
  • I. INTRODUCTION: Event-driven deviations such as interventions, outages, and demand shifts require future-oriented hypotheses and localized constraints rather than direct numerical forecasts from LLMs.The framework targets abrupt changes that may be weakly reflected in historical values.
  • I. INTRODUCTION: SCENARIODIFF organizes context into historical evidence, a future scenario, and local anchor constraints through frozen Historical Context, Scenario, and Anchor Guidance Agents.The Historical Context Agent produces stepwise summaries aligned with observed history, while the Scenario Agent synthesizes historical series and contextual information.
  • I. INTRODUCTION: A Multimodal Diffusion Transformer conditions iterative denoising on structured agent signals, while Anchor Blended Sampling locally refines generated trajectories without direct LLM numerical generation.The structured signals combine numerical history with scenario-level textual evidence.
  • I. INTRODUCTION: On the Time-MMD benchmark, SCENARIODIFF shows strong performance, with clearest gains in event-driven domains containing actionable textual signals about future dynamics.The result supports the proposed emphasis on explicit hierarchical scenario guidance when external events drive changes weakly reflected in numerical history.
  • II. RELATED WORK: Related work includes LLM-based forecasting methods that transfer language-model backbones, reprogram numerical histories, or align temporal tokens with textual representations.Examples include OFA, Time-LLM, and CALF.
  • II. RELATED WORK: Multimodal time series forecasting methods incorporate contextual text through benchmarks, timestep alignment, text embeddings, and agent-based summarization pipelines.Time-MMD, TaTS, and TimeCAP represent these directions in the related work.
  • II. RELATED WORK: Unlike prior diffusion approaches, SCENARIODIFF uses structured signals encoding historical evidence and future hypotheses, then steers trajectories toward sparse contextual anchor bands at inference time.This guidance is interpretable, flexible, and compatible with standard diffusion backbones.

III. PRELIMINARIES · A. Denoising Diffusion Probabilistic Models · B. Conditional DDPM For Time Series Forecasting

The preliminaries describe DDPMs as Markov-chain generative models that add and remove Gaussian noise, then extend them into conditional time-series forecasting with textual event context.

  • A. Denoising Diffusion Probabilistic Models: DDPMs iteratively add Gaussian noise to a clean sample x(0) over T steps through a Markov chain.The noise schedule is denoted {βt}T.
  • A. Denoising Diffusion Probabilistic Models: The forward diffusion process permits direct sampling of x(t) from the clean sample x(0).The passage introduces the corresponding forward-process sampling expression.
  • A. Denoising Diffusion Probabilistic Models: DDPM reverses diffusion by progressively removing noise through a reverse Markov chain.The reverse process is defined step by step to recover cleaner samples.
  • A. Denoising Diffusion Probabilistic Models: The reverse process starts from pθ(x(T )) = N(0, I), predicts clean x(0) from noisy x(t), and typically uses fixed variance Σt.Here, xθ(x(t), t) denotes the predicted clean sample.
  • B. Conditional DDPM For Time Series Forecasting: For forecasting, conditional DDPM predicts the future horizon x(0) ≡xL+1:L+H from noisy future values and conditioning signals c.The future horizon is ⟨xL+1, xL+2, . . . xL+H⟩, and the denoising objective is formulated accordingly.
  • B. Conditional DDPM For Time Series Forecasting: Beyond the lookback window x1:L, the condition incorporates textual context through local bags Mn containing event descriptions for each historical timestep n.Each timestep n ∈ {1, . . . , L} has a local bag Mn = {si}|Mn| i=1.

IV. METHODOLOGY · A. Framework Overview · B. Hierarchical Contextual Reasoning

SCENARIODIFF uses three hierarchical agents to transform aligned documents and historical observations into stepwise evidence, a horizon-level scenario, and sparse future anchors. These signals condition a diffusion forecaster, with anchors providing localized inference-time refinement while using only information available through the lookback window.

  • A. Framework Overview: SCENARIODIFF organizes contextual reasoning into three levels: historical summaries, a qualitative scenario, and sparse anchor points.The frozen agents respectively extract document-aligned evidence, project a coarse future prior, and generate localized constraints.
  • B. Hierarchical Contextual Reasoning: The Scenario Agent produces a short natural-language description of expected future direction, volatility, or disruptions and encodes it as a horizon-level prior.The scenario representation conditions the diffusion forecaster at the forecast-horizon level.
  • A. Framework Overview: The numerical forecaster is a conditional denoising diffusion model that generates future trajectories from historical observations and contextual conditions.Denoising receives the observed history, noisy future, historical context, and scenario description, starting from Gaussian noise.
  • A. Framework Overview: Anchor Blended Sampling locally refines generated trajectories using sparse anchor points after diffusion sampling.This inference-time refinement makes the trajectory consistent with event-relevant localized constraints.
  • B. Hierarchical Contextual Reasoning: The Historical Context Agent compresses each timestep’s verbose or weakly relevant document set into a concise summary, emitting [NO_EVENT] when no useful evidence exists.The resulting sequence provides a temporally grounded semantic representation of past evidence and is encoded into continuous features injected into historical diffusion tokens.
  • B. Hierarchical Contextual Reasoning: The framework uses the same history-only procedure during training and inference, thereby avoiding temporal leakage.Only information available up to timestep L is used; the Scenario Consistency Score is reserved for analysis and uses an oracle scenario unavailable during training or inference.
  • B. Hierarchical Contextual Reasoning: The Anchor Guidance Agent converts contextual information into sparse anchors that specify future timesteps, plausible value intervals, and confidence weights.Tighter intervals yield stronger guidance, complementing the global scenario with time-localized structure during inference.

C. Multimodal Diffusion Transformer

The Multimodal Diffusion Transformer tokenizes normalized time series and conditions denoising on compact textual representations. It injects historical context and scenario guidance through distinct mechanisms, with adaptive normalization controlling computation across diffusion steps.

  • Architecture: RevIN normalizes the input series to reduce temporal distribution shift and outlier effects before Transformer-based denoising.The normalized series is then converted into diffusion-step-aware tokens by Φts(·).
  • Token Distiller: Learnable prototype queries compress long historical and scenario token sequences into Q distilled tokens, where Q ≪ L.Cross-attention selectively aggregates full-context information while masking padding and fully masked sequences for stability.
  • Condition Injection: The model adds historical context to history tokens, applies cross-attention between future tokens and scenario representations, and jointly updates both through AdaLN Transformer blocks.A lightweight flatten-linear decoder maps final hidden representations back to the reconstructed denoised time series.
  • Condition Injection: During reverse denoising, AdaLN derives scale and shift parameters from diffusion-step embeddings and distilled context to adapt Transformer states.This conditions computation on both the temporal noise level and multimodal context.

D. Anchor Blended Sampling

Anchor Blended Sampling post-processes a diffusion forecast by briefly editing anchor-relevant temporal regions according to anchor constraints. It preserves non-anchor structure and requires no retraining.

  • D. Anchor Blended Sampling: The method starts from a standard reverse-diffusion trajectory, noises it to level T_e, and runs a short reverse editing process.The source trajectory is the initial forecast x_θ^(0).
  • D. Anchor Blended Sampling: At each editing step, the posterior mean is shifted by the negative gradient of the distance-to-band anchor loss, scaled by γ_t.The update uses µ_t + γ_tΣ_tg_t, where g_t = −∇_x(t)L_anch.
  • D. Anchor Blended Sampling: A mask confines guidance to anchor-relevant regions, leaving non-anchor timesteps undisturbed and locally correcting the trajectory without retraining.This blending imposes an inpainting-style constraint while preserving the remaining temporal structure.
  • D. Anchor Blended Sampling: The anchor loss penalizes trajectory values outside each specified interval within a local temporal window, while allowing small temporal shifts.Anchors are represented as A = {(t_j, l_j, u_j, w_j)}.

V. EXPERIMENTS · A. Experimental Settings

Experiments use the multi-domain Time-MMD benchmark with domain-specific forecasting horizons, comparisons against numerical, LLM-based, multimodal, and diffusion baselines, and standard deterministic or probabilistic metrics. Models are implemented with specified diffusion, language-agent, leakage-prevention, and anchor-editing configurations.

  • A. Experimental Settings: Time-MMD pairs numerical time series with aligned textual facts and reports across Economy, Energy, Security, Social Good, and Traffic domains.Series are sampled weekly or monthly and span 1950–2024.
  • A. Experimental Settings: Monthly series use L = 36 with H ∈ {6, 12, 18}, while weekly series use L = 96 with H ∈ {12, 24, 48}.Reported results are averaged over prediction horizons within each domain.
  • A. Experimental Settings: SCENARIODIFF is compared with numerical-only Transformers, LLM-prior methods, paired-text multimodal forecasters, and probabilistic diffusion models.Named baselines include Informer, FEDformer, iTransformer, Time-LLM, S2IP-LLM, TaTS, and MM-TSF.
  • A. Experimental Settings: Deterministic forecasting uses MSE and MAE on standardized series, while stochastic models additionally use CRPS; lower values are better for every metric.CRPS measures predictive-distribution quality.
  • A. Experimental Settings: Models are implemented in PyTorch, trained on one NVIDIA H100 GPU, and use six residual blocks, 8 attention heads, Adam optimization, and 60–150 training epochs.The hidden dimension is selected from d ∈ {64, 128, 256}.
  • A. Experimental Settings: Training uses batch sizes from {16, 32}, learning rates from {10^-4, 5 × 10^-4}, weight decay 10^-6, and DDIM generation with 50 sampling steps.These settings complete the reported diffusion forecaster implementation configuration.
  • A. Experimental Settings: The three guidance agents use Gemini 2.5 Flash offline, while truncated 512-token inputs are encoded by frozen bert-base-uncased representations with d_emb = 768.Forecast origins use only documents aligned with observed timesteps to avoid temporal leakage.
  • A. Experimental Settings: Anchor Blended Sampling retains at most five anchor intervals, uses radius r = 2 and τ = 0.10, and performs N_edit = 6 guided DDIM reverse-editing steps.Mask-restricted, unit-norm-clipped anchor gradients use guidance scale γ = 50, with non-anchor regions blended with the noised source forecast.

B. Main Results

SCENARIODIFF achieves the strongest overall horizon-level performance across the five Time-MMD domains, with particularly clear gains in event-driven Economy and Security. Its hierarchical scenario guidance also improves probabilistic forecasting in event domains, although results remain domain-dependent.

  • Overall forecasting performance: SCENARIODIFF records the largest number of MSE/MAE wins across horizons and domains, with especially strong gains in Economy and Security.It also shows competitive performance on Traffic.
  • Overall forecasting performance: Explicit scenario-level signals make noisy documents more effective for forecasting than unstructured-text fusion.LLM agents extract historical evidence, generate scenario descriptions, and produce anchor points instead of directly predicting numerical values.
  • Probabilistic forecasting: SCENARIODIFF achieves the best event-domain average CRPS among representative diffusion-based probabilistic forecasting models.The improvement is not uniform because strong numerical or diffusion-based baselines remain competitive in some domains.

C. Ablation Studies

Ablations show that all three SCENARIODIFF agents improve forecasting, with historical evidence contributing most and scenario guidance helping especially in event-driven domains. Additional diagnostics show that anchor and textual guidance are useful but depend on alignment, blending choices, and noise levels.

  • Contribution of Components: Removing any agent worsens accuracy, with the Historical Context Agent causing the largest degradation in most domains.The Scenario Agent also matters especially in event-driven domains, while removing Anchor Guidance produces a smaller but consistent drop.
  • Event-driven Subset Analysis: SCENARIODIFF achieves the strongest mean rank on event-driven domains, outperforming representative numerical-only, LLM-prior, multimodal, and diffusion-based baselines.The event-driven partition comprises Economy, Energy, and Security; the non-event-driven partition comprises Social Good and Traffic.
  • Event-driven Subset Analysis: In non-event-driven domains, SCENARIODIFF’s advantage is less pronounced because regular temporal patterns dominate and textual evidence provides fewer actionable signals.Strong numerical-only, multimodal, and diffusion-based baselines achieve better average ranks in this partition.
  • Anchor Quality Analysis: Anchor diagnostics find useful local signals in Economy and Traffic but higher errors and negative endpoint gains in Energy.Anchors are evaluated within local radius r = 2 using MSE@r2 and MAE@r2, with endpoint gains compared against a mean-value baseline.
  • Parameter Sensitivity of Anchor Guidance: Moderate and wide blending windows generally improve Economy forecasting more reliably than very narrow windows.The analysis recommends a compact anchor set and sufficiently wide blending window so anchor information is incorporated smoothly into the diffusion trajectory.
  • Robustness Analyses: Clean source text yields the lowest errors, perturbation generally increases forecasting error, and severe noisy-anchor contamination substantially degrades performance.Moderate anchor contamination has limited impact, indicating tolerance to small amounts of noisy guidance, although the perturbation trend is not strictly monotonic.

D. Computational Cost

SCENARIODIFF separates one-time LLM preprocessing from online forecasting. The three agents generate cached outputs with average latencies of 8–10, 13–15, and around 15 seconds, respectively.

  • Offline preprocessing: The three LLM agents are called once per input instance to generate and cache stepwise context summaries, a scenario description, and anchor points.Their average latencies are 8–10 seconds, 13–15 seconds, and around 15 seconds, respectively.
  • Online forecasting: Online forecasting loads cached agent outputs, encodes them with a frozen text encoder, and performs diffusion sampling.The passage specifies evaluation on a single NVIDIA H100 GPU with batch size 64.

E. Case study · VI. CONCLUSION

The Energy case study shows how SCENARIODIFF’s three-level contextual reasoning and Anchor Blended Sampling guide forecasts toward an upward event-driven change. The conclusion characterizes the framework as conditioning diffusion forecasting on structured, interpretable signals, with strongest benefits in event-driven domains.

  • E. Case study: SCENARIODIFF’s Energy case study demonstrates joint use of historical evidence, scenario guidance, and anchor refinement for event-driven forecasting.The case compares ground truth, an unblended diffusion forecast, an anchor-refined forecast, and PatchTST.
  • E. Case study: Anchor Blended Sampling refines the diffusion trajectory toward local anchor bands and better follows the realized increase than PatchTST.PatchTST remains near the historical range and misses the upward transition.
  • E. Case study: Stepwise context summaries consistently indicate continued gasoline-price increases, providing directional evidence for an upward regime.The realized jump shortly after the forecast origin supports their role as historical grounding evidence.
  • E. Case study: The Historical Context Agent extracts recent price-increase evidence, while the Scenario Agent forms an upward forecast-horizon scenario.These are the first two levels of the case study’s three-level reasoning process.
  • E. Case study: The Anchor Guidance Agent provides local bands for event-relevant future steps, improving the direction and timing captured by the forecast.The full shock magnitude remains underestimated.
  • VI. CONCLUSION: SCENARIODIFF organizes contextual information into historical evidence, a future hypothesis, and local constraints.These correspond to stepwise context summaries, a scenario description, and anchor points.
  • VI. CONCLUSION: The framework conditions a Multimodal Diffusion Transformer on structured, interpretable signals rather than raw or implicitly fused text.Anchor Blended Sampling performs inference-time trajectory refinement grounded in classifier-guided diffusion.
  • VI. CONCLUSION: Experiments on Time-MMD confirm that hierarchical scenario guidance yields the greatest benefit in event-driven domains.This conclusion aligns with the Energy case study’s event-driven forecasting behavior.
Loading 2608.17164v1…