PaperlayerView exact revision on arXiv

Source-linked AI summary

ConceptTS: LLM-Guided Concept Bottlenecks for Interpretable Multivariate Time-Series Forecasting

Yichen Jiang, Yueqiao Chen, Dongyu Liu

arXiv:2608.21277v1cs.LG

TL;DR

Opaque multivariate time-series forecasters provide limited insight into why forecasts are produced. ConceptTS uses an offline LLM to generate named concepts and executable labeling rules across three temporal scopes, then forecasts from their bottleneck representations. On Beijing air-quality data, it achieves competitive accuracy while supporting concept-level inspection and intervention.

  • Problem

    Existing multivariate forecasters capture complex dependencies but expose limited evidence about which temporal conditions and variable interactions shape a forecast.

  • Method

    ConceptTS uses an offline LLM to generate named temporal concepts and executable labeling rules, organizing them into historical, local-horizon, and global-horizon bottlenecks with a shared decoder.

  • Results

    ConceptTS achieves competitive accuracy across three air-quality sites and two settings; inverted concept labels increase MAE from 15.371 to 99.661.

  • Takeaways & Limitations

    The forecasting pathway supports concept-level inspection and intervention without requiring manual segment-level annotation or an LLM at inference time.

Abstract

from arXiv · show

State-of-the-art multivariate time-series forecasters can model complex temporal and cross-variable dependencies, yet their opaque representations provide limited insight into why a particular forecast is produced. This lack of transparency restricts their use in settings where practitioners must understand and assess the factors underlying a prediction. We introduce ConceptTS, an interpretable forecasting framework that organizes its predictions around named, human-readable concepts. ConceptTS uses a large language model to propose task-relevant concepts and generate executable labeling rules, translating the language model's domain knowledge into direct supervision without costly manual concept annotation. The proposed concepts are organized into three complementary bottlenecks that describe the historical context, local forecast intervals, and the full forecast horizon. A shared decoder combines representations derived from their predicted activations to construct the forecast, making the model's decision process explicit and supporting direct concept-level interventions. Experiments on the Beijing Multi-Site Air Quality dataset show that ConceptTS achieves accuracy competitive with strong black-box baselines while producing semantically meaningful concept activations.

I. INTRODUCTION

ConceptTS addresses the interpretability gap in multivariate forecasting by organizing predictions around named temporal concepts generated and labeled with LLM assistance. It combines these concepts in an intrinsic forecasting pathway and evaluates both predictive accuracy and concept reliance.

  • Motivation: ConceptTS targets opaque forecasts whose influential temporal conditions and variable interactions are not exposed as human-understandable evidence.The paper contrasts concept-level explanations with importance scores over variables, timestamps, or selected instances.
  • Contribution: An offline LLM proposes named concepts, natural-language definitions, and executable labeling rules from domain and dataset context, avoiding manual segment-level annotation.The generated predicates automatically label training segments.
  • Contribution: ConceptTS organizes concepts into historical, local forecast-interval, and global forecast-horizon bottlenecks, with a shared decoder constructing forecasts from their representations.The architecture preserves temporal and variable structure while supporting intervention around the concept pathway.
  • Evaluation: Experiments across three Beijing air-quality sites and two input regimes find competitive forecasting performance, including MAE within 5% of the best look-back-only baseline.The evaluation also includes ablations, case studies, concept sweeps, and controlled interventions.
  • Evaluation: Replacing activations with inverted ground-truth concept labels increases test MAE from 15.371 to 99.661, indicating reliance on the concept-activation channel in the tested configuration.This intervention directly probes whether the decoder uses the displayed activation pathway.

A. Formulation and Framework

ConceptTS forecasts an endogenous target from historical target and covariate sequences plus, when available, future covariates. Its framework uses training-only statistical summaries and clustering to prompt an offline LLM for multiscale concepts and executable predicates.

  • 1) Problem Formulation: ConceptTS predicts a target trajectory from look-back targets, look-back covariates, and future covariates, with a separate look-back-only evaluation setting.The future-aware formulation supplies covariates over the forecast horizon, whereas the alternative setting removes that access.
  • 2) Framework Overview: The pipeline encodes inputs, predicts activations in three parallel concept bottlenecks, and assembles concept contexts with a BiGRU and regularized residual channel.A shared residual sub-decoder produces the final forecast end-to-end.
  • 3) LLM Concept Proposal Generation: Dataset statistics and executable Python predicates let the offline LLM generate concepts and automatically label every training segment without human-authored segment labels.The proposer is invoked once; predicates are executed over precomputed segment statistics.
  • 1) Per-Segment Statistical Summary: Segment summaries include per-channel look-back statistics, FFT descriptors, forecast statistics, and sub-window statistics.These features expose level, variability, trends, ranges, cyclic patterns, and finer-grained horizon descriptions.
  • 1) Per-Segment Statistical Summary: Cross-channel correlations, timestamps, and seasonal tags supplement the segment summaries used for concept proposal and activation labeling.The combined information gives the LLM temporal, cross-variable, and seasonal context.
  • 2) Summary Vector Clustering and Profiling: Training-only K-means clustering supplies cluster sizes, continuous-channel z-scores, and categorical-channel purity profiles to guide LLM proposals without leakage.Cluster and global statistics referenced in this process are computed on the training split only.
  • 3) LLM Concept Proposal Generation: The LLM generates look-back concepts for history, sub-window concepts for local covariate patterns, and global concepts for full-horizon behavior.Examples include heat or rain in history, morning rush periods, rising wind, extreme temperature changes, and pressure swings.
  • 3) LLM Concept Proposal Generation: Forecast predicates use context covariates rather than the unavailable target, and concepts are filtered by training positive rates before supervising binary activations.Only concepts with positive rates in [rmin, rmax] are retained.

4) Concept Filtering:

The encoder preserves variable identity and temporal structure, then attention-weighted pooling creates sharper summaries for the historical, sub-window, and global concept bottlenecks. Separate encoders impose an information barrier between look-back and forecast inputs.

  • Encoder: ModernTCN-style encoding preserves channel identity and captures broad temporal context through variable-independent embeddings and large-kernel depthwise convolutions.Grouped pointwise bottlenecks mix feature and variable information while residual blocks preserve the representation pathway.
  • Encoder: Parallel encoders separately process look-back and forecasting inputs to manually impose an information barrier between the two time slots.This separation is used in the default setting.
  • Concept Bottlenecks: Three parallel bottlenecks summarize recent history, local forecast sub-windows, and global forecast-horizon trends.The sub-window bottlenecks capture covariate patterns in individual prediction intervals, while the global bottleneck captures macro behavior.
  • Attention-weighted Context Pooling: AttentionPool converts encoder embeddings into context vectors by learning additive attention scores across timesteps and returning a weighted sum.The attention MLP uses two layers, W1 and W2.
  • Attention-weighted Context Pooling: Additive attention emphasizes semantically important timesteps and suppresses irrelevant ones, avoiding the dilution caused by uniform mean pooling.This gives concept bottlenecks a sharper temporal summary for localized signals.

2) Concept Embedding Module:

ConceptTS uses concept embeddings for active and inactive states, with input-independent embeddings intended to limit information leakage. Sub-window contexts are modeled sequentially before entering a decoder that retains a controlled residual path.

  • Concept Embedding Module: The concept embedding module represents each concept with separate learned embeddings for its active and inactive states.This adapts the concept embedding model structure rather than the original binary concept bottleneck alone.
  • Concept Embedding Module: Input-independent concept embeddings remain static during inference so the displayed activation pathway is less vulnerable to information leakage.The design aims to preserve the strength of interpretability.
  • Concept Embedding Module: A scoring head maps an encoder embedding to a scalar logit, applies a sigmoid for activation probability, and mixes the active and inactive embeddings accordingly.Stacking these mixtures across concepts forms the bottleneck output.
  • Sub-window Context: A bidirectional GRU models temporal correlations among forecast sub-window concept contexts and supplies cross-sub-window information to each sub-decoder.The design captures dependencies such as effects across neighboring forecast intervals.
  • Sub-window Context: Using bottleneck outputs rather than encoder embeddings as BiGRU inputs prevents information from bypassing the concept pathway.This preserves the intended bottleneck for cross-sub-window context.

4) Residual Sub-decoder Stack:

The residual sub-decoder combines multiscale concept bottleneck embeddings with contextual and limited residual information, refining forecasts through stacked feedback blocks. Its hybrid objective supervises both forecast accuracy and concept activations while regularizing information outside the bottlenecks.

  • Residual Sub-decoder Stack: The sub-decoder concatenates three concept-bottleneck outputs with cross-sub-window context, positional embeddings, and a narrow residual bypass.The bypass includes shared look-back and per-sub-window projections of encoder features.
  • Residual Sub-decoder Stack: The residual bypass preserves continuous information that scalar concept activations cannot fully express while its regularization weight controls the accuracy–interpretability trade-off.The bypass is low-dimensional and regularized to discourage excessive reliance on it.
  • Residual Sub-decoder Stack: Ndec stacked MLP blocks iteratively refine predictions, with later blocks receiving the running forecast and neighboring sub-window context.Partial predictions are concatenated along the time axis to form the final forecasting curve.
  • Residual Sub-decoder Stack: The two encoders, three CEM modules, and decoder are trained jointly end-to-end using a hybrid objective.The objective includes forecast, concept, boundary-smoothness, and variance-related terms.
  • Residual Sub-decoder Stack: Table I and Table II report forecasting performance for future-aware and look-back-only settings, respectively, using the study’s baseline-comparison conventions.Table I covers three Beijing monitoring stations, while Table II covers the look-back-only setting; lower values are better.

IV. EXPERIMENTS

Experiments evaluate ConceptTS on Beijing air-quality forecasting under future-aware and look-back-only data regimes. The study measures forecasting error across monitoring sites and examines concept behavior through ablations and interventions.

  • Dataset and Evaluation: The Beijing dataset contains six pollutant concentrations and five meteorological features collected at twelve monitoring sites, with PM2.5 as the target.The experiments use additional pollutants and meteorological measurements as context features.
  • Dataset and Evaluation: Overall forecasting performance is benchmarked across the Aotizhongxin, Dingling, and Tiantan monitoring sites.Aotizhongxin is used for subsequent ablation and case-study experiments.
  • Dataset and Evaluation: Forecasts are evaluated using mean absolute error (MAE) and root mean squared error (RMSE), reported in raw µg/m3.MAE measures average deviation, while RMSE weights larger errors more heavily.
  • Experimental Settings: The future-aware setting assumes forecast-horizon exogenous features are known, whereas the look-back-only setting removes access to those future covariates.The two regimes reflect different data-availability conditions and use different concept bottleneck types.
  • Experimental Settings: The experiments compare against different baseline sets in each setting because most Time-Series-Library baselines consume only the look-back window.This separates comparisons according to whether future covariates are available.

4) Model Evaluation:

ConceptTS remains competitive with black-box forecasters in both future-aware and look-back-only settings while exposing concept activations and supporting intervention-oriented analysis. Ablations show that forecast-subwindow concept capacity and residual weighting govern the balance between accuracy and interpretability.

  • Benchmark forecasting performance: 12.71µg/m3 MAE matches strong baselines and outperforms several competitors for Aotizhongxin under the future-aware setting.ConceptTS beats Informer, Random Forest, DeepAR, and NHiT, while matching XGBoost, TFT, and LightGBM.
  • Benchmark forecasting performance: 55.59µg/m3 MAE beats four Time-Series-Library baselines in the look-back-only setting, while remaining within 0.81µg/m3 of the strongest listed competitors.ConceptTS outperforms TimeXer, iTransformer, DLinear, and LSTM, but trails Informer and Crossformer slightly.
  • Number of concepts: Increasing forecast sub-window concept count monotonically decreases test MAE when look-back and global concept counts are fixed at 10.The residual connection is disabled, so forecasting must pass entirely through the concept bottleneck in this ablation.
  • Accuracy–interpretability trade-off: Lowering αresidual improves full-model MAE from 15.35 to 12.71 but widens the concept-only gap from 0.5% to 161.0%.The residual weight therefore controls how strongly the decoder relies on the concept bottleneck versus the residual path.
  • Accuracy–interpretability trade-off: As αresidual increases, concept-only forecasts converge toward full-model forecasts, indicating greater reliance on concept bottlenecks.Figure 3 compares ground truth, full-model, and concept-only curves at αresidual values of 0.01, 0.1, and 1.0.
  • Concept activation case study: In a cold, dry test window, all three bottlenecks activate matching concepts, while the look-back bottleneck also identifies a winter block and wind–PM2.5 anticorrelation.The activation pattern corresponds to near-zero wind followed by rising wind and sharply falling PM2.5.

2) Concept Sweep Study:

ConceptTS supports interpretability by sweeping individual concept activations and measuring how forecasts change. The intervention diagnostic further tests whether predictions rely on the concept activation channel.

  • Concept Sweep Study: Sweeping one concept’s activation probability from 0 to 1 reveals its direct influence on the forecast while other predicted probabilities remain fixed.The study uses two contrasting test windows and manually varies a single activation.
  • Concept Sweep Study: “is calm block” raises predicted PM2.5 as its activation increases in both low- and high-pollutant windows, matching the stated physical interpretation.The concept denotes consistently low wind speed during a sub-window; the windows have ground-truth means of approximately 12 and 332 µg/m3.
  • Interpretability Evidence: The case studies jointly indicate that concept activations describe inputs and that activation interventions expose how concepts contribute to forecasts.Fig. 4 displays the top-five concepts for each of three concept sets, while Fig. 5 displays forecast curves under activation sweeps.
  • Concept Intervention Diagnostic: The intervention diagnostic replaces predicted activations with baseline, oracle, flip, or random regimes to test the decoder’s reliance on concepts.The flip regime uses 1−ground-truth labels, and the random regime samples Bernoulli(0.5) activations.
  • Concept Intervention Diagnostic: 84.990 µg/m3: flipping concept activations increases test MAE from 15.371 to 99.661 µg/m3, while random activations increase it to 60.818 µg/m3.The reported degradations are significant and support the conclusion that the activation channel carries most predictive signal.
  • Concept Intervention Diagnostic: Continuous activation probabilities outperform hard binary labels in the diagnostic, with oracle ground-truth labels causing a slight degradation relative to the model’s soft assignments.The authors interpret soft probabilities as representing degrees of a concept, such as “very cold” or “slightly cold.”

APPENDIX A PROTOTS FAILURE MODE

The ProtoTS analysis motivates ConceptTS by identifying interpretability and expressivity problems in prototype-based forecasting. ConceptTS instead uses smaller combinations of named concepts, while concept analysis exposes a remaining weakness for temporal-shape concepts.

  • ProtoTS Architecture: ProtoTS independently embeds channels, mixes time and feature axes with residual MLP-mixer blocks, and routes representations through hierarchical prototypes.Its prototype layer compares an input-derived key with prototype embeddings and combines routed prototype values into forecasts.
  • Motivations from ProtoTS: Randomly initialized ProtoTS prototypes lack specified semantic meaning, so interpreting them requires post-training inspection that can be time-consuming and ambiguous.Experts must examine activation patterns together with corresponding prototype-value shapes.
  • Motivations from ProtoTS: Mixing time and feature axes captures joint patterns but makes specific cross-channel semantics harder to isolate from a single learned prototype.The paper gives “extremely low temperature leads to an electricity-load spike” as an example of a difficult post-hoc interpretation.
  • Motivations from ProtoTS: ProtoTS forecasts combine activation score, value bias, and prototype value, allowing an auxiliary value-bias pathway to carry input-dependent predictive information.The authors observed nearly uniform activation scores while most forecasting signal flowed through value bias × prototype value.
  • Motivations for ConceptTS: ConceptTS replaces potentially uncontrolled prototype growth with smaller combinations of concepts, preserving broad scenario coverage through on/off combinations.The paper states that 20–30 concepts can span tens of thousands of combinations.
  • Concept Failure Mode Analysis: Shape-dependent concepts are consistently harder to learn than level-based concepts, whose AUROC routinely exceeds 0.9.Examples of difficult concepts include warming, rising pressure, and stable temperature, whereas history-dry and cold-block concepts perform better.
  • Concept Failure Mode Analysis: Statistics-based predicates inadequately express temporal curve shape, producing internally inconsistent labels and noisier learning targets for shape-dependent concepts.The authors propose adding shape descriptors to per-segment summaries so generated predicates can represent these conditions more precisely.

APPENDIX C MODERNTCN ENCODER STRUCTURE

The appendix specifies the ModernTCN-style encoder, training losses, and data-processing settings used by ConceptTS. Its design mixes temporal, feature, and variable information while supervising forecast and concept predictions.

  • ModernTCN encoder: Depthwise convolution uses kernel size κ = 51 to mix temporal patterns separately for each variable-feature pair.The tensor is reshaped to (B, C × D, N), and Conv1D uses groups = CD.
  • ModernTCN encoder: Grouped pointwise convolutions first mix features within each variable, then mix variables within each feature.The two inverted-bottleneck modules use groups = C and groups = D, respectively.
  • Loss composition: The forecast loss is mean L1 error on the z-scored target, while concept losses use binary cross-entropy against LLM-derived binary labels.Concept losses cover look-back, forecast sub-window, and global forecast bottlenecks.
  • Loss composition: Boundary smoothness penalizes discontinuities between adjacent forecast sub-windows, and variance regularization prevents concept activations from collapsing across samples.The variance penalty applies when batch-wise activation variance falls below σmin.
  • Loss composition: Residual regularization limits reliance on the low-dimensional residual channel, with αres controlling the trade-off between forecast accuracy and interpretability.The residual vector receives an L2 penalty.
  • Experimental setup: Experiments use chronological station-level splits, training-only normalization statistics, 96-hour look-backs and horizons, and six-hour window extraction.The Beijing Multi-Site Air Quality data are split into 70% training, 10% validation, and 20% test sets.

B. Model Architecture Hyperparameters •

The appendix gives the principal ConceptTS hyperparameters and evaluation settings, including encoder dimensions, bottleneck sizes, optimization choices, baselines, and ablation procedures. It also reports the relative importance of the three bottlenecks.

  • Architecture: The encoder uses D = 32, patch size P = 4, stride S = 4, κ = 51, four ModernTCN blocks, and Denc = 128.Separate encoders process the look-back and forecast windows.
  • Bottlenecks: Concept embeddings have dimension dc = 16, with 20 look-back, 50 forecast sub-window, and 20 global-forecast concepts.Forecast sub-window predictors use separate weights per sub-window while sharing concept embeddings.
  • Optimization: Training uses Adam with learning rate 3 × 10^-4, weight decay 1 × 10^-4, batch size 16, and up to 60 epochs with validation-MAE early stopping.The default residual weight is αres = 0.01.
  • Evaluation: ConceptTS and baselines are trained three times per station, with concept proposals fixed across ConceptTS runs; the main experiments use Nsw = 24.The selected sub-window count balances accuracy and parameter count.
  • Bottleneck ablation: Test-time bottleneck blocking estimates each bottleneck’s contribution without retraining, avoiding information rerouting through the remaining concepts.A larger error increase indicates heavier reliance on the blocked bottleneck.
  • Bottleneck ablation: 43.193 µg/m3 MAE is the degradation from blocking the forecast sub-window bottleneck, compared with 1.412 µg/m3 for global forecast and 0.037 µg/m3 for look-back blocking.The forecast sub-window bottleneck is therefore the most influential in this setting.

B. Number of Sub-windows

ConceptTS performance improves as the forecast horizon is divided into more sub-windows, but the gains flatten while parameter count and training cost grow. The appendix also specifies how forecast-side concepts are generated and applied.

  • B. Number of Sub-windows: Larger Nsw values shorten each sub-window, enabling forecast concepts to describe finer-grained local trends and events.The comparison disables the residual channel and reuses the same concept sets across runs.
  • B. Number of Sub-windows: MAE decreases from 25.11 to 15.00 µg/m3 and RMSE from 41.37 to 24.03 µg/m3 as Nsw increases from 2 to 32.Both metrics decrease monotonically across the tested range.
  • B. Number of Sub-windows: Nsw increases concept-scoring heads and CEM embeddings linearly, raising training time and total parameter count; the main experiments therefore use Nsw = 24.This choice lies where the accuracy curve has largely flattened while parameter count remains manageable.
  • Concept organization: In future-aware mode, ConceptTS uses one look-back, N forecast sub-window, and one global-forecast bottleneck, with shared embeddings and per-sub-window predictor heads.The global bottleneck spans the whole forecast horizon.
  • Concept labeling: Forecast-side concepts cannot reference future PM2.5 and instead use meteorology, other pollutants, and calendar variables available over the forecast window.The forecast window’s PM2.5 values are masked at inference time.
  • Concept labeling: Each concept rule is an executable matches function with one boolean expression, defensive access, no imports, and a target positive rate between 5% and 60%.The prompt also requires diverse concept categories.

APPENDIX H TEST ON ELECTRICITY LOAD DATASET

On the LOF electricity-load dataset, ConceptTS trails every baseline. The appendix attributes this result to distribution shift and the model’s short-term concept vocabulary, which does not directly represent long-term periodicities.

  • Results: ConceptTS trails every baseline on the LOF-PC electricity-load dataset.The evaluation uses the future-aware setting with future exogenous features available at inference time.
  • Interpretation: The authors attribute the performance drop partly to a training–testing distribution shift that ConceptTS handles less gracefully than tree-based and Transformer baselines.This attribution is specific to the LOF evaluation.
  • Scope limitation: ConceptTS concepts describe short-term local behavior and cannot directly express seasonal or annual electricity-load patterns.The paper contrasts this limitation with PM2.5 forecasting, which depends mainly on short-term cross-channel context.
Loading 2608.21277v1…