PaperlayerView exact revision on arXiv

Source-linked AI summary

KITE: A Knowledge-Injected Temporal Encoder for Time-Series Event Forecasting

Jonas Petersen, Pratik Jawahar, Riccardo Maggioni, Gian-Alessandro Lombardi, Matteo Cozzi, Marek Gajewski, Federico Martelli, Camilla Mazzoleni, Philipp Christian Petersen

arXiv:2610.12738v1cs.LGcs.AI

TL;DR

Rare, high-cost events are difficult to forecast from sparse labels, although engineers possess knowledge that precedes them. KITE injects such knowledge as switchable inputs to a self-supervised event forecaster. Informative priors improve short-context results and can substitute for feature transfer, while redundant or unsuitable priors remain inert.

  • Problem

    Rare, high-cost events are sparsely labeled, and existing approaches handle only slices of the heterogeneous domain knowledge engineers hold about system variables and their relations.

    Source: §1 · p. 1

  • Method

    KITE routes semantic priors through switchable covariate and attention-bias mechanisms in one self-supervised backbone, each identity at its null.

    Source: §1 · p. 2, §3.3 · p. 3

  • Results

    In short context, the true-phase Occupancy clock raises h-AUROC from 0.762 to 0.866, while domain-invariant priors also improve applicable out-of-domain transfer cells.

    Source: §4.1 · pp. 3–4

  • Takeaways & Limitations

    A prior helps when it supplies information the observed window lacks, and KITE offers a switchable, attributable route to contextual conditioning.

    Source: §5 · p. 4

    The evidence is scoped to short context and two priors; several taxonomy categories remain conceptual, the attention-bias mechanism is undetermined, and the strongest transfer target is synthetic.

    Source: §4.2 · p. 4

Abstract

from arXiv · show

Forecasting rare, high-cost events in multivariate time-series systems is hard because their precursors are seldom observed, so a purely data-driven model has little to learn from. The engineers who run these systems, however, hold knowledge that precedes such events, information absent from the scarce labels. We call this a semantic prior and inject it into a self-supervised event forecaster through two switchable mechanisms, a covariate mechanism and an attention-bias mechanism, each the identity at its null, so one architecture serves every prior type with no per-dataset architecture search. A prior helps when it supplies information the observed window does not already contain: in a short context an informative prior beats no prior in-domain (pooled h-AUROC over the no-prior baseline: a periodicity clock +14%, p<0.001; a conservation residual +47% at a 0.25 label fraction), and a working prior transfers out-of-domain (+53%), substituting for feature transfer, while priors the window already recovers stay inert and the attention-bias mechanism shows no measurable gain. KITE is a switchable, attributable route to contextual conditioning, and a substrate for scaling toward cross-domain, pretrained time-series foundation world models that carry their governing structure explicitly.

1 Introduction

Rare, costly events leave purely data-driven forecasters with little labeled evidence, while engineers possess relevant knowledge before data is observed. The introduction frames KITE as a way to inject heterogeneous semantic priors through a single architecture rather than prior-specific designs.

1 Introduction

  • Rare, high-cost events are sparsely labeled, while engineers hold prior knowledge about system variables and their relations.Examples include variable identity, thresholds, periodicity, conservation laws, and causal edges.Source: §1 · p. 1
  • Existing approaches handle only slices of the problem, using a single always-on pathway entangled with each backbone.The authors identify the lack of a unified covariate and relational route across prior types.Source: §1 · p. 1
  • KITE treats domain knowledge as a switchable input to one self-supervised backbone, with covariate and attention-bias mechanisms that are identity at their null.The design supports prior-versus-no-prior ablations without per-dataset architecture search.Source: §1 · p. 2
  • The empirical organizing principle is that priors help when they supply information absent from the observed window, including in short-context and out-of-domain settings.The introduction also reports inert cases, including redundant static injection and graph bias.Source: §1 · p. 2

2 Related work

The related-work section contrasts pretrained forecasters without switchable prior pathways with prior-specific approaches whose pathways are always on. KITE combines several prior types as switchable inputs to one backbone.

2 Related work

  • Pretrained time-series models can transfer with little retraining, but forecast future values and expose no switchable pathway for a-priori domain knowledge.The section names both any-variate and variate-token designs, as well as industrial pretraining corpora.Source: §2 · p. 2
  • Prior injection is established, but cited approaches typically handle one type through an always-on pathway trained into the model weights.Examples include covariate-conditioned, graph-conditioned, physics-informed, periodicity-baked, and text-conditioned approaches.Source: §2 · p. 2
  • KITE unifies covariate, relational, conservation, and periodicity priors as switchable inputs, each identity at its null.This makes a prior’s contribution separable from the capacity of its pathway.Source: §2 · p. 2

3 Method

KITE organizes priors by when they are knowable and whether they are per-variable or relational, then routes them through a shared event-JEPA backbone. Covariates and attention bias provide switchable conditioning paths, each identity at its null.

3.1 The semantic-prior taxonomy

  • The taxonomy sorts twelve prior categories by knowability and whether they are per-variable or relational, and implements four for evaluation.The evaluated categories are periodicity, conservation, a structural descriptor, and causal-graph attention bias; static per-variable priors tested inert.Source: §3.1 · p. 2

3.2 Backbone: encoder, predictor, and event head

  • The backbone uses a permutation-invariant, cardinality-free variate-token encoder and a horizon-conditioned predictor, followed by a discrete-survival event head.Stage 1 self-supervisedly pretrains the encoder and predictor; Stage 2 freezes the encoder and trains the head on labels.Source: §3.2 · p. 2
  • The architecture emits a monotone event-probability surface by composing per-horizon hazards.Evaluation uses rank-based horizon-AUROC.Source: §3.2 · p. 2, Figure 1 · p. 3

3.3 Semantic-prior mechanisms

  • The covariate mechanism routes static descriptors to the encoder, future-known signals to the predictor, and past-only derived laws as covariates.The future-known periodicity clock is evaluated at t+∆t, while the attention-bias mechanism adds a scalar-gated driver-adjacency term to attention logits.Source: §3.3 · p. 3
  • Both mechanisms are identity at their null, and the graph-attention bias does not measurably help in the reported setting.Its measured effect is bounded to approximately 0.006 h-AUROC.Source: §3.3 · p. 3, Figure 1 · p. 3

4 Results and Discussion

In short context, informative priors improve performance over the prior-free control, and domain-invariant priors can substitute for feature transfer. The gains are bounded: redundant, misaligned, or present-state priors may not help, and the evidence has limited scope.

4.1 Claim A: an informative prior beats no prior in short context

  • The true-phase periodicity clock raises Occupancy silent-onset h-AUROC from 0.762 to 0.866 under a short w = 360 context.The gain depends on phase and horizon placement: a shuffled clock falls 0.346 below the true phase, while the clock at the predictor scores 0.866.Source: §4.1 · p. 3, Table 1 · p. 4
  • On BATADAL at w = 512, the conservation residual lifts the prior-free null to 0.877 (p = 1.2 × 10−5, n = 5) and still helps at a 0.25 label fraction (+0.207).A synthetic structural descriptor also confirms the static path works when a genuine structural prior exists (p = 0.030).Source: §4.1 · p. 4
  • None of 20 external-baseline arms beats KITE at BH-FDR 0.05, but the authors claim prior gains over KITE’s own null rather than superiority to foundation models.External arms sit at or below No prior on Occupancy and well above it elsewhere, with some point estimates above KITE’s.Source: §4.1 · p. 4, Table 1 · p. 4
  • A domain-invariant prior lifts the applicable transfer cells by +0.12 to +0.39, while a static per-variable prior does not (+0.006).A prior without pretraining beats transfer with no prior in eight of ten cells; the stronger claim that priors help more out of domain than in does not hold.Source: §4.1 · p. 4

4.2 Where priors do not help

  • Priors can be inert when the window already contains their information, when they are misaligned, or when present-state signals are asked to anticipate onset.The evidence is scoped to short context and two priors; several taxonomy categories remain conceptual, mechanism II is undetermined, and the strongest transfer target is synthetic.Source: §4.2 · p. 4

5 Conclusion

KITE routes semantic priors through switchable covariate and attention-bias mechanisms, with identity behavior at their null settings. Priors help when they add information missing from the observed window, while redundant static injection and the graph bias show no gain.

5 Conclusion

  • KITE’s conclusion is that informative priors help in short contexts and can substitute for feature transfer out of domain, while redundant static injection and graph bias do not help.The paper presents switchable prior conditioning as a substrate for pretrained, cross-domain time-series foundation world models carrying governing structure explicitly.Source: §5 · p. 4

A Mechanism details

  • Both mechanisms are identity at their null, but trained no-prior comparisons require separately trained arms because weights adapt to the prior.At initialization, zeroing the prior leaves the conditioned model exactly unconditioned.Source: A Mechanism details · p. 7
  • Static per-variable descriptors enter the encoder through an identity-at-initialization pathway, and are inert when the window already carries their information.The static path is present during Stage-1 pretraining so the frozen encoder can learn to use the prior.Source: A Mechanism details · p. 7
  • A future-known covariate enters the predictor at the forecast horizon, whereas a past-computable conservation residual detects an ongoing violation rather than anticipating onset.The future-known pathway is tested with a periodicity clock; the residual example is P_out − dL/dt.Source: A Mechanism details · p. 7
  • The attention-bias mechanism adds scaled causal driver-adjacency to query–key logits and is identity at β = 0.Future-known predictor inputs are not yet cardinality-compatible across datasets with differing covariate counts.Source: A Mechanism details · p. 8, Figure 2 · p. 8

B Dataset–prior alignment

The dataset–prior alignment section maps datasets to semantic-prior categories and documents several conditions under which priors do not help. The evidence includes redundant information, misaligned priors, present-state-only signals, and limited capacity.

B Dataset–prior alignment

  • Table 2 maps datasets to semantic-prior categories, while Table 3 adds implementation status to the extended taxonomy.The taxonomy distinguishes evaluated, implemented, inert, and conceptual categories.Source: B Dataset–prior alignment · p. 8, Table 2 · p. 8
  • Static injections survive 0/19 ablations, the full-context clock survives 0/3 cells, and graph-bias runs yield no positive cell in 19 when the bias is switched on.The static real prior is indistinguishable from a shuffled prior; earlier graph-correctness ablations at β = 0 were byte-identical.Source: B Dataset–prior alignment · p. 8
  • A wrong FFT-estimated period reverses the Occupancy win to −0.022, while the Weather negative control is null at −0.017 (p = 0.69).The paper attributes the period reversal to phase precession across the split.Source: B Dataset–prior alignment · p. 8
  • The conservation residual detects ongoing violations (+0.741 on BATADAL) but does not forecast onset; static-identity transfer is also null at +0.006 (p = 0.56).The onset-stratum values are BATADAL −0.053 and GECCO −0.136; the static prior is absorbable by the next linear layer.Source: B Dataset–prior alignment · p. 8

C Cross-domain transfer: full results

The transfer results test whether a prior can substitute for encoder feature transfer and whether its benefit amplifies out of domain. The adjacent taxonomy and illustration describe prior categories and route static, past-known, and future-known covariates differently.

C Cross-domain transfer: full results

  • A prior without pretraining beats transfer without a prior in eight of ten cells, by +0.027 to +0.381, and is null where the prior is known not to work.Figure 3 summarizes the in-domain and out-of-domain prior comparisons; Weather is null at +0.003 (p = 0.72).Source: Figure 3 · p. 9, C Cross-domain transfer: full results · p. 9
  • Only frozen-encoder BATADAL→SYNTH-CON passes both interpretability gates, with OOD−ID = −0.002 [−0.024, +0.020] (p = 0.82), a null amplification result.The setup holds the target fixed while varying encoder pretraining, and pairs prior arms from one cached Stage-1 checkpoint.Source: C Cross-domain transfer: full results · pp. 8–9
  • The competence-gradient result is suggestive only because its pre-registered cell fails G1 through a ceiling effect; the primary conservation target is synthetic.WADI and SWaT were not licensable in the available time, although the BATADAL source is real.Source: C Cross-domain transfer: full results · p. 9

D Extended semantic-prior taxonomy

  • The extended taxonomy lists twelve categories and marks whether each is evaluated, implemented, found inert, or a conceptual scaffold.Static per-variable priors were found inert in ablations; category #1 is evaluated through a structural descriptor on synthetic data.Source: D Extended semantic-prior taxonomy · p. 9, Table 3 · p. 9

E Covariate mechanism illustration

  • Static and past-known descriptors condition the encoder, while the Occupancy clock bypasses it and enters the predictor at the forecast horizon.The clock supplies future phase for a daily cycle that may not span the context window; separate routes support attribution to the conditioning mechanism.Source: E Covariate mechanism illustration · p. 10, Figure 4 · p. 10

F Proofs

The proofs establish KITE’s variable-order invariance and variable-count flexibility, then describe the identity-at-null graph bias and bound its measured contribution. Under the tested graph-bias settings, results remain undetermined rather than refuting the mechanism.

F.2 Permutation invariance and cardinality freedom

  • Reordering variables only reorders tokens and outputs, while symmetric pooling makes the final representation order-independent and parameters remain valid for changed variable counts.The same trained parameters run at C_test ≠ C_train without modification.Source: §F.2 · pp. 10–11

F.1 KITE tokenization

  • Shared patch embeddings tokenize each variable–patch pair, while temporal patches are ordered and variables form an unordered set.The token grid is fed to the encoder; no per-variable parameter is used.Source: §F.1 · p. 10, §F.2 · p. 10, Figure 5 · p. 11

F.3 Identity at β = 0

  • The graph mechanism adds β-scaled driver-adjacency to attention logits; at β = 0, the forward pass matches the unconditioned encoder at the same weights.The adjacency contributes no learned weights of its own; the prior enters through scalar β.Source: §F.3 · p. 11

G Graph mechanism: correctness, correction, and the bounded null

  • Graph correctness is defined by overlap in flagged target–driver pairs, excluding unflagged pairs from the score.The mechanism depends only on driver-adjacency, so graph changes that leave that matrix unchanged do not affect the model.Source: G Graph mechanism: correctness,… · pp. 11–12
  • With β = 2, graph-edge attention rises, but Weather adjacency variants differ by at most ±0.0013 h-AUROC and the mechanism remains undetermined with a small upper bound.Closing cross-variable paths costs −0.0011 h-AUROC (95% CI [−0.0063, +0.0040], n = 5); the paper estimates a bound of about 0.006, not a refutation.Source: G Graph mechanism: correctness,… · p. 12

H Implementation alignment

The implementation-alignment study re-ran idealised flagship variants against identity-at-null controls, isolating changes to the design. It finds costs from scalar-gated fusion, no detectable cost from removing the slot or adding a Stage-1 covariate, and distinct limits on what BATADAL demonstrates.

H Implementation alignment

  • Removing the per-variable slot and supplying the future-known covariate during Stage-1 pretraining produce no detectable change (all p ≥0.08), not demonstrated equivalence.The paper describes the slot-free encoder and does not adopt the Stage-1 covariate variant.Source: H Implementation alignment · p. 12
  • Scalar-gated fusion costs accuracy on every flagship, so the paper retains zero-initialized fusion.The companion study tested idealised variants against the identity-at-null control.Source: H Implementation alignment · p. 12
  • The static descriptor is concatenated with observed variables during Stage 1, while the future-known predictor covariate is kept out to preserve identical Stage-1 data across arms.The predictor is retrained at Stage 2; supplying its covariate during Stage-1 pretraining had no detectable cost.Source: H Implementation alignment · p. 12
  • The faithfully implemented scalar gate ends at magnitudes of 0.029 on Occupancy and 0.007 on BATADAL; wider-open seeds score higher, correlationally.The reported across-seed Pearson correlations are +0.89 and +0.81 (n = 10); the passage cautions this is not causal evidence.Source: H Implementation alignment · pp. 12–13
  • At g = 0, the gate gradient is proportional to descriptor usefulness if admitted; zero-initialized fusion instead admits the descriptor at full scale from the first step.Only the fusion weights are learned in the zero-initialized form.Source: H Implementation alignment · p. 13
  • Neither SIGReg nor its Sliced-Wasserstein variant is better on any flagship (all p ≥0.08), so the adoption rule retains VICReg.Both sketching objectives raise per-feature standard deviation from 0.089 to 0.52–0.64; Sliced-Wasserstein also lowers mean pairwise cosine from 0.956 to 0.734.Source: H Implementation alignment · p. 13
  • The slot-free encoder is invariant to variable reordering to about 2 × 10^-7 and runs at tested variable counts from 2 to 10, unlike the slot encoder.Across 50 test-set permutations, the slot-free model moves by exactly 0 h-AUROC; the slot model moves by at most 8 × 10^-4.Source: H Implementation alignment · p. 13
  • On BATADAL’s ongoing stratum, a zero-parameter residual threshold scores 0.921 versus the model’s 0.873, while both score 0.489 on onset; this supports detection, not anticipation.The result uses the same evaluation code, windows, strata, and re-run as the shipped row.Source: H Implementation alignment · p. 13

I Prior-only and generic-routing baselines

Prior-only and generic-routing baselines test what the prior contributes without sensors or an encoder, and where it helps when the architecture is held fixed. The two flagships differ: clock routing helps on Occupancy, while the BATADAL residual acts as a detection feature.

I Prior-only and generic-routing baselines

J Reproducibility

The reproducibility materials cover public real and synthetic benchmarks, typed priors, causal graphs, and the evaluation protocol. Configuration files, checkpoint manifests, and per-seed result tables are slated for release with the code.

J Reproducibility

  • Experiments use public real benchmarks and two synthetic datasets generated by released code from fixed seeds.The real datasets include Occupancy, BATADAL, ETTm1, Beijing air quality, SKAB, GECCO, Weather, CMAPSS, and Tennessee Eastman.Source: J Reproducibility · p. 14
  • Typed priors are provided for six benchmarks, while causal graphs are provided for all 22 datasets.The typed-prior datasets are BATADAL, Beijing air quality, ETTm1, GECCO, SKAB, and Weather.Source: J Reproducibility · p. 14
  • The primary metric is horizon-pooled h-AUROC, and prior-versus-null comparisons are paired by seed with Benjamini–Hochberg FDR control at 0.05.In-domain results use 5 to 10 seeds and transfer results use 5; configuration files, checkpoint manifests, and per-seed tables will be released with the code.Source: J Reproducibility · p. 14

Related papers

Loading 2610.12738v1…