ATEMPO: Amortized Transfer Entropy Model Per-lag Over-Time

• Fixed-width windows conflate whether a variable is informative with when its information arrives, potentially including many irrelevant intervening lags.The passage notes that extra inputs can increase estimation difficulty and obscure delayed signals, particularly in high-dimensional settings.Source: §1 · p. 1
• Estimating the TE(k, l, t) surface requires varying source depth, target-conditioning depth, and time, while existing estimators resolve individual surface points.Local windows improve temporal resolution but provide fewer samples for estimating TE.Source: §1 · p. 1
• ATEMPO conditions a shared score model on source depth, target-conditioning depth, and temporal position, sampling these configurations during training to estimate the full surface in one fit.Differencing the surface along source depth recovers the time-varying delay profile.Source: §1 · p. 2, Figure 1 · p. 2
• ATEMPO achieves the lowest mean relative error, 8.3 ± 3.3%, in one fit; other estimators require 18 fits, and it maintains perfect lag-detection accuracy across window lengths.On WEATHER-5K it reduces MSE by 7.25%; on three Neural Latent Benchmark tasks it improves mean R2 by 4.1% over the strongest competing baseline.Source: §1 · p. 2

Lag selection with transfer entropy

• Granger causality is limited to linear couplings, while time-delayed mutual information handles nonlinear interactions but cannot separate source contributions from shared-input effects.Transfer entropy conditions on the target’s own past; other approaches include momentary information transfer, TESPO, non-uniform embedding, and graph-based searches.Source: §2 · p. 3

Time-varying coupling

• Estimating time-resolved TE with local or sliding windows trades temporal resolution against the samples available in each window.ATEMPO conditions a shared estimator on position across the full recording and uses a local window only to aggregate estimates at evaluation.Source: §2 · p. 3

Estimating transfer entropy

• Nearest-neighbor TE estimation needs exponentially many samples as conditioning grows, variational bounds can loosen at high information, and cross-entropy estimators discretize continuous data.TENDE applies conditional diffusion scores to TE but fixes history configuration and window size before training, requiring one fit per hypothesis and window.Source: §2 · p. 3
• ATEMPO amortizes the TE query itself: source depth and conditioning depth specify the conditioning set, while time specifies temporal position, enabling recovery of TE(k, l, t) without refitting.The related-work passage states that existing methods do not vary all three axes.Source: §2 · p. 3

• Transfer entropy measures source-history information about the target beyond information in the target’s own past.The setup considers a single recording of a bivariate process and allows empty histories at depth zero.Source: §3 · p. 3
• Differencing adjacent TE values across source depth decomposes cumulative information into a delay profile, where a large entry at k indicates information arriving k steps later.A run of large entries indicates influence persisting beyond information provided by earlier lags.Source: §3 · p. 4
• The target is to recover TE(k, l, t) across source depth k, target-history depth l, and recording position t, then difference along k to obtain delay profiles.The target-history depth matters because too little conditioning may attribute existing target-history information to the source, while deeper conditioning increases estimation dimension.Source: §3 · p. 4
• Estimating one model per surface cell costs 2(K+1)(L+1)|I| fits, while widening local windows increases samples at the expense of temporal resolution.The conditioning dimension grows with source and target-history depths.Source: §3 · p. 4

4 METHOD

• ATEMPO estimates TE across source depth, target-conditioning depth, and position using one score model rather than a separate fit for each hypothesis.Diffusion-based TE estimation expresses the quantity through two score functions that differ in conditioning information and operates directly on continuous variables.Source: §4 · p. 4

4.1 TRANSFER ENTROPY FROM SCORE FUNCTIONS

• Transfer entropy is an expected divergence between conditional target distributions that differ in whether source history is supplied.This score-based formulation permits varying source and target-history depths without fixing them during training.Source: §4.1 · p. 4

4.2 AMORTIZING OVER THE TRANSFER-ENTROPY SURFACE

• ATEMPO shares a score model across conditioning configurations because changing source and target-history depths changes only the variables conditioned on, not the score-estimation problem.Depths are sampled per training example; temporal position is also supplied as a conditioning argument, replacing the grid of fits with one fit.Source: §4.1 · p. 5, §4.2 · p. 5
• Recording position conditions the shared score network, while the local window only determines which estimates are averaged and can be chosen at evaluation.This temporal sharing assumes variation across the recording can be represented by a shared function of observations and position.Source: §4.2 · p. 6

5.1 RESULTS

• 7.25% lower forecasting MSE than the WEATHER5K default across 5,672 stations; on 300 stations, 120 ATEMPO features beat the 240-feature default by 4.6%.The 300-station result also reports a 6.10% reduction using 240 features.Source: Figure 2 · p. 7, §5.1 · p. 7
• ATEMPO gives the most accurate TE estimates and best lag detection on synthetic data, using one fit for K=6 and L=3 versus 18 separate fits.The 18 fits cover the same source-depth and target-depth grid.Source: Table 2 · p. 8, §5.1 · p. 8
• ATEMPO’s five selected lags yield best or equal-best R² on three MC tasks, with mean R² 4.1% above Liang-Kleeman.Each method’s five selected lags were evaluated with the same ridge forecaster on held-out data.Source: Table 2 · p. 8, §5.1 · p. 8
• ATEMPO recovers every switched lag at every window length, while the best refit baseline falls from 0.92 to 0.68 detection as windows shrink from 1,000 to 125 samples.Results average across three time-varying synthetic systems.Source: Figure 3 · p. 8, §5.1 · pp. 8–9
• Across three time-varying systems, lag-detection accuracy correlates with oracle-relative forecasting MSE at r = [−0.82, −0.99, −0.92].Each correlation is a per-system least-squares fit across estimator and window-length results.Source: Figure 4 · p. 8, §5.1 · p. 9
• On MMTT-Bench, ATEMPO recovered the planted lag for five of six lags and zero lags for null descriptions; correct-text TE was 0.52 versus 0.14 nats, with 41% MSE improvement.It equaled TENDE and beat all other baselines except PCMI in comparable processing time.Source: §5.1 · p. 9
• At full-recording fit, lag-only amortization detects moving lags in 0.58–0.64 of windows; adding time conditioning reaches 1.00 at every evaluation-window length.The ablation separates amortization over lag from conditioning on temporal position.Source: §5.1 · p. 9
• Exact TE error evaluation is limited to synthetic, approximately Gaussian systems with one informative dimension; real-data utility is assessed through alignment or forecasting because ground-truth TE is unavailable.The paper also notes that large conditioning dimensionality requires sufficient samples and model capacity.Source: §5.1 · p. 10

6 CONCLUSION

• ATEMPO achieves the best mean TE estimation and lag-detection accuracy among eight baselines.The recovered delays reduce WEATHER5K forecasting MSE by 5–7% and improve mean R2 by 4.1% across three Neural Latent Benchmark tasks.Source: §6 · p. 10
• One shared estimator conditions on lag, target-conditioning depth, and temporal position to estimate TE(k, l, t).Differencing the resulting surface along k recovers a time-varying delay profile.Source: §6 · p. 10
• ATEMPO separates how much history to expose from where within that history useful information lies.Source: §6 · p. 10

AI USE STATEMENT

• Generative AI checked implementations of TE-estimator baselines without open-source code and reviewed synthetic dataset generation.Source: AI USE STATEMENT · p. 11
• Generative AI created some Figure 1 diagrams and formatting code for Figure 2, and helped clean and comment the codebase.Source: AI USE STATEMENT · p. 11
• The authors state that humans made manuscript changes, verified generated code, reviewed AI-assisted work, and take responsibility for the final content.Source: AI USE STATEMENT · p. 11

ETHICS STATEMENT

• The evaluation uses synthetic data and public benchmarks; the NLB dataset includes animal-derived data collected under institutionally approved procedures.Source: ETHICS STATEMENT · p. 11
• For high-stakes applications, the authors advise against relying on TE estimates alone and recommend human-in-the-loop safeguards.They also advise considering privacy, fairness, and domain-specific requirements when applying the method to sensitive data.Source: ETHICS STATEMENT · p. 11
• The authors state that the method, experiments, and figure and table generation are reproducible and available on GitHub.The repository includes per-dataset reproduction scripts, a plotting notebook, and a script for running ATEMPO on new data.Source: REPRODUCIBILITY STATEMENT · p. 11

A DATASET DETAILS

• The synthetic suite varies coupling nonlinearity, dimensionality, and delay count, and includes time-varying-lag systems.Seven non-time-varying datasets are stationary; five additional datasets are used for lag detection and forecasting.Source: A DATASET DETAILS · p. 15, Table 4 · p. 15

B.1 DATASETS

• WEATHER5K lag selection fits 25 ordered variable pairs using 300 stations, then evaluates selected lags on the subset and all 5,672 stations.Lag selection uses training data only; MSE is measured on held-out 2023 data, and iTransformer accepts the non-consecutive selected lags.Source: §B.1 · pp. 15–16
• Neural Latent Benchmark tasks use distinct neural and behavioral signals, train/validation partitions, and task-specific lag-selection setups across MC_RTT and MC_Maze.The tasks include velocity forecasting, neural decoding, and movement-onset hand-velocity decoding.Source: §B.1 · p. 16
• Synthetic forecasting uses chronological 70/15/15 train, validation, and test splits, with ridge and MLP predictors using target history and the detected source lag.Each system/window pair is averaged over three seeds, except ATEMPO uses one model across window sizes.Source: §B.1 · p. 16
• Three switching systems change their planted delay from lag 2 to lag 5 halfway through 8,000 samples and are evaluated across 8–64 windows.The systems cover linear, mean-blind squared, and effect-size-matched squared coupling; detections are scored against the planted lag at each window center.Source: §B.1 · pp. 16–17
• MMTT-Bench plants text-to-signal delays from the benchmark's native one-step relationship and tests correct, incorrect, irrelevant, and time-shuffled text conditions.Detection uses four PCA dimensions, while forecasting uses sixteen; the latter predicts from recent target values and text embeddings at selected lags.Source: §B.1 · p. 17

B.2 BASELINES

• The baseline suite includes estimators that differ in how they estimate TE or lag profiles, and Liang-Kleeman and DTW are excluded from ground-truth TE error comparisons because they do not estimate TE directly.NJEE uses a difference of conditional entropies; MINE estimates TE as a difference of two mutual-information estimates, while CCMI estimates conditional mutual information directly.Source: §B.2 · pp. 17–19

Definitions

• A delay attribution is a decomposition when its nonnegative shares sum to total TE; the set function scores each subset of source delays conditional on the target history.Adding a delay to a subset changes the set-function value by its conditional information about the target.Source: C ATTRIBUTION OF TRANSFER ENTROPY… · p. 19

Attribution choice

• When adjacent delays repeat the same information, isolation-based attribution assigns each delay 1 nat, while conditioning on all other delays assigns both 0, although total TE is 1 nat.The example illustrates why these alternatives are not decompositions of total TE.Source: C ATTRIBUTION OF TRANSFER ENTROPY… · p. 19
• The recency-ordered increment and Shapley value both decompose total TE, but only the increment preserves precedence needed to interpret when a source delay acts.The increment uses the recency ordering; Shapley averages over all orderings and divides redundant information equally across repeated delays.Source: C ATTRIBUTION OF TRANSFER ENTROPY… · p. 19

Estimation consequences

• Evaluating Eq. 14 requires 2^K−1 coalitions per delay; at the depths used here (K = 336 on WEATHER-5K), no sampling scheme attains the precision a delay estimate requires.Source: C ATTRIBUTION OF TRANSFER ENTROPY… · p. 19
• 0.008 against 0.011 nats: ATEMPO’s delay profile is less noisy than its TE surface, while per-cell fits produce a noisier profile than surface (0.026 against 0.019 nats).Adjacent-cell error correlation is 0.72 for ATEMPO versus 0.05 for per-cell fits; the delay-profile error standard deviation is three times lower with ATEMPO.Source: C ATTRIBUTION OF TRANSFER ENTROPY… · p. 20

D SYNTHETIC DATASET RESULTS

• ATEMPO has the lowest mean relative TE error across the seven datasets with known TE, while NPEET and NJEE tend to underestimate and CCMI tends to overestimate.The signed-error comparison is especially noted for multidimensional systems.Source: D SYNTHETIC DATASET RESULTS · p. 21, Table 6 · p. 21, Table 7 · p. 21
• Lag-recovery accuracy counts a success only when the true lag has the highest ΔTE estimate; time-varying results use one seed and eight windows.A score of 6/8 means the true lag produced the maximum TE in six windows.Source: D SYNTHETIC DATASET RESULTS · p. 21, Table 8 · p. 21

D.1 ABLATION

• ATEMPO detects the true lag in every evaluated dataset and window across three synthetic time-varying systems.The ablation reports degraded detection without time conditioning and with per-window training, particularly as window size decreases.Source: §D.1 · p. 21, Table 9 · p. 22
• Across the time-varying datasets, ATEMPO’s selected lags yield the lowest forecasting MSE and remain robust to window size.Per-window ablations match this performance only at the largest window; removing time conditioning performs worst.Source: §D.1 · p. 22, Table 10 · p. 22

MMTT-Bench lag recovery

• ATEMPO and TENDE each miss only the one shifted lag, narrowly behind PCMI, which detects all planted lags.The lag-recovery results are reported per planted delay alongside shuffled-text null rejection and relative cost.Source: E MMTT-BENCH FULL RESULTS · p. 23, Table 11 · p. 23

MMTT-Bench by text condition

• ATEMPO estimates the strongest information for correct text, weaker nonzero information for incorrect text, and null information for irrelevant and shuffled text.The benchmark’s forecasting comparison reports gains only when adding a lag for correct text; none of the three controls benefits from a lag choice.Source: E MMTT-BENCH FULL RESULTS · p. 23, Table 12 · p. 23

F WEATHER5K FULL RESULTS

• 3.8% with ridge and 4.6% with iTransformer: ATEMPO’s 120-feature inputs improve over the 240-feature WEATHER-5K baseline recent(48h).The ridge comparison uses a 300-station subset; its improvement is reported as comparable to iTransformer’s and robust across models.Source: Table 13 · p. 24, F WEATHER5K FULL RESULTS · p. 26, Table 14 · p. 27
• The WEATHER-5K comparison limits other baselines to 67 candidate lags because TENDE is computationally intractable on even the 300-station subset; both 67- and 336-candidate searches are run.The smaller candidate search is strictly easier than searching all 336 lags.Source: F WEATHER5K FULL RESULTS · p. 26

F.1 PHYSICAL ANALYSIS OF SELECTED LAGS

• ATEMPO is the only estimator reported to spread selected lags across all defined weather-pattern categories, while boundary lags are the least valuable band in the removal ablation.The boundary-band result is based on the smallest positive cost across the reported band-removal columns.Source: §F.1 · pp. 28–29, Table 15 · p. 28, Table 16 · p. 28

G ADDITIONAL MODELS

• The additional-model tests compare 48-hour default, random, ATEMPO, and CCMI selections against the full 336-hour window at one seventh the input size.The experiments ask whether rankings persist across architectures and input layouts, and whether a pretrained foundation model benefits similarly.Source: G ADDITIONAL MODELS · p. 29

G.1 MODEL DETAILS

• Across trained forecasters except Pyraformer, selector rankings match the main comparison; iTransformer–DLinear ranking correlations are 0.82 at 48 lags and 0.88 at 24.Chronos-2 is an exception to the selected-hours comparison: its full-window forecast gains 18.0% over any 48-hour selection.Source: §G.1 · p. 30, Table 18 · p. 30, Table 17 · p. 30

H NLB FULL RESULTS

• At five additional lags, NLB velocity reaches R2 = 0.812 versus 0.794 for both recent-lags and cross-correlation selections.The deeper ATEMPO picks are 120 and 160 ms, or 320–360 ms before the forecast target.Source: H NLB FULL RESULTS · p. 31
• At five selected lags, NLB decoding reaches R2 = 0.47, near the all-20-lag score of 0.48; ATEMPO’s advantage over the next-best estimator exceeds paired standard errors on both decoding tasks.Cross-correlation is strongest at one to three additional lags, while ATEMPO performs best among reported selectors at five.Source: H NLB FULL RESULTS · p. 31
• On MC_Maze, ATEMPO’s phase-conditioned ranking reaches R2 = 0.619 at five lags, while five evenly spaced lags score 0.626.The results suggest precise lag choices matter less once redundancy between neighboring bins is avoided on this task.Source: H NLB FULL RESULTS · p. 31
• These neural results compare input-selection strategies using smoothed spike counts, not state-of-the-art decoding under the benchmark’s inferred firing rates and fixed lead.The benchmark uses an approximately 140 ms neural lead.Source: H NLB FULL RESULTS · p. 31
• Incremental TE measures information beyond shallower lags, so its largest value need not identify the best single input for smooth, redundant signals.The reported results show this mismatch in cross-correlation’s single-lag advantage and the strength of evenly spaced MC_Maze lags.Source: H NLB FULL RESULTS · p. 32

I.1 TIME SERIES VARIABLE SELECTION

• For variable selection, ATEMPO ranks each candidate using one fit of its past 24 lags against the target one step ahead, conditioned on the target’s past 24 lags.The ranking measures information added beyond the target’s history and is compared across five benchmarks.Source: §I.1 · p. 32
• On Electricity, ridge with all 320 series scores R2 = 0.824 versus 0.882 for target-only; ATEMPO is the strongest selector at budgets 1, 2, and 50.Figure 5 plots median ridge test R2 against selected-series count across all five datasets.Source: §I.1 · pp. 32–33, Figure 5 · p. 33
• On Traffic, ATEMPO-selected subsets underperform target-only at smaller budgets amid test-period variance shifts absent from training.Four of ATEMPO’s 50 top-five picks across ten targets have test-to-train standard-deviation ratios above 2, versus one for ChronoEpilogi and none for correlation or GroupLasso; the authors treat this as a distribution-shift robustness finding, not a selection-quality comparison.Source: §I.1 · p. 33