PaperlayerView exact revision on arXiv

Source-linked AI summary

MercerFlow: Flow Matching in a Kernel-Induced Latent Space for Probabilistic Forecasting

Ilya Kuleshov, Egor Serov, Alexey Zaytsev

arXiv:2610.06039v1cs.LGcs.AI

TL;DR

With a GP prior, the choice of linear latent map matters because the map must decorrelate the prior structure. MercerFlow uses the prior kernel's eigenbasis for MLP-based conditional flow matching; its MLP codecs reduce memory and epoch time versus TSFlow at matched batch size.

  • Problem

    Under a GP prior, a latent map must decorrelate the prior structure, but Fourier and DCT best diagonalise OU-like priors and PCA depends on the training path.

    Source: §1 · p. 2

  • Method

    MercerFlow projects the joint context and forecast window into the GP kernel's Mercer eigenbasis and learns conditional flow matching with an MLP.

    Source: §1 · p. 2

  • Results

    At matched batch size, MLP codecs use about 4.7× less peak GPU memory and 3.5×–4.4× less time per epoch than TSFlow.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The Mercer basis is a data-free codec that works with non-OU priors, unlike fixed harmonic transforms that best diagonalise OU-like priors.

    Source: §1 · p. 2

    The primary benchmark runs adopt the stationary exponential (OU) kernel.

    Source: §2.1 · p. 3

Abstract

from arXiv · show

Recent work has shown that probabilistic flow matching for time series forecasting benefits from a data-matched prior. The resulting prior introduces local correlations, which a sequential architecture usually absorbs: a recurrent neural network (RNN), a structured state-space model (S4), or a Transformer. However, such a backbone costs GPU memory and time per epoch. A cheaper alternative is MLP-based latent-space flow matching: embed the time series via an invertible map to a single latent vector and learn the flow there, so a tabular MLP can treat the series as a set of features. The relationship between the prior and the choice of linear latent map is understudied in conditional flow matching (CFM) forecasting, yet we found it strongly affects performance. Fixed transforms such as Fourier or discrete cosine (DCT) are only well-conditioned for Ornstein--Uhlenbeck priors, while a principal-component (PCA) map fit to the data is a strong but training-set-dependent reference sensitive to train--test shift. Instead, we propose to use the Mercer eigenbasis of the prior kernel: it diagonalises the centred covariance exactly, decouples from training data, and adapts to non-stationary and periodic priors. On five GluonTS benchmarks (ETTh1, ETTh2, Weather, Electricity, Traffic) under a shared protocol with TSFlow, the resulting MLP matches or beats it on CRPS at about $4.7\times$ less training memory and $3.5\times$--$4.4\times$ less time per epoch.

1 INTRODUCTION

A GP prior gives flow-matching forecasts useful local structure, but correlated time coordinates make an MLP spend capacity on smoothness the prior already encodes. MercerFlow studies kernel-matched linear codecs for an MLP backbone and finds that codec effects depend on the optimizer.

1 INTRODUCTION

  • A GP prior can bring the flow closer to the target, but its local temporal correlations make a global MLP spend capacity on smoothness already encoded by the prior.TSFlow uses a bidirectional state-space backbone to absorb those correlations, at greater memory and epoch-time cost.Source: §1 · p. 1
  • The Mercer eigenbasis makes coefficients uncorrelated under the corresponding GP, exactly for the centred prior but approximately for the conditional law and training interpolant.It is data-free and supports non-OU priors, whereas Fourier and DCT best diagonalise OU-like priors and PCA fits only the training path.Source: §1 · p. 2
  • Under Adam, decorrelating codecs have lower first-layer κ(DAD) and better accuracy than Raw on challenging datasets, while their advantage disappears under SGD.The proposed explanation is that Adam's coordinate-wise Jacobi scaling depends on codec-specific input diagonals, unlike Euclidean SGD.Source: §1 · p. 2
  • The study compares Mercer, Fourier, DCT, PCA, and Raw codecs on five GluonTS benchmarks under matched architecture, prior, and protocol, using paired t-test ranks.The introduction frames the comparison around whether codec choice interacts with the optimizer and balances predictive accuracy with computational cost.Source: §1 · p. 2
  • Against TSFlow, MLP codecs use about 4.7× less peak GPU memory and 3.5×–4.4× less time per epoch at matched batch size.The authors present this as making MLP flow matching in linearly projected latents a practical alternative to sequential S4 backbones.Source: §1 · p. 3

2 METHOD

MercerFlow conditions a GP prior on the observed past, encodes joint windows in the kernel eigenbasis, and trains a lightweight MLP flow in coefficient space.

2.1 CONDITIONAL GP PRIOR

  • The conditional GP prior supplies future-window source samples given the observed context; benchmark runs use a stationary exponential OU kernel and select its length scale by validation likelihood.Diagonal jitter is used for numerical stability.Source: §2.1 · p. 3

2.2 LINEAR CODEC ON THE JOINT WINDOW

  • A shared orthonormal codec projects joint windows; Mercer diagonalizes the prior, DCT approximates OU-like bases, Fourier uses harmonic modes, and PCA is training-dependent and shift-sensitive.The Mercer coefficient covariance equals the kernel eigenvalue matrix, so prior decorrelation is exact.Source: §2.2 · p. 4

2.3 FLOW MATCHING IN COEFFICIENT SPACE

  • Independent pairing with the conditional GP source yields straight paths without optimal-transport overhead, while an MLP learns the coefficient-space velocity and orthogonal codecs preserve loss magnitude.At inference, the learned velocity is integrated with an Euler solver before decoding the forecast horizon.Source: §2.3 · pp. 4–5

2.4 WHY THE ROTATION CAN MATTER: ADAM’S DIAGONAL

  • Under Adam’s Jacobi scaling, Mercer gives κ(DAD)=1 while Raw’s condition number grows with prior length scale; Euclidean SGD instead preserves identical trajectories across orthonormal codecs.AdamW is not rotation-equivariant because its coordinate-wise scaling changes with the input basis.Source: §2.4 · pp. 5–6

3. Fourier and DCT: The respective transformation bases satisfy the asymptotic bounds

The section relates codec conditioning to spectral structure: bounds loosen for low-frequency-concentrated spectra, while observed codec differences depend on the optimizer.

3. Fourier and DCT: The respective transformation bases satisfy the asymptotic bounds

  • The asymptotic bound loosens when the spectral density concentrates at low frequencies; the accompanying table summarizes second-moment structures and condition numbers across five codecs.Source: §3 · p. 6
  • Within 0.2%, CRPS ties across codecs on synthetic stationary OU paths, where empirical κ(DAD) matches the predicted ordering.These controlled paths reproduce behavior observed on easy real datasets.Source: §3 · p. 6
  • Under AdamW, first-layer κ(DAD) separates Raw from decorrelating codecs on challenging benchmarks, whereas SGD eliminates performance discrepancies across codecs.The reported pattern aligns with Proposition 1.Source: §3 · p. 6

3 EXPERIMENTS

Experiments compare five codecs under a shared architecture, prior, and protocol, then assess forecast quality, efficiency, prior alignment, and robustness. MercerFlow achieves the strongest reported overall ranks while using less training memory and time per epoch than the sequential TSFlow backbone.

3 EXPERIMENTS

  • Experiments use five GluonTS benchmarks, channel-independent series, and fixed context and forecast lengths of 192.All five codec variants share the MLP architecture, joint-window conditioning, and GP prior; probabilistic evaluation uses 100 forecast trajectories per test window.Source: §3 · p. 7

3.1 MAIN RESULTS: FORECASTING QUALITY AND EFFICIENCY

  • Across five benchmarks, MLP codecs achieve mean CRPS ranks of 1.00–1.80 vs 5.80 for TSFlow, while MercerFlow ranks 1.00 with 5/5 wins on CRPS and MAE.Table 1 summarizes predictive performance across ETTh1, ETTh2, Weather, Electricity, and Traffic.Source: Table 1 · p. 7, §3.1 · pp. 7–8
  • At batch size 512, the MLP uses 1.5 GB vs 7.1 GB peak GPU memory and trains 3.5×–4.4× faster per epoch than the sequential state-space model.The MLP has approximately 55k parameters, compared with 186k for the sequential model.Source: Table 2 · p. 7, §3.1 · p. 8

3.3 EXTENDED ANALYSES AND APPENDIX OVERVIEW

  • Matched-budget Nesterov SGD ties all five codecs within one standard deviation on ETTh1 and ETTh2, whereas AdamW breaks rotational equivalence.Under Adam, first-layer conditioning distinguishes Raw from decorrelating codecs on harder benchmarks.Source: §3.3 · p. 8
  • At η = 10^-1, Raw’s CRPS explodes to 1.22 on ETTh1 and 0.246 on Traffic, while all decorrelating codecs remain stable.Source: §3.3 · p. 8
  • On synthetic OU paths, Mercer achieves κ(DAD) = 1.00; misspecifying the length scale by 10× degrades CRPS by only 0.006.Straight probability paths saturate calibration in 4–10 integration steps, outperforming score-based diffusion at 128 steps.Source: §3.3 · p. 8
  • Across diagonal jitter δ ∈ [10^-4, 1.0], damping the prior cannot rescue an unrotated basis: Raw stays ill-conditioned at κ ≈ 10^5 (Appendix F.3, Table 20).Source: §3.3 · p. 8

3.2 WHY MERCER? CODEC UNIVERSALITY AND PRIOR ALIGNMENT

  • On Traffic, the periodic prior raises DCT’s κ(DAD) to 5.2 × 10^4 versus 1.4 × 10^3 for Mercer, while Mercer recomputes its basis from the chosen kernel.Fourier faces boundary-leakage ill-conditioning on stationary long-horizon series, while PCA’s condition number rises from 481 to 1290 on held-out windows.Source: §3.2 · p. 8, Figure 2 · p. 9

4 CONCLUSION

The conclusion argues that aligning latent coordinates with the prior kernel can give a lightweight MLP stable preconditioning while matching or surpassing sequential models on calibration. Its scope is limited by idealized optimizer assumptions and evaluations on univariate channels with one fixed-width MLP.

4 CONCLUSION

  • Mercer coordinates align the latent representation with the prior kernel, allowing an MLP to match or surpass sequential state-space models on probabilistic calibration while reducing memory and epoch time.The reported reductions are 4.7× in peak GPU memory and 3.5×–4.4× in per-epoch training time.Source: §4 · p. 9

LIMITATIONS

  • The analysis idealizes Adam’s coordinate rescaling as exact Jacobi preconditioning and assumes rotation-invariant SGD initialization.Evaluations are also restricted to univariate channels and a single fixed-width MLP backbone.Source: LIMITATIONS · p. 9

REPRODUCIBILITY

  • The authors state that their anonymous repository is being published for reproducibility and that all datasets are freely available.They report that most training runs take only a few minutes.Source: REPRODUCIBILITY · p. 10

AI USE STATEMENT

  • The authors report using AI tools for literature review, code maintenance, and LaTeX drafting, with co-author review of cited papers, code, and text.They state that two co-authors manually reviewed the code and that most of the text was rewritten after review.Source: AI USE STATEMENT · p. 10

A RELATED WORK

  • Prior work studies generative models in orthonormal latents, but the authors identify the interaction between forecasting GP priors and latent projections under flow matching as unexamined.They compare five full-order codecs using one MLP and one conditional flow-matching loss.Source: A RELATED WORK · p. 12

B.1 PROOF OF PROPOSITION 1 (SGD INVARIANCE)

  • Under Euclidean SGD with rotation-invariant initialization, orthogonal codecs yield identical training-loss trajectories and decoded probability flows.The proof uses an isometric latent transformation that preserves the conditional flow-matching regression objective.Source: §B.1 · p. 13

B.2 SGD CONTROL ON REAL BENCHMARKS

The matched-budget Nesterov SGD control supports the predicted codec equivalence on ETTh1 and ETTh2: all five codecs fall within one standard deviation on both datasets. On these two datasets, the shared SGD level is worse than AdamW.

B.2 SGD CONTROL ON REAL BENCHMARKS

  • The control uses untuned Nesterov SGD with momentum 0.9 and learning rate 10−3, changing only the optimizer from the AdamW runs.The comparison uses a matched training budget on ETTh1 and ETTh2.Source: §B.2 · p. 14, Table 3 · p. 14
  • All five codecs land within one standard deviation of each other on ETTh1 and ETTh2 under Nesterov SGD, while the shared SGD level is worse than AdamW on both.On ETTh1, Mercer’s test CRPS is 0.2255 with SGD versus 0.188 with AdamW, about 20% above AdamW.Source: §B.2 · p. 14, Table 3 · p. 14

B.3 PROOF OF PROPOSITION 2 (JACOBI PRECONDITIONING)

The proposition evaluates first-layer conditioning after Jacobi scaling under the centered prior, with the condition number depending on how each codec represents that covariance. The appendix contrasts exact Mercer diagonalization with the behavior of Fourier, DCT, PCA, and Raw, and separately defines evaluation metrics.

B.3 PROOF OF PROPOSITION 2 (JACOBI PRECONDITIONING)

  • The proposition models Adam’s input-side rescaling as Jacobi preconditioning of the centered-prior second-moment matrix, and κ(DAD) measures the resulting Gram condition number.The uncentered conditional interpolant used in training is outside the proposition’s claim.Source: §B.3 · p. 14
  • For OU priors, Fourier’s κ(DAD) grows with length scale, from 13.4 on Traffic at ℓ = 7 to 268.5 on ETTh1 at ℓ = 336, while DCT remains in [1.90, 2.12] across benchmarks.The analysis attributes Fourier’s growth to low high-frequency spectral density and DCT’s stability to its reflective-boundary approximation.Source: §B.3 · p. 15
  • PCA’s condition number is 8.15–46.98 on the centered prior because its training-covariance basis is generally misaligned with the prior kernel; it is 1 only on its own empirical-covariance population.Source: §B.3 · p. 15
  • Mercer gives κ(DAD) = 1 on every dataset under the centered prior, whereas DCT is 1.9–2.1, PCA 8–47, Fourier ranges from tens to hundreds, and Raw ranges from 1.96 × 10^2 to 1.72 × 10^5.These condition numbers are for the centered prior population, not the conditional training inputs.Source: §B.3 · p. 15, Table 5 · p. 16

C.1 EVALUATION METRICS

  • The evaluation section defines normalized CRPS as mean weighted quantile loss over nine quantiles and evaluates deterministic forecasts using the empirical median’s unnormalized MAE.Both metrics are averaged across rolling test horizons.Source: §C.1 · p. 16

C.2 EXPERIMENTAL PROTOCOL

The experimental protocol uses a shared OU GP prior and a matched lightweight MLP for five codecs, compared with TSFlow on the same forecast window. The MLP’s post-first-layer time modulation satisfies the structural condition used in the SGD equivariance proof.

C.2 EXPERIMENTAL PROTOCOL

  • All channels are treated as independent univariate series, and the OU prior length scale is selected per dataset by validation likelihood and shared across the five MLP codecs.The context and forecast horizons are each 192 steps, giving a joint window of 384; source and target trajectories are paired independently.Source: §C.2 · p. 16
  • The five codecs use the same square orthonormal joint-window map and a matched two-layer MLP with width 64 and about 55k parameters, while TSFlow uses a 186,833-parameter bidirectional S4 backbone on the same window and prior.The evaluation uses normalized CRPS, median-trajectory MAE, 20 Euler steps, 100 sample trajectories, and seeds 42–48.Source: §C.2 · pp. 16–17

C.3 ADALN ARCHITECTURAL PLACEMENT

  • The MLP applies adaptive flow-time modulation after its first linear layer, with scale and shift generated from the flow-time embedding, so the hidden operation acts entrywise as required by Proposition 1.Source: §C.3 · p. 17

C.4 DATASET PROPERTIES AND VALIDATION LIKELIHOOD

Validation likelihood favors different kernel structures across benchmarks. Under a periodic prior, Mercer coordinates maintain conditioning and forecasting quality where fixed harmonic codecs degrade.

C.4 DATASET PROPERTIES AND VALIDATION LIKELIHOOD

  • Validation selects periodic, RBF, or OU kernels by benchmark; primary runs use domain-aligned OU length scales and shared jitter ε = 10^-4.Candidate kernels are scored by validation-slice Gaussian negative log density; the primary-run OU parameters are summarized in Table 8.Source: §C.4 · p. 17, Table 6 · p. 17
  • Under the weekly periodic Traffic prior, DCT’s condition number rises from 712 under OU to 2.45 × 10^4, while MercerFlow’s is 1.18 × 10^3.The periodic prior uses p = 168, ℓ = 0.1, and ε = 0.05.Source: §C.4 · pp. 17–18, Table 9 · p. 18
  • At η = 10^-3, the periodic prior improves CRPS across all five codecs, and MercerFlow achieves the best Traffic forecast at 0.0821 CRPS.At η = 10^-1, DCT reaches 0.1641 CRPS and Fourier 0.7773, while MercerFlow maintains 0.0951 CRPS.Source: §C.4 · p. 19, Table 10 · p. 19
  • Fourier’s boundary leakage raises its condition number to 268.5 on ETTh1 and 471 under conditional draws, versus 1.00 and 14.8 for MercerFlow.The passage attributes the instability under learning-rate stress to amplified high-frequency coordinates along leakage directions.Source: §C.4 · p. 19

C.5 DATASET HARDNESS AND FIRST-LAYER CONDITIONING

The section relates prior-mean residuals and first-layer conditioning to forecast difficulty, then introduces comprehensive per-dataset predictive results.

C.5 DATASET HARDNESS AND FIRST-LAYER CONDITIONING

  • Residual MSE is 0.33 on ETTh2 and 0.44 on Weather, classified as easy; ETTh1, Electricity, and Traffic residuals range from 1.00 to 1.46 and are classified as hard.Hardness is measured using the frozen analytical OU conditional mean on the held-out final 15%.Source: §C.5 · p. 19, Table 11 · p. 19
  • PCA’s condition number rises from 115 to 285 on Electricity and from 481 to 1290 on Traffic between train and test, while MercerFlow maintains consistent preconditioning across splits.PCA’s basis is fit on data windows, whereas MercerFlow’s is derived analytically from the prior kernel.Source: §C.5 · p. 20, Table 12 · p. 20
  • On hard datasets, conditioning and test CRPS have mean within-dataset Spearman ρ = 0.79 on held-out splits; on easy datasets, ρ = −0.08.The authors attribute the difference to the prior mean already accounting for nearly all predictive capacity on easy datasets.Source: §C.5 · p. 20, Table 13 · p. 20

D FULL PREDICTIVE PERFORMANCE

  • This section provides comprehensive per-dataset probabilistic and point-forecasting results across all five GluonTS benchmarks.The main text condenses benchmark evaluations into summary ranks and computational footprint.Source: D FULL PREDICTIVE PERFORMANCE · p. 20

D.1 PROBABILISTIC FORECASTING PERFORMANCE (CRPS)

MercerFlow ranks first among conditional flow-matching models on all five benchmarks for probabilistic forecasting. It also maintains competitive point accuracy and calibrated intervals, while latent codecs help at extended horizons.

D.1 PROBABILISTIC FORECASTING PERFORMANCE (CRPS)

  • Rank 1 across all five benchmarks: MercerFlow consistently matches or outperforms the other conditional flow-matching models on CRPS.Table 14 reports normalized test CRPS; parenthetical ranks use paired two-sided t-tests on shared seeds.Source: §D.1 · p. 20, Table 14 · p. 21

D.2 POINT FORECASTING PERFORMANCE (MAE)

  • MercerFlow maintains competitive point accuracy across datasets while preserving calibrated prediction intervals, with top rank 1.00 among generative models.Deterministic baselines can achieve strong MAE on smooth benchmarks but show severe distributional degradation.Source: §D.2 · p. 21, Table 15 · p. 21

D.3 SENSITIVITY TO EXTENDED FORECASTING HORIZONS

  • At horizon Lf = 720 on Electricity, RAW MAE degrades to 206.9 ± 15.6, compared with 175.7 ± 5.6 for DCT; orthogonal latent codecs maintain calibration as windows lengthen.The comparison extends the forecast window from Lf = 192 while keeping context length Lp = 192.Source: §D.3 · p. 21, Table 16 · p. 22

E OPTIMIZER DYNAMICS AND STABILITY

Under severe learning-rate stress, the unrotated Raw baseline collapses on harder benchmarks while decorrelating codecs remain stable. At elevated learning rates, MercerFlow’s training loss also converges monotonically where Raw is unstable.

E OPTIMIZER DYNAMICS AND STABILITY

  • At η = 10^-1, Raw CRPS explodes to 1.22 on ETTh1, 0.065 on Electricity, and 0.246 on Traffic, while decorrelating codecs avoid numerical instability.The comparison uses matched mid-run evaluation checkpoints across the five benchmarks.Source: E OPTIMIZER DYNAMICS AND STABILITY · p. 22
  • Raw’s first-layer condition number is two to four orders of magnitude larger than that of orthonormal codecs under the stressed OU-prior setting.The study raises AdamW’s learning rate from η = 10^-3 to η = 10^-1.Source: E OPTIMIZER DYNAMICS AND STABILITY · p. 22
  • At η = 10^-2, MercerFlow’s loss converges rapidly and monotonically, while RAWFLOW has a visible ETTh1 spike around epoch 60.The figure contrasts MercerFlow’s trajectory with the unrotated baseline.Source: E OPTIMIZER DYNAMICS AND STABILITY · pp. 22–23, Figure 3 · p. 23

F.1 SYNTHETIC ORNSTEIN–UHLENBECK COMPARISON

Controlled OU tests show Mercer achieves optimal centred-prior conditioning while codec CRPSs nearly tie under AdamW; flow matching saturates in few inference steps, and extreme jitter degrades accuracy.

F.1 SYNTHETIC ORNSTEIN–UHLENBECK COMPARISON

  • κ(DAD)=1.00 for Mercer on the centred prior, versus 1.82 for DCT, 24.9 for Fourier, and 1,501.0 for Raw; conditional interpolants preserve that ordering.On conditional interpolants, Mercer is 98.5, DCT 105.0, Fourier 654.0, and Raw 3,515.0.Source: §F.1 · p. 23, Table 18 · p. 24
  • Under AdamW, all four codecs tied within 0.2% on test CRPS: DCT 0.6062 ± 0.0024, Mercer 0.6063 ± 0.0025, Fourier 0.6064 ± 0.0023, and Raw 0.6069 ± 0.0030.The passage attributes the flat scoreboard to the exact prior match leaving minimal residual dynamics for the flow to resolve.Source: §F.1 · p. 23, Table 18 · p. 24
  • A 10× length-scale mismatch raised Mercer prior/interpolant κ(DAD) from 1.00/98.5 to 1.71/192.0 and CRPS from 0.6063 to 0.6125±0.0057, within seed standard deviation.The model and Mercer basis used ℓ=320 against data generated at ℓ=32.Source: §F.1 · p. 24
  • With Nesterov SGD, Mercer, Fourier, and Raw achieved identical test CRPS of 0.6033±0.0034, with a relative gap from Raw of 0.00%.The authors present this as empirical support for Proposition 1’s rotation-equivariance result.Source: §F.1 · p. 24, Table 18 · p. 24

F.2 SAMPLING EFFICIENCY: FLOW MATCHING VS. DIFFUSION

  • 4–10 Euler steps let MERCERFLOW saturate its predictive distribution (ETTh1 CRPS 0.199), while MERCERDIFF remains suboptimal at NFE 128 (0.47).The comparison uses horizon Lf = 192.Source: §F.2 · p. 24, Table 19 · p. 24

F.3 SENSITIVITY TO PRIOR DIAGONAL JITTER

  • For ε ∈ [10−4, 10−2], Mercer κ(DAD) stayed ≤30.6 on ETTh1 and ≤95.3 on ETTh2, with ETTh1 CRPS 0.186–0.188.On ETTh2, CRPS remained 0.131–0.135 across codecs.Source: §F.3 · p. 25, Table 20 · p. 25
  • At ε ≥0.1, Mercer κ(DAD) reached 3.64 × 10^3 on ETTh2, while ETTh1 CRPS worsened 0.188→0.255 for Mercer and 0.192→0.256 for Raw as the prior neared white noise.The authors attribute the degradation to the loss of the data-matched starting state.Source: §F.3 · p. 25, Table 20 · p. 25
Loading 2610.06039v1…