PaperlayerView exact revision on arXiv

Source-linked AI summary

ALPHABET: A Laplace-Pole History Aggregator with Banked Exponential Transport

Daehwa Ko, JaeHyeon Kim, Oh Seong Kwon, Jay Hoon Jung

arXiv:2608.24051v1cs.LG

TL;DR

Compact sequence models still lack a clear account of what temporal information their retained states preserve. ALPHABET addresses this with stable pole-based modal statistics and shows competitive prediction alongside substantially lower latency than evaluated baselines.

  • Problem

    The paper asks whether compact sequence states can preserve discriminative temporal structure while avoiding the size, speed, and opacity of modern backbones.

    Source: Introduction · p. 1

  • Method

    ALPHABET uses direct and independent cascaded banks of stable complex Laplace-pole modes, with an affine interface reading modal energies and lag moments.

    Source: Introduction · p. 1

  • Results

    Across the evaluated registry and audits, ALPHABET provides competitive temporal prediction and substantially lower latency than all evaluated baselines.

    Source: Introduction · p. 2, Conclusion · p. 7

  • Takeaways & Limitations

    The model’s modal readout has a characterized spectral interpretation, while its predictions decompose into per-pole contributions for an auditable representation.

    Source: Introduction · p. 1, Conclusion · p. 7

    The spectral identification and affine-consistency guarantees assume a fixed encoder with a stationary, fully observed, equal-step feature process and concern learned features rather than raw inputs.

    Source: Discussion and Limitations · p. 7

Abstract

from arXiv · show

Can a sequence model remain competitive with only a few thousand parameters and an explicitly auditable prediction interface? We introduce ALPHABET, a compact linear-time model that compresses temporal history into stable complex pole modes: a direct bank synthesizes its modal states back into the feature trajectory, an independent cascaded bank analyzes the transformed trajectory without resynthesis, and an affine head reads only modal energies and lag moments from both banks. We characterize the temporal information this descriptor retains: for a stationary, fully observed feature process, each mode energy is a frequency-localized measurement of the second-order spectrum, the continuum of such measurements identifies the spectrum, and almost every mode separates any fixed finite set of spectrally distinct classes. On a Gaussian control with matched low-lag statistics, the learned descriptor approaches the Bayes oracle where raw autocovariances remain at chance. Across the fixed 82-task registry, ALPHABET attains mean rank 3.97 in the complete ten-family comparison. At the common-width D=64 runtime anchor, its 6,437 parameters deliver 5.02 times faster inference and 3.93 times faster complete training steps than the nine baselines on average.

Introduction

ALPHABET studies whether a deliberately compact, stable state can preserve discriminative temporal structure through an auditable prediction interface. It combines two stable Laplace-pole banks with spectral interpretations and generic population-separation guarantees.

Motivation

  • ALPHABET compresses recurrent history into a small prediction interface whose temporal measurements have explicit meaning, stability, and empirical utility.The design targets compact prediction rather than reconstruction of every input token.Source: Introduction · p. 1

Architecture

  • Two complex recurrent banks use stable Laplace poles: a direct bank synthesizes projected modal states, while an independent cascaded bank analyzes the transformed trajectory without resynthesis.Pole locations determine modal decay and oscillation; spectral coverage and modal moment readout are essential to the construction.Source: Introduction · p. 1

Theory

  • For stationary feature processes, each pole energy is a directional, Poisson-localized measurement of the matrix-valued second-order spectrum.The continuum of admissible measurements uniquely identifies that spectrum.Source: Introduction · pp. 1–2
  • Almost every admissible mode separates any fixed finite collection of distinct class spectra, and a finite bank containing such a mode admits an exactly separating affine head.These guarantees apply to population prototypes under the stated stationary-process setting.Source: Introduction · p. 1

Architecture

  • The affine prediction interface reads only mode-wise energies and lag moments from both banks, yielding a linear-time sequence model.The direct and cascaded banks provide complementary modal descriptors for the fixed interface.Source: Introduction · p. 1, Figure 1 · p. 2

Theory

  • Stable interval-aware dynamics and normalized excitation provide sequence-length-independent bounds for realized finite-bank prediction interfaces.This result accompanies the spectral characterization for fixed, fully observed, equal-step, second-order stationary feature processes.Source: Introduction · p. 2

Related Work

Prior temporal models construct history through gated recurrence, attention, or learned linear and selective dynamics. Related spectral approaches seek compact dynamical descriptions using reduction methods, decompositions, filter banks, covariance features, and pole-based representations.

Temporal state construction

  • Recurrent models propagate gated hidden states, Transformers build contextual representations through attention, and state-space models encode temporal history with learned linear or selective dynamics.The passage names LSTMs, GRUs, Transformers, HiPPO, S4, DSS, S4D, S5, LRUs, and Mamba.Source: Related Work · p. 2

Structured spectral representations

  • Compact spectral descriptions include balanced reduction, dynamic mode decomposition, Laguerre or Kautz bases, filter-output covariance, state covariance, wavelet variances, random autoregressive filter banks, and recurrent covariance features.These methods are presented as structured spectral representations and classical spectral interpolation objects.Source: Related Work · p. 2

ALPHABET

ALPHABET represents temporal inputs with two stable complex-pole banks and predicts from modal moment descriptors. Observation and validity masks control injection and feature statistics while elapsed-time transitions continue, supporting irregular sampling.

Architecture

  • ALPHABET uses direct and cascaded banks indexed by r = 1, 2, with M complex modes per bank and D-dimensional real features.The direct and cascaded banks are distinguished explicitly, while αm and ωm parameterize each mode’s decay rate and angular frequency.Source: ALPHABET · pp. 2–3, Stable Laplace-pole dynamics · p. 3

State transport

  • The exact pole-state evolution remains stable under elapsed-time transitions, with regular sampling fixing pm and irregular sampling adapting pm,t to ht.The cascaded bank omits the observation mask to preserve memory transported by the direct bank.Source: Stable Laplace-pole dynamics · p. 3

Masking

  • Validity masking blocks injection at invalid tokens, while event availability additionally blocks direct-bank injection when all channels are unobserved.Neither mask stops the elapsed-time state transition; the observation mask also removes unavailable raw entries and gates external excitation.Source: ALPHABET · p. 2, Stable Laplace-pole dynamics · p. 3

Modal readout

  • Each mode contributes empirical energy and phase-sensitive lag moments, producing seven real coordinates per mode for each bank.With T = {1, 2, 4}, each bank has 7M descriptor coordinates, and the concatenated descriptor has dimension 14M.Source: Modal moment readout · p. 3

Prediction

  • The affine head consumes the concatenated direct- and cascaded-bank descriptors to produce K-dimensional logits.The final mapping is fΘ(X) = Wheadg(X) + bhead, where g(X) combines ϕ(1)(X) and ϕ(2)(X).Source: Modal moment readout · p. 3

Theoretical Properties of Modal Compression

ALPHABET’s modal descriptor retains localized spectral mass and phase-sensitive lag information, with a continuum of directional modes that identifies the full spectral measure and generically separates spectrally distinct classes. Stable filtering, conditioned moments, and RMS-normalized prediction provide finite-sample consistency and a bounded interface independent of sequence length and raw-input amplitude.

Spectral representation

  • Modal energy measures Poisson-localized spectral mass, while complex moments retain phase-sensitive lag information.The localization sharpens as pole damping approaches one, and moments add coordinates to the finite bank.Source: Pole moments as spectral measurements · p. 4

Characteristicness and separation

  • A positive-volume set of directional pole transforms identifies the full matrix-valued spectral measure and separates almost every pair of spectrally distinct classes.Injectivity also holds at fixed damping radius when frequency and direction pairs vary.Source: Characteristicness and affine separation · p. 4
  • A generic modal energy separates classes sharing the same feature mean and Γ(0) but differing in some Γ(k), k ≠ 0.The class-independent positive factor transfers transform separation to modal energies.Source: Characteristicness and affine separation · p. 4

Affine realizability and consistency

  • An affine log-energy head classifies every population prototype exactly when one mode has pairwise distinct class energies.For stationary ergodic processes with bounded excitation and summable energy autocovariance, the descriptor’s error probability has an O(T^-1) bound.Source: Characteristicness and affine separation · p. 4

Bounded prediction interface

  • Stable zero-initialized states remain bounded independently of token count, and RMS normalization yields a bounded prediction interface uniformly over finite raw-input amplitudes and sequence configurations.The bound applies to both banks and is independent of upstream operator norms because normalization precedes each scan.Source: Stable dynamics and bounded prediction · pp. 4–5

Experiments

Across a fixed 82-task registry, ALPHABET achieves strong predictive rank while remaining exceptionally compact and fast. Controlled diagnostics support higher-lag spectral access and a task-dependent benefit from its two-scan design, with pole-level attribution and corruption audits exposing both mechanisms and limitations.

Benchmark coverage

  • 82 tasks span univariate and multivariate signals, regular and irregular observations, and classification, multilabel, and forecasting objectives.The registry includes 30 UCR datasets, 31 multivariate sequence and vibration/fault tasks, and 21 additional sequence, ECG, clinical/activity, and forecasting tasks.Source: Evaluation protocol · p. 5

Predictive performance

  • 3.97 mean rank, 14 Top-1 placements, and 43 Top-3 placements make ALPHABET competitive across the complete ten-family comparison.These results summarize tasks with complete results for all ten model families.Source: Predictive rank, capacity, and systems… · p. 5

Efficiency

  • 6,437 trainable parameters yield 5.02× faster inference and 3.93× faster complete training steps than nine baselines on average at D=64.Measurements use steady-state FP32 execution on one RTX 4090 under an identical harness; the anchor is D=64, B=32, T=512.Source: Predictive rank, capacity, and systems… · p. 5

Controlled spectral recovery

  • .856 balanced accuracy at ϵ = .4 exceeds LRU’s .787 and S4D’s .750 on the moment-matched MA(5) control.The control matches population autocovariances through lag 4 and first differs at lag 5, testing higher-lag spectral access.Source: Controlled spectral recovery under… · p. 5

Structural ablations

  • 3.33 points is the full model’s advantage over the capacity-matched one-scan control, while no reported contrast has Holm-adjusted p < .05.The full model also exceeds no-synthesis and energy-only controls by 1.67 and 5.48 points, respectively; fixed poles trail by 1.67 points but remain statistically competitive.Source: Focused structural ablation · pp. 5–6

Pole attribution

  • .990 mean rank correlation shows optimization-fold pole rankings transfer to validation, while removing the top eight poles reduces balanced accuracy by 26.9 points more than random removal.Retaining only the top eight improves prediction agreement with the unmasked model by 23.3 points over random retention.Source: Pole-level prediction audit · p. 6

Discussion and Limitations

ALPHABET’s guarantees apply to fixed encoders producing stationary, fully observed, equal-step feature processes, and characterize learned-feature spectra rather than raw inputs. Empirically, additive noise limits performance, post-hoc shrinkage fails to transfer, and prospective input-conditioned debiasing remains needed.

Theoretical scope

  • The spectral theory assumes a fixed encoder, stationary fully observed features, and equal-step sampling, and identifies the learned feature process’s spectrum rather than the raw input’s.The characterization concerns what the modal readout preserves about encoder output, not what the encoder discards.Source: Discussion and Limitations · p. 7
  • The guarantee is generic rather than constructive: optimization need not find a separating mode, while higher-order non-Gaussian differences fall outside the second-order theory.Interval-aware recurrence, descriptor, and logit bounds still hold on irregular grids without stationarity.Source: Discussion and Limitations · p. 7

Empirical boundaries

  • Across 17 tasks, the two-scan advantage remains positive, but no ablation contrast survives Holm correction and fixed poles remain competitive.Attribution in the fixed-pole variant is similarly concentrated, supporting an auditable stable-bank interface.Source: Discussion and Limitations · p. 7
  • Additive noise contaminates modal statistics at both evaluated noise levels, so the representation is not recommended in low-SNR settings.This is identified as the clearest empirical boundary.Source: Discussion and Limitations · p. 7

Prospective debiasing

  • A post-hoc rule suppressing weak lag moments failed on dataset-disjoint validation because weak lag coherence can represent nuisance noise or weak class signal.Input-conditioned, pole-response-aware debiasing of Q_m and R_m,τ should therefore be frozen and tested prospectively.Source: Discussion and Limitations · p. 7

Conclusion

ALPHABET shows that competitive temporal prediction can use a compact, auditable architecture rather than a large or opaque backbone. Its two stable Laplace-pole banks yield spectrally interpretable modal statistics, while predictions decompose exactly into per-pole contributions and achieve substantially lower latency than evaluated baselines.

Conclusion

  • Two banks of stable Laplace poles compress temporal history into fixed modal statistics with characterized spectral meaning.The architecture uses two pole banks to produce the temporal descriptor.Source: Conclusion · p. 7
  • Predictions decompose exactly into contributions from individual poles, providing an auditable prediction interface.This decomposition supports inspection of how each pole contributes to a prediction.Source: Conclusion · p. 7
  • ALPHABET achieves substantially lower latency than all evaluated baselines under the matched evaluation settings.The latency comparison applies to the evaluation settings considered in the work.Source: Conclusion · p. 7

Proofs and Mathematical Boundaries

The proofs establish injective, non-expansive moment conditioning and generic spectral identification, while showing that any finite pole bank cannot be globally injective over all analytic spectra. They also bound descriptor magnitude and perturbation under stable, normalized dynamics.

Moment conditioning

  • Lemma 2 makes radial-log moment conditioning injective and non-expansive on bounded descriptor regions.The map preserves zero and has Jacobian eigenvalues bounded between (1 + U)^−1 and 1.Source: Radial-log geometry · p. 8

Spectral identification

  • Almost every direction and pole separates pairwise distinct class spectra, and when 2M ≤D a separating pair can be completed to an admissible M-mode frame.This is an existence result; it does not assert that optimization discovers the separating frame.Source: Spectral embedding, generic… · p. 8

Finite-bank boundary

  • Any fixed finite pole bank fails global injectivity: distinct strictly positive, real-analytic, full-support spectra can share all finite modal descriptors.The limitation does not preclude separation on finite task families or under the stated generic conditions.Source: Spectral embedding, generic… · pp. 8–9

Stability and perturbation

  • Under stated normalization, stability, and parameter bounds, descriptor norm is independent of raw-input amplitude and token count, while affine scores admit perturbation bounds.Invalidity masks with no eligible pairs contribute zero under the moment-definition denominator convention.Source: Cumulative-step and… · p. 9

Model Implementation

ALPHABET uses two independently maintained pole banks: only the direct bank synthesizes its modal states, while the affine head reads fixed-lag modal descriptors without a pooled D-dimensional path. Its implementation supports physical-time interpolation and a specialized static-pole CUDA path using associative scans, while retaining linear-time sequencedependent compute.

Banked implementation

  • Only the direct bank synthesizes through A⊤, while the affine head uses fixed offsets T = {1, 2, 4} and has K(14M + 1) parameters.The cascaded bank analyzes the transformed trajectory without direct resynthesis, and no pooled D-dimensional path is used.Source: Model Implementation · p. 9

Time handling

  • When time_delta is supplied, local lifts and modal readouts switch from token offsets to fixed physical-time offsets with interpolation and validity masking.The exact-ZOH transition uses the same elapsed-time metadata, and empty interpolation support returns zero.Source: Model Implementation · p. 9

Complexity

  • O(T) sequencedependent compute is achieved with width-5 lifts, diagonal pole recurrences, direct-bank synthesis, fixed-lag costs, and streamed modal-state memory.The listed costs are O(TD), O(TMD), O(TM), and O(MK), with training potentially retaining trajectories for automatic differentiation.Source: Model Implementation · p. 10

CUDA specialization

  • O(log T) parallel scan depth replaces sequential recurrence depth for each bank, while the cascaded bank remains serially composed because it depends on the synthesized direct trajectory.The associative scan also accumulates seven per-mode radial-log statistics, and the specialized path is numerical-parity gated because reassociation changes FP32 rounding order.Source: Final radial-log kernel specialization · p. 10

Validation

  • 8.345 × 10−7 maximum logit error was observed, while gradients and three-step optimization trajectories passed at both anchors across seeds {7, 11, 19}.The fail-closed 140-group evaluator reported status=pass.Source: Final radial-log kernel specialization · p. 11

Evaluation Protocol

ALPHABET is evaluated on a fixed 82-task registry under a common candidate-selection protocol, with matched candidate and final-seed counts across nine control families. Configuration selection uses staged validation evaluation, while official TEST data remain sealed until final evaluation.

Candidate design

  • 18 candidates and five final-training seeds are used in every task–family cell across ALPHABET and the nine control families.The control families are CNN1D, TCN, Transformer, Mamba, S4D, S5, LRU, GRU, and LSTM; native parameters and compute costs are not matched.Source: Evaluation Protocol · p. 11, Table 3 · p. 11

Configuration space

  • Each control family uses widths {32, 64, 128}, two structural choices, and recipes A/B/C, producing 18 candidates per task–family cell.Recipes use learning-rate and clipping pairs (10−3, 0.5), (3×10−3, 1), and (10−2, 2), with weight decay 10−4 and effective batch size 64.Source: Table 3 · p. 11, Tasks, objectives, and data sealing · p. 11

Data governance

  • Official TEST data remain sealed until configuration selection is complete, with UCR and UEA selection restricted to official TRAIN data.Forecasting windows preserve chronological splits, and normalization plus data-dependent preprocessing exclude TEST data.Source: Evaluation Protocol · p. 11, Model configuration ledger · p. 12

Benchmark registry

  • 82 independently trained tasks form the fixed registry, spanning UCR, general sequence, ECG, clinical/activity, forecasting, multivariate, and vibration/fault tasks.Registry membership follows public availability, reproducible splits and preprocessing, common-interface compatibility, and broad task coverage.Source: Tasks, objectives, and data sealing · p. 11

Configuration selection

  • Stage 1 evaluates all 18 candidates at seed 7, retains the validation Top-6, and Stage 2 evaluates those six at seeds 11 and 19 before freezing the best mean validation configuration.Exact ties are resolved with the predeclared configuration key.Source: Model configuration ledger · p. 11

Diagnostic Protocols

The diagnostic protocols use TRAIN-only, fixed-split evaluations to test temporal-dependence separation, descriptor ablations, pole-block contributions, and robustness under controlled corruptions. A Gaussian control matches low-lag autocovariances while differing at lag 5, enabling comparison of spectral descriptors with raw autocovariances and Bayes likelihoods.

Common protocol

  • TRAIN-only UCR diagnostics use a stratified 80/20 official-TRAIN split, TRAIN-fitted preprocessing, and no official TEST data.Unless stated otherwise, experiments use (D, M) = (64, 16), 100 epochs, seeds {7, 11, 19}, and confirmatory trial 4.Source: Diagnostic Protocols · p. 12

Gaussian control

  • The Gaussian pair is stationary, Gaussian, and unit variance, with matching population autocovariances for |k| ≤4 and the first difference at lag 5.The construction is Xt = aZt + bZt−5, with a2 + b2 = 1 and 2ab = ϵ; ϵ ∈{.1, .2, .4, .8} is swept across five seeds.Source: Moment-matched spectral control · p. 12
  • The Gaussian control compares exact Gaussian Bayes likelihood ratios, nearest-prototype classification from empirical raw bΓ(0:4), and a capacity-matched energy-only model.Each seed uses 512 optimization and 256 validation paths of length 128, with the estimators sharing the same realized samples.Source: Moment-matched spectral control · p. 12

Descriptor ablations

  • The two-scan comparison gives a one-scan baseline the same 224 real coordinates with M′ = 32 and equalizes trainable parameters using D′ = 72.The main-paper ablation reports the resulting task-paired comparison.Source: Capacity-matched structural ablation · p. 12

Corruption evaluation

  • Robustness evaluates 850 checkpoints from 10 × 17 × 5 training runs without retraining under Gaussian noise, amplitude changes, and half-rate downsampling with restoration.Configurations are frozen from TRAIN-derived validation, and the section records the TEST-free evaluation contract.Source: Synthetic-corruption protocol · p. 12

Complete TEST Results

Complete TEST evaluation uses a frozen two-stage validation protocol across 82 tasks and ten model families, totaling 28,700 fits. Reporting follows dataset-specific metrics, leakage controls, and rank-based aggregation procedures.

Evaluation protocol

  • 28,700 fits span 82 tasks and ten model families under the prescribed selection and final-evaluation protocol.Each task–family cell uses 18 Stage 1 fits, 12 Stage 2 fits, and five final-training fits.Source: Configuration freezing and final… · p. 12
  • Only the configuration selected by two-stage validation contributes to TEST results, while TEST data remain inaccessible during selection.Final-training seed sets are fixed before TEST evaluation, and nonselected configurations do not affect reporting or cross-model ranking.Source: Configuration freezing and final… · pp. 12–13

Metrics

  • Metrics are fixed by dataset: balanced accuracy for several classification families, MSE for forecasting, macro AUPRC for AudioSet, and accuracy or AUROC for remaining specified rows.Every Table 4 row names its metric and direction; selection may use a different imbalance-sensitive metric than the primary reporting endpoint.Source: Final metrics and PhysioNet handling · p. 13

Dataset-specific evaluation

  • PhysioNet 2012 evaluation retains original minute timestamp groups and combines standardized values, observation indicators, elapsed time, and timestep validity.Set C remains sealed until configuration freezing, and preprocessing is standardized using optimization records only.Source: Dataset-specific protocol: PhysioNet… · p. 13

Statistical aggregation

  • Registry summaries average per-task ranks rather than heterogeneous raw metrics, with Friedman and corrected pairwise signed-rank tests applied to taskwise rank differences.Pairwise tests retain zeros by Pratt’s convention and use Holm correction.Source: Completeness, ranking, and statistical… · p. 14

Related papers

Loading 2608.24051v1…