Source-linked AI summary
FourierQK: Filter Shape, Admissibility and the Leakage-Coverage Law
Athanasios Zeris
TL;DR
The paper asks which filter properties matter for spectral attention and examines frequency-selective filtering and bilateral FFT leakage. It finds that effective bilateral FFT attention under clean conditions depends on filter properties, while broader spectral coverage is associated with more leakage.
Problem
The paper asks which filter properties are necessary and sufficient for spectral attention and whether combining scales helps.
Source: §1 · pp. 1–2Method
Results
Under clean bilateral FFT conditions (gap > +4), effective attention requires DC and Nyquist suppression, intermediate bandwidth, paragraph-scale centre frequency, and admissible shape; multi-scale coverage adds gain but introduces leakage.
Source: §5 · p. 7Takeaways & Limitations
FourierQK is effective with bidirectional context, while autoregressive generation requires a causal spectral variant such as MorletQK [Zeris, 2026f].
Source: §1 · p. 2Bilateral FFT attention includes future positions, creating a train/inference mismatch for autoregressive generation.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Frequency-collapse attention [Zeris, 2026e] achieves large gains over standard dot-product attention by replacing the Q/K dot product with a bandpass-filtered inner product at a learned frequency. A natural follow-up question is: which filter shape works best, and why? We test five hypotheses about filter properties -- DC suppression, Nyquist suppression, bandwidth, centre frequency, and multi-scale coverage -- using a controlled ablation on character-level language modelling (TinyShakespeare, 6-layer GPT). Our main findings are: (1) DC and Nyquist components are actively harmful (val ~= 2.0, equivalent to phase randomisation), confirming that oscillatory bandpass structure is essential, not just any low-dimensional spectral summary; (2) the optimal single-scale bandwidth is sigma ~= 2 bins centred at paragraph scale (~70 tokens), giving a clean gain of Delta = +1.15 nats over BASE-DOT; (3) admissible filters (zero-mean, Mexican Hat DOG m = 2) outperform non-admissible Gaussians at the same scale and provide partial protection against bilateral FFT leakage; (4) bilateral FFT leakage scales monotonically with spectral coverage -- narrowband filters (gap > +4) are clean, wideband filters (gap < +2) are leaky; and (5) causal time-domain Morlet at character scale cannot beat BASE-DOT (K=128 taps covers 50% of T=256 context), motivating word-level experiments in the companion MorletQK paper [Zeris, 2026f]. Together, findings (1)-(5) characterise FourierQK as effective in bidirectional attention settings (encoder-style, e.g. BERT), where full-sequence context is available at both training and inference time; autoregressive generation requires a causal spectral variant such as MorletQK [Zeris, 2026f] (decoder-style, e.g. GPT). Code available at: https://github.com/AthanasiosZeris/energy-gated-attention
1 Introduction
Prior work reports large, reproducible gains from frequency-collapse attention over standard attention [Zeris, 2026e]; this paper asks which filter properties are necessary and sufficient for those gains and characterizes their deployment implications.
1 Introduction
- Frequency-collapse attention produced large, reproducible gains over standard attention, with phase coherence, global sequence mixing, and a learned near-dyadic frequency hierarchy identified as key mechanisms [Zeris, 2026e].Phase randomization destroys the gain, while causal convolution provides no gain.Source: §1 · p. 1
- The paper asks which filter properties are necessary and sufficient once spectral attention is known to work.The question has concrete implications for deployment.Source: §1 · p. 1
- Bilateral FFT attention uses future positions, fitting bidirectional encoders but creating a train/inference mismatch for autoregressive generation; MorletQK instead uses causal FIR convolution [Zeris, 2026f].Bidirectional settings have the full sequence available at training and inference time.Source: §1 · p. 2
- The controlled ablation tests DC suppression, Nyquist suppression, bandwidth, centre frequency, and multi-scale coverage.The multi-scale question is whether combining scales helps.Source: §1 · p. 2
- A shuffled-gap diagnostic is used to characterize the leakage structure of bilateral FFT methods.The diagnostic serves as an inverse proxy for leakage severity.Source: §1 · p. 2
2 Background and Architecture
The section specifies the GPT setup and spectral scoring operation, then defines a shuffled-gap diagnostic for assessing bilateral FFT leakage.
2 Background and Architecture
- Experiments use a 6-layer GPT on TinyShakespeare with block size T = 256, and spectral variants replace Q/K dot-product scoring with frequency-collapse scoring.The attention score is the scaled inner product of filtered Q and K vectors.Source: §2 · p. 2
2.1 Leakage diagnostic
- Bilateral FFT filtering is non-causal because a filtered vector at position i depends on Q/K values across the full sequence, including future positions.The shuffled gap compares validation performance on ordered and fixed-permutation validation data.Source: §2.1 · p. 2
- A shuffled gap above +4 indicates use of temporal ordering consistent with legitimate learning, while a gap below +2 is consistent with leakage.The diagnostic is described as being in the spirit of surrogate-data testing.Source: §2.1 · p. 2
3 Frequency-Selective Filtering: Filter Shape Ablations
The ablations test which filter properties preserve phase-coherent signal and which introduce bilateral FFT leakage. They find that clean frequency-collapse attention depends on bandpass filtering, with bandwidth, centre frequency, and filter shape determining the results.
3 Frequency-Selective Filtering: Filter Shape Ablations
- The paper calls narrow informative frequency-band selection “spectral compression,” meaning bandpass selection rather than dimensionality reduction or lossy coding.The ablation tests whether bandpass compression specifically, rather than any low-dimensional spectral summary, helps.Source: §3 · p. 3
- DC gives val=2.017, identical to phase randomisation, while Nyquist gives val=1.808, still below BASE-DOT.The real-only results indicate that mean-pooled Q/K lacks phase-coherent oscillatory information, whereas character alternation supplies a weak signal.Source: §3 · p. 3
- The DC and Nyquist null results require real-only scoring; the 1j branch tests different hypotheses despite using the same frequency-bin labels.For DC, the 1j branch yields a time-varying signal encoding departures from the sequence mean, not the mean itself.Source: §3 · pp. 3–4
- Among clean models, val=0.326 for Init4 (σ=2) is better than val=0.840 for GaussNarrow (σ=0.5), making intermediate bandwidth optimal.Too-narrow filtering selects a single frequency bin.Source: §3 · p. 4
- GaussWide (σ=16) reaches val=0.023 but is leaky (gap=+0.93), covering 96/129 bins; MexHat-K4 is less leaky (gap=+1.50).The zero-DC MexHat filter provides partial leakage mitigation compared with GaussWide.Source: §3 · p. 4
- The clean regime requires suppressed DC and Nyquist, σ≈2-bin bandwidth, paragraph-scale centre frequency, and multiple scales for maximum gain.The reported paragraph-scale periods are approximately 65–73 tokens; clean multi-scale MexHat-K4 would require a causal implementation.Source: §3 · p. 4
4 Wavelet Collapse and Admissibility
The section compares admissible wavelet filters with Gaussian filters, tests causal filters at character scale, and relates bilateral FFT leakage to spectral coverage. It finds that admissibility partially mitigates leakage, while character-scale causal filters do not beat BASE-DOT.
4 Wavelet Collapse and Admissibility
- MexHat and Paul are admissible, unlike the non-zero-mean Gaussian; the filter-shape ranking is MexHat > Paul ≫ Gauss.The paper relates that ranking to the admissibility condition ˆψ(0)=0.Source: §4 · pp. 4–5
4.1 Causal filters: no gain at character scale
- All tested causal filters perform below BASE-DOT at character scale: Gaussian-QK val=1.522, MexHat-QK val=1.540, and Morlet-causal val=1.512.A fixed causal Morlet scale sweep also remains below BASE-DOT; its best result is a=32 (val=1.487, Δ=−0.020).Source: §4.1 · p. 5
- At a=32, causal Morlet improves monotonically with scale but remains worse than BASE-DOT; K=128 taps cover only 50% of the T=256 context.By comparison, global Fourier-QK mixing uses 100% of context and achieves val=0.326 (clean, Δ=+1.15).Source: §4.1 · p. 5
4.2 Leakage severity as spectral coverage
- The shuffled gap decreases monotonically as spectral coverage rises; at the same 15% coverage, admissible MexHat-K4 has a higher gap than non-admissible filters.Table 2 defines gap as shuffled_val − ordered_val and spectral coverage as the fraction of bins with significant filter response.Source: §4.2 · p. 5, Table 2 · p. 5
- Three independently written implementations agree that K=1 at σ=2 bins (3.9% coverage) remains clean, while K=4 (about 15% coverage) becomes leaky.The independent reviewer’s K=4 run reaches val=0.020 with gap=+0.58, matching the reported multi-seed range within run-to-run variation.Source: §4.2 · p. 6
5 Discussion
Effective bilateral frequency-collapse attention requires DC and Nyquist suppression, intermediate bandwidth, paragraph-scale frequency, and admissible filters. Leakage rises with spectral coverage, while multi-scale filters add gain but introduce leakage in the bilateral regime.
5 Discussion
- DC and Nyquist suppression, intermediate bandwidth (σ ≈2 bins), paragraph-scale centre (∼65–73 tokens), and admissible filter shape define effective bilateral frequency-collapse attention under clean conditions.Multi-scale coverage provides additional gain but introduces leakage in the bilateral FFT regime and is cleanly testable only in the causal regime.Source: §5 · p. 7
- Bilateral FFT leakage scales with spectral coverage: gap > +4 classifies models as clean, while gap < +2 marks them as leaky.The paper reports that leaky models' absolute validation values overstate genuine gain.Source: §5 · p. 7
- Admissible filters suppress DC through their zero-mean property, partially blocking mean-pooling leakage in the bilateral FFT regime.This gives admissibility a leakage-control role in attention, beyond its traditional wavelet-theory motivation.Source: §5 · p. 7
- At word-level tokenisation, K = 128 taps covers more than one period at paragraph scale, resolving the character-scale aliasing problem for causal Morlet filters.These filter conditions specify the target for MorletQK [Zeris, 2026f], which compares with causal Haar WaveletGPT [Verma, 2024].Source: §5 · p. 7
A The 1j-branch frequency ablation regime
In the 1j-branch regime, leakage varies with centre frequency even at fixed bandwidth, while the learned frequency remains stable near bin 4. The cleanest reported result favors a global mean-free component.
A The 1j-branch frequency ablation regime
- This appendix reports a separate fixed-bin frequency ladder using two-branch complex scoring, rather than the real-only scoring used throughout §3.The reported findings are included for completeness and to support the stated caveat.Source: A The 1j-branch frequency ablation… · p. 7
- With identical σ = 2-bin bandwidth and spectral coverage, the shuffled gap falls from +4.37 at DC to +2.31 at bin=64.The authors attribute the frequency-linked leakage to sharper temporal localization from the Hilbert transform at higher frequencies, a mechanism distinct from coverage-driven leakage.Source: A The 1j-branch frequency ablation… · p. 7
- Under the clean gap > +4 restriction, bin=1 (period=256 tokens) gives the best honest result, val = 0.312.The passage relates this result to global low-frequency structure rather than high-frequency local patterns.Source: A The 1j-branch frequency ablation… · p. 8
- The learned frequency stays near bin ≈4.1 (period ≈63 tokens) across all six layers from step 0 through step 5000.It was initialized at bin=4 with log f excluded from weight decay, and the stability mirrors Zeris [2026e].Source: A The 1j-branch frequency ablation… · p. 8
B Conclusion
The conclusion identifies filter conditions for effective bilateral frequency-collapse attention and distinguishes its fit for bidirectional encoders from its train/inference mismatch in autoregressive generation. It also presents shuffled-gap as a model-independent diagnostic of bilateral leakage.
B Conclusion
- For single-scale frequency-collapse attention, DC and Nyquist suppression are necessary because both are harmful; intermediate bandwidth (σ ≈2 bins) at paragraph scale (∼70 tokens) is optimal.Source: B Conclusion · p. 8
- Causal convolution at character scale cannot replicate the bilateral FFT's global sequence mixing.This motivates a causal spectral variant rather than using bilateral FFT attention for autoregressive generation.Source: B Conclusion · p. 8
- Bilateral spectral Q/K suits bidirectional encoders with full-sequence context, but creates train/inference mismatch for autoregressive generation.MorletQK [Zeris, 2026f] replaces the symmetric kernel with a causal FIR convolution and carries over the identified filter conditions.Source: B Conclusion · p. 8
- The shuffled-gap diagnostic measures leakage severity across spectral attention variants; leakage increases monotonically with the fraction of spectrum covered.The paper recommends narrow admissible bandpass filters and verifying gap > +4 before treating validation improvements as genuine language-modelling gains.Source: B Conclusion · p. 8