PaperlayerView exact revision on arXiv

Source-linked AI summary

Contrastive Attention Mitigates Spectral Bias in Spiking Transformers

Xiaoli Liu, Malu Zhang, Yang Yang

arXiv:2610.01403v1cs.AI

TL;DR

Spiking Transformers retain a performance gap relative to ANNs, and the paper attributes this in part to low-pass behavior that dissipates high-frequency information. It proposes SCA, which combines contrastive aggregation with differential refinement. The authors report improvements across several tasks and identify larger-scale evaluation and hardware deployment as limitations.

  • Problem

    Spiking Transformers have a performance gap relative to ANNs, and spiking neurons and self-attention suppress high-frequency components.

    Source: §1 · pp. 1–2

  • Method

    SCA aggregates global context through contrast prototypes and uses differential refinement to emphasize high-frequency information.

    Source: §1 · p. 2

  • Results

    SCA improves Spiking Transformers across image classification, semantic segmentation, and event-based tracking while mitigating high-frequency attenuation.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The authors present SCA as a potential fundamental building block for Spiking Transformers.

    Source: §1 · p. 2

    The study has not tested SCA on larger-scale models such as spike-based large language models or evaluated its hardware deployment.

    Source: §A.7 · p. 21

Abstract

from arXiv · show

Spiking Transformers merge the energy-efficiency of spiking neural networks (SNNs) with the representational power of self-attention, creating a promising architecture for high-performance, energy-efficient computation. However, a performance gap persists versus its counterparts in artificial neural networks (ANNs). Unlike prior works attributing this to binary activations, we reveal that both spiking neurons and spiking self-attention (SSA) act as low-pass filters through multiscale spectral analysis. This characteristic leads to the dissipation of high-frequency components. To address this issue, we propose the Spiking Contrastive Attention (SCA) paradigm, which draw inspiration from the edge-detection and differential sensing properties of biological visual system. By extracting contrast prototypes via global contrastive aggregation and applying local differential refinement, SCA effectively enhances high-frequency information. Extensive experiments show that SCA is a general module that consistently boosts Spiking Transformers across image classification, semantic segmentation, and event-based tracking. Furthermore, it achieves lower complexity, offering superior efficiency over original SSA. These results establish its potential as a fundamental building block for energy-efficient Spiking Transformers.

1. Introduction

Spiking Transformers are energy-efficient, but SNNs retain a performance gap relative to ANNs. The paper attributes this limitation to low-pass behavior in spiking neurons and spiking self-attention, and introduces SCA to enhance high-frequency information.

1. Introduction

  • SNNs offer an energy-efficient alternative to ANNs through spike-based processing, motivating their integration with Transformers.Spikes transmit information asynchronously, with energy consumed only when receiving or emitting spikes.Source: §1 · p. 1
  • Spiking neurons and representative spiking self-attention mechanisms act as low-pass filters, suppressing high-frequency visual information such as textures, edges, and fine structures.The resulting spectral bias is identified as a factor limiting Spiking Transformer performance.Source: §1 · pp. 1–2
  • SCA uses contrast prototypes and differential fusion to refine local representations and emphasize edge-related high-frequency information.Its two stages aggregate global key-value information and then interact with full-resolution query tokens.Source: §1 · p. 2
  • SCA replaces quadratic token interactions with interactions between N query tokens and n contrast prototypes, reducing complexity from O(TN^2C) to O(TNnC), where n ≪ N.Source: §1 · p. 2
  • SCA is reported to improve performance across image classification, semantic segmentation, and event-based tracking.The paper presents it as a potential fundamental building block for Spiking Transformers.Source: §1 · p. 2

2. Preliminary and Related Works

This section introduces LIF neurons as the paper’s spiking-neuron model and situates Spiking Transformers among work on spike-based attention and representation.

2.1. Spiking Neuron Models

  • The paper focuses on Spiking Transformers using LIF neurons, a widely adopted abstraction balancing biological plausibility and computational efficiency.The neuron’s membrane potential and spike/reset behavior are described in the model equations.Source: §2.1 · pp. 2–3

2.2. Spiking Transformer

  • Prior Spiking Transformer work includes spike-based self-attention, architectural changes, spike coding, and multi-spike neurons; Max-Former addresses high-frequency information loss with a MetaFormer-style architecture.This paper positions its frequency-domain analyses and plug-and-play module as another approach to spectral bias.Source: §2.2 · p. 3

3. Problem Analysis

The analysis treats spiking neurons and softmax-free spiking self-attention as sources of low-pass behavior. It also contrasts this with softmax’s capacity to mitigate high-frequency loss.

3. Problem Analysis

  • The section frames the analysis as covering both neuronal filtering and representative softmax-free attention mechanisms.It identifies LIF charging dynamics and high-frequency suppression by attention as the two analysis levels.Source: §3 · p. 3

3.1. Analysis of Spiking Neuron

  • LIF charging is modeled as a first-order IIR low-pass filter whose squared amplitude-frequency response decreases strictly with angular frequency ω ∈ [0, π].For an L-layer SNN, the overall frequency response is exponentially modulated by the low-pass factor.Source: §3.1 · p. 3

3.2. Analysis of Softmax-free Spiking Self-Attention

  • When feature dimension d_k < N, softmax-free attention can truncate high-frequency components, whereas softmax’s element-wise exponential maps a rank-d_k matrix to full rank and can mitigate that loss.The paper states that softmax can, to a certain extent, mitigate high-frequency information loss.Source: §3.2 · p. 3
  • The analysis represents softmax-free attention variants as graph-shift operators O = A · V, with attention-specific generalized adjacency matrices A constructed from Q and K.The formulations are considered in the frequency domain.Source: §3.2 · p. 4
  • SSA, SDSA, and Meta-SDSA behave as low-pass filters in the frequency domain.SDSA uses a diagonal adjacency matrix, while SSA and Meta-SDSA apply factorized low-rank adjacency matrices to value signals.Source: §3.2 · p. 4

4. Methodology

The methodology targets spectral bias in spiking self-attention through SCA’s two-stage global-to-local differential attention, then analyzes its cost against vanilla SSA.

4.1. Differential for Mitigating High-frequency Losses

  • Differential attention computes scores as the difference between two attention maps, enhancing high-frequency information and amplifying attention to relevant context.The method seeks this effect without softmax, whose exponentiation and division conflict with hardware-efficient SNN computation.Source: §4.1 · p. 4

4.2. Spiking Contrastive Attention Mechanism

  • SCA first aggregates global information with positive and negative contrast prototypes, then uses them to differentially refine full-resolution query features.This global-to-local design is described as enhancing high-frequency representations while maintaining spike-driven computation.Source: §4.2 · p. 4

4.2.1. Stage I: Global Contrast Aggregation

  • Stage I spatially pools query features into contrast tokens, adds distinct positive and negative positional embeddings, and aggregates global key-value information in two streams.The streams are integrated by differential membrane-potential fusion to generate a global contrast map.Source: §4.2.1 · p. 5

4.2.2. Stage II: Local Differential Refinement

  • Stage II uses positive and negative contrast tokens as keys and the global contrast map as the value to refine query tokens differentially.The resulting membrane potential drives output neurons to produce spikes for the next layer.Source: §4.2.2 · pp. 5–6

4.3. Complexity Analysis

  • SCA has complexity O(T·NnC), versus SSA’s O(TN^2C), reducing attention cost by a factor of N/n.The complexity analysis compares prototype-based interactions with original SSA.Source: §4 · p. 4, §4.3 · p. 6

5. Experiments

SCA improves results across image classification, semantic segmentation, and event-based tracking; ablations examine the two-stage design and the roles of prototype construction and component choices.

5. Experiments

  • SCA improves results in ImageNet-1K classification, ADE20K segmentation, and event-based tracking, as reported in Tables 2–4.The tracking comparison reports that SCA-enhanced SDTrack outperforms the original architecture on multiple metrics.Source: §5.1 · p. 7, Table 3 · p. 7, §5.2 · pp. 7–8

5.1. Image Classification

  • Across three architectures, SCA improves SDT-V1 by 1.03% and 0.92%, and SDT-V3 by 0.61%, 0.63%, and 0.86% for 5M, 10M, and 19M parameter configurations.QKFormer also gains 0.69% and 0.73% in its two configurations.Source: §5.1 · p. 7

5.2. Semantic Segmentation and Event-based Tracking

  • On ADE20K, SCA improves MIoU by 2.68%, 1.13%, and 1.36% for 5M, 10M, and 19M backbone configurations, respectively.The evaluation follows the SDT-V3 experimental protocol.Source: §5.2 · p. 7, Table 3 · p. 7
  • Across FE108, FELT, and VisEvent, SCA-enhanced SDTrack outperforms the original architecture on multiple metrics.The tracking pipeline processes event streams into event frames using the Global Trajectory Prompt method.Source: §5.2 · pp. 7–8, Table 4 · p. 8

5.3. Ablation Study

  • Using only Stage I or only Stage II decreases performance by 1.15% or 1.07%, respectively, indicating both stages are important in the tested Spikformer setting.The ablation uses Spikformer with 4 blocks and 384 dimensions on CIFAR100.Source: §5.3 · p. 8, Figure 6 · p. 9
  • Prototype performance peaks when both streams combine pooling and learnable embeddings; using embeddings alone decreases performance by 0.52%.Replacing only the positive or negative stream with pooling decreases performance by 0.17% or 0.43%, respectively.Source: §5.3 · p. 8, Figure 6 · p. 9
  • The component ablation identifies differential fusion as the main driver of accuracy gains, while compression and learnable embeddings primarily reduce complexity and stabilize token routing.Contrast-token compression reduces complexity to O(N), and the passage reports that low-rank or global-local design alone slightly degrades accuracy.Source: §5.3 · p. 8, Table 5 · p. 9

6. Conclusion

The conclusion identifies spectral bias as a key performance limitation for Spiking Transformers and presents SCA as a lower-complexity approach evaluated across three task types.

6. Conclusion

  • The paper identifies spectral bias as a key factor limiting Spiking Transformer performance.The conclusion frames this as an explanation of the performance gap with ANN counterparts.Source: §6 · p. 9
  • SCA is presented as a biologically inspired mechanism that aggregates global context at lower complexity and mitigates high-frequency information attenuation.The conclusion connects its inspiration to differential perception in biological neurons.Source: §6 · p. 9
  • Experiments across image classification, semantic segmentation, and event-based tracking show improved model performance with SCA.The conclusion also states that SCA mitigates attenuation of high-frequency information.Source: §6 · p. 9

A.1. Discrete Time Fourier Transform

The section introduces the DTFT as a tool for mapping non-periodic discrete-time signals into the frequency domain, supporting analysis of spectral bias and filtering.

A.1. Discrete Time Fourier Transform

  • The Discrete-Time Fourier Transform maps discrete sequences from the time domain to continuous functions in the frequency domain.The passage presents it as a fundamental tool for analyzing non-periodic discrete-time signals.Source: §A.1 · p. 12
  • The DTFT provides theoretical underpinning for understanding spectral bias and frequency-domain filtering characteristics in deep neural networks.Source: §A.1 · p. 12
  • The section positions frequency-domain analysis as a way to study properties of non-periodic discrete-time signals.Source: §A.1 · p. 12

A.1.1. Definition

The DTFT is introduced for a discrete-time signal x[n], with its frequency variable and spectral components described.

A.1.1. Definition

  • For a discrete-time signal x[n], where n ∈ Z, the DTFT is denoted X(ejω).Source: §A.1.1 · p. 12
  • ω represents normalized angular frequency in radians per sample.Source: §A.1.1 · p. 12
  • X(ejω) is complex-valued and comprises magnitude and phase spectra.Source: §A.1.1 · p. 12

A.1.2. Convergence Conditions

DTFT convergence is described under absolute-summability and finite-energy conditions, with different convergence modes.

A.1.2. Convergence Conditions

  • Absolute summability is presented as the typical condition required for the DTFT to exist.Source: §A.1.2 · p. 12
  • When absolute summability holds, the series converges uniformly to X(ejω).Source: §A.1.2 · p. 12
  • For finite-energy, square-summable signals, the DTFT converges in the mean-square sense.Source: §A.1.2 · p. 12
  • In this context, the spectrum typically precludes Dirac δ functions.Source: §A.1.2 · p. 12

A.1.3. Convolution Theorem

This section analyzes how attention mechanisms and LIF dynamics affect frequency components, including low-pass filtering and differential attention’s frequency response. It also presents experiments on SCA’s effects across high- and low-pass inputs and convolutional architectures.

A.1.3. Convolution Theorem

  • Convolution corresponds to frequency-domain multiplication, while Schur multiplication is element-wise and preserves positive semidefiniteness.This provides a mathematical basis for interpreting convolution as frequency weighting and element-wise products of PSD matrices as PSD.Source: §A.1.3 · p. 12, §A.2.1 · p. 12

A.4. Softmax-free Spiking Self-Attention

  • Vanilla self-attention captures global dependence but is unsuitable for SNNs because floating-point softmax operations and quadratic sequence complexity violate their constraints.The section introduces softmax-free spiking attention as an alternative.Source: §A.3 · p. 13, §A.4 · p. 13

A.4.1. Spiking Self-Attention

  • Spikformer’s SSA uses spiking queries, keys, and values without softmax, while SDSA uses spike tensors and mask-equivalent Hadamard products.SDSA computes attention using masks and addition operations.Source: §A.4.1 · p. 13, §A.4.2 · p. 13

A.4.3. Meta Spike-driven Self-Attention

  • Meta-SDSA is the default Meta-SpikeFormer attention mechanism analyzed, and it incorporates the scaling factor into the spiking-neuron threshold.The section distinguishes Meta-SDSA from SDSA and identifies it as SDSA-3 in.Source: §A.4.3 · pp. 13–14

A.5.1. Proof of Theorem 3.1

  • A LIF neuron's charging process is a first-order IIR low-pass filter, and cascading layers exponentially strengthens suppression of high-frequency signals.Reset preserves the low-pass trend and further attenuates sustained high-frequency fluctuations through nonlinear amplitude truncation.Source: §A.5.1 · pp. 14–15

A.5.2. Proof of Lemma 3.3

  • When dk < N, softmax-free attention is low-rank, so its null space can remove high-frequency components and constrain output Dirichlet energy.The proof attributes this loss to high-frequency variations lying in directions orthogonal to the operator's principal directions.Source: §A.5.2 · p. 15

A.5.3. Proof of Theorem 3.4

  • Softmax’s element-wise exponential restores the attention matrix rank from dk to N, giving it a trivial kernel and mitigating high-frequency information loss.The Schur-product expansion explains how exponentiation of a low-rank score matrix can yield a full-rank matrix.Source: §A.5.3 · pp. 15–16

A.5.4. Proof of Theorem 3.5

  • SSA, SDSA, and Meta-SDSA behave as low-pass filters in the frequency domain.This theorem states the shared spectral behavior of the three spiking self-attention mechanisms.Source: §A.5.4 · p. 16
  • From the graph signal processing perspective, softmax-free attention and SDSA act as low-pass filters that suppress high-frequency components.For softmax-free attention, the rank constraint permits at most d nonzero eigenvalues; SDSA’s sparse diagonal mask gates matching query-key positions and attenuates most randomly distributed high-frequency noise. Meta-SDSA changes the computational path from O(N^2) to O(N) without changing the graph connectivity definition.Source: §A.5.4 · p. 17

A.5.5. Proof of Theorem 4.1

  • Differential attention suppresses the dominant low-frequency/DC component and retains high-frequency response, yielding band-pass or high-pass behavior.Under the stated conditions, it also removes common-mode noise and guarantees SNRdiff > SNRorig when the second attention branch contains less valid information than the first.Source: §A.5.5 · p. 18

A.6.1. Image Classification

  • Experiments used ImageNet-1K and compared attention modules in unchanged Spike-driven Transformer V1, QKFormer, and V3 architectures.Image classification ran on eight NVIDIA H100 GPUs; the study reports PyTorch and hyperparameter comparisons across architectures.Source: §A.6 · p. 18, §A.6.1 · pp. 18–19, Table 6 · p. 19

A.6.2. Semantic Segmentation

  • Semantic segmentation used ImageNet-1K-pretrained SDT-V3 with PVT on ADE20K, while event tracking used the SDTrack pipeline on FE108, VisEvent, and FELT.Segmentation training used batch size 20 for 160,000 iterations; event-tracking schedules were 100 epochs for FE108 and VisEvent and 300 for FELT. Segmentation and tracking experiments ran on eight NVIDIA 4090 GPUs.Source: §A.6 · p. 18, §A.6.2 · p. 19, §A.6.3 · p. 19

A.6.4. Source of Gain

  • +6.35% on high-pass inputs versus +0.06 on low-pass inputs measures QKFormer+SCA’s gain over QKFormer, supporting a high-frequency-specific benefit.The study filters CIFAR100 test images to retain high-frequency edges and textures or remove high frequencies with Gaussian blur. Table 7 presents the performance comparison under these test-set conditions.Source: §A.6.4 · pp. 19–20, Table 7 · p. 21

A.6.5. Generality across Convolutional Architectures

  • Adding SCA’s differential membrane-potential mechanism improved accuracy by 2.99% and 2.46% for SEW-ResNet on CIFAR-10 DVS at T = 4 and T = 8, respectively.For MS-ResNet-18 at T = 4, the improvements were 1.97% on CIFAR-10 and 2.09% on CIFAR-100.Source: §A.6.5 · p. 21, Table 8 · p. 22

A.6.6. Throughput and Inference Latency

  • On a single NVIDIA RTX 4090 with batch size 64, the reduction from O(TN^2C) to O(TNnC) translated into measurable latency improvements.Figure 10 compares throughput and latency for the baselines and the proposed method; the passage gives no specific values.Source: §A.6.6 · p. 21, Figure 10 · p. 22

A.7. Limitations

  • The study leaves testing SCA on larger-scale models and hardware deployment, including FPGA platforms, for future work.The authors identify optimizing data read/write streams and memory-access schemes to improve inference speed as future work.Source: §A.7 · p. 21
Loading 2610.01403v1…