PaperlayerView exact revision on arXiv

Source-linked AI summary

Chiaroscuro Attention: Spending Compute in the Dark

Prateek Kumar Sikdar

arXiv:2606.08327v2cs.CLcs.AIcs.LG

TL;DR

CHIAR-Former addresses the cost of fixed-pattern quadratic attention by routing tokens between DCT spectral mixing and full attention using spectral entropy, while adding a learned task-level gate. It achieves 37% FLOP reduction at 400M with a 3.93 PPL cost on WikiText-103, and mixed-dataset training shows strong spectral regularisation on small corpora. The paper identifies DCT+Attention as the effective operator pair and reports a stable MetaRouter equilibrium at g ≈0.22.

  • Problem

    Fixed-pattern self-attention has quadratic cost O(n^2d) and does not adapt computation to individual token information content.

  • Method

    CHIAR-Former routes tokens between DCT spectral mixing and full attention using spectral entropy, and uses a learned MetaRouter to soft-blend spectral and identity paths.

  • Results

    37% FLOP reduction at 400M accompanies a 3.93 PPL cost on WikiText-103, while mixed-dataset training shows strong spectral regularisation on small corpora.

  • Takeaways & Limitations

    DCT+Attention forms the effective operator pair, and MetaRouter stabilization at g ≈0.22 indicates complementary spectral and attention roles at scale.

  • Takeaways & Limitations

    Future work remains to hard-gate the MetaRouter at inference, scale beyond 1B parameters, and evaluate downstream NLP benchmarks.

Abstract

from arXiv · show

We introduce CHIAR-Former (CHIAroscuro Attention-based tRansFormer), an efficient transformer that routes each token to either DCT spectral mixing (O(d log d), sub-quadratic) or full self-attention (O(n^2 d), quadratic in sequence length n) based on per-token spectral entropy H(x) in [0,1], which measures the frequency-domain complexity of each token embedding x. We make three contributions: (1) we discover routing collapse -- a three-operator system collapses to DCT+Attention, revealing the optimal operator subset; (2) we propose a learned task-level MetaRouter g = sigma(Linear(x-bar)) in [0,1], where x-bar is the batch-mean embedding and g soft-blends spectral and identity paths end-to-end; and (3) we demonstrate 35-40% FLOP reduction at 400M parameters with a 3.93 PPL cost on WikiText-103 (Test PPL 27.51 vs. 23.58). Under mixed-dataset training, CHIAR-Former dramatically outperforms full attention on small corpora, confirming the regularisation value of spectral mixing. The MetaRouter stabilises at g ~ 0.22, indicating that at scale the model reaches a robust compute-quality equilibrium: attention layers absorb representational complexity while spectral preprocessing efficiently anchors low-frequency structure.

1 Introduction

CHIAR-Former addresses the quadratic cost of fixed-pattern self-attention by routing tokens according to spectral entropy, using cheap DCT mixing for smooth signals and full attention for complex ones. Its contributions include entropy-based routing, a learned task-level MetaRouter, and scale-dependent efficiency experiments.

  • Self-attention costs O(n^2d), while existing efficient transformers use fixed computational patterns regardless of token information content.
  • Spectral entropy measures a token embedding’s frequency-domain complexity and distinguishes tokens suited to cheap spectral mixing from those benefiting from full attention.
  • CHIAR-Former applies the chiaroscuro principle by spending attention compute on high-entropy signals and DCT mixing on smooth, low-entropy tokens.
  • The paper contributes routing-collapse analysis, theory-grounded entropy routing, a learned MetaRouter, and scale experiments spanning efficiency–quality trade-offs.

2 Related Work

Related work includes foundational attention models, fixed-pattern efficient attention, spectral mixers, mixture-of-experts routing, state-space alternatives, and positional encodings. CHIAR-Former differs by combining spectral and attention operators with content-based routing within a standard transformer.

  • Foundational transformers established multi-head attention, bidirectional pre-training, few-shot learning at scale, and compute-based scaling laws.
  • Efficient attention methods reduce cost through sparse, low-rank, randomized, hashed, or IO-aware patterns, but apply fixed computation without per-token content routing.
  • Spectral mixers replace or modify attention with Fourier-based operations, while CHIAR-Former adds entropy-based routing and a learned spectral filter.
  • Mixture-of-experts models route tokens to expert networks, whereas CHIAR-Former routes them between spectral and dynamic operators using a theory-driven signal.
  • State-space models provide linear-time sequence modeling alternatives, while CHIAR-Former hybridizes spectral and attention operators within the standard transformer architecture.
  • CHIAR-Former uses RoPE for relative positional encoding, alongside ALiBi as an alternative attention-bias approach.
  • The MetaRouter’s stabilization at g ≈0.22 is presented as consistent with attention absorbing complexity while spectral preprocessing contributes stable low-frequency structure at scale.

3 CHIAR-Former Architecture

CHIAR-Former combines spectral entropy, DCT mixing, per-token operator routing, a task-level MetaRouter, and a final attention anchor within a parameter-matched transformer. The architecture uses spectral processing for low-complexity structure while retaining attention where greater expressiveness is needed.

  • 3.1 Spectral Entropy: Spectral entropy normalizes a token’s DCT power spectrum to quantify frequency-domain complexity between 0 and 1.
  • 3.1 Spectral Entropy: H = 0 denotes a pure single-frequency signal, whereas H = 1 denotes a uniform, maximally complex spectrum.
  • 3.2 Theoretical Motivation: Under an approximately Toeplitz covariance assumption, DCT provides near-optimal energy compaction for low-entropy signals in O(d log d).
  • 3.3 Layer Architecture: The model evolves through versions to an N-layer architecture with DCT mixing at L1, routed DCT or attention in intermediate layers, and full attention at LN.
  • 3.3 Layer Architecture: Each layer shares an FFN, while the spectral branch applies a learned frequency filter, inverse DCT, FFN, and layer normalization to sequence embeddings.
  • 3.3 Layer Architecture: The SpectralRouter sets τ from entropy percentiles so approximately half of tokens use each operator, with τ = 0.8954 at 400M scale.
  • 3.4 Learned MetaRouter: The MetaRouter computes a task-level gate from the batch-mean embedding and soft-blends DCT-mixed and identity paths before subsequent layers.
  • 3.5 Parameter Parity: At N = 28 and d = 1024, parameter parity is maintained: Baseline has 404M parameters and CHIAR has 400M.

4 Routing Collapse

A three-operator routing design collapsed to a two-operator system during training, with all tokens selecting either DCT or attention. The collapse identified DCT+Attention as the operator subset used thereafter.

  • RBF routing fell to 0% by epoch 3 in a 16M run, leaving all tokens divided between DCT and attention.
  • The collapse was interpreted as revealing DCT+Attention as the optimal operator subset, so subsequent experiments removed RBF routing.

5 Experiments

Experiments evaluate CHIAR across 16M and 400M WikiText settings, showing small-scale performance costs and a larger-scale efficiency–quality trade-off. Mixed-dataset training further improves generalisation on small corpora through spectral regularisation.

  • Scaling and ablations: CHIAR variants are slightly worse at 16M, while the efficiency trade-off is evaluated at 400M.The 16M experiments establish routing collapse and ablate routing mechanisms.
  • 400M WikiText-103: 27.51 Test PPL vs. 23.58, with 4M fewer parameters and ∼37% fewer total FLOPs at 400M.Hard routing executes exactly one operator per token in L2–L27, reducing attention FLOPs in routing layers by 62.5%.
  • Scaling and ablations: 3–4 points: the approximate Test PPL gap across 16M and 400M scales, while compute savings increase with model size.Figure 1 summarizes the scaling comparison.
  • Training stability: Stable convergence is observed for the baseline, CHIAR standalone, and CHIAR mixed-training conditions during 400M training.Figure 2 reports validation PPL trajectories for these three conditions.
  • Mixed-dataset training: 28.56 Test PPL vs. 27.51 for single-dataset CHIAR, a 1.05 PPL cost for mixed-dataset breadth on WikiText-103.Mixed batches draw from WikiText-103, WikiText-2, IMDB, and ListOps.
  • Mixed-dataset training: Mixed training with spectral regularisation dramatically improves generalisation over full attention on WikiText-2, where overfitting is the primary challenge.WikiText-2 contains 2.4M tokens, and the result is attributed to DCT energy compaction acting as a structural prior on limited data.

6 Analysis

The analyses show entropy-aligned routing, stable MetaRouter adaptation, smooth convergence, and distinct efficiency–quality regimes across scales.

  • Routing Heatmap: Function words route to DCT and content words to Attention, producing a 50/50 split on the prompt.
  • MetaRouter Learning Dynamics: g descends from 0.50 to a stable equilibrium near 0.22, indicating continued but limited spectral preprocessing at scale.The soft-blend formulation still computes DCT regardless of g; hard inference gating is left for future work.
  • Training Curves: All CHIAR variants converge smoothly during the 16M ablation training.
  • Scale Results: Test PPL is 27.51 for CHIAR versus 23.58 for the baseline on the same 400M schedule.
  • Operating Regimes: 37% FLOP reduction at 400M incurs a 3.93 PPL cost on WikiText-103, while spectral mixing provides stronger regularisation on small corpora.

7 Conclusion

CHIAR-Former routes tokens between spectral mixing and full attention according to spectral entropy, combining efficient processing with targeted representational capacity. Its results identify DCT plus attention as the effective operator pair, with substantial compute savings and stable scale-dependent behaviour, while future work targets harder gating and broader evaluation.

  • CHIAR-Former routes tokens between DCT spectral mixing and full self-attention using per-token spectral entropy H(x).
  • Routing collapse identifies DCT+Attention as the optimal operator pair.
  • 37% FLOP reduction at 400M comes with a 3.93 PPL cost on WikiText-103, alongside strong spectral regularisation on small corpora.
  • The MetaRouter stabilises near g ≈0.22, reflecting complementary roles for spectral preprocessing and attention at scale.
  • Future work includes hard-gating the MetaRouter at inference, scaling beyond 1B parameters, and fine-tuning on GLUE and SuperGLUE.

A CHIAR-Former Architecture Diagrams (v1, v2, v3)

CHIAR-Former evolves from a three-operator v1 to a DCT-plus-attention v2 and a 400M-parameter v3 with RoPE and a learned MetaRouter. Across versions, routing progressively combines cheaper spectral processing with attention while preserving a fixed full-attention accuracy anchor.

  • v1 — Original CHIAR-Former: v1 routes tokens among DCT Mixing, RBF Mixing, and Full Attention, with fixed DCT processing at L1 and a fixed Full Attention anchor at L4.The learned Spectral Router operates at L2 and L3; v1 has 16M parameters and Absolute PE.
  • Evolution across versions: RBF routing collapsed to 0% in the initial 16M experiments, leading subsequent versions to use DCT+Attention as the operator subset.The collapse was treated as evidence identifying the sufficient two-operator design rather than as a failure.
  • v2 — DCT+Attn Validated: v2 removes RBF Mixing and uses binary spectral-entropy gating: H(x) ≤τ selects DCT Mixing, while H(x) > τ selects Full Attention.L1 remains all-DCT, and L4 remains the full-attention accuracy anchor.
  • v3 — RoPE + Learned MetaRouter: v3 replaces Absolute PE with RoPE and adds a learned MetaRouter that soft-blends DCT Mixing with an Identity bypass at L1.The gate uses the batch-and-sequence mean embedding and sigmoid, while later layers retain per-token spectral routing.
  • v3 — RoPE + Learned MetaRouter: 37% FLOP reduction is reported for v3 at 400M parameters, with Test PPL 27.51 versus the 23.58 baseline.The MetaRouter stabilises at g ≈0.22, and L28 remains a fixed Full Attention accuracy anchor.
Loading 2606.08327v2…