PaperlayerView exact revision on arXiv

Source-linked AI summary

Hyena Hierarchy: Towards Larger Convolutional Language Models

Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y. Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, Christopher Ré

arXiv:2302.10866v3cs.LGcs.CL

TL;DR

Transformers’ attention is powerful but has quadratic sequence-length cost, while existing subquadratic alternatives do not match Transformer quality without hybridization. Hyena replaces attention with interleaved implicitly parametrized long convolutions and data-controlled gating, matching Transformer-quality language modeling with lower compute and much faster long-sequence execution.

  • Problem

    Attention’s quadratic cost limits accessible context, while existing subquadratic methods require dense-attention hybridization to reach Transformer quality.

  • Method

    Hyena is a subquadratic attention replacement that interleaves implicitly parametrized long convolutions with data-controlled multiplicative gating.

  • Results

    Hyena matches Transformer quality on language modeling, including The Pile with 20% fewer FLOPs at 335M parameters, and achieves 100x speedup over FlashAttention at sequence length 64k.

  • Takeaways & Limitations

    The results support attention-free convolutional architectures as a viable route to Transformer-quality modeling with subquadratic computation and unrestricted long context.

  • Takeaways & Limitations

    Self-attention accesses distant information with O(L^2) operations, while autoregressive Hyena requires causal convolutions to ensure outputs depend only on the past.

Abstract

from arXiv · show

Recent advances in deep learning have relied heavily on the use of large Transformers due to their ability to learn at scale. However, the core building block of Transformers, the attention operator, exhibits quadratic cost in sequence length, limiting the amount of context accessible. Existing subquadratic methods based on low-rank and sparse approximations need to be combined with dense attention layers to match Transformers, indicating a gap in capability. In this work, we propose Hyena, a subquadratic drop-in replacement for attention constructed by interleaving implicitly parametrized long convolutions and data-controlled gating. In recall and reasoning tasks on sequences of thousands to hundreds of thousands of tokens, Hyena improves accuracy by more than 50 points over operators relying on state-spaces and other implicit and explicit methods, matching attention-based models. We set a new state-of-the-art for dense-attention-free architectures on language modeling in standard datasets (WikiText103 and The Pile), reaching Transformer quality with a 20% reduction in training compute required at sequence length 2K. Hyena operators are twice as fast as highly optimized attention at sequence length 8K, and 100x faster at sequence length 64K.

1 Introduction

Transformers provide strong scaling and in-context learning, but attention’s quadratic sequence-length cost limits accessible context. Hyena addresses this gap with subquadratic convolutions and gating, matching Transformer quality while reducing compute and improving long-sequence speed.

  • Motivation: Quadratic attention cost in sequence length limits the context Transformers can process.The paper identifies breaking this barrier as important for applications requiring very long inputs.
  • Motivation: Existing linearized, low-rank, and sparse attention alternatives trade expressivity for speed and require hybridization with standard attention to reach Transformer quality.
  • Hyena hierarchy: Hyena composes long convolutions and element-wise multiplicative gating into a data-controlled, subquadratic operator.The recurrence depth controls operator size, and fast convolution algorithms enable efficient evaluation without materializing the full matrix.
  • Experiments: Over 50% accuracy improvement is reported on the most challenging hundreds-of-thousands-token reasoning settings versus state-space, frequency-domain, and standard convolution operators.
  • Experiments: 20% fewer FLOPs achieves Transformer perplexity on The Pile at 335M parameters, establishing a state-of-the-art for dense-attention-free language architectures.
  • Long-context efficiency: 100x speedup over FlashAttention is reported at sequence length 64k, where standard PyTorch attention runs out of memory.At length 8192, the paper reports 5x speedup over dense self-attention and 2x over highly optimized FlashAttention2.

2 Preliminaries and Related Work

This section introduces convolutional representations, fast FFT-based evaluation, and implicit parameterizations for long sequences, then relates them to data-controlled self-attention and subquadratic alternatives.

  • Convolutions: A discrete convolution combines a length-L input signal with a learnable filter, and can be represented as multiplication by an induced Toeplitz matrix.The discussion specializes to single-input single-output layers while noting that the multiple-input multiple-output case follows directly.
  • Explicit and implicit parameterizations: Explicit filters learn response values at prescribed steps, whereas implicit filters represent ht as a parameterized function of time t.Implicit parameterization decouples filter length from parameter count, while its function class affects expressivity and computational complexity.
  • Memory and scaling: Implicit convolutions separate memory length from parameter count, unlike finite impulse response filters whose memory is limited by filter size M.FIR filters have O(ML) computation, but their parameter count scales linearly with M.
  • Fast convolution: O(L log2 L) FFT-based convolution avoids materializing the Toeplitz operator and accelerates long convolution through the convolution theorem.Zero-padding converts aperiodic convolution into circular convolution, which is diagonalized in the discrete Fourier basis.
  • Self-attention and alternatives: Self-attention is a data-controlled operator that indexes a family of dense linear transformations using input-derived query, key, and value projections.It accesses the whole sequence without entangling distant-context access with parameter count, but requires O(L2) operations.
  • Self-attention and alternatives: Subquadratic alternatives modify how data control is implemented, including combinations of gating, SoftMax, explicit convolution, and state-space mechanisms.These approaches are presented as alternatives that alter the nonlinear dependence of the operator on the input.

3 Hyena: Definition and Properties

Hyena is a data-controlled operator built by interleaving implicit long convolutions with element-wise multiplicative gating. Its recurrence provides unbounded context with subquadratic evaluation and causal autoregressive use.

  • Definition: Hyena interleaves long convolutions and element-wise multiplicative gating to define a data-controlled operator without materializing its matrix.The convolutions are implicitly parameterized so parameter scaling remains sublinear in sequence length.
  • Definition: An order-N Hyena recurrence uses N projected inputs plus a value projection, with its depth controlling the operator size.Short recurrences recover existing models as special cases, including GSS as Hyena1 and H3 as Hyena2.
  • Complexity and memory: O(NL log2 L) is the time complexity of a Hyena recurrence when each long convolution is evaluated through the Fourier domain.The recurrence alternates time-domain multiplication with convolution, whose frequency-domain counterpart helps expand memory length.
  • Complexity and memory: Hyena operators have unbounded context and can learn long-range dependencies between any elements of the value sequence through long convolutions.The operator is not artificially restricted by locality.
  • Filter parameterization: Hyena filters are generated by an FFN applied to positional encodings and modulated by a window, decoupling filter length from parameter cost.Exponential decay windows and varied channel-wise decay rates specialize filters toward different lengths and behaviors.
  • Causality and implementation: A Hyena operator is causal when every filter is causal, ensuring outputs depend only on the past for autoregressive language modeling.FFT evaluation uses sequence and filter zero-padding to compute the convolution efficiently.
  • Causality and implementation: Hyena evaluates filters with parallel FFN passes and uses FFT-based convolutions to improve hardware utilization during the forward pass.The algorithm computes filters across sequence positions and operator orders before splitting them into convolution filters and projections.

4 Experiments

The experiments evaluate Hyena’s long-convolution parametrizations and attention-free performance across reasoning, language-modeling, runtime, and vision tasks. Hyena matches or improves on Transformer-quality results while offering advantages on long sequences and compute.

  • Experimental setup: Hyena combines implicit FFN-based long-convolution parametrizations with short explicit filters.The compared parametrizations include explicit convolutions, frequency-domain filters, state-space models, transfer functions, FFNs, and Hyena’s combined design.
  • In-context learning: 80 points: Hyena outperforms CKConv on associative recall at sequence length 131k.The gap between convolution parametrization schemes widens on extremely long sequences.
  • In-context learning: Hyena is the only evaluated operator able to solve the associative-recall task in the operator comparison.The comparison includes order 2 Hyena, GSS, H3, AFT-conv, RWKV, and standard GPT with FlashAttention.
  • Vision: 91%: Hyena reaches standard S4 accuracy on sequential CIFAR with the same model size.The same operator defined for language is applied to flattened pixel sequences.
  • Language modeling: 20% reduction in total FLOPs: Hyena matches GPT quality on The Pile while setting a state-of-the-art for dense-attention-free architectures.The reported result concerns autoregressive language modeling and uses different runs at 5, 10, and 15 billion tokens.
  • Runtime: 100×: Hyena’s runtime speedup reaches this level at sequence length 64K, with crossover against attention at length 2048.The benchmark compares order 2 Hyena with attention and FlashAttention using batch size 64; speedups emerge on longer sequences because Hyena has lower hardware utilization.
  • Vision: Hyena matches ViT performance when replacing its attention layers and improves on S4ND in CIFAR-2D with 8% speedup and 25% fewer parameters.The ViT replacement uses Hyena without changes from its language counterpart.

5 Discussion and Conclusion

Hyena is presented as an attention-free, subquadratic replacement that learns in-context on very long sequences. At sub-billion scale, it matches Transformer language-modeling quality with reduced training compute.

  • Hyena operators combine gating with implicitly parametrized long convolutions and can be evaluated efficiently in subquadratic time.
  • Hyena learns in-context on very long sequences without attention.
  • On The Pile, deep Hyena stacks match Transformer perplexity and downstream performance while substantially reducing training compute.
  • Results at the sub-billion parameter scale suggest that attention-free convolutional architectures can support efficient large-model development.

A Experimental Details

The paper provides an implementation of Hyena through an external link.

  • An implementation of Hyena is available through the linked repository.

A.1 Mechanistic Design Synthetic Benchmarks

The synthetic benchmark suite tests mechanistic capabilities such as recall, induction, counting, function in-context learning, and arithmetic across increasingly difficult sequence settings. Hyena’s implicit convolutional parameterization performs especially well on long associative recall, while Transformer performance depends on sequence length and available data.

  • Mechanistic Design Synthetic Benchmarks: The evaluation covers associative recall, majority voting and counting, in-context learning of linear functions, and arithmetic.
  • Mechanistic Design Synthetic Benchmarks: Difficulty increases through sequence lengths from 1024 to 131136 tokens and vocabulary sizes from 10 to 40.
  • Mechanistic Design Synthetic Benchmarks: Repeated key-value tuples in long associative-recall prompts let models with farther effective context see more data.
  • Mechanistic Design Synthetic Benchmarks: Hyena combines implicit FFN-based parameterization with exponentially decayed modulation and short explicit filters.
  • Mechanistic Design Synthetic Benchmarks: Implicit convolutional parameterizations outperform explicit ones on associative recall, with CKConv and Hyena improving extraction of key-value relations.
  • Mechanistic Design Synthetic Benchmarks: Transformers solve longer associative-recall sequences when memory permits and training includes enough examples, but struggle under the fixed 2000-sample regime.
  • Mechanistic Design Synthetic Benchmarks: For shorter sequences, Transformers solve the task easily with limited data, comparably to Hyena.
  • Mechanistic Design Synthetic Benchmarks: For sequences up to 8k, hybridizing Hyena with attention-free and attention-based token-mixing layers can improve downstream performance when that is the only metric.

A.2 Language Modeling

Hyena is evaluated for language modeling on WikiText103, The Pile, and PG-19 using standard-scale training setups. The experiments include compute accounting, preliminary larger-model results, and an explicit caveat that GPT-tuned hyperparameters may be suboptimal for Hyena.

  • Language Modeling: WikiText103 experiments train 125M-parameter models and compare Hyena perplexity with Transformers, hybrids, and other subquadratic attention variants.
  • Language Modeling: The Pile experiments train 125M- and 355M-sized models, with preliminary results also reported at 1.3B parameters.
  • Language Modeling: Training uses 5, 10, and 15 billion tokens at sequence length 2024, with global batch size 256 on eight A100 80GB GPUs.
  • Language Modeling: Standard GPT hyperparameters are used for both models despite evidence that they are likely suboptimal for Hyena.
  • Language Modeling: Hyena 153M reaches a test perplexity of 14.6 on PG-19 with a 16k-token context length after 8 epochs.
  • Language Modeling: The experiments use sine activations in Hyena filter FFNs and document the corresponding architecture hyperparameters.
  • Language Modeling: The reported FLOP accounting replaces attention with short projection convolutions and FFT-based long convolutions.

A.3 Downstream Evaluation

Hyena is evaluated on language understanding, language modeling, and vision benchmarks against similarly sized sequence models and attention-based architectures. The evaluations cover both task-specific performance and cross-domain replacement of attention layers.

  • Language evaluation: SuperGLUE evaluation uses greedy decoding for most tasks, while WIC, CB, and BoolQ use logit scoring.The parsing pipeline follows Arora et al. (2022).
  • Language evaluation: Hyena, GPTNeo, and RWKV are compared using checkpoints trained on The Pile at similar parameter scales.GPTNeo has 125M parameters, RWKV-v4 has 169M, and Hyena has 153M.
  • Language evaluation: 44.64% accuracy is reached by the small Hyena model on LAMBADA after training on 137B tokens.The evaluation filters stop words and requires all tokens of the final word to match the ground truth.
  • Vision evaluation: On ImageNet-1k, Hyena replaces ViT attention layers in a similarly sized model, using 88M parameters versus 87M for ViT-B.The models are trained from scratch without outside data on eight Nvidia A100 GPUs.
  • Vision evaluation: On CIFAR-10, sequential experiments compare Hyena with an equal-sized S4 model, while 2D experiments use windowed filters without language-task gating.The sequential setup swaps layers in residual blocks; the 2D setup learns filters along both spatial dimensions.

B Theoretical Results and Details

The theoretical treatment characterizes Hyena as alternating diagonal and Toeplitz operators, establishing causality under causal filters and relating the construction to a surrogate attention mechanism. Fourier analysis explains how gating prevents the layer from collapsing into a simple convolution.

  • Causality: A Hyena operator is causal when every filter in its recurrence is causal.Causality follows because causal filters induce lower-triangular Toeplitz matrices, whose alternating product remains lower triangular.
  • Surrogate attention: The surrogate attention map is conditioned on queries, keys, filters, and values, and is represented as a linear operator mapping v to y.Its factors depend on projections of the input and can be decomposed into alternating diagonal and convolutional terms.
  • Surrogate attention: The surrogate attention matrix combines diagonal query and key gates with Toeplitz convolution kernels.The construction uses matrices formed from Dq, Dk, Sψ, and Sϕ, with causal filters producing lower-triangular kernels.
  • Fourier analysis: Fourier decomposition diagonalizes the convolution operators, expressing the matrix as DqW*DΨWDkW*DΦW.The Fourier-domain representation exposes the interaction between frequency responses and data-dependent gates.
  • Fourier analysis: Non-commutativity between gating terms and the Fourier transform acts as a non-linearity in the chain of convolution operators.If the operators commuted, the entire layer would reduce to a simple convolution.

C Discussion and Additional Results

Additional experiments examine whether synthetic reasoning tasks predict scaled language-model performance and how Hyena handles recall, arithmetic, and matrix behavior. Results show strong recall and arithmetic capability, while visualizations distinguish Hyena’s data-controlled matrices from attention.

  • Scaling and recall: Synthetic benchmark results correlate with language-model loss at scale and may predict performance when Hyena replaces attention in other domains.The paper reports similar behavior on other mechanistic design tasks and links language synthetics to image-classification results.
  • Scaling and recall: Associative-recall performance at vocabulary sizes 10, 20, 30, and 40 is compared with The Pile test loss after 5B tokens.The sequence length is fixed at 2048 for both the recall experiments and The Pile training.
  • Scaling and recall: A single width-64 Hyena layer solves associative recall completely with vocabulary size 40.The paper leaves the exact mechanism behind this single-layer recall ability for future work.
  • Arithmetic: A single Hyena layer learns addition with up to 4 digits, while longer numbers require deeper models.Alternative architectures such as AFT-conv struggle to learn arithmetic.
  • Matrix visualizations: Hyena matrices are visualized as element-wise absolute values, unlike attention matrices shown after softmax.Hyena entries can be positive or negative and have unconstrained magnitude; pretrained matrix magnitudes are around 10^-3.

D.2 Hyena Filters

Hyena filter behavior depends strongly on initialization and positional encoding. Training learns structured lower-order filters, while positional features control spectral bias and create a trade-off between initialization quality and parameter count.

  • Filter initialization: Up to 5% perplexity separates different filter initialization schemes.Excessively smooth initial filters produce worse solutions and slower convergence.
  • Filter initialization: At convergence, Hyena learns collections of similarly structured lower-order filters that can speed inference after training.The learned structure provides an opportunity for post-training optimization.
  • Positional encoding: The positional encoding maps time to 2K + 1 features using a truncated complex exponential basis.The real and imaginary parts of the basis functions form the positional features.
  • Positional encoding: Increasing K biases initialized filters toward low-pass behavior with an approximate cutoff frequency of 2K + 1.The choice of K preconditions the filter spectrum and affects initialization and training performance.
  • Positional encoding: Richer high-frequency initialization can improve training dynamics, but increasing K enlarges the feed-forward networks and parameter count.Increasing the sinusoidal activation frequency is presented as a more efficient alternative.
  • Filter visualizations: Figures compare filters at initialization and after 130B-token training, alongside initializations using different positional-feature counts and activation frequencies.The visualizations include K = 8, K = 32, K = 64, and a frequency setting of 10.
Loading 2302.10866v3…