HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution

Eric Nguyen; Michael Poli; Marjan Faizi; Armin Thomas; Callum Birch-Sykes; Michael Wornow; Aman Patel; Clayton Rabideau; Stefano Massaroli; Yoshua Bengio; Stefano Ermon; Stephen A. Baccus; Chris Ré

Source-linked AI summary

HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution

Eric Nguyen, Michael Poli, Marjan Faizi, Armin Thomas, Callum Birch-Sykes, Michael Wornow, Aman Patel, Clayton Rabideau, Stefano Massaroli, Yoshua Bengio, Stefano Ermon, Stephen A. Baccus, Chris Ré

arXiv:2306.15794v2Jun 27, 2023cs.LGq-bio.GN

TL;DR

Existing genomic models are limited by attention’s quadratic scaling and token aggregation, restricting long-range modeling and single-nucleotide resolution. HyenaDNA addresses these constraints with long-context, single-nucleotide genomic modeling and achieves strong results across downstream benchmarks, while remaining limited by pretraining on one human reference genome.

Problem
Attention scales as O(L^2), limiting genomic context lengths, while tokenizers and fixed k-mers lose single-nucleotide resolution needed to represent fine genetic variation.
Method
HyenaDNA uses a decoder-only Hyena architecture with single-character DNA tokens, ultralong contexts up to 1 million tokens, sequence-length warm-up, and downstream soft-prompting or in-context adaptation.
Results
HyenaDNA reaches state-of-the-art on 12 of 18 Nucleotide Transformer datasets and surpasses state-of-the-art on 7 of 8 GenomicBenchmarks datasets by an average of +10 accuracy points.
Takeaways & Limitations
HyenaDNA demonstrates that genomic foundation models can combine million-token context with single-nucleotide resolution and support in-context learning for downstream adaptation.
Takeaways & Limitations
HyenaDNA was pretrained on only one human reference genome, which may constrain generalizability and contribute to bias in learned features.

Abstract

from arXiv · show

Genomic (DNA) sequences encode an enormous amount of information for gene regulation and protein synthesis. Similar to natural language models, researchers have proposed foundation models in genomics to learn generalizable features from unlabeled genome data that can then be fine-tuned for downstream tasks such as identifying regulatory elements. Due to the quadratic scaling of attention, previous Transformer-based genomic models have used 512 to 4k tokens as context (<0.001% of the human genome), significantly limiting the modeling of long-range interactions in DNA. In addition, these methods rely on tokenizers or fixed k-mers to aggregate meaningful DNA units, losing single nucleotide resolution where subtle genetic variations can completely alter protein function via single nucleotide polymorphisms (SNPs). Recently, Hyena, a large language model based on implicit convolutions was shown to match attention in quality while allowing longer context lengths and lower time complexity. Leveraging Hyena's new long-range capabilities, we present HyenaDNA, a genomic foundation model pretrained on the human reference genome with context lengths of up to 1 million tokens at the single nucleotide-level - an up to 500x increase over previous dense attention-based models. HyenaDNA scales sub-quadratically in sequence length (training up to 160x faster than Transformer), uses single nucleotide tokens, and has full global context at each layer. We explore what longer context enables - including the first use of in-context learning in genomics. On fine-tuned benchmarks from the Nucleotide Transformer, HyenaDNA reaches state-of-the-art (SotA) on 12 of 18 datasets using a model with orders of magnitude less parameters and pretraining data. On the GenomicBenchmarks, HyenaDNA surpasses SotA on 7 of 8 datasets on average by +10 accuracy points. Code at https://github.com/HazyResearch/hyena-dna.

1 Introduction

HyenaDNA addresses the challenge of modeling genomic sequences that are both extremely long and sensitive to single-nucleotide changes. It combines Hyena operators, single-nucleotide resolution, long-context training, and downstream adaptation procedures, achieving strong benchmark performance.

Limitations of current models: Current genomic models typically use 512 to 4,096 tokens, or <0.001% of the human genome, because attention scales quadratically with sequence length.Fixed k-mers and tokenizers also aggregate nucleotides, potentially losing single-nucleotide resolution relevant to SNPs and mutations.
Toward longer context models: HyenaDNA uses a stack of Hyena operators with implicit long convolutions and data-controlled gating to process long genomic contexts.The long convolutions are parameterized through an MLP and evaluated with FFT convolution, while gating provides context-specific operations over tokens.
HyenaDNA: HyenaDNA is pretrained on the human reference genome with up to 1 million single-nucleotide tokens, an up to 500x increase over dense-attention genomic models.The model scales sub-quadratically, trains up to 160x faster than attention at sequence length 1M, and provides a global receptive field at each layer.
HyenaDNA: A sequence-length warm-up scheduler stabilizes ultralong-sequence training; at length 450k, training time falls 40% while species-classification accuracy rises 7.5 points.The model also uses downstream adaptation procedures based on soft prompting and in-context learning rather than standard fine-tuning.
Genomic downstream tasks: HyenaDNA achieves state-of-the-art results on 12 of 18 Nucleotide Transformer datasets and surpasses SotA on 7 of 8 GenomicBenchmarks datasets.On GenomicBenchmarks, the average improvement is +10 accuracy points, reaching as much as +20 points for enhancer function identification.

2 Preliminaries and Related Work

Attention provides global, high-resolution context but scales quadratically with sequence length, limiting genomic context. Long-context alternatives use convolutions or aggregation strategies, but existing genomic approaches often sacrifice single-nucleotide resolution.

Transformers and Attention: Attention compares every token pair, giving global context at high resolution but scaling as O(L^2) and limiting context on current hardware.The embedding dimension is D, and learned projections Wq, Wk, and Wv define the attention operation.
Transformers and Attention: Sparse and linear attention reduce time complexity by approximating dense attention, trading longer sequences for reduced expressivity.
Long Context Strategies in Genomics: Genomic models extend context through fixed k-mers or BPE tokenization, and through dilation or downsampling that aggregates or skips sequence elements.Enformer uses dilation and downsampling to reach 100k-nucleotide contexts for gene-expression prediction.
Long Context Strategies in Genomics: These genomic strategies sacrifice single-nucleotide resolution, which limits fine-grained representation while targeting longer context.
Large Convolutional Models: Hyena combines long convolutions with implicit parameterization to model long sequences without attention, motivating attention-free genomic models with extended context.HyenaDNA uses these capabilities to explore in-context learning for adapting to genomic tasks without updating pretrained models.

3 HyenaDNA Long-Range Genomic Foundation Models

HyenaDNA is a decoder-only genomic model that replaces attention with Hyena operators, processes nucleotides directly, and uses sequence-length warm-up for ultralong training. Its design combines long convolutions, data-dependent gating, and efficient downstream adaptation.

The HyenaDNA Model: HyenaDNA uses decoder-only blocks with a Hyena operator followed by a feed-forward network.
The HyenaDNA Model: The Hyena2 operator combines projected inputs, a learnable long convolution filter, and data-dependent diagonal gating.The filter is generated by a neural network from position indices, while the gating matrices are constructed from input projections.
The HyenaDNA Model: O(L log^2 L) is the evaluation time of a Hyena operator, making efficient computation important for extremely long genomic sequences.For embedding dimension D > 1, independent Hyena operators are applied across dimensions after linear projections.
Tokenization: HyenaDNA processes ultralong DNA sequences at single-nucleotide resolution without frequency-based aggregation tokenizers.Its vocabulary includes A, G, C, T, N, and special tokens, with nucleotides mapped to embedding dimension D.
Training Long Sequence Models: Sequence-length warm-up gradually doubles the window from L1 = 64 while keeping global batch size constant.At 450k sequence length, the schedule reduces training time by 40% and improves species-classification accuracy by 7.5% points.
Downstream Adaptation: Soft prompting optimizes trainable prompt parameters while keeping all pretrained model parameters fixed.The prompt is prepended after embedding and optimized using a small subset of prompt-label pairs.

4 Experiments

The experiments evaluate HyenaDNA across pretraining, short-range classification, in-context learning, chromatin prediction, embedding quality, and ultralong-range species classification. Results examine efficiency, single-nucleotide resolution, and the capabilities enabled by longer genomic context.

Pretraining on the Human Genome: At sequence length 1M, HyenaDNA is 160x faster than its Transformer counterpart.The runtime comparison uses 2 layers, width=128, gradient checkpointing, batch size=1, and an A100 80GB.
Pretraining on the Human Genome: As context length increases, perplexity improves during pretraining, although longer contexts require more training time and tokens.Models that are too shallow to process longer contexts effectively can instead show perplexity degradation.
Single Nucleotide Resolution: HyenaDNA surpasses SotA on 12 of 18 Nucleotide Transformer datasets using a model with orders of magnitude less parameters and pretraining data.The benchmarks cover regulatory-element prediction for enhancers, promoters, epigenetic marks, and splice sites from 200-600 nucleotide sequences; Table 4.2 uses MCC or F1-score depending on the dataset.
In-context Learning for Genomic Sequences: Performance on novel tasks improves as more tuneable tokens are added and saturates close to baseline performance, except on the Human Regulatory dataset.The in-context-learning experiments use soft prompting or instruction fine-tuning because DNA’s small vocabulary lacks new classification symbols.
Species Classification: HyenaDNA effectively solves species classification with context lengths of 450k to 1 million, while Transformer cannot because of infeasible training time.Both models struggle at length 1024; longer contexts expose distinct mutational profiles despite low cross-species sequence divergence.

5 Conclusion

HyenaDNA demonstrates long-context, single-nucleotide genomic modeling and introduces in-context learning for downstream adaptation. The study identifies broader pretraining data and multimodal biological sequences as future directions.

5 Conclusion: HyenaDNA supports context lengths up to 1 million tokens at single nucleotide resolution and enables in-context learning for genomic tasks.The model is presented as a genomic foundation model pretrained on the human reference genome.
5 Conclusion: Incorporating genomes from multiple humans and species could increase feature generalizability and reduce bias.
5 Conclusion: Extending the framework beyond DNA to proteins and drug molecules could support multimodal biological foundation models.
5 Conclusion: Long-context generative modeling may support designing synthetic regulatory elements, genes, and protein complexes.

A Appendix: Experimental Details

The experiments use PyTorch-based training across several NVIDIA GPU types with cross-entropy as the default objective, and the code is publicly available.

A Appendix: Experimental Details: The experiments use PyTorch and PyTorch Lightning throughout.
A Appendix: Experimental Details: Training uses a mix of NVIDIA A100, V100, and T4 GPUs.
A Appendix: Experimental Details: Unless otherwise stated, the objective is cross entropy, and the repository is publicly available.

A.1 Pretraining Details

The appendix describes pretraining data, compact HyenaDNA architectures, long-sequence training, and compute comparisons across pretrained models.

A.1 Pretraining Details: Pretraining uses one human reference genome, with chromosome 14 and X reserved for non-overlapping test sequences.
A.1 Pretraining Details: HyenaDNA models span 2–8 layers, widths of 128–256, 400k–6.6M parameters, and sequence lengths from 1,024 to 1M.
A.1 Pretraining Details: The largest 1M-context model was trained on 2T tokens over 4 weeks while maintaining a consistent global batch size.
A.1 Pretraining Details: Training-efficiency comparisons use pretraining compute resources and GPU-hours required to reach competitive short-range-task performance.

A.2.1 GenomicBenchmarks experiment

The GenomicBenchmarks experiment evaluates sequence-level regulatory-element classification and species classification using PyTorch-aligned baselines and a pretrained HyenaDNA backbone.

A.2.1 GenomicBenchmarks experiment: GenomicBenchmarks contains 8 regulatory-element sequence-classification datasets and one binary species task.
A.2.1 GenomicBenchmarks experiment: The appendix provides GPU and runtime comparisons for short-range models and hyperparameter settings for HyenaDNA and a FlashAttention Transformer baseline.
A.2.1 GenomicBenchmarks experiment: The study compares against the PyTorch-based benchmark results because its implementation uses PyTorch.
A.2.1 GenomicBenchmarks experiment: The HyenaDNA classifier uses a pretrained 2-layer, width-128 model trained at sequence length 1024, pooled into a classification token with a linear decoder.
A.2.1 GenomicBenchmarks experiment: The experiment sweeps learning rate, global batch size, dropout, weight decay, and reverse-complement augmentation.

A.2.2 Ablations on the GenomicBenchmarks

Ablations show that single-nucleotide tokenization and causal modeling are important components of HyenaDNA’s GenomicBenchmarks performance, while pretraining provides only mild to moderate gains.

Pretraining: Pretrained models provide mild to moderate gains over models trained from scratch on the GenomicBenchmarks.The authors attribute the limited gains likely to near-saturated benchmark performance.
Tokenization: Up to 10 accuracy points are lost across most datasets when single-nucleotide tokenization is replaced with a k-mer tokenizer, although one dataset improves.The k-mer tokenizer boosts Human Enhancer Ensembl while reducing performance elsewhere.
Bidirectional: The bidirectional variant degrades performance on 7 of 8 datasets, averaging 3.8 accuracy points below standard causal HyenaDNA.Both variants are trained from scratch for this comparison.
Bidirectional: The bidirectional implementation uses circular FFT convolution with symmetric padding to provide a bidirectional receptive field.The input is padded by half the sequence length on both the left and right sides before convolution.

A.2.3 Downstream prediction tasks for Nucleotide Transformer benchmark

The Nucleotide Transformer benchmark suite covers regulatory, epigenetic, splice-site, in-context, chromatin, and gene-biotype prediction tasks. Across these evaluations, HyenaDNA benefits from pretraining on harder tasks and achieves strong performance with compact representations and models.

Datasets: The benchmark suite combines datasets from four sources covering promoter, enhancer, epigenetic-mark, and splice-site prediction.Promoter data include human and mouse sequences; splice-site data span more than 100 organisms.
Regulatory prediction: Promoter tasks classify TATA-box-containing, TATA-box-lacking, and non-promoter sequences, while enhancer tasks distinguish enhancer status and enhancer types.The promoter sequences are drawn from regions around transcription start sites.
Epigenetic prediction: Epigenetic-mark tasks comprise 10 binary classifications of histone occupancy and modification states in the yeast genome.The tasks include unmodified H3 and H4 and several acetylated or methylated histones.
Preprocessing: The evaluation used generated 90:10 train-test splits because exact Nucleotide Transformer splits were unavailable, except for the enhancer dataset.Negative promoter samples were generated using the cited procedure because they were not available.
Pretraining ablations: Up to 21 MCC points of pretraining gain occur on H3K4me3, whereas splice-site and promoter gains are 0 to 1 accuracy points.The larger gains appear on more challenging histone-mark tasks, while simpler tasks are near saturation.
Chromatin profile prediction: The smallest 1024 bp model outperforms DeepSEA and BigBird on TF and DHS prediction, while a 32k-context model outperforms BigBird on long-range HM prediction.The 32k-context model has four layers and can show degraded performance on short-range tasks; the models use 5–30× fewer parameters than DeepSEA and BigBird.
Biotype classification: HyenaDNA achieves the highest F1 score on biotype classification using 256-dimensional embeddings, versus dimensions 1029 and 1280 for DNABERT and Nucleotide Transformer.The result indicates that HyenaDNA embeddings contain features informative of biological function.