PaperlayerView exact revision on arXiv

Source-linked AI summary

DNABERT-2: Efficient Foundation Model and Benchmark For Multi-Species Genome

Zhihan Zhou, Yanrong Ji, Weijian Li, Pratik Dutta, Ramana Davuluri, Han Liu

arXiv:2306.15006v2q-bio.GNcs.AIcs.CEcs.CL

TL;DR

Genome foundation models rely on k-mer tokenization, which introduces computational and sample inefficiencies and complicates fair comparison across species and tasks. This paper replaces k-mers with BPE, introduces DNABERT-2 and the GUE benchmark, and reports comparable state-of-the-art performance with 21× fewer parameters and approximately 92× less pre-training GPU time.

  • Problem

    Existing genome foundation models face inefficient, information-leaking k-mer tokenization and lack a comprehensive standardized multi-species benchmark for fair evaluation.

  • Method

    The paper introduces DNABERT-2, using BPE tokenization with ALiBi and related efficiency strategies, alongside the 9-task, 36-dataset GUE benchmark.

  • Results

    Across GUE tasks, DNABERT-2 performs comparably to the state-of-the-art model while being 21× smaller and requiring approximately 92× less pre-training GPU time.

  • Takeaways & Limitations

    DNABERT-2 and GUE provide an efficient multi-species genome modeling and evaluation framework spanning varied sequence lengths and classification tasks.

  • Takeaways & Limitations

    Non-overlapping k-mer and BPE tokenization can reduce retained sequence information and hinder detection of subtle signals in short inputs.

Abstract

from arXiv · show

Decoding the linguistic intricacies of the genome is a crucial problem in biology, and pre-trained foundational models such as DNABERT and Nucleotide Transformer have made significant strides in this area. Existing works have largely hinged on k-mer, fixed-length permutations of A, T, C, and G, as the token of the genome language due to its simplicity. However, we argue that the computation and sample inefficiencies introduced by k-mer tokenization are primary obstacles in developing large genome foundational models. We provide conceptual and empirical insights into genome tokenization, building on which we propose to replace k-mer tokenization with Byte Pair Encoding (BPE), a statistics-based data compression algorithm that constructs tokens by iteratively merging the most frequent co-occurring genome segment in the corpus. We demonstrate that BPE not only overcomes the limitations of k-mer tokenization but also benefits from the computational efficiency of non-overlapping tokenization. Based on these insights, we introduce DNABERT-2, a refined genome foundation model that adapts an efficient tokenizer and employs multiple strategies to overcome input length constraints, reduce time and memory expenditure, and enhance model capability. Furthermore, we identify the absence of a comprehensive and standardized benchmark for genome understanding as another significant impediment to fair comparative analysis. In response, we propose the Genome Understanding Evaluation (GUE), a comprehensive multi-species genome classification dataset that amalgamates $36$ distinct datasets across $9$ tasks, with input lengths ranging from $70$ to $10000$. Through comprehensive experiments on the GUE benchmark, we demonstrate that DNABERT-2 achieves comparable performance to the state-of-the-art model with $21 \times$ fewer parameters and approximately $92 \times$ less GPU time in pre-training.

1 INTRODUCTION

Genome foundation models have advanced genomics applications but remain constrained by tokenization, input-length, benchmarking, and efficiency limitations. DNABERT-2 addresses these challenges with BPE tokenization, multi-species pretraining, and the GUE benchmark, achieving state-of-the-art performance with substantially lower resource use.

  • Background: Earlier genome foundation models supported applications including gene-expression, DNA-methylation, chromatin-state, regulatory-interaction, binding, variant-effect, and gene-network prediction.These models provide numerical genome-sequence embeddings applicable to diverse genomics prediction and analysis tasks.
  • Limitations and related work: DNABERT was pretrained only on the human reference genome, while Nucleotide Transformers scaled to 500M–2500M parameters but retained a hard input-length limitation.Nucleotide Transformers pretrained on genomes from 850 species and used non-overlapping k-mer tokenization to reduce tokenized sequence length.
  • Contributions: DNABERT-2 replaces k-mer tokenization with BPE to address known tokenization issues while maintaining non-overlapping tokenization’s computational efficiency.BPE is described as a data-compression algorithm widely used by large language models.
  • Contributions: The Genome Understanding Evaluation benchmark addresses inconsistent preprocessing and weak comparability through 9 tasks and 36 datasets for multi-species genome classification.The benchmark is intended to facilitate research in genome foundation models.
  • Contributions: DNABERT-2 delivers performance on par with the state-of-the-art model while being 21× smaller and using approximately 92× less GPU time.The contribution statement presents these reductions as properties of the efficient pre-trained multi-species genome foundation model.

2 BACKGROUND

Genome language models commonly tokenize DNA with k-mers, but overlapping and non-overlapping variants introduce information leakage and sample inefficiency. The paper therefore adapts SentencePiece BPE to construct variable-length genome tokens by merging frequent nucleotide and genome-segment pairs.

  • K-mer tokenization: Most genome language models use k-mer tokenization, representing contiguous k-length genome segments as tokens from the four nucleotide bases A, T, C, and G.A sliding window with window size k and stride t converts genome sequences into tokens.
  • K-mer tokenization: Overlapping k-mers cause substantial masked-language-modeling information leakage because adjacent tokens share k−1 characters.A masked token can be entirely or partially revealed by unmasked neighboring tokens, depending on whether both or only one adjacent side remains visible.
  • K-mer tokenization: Non-overlapping k-mers reduce sequence length by a factor of k but are sample-inefficient because small input changes can produce inconsistent tokenized representations.This makes it harder for the model to align identical or near-identical inputs, impeding efficient learning and potentially overall performance.
  • BPE tokenization: The paper adapts SentencePiece with BPE [Sennrich et al., 2016] to replace k-mer tokenization for genome sequences.BPE iteratively merges frequent nucleotide and genome-segment pairs into a variable-length vocabulary representing the genome dataset.

3 METHOD

DNABERT-2 uses SentencePiece-based BPE tokenization for raw DNA streams and evaluates vocabulary sizes empirically. Its Transformer architecture combines ALiBi, FlashAttention, low-precision normalization, LoRA, and GEGLU to improve efficiency, sequence-length handling, and capability.

  • 3.1 Tokenization: DNABERT-2 adapts SentencePiece with BPE to tokenize DNA as a raw stream without predefined words or sentences.SentencePiece is language-agnostic, while BPE constructs tokens through data-driven segmentation.
  • 3.1 Tokenization: Eight multi-species BPE vocabularies with target sizes from 28 to 215 were constructed to evaluate how vocabulary size affects tokenization and GUE performance.Figure 3 reports average token length, sequence-length reduction, and GUE performance across vocabulary sizes.
  • 3.1 Tokenization: Larger vocabularies produce sparser embedding updates, so variants with vocabulary sizes 28, 212, and 215 were pre-trained and evaluated on GUE.The comparison used batch size 2048 for 150000 steps on multi-species genomes.
  • 3.2 Model architecture: The Transformer architecture replaces learned positional embeddings with ALiBi, uses FlashAttention and low-precision layer normalization, and incorporates GEGLU.These changes target input-length limitations, computation, memory efficiency, and Transformer performance.
  • 3.2 Model architecture: DNABERT-2 adopts LoRA for parameter-efficient fine-tuning, reducing computation and memory costs with an ignorable performance sacrifice.LoRA represents task-specific weight changes as a low-rank update to the pretrained matrix.

4 DATA

The paper releases human and multi-species genome datasets for foundation-model pre-training and introduces the GUE/GUE+ benchmarks for genome understanding across varying input lengths. Benchmark datasets were selected and calibrated to distinguish current models’ capabilities rather than produce uniformly saturated scores.

  • 4.1 Pre-training datasets: The multi-species pre-training dataset covers 135 species across 6 categories and 32.49B nucleotide bases, while the human dataset contains 2.75B bases.The human dataset is the one used in DNABERT.
  • 4.2 Benchmark datasets: GUE contains 7 genome-sequence classification problems across 28 datasets with input lengths of 70–1000, while GUE+ targets longer sequences of 5000–10000.The benchmark is split because existing genome foundation models have input-length limits.
  • 4.2 Benchmark datasets: The benchmark was calibrated to avoid tasks that were too simple or too difficult, since Nucleotide Transformer variants reached 97%–98% accuracy on Splice Site Prediction.Promoter Prediction and Enhancer Prediction also produced high scores, whereas Section 5 reports greater variation on more discerning datasets.
  • 4.2 Benchmark datasets: Benchmark construction aggregated biologically important datasets, evaluated DNABERT and Nucleotide Transformer, and retained datasets with moderate, distinguishable model scores.Datasets failing these criteria were subjected to a subsequent restriction process described in the passage.

5 EXPERIMENTS

Experiments show that DNABERT-2 delivers state-of-the-art-comparable genome understanding with substantially lower computational cost, while additional in-domain pre-training achieves the highest aggregate performance. Its BPE- and ALiBi-based design also supports long-sequence extrapolation and strong multi-species performance, with limitations on subtle signals in short inputs.

  • 5 EXPERIMENTS: 21× fewer parameters and 19× fewer FLOPs enable DNABERT-2 to match the state-of-the-art model while surpassing other baselines; additional GUE pre-training yields the highest average performance and top-two rankings across 28 tasks.The additional pre-training uses only 0.41B tokens, or 0.08% of the full training process.
  • 5 EXPERIMENTS: The evaluation compares computational cost using parameter counts and forward-pass FLOPs on 500-length sequences, alongside downstream performance across the GUE benchmark.DNABERT-2 is compared with DNABERT and Nucleotide Transformer, with LoRA used to make Nucleotide Transformer fine-tuning more efficient.
  • 5 EXPERIMENTS: DNABERT-2 and NT-2500M-multi achieve top-tier performance across most GUE tasks, especially non-human genome analyses, while multi-species pre-training also transfers strongly to human genome tasks.DNABERT variants perform best on Core Promoter Detection with length-70 inputs, but their performance declines on length-300 Promoter Detection.
  • 5 EXPERIMENTS: Despite 30% more parameters than DNABERT, DNABERT-2 requires only one-third the FLOPs, demonstrating the efficiency of BPE tokenization over overlapping k-mer tokenization.Combined with ALiBi, the new tokenization enables effective processing of arbitrarily long genome sequences.
  • GUE+ Benchmark: DNABERT-2 surpasses DNABERT and Nucleotide Transformer on every GUE+ dataset, including sequences up to 10000 bp, despite pre-training only on 700 bp sequences.A few fine-tuning epochs suffice, indicating ALiBi-enabled extrapolation to long DNA sequences.

6 CONCLUSION

The paper introduces DNABERT-2, an efficient multi-species genome foundation model that replaces inefficient k-mer tokenization with BPE for DNA language modeling. It builds the model from empirical analyses and integrates techniques including ALiBi and Low-Rank Adaptat.

  • 6 CONCLUSION: DNABERT-2 is introduced as an efficient foundational model pre-trained on multi-species genomes.The model is presented as the paper’s central contribution.
  • 6 CONCLUSION: The paper identifies computational and sample inefficiencies in k-mer tokenization and adapts BPE for DNA language modeling.The conclusion frames BPE adaptation as a response to limitations of the existing tokenization method.
  • 6 CONCLUSION: DNABERT-2 is built from comprehensive empirical analyses and integrates techniques including ALiBi and Low-Rank Adaptat.The supplied passage ends mid-term after “Low-Rank Adaptat,” so the technique name is preserved exactly as provided.

A EXPERIMENTS · A.1 ALL EXPERIMENT RESULTS · H3 H3K14ac H3K36me3 H3K4me1 H3K4me2 H3K4me3

This experimental block evaluates DNABERT-2 and baseline models on the GUE benchmark, covering epigenetic-mark prediction, promoter and core-promoter detection, transcription-factor prediction, and related tasks. Table 6 reports the comparative benchmark results, while Table 7 documents the training-step schedules used across tasks.

  • A.1 ALL EXPERIMENT RESULTS: The experiments include Epigenetic Marks Prediction as a major GUE evaluation task.
  • H3 H3K14ac H3K36me3 H3K4me1 H3K4me2 H3K4me3: The reported task coverage includes promoter detection alongside epigenetic-mark prediction.
  • H3 H3K14ac H3K36me3 H3K4me1 H3K4me2 H3K4me3: The listed promoter-related datasets distinguish tata from notata and all-data variants, including both promoter and core-promoter detection.
  • H3 H3K14ac H3K36me3 H3K4me1 H3K4me2 H3K4me3: The benchmark scope also includes transcription-factor prediction on the human genome and core-promoter detection.
  • H3 H3K14ac H3K36me3 H3K4me1 H3K4me2 H3K4me3: Table 6 presents DNABERT-2 and baseline performance across the GUE benchmark.The table caption specifies that ♦ marks models receiving further pre-training on GUE training sets.
  • A.1 ALL EXPERIMENT RESULTS: Table 7 records the training steps used for Epigenetic Marks Prediction, transcription-factor prediction, promoter detection, core-promoter detection, Covid Variants Classification, and Splice Site Prediction.The schedule distinguishes human and mouse transcription-factor prediction and tata, notata, and all-dataset variants for promoter and core-promoter detection.

A.2 IMPLEMENTATION · A.3 HYPERPARAMETERS

DNABERT-2 pre-training uses independently masked-token MLM with a 15% mask ratio, large batches, and a 500,000-step AdamW schedule. Fine-tuning uses shared optimization settings across datasets, with standard learning rates specified for DNABERT and DNABERT-2.

  • A.2 IMPLEMENTATION: DNABERT-2 is pre-trained with Masked Language Modeling using a 15% mask ratio and independently masked tokens rather than continuous spans.This differs from span masking such as Ji et al. (2021).
  • A.2 IMPLEMENTATION: Pre-training uses a batch size of 4096 and a maximum sequence length of 128.
  • A.2 IMPLEMENTATION: The model is trained for 500000 steps with AdamW, β1 = 0.9, β2 = 0.98, ϵ = 1e-6, and weight decay of 1e-5.
  • A.2 IMPLEMENTATION: The pre-training learning rate increases linearly from 0 to 5e-4 during the first 30000 steps.
  • A.3 HYPERPARAMETERS: Fine-tuning uses AdamW, batch size 32, warmup step 50, and weight decay 0.01 across models and datasets.Table 7 reports the number of training steps for each task, but those values are not provided here.
  • A.3 HYPERPARAMETERS: DNABERT and DNABERT-2 use standard fine-tuning with a learning rate of 3e-5.

A.4 PRELIMINARY EXPERIMENTS ON NUCLEOTIDE TRANSFORMER · A.5 COMPARING DNABERT-2 WITH OTHER BASELINES

The experiments establish a fair comparison with Nucleotide Transformer using shared epigenetic-mark prediction and show that DNABERT-2 consistently outperforms HyenaDNA, CNN, and an unpretrained DNABERT-2 baseline.

  • A.4 PRELIMINARY EXPERIMENTS ON NUCLEOTIDE TRANSFORMER: The Nucleotide Transformer comparison uses epigenetic-mark prediction because it is the only task shared with that model’s reported evaluation.Open-sourced Nucleotide Transformer checkpoints are trained with the authors’ code base using LoRA because no official fine-tuning code is available.
  • A.4 PRELIMINARY EXPERIMENTS ON NUCLEOTIDE TRANSFORMER: On ten Yeast-genome epigenetic-mark datasets, the authors’ Nucleotide Transformer implementation performs better than the results reported in the original paper.The table marks results taken from Dalla-Torre et al. (2023) with an asterisk.
  • A.4 PRELIMINARY EXPERIMENTS ON NUCLEOTIDE TRANSFORMER: The stronger Nucleotide Transformer implementation results support the fairness of the subsequent comparison.The authors explicitly use this agreement check to compare reported-paper results with performance under their implementation.
  • A.5 COMPARING DNABERT-2 WITH OTHER BASELINES: HyenaDNA is implemented from its official Hugging Face checkpoint and implementation, while the CNN uses official GitHub code and default hyperparameters from both papers.Both baselines are trained with Hugging Face Trainer or the authors’ provided code as applicable.
  • A.5 COMPARING DNABERT-2 WITH OTHER BASELINES: DNABERT-2 consistently outperforms HyenaDNA, CNN, and DNABERT-2 without pre-training across the reported baseline comparisons.The comparison includes HyenaDNA, a DNA-classification CNN, and an unpretrained DNABERT-2 model.
  • A.5 COMPARING DNABERT-2 WITH OTHER BASELINES: The official HyenaDNA Hugging Face checkpoint does not reproduce its reported EMP-task performance.Possible explanations include non-open-source checkpoint differences, improper Hugging Face Trainer optimization, or the need for more extensive hyperparameter searches.

A.6 ABLATION STUDY ON TOKENIZATION METHOD · B DATA

The ablation compares overlapping 6-mer and BPE tokenization under matched DNABERT-2 training conditions, showing that BPE improves data efficiency while requiring 3–4 times less computation than K-mer tokenization. The section also includes benchmark and ablation tables covering DNABERT-2 comparisons and tokenization methods.

  • A.6 ABLATION STUDY ON TOKENIZATION METHOD: The ablation trains two DNABERT-2 variants using identical training data, architecture, and hyperparameters.The variants differ in tokenization method: overlapping K-mer tokenization used in DNABERT versus BPE tokenization used in DNABERT-2.
  • A.6 ABLATION STUDY ON TOKENIZATION METHOD: The matched ablation uses batch size 4096, maximum sequence length 128, and 120,000 training steps per model.The setup controls these training conditions while comparing the tokenization methods.
  • A.6 ABLATION STUDY ON TOKENIZATION METHOD: Six-mer tokenization is selected to keep parameter counts sufficiently similar to BPE’s vocabulary size of 4096.The passage explains that BPE has vocabulary size 4096 and that 6-mer tokenization is used for comparability.
  • B DATA: Table 9 benchmarks DNABERT-2 against HyenaDNA, CNN, and DNABERT-2 without pre-training on GUE.The caption identifies the benchmark scope but does not provide comparative cell values.
  • A.6 ABLATION STUDY ON TOKENIZATION METHOD: BPE tokenization achieves greater data efficiency than K-mer tokenization in the ablation study.The supplied passage states that BPE shows its data efficiency, but does not provide the underlying numerical values.
  • A.6 ABLATION STUDY ON TOKENIZATION METHOD: 3–4 times less computational cost is required by BPE than by K-mer tokenization.This comparison is reported with reference to Table 6 of the paper.
  • A.6 ABLATION STUDY ON TOKENIZATION METHOD: Table 10 presents the ablation study on tokenization methods.The supplied table caption identifies the table’s scope but does not provide cell values or a winner.

B.1 MULTI-SPECIES GENOME FOR PRE-TRAINING

The pre-training corpus randomly selected 135 species across 7 categories and recorded the nucleotide yield obtained from each species. Table 11 summarizes these dataset statistics.

  • Dataset composition: 135 species across 7 categories were randomly selected for genome foundation model pre-training, with nucleotide counts recorded for each species.The corpus includes genomes from protozoa, vertebrates, and other taxonomic groups.
  • Dataset statistics: 31,372.10 million nucleotides were recorded for Human, compared with 28,768.00 million for Bubalus Bubalis and 26,545.98 million for Mus Musculus.These examples illustrate the substantial genome-size variation represented in the pre-training corpus.
  • Dataset statistics: Table 11 presents the multi-species pre-training dataset’s detailed statistics, including species identities and nucleotide quantities.Reported genome sizes range from 5.30 million nucleotides for Legionella Endosymbiont Of Polyplax Serrata to 31,372.10 million for Human.

B.2 GENOME UNDERSTANDING EVALUATION (GUE & GUE+)

GUE is a standardized multi-species benchmark comprising 36 datasets across 9 genome-analysis tasks, with sequence lengths ranging from 70 to 10,000 bases. Its tasks cover regulatory-element detection, splicing, species classification, epigenetic prediction, and viral variant classification.

  • B.2 GENOME UNDERSTANDING EVALUATION (GUE & GUE+): GUE contains 36 datasets spanning 9 biologically important genome-analysis tasks and input lengths from 70 to 10,000 bases.Table 12 reports the training, validation, and test sample counts for each evaluation dataset.
  • B.2 GENOME UNDERSTANDING EVALUATION (GUE & GUE+): The human regulatory tasks include proximal and core promoter detection, transcription-factor binding-site prediction, and enhancer–promoter interaction classification.Core promoter detection uses a shorter window around the transcription start site, while enhancer–promoter interaction is formulated as sequence-pair binary classification.
  • B.2 GENOME UNDERSTANDING EVALUATION (GUE & GUE+): GUE also evaluates human splice-site prediction using 400-bp Ensembl GRCh38 sequences to identify donor and acceptor sites.The task targets the exact genomic locations where alternative splicing occurs.
  • B.2 GENOME UNDERSTANDING EVALUATION (GUE & GUE+): Cross-species tasks classify virus and fungi genome segments, predict mouse transcription-factor binding sites, and predict yeast epigenetic marks.Mouse negative examples use dinucleotide shuffling, whereas the yeast datasets are split into training, validation, and test sets at an 8:1:1 ratio.
  • B.2 GENOME UNDERSTANDING EVALUATION (GUE & GUE+): The viral task predicts 9 SARS-CoV-2 variant types from 1,000-length genome sequences.The variants include Alpha, Beta, Delta, Eta, Gamma, Iota, Kappa, Lambda, and Zeta.
Loading 2306.15006v2…