PaperlayerView exact revision on arXiv

Source-linked AI summary

MergeDNA: Context-aware Genome Modeling with Dynamic Tokenization through Token Merging

Siyuan Li, Kai Yu, Anna Wang, Zicheng Liu, Chang Yu, Jingbo Zhou, Qirong Yang, Yucheng Guo, Xiaoming Zhang, Stan Z. Li

arXiv:2511.14806v1q-bio.GNcs.AIcs.LG

TL;DR

Genomic modeling faces uneven information density and no predefined vocabulary units, while sequences also contain long-range dependencies. MergeDNA jointly learns dynamic tokenization and context-aware pre-training in a hierarchical model. It reports state-of-the-art results across DNA benchmarks and generalization across species and modalities, with genome-wide resolution beyond its current scope.

  • Problem

    Genomic sequences have uneven information density, no predefined vocabulary units, long-range dependencies, and naive pre-training objectives may fail to focus on important regions.

    Source: §1 · p. 1

  • Method

    MergeDNA combines differentiable local token merging, a global-context latent Transformer, and Merged Token Reconstruction and Adaptive Masked Token Modeling.

    Source: §1 · p. 2

  • Results

    MergeDNA achieves state-of-the-art performance on three standard DNA benchmarks and strong generalization across species and modalities.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The model dynamically adjusts token granularity and focuses on salient regions across diverse genomic contexts.

    Source: §5 · p. 8

    Pre-training is limited to sequences up to 4k bases; evaluations reach tens of kilobases through windowed processing and aggregation, not chromosome- or genome-wide resolution.

    Source: §5 · p. 8

Abstract

from arXiv · show

Modeling genomic sequences faces two unsolved challenges: the information density varies widely across different regions, while there is no clearly defined minimum vocabulary unit. Relying on either four primitive bases or independently designed DNA tokenizers, existing approaches with naive masked language modeling pre-training often fail to adapt to the varying complexities of genomic sequences. Leveraging Token Merging techniques, this paper introduces a hierarchical architecture that jointly optimizes a dynamic genomic tokenizer and latent Transformers with context-aware pre-training tasks. As for network structures, the tokenization module automatically chunks adjacent bases into words by stacking multiple layers of the differentiable token merging blocks with local-window constraints, then a Latent Encoder captures the global context of these merged words by full-attention blocks. Symmetrically employing a Latent Decoder and a Local Decoder, MergeDNA learns with two pre-training tasks: Merged Token Reconstruction simultaneously trains the dynamic tokenization module and adaptively filters important tokens, while Adaptive Masked Token Modeling learns to predict these filtered tokens to capture informative contents. Extensive experiments show that MergeDNA achieves superior performance on three popular DNA benchmarks and several multi-omics tasks with fine-tuning or zero-shot evaluation, outperforming typical tokenization methods and large-scale DNA foundation models.

1 Introduction

Genomic modeling must handle uneven information density, context-dependent sequence units, long-range dependencies, and pre-training objectives that may miss important regions. MergeDNA jointly adapts tokenization and context-aware pre-training in a hierarchical architecture, with reported strong results across DNA benchmarks and multi-omics tasks.

1 Introduction

  • Genomic sequences combine uneven information density, absent predefined vocabulary units, long-range dependencies, and potentially unfocused naive pre-training objectives.Coding sequences comprise around 2% of the human genome, while biologically relevant units can range from 3-base codons to 6–10-base binding sites or longer sequences.Source: §1 · p. 1
  • Prior work often optimizes long-context architectures, tokenization, and pre-training objectives in isolation rather than addressing all three challenges jointly.The introduction argues for a context-sensitive tokenizer and adaptive objectives that prioritize information-dense regions.Source: §1 · pp. 1–2
  • MergeDNA combines a differentiable local tokenizer, a global-context latent Transformer, and two pre-training objectives in one hierarchical framework.Merged Token Reconstruction trains tokenization while filtering redundancy; Adaptive Masked Token Modeling predicts important tokens selected through merging.Source: §1 · p. 2
  • MergeDNA is reported to achieve competitive performance across three DNA benchmarks and generalize to RNA and protein tasks, outperforming prior tokenization methods and foundation models.The contribution summary describes evaluation in short- and long-context settings.Source: §1 · p. 2

2 Related Work

Related work spans long-sequence architectures, DNA tokenization, pre-training objectives, and application domains. Byte-level methods also provide a route to end-to-end modeling without fixed subword tokenizers.

2 Related Work

  • Long-sequence DNA models use linear-time state-space models, hierarchical attention, or hybrid designs to balance sequence length, motif capture, and memory.Examples include HyenaDNA and Caduceus for linear complexity, and hybrid SSM-attention designs for motifs and chromosome-level structure.Source: §2 · p. 2
  • DNA tokenization ranges from byte-level inputs and k-mers to learnable vocabularies, while pre-training uses BERT-style or autoregressive-like masked objectives.Related methods also include loss reweighting, masking curricula, contrastive learning, and cross-modality alignment.Source: §2 · p. 2
  • DNA foundation models span human, multispecies, and specialized-domain corpora, while multi-omics work links DNA, RNA, protein, and epigenome tasks.Examples include prokaryotic, plant, and metagenomic models, as well as gene-to-expression and genome-to-protein pipelines.Source: §2 · p. 2
  • Byte-level approaches avoid fixed subword tokenization; learned chunking methods include entropy-balanced patches and differentiable segmentation.DNA models that take raw nucleotides as input are also categorized as byte-level architectures.Source: §2 · p. 2

3 Methodology

MergeDNA combines context-dependent local tokenization with long-range latent modeling in a hierarchical autoencoder. Its pre-training reconstructs compressed sequences and prioritizes informative tokens for prediction.

3.1 Preliminary

  • MergeDNA represents DNA as base sequences and uses masked token modeling to infer masked bases from surrounding context.The model formalizes a tokenizer as mapping a base sequence to a sequence of tokens.Source: §3.1 · p. 3

3.2 Architectural Overview

  • A Local Encoder merges adjacent bases into context-dependent tokens, while a full-attention Latent Encoder models long-range dependencies across them.The Latent Decoder reconstructs token representations and the Local Decoder unmerges them to reconstruct bases; inference can omit the decoders for sequence-level prediction.Source: §3.2 · p. 3

3.3 MergeDNA Tokenization

  • Local-window Token Merging progressively merges similar token pairs within windows, and Merged Token Reconstruction trains the model to recover the original sequence from compressed tokens.Training samples compression ratios to expose the tokenizer to a range of segmentations and reduce overfitting to one rate.Source: §3.3 · pp. 3–4

3.4 Adaptive Context Modeling

  • Global latent merging selects salient tokens for reconstruction, while Adaptive Masked Token Modeling assigns higher masking probability to tokens deemed more informative.The masking loss focuses on selected tokens and their constituent bases; the latent reconstruction step holds the Local Encoder fixed.Source: §3.4 · p. 4
  • The full pre-training objective combines sequence reconstruction, latent reconstruction, and adaptive masked-token prediction, down-weighting latent reconstruction with λ = 0.25.The down-weighting is intended to recover dropped information without overweighting low-information content in the training signal.Source: §3.4 · pp. 4–5

4 Experiments

Across fine-tuned genomic benchmarks and multi-omics evaluations, MergeDNA reports leading results, including in zero-shot protein-fitness prediction. Tokenization analysis and ablations examine how context-dependent merging and pre-training choices relate to performance.

4 Experiments

  • MergeDNA is compared with state-of-the-art genomics models and specialist models in evaluation experiments on DNA benchmarks and multi-omics tasks using supervised fine-tuning (SFT) or zero-shot protocols.Source: §4 · p. 5, §4.1 · p. 5

4.2 Comparison Results on Genomic Benchmarks

  • MergeDNA achieves 90.87% overall accuracy across eight Genomic Benchmark tasks, the highest among prior DNA foundation models.On enhancer tasks, it scores 85.11% versus 84.87% for the second-best model.Source: §4.2 · p. 6
  • MergeDNA scores 78.39 on the Nucleotide Transformer benchmark versus MxDNA’s 78.14, and leads GUE with 77.11% versus HybriDNA-7B’s 76.42%.On GUE, MCC (%) or F1 score (%) is averaged across subtasks with supervised fine-tuning.Source: §4.2 · p. 6, Table 3 · p. 6

4.3 Multi-omics Downstream Tasks

  • For SpliceAI splice-site prediction, MergeDNA achieves mean AUROC 69.8 versus 63.2 for SpliceAI and exceeds prior DNA foundation models.It nearly matches 7B-parameter Evo on donor sites and exceeds it on acceptor sites.Source: §4.3 · p. 6, Table 4 · p. 6
  • On long-range expression tasks, MergeDNA reaches AUROC 0.75 versus 0.74 for eQTL and R2 0.62 versus 0.60 for bulk expression.The tasks use kilobase-scale contexts.Source: §4.3 · p. 6, Table 6 · p. 7
  • In zero-shot protein fitness prediction, MergeDNA obtains 42.7% SRCC on the bacterial protein and 20.6% on the human protein, without protein-data fine-tuning.ESM2 scores highest; MergeDNA trails Evo2 on the human protein but outperforms earlier DNA models there.Source: Table 5 · p. 6, §4.3 · p. 7

4.4 Empirical Analysis of Tokenization

  • MergeDNA’s token-length distributions vary by genomic context: peaks are k=4 for splice sites, k=7 for promoters, and k=9 for enhancers, unlike static baselines.BPE peaks around length 6 regardless of sequence type; MxDNA also varies minimally across contexts.Source: §4.4 · p. 7, Figure 3 · p. 7

4.5 Ablation Study

  • On Genomic Benchmark ablations, adding Merged Token Reconstruction improves accuracy by +1.03 and Adaptive MTM raises the gain to +1.57 over baseline.Local token merging improves performance by +0.39 at the same parameter budget; latent LMTR loss weight λ=0.25 yields the best result, and reducing the tokenizer to two merging layers degrades performance.Source: §4.5 · pp. 7–8, Table 7 · p. 7

5 Conclusions and Limitations

MergeDNA combines dynamic local tokenization, global context modeling, and two complementary pre-training tasks; experiments report state-of-the-art performance across DNA benchmarks and multi-omics tasks. The paper also identifies boundaries in context length, token interpretability, architecture exploration, and evaluation scope.

5 Conclusions and Limitations

  • MergeDNA achieves state-of-the-art performance on three standard DNA benchmarks and several multi-omics tasks, with strong generalization across species and modalities.The authors describe the approach as a scalable and principled framework for genome-scale representation learning.Source: §5 · p. 8
  • A differentiable local tokenizer and global latent Transformer are trained through Merged Token Reconstruction and Adaptive Masked Token Modeling.The architecture is intended to adjust token granularity and focus on salient regions across genomic contexts.Source: §5 · p. 8
  • Pre-training is limited to sequences up to 4k bases; evaluation reaches tens of kilobases through windowing and aggregation, not chromosome- or genome-wide resolution.The authors suggest longer contexts may require additional hierarchical stages, structured token compression, and improved memory management.Source: §5 · p. 8
  • The paper does not systematically interpret merged tokens biologically or explore hybrid designs that use MergeDNA with tokenizer-free long-sequence models.Whether merged tokens align with canonical motifs or how tokenization behaves in repetitive and low-complexity regions remains open.Source: §5 · p. 8
  • Evaluation uses a representative but limited set of genomic and multi-omics benchmarks; broader evaluation is needed across single-cell, epigenomic, metagenomic, and clinical variant interpretation settings.The pre-training corpus is Multi-Species Genomes.Source: §5 · p. 8

A.1 Pre-training Settings

MergeDNA is pre-trained on reference sequences from the Multi-species Genomes corpus with a large-scale AdamW training run. The architecture configuration distinguishes local-window attention from self-attention blocks.

A.1 Pre-training Settings

  • Pre-training uses Multi-species Genomes reference sequences and runs AdamW for 100,000 iterations at a 1 × 10−4 learning rate with batch size 256.The run uses eight Nvidia A100-80G GPUs and takes nearly five days.Source: §A.1 · p. 12
  • The architecture configuration labels Local-Attn and Attention blocks as local-window and self-attention blocks, respectively, with FlashAttention implementation.Source: Table A1 · p. 12

A.2 Evaluation Setups

Evaluation generally transfers pre-trained models through supervised fine-tuning with task-specific heads and parameter-efficient LoRA. Protein fitness is evaluated zero-shot, and reported results are averaged over three runs.

A.2 Evaluation Setups

  • Most downstream evaluations use supervised fine-tuning with frozen encoder blocks and LoRA, adding a task-specific decoder head such as an MLP.When available, benchmark and model hyperparameters follow official settings; otherwise learning rate and weight decay are selected from stated candidate values.Source: §A.2 · p. 12
  • Protein fitness prediction is evaluated zero-shot using linear regression on Latent Encoder embeddings, with results averaged over three runs.Source: §A.2 · p. 12

B.1 DNA Tasks with Genomics Benchmark

The Genomics Benchmark evaluates basic genomic classification across enhancer, species, and regulatory-element tasks. Examples are labeled 200bp central genome sequences, and performance is measured with top-1 accuracy.

B.1 DNA Tasks with Genomics Benchmark

  • The benchmark groups balanced binary classification into enhancer prediction, species classification, and regulatory-element classification.Tasks include enhancer regions, coding versus intergenic or human versus worm sequences, and regulatory annotations, open chromatin, or nonTATA promoters.Source: §B.1 · p. 12
  • Each example is a 200bp central sequence from a reference genome labeled by high-throughput functional assays.Source: §B.1 · p. 12
  • The benchmark uses balanced binary classification and reports top-1 accuracy.Source: §B.1 · p. 12

B.2 DNA Tasks with Nucleotide Transformer (NT) Benchmark

The NT benchmark evaluates DNA models across 18 tasks spanning regulatory logic, histone marks, promoters, enhancers, and splice sites, using chromosome-wise splits and task-appropriate metrics.

B.2 DNA Tasks with Nucleotide Transformer (NT) Benchmark

  • The NT benchmark spans 18 tasks covering histone-mark prediction, promoter/enhancer subtasks, and canonical splice-site detection.The benchmark probes both local and distal regulatory logic.Source: §B.2 · p. 12
  • Sequences of 1–4 kbp around epigenetic peaks are sampled from hg38, with chromosome-wise train/validation/test splits to prevent leakage.Source: §B.2 · p. 12
  • MCC is used for imbalanced or noisy histone-mark and splice-site labels, while promoter/enhancer tasks use F1 score.Table 2 presents full NT benchmark results with popular DNA models.Source: §B.2 · p. 12

B.3 DNA Tasks with GUE Benchmark

GUE covers 24 datasets across seven genome-analysis tasks and four species, with official splits and task-specific metrics; the supplied tables describe result coverage but provide no scores.

B.3 DNA Tasks with GUE Benchmark

  • GUE comprises 24 datasets across seven practical genome-analysis tasks and four species, using an official 70/15/15 train/validation/test split.Input sequences range from 70 bp to 1 kb to probe local and distal regulatory logic.Source: §B.3 · p. 12
  • Binary tasks are scored with MCC and the multi-class Covid variant task with macro-F1; Table A3 covers the GUE task groups without exposing scores in its caption.Source: §B.3 · p. 12, Table A3 · p. 13
  • Table A2 reports top-1 accuracy averaged across three trials for recent DNA foundation models on Genomic Benchmarks.Its caption says best and second-best results are marked, but does not provide their values.Source: Table A2 · p. 13
  • The human tasks cover promoter and core-promoter detection, transcription-factor binding, and splice-site prediction; the benchmark also includes mouse TF and yeast histone-mark tasks.Human promoter datasets distinguish TATA, non-TATA, and combined cases; splice-site data adds hard negatives.Source: §B.3 · pp. 12–13
  • GUE also includes SARS-CoV-2 lineage classification from 1 kb genome snippets, while yeast tasks cover ten histone marks or nucleosome-occupancy tracks.Source: §B.3 · p. 13

B.4 Multi-omics Downstream Tasks

The paper specifies multi-omics and zero-shot evaluation tasks, reviews DNA foundation-model approaches, and distinguishes two broad directions for DNA-centered multi-omics modeling.

B.4 Multi-omics Downstream Tasks

  • The downstream evaluations span splice-site prediction, long-range variant and expression tasks, and zero-shot protein fitness prediction.They use chromosome-held-out or unseen-chromosome evaluation where specified, and hidden fitness values during zero-shot inference.Source: §B.4 · pp. 13–14

C.1 DNA Foundation Models

  • DNA foundation models vary in long-sequence architectures, tokenization, pre-training objectives, and training domains and downstream tasks.The review contrasts base, k-mer, BPE, and learned dynamic tokenization, alongside masked or autoregressive objectives and other pre-training strategies.Source: §C.1 · p. 14

C.2 Multi-omics Modeling

  • DNA-centered multi-omics models fall into protein-function prediction from DNA and gene-to-expression modeling of cellular or molecular readouts.The reviewed approaches include genome-to-proteome mappings and prediction of expression or other cell-level assay results from DNA.Source: §C.2 · p. 15

C.3 Byte Architectures

Byte architectures address token-boundary constraints by processing byte sequences directly or by learning segmentation within multiscale models.

C.3 Byte Architectures

  • Precomputed tokenizers and separate dynamic tokenization modules retain discrete token boundaries, which may introduce information loss or segmentation artifacts.Source: §C.3 · p. 15
  • Byte-level architectures eliminate tokenization, with multiscale Transformers and state-space models supporting contexts on the order of millions of characters.Source: §C.3 · p. 15
  • Entropy-based segmentation and fully differentiable tokenization offer alternatives that learn how byte sequences are chunked during modeling.Source: §C.3 · p. 15
Loading 2511.14806v1…