PaperlayerView exact revision on arXiv

Source-linked AI summary

BEND: Benchmarking DNA Language Models on biologically meaningful tasks

Frederikke Isa Marin, Felix Teufel, Marc Horlacher, Dennis Madsen, Dennis Pultz, Ole Winther, Wouter Boomsma

arXiv:2311.12570v4q-bio.GNcs.LG

TL;DR

Experimental annotation has not kept pace with the growth of genomic sequence data, and DNA LMs face long, sparse genomic signals. BEND benchmarks DNA LMs on biologically meaningful human-genome tasks and finds that model performance is promising on some tasks but uneven overall, with long-range reasoning still challenging.

  • Problem

    Experimental annotation has not kept pace with genomic sequence availability, while DNA LM evaluation varies across studies and may not reflect genome annotation's length, scale, and sparsity.

    Source: §1 · p. 1, §2.3.4 · p. 5

  • Method

    BEND provides a standardized collection of realistic, biologically meaningful downstream tasks on the human genome to assess DNA LM representations.

    Source: §1 · p. 1, §3.1 · p. 5, §3.6 · p. 7

  • Results

    DNA LMs show promising but inconsistent performance across tasks, sometimes approaching expert methods, while functional-genomics embeddings do not consistently improve on Basset with a two-layer CNN.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    NT-MS is a strong default LM overall, but can underperform the much smaller DNABERT on some tasks, and pretraining data and tokenization may merit further attention.

    Source: §6 · p. 9

    BEND focuses on the human genome, leaving DNA LM generalization across organisms unanswered.

    Source: §7 · p. 9

Abstract

from arXiv · show

The genome sequence contains the blueprint for governing cellular processes. While the availability of genomes has vastly increased over the last decades, experimental annotation of the various functional, non-coding and regulatory elements encoded in the DNA sequence remains both expensive and challenging. This has sparked interest in unsupervised language modeling of genomic DNA, a paradigm that has seen great success for protein sequence data. Although various DNA language models have been proposed, evaluation tasks often differ between individual works, and might not fully recapitulate the fundamental challenges of genome annotation, including the length, scale and sparsity of the data. In this study, we introduce BEND, a Benchmark for DNA language models, featuring a collection of realistic and biologically meaningful downstream tasks defined on the human genome. We find that embeddings from current DNA LMs can approach performance of expert methods on some tasks, but only capture limited information about long-range features. BEND is available at https://github.com/frederikkemarin/BEND.

1 INTRODUCTION

Genome sequencing has expanded faster than experimental annotation, while DNA LMs face sparse signals and long sequences. BEND standardizes biologically meaningful tasks and finds promising but uneven performance across models.

1 INTRODUCTION

  • Genome sequencing has expanded genomic data, but experimental annotation has not kept pace.Source: §1 · p. 1
  • BEND provides seven curated tasks probing DNA functional elements across a variety of length scales.The benchmark also evaluates LMs from six sources alongside baseline methods and offers an adaptable framework for embeddings and lightweight supervised models.Source: §1 · p. 2
  • DNA LMs approach expert-method performance on some tasks, but no model consistently leads and very-long-context reasoning remains challenging.The reported challenge includes finding enhancers.Source: §1 · p. 2
  • DNA LM embeddings can capture distinct features, with some focused more on gene structure and others on noncoding regions.Source: §1 · p. 2

2 BACKGROUND

Eukaryotic genomes combine genes and regulatory elements across long, compacted DNA; these properties make DNA modeling different from protein modeling. Existing methods and benchmarks cover varied tasks, but BEND targets realistic tasks across length scales and genomic properties.

2.1 EUKARYOTIC DNA ORGANIZATION AND TERMINOLOGY

  • Genes combine introns and exons, while regulatory elements include nearby promoters and enhancers that may lie thousands of base pairs away; DNA is packaged with histones into chromosomes.Chromatin organization and looping bring distant sequence regions into physical proximity and affect DNA accessibility.Source: §2.1 · p. 2, Figure 1 · p. 3

2.2 LANGUAGE MODELING FOR BIOMOLECULAR SEQUENCES: FROM PROTEINS TO DNA

  • DNA's four-letter alphabet, long unbounded sequences, and sparse signals contrast with shorter, denser protein sequences, creating distinct modeling challenges.Protein sequences typically contain 400–500 amino acids; DNA sequences such as genes are considerably longer.Source: §2.2 · pp. 2–3

2.3.1 DNA LANGUAGE MODELS

  • DNA LMs range from k-mer BERT models to multispecies transformers, BPE-tokenized models, and single-nucleotide autoregressive models.Reported input lengths range from 512 bases for DNABERT to up to 1 million nucleotides for HyenaDNA.Source: §2.3.1 · pp. 3–4

2.3.2 SUPERVISED LEARNING ON DNA

  • Supervised DNA methods include CNN-based predictors for binding and chromatin features, methylation models, and systems integrating long-range genomic tracks.Basenji and Enformer integrate information across up to 200 kilobases for gene-expression prediction.Source: §2.3.2 · p. 4

2.3.3 BENCHMARK COLLECTIONS ON DNA

  • Existing DNA benchmark collections differ in task coverage and model comparisons, and some omit non-LM task-specific baselines.Genomic Benchmarks reports a supervised neural baseline; GUE and the NT leaderboard do not include non-LM task-specific baselines.Source: §2.3.3 · p. 4

2.3.4 MOTIVATION OF BEND

  • Short-sequence tasks do not assess long-range genomic understanding; BEND instead spans biologically relevant length scales and DNA properties.Its tasks require less prior knowledge of feature positions and include both single-base and longer-stretch resolutions; the benchmark focuses on the human genome.Source: §2.3.4 · p. 5

3 TASKS AND DATASETS

BEND defines tasks spanning gene structure, regulatory elements, chromatin and epigenetic marks, and noncoding variant effects. The collection uses human genomic data and task-specific labels and evaluation metrics.

3 TASKS AND DATASETS

  • Gene finding labels each nucleotide as exon, intron, donor or acceptor splice site, or noncoding, with strand-specific labels for gene features.GENCODE annotations are partitioned at 80% identity, and multi-class MCC is used to account for highly uneven label ratios.Source: §3.1 · p. 5

3.2 ENHANCER ANNOTATION

  • Enhancer annotation predicts enhancer membership for each 128-bp segment in a 100,096-bp gene-adjacent sequence centered on the TSS.Validated enhancer-gene pairs are used, and performance is evaluated by 10-fold cross-validation with AUPRC.Source: §3.2 · p. 6

3.3 CHROMATIN ACCESSIBILITY PREDICTION

  • Chromatin accessibility prediction classifies 512-bp sequences as open or closed across 125 cell types.Labels come from ENCODE DNase I hypersensitive sites; AUROC is computed per label and averaged.Source: §3.3 · p. 6

3.4 HISTONE MODIFICATION PREDICTION

  • Histone modification prediction assigns one or more histone marks to each 512-bp sequence, using 11 marks and 18 K562 replicates.The task uses chromosome-partitioned samples and averages AUROC across labels.Source: §3.4 · p. 6

3.5 CPG METHYLATION PREDICTION

  • CpG methylation prediction labels a centered 512-bp sequence for methylation status across seven human cell lines.Bisulfite sequencing labels are chromosome-partitioned, and AUROC is averaged across labels.Source: §3.5 · p. 6

3.6 NONCODING VARIANT EFFECTS (EXPRESSION AND DISEASE)

  • Noncoding variant-effect tasks classify single-base mutations for expression or disease effects using the cosine distance between variant and reference embeddings.Each variant uses 512 bp of adjacent context; AUROC is reported overall and by consequence type.Source: §3.6 · pp. 6–7

4 MODELING

BEND evaluates DNA language models using frozen embeddings and lightweight supervised models, alongside supervised models trained without pre-training. The benchmark also includes zero-shot variant-effect prediction and task-specific expert comparisons where Basset does not apply.

4 MODELING

  • The benchmark compares available human-genome DNA LMs and trains AWD-LSTM and dilated-CNN baselines; Table 3 reports results across tasks.The AWD-LSTM is trained on three species and the dilated CNN on the human genome; Table 3 highlights the best-performing DNA LM for each task.Source: §4 · p. 7, Table 3 · p. 8
  • A two-layer CNN is trained on frozen LM embeddings for each task; variant-effect prediction instead uses embedding cosine distance as a zero-shot predictor.Embeddings reduced in length by tokenization are upsampled; sequence-level tasks use average pooling, and enhancer annotation uses fewer channels to prevent overfitting.Source: §4 · p. 7
  • Two supervised models are trained without pre-training: a two-layer CNN on one-hot sequences and task-specific alternatives where applicable.Basset is used for chromatin accessibility, histone modification and CpG methylation; ResNet CNNs are used for gene finding and enhancer annotation, with expert methods reported where Basset does not apply.Source: §4 · p. 7

5 RESULTS

DNA LMs approach expert performance on some tasks but show uneven results across gene finding, enhancer annotation, chromatin accessibility, histone modification, CpG methylation, and variant-effect prediction. The findings also point to limits in long-range prediction and differences in the sequence features captured by different models.

Gene finding

  • NT-MS and NT-V2 lead the DNA LMs for gene finding but remain behind AUGUSTUS; LM embeddings outperform the raw CNN baseline, suggesting they capture some long-range information.Performance varies across classes, some embeddings capture splice sites despite their low frequency, and HyenaDNA-large shows only modest performance despite covering the task input length.Source: §5 · p. 8

Enhancer annotation

  • All investigated models perform poorly on enhancer annotation, where long sequence scale, sparse signal, and a small dataset challenge prediction.The supervised baseline's receptive field is large enough for long-range interactions, but dataset size is prohibitive; Enformer performs comparably without direct enhancer training, and gene-specific enhancer prediction from sequence alone remains difficult.Source: §5 · p. 8

Chromatin accessibility

  • DNABERT reaches 0.85 on chromatin accessibility, on par with Basset; other LMs perform worse and offer no advantage over Basset.Source: §5 · p. 8

Histone modification

  • 0.74 is the highest histone-modification performance for NT-MS and DNABERT, above Basset at 0.72.The passage characterizes the potential improvement from LM embeddings as marginal.Source: §5 · p. 8

CpG methylation

  • NT-MS scores 0.92 on CpG methylation across included cell lines, while Basset scores 0.93 and outperforms it.DNABERT, GENA-LM, and HyenaDNA-large also perform competitively, indicating embeddings capture information about CpG island methylation patterns.Source: §5 · p. 8

Variant effect prediction

  • DNA LMs show some unsupervised signal for noncoding variant effects, but performance is only partly consistent across expression and disease datasets, with DNABERT and NT-MS best, respectively.DNABERT matches DeepSEA's unsupervised expression AUROC of 0.6; on disease variants, multiple LMs approach DeepSEA's Disease Impact Score, with NT-MS outperforming it. NT-MS performance is driven by splice-site and intron variants, while DNABERT performs better in UTR and nearby regions, suggesting distinct learned sequence features.Source: §5 · pp. 8–9

6 DISCUSSION

Current DNA language models perform promisingly on some tasks but do not improve consistently across tasks or reliably surpass specialized methods. Their long-range understanding remains limited, although NT-MS shows promising gene-finding performance and models learn distinct genomic features.

6 DISCUSSION

  • DNA LMs show promise on some tasks but do not consistently improve over supervised baselines or surpass specialized prediction methods; NT-MS is a strong default but can trail DNABERT.Pre-training data and tokenization are key architectural differences that may merit more attention.Source: §6 · p. 9
  • In functional genomics, LM embeddings do not consistently improve performance over Basset when paired with a two-layer CNN.The comparison uses Basset trained on the same data.Source: §6 · p. 9
  • NT-MS with a two-layer CNN shows promising gene-finding performance compared with AUGUSTUS, suggesting current DNA LMs model long-range dependencies to some extent.AUGUSTUS was identified as state of the art in a recent benchmark.Source: §6 · p. 9
  • Enhancer annotation indicates that long-range understanding still needs improvement for sparse problems with limited data.The authors raise the question of whether current LM objectives incorporate distant, sparse signals or whether supervision is needed.Source: §6 · p. 9

7 LIMITATIONS AND OUTLOOK

BEND focuses on the human genome because comprehensive benchmark curation requires experimental ground truth and most published models use human data. Its embedding-based evaluation is method-agnostic, while direct fine-tuning may yield gains beyond the reported results.

7 LIMITATIONS AND OUTLOOK

  • BEND is restricted to the human genome because comprehensive task curation requires experimental ground truth and most published models use human data.Whether DNA LMs generalize across organisms remains unanswered; the authors hope to test transfer learning in future benchmark extensions.Source: §7 · p. 9
  • BEND evaluates embeddings with downstream predictors, an approach agnostic to the LM methodology and scalable to models of any size.The evaluation focuses on features embeddings provide to downstream prediction models.Source: §7 · p. 9
  • Direct task fine-tuning may yield gains beyond the results observed here, and deeper analysis of pre-training feature learning remains future work.The authors note that fine-tuning may conflate representation content with a task model architecture's inductive bias (Zhou et al., 2023).Source: §7 · p. 9

A.1 DATASET DOCUMENTATION

Dataset documentation follows the Datasheets for Datasets framework, with sections tailored to the information appropriate for each dataset.

A.1 DATASET DOCUMENTATION

  • The dataset documentation follows the Datasheets for Datasets framework, covering motivation, composition, collection, preprocessing, and uses as appropriate.The framework is attributed to Gebru et al. (2018).Source: §A.1 · p. 19
  • Because the study acquired no new experimental data, collection sections list relevant sources rather than documenting original experimental protocols.Documenting those protocols was beyond the scope of a datasheet.Source: §A.1 · p. 19
  • Distribution and maintenance sections are omitted because they are identical for each dataset.Source: §A.1 · p. 19

A.1.1 GENE FINDING •

The appendix documents task motivations, dataset composition, sources, and preprocessing for gene finding, enhancer annotation, chromatin accessibility, and histone modification prediction.

A.1.1 GENE FINDING •

  • The gene-finding dataset predicts gene structure by labeling nucleotides as exons, splice sites, introns, or noncoding regions, using 5,976 human genes with lengths from 1,433 to 14,000 nucleotides.Annotations came from GENCODE release 44, with experimentally supported transcripts selected and sequence-identity-based train, test, and validation splits.Source: §A.1.1 · p. 19
  • Enhancer annotation labels 100,096-nucleotide sequences around transcription start sites in 782 bins of 128 bp as enhancer or non-enhancer.Enhancer locations derive from CRISPRi experiments; only experimentally validated pairs were retained, and pairs over 50,048 bp away were discarded.Source: §A.1.1 · pp. 19–20
  • Chromatin accessibility prediction labels 2,005,617 human-genome windows of 512 nucleotides across 125 cell types as open or closed, based on DNase I hypersensitivity.Data came from ENCODE and was split by chromosome into training, validation, and test sets.Source: §A.1.1 · p. 20
  • Histone modification prediction uses 512-nucleotide human-genome windows with binary labels for 18 histone marks.The ENCODE ChIP-seq data were processed into merged peaks and split by chromosome; the dataset was established in this study.Source: §A.1.1 · p. 20

A.1.5 CPG METHYLATION •

The CpG methylation benchmark evaluates whether models can predict methylation at CpG sites across human cell lines using 512-bp sequence windows. It combines seven ENCODE bisulfite-sequencing experiments with stringent filtering and chromosome-based splits.

Composition

  • Each instance is a 512-nucleotide sequence centered on a CpG site, with labels indicating methylation status across cell lines.The composition passage specifies a binary label vector of size 19; the dataset collection passage describes seven selected cell-line experiments.Source: §A.1.5 · p. 21

Collection

  • Seven human cell-line whole-genome bisulfite-sequencing experiments from ENCODE supply the methylation data.Experiments marked as having extremely low or insufficient coverage were excluded.Source: §A.1.5 · p. 21

Preprocessing

  • The dataset contains 959,039 CpG sites with 512-bp windows, split by chromosome into 743,095 training, 109,717 validation, and 106,227 test samples.Sites had to pass filtering in all seven experiments; chromosomes 4, 13, 19, and 21 form the test set, while chromosomes 5, 9, and 22 form validation.Source: §A.1.5 · p. 21

Uses

  • The dataset was established in this study from publicly available ENCODE data and had not been used before.The underlying ENCODE data source has seen widespread use.Source: §A.1.5 · p. 21

A.1.6 NONCODING VARIANT EFFECTS (EXPRESSION) •

The appendices describe two zero-shot noncoding-variant datasets, benchmark packaging and model-evaluation procedures, and supplementary results. They also document model selection and training, data-use considerations, and societal cautions for possible clinical applications.

A.1.6 NONCODING VARIANT EFFECTS (EXPRESSION) •

  • The expression variant dataset pairs reference and alternate nucleotides with labels for eQTL effects versus background variation, using SNPs from DeepSEA.It includes 8,000 effect variants and 98,221 background SNPs; the passage notes potential ethnicity bias in 1000 Genomes and European-ancestry bias in GRASP.Source: §A.1.6 · p. 21

A.1.7 NONCODING VARIANT EFFECTS (DISEASE) •

  • The disease variant dataset uses ClinVar benign/pathogenic labels and filters variants annotated as coding before zero-shot prediction from 512-bp context.The full dataset contains 274,399 benign and 21,524 pathogenic SNPs; variants are mapped to GRCh38 and assigned the most severe VEP consequence.Source: §A.1.7 · p. 22

A.2 FORMATTING

  • BEND stores task coordinates, splits, and labels in a shared bed-based format, with complex labels optionally held in indexed HDF5 files.This format allows users to include additional flanking context without reprocessing the data.Source: §A.2 · p. 30

A.3 LICENSE

  • BEND contributions are generally CC BY 4.0, while underlying dataset sources retain their own terms; data and experiment code are made available online.Users of the provided datasets are required to cite the original sources.Source: §A.3 · p. 30, §A.4 · p. 30

A.5 SOCIETAL IMPACT

  • The authors advise population-diverse evaluation and bias checks before clinical diagnostic use, and consent when private human genomes are used for pretraining.They frame biomedical research use as a potential application of DNA-LM predictors.Source: §A.5 · p. 30

A.6.1 LMS TRAINED IN THIS WORK

  • The study trained an AWD-LSTM on three species and a dilated ResNet LM on human-genome sequences, with the latter using 10,000-nucleotide training sequences.The AWD-LSTM training used full genomes of human, mouse, and fruit fly; the ResNet LM used 256 hidden channels and 50,000 steps.Source: §A.6.1 · p. 30

A.6.2 LM CHECKPOINT SELECTION

  • Checkpoint selection targeted publicly available human-genome DNA LMs, while embedding upsampling aligned tokenized-model outputs with nucleotide-level labels.For 6-mer tokenization each embedding is repeated six times; the appendix also describes variant-position embeddings and task-specific baseline procedures.Source: §A.6.2 · p. 31, §A.6.3 · p. 31, §A.7 · p. 32

A.8 EXTENDED RESULTS

  • On the disease dataset filtered to exclude ClinVar “Likely” labels, NT-MS outperforms DeepSEA, while ResNet-LM and DNABERT also show strong performance.The reduced dataset contains 100,623 benign and 8,188 pathogenic variants.Source: Table A14 · p. 36

Related papers

Loading 2311.12570v4…