PaperlayerView exact revision on arXiv

Source-linked AI summary

What DNA Foundation Models Learn Beyond Sequence Composition

Vincenzo Yuto Civale, Andrew David Bagdanov, Alberto Magi

arXiv:2610.15711v1cs.LGcs.AI

TL;DR

Frozen DNA foundation models are evaluated against normalized k-mer controls to test whether their representations add predictive value beyond sequence composition. Across benchmark classification tasks, NTv3 provides the clearest incremental gains, while the paper’s decomposition and shuffle tests offer complementary diagnostics with bounded interpretation.

  • Problem

    It remains unclear when frozen DNA foundation-model representations provide predictive value beyond sequence composition, and normalized k-mer controls are not consistently included in FM evaluations [13].

    Source: §1 · p. 1

  • Method

    The paper evaluates five frozen DNA FMs on 57 genomic classification datasets and uses matched 6-mer-versus-concatenation tests, ridge decomposition, and composition-preserving perturbations.

    Source: §1 · pp. 1–2, §3.3 · p. 3

  • Results

    Across 57 tasks, NTv3 improves 46/57 with mean ∆add = +0.072 MCC; Caduceus gains are smaller, and the other models do not increase mean performance.

    Source: §4.2 · p. 5

  • Takeaways & Limitations

    Compositional controls and incremental-utility tests complement diagnostics of linear recoverability and order sensitivity, without establishing a specific biological mechanism.

    Source: §6 · p. 9

    The study covers frozen last-layer representations, not fine-tuning or parameter-efficient adaptation, and is not an upper bound on foundation-model performance.

    Source: §5.4 · p. 8

Abstract

from arXiv · show

Do genomic foundation models (FMs) provide predictive information beyond simple sequence composition? We evaluate frozen representations from five DNA FMs against normalized k-mer frequencies across 57 genomic classification datasets and two exploratory gene-expression regression tasks. Simple k-mer features remain competitive with most models, whereas NTv3 shows a consistent advantage; the ranking is robust to globally fixed and multi-scale k-mer baselines and alternative downstream probes. To test incremental predictive utility directly, we compare global 6-mer features with their concatenation with each FM embedding under a matched Random Forest protocol. NTv3 improves 46 of 57 datasets (mean $\Delta$MCC = +0.072), while Caduceus provides smaller gains and the other models do not improve mean MCC. We separately decompose embeddings into a component linearly predictable from 6-mer frequencies and a ridge residual, finding that linear recoverability does not reliably indicate downstream utility. Exact 6-mer-preserving shuffles further reveal task-relevant sequence-order sensitivity, most pronounced for NTv3, although order sensitivity alone does not guarantee predictive improvements. Our results establish compositional features as essential controls and offer complementary diagnostics for evaluating the added value of frozen genomic representations, without making claims about fine-tuning capacity.

1 Introduction

The paper asks whether frozen DNA foundation-model representations add predictive value beyond sequence composition, using normalized k-mer frequencies as an interpretable control. It tests standalone comparisons, incremental value, linear recoverability, and order sensitivity.

1 Introduction

  • The central question is whether frozen DNA foundation-model representations provide predictive value beyond basic sequence composition.Normalized k-mer frequencies offer an inexpensive, interpretable control, but are not consistently included in FM evaluations [13].Source: §1 · p. 1
  • The study evaluates five architectures across 57 genomic classification datasets, with two gene-expression regression tasks as case studies.Classification comparisons use globally fixed k = 6, with fixed-k, multi-scale, and strand-aware checks.Source: §1 · p. 1
  • The controlled comparison tests when frozen DNA FM embeddings outperform fixed and multi-scale k-mer baselines, with checks across probes and pooling choices.The paper does not claim novelty for k-mer competitiveness itself; prior work compared k-mer profiles and genome FMs in metagenomic binning [6].Source: §1 · pp. 1–2
  • The paper tests incremental value by comparing 6-mer features with their concatenation with each FM embedding under matched classifiers.A ridge-based decomposition and composition-preserving perturbations assess linear recoverability and order sensitivity.Source: §1 · p. 2
  • The composition-preserving perturbation test examines downstream-relevant order sensitivity, especially for NTv3, without attributing unexplained variation to a specific biological mechanism.The paper treats this as a separate diagnostic from incremental predictive utility.Source: §1 · p. 2

2 Related Work

Prior evaluations of genomic FMs do not consistently include explicit compositional controls, while classical sequence representations include methods beyond normalized k-mer histograms. This paper studies supervised tasks and treats ridge residuals as mapping-dependent diagnostics rather than inherently biological signal.

2.1 Genomic Foundation Models and Benchmarks

  • Broad genomic FM benchmarks compare models across multiple tasks but do not consistently use explicit compositional controls.The paper’s conclusions concern frozen, task-agnostic feature extraction, not fine-tuning or parameter-efficient adaptation.Source: §2.1 · p. 2

2.2 Classical Sequence Features and Representation Diagnostics

  • Classical sequence methods include mismatch-tolerant kernels, gapped and position-aware variants, and gapped-k-mer methods for regulatory analysis.Prior work compared scalable k-mer profiles with genome FMs for unsupervised metagenomic binning [6]; this paper instead studies supervised functional and regulatory classification.Source: §2.2 · p. 2
  • The normalized k-mer histogram is a minimal compositional control, not an upper bound on classical sequence methods.The paper evaluates incremental value beyond a specified global k-mer baseline.Source: §2.2 · p. 2
  • The ridge residual is mapping-dependent and must not be interpreted as inherently biological or non-compositional signal.Ridge regression predicts an FM embedding from a defined k-mer representation, rather than probing a task-specific biological attribute [1] [16] [3].Source: §2.2 · p. 2

3 Methods

The methods compare frozen FM features with normalized k-mer controls across benchmark tasks and use matched classifiers to test incremental utility. Separate diagnostics examine linear recoverability and order sensitivity while preserving exact 6-mer composition.

3.1 Datasets and Features

  • The study uses 57 classification datasets from the DNA Foundation Models Benchmark and two exploratory gene-expression regression tasks.The Caduceus Bulk RNA result could not be certified against the official dataset configuration and is not used to support the conclusions.Source: §3.1 · p. 3

3.2 Feature Representations

  • The primary compositional baseline is global k = 6, alongside fixed k = 4 and 5, multi-k, and reverse-complement-canonicalized features.Frozen final-layer embeddings come from five architectures, with mean pooling as the primary token aggregation; alternative pooling and token choices are assessed separately.Source: §3.2 · p. 3

3.3 Evaluation and Incremental Predictive Utility

  • Incremental utility compares the same classifier on L1-normalized 6-mer features alone and concatenated with mean-pooled FM embeddings.T6 matches splits, seeds, tuning criteria, and train/test partitions; positive ∆add supports utility only for this classifier and feature protocol, not information-theoretic uniqueness.Source: §3.3 · p. 3

3.4 Linear k-mer Recoverability and Ridge Residual

  • The ridge diagnostic predicts frozen embeddings from 6-mer frequencies and evaluates the predicted component and residual separately as classifier features.The decomposition is conditional on the feature basis and mapping; residuals can include nonlinear composition, order effects, noise, or modeling artifacts, and per-dimension R2 values are averaged without necessarily measuring total variance explained.Source: §3.4 · pp. 3–4
  • Table 1 summarizes dataset loss counts for compositional baselines against standalone FM embeddings; best-k is a retrospective task-specific oracle, not a significance estimate.The table caption describes the counts as datasets where the indicated baseline outperforms the FM.Source: Table 1 · p. 4

3.5 Composition-Preserving Order Perturbation

  • Eulerian-path shuffles preserve each sequence’s exact global 6-mer multiset, allowing the study to test order sensitivity with frozen and retrained linear probes.Three shuffle seeds are averaged within each dataset; the test does not distinguish absolute positions, relative motif arrangement, or a particular regulatory mechanism.Source: §3.5 · p. 4

4 Experimental Results

Across 57 genomic classification tasks, frozen DNA foundation models vary in whether they outperform compositional baselines, with NTv3 showing the clearest gains. Direct concatenation, embedding decomposition, composition-preserving shuffles, and robustness checks further distinguish predictive utility from recoverability and order sensitivity.

4.1 Frozen Embedding Comparisons Across Compositional Baselines

  • Under globally fixed k = 6, k-mers outperform NTv3 on 13 datasets, compared with 40 for HyenaDNA, 44 for DNABERT-2, 26 for Caduceus, and 52 for Evo2.The broad ranking also holds across k = 4, 5, 6 and multi-k controls; the best-k oracle is retrospective, not prospective.Source: §4.1 · p. 4, Figure 2 · p. 5

4.2 Added Value of FM Embeddings Beyond 6-mer Features

  • NTv3 improves 46/57 tasks with mean ∆add = +0.072 MCC, while Caduceus improves 35/57 (+0.022) and the other models do not increase mean performance.These are T6 held-out results comparing [6-mer; FM] with 6-mer alone under the same Random Forest protocol.Source: Table 2 · p. 5, §4.2 · p. 5

4.3 Linear Recoverability Is Not Predictive Utility

  • Median held-out dimension-averaged ridge R2 ranges from 0.75 for HyenaDNA to 0.17 for Evo2, but this ordering does not track standalone quality or concatenation gains.In E3, NTv3's descriptive mean full-minus-prediction MCC is +0.030 across 57 tasks; its ridge residual has median MCC slightly below the 6-mer baseline (−0.002).Source: §4.3 · pp. 5–6, Figure 4 · p. 7

4.4 Order Sensitivity Under Exact Composition Preservation

  • Across three shuffled replicas, frozen-probe MCC decreases by 0.143 for NTv3, 0.063 for HyenaDNA, and 0.024 for DNABERT-2, while retrained-probe drops are smaller.NTv3 loses accuracy on 53/57 datasets after retraining; the intervention indicates downstream-relevant order sensitivity, not specific motif arrangements or absolute genomic position.Source: §4.4 · p. 6, Figure 3 · p. 6

4.5 Robustness to Probes, Pooling, and Strand Treatment

  • NTv3's mean ∆MCC versus fixed 6-mers is +0.061, +0.096, and +0.069 under Random Forest, L2 logistic regression, and MLP probes, respectively.Alternative pooling changes DNABERT-2 and Evo2 mean MCC, while canonical 6-mers improve 20 of 26 prespecified strand-invariant datasets (median +0.018 MCC); the model-level conclusion is unchanged.Source: §4.5 · pp. 6–7

5 Discussion

The discussion treats simple k-mer composition as an essential control and identifies NTv3 as the clearest source of incremental predictive utility in the tested frozen-feature protocol. It cautions that ridge residuals and order sensitivity do not by themselves establish useful non-compositional signal, and limits conclusions to the evaluated setting.

5.1 Why Composition Is a Necessary Control

  • Matching or exceeding frozen FM embeddings with an inexpensive k-mer histogram is an evaluation outcome, not proof that tasks are determined by composition alone.The 6-mer histogram is a reproducible lower-complexity reference, not the strongest conceivable non-neural baseline.Source: §5.1 · p. 7

5.2 What Does the Incremental Test Establish?

  • Adding NTv3 to composition improves mean MCC and a large majority of tasks under the tested frozen-feature Random Forest protocol.Caduceus yields a smaller gain, while the other three models do not.Source: §5.2 · p. 7
  • Order-dependent variation in an embedding is not equivalent to added predictive value beyond the selected composition features.The significant perturbation response of HyenaDNA illustrates this limitation.Source: §5.2 · pp. 7–8

5.3 Interpreting the Ridge-Based Diagnostic

  • A ridge residual is conditional on the specified linear map from global 6-mers and may reflect nonlinear recoverability, finite-sample error, or artifacts rather than biology.Low mean per-dimension ridge R2 does not establish useful or predominantly order-specific features, and a stronger full embedding does not establish a useful standalone residual.Source: §5.3 · p. 8

5.4 Scope and Limitations

  • The study covers frozen last-layer representations and specified pooling and probe protocols, not fine-tuning, adaptation, or systematically probed intermediate layers.Its benchmark splits are not guaranteed locus- or chromosome-disjoint, and only two regression tasks are available.Source: §5.4 · p. 8
  • Richer nonlinear or position-aware compositional controls remain untested, and the shuffle test cannot separate absolute location from relative motif organization.The authors caution that conclusions should remain within the evaluated frozen-representation setting.Source: §5.4 · p. 9

6 Conclusion

The conclusion presents compositional controls and matched incremental-utility tests as complementary tools for evaluating frozen genomic foundation-model representations.

6 Conclusion

  • Across 57 classification tasks, k-mer frequencies provide a strong baseline, while matched concatenation shows substantial incremental utility for NTv3 and smaller gains for Caduceus.The ridge decomposition measures linear recoverability from the chosen features, while composition-preserving perturbations indicate useful order-sensitive variation for NTv3 without identifying a unique mechanism.Source: §6 · p. 9

Funding

The research received no specific funding or external financial support.

Funding

  • The research received no specific funding or external financial support.Source: Funding · p. 9

Code and Data Availability

The appendix provides reproducibility information and sensitivity analyses covering compositional baselines, an independent incremental-utility check, downstream probes, token pooling, and embedding recoverability.

Code and Data Availability

  • The code and experiment instructions are publicly available, and the study uses public benchmark datasets and pretrained models with access instructions and model identifiers in the repository.Source: Code and Data Availability · p. 9

A.1 Multi-scale Compositional Baseline

  • Compared with standalone FMs, concatenated k = 4, 5, and 6 features have larger MCC on 53/57 Evo2 tasks (+0.241) and 53/57 DNABERT-2 tasks (+0.068).The best single k retrospectively is 4 on 28 datasets, 5 on 13, and 6 on 16; that oracle is not an actionable selection procedure.Source: §A.1 · p. 12

A.2 Independent E8 Incremental-Utility Check

  • The independent E8 rebuttal analysis uses k = 6 and mean pooling with a Random Forest grid and CV scaling; its paired incremental MCC results cover 57 datasets and are not pooled with T6.Source: §A.2 · p. 12, Table 3 · p. 12

A.3 Downstream Probe Robustness

  • Table 4 reports mean paired MCC differences for standalone FM embeddings versus fixed 6-mer features, separately from retrospective best-k contrasts.Source: Table 4 · p. 12

A.4 Alternative Pooling and Token Representations

  • For DNABERT-2, CLS outperforms fixed 6-mers on 19/54 tasks (mean ΔMCC = −0.047), compared with 12/54 for mean pooling (mean ΔMCC = −0.082).The alternative pooling table has uneven model coverage, so its means are not matched 57-task comparisons.Source: §A.4 · p. 12, Table 5 · p. 12

A.5 Distribution of Embedding Recoverability

  • The held-out average per-dimension ridge R2 distribution measures linear predictability from the specified compositional basis, not downstream relevance or genuinely non-compositional information.Source: Figure 6 · p. 13

A.6 Residual Robustness and Order-Perturbation Details

Residual and order-perturbation analyses show that NTv3’s residual performance changes under sequence perturbation, but do not identify the responsible sequence attribute. The sequence-length trend weakens after category adjustment, and the exploratory regression analyses are not used for the paper’s conclusions.

A.6 Residual Robustness and Order-Perturbation Details

  • NTv3’s ridge residual showed MCC decreases of 0.161 before retraining and 0.083 after retraining under perturbation, while the global 6-mer histogram remained unchanged.The residual is conditional on the mapping and is not a pure non-compositional feature; the perturbation does not identify which sequence-order attribute is responsible.Source: §A.6 · p. 13, Table 6 · p. 13, Table 7 · p. 13

A.7 Sequence Length Is Not an Explanatory Variable

  • The weak sequence-length trend disappears after adjustment for task category, and the two regression tasks are too few to establish broad predictive trends.Table 8 reports dataset-level Spearman correlations and states that no association passes the specified multiple-test control.Source: §A.7 · p. 13, Table 8 · p. 14

A.8 Original Oracle Scatter and Exploratory Regression Case Studies

  • The original scatter uses a retrospective best-k comparison for sensitivity, while the exploratory regression results are excluded from the confirmatory analysis and mechanistic conclusions.The Caduceus Bulk RNA estimate could not be independently audited against the benchmark configuration and would require a source-verified rerun for quantitative interpretation.Source: §A.8 · p. 14, Figure 7 · p. 14

Appendix B Model Architectures

The five evaluated foundation models span distinct architectures, tokenization schemes, pretraining corpora, and embedding dimensions. Their model-specific design and pretraining details frame interpretation of the decomposition analyses.

Appendix B Model Architectures

  • NTv3-650M is a hybrid U-Net/Transformer encoder with 650M parameters, trained on a multi-species corpus using masked diffusion language modelling and 6-mer tokenization.The extracted embeddings are mean-pooled over non-padding tokens (d = 1536), and the model supports contexts up to 1 million base pairs.Source: Appendix B · pp. 14–15
  • HyenaDNA replaces self-attention with a data-controlled long convolution and uses single-nucleotide tokenization; classification and regression use its 160k and 1M variants, respectively.The model has 1.6M parameters and was pretrained on the human reference genome; mean pooling is applied over sequence positions.Source: Appendix B · p. 15
  • DNABERT-2 is a 117M-parameter BERT encoder with multi-species Byte Pair Encoding and ALiBi positional embeddings; its tokens are variable-length rather than fixed 6-mers.The passage cautions that performance differences are not attributed to token-boundary effects without controlled token-level analysis.Source: Appendix B · p. 15
  • Caduceus-Ph is a bidirectional reverse-complement-equivariant Mamba model with character-level tokenization, reverse-complement augmentation, and mean-pooled embeddings.It has approximately 35M parameters, is pretrained on the human reference genome, and has an approximately 131,000-bp context window.Source: Appendix B · p. 15
  • Evo2-1B is an autoregressive StripedHyena 2 model combining input-dependent convolutions with sparse attention, pretrained on a multi-domain corpus at single-nucleotide resolution.The model has 1 billion parameters and yields mean-pooled final-layer embeddings of dimension 1920.Source: Appendix B · p. 15

C.1 FM vs. K-mer: Wilcoxon and Mixed-Effects Results

The appendix reports statistical comparisons under a retrospective best-k benchmark, probe differences across models, category-specific k-mer patterns, strand-aware canonicalization results, and the scope of per-dataset tables. These analyses qualify rather than replace the paper’s fixed-k comparison.

C.1 FM vs. K-mer: Wilcoxon and Mixed-Effects Results

  • Under the retrospective best-k oracle, negative shifts for HyenaDNA, DNABERT-2, Caduceus, and Evo2 meet stated significance thresholds; NTv3’s positive shift is significant in Wilcoxon but attenuated in the mixed model.NTv3’s advantage is task-selective, with wins concentrated in splice-site and promoter tasks and none on virus or iDNA-ABF datasets.Source: §C.1 · p. 15, Table 9 · p. 16
  • Caduceus’s Wilcoxon p-value is 0.021 while its mixed-effects estimate has p = 0.006, consistent with more uniform losses but category-concentrated wins.The mixed-effects model uses a dataset-level random intercept to account for task-difficulty variability.Source: §C.1 · pp. 15–16

C.2 Classifier Choice: Random Forest vs. Linear Probe

  • Probe outcomes vary by model: RF significantly outperforms LP for HyenaDNA and DNABERT-2, whereas LP outperforms RF for Caduceus; NTv3 and k-mers show no systematic probe advantage.Caduceus’s LP advantage does not translate into systematic superiority over k-mer features.Source: §C.2 · p. 16, Table 10 · p. 16, Figure 8 · p. 17

D.1 k-mer Selection by Biological Category

  • Optimal k varies by biological category: histone-modification and coding tasks favor k = 6, methylation favors k = 4, and promoter and enhancer tasks are largely insensitive.Splice-site tasks show high within-category variance at all tested k values, consistent with positional structure that compositional features cannot fully capture.Source: §D.1 · p. 17

D.2 Strand-Aware Canonical k-mers

  • Canonical 6-mers improve 20/26 strand-invariant tasks, with median +0.018 MCC, but show zero median improvement for strand-specific tasks.The stratified analysis excludes two ambiguous tasks; ordinary non-canonicalized 6-mers remain the reference across all tasks.Source: §D.2 · pp. 17–18, Figure 10 · p. 18

Appendix E Per-Dataset Classification Results

  • The per-dataset table reports historical standalone results against a retrospective best-k oracle, not T6 concatenation outcomes; it marks the best k-mer and overall result for each dataset.Its category groupings and negative MCC values describe the table’s scope and scoring, not a diagnosis of prediction errors.Source: Appendix E · pp. 19–22, Table 11 · p. 20

Appendix F Computational Efficiency

The supplied cost estimates place 6-mer features and NTv3 on the plotted cost-performance Pareto frontier, but they are illustrative operation counts rather than measured runtime. The checklist also records limits in compute-resource verification and repository availability.

Appendix F Computational Efficiency

  • Under supplied 500-bp estimates, 6-mer features and NTv3 lie on the cost-performance Pareto frontier, while the other evaluated points are dominated.Mean MCC is 0.500 for fixed 6-mers and 0.562 for NTv3; these are illustrative complexity estimates, not hardware-normalized latency measurements.Source: Appendix F · p. 22, Figure 11 · p. 22

NeurIPS Paper Checklist

  • The cost comparison relies on externally supplied GFLOP estimates, not measured runtime, memory, or hardware-normalized throughput; verified resource details are unavailable for every experiment.The checklist says operation-count conventions and sequence settings require audit before universal speedup claims.Source: Appendix F · p. 22, NeurIPS Paper Checklist · p. 24
Loading 2610.15711v1…