PaperlayerView exact revision on arXiv

Source-linked AI summary

VANDAM: Viewing a nucleotide sequence with DNA molecular priors

Jeremy Levy, Ariel Larey, Yury Nahshan, Raizy Kellerman, Elay Dahan, Amit Bleiweiss, Guy Leib, Omri Nayshool, Dan Ofer, Tal Zinger, Dan Dominissini, Gideon Rechavi, Marissa Wirth, Simon Lee, Dung Hoang, Noam D. Beckmann, Shane O'Connell, Nicole Bussola, Alexander W. Charney, Yoli Shavit, Nati Daniel

arXiv:2610.00411v1cs.LGstat.ML

TL;DR

Genomic foundation models largely train by predicting nucleotide tokens, without directly requiring representations to preserve molecular properties relevant to genomic function. VANDAM adds regional-property prediction during self-supervised training and local feature injection when functional labels can reward their use. It improves downstream performance across architectures and held-out tasks, while probing shows increased access to held-out molecular descriptors.

  • Problem

    Token-identity objectives do not directly require genomic representations to preserve molecular properties relevant to functional and phenotypic tasks, although many DNA properties can be estimated from sequence.

    Source: §1 · p. 1

  • Method

    VANDAM predicts sequence-derived regional molecular properties from pooled representations and additionally injects local molecular descriptors when functional labels can reward their use.

    Source: §1 · p. 2

  • Results

    Across four backbone families and nine held-out tasks, VANDAM improves point estimates in 29 of 36 pre-training and 33 of 36 multi-task comparisons.

    Source: §1 · p. 2

  • Takeaways & Limitations

    VANDAM's molecular-prior interventions improve downstream genomic representations, and RPP increases access to held-out molecular descriptors.

    Source: §6 · p. 9

    The study focuses on local genomic-tile properties and comparatively small backbones with contexts of at most 8,192 bp, so it does not establish behavior as prior scale, model capacity, and context grow together.

    Source: §5 · p. 8

Abstract

from arXiv · show

Contemporary Genomic Foundation Models (GFMs) rely on a DNA-as-a-string paradigm that employs masked token prediction objectives for pretraining. However, this abstraction does not explicitly model the biochemical, structural, and physical properties essential to biological function. Many molecular properties can be estimated from sequence using established biophysical models, so their utility lies not in providing an independent modality, but in introducing priors that training objectives can explicitly exploit. We introduce VANDAM, a framework that extends the training of GFMs with DNA molecular priors. In self-supervised training, VANDAM predicts regional molecular properties from pooled representations. When functional labels are available and can reward retaining molecular priors, local features are additionally injected at the input. VANDAM consistently improves downstream performance across four architecture families and nine held-out genomic tasks by complementing token-based objectives. Probing experiments further demonstrate that the use of molecular priors generalizes to other unseen molecular properties.

1 Introduction

Genomic foundation models learn from token-prediction objectives, which do not directly require representations to retain molecular properties relevant to genomic function. VANDAM adds sequence-derived molecular priors through regional-property prediction and, when functional labels are available, local-feature injection; it improves performance across held-out tasks and architectures.

1 Introduction

  • Token prediction does not directly require representations to preserve molecular properties relevant to functional and phenotypic genomic tasks.GFMs largely inherit masked language modeling and next-token prediction objectives that train recovery or prediction of token identity.Source: §1 · p. 1
  • Many DNA molecular properties, including duplex free energy and helical shape, can be estimated directly from sequence using established biophysical models.The paper argues GFMs should represent DNA as a molecule as well as a string of symbols.Source: §1 · p. 1
  • VANDAM combines Local Feature Injection, which adds local molecular descriptors to token embeddings, with Regional Property Prediction, an auxiliary objective for predicting regional properties.RPP is used in self-supervised pre-training, while LFI is additionally used in multi-task training when functional labels can reward using the descriptors.Source: §1 · p. 2
  • Across four backbone families and nine held-out tasks, VANDAM improves point estimates in 29 of 36 compute-matched pre-training comparisons and 33 of 36 multi-task comparisons.The improvements are statistically significant in 25 and 23 comparisons, respectively.Source: §1 · p. 2
  • Matched non-biological controls indicate that the gains depend on biological target content rather than auxiliary capacity alone.Frozen-representation probes also show increased linear access to six of eight held-out qDNA descriptors on an independent confirmation split.Source: §1 · p. 2

2 Related Work

Genomic foundation models vary in architecture and input representation, yet most retain token-identity objectives. Prior biophysical-supervision work is narrower in modality or task, while VANDAM frames a broader framework for sequence-derived DNA molecular properties.

2 Related Work

  • Despite diverse architectures and tokenization schemes, prominent genomic foundation models primarily use masked recovery or autoregressive prediction of token identity, without explicitly supervising DNA's physical properties.Examples include DNABERT-2, NTv3, Caduceus, and Evo2.Source: §2 · p. 2
  • Prior approaches alter genomic training signals or representations through functional-embedding supervision, diffusion, population variation, biologically informed tokens, or evolutionary information.DNAGPT adds sequence-order classification and GC-content regression; these studies point to training signal and genomic context as factors in representation quality.Source: §2 · p. 2
  • Biophysical supervision has used sequence-derived RNA structure and free-energy targets or aligned DNA representations with physicochemical features for bendability prediction.UTR-LM addresses RNA, while MIXBend is tailored to DNA bendability; VANDAM targets a broader set of DNA properties under self-supervised and functionally supervised training.Source: §2 · pp. 2–3
  • DNA-shape descriptors have improved task-specific supervised predictions, including transcription-factor binding and core-promoter prediction.The cited evidence primarily concerns task-specific supervised settings, whereas Figure 1 depicts VANDAM's local-feature and regional-property routes.Source: §2 · p. 3, Figure 1 · p. 3

3 Methods

VANDAM grounds genomic models in sequence-derived molecular properties through local input features and regional prediction targets. Which mechanism is used depends on whether the training objective can make those properties consequential.

3.1 From DNA as a String to DNA as a Molecule

  • VANDAM frames molecular priors as a complement to DNA-as-a-string modeling, providing biophysical inductive bias without an additional experimental modality.The section introduces the molecular features before presenting LFI and RPP and their use under different supervision regimes.Source: §3.1 · p. 3

3.2 DNA Molecular Priors

  • VANDAM defines 14 per-nucleotide molecular profiles: 13 DNAshapeR structural features and minor-groove electrostatic potential.The profiles cover minor-groove environment, intra-base-pair geometry, and inter-base-pair geometry; they are model-derived priors, not direct measurements, and omit cell-specific context, environmental conditions, and conformational variability.Source: §3.2 · p. 4

3.3 Local Feature Injection

  • Local Feature Injection adds sequence-derived biological priors at the input without modifying the backbone's internal blocks.Nucleotide-level features are mean-pooled over each token's nucleotide span, encoded, concatenated with token embeddings, and projected to the backbone input width; features use only nucleotides visible under the native objective.Source: §3.3 · p. 4

3.4 Region Property Prediction

  • Regional Property Prediction improves on local masked-feature prediction by supervising tile-level properties, encouraging pooled representations to retain information distributed across broader sequence context.The paper notes that a short visible sequence window could suffice for predicting local masked features, making downstream effects hard to distinguish from generic auxiliary regularization.Source: §3.4 · p. 4

3.4.1 Regional Properties

  • The regional targets comprise six sequence-derived properties spanning geometry, composition, duplex stability, and non-B-DNA structural propensity.They are intrinsic curvature, AT content, melting propensity, and G-quadruplex, cruciform, and Z-DNA propensity.Source: §3.4.1 · p. 4

3.4.2 Learning with Regional Properties

  • RPP partitions sequences into fixed non-overlapping tiles, pools backbone hidden states for each tile, predicts sequence-derived properties, and combines its mean-squared-error loss with the native token loss.The RPP coefficient warms linearly from zero during the first 1,000 optimizer steps; both losses use the same forward pass.Source: §3.4.2 · p. 5

3.5 VANDAM Training Regimes

  • VANDAM combines token learning with regional molecular-property prediction during pre-training and adds local feature injection in multi-task training when functional labels can reward using the descriptors.VANDAM-PT omits LFI because its deterministic descriptors are already available from context and are not themselves supervised; VANDAM-MT combines LFI with RPP.Source: §3.5 · p. 5

4 Experiments

Experiments compare VANDAM with compute-matched baselines across four backbones and two training regimes, then test whether biological target content matters and whether RPP transfers to unseen qDNA descriptors.

4.1 Setup

  • The setup spans four architecture families and nine held-out tasks, using compute-matched controls for continual pre-training and mixed-objective multi-task training.Evaluation includes zero-shot variant-effect tasks and frozen-representation linear probes; paired bootstrap and randomization tests with within-regime BH adjustment assess differences.Source: §4.1 · pp. 5–6

4.2 Downstream Performance and Component Ablations

  • VANDAM improved point estimates in 29 of 36 pre-training and 33 of 36 multi-task task–backbone comparisons, with statistically significant gains in 25 and 23, respectively.The compute-matched baseline was significantly superior on three pre-training and one multi-task comparison.Source: §4.2 · p. 6, Table 2 · p. 6, Table 3 · p. 7
  • Token-objective metrics vary by backbone and regime rather than increasing uniformly, so downstream gains are not explained solely by improved token-identity prediction.Table 4 compares LFI and RPP in isolation and in combination on NTv3-8M.Source: §4.2 · pp. 6–7, Table 4 · p. 7

4.3 Does Biological Content Matter?

  • During pre-training, biological RPP exceeds algorithmic and shuffled-table controls by 3.4 and 4.7 mean-AUROC points; under multi-task training, randomizing LFI reduces mean AUROC by 3.9 points.Replacing biological RPP with shuffled-table or algorithmic targets in multi-task training reduces mean AUROC by 1.1 and 1.6 points, respectively.Source: §4.3 · p. 7, Figure 2 · p. 7

4.4 RPP Transfers to Unseen Molecular Descriptors

  • VANDAM-PT improves predictability for six of eight qDNA descriptors, and the reported transfer persists after residualization against the six RPP targets.The study uses qDNA electronic and photophysical observables as held-out descriptors; residualization tests whether gains are explained solely by linear recombination of the supervised targets.Source: §4.4 · pp. 7–8, Table 5 · p. 8

5 Discussion

Molecular grounding improves downstream performance in most comparisons across four architecture families, tokenization schemes, and token objectives, while complementing rather than replacing token recovery. The discussion relates this pattern to what training objectives require representations to preserve and notes limits on the study’s scale and scope.

5 Discussion

  • Molecular grounding improves performance in most downstream comparisons across four architecture families, two tokenization schemes, and both MLM and NTP objectives.The RPP-without-MLM ablation underperforms Baseline-PT, indicating that molecular grounding complements rather than replaces the token objective.Source: §5 · p. 8
  • 60.5 mean AUROC for both LFI and Baseline-PT during continual pre-training contrasts with 64.2 versus 63.0 for LFI and Baseline-MT during multi-task training.The discussion attributes the difference to the objective: token recovery can use the same bounded local context as an injected descriptor, whereas functional labels can reward biologically meaningful features.Source: §5 · p. 8
  • Biological-content controls and qDNA probing suggest RPP promotes representations retaining broader molecular structure, and transfer persists after residualization.Source: §5 · p. 8
  • The study covers local-tile properties and backbones with contexts up to 8,192 bp, so it does not establish behavior as prior scale, model capacity, and context grow together.Under MLM, masking makes approximately 47% of interior LFI vectors zero; feature-only imputation was tested only on NTv3-8M, with task-specific effects.Source: §5 · p. 8
  • Supervision for long-range and three-dimensional genome organization is identified as a direction for future work.Source: §5 · p. 9

6 Conclusion

VANDAM grounds genomic representations in DNA as both a sequence and a molecule, using sequence-derived molecular priors in training. The section describes its local descriptors, their alignment to different backbones, and regional properties used for supervision.

6 Conclusion

  • VANDAM uses regional-property supervision during self-supervised continual pre-training and local molecular descriptors when functional labels can reward their use; these interventions improve downstream representations across diverse architectures and token objectives.RPP also increases access to held-out molecular descriptors.Source: §6 · p. 9

A Local Molecular Descriptor Definitions

  • The 14 DNAshapeR channels serve as local features; rotational quantities are measured in degrees and translational quantities in Å.The SantaLucia nearest-neighbor duplex free-energy descriptor is an additional local feature alongside the 14 channels.Source: Table 6 · p. 12, A Local Molecular Descriptor Definitions · p. 12

B Local-Feature-Injection Alignment and Backbone-Specific Handling

  • LFI features are computed only from the corrupted sequence available to the backbone, excluding masked or future bases as appropriate; nucleotide-level features are pooled across BPE tokens, and HyenaDNA uses a past-only window.For Caduceus-PS, the projection is applied independently to each strand in its natural orientation to preserve reverse-complement equivariance.Source: B Local-Feature-Injection Alignment… · p. 12

C Regional Biological Feature Definitions

  • The six regional properties summarize geometry, composition, duplex stability, and non-B-DNA propensity; nearest-neighbor free energy and purine–pyrimidine classification are intermediate inputs, not separate RPP targets.The properties include intrinsic curvature, AT content, melting propensity, G-quadruplex propensity, cruciform propensity, and Z-DNA propensity.Source: C Regional Biological Feature… · pp. 12–13
  • The Z-DNA score measures alternating purine–pyrimidine tract length as a coarse sequence proxy and does not model sequence-dependent Z-DNA stability.Cruciform propensity measures one arm of an inverted-repeat stem; long stems can extrude as cruciform or hairpin structures under supercoiling.Source: C Regional Biological Feature… · p. 14

D RPP Implementation and Backbone Alignment

RPP builds regional targets on base-defined tiles and pools backbone states to predict them. Tile construction and token alignment keep the regional targets consistent across backbones.

D RPP Implementation and Backbone Alignment

  • RPP targets use transformations and calibration statistics estimated offline from training tiles, with target bases restricted to the input crop.Tiles with fewer than 50% valid A/C/G/T bases are excluded from the RPP loss.Source: D RPP Implementation and Backbone… · p. 14
  • For each valid tile, scalar-attention pooling aggregates its hidden states, with attention weights normalized within the tile and a scorer shared across target dimensions.Source: D RPP Implementation and Backbone… · p. 14
  • DNABERT-2 tiles remain defined in nucleotide coordinates, assigning each decoded BPE token wholly to the tile containing its span midpoint.Special, padding, and non-nucleotide tokens are excluded; this lets native token states predict the same type of base-defined regional targets across backbones.Source: D RPP Implementation and Backbone… · p. 15
  • DNABERT-2 uses 150-bp RPP tiles rather than 300 bp elsewhere, and λmax is set per backbone to keep the RPP-to-native-token-loss ratio approximately constant.Source: D RPP Implementation and Backbone… · p. 15, Table 8 · p. 15

E Training Configuration

Training continues from released backbone checkpoints under fixed within-regime settings, while the multi-task training pool is filtered against evaluation examples. The configurations preserve each backbone’s native sequence objective and context choices.

E Training Configuration

  • Both training regimes continue from released checkpoints, and within each regime the VANDAM arms hold data, optimizer, schedule, and update count fixed while varying biological components.Per-backbone compute is used for both Baseline and VANDAM.Source: E Training Configuration · p. 15, Table 10 · p. 15, Table 9 · p. 15
  • Training uses OpenGenome2 with backbone-specific context lengths and training-only random crops and reverse-complementation; evaluation is forward-strand.NTv3 samples four sequence lengths, Caduceus and HyenaDNA use 8,192-bp contexts, and DNABERT-2 uses 3,072-bp crops with 512 BPE tokens.Source: E Training Configuration · pp. 15–16
  • NTv3, Caduceus, and DNABERT-2 use 15% random token masking, whereas HyenaDNA retains autoregressive next-token prediction.Source: E Training Configuration · p. 16
  • Every tenth optimizer step is task-only; remaining steps train the native sequence objective, with VANDAM-MT additionally training shared RPP losses on sequence-only batches.The backbone is frozen during the first 1,000 task-only steps; zero-shot variant tasks use reference-versus-alternate sequence log-likelihood ratios instead of supervised heads.Source: E Training Configuration · p. 16

F Multi-Task Training Pool and Overlap Control

  • The eight-task multi-task pool is task-disjoint from evaluation, and record filtering removes examples overlapping evaluation under allele, position, sequence, locus, and window rules.For coordinate-resolvable examples, retained training records are locus/window-disjoint under the declared rules.Source: F Multi-Task Training Pool and Overlap… · p. 16, Table 11 · p. 16

G.1 Tasks

The evaluation suite covers zero-shot variant scoring and supervised functional-genomics tasks. It uses backbone-specific prediction protocols and task-appropriate metrics, including separate macro metrics for histone marks and CAGE.

G.1 Tasks

  • The nine-task suite combines four zero-shot variant tasks—BRCA1, ClinVar, OMIM, and TraitGym—with five supervised tasks spanning TF binding, eQTL, gene-expression variants, histone marks, and CAGE.Source: §G.1 · pp. 16–17, Table 13 · p. 17

G.2 Prediction Protocols

  • Zero-shot variants are scored by cosine dissimilarity between pooled reference and alternate representations, with larger perturbations ranking as more likely pathogenic and no task-specific fitting.The single-nucleotide change can affect multiple contextualized states, and DNABERT-2 may also change local BPE segmentation.Source: §G.2 · p. 17
  • For supervised tasks, frozen backbone representations feed trained linear heads, with paired variant tasks using concatenated alternate and reference representations.Histone marks use 20 sigmoid outputs, while CAGE predicts binned log 1p counts from bin-pooled states.Source: §G.2 · pp. 17–18

G.3 Evaluation Metrics

  • AUROC evaluates variant ranking and binary probes; histone marks use track-averaged AUROC, while CAGE reports mean per-track Pearson correlation.CAGE chance is zero, so it is not included in the AUROC mean.Source: §G.3 · p. 18

H Paired Statistical Inference for Main Results

The section specifies paired inference for the main comparisons, contrasts local feature reconstruction with regional property prediction, and describes learned use of DNAshape inputs. The reported pathway measurements are descriptive rather than evidence of causal task importance.

H Paired Statistical Inference for Main Results

I Masked Feature Modeling and the Locality Control

  • 63.8 mean AUROC for biological RPP exceeded algorithmic and shuffled-table regional controls by 3.4 and 4.7 points, while MFM’s 1.2-point gain over random 5-mer control was concentrated on ClinVar.RPP’s largest improvements over both regional controls occurred on ClinVar and TraitGym; the authors interpret the pattern as consistent with regional supervision better serving broader-context integration.Source: Table 16 · p. 21, I Masked Feature Modeling and the… · p. 21

J Learned Utilization of the DNAshape Feature Pathway

  • In a single trained checkpoint, DNAshape’s centered branch contribution had median magnitude 0.51× the sequence branch (IQR 0.42–0.66×); this descriptive measure does not establish causal importance for an individual feature or downstream task.ProT induced the largest mixed-embedding change under a one-standard-deviation feature perturbation (0.393), while Buckle induced the smallest (0.135).Source: J Learned Utilization of the DNAshape… · p. 22, Figure 3 · p. 22

K RPP Target Ablation

Removing RPP targets has task-dependent effects: no target removal lowers every score, and different targets contribute to performance across tasks. Folding and curvature supervision show the largest reported declines on OMIM.

K RPP Target Ablation

  • OMIM scores fall from 59.00 to 49.72 without folding supervision and to 49.51 without curvature supervision.These are the largest reported declines among the target removals.Source: K RPP Target Ablation · p. 22, Table 17 · p. 22
  • Removing any target group decreases eQTL, histone, and CAGE performance, although no target removal lowers every score.The ablation effects are task-dependent.Source: K RPP Target Ablation · p. 22, Table 17 · p. 22
  • The results indicate that regional properties provide complementary signals rather than one uniformly responsible target for downstream gains.Source: K RPP Target Ablation · p. 22

L Intrinsic Token-Objective Evaluation

The section evaluates native-token performance, biological-content controls, qDNA probing, and masked-base handling in local feature injection. It reports qDNA probing gains for selected descriptors and task-dependent effects from imputation.

L Intrinsic Token-Objective Evaluation

M Biological-Content Controls

  • Matched controls hold backbone, data, and training budget fixed while randomizing injected descriptors or replacing RPP targets to test biological content.The shuffled-table control preserves target distributions and correlations but disrupts pentamer-to-shape correspondence; the algorithmic control uses generic regional statistics.Source: M Biological-Content Controls · p. 23

N qDNA Representation Probing

  • Six qDNA descriptors pass both selection stages in the original and residual-target analyses; residual-target effects range from 0.051 to 0.091 absolute R2 (mean 0.069).Both maximum-transfer channels remain exceptions; qDNA descriptors are distinct from the properties used for RPP supervision.Source: N qDNA Representation Probing · p. 24, Table 20 · p. 25

O Sensitivity to Masked-Base Handling in LFI

  • Masked-base imputation improves BRCA1, gene-expression variants, and histone prediction, but reduces TF-binding and CAGE performance.Conditional sampling does not consistently outperform uniform sampling; because the sensitivity analysis is limited to NTv3-8M, zeroing remains the multibackbone policy.Source: O Sensitivity to Masked-Base Handling… · p. 25, Table 21 · p. 25
Loading 2610.00411v1…