Source-linked AI summary
When Genomic Masking Priors Fail to Transfer: Strong Variant Prediction, Weak Functional Generation
Susu Hu, Preetam Gattogi, Jens Lehmann, Sahar Vahdati, Stefanie Speidel, Julien Vibert
TL;DR
Genomic benchmarks do not establish that generated sequence preserves locus-specific function, motivating a separate test of functional inpainting. The paper develops GenDA, an absorbing diffusion model with entropy-guided span masking, and finds strong fine-tuned ClinVar prediction but no reliable transfer to zero-shot functional restoration over composition-preserving controls.
Problem
Existing likelihood and variant-prediction benchmarks do not establish whether genomic generators preserve locus-specific molecular function.
Method
GenDA uses bidirectional absorbing diffusion with contiguous entropy-guided masking to test variant prediction and zero-shot functional inpainting.
Results
Strong fine-tuned ClinVar prediction does not transfer reliably to functional generation, and Random Span reaches 0.777 ClinVar AUROC versus 0.774 for GenDA.
Takeaways & Limitations
Discriminative and generative capabilities require separate validation using composition-preserving controls and control-normalized restoration.
Takeaways & Limitations
The study cannot disentangle the effects of 4,096-bp context, 1-mer tokenization, and the 300-bp span cap, and its claims remain limited to the evaluated models, predictor, and loci.
Abstract
from arXiv · showhide
Bidirectional discrete diffusion model appears naturally suited to genomic modeling because it can reconstruct missing sequence from both flanks. We developed GenDA (Genomic Density-optimized Absorbing Diffusion) under the additional hypothesis that entropy-guided span placement would concentrate reconstruction pressure on compositionally complex regions, improving both downstream variant-effect prediction and functional sequence generation. Our results only partially support this premise. After supervised fine-tuning, the 202M-parameter GenDA model reaches a pooled ClinVar SNV AUROC of 0.774, exceeding a similarly scaled autoregressive model by 0.103. However, a matched random-span variant reaches 0.777, providing no evidence that entropy guidance causes the ClinVar improvement. More unexpectedly, GenDA fails a zero-shot functional inpainting stress test: across promoters, enhancers, exon boundaries, and intron boundaries, it does not consistently outperform a control that shuffles the native gap while exactly preserving 3-mer composition. Failure is already present for 50--500-bp gaps, although enhancer degradation worsens at longer gaps. Diagnostics identify several boundary conditions: entropy measures local sequence complexity rather than functional importance; 1-mer tokenization limits physical context; training spans are capped at 300 bp; and high absolute AlphaGenome fidelity can coexist with negative control-normalized restoration. These results show that strong fine-tuned variant prediction, a plausible corruption prior, and functional generation are distinct claims that require separate validation.
1 Introduction
The paper tests whether entropy-guided bidirectional diffusion improves both variant prediction and zero-shot functional generation. GenDA shows strong fine-tuned ClinVar performance, but the entropy prior is not supported as its cause and functional generation remains a separate challenge.
- Motivation: Genomic benchmarks such as likelihood and variant prediction do not establish preservation of locus-specific molecular function during sequence generation.Many statistically plausible sequences may not preserve promoter output, enhancer activity, or splice usage.
- Motivation: Bidirectional masked diffusion can reconstruct missing sequence from both flanks while iterative span corruption encourages broader contextual integration.This contrasts with left-to-right autoregressive generation and isolated-token masking.
- Design premise: GenDA hypothesizes that entropy-guided masking concentrates reconstruction pressure on compositionally complex regions, benefiting variant prediction and functional inpainting.The hypothesis treats reconstruction difficulty as a label-free signal rather than assuming local entropy directly identifies function.
- Results: 0.774 pooled ClinVar SNV AUROC is achieved by GenDA after fine-tuning, versus 0.671 for a similarly scaled autoregressive baseline.A matched Random Span variant reaches 0.777, so the improvement cannot be attributed to entropy guidance.
- Implications: The paper recommends separate validation of discriminative and generative capabilities, alongside composition-preserving controls and control-normalized restoration metrics.It also highlights corruption-scale matching and diagnostic checks as actionable evaluation practices.
2 Related Work
Prior genomic foundation-model work establishes useful representations and scalable generation across masked, autoregressive, and diffusion architectures. This paper focuses on what evidence is needed to assess functional generation rather than relying on proxy benchmarks alone.
- Genomic representation and generation: Masked genomic models learn useful representations, while autoregressive models emphasize scalable DNA generation and other architectures add specialized inductive biases.The cited lines include DNABERT-2, Nucleotide Transformer, HyenaDNA, Evo, Evo 2, and Caduceus.
- Discrete genomic diffusion: Discrete diffusion research spans D3PM, MaskGIT, MDLM, absorbing-state models, and genomic adaptations such as D3 and D3LM.Continuous and simplex alternatives include DNA-Diffusion and Dirichlet flow matching.
- Masking comparison: Figure 1 compares GenDA density-guided spans with independent uniform masking and Random Span.The comparison frames the paper’s masking-policy analysis.
- Predictor-based evaluation: Sequence-to-function predictors provide scalable evaluation of generated DNA, but their outputs remain proxies rather than experimental measurements.The paper uses AlphaGenome for locus-resolved CAGE, chromatin, and splice predictions and compares reconstructions with composition-preserving corruptions.
3 Design Premise
GenDA combines absorbing discrete diffusion with contiguous, density-guided masking and progressive corruption to test whether harder local reconstruction improves genomic prediction and generation. Its decoding procedure iteratively retains confident predictions while revisiting uncertain positions.
- Design premise: GenDA tests whether contiguous spans, entropy-guided starts, and the resulting pretraining configuration improve variant prediction and zero-shot functional inpainting.The third expectation also depends on sufficient context, matched corruption geometry, stable decoding, and a sensitive evaluator.
- Diffusion objective: The diffusion objective reconstructs masked tokens from remaining context at a training-example-specific masking ratio.The masked set defines the reconstruction targets.
- Density-optimized masking: GenDA samples contiguous spans whose locations favor locally complex, nucleotide-dense regions instead of independently masking positions.The design aims to make reconstruction integrate broader context and excludes non-ACGT regions through density guidance.
- Progressive corruption curriculum: The masking curriculum shifts from high ratios for global scaffolding to lower ratios for structural reconstruction and high-resolution refinement.The final stage retains 10% fully masked examples for complete-reconstruction exposure.
- Span sampling: Span starts are sampled categorically and lengths from a clipped Poisson distribution between 1 and 300 bp until the masking budget is reached.The final span may slightly exceed the budget.
- Density-optimized masking: Individual spans are capped at 300 bp, so the objective targets local-to-intermediate reconstruction rather than multikilobase contiguous gaps.The final configuration uses 70% density-guided contiguous spans and 30% other masking, as stated in the full method passage.
- Iterative decoding: During decoding, low-confidence positions return to [MASK] while high-confidence predictions are retained, with refinement steps adapted to the initial gap length.This schedule changes inference without additional training.
4 Evaluation and Results
The evaluation separates fine-tuned ClinVar variant prediction from zero-shot functional inpainting. GenDA performs strongly on the former, but neither entropy guidance nor the complete model reliably improves restoration over a composition-preserving control.
- ClinVar evaluation: GenDA and AR Llama are fine-tuned on identical ClinVar splits, while Evo 2 is evaluated zero-shot using centered 256-bp variant sequences.The models contain 202M and 151M parameters for GenDA and AR Llama, respectively.
- ClinVar evaluation: GenDA scores variants with an inverted log-likelihood ratio and reports AUROC by functional region and across the pooled benchmark.Larger scores indicate that the alternative allele is less compatible with surrounding context.
- ClinVar results: 0.774 overall AUROC is achieved by fine-tuned GenDA, exceeding fine-tuned AR Llama by 0.103 and zero-shot Evo 2 by 0.041.This establishes strong downstream classification for the complete GenDA configuration without identifying which component causes the gain.
- Masking ablation: Exclusive independent 1-mer masking collapses during unconditional generation in the evaluated run, whereas GenDA’s span-dominated mixture does not.This result distinguishes span-dominated masking from the independent-masking configuration in that generation test.
- Entropy guidance: 0.777 overall ClinVar AUROC is reached by the matched random-span variant, compared with 0.774 for GenDA, while inpainting advantages vary by modality.Random spans perform better on enhancers, whereas GenDA has higher median restoration gain for promoters, exon boundaries, and intron boundaries.
- Entropy guidance: ClinVar scoring may be comparatively local because variants are evaluated in centered 256-bp windows, allowing random contiguous spans to train reconstruction at a similar scale.This is a task-specific interpretation and does not imply that all variant effects are local.
- Functional inpainting: The inpainting test reconstructs 50–3,500-bp gaps at 200 held-out sites spanning promoters, enhancers, exons, and introns.GenDA uses both flanks, while AR Llama and Evo 2 generate from left context.
- Functional inpainting: Restoration gain is measured against native-gap shuffles that preserve 3-mer composition, with zero denoting parity and positive values indicating improved restoration.Site-level medians and 95% bootstrap confidence intervals are reported.
5 Diagnosing the Failed Premise
The experiments identify several constraints that complicate functional inpainting and weaken the entropy-guided premise. These include mismatched corruption geometry, a complexity heuristic that does not directly identify function, evaluator sensitivity, and modality-specific effects from inference-time rules.
- Context and corruption geometry: At a 3,500-bp gap, fewer than 600 observed flank nucleotides remain, while training spans average 100 bp and are capped at 300 bp.Only the 50- and 200-bp evaluation gaps fall within the training-span cap.
- Entropy guidance: Local entropy targets compositionally complex sequence but does not identify transcription start sites, enhancer activity, splice regulation, or AlphaGenome-sensitive positions.Random Span slightly exceeds GenDA on pooled ClinVar, 0.777 versus 0.774, and improves enhancer RG from −130.8 to −58.4.
- Evaluator sensitivity: 90.1–95.7 H3K27ac fidelity can coexist with negative enhancer RG because unchanged context, smooth profiles, and weakly responsive channels may dominate predictions.The composition-preserving shuffle makes this control-level comparison explicit.
- Metric caveat: RG can become extremely negative when control distances are small, so values near the 0.01 exclusion threshold remain unstable.The study reports absolute fidelity alongside RG and recommends the non-ratio difference Dcontrol −Dmodel.
- Inference-time guidance: BioCheck improves enhancer RG from −130.8 to −42.6 but degrades promoter RG from −37.9 to −69.4, leaving splice usage essentially unchanged.The heuristic shifts performance between modalities rather than producing general functional restoration.
6 Actionable Takeaways and Limitations
The paper recommends inexpensive controls that align evidence with the claimed capability and expose failures caused by context, corruption scale, evaluator sensitivity, or modality trade-offs. It also limits its empirical conclusions to the evaluated models, predictor, and loci.
- Evidence matching: Variant prediction, sequence likelihood, native-profile similarity, and functional generation measure different capabilities and should not substitute for one another.Success on one capability can motivate, but does not replace, direct evaluation of another.
- Experimental reporting: Physical context and corruption geometry should be reported together because token-window length and total masking ratio do not specify DNA coverage or a single missing interval.Gap length, contiguity, physical coverage, and remaining flank context are recommended reporting dimensions.
- Evaluator calibration: Composition-preserving perturbations should calibrate evaluator sensitivity before generator evaluation, since barely affected controls make high fidelity and ratio-based improvement uninformative.The exact 3-mer-preserving shuffles retain local sequence composition while testing whether the predictor responds to the corrupted gap.
- Stratified evaluation: Gap regimes and functional modalities should be stratified because short-gap failures, long-gap enhancer degradation, and splice insensitivity can be concealed by aggregate scores.Separate reporting distinguishes general failure, scale-specific degradation, and evaluator insensitivity.
- Multi-objective guidance: Biological guidance should be evaluated across modalities and reported as trade-offs rather than as general improvement.A rule or reward that improves one predictor output may degrade another.
- Practical consequence: These checks can reject weak generative claims before investment in larger models, additional predictors, or experimental validation.The proposed protocol is intended as an inexpensive diagnostic screen.
- Limitations: The study cannot separate the effects of 4,096-bp context, 1-mer tokenization, and the 300-bp span cap because matched alternative models were not retrained.Its empirical claims remain limited to the evaluated models, predictor, and loci.
7 Conclusion
GenDA supports a discrimination–generation disconnect: it performs strongly on supervised ClinVar prediction, but entropy guidance is not validated and reliable zero-shot functional inpainting is absent. The conclusion is bounded by unverified boundary conditions and by the evaluated uniform-masking experiment.
- Conclusion: 0.777 pooled ClinVar SNV AUROC for Random Span slightly exceeds GenDA’s 0.774, so the entropy prior is not validated.Neither masking strategy produces reliable zero-shot functional inpainting.
- Discrimination versus generation: Strong fine-tuned variant prediction does not establish that a pretrained model has learned a useful conditional sequence distribution.Compositionally complex sequence is not necessarily functionally important.
- Scope of explanation: Limited physical context, corruption-geometry mismatch, and evaluator sensitivity may contribute to failure, but none is established as its sole cause.The paper recommends testing masking priors against random-span controls and evaluating discriminative and generative capabilities separately.
- Masking experiment: Exclusive independent 1-mer masking causes the evaluated uniform-masking model to concentrate predictions on a single nucleotide throughout denoising.The result applies to the evaluated GenDA setting and does not establish that limited uniform masking is harmful elsewhere.
A.2 Random-Span and Scaling Ablations
Random-span and scaling ablations show no established advantage for density-guided masking, while increased model depth improves final-stage ClinVar performance. Inpainting differences remain task-dependent, and BioCheck changes restoration unevenly across modalities.
- Random-span comparison: 0.777 overall ClinVar AUROC for Random Span slightly exceeds 0.774 for GenDA, so density-guided placement is not established as superior.These are sequential curriculum checkpoints, not compute-matched standalone models.
- Scaling: 0.786 overall ClinVar AUROC makes GenDA-large the strongest final-stage result across all four regions.GenDA-large uses twice the depth of GenDA, while masking policy and capacity are not jointly controlled.
- Inpainting comparison: GenDA has higher median restoration gain for promoters, exon boundaries, and intron boundaries, whereas Random Span performs better on enhancers.Confidence intervals overlap in all four categories, so these are task-dependent trends rather than statistically established differences.
- BioCheck: BioCheck improves enhancer restoration but degrades promoter restoration and has little effect on splice usage.The intervention is interpreted as modality-specific rather than evidence of improved biological validity.
B AlphaGenome Evaluation Details
The AlphaGenome evaluation inserts reconstructions into matched genomic contexts and compares predictor-based restoration against 3-mer-preserving shuffled controls. The protocol reports fidelity and control-normalized gains across genomic categories and gap lengths, but these scores do not establish experimental biological activity.
- Protocol: 200 held-out sites cover 50 promoters, 50 enhancers, 50 exons, and 50 introns across gaps from 50 to 3,500 bp.Each site–model–gap combination has three reconstructions inserted into the same 524,288-bp hg38 context.
- Controls and metrics: Absolute fidelity is scaled so c = 100 denotes an AlphaGenome prediction identical to the native profile, while c = 0 denotes maximal normalized disagreement.Unlike restoration gain, absolute fidelity does not depend on a corrupted control.
- Controls and metrics: Restoration gain compares each reconstruction with a native-gap shuffle that exactly preserves 3-mer composition.Positive RG indicates improvement over the corrupted control, with cases of control distance below 0.01 excluded as unstable.
- Interpretation: AlphaGenome scores measure predictor-based restoration rather than sequence realism, diversity, low-complexity collapse, experimental biological activity, or general sequence quality.The ratio form of RG is sensitive to small control distances despite the eligibility threshold.
- Gap-length analysis: Negative promoter and enhancer restoration gain is already present for short 50–500-bp gaps, while GenDA enhancer gain deteriorates sharply for 2,000–3,500-bp gaps.Splice-usage gain remains near zero across ranges, and extreme enhancer ratios may be unstable when control distances are small.
C.1 Data Curation and Splitting
The study curates diverse human genomic anchors with gene-aware splitting and fixed 4,096-token contexts, while benchmarking multiple controlled and external model comparisons. The resulting setup preserves nucleotide resolution but imposes physical-context and feature-length constraints.
- Data curation and splitting: Anchors include promoters, exons, enhancers, introns, and repeats extracted from hg38, with sequences containing more than 5% uncalled bases discarded.
- Context construction: Sequences are cropped or padded to L = 4,096 tokens, although many introns do not fit completely within the model window.Most promoters, enhancers, and exons fit within the window, whereas many introns do not.
- Sequence formulation and tokenization: A 1-mer tokenizer preserves exact nucleotide resolution for ClinVar scoring and reconstruction but limits the physical context to approximately 4,096 bp.Coarser tokenization could cover more bases but would alter masking, decoding, and allele-level likelihoods; that trade-off is not evaluated.
- Data curation and splitting: Approximately 3.2 million genomic anchors are partitioned with a strict gene-aware split to prevent structural leakage.The dataset uses roughly 80% training, 10% validation, and 10% test partitions by gene_id.
- Model comparisons: GenDA uses a bidirectional ModernBERT backbone, while Llama provides a native causal baseline and external generators are treated as descriptive comparisons.The controlled comparison matches core attention scale rather than exact parameter count.
C.4 Training Hyperparameters and Masking
GenDA is trained with a three-stage curriculum and a heuristic mixture of density-guided and uniform masking. The masking configuration emphasizes locally complex sequence while retaining independent 1-mer masking examples.
- Masking distribution: The density-guided scanner uses a sliding window of W = 9 and clipped-Poisson span lengths with λ = 100.The final configuration applies density-guided spans to 70% of examples and independent position-uniform 1-mer masking to 30%.
- Masking distribution: Density guidance is a heuristic configuration rather than an exhaustively tuned masking policy.The selected masking proportions and distribution parameters were chosen heuristically.
D Reproducibility Statement
The paper states that its empirical claims and experimental details are documented through appendices, checklist responses, and planned code release. It also discusses limitations, societal risks, and the absence of theoretical results.
- Reproducibility: The appendix documents data splits, model sizes, tokenization, training, masking, decoding, evaluation contexts, controls, samples, and bootstrap procedures.Code and processed evaluation metadata are planned for release upon publication, subject to licensing constraints.
- Limitations and societal impact: The authors identify predictor scores as insufficient evidence that generated sequences are biologically valid or safe.They call for experimental validation, biosafety review, and application-specific oversight.
- Authorship and verification: Language models assisted with editing and organization but did not generate measurements, select results, or replace author verification.
- Theoretical scope: The study is primarily empirical and introduces no novel theoretical results or formal proofs.
- Statistical reporting: The paper reports 95% confidence intervals for experimental results.