PaperlayerView exact revision on arXiv

Source-linked AI summary

RAGenome: Scaling Retrieval-Based Genomic Language Models to Long Contexts

Frederikke Isa Marin, Panagiotis Antoniadis, Dionysia Danai Brilli, Andreas Bjerregaard, Rachael DeVries, Yan Li, Ole Winther, Wouter Boomsma

arXiv:2610.11761v1cs.LG

TL;DR

Genomic language models can be computationally expensive, and MSA-based models are limited to short contexts. RAGenome retrieves homologous sequences from whole-genome alignments to extend context while modeling evolutionary relationships. It improves gene finding over GPN-Star while retaining strong performance on evolutionary-driven variant prioritization.

  • Problem

    Genomic language models can require enormous compute and still fall behind traditional methods on some tasks, while MSA-based models are restricted to short contexts.

    Source: §1 · p. 1

  • Method

    RAGenome retrieves homologous sequences from whole-genome alignments of 100 vertebrates, scaling pretraining to longer contexts while explicitly modeling evolutionary relationships.

    Source: §1 · p. 2

  • Results

    RAGenome exceeds GPN-Star on gene finding (MCC 0.60 vs. 0.45) and outperforms NT-MS and Evo2 on variant effect prediction.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    RAGenome indicates that retrieval-based gLMs can unify evolutionary modeling and long-range capabilities within a single, efficient architecture.

    Source: §5.3 · p. 9

    RAGenome requires a whole-genome alignment at inference, and its behavior on species other than human remains untested.

    Source: §6 · p. 9

Abstract

from arXiv · show

The genome holds the blueprint that governs the biological properties of the cell. Consequently, advancing our knowledge of genomic function is crucial both for a broader understanding of biology and for continued biomedical advances. The success of large language models on natural language and protein sequences has motivated similar efforts on genomic data. However, standard genomic language models (gLMs) often require extremely large computational resources and still fall behind traditional methods on some downstream tasks. Recently, MSA-based pretraining has been proposed as an efficient alternative, but existing models are limited to short input contexts, restricting their use to short-range tasks, such as variant effect prediction. In this work, we present RAGenome, the first retrieval-based gLM that scales pretraining to longer contexts (100$\times$ longer than existing MSA-based gLMs), allowing it to capture both across-species evolutionary relationships and within-species longer-range interactions. Trained on whole-genome alignments from 100 vertebrates, RAGenome substantially improves the long-range capabilities of MSA-based gLMs, raising gene finding performance from 0.45 to 0.60, while remaining competitive on purely evolutionary-based tasks like prioritizing pathogenic variants. RAGenome provides competitive gLM performance at a fraction of the training cost, unifying evolutionary modeling and long-range capabilities within a single, flexible, scalable framework. Code is available at https://github.com/PanosAntoniadis/RAGenome.

1 INTRODUCTION

Genomic sequences remain difficult to interpret despite growing sequence data, while genomic language models can be costly and MSA-based models are limited to short contexts. RAGenome uses retrieval to extend context while retaining evolutionary modeling, improving gene-finding performance and remaining strong on variant prioritization.

1 INTRODUCTION

  • Despite growing genome-sequence availability, the ability to interpret genomic sequences remains incomplete, with implications for understanding biology and disease.Source: §1 · p. 1
  • Genomic language models can require substantial training compute, yet large-scale models still fail on some tasks, including prioritizing pathogenic variants.Evo2 is cited as requiring a billion-parameter model trained on trillions of nucleotides over months of compute.Source: §1 · p. 1
  • MSA-based models capture evolutionary relationships but process alignment tokens including gaps, restricting them to short contexts that miss long-range genomic interactions.Their computational cost grows with context length and species count, limiting windows to around 128–256 nucleotides.Source: §1 · p. 2
  • RAGenome surpasses GPN-Star on gene finding (MCC 0.60 vs. 0.45) and maintains strong performance on prioritizing pathogenic variants.The authors describe it as narrowing the remaining gap to large-scale, long-context gLMs.Source: §1 · p. 2
  • RAGenome retrieves homologous sequences from whole-genome alignments of 100 vertebrates, scaling to context windows 100× longer than MSA-based methods.The retrieval approach selectively uses relevant evolutionary information rather than conditioning directly on the full alignment.Source: §1 · p. 2

2 BACKGROUND AND PRELIMINARIES

The background introduces masked language modeling as a training objective for genomic language models and multiple sequence alignments as a source of evolutionary information. Whole-genome alignments extend that alignment framework across species and supply RAGenome's human-query training data.

2.1 MASKED LANGUAGE MODELING FOR GENOMIC SEQUENCES

  • Masked language modeling trains a genomic language model by masking a subset of DNA tokens and minimizing their negative log-likelihood.The text says roughly 15% of tokens are usually masked; short contiguous spans are used to encourage learning dependencies beyond adjacent bases.Source: §2.1 · p. 2

2.2 MULTIPLE SEQUENCE ALIGNMENTS

  • Multiple sequence alignments align homologous positions across biological sequences, revealing conservation patterns that provide an explicit evolutionary signal.Extending this approach genome-wide yields whole-genome alignments across hundreds of species.Source: §2.2 · p. 2
  • RAGenome uses a human-referenced whole-genome alignment of 100 vertebrates, with the human sequence as query and the remaining rows as retrieved homologs.The alignment vocabulary includes a gap token marking the absence of an aligned nucleotide due to an insertion or deletion.Source: §2.2 · p. 3

3 RELATED WORK

Prior genomic language models pursued longer contexts through architectural changes or broader species coverage, while MSA-based models explicitly learned from homologous regions. The paper positions RAGenome alongside these approaches and distinguishes its self-supervised sequence pretraining from supervised functional-track training.

3.1 LONG-CONTEXT GENOMIC LANGUAGE MODELS

  • Long-context genomic language models use approaches including long convolutions, multi-species training, and large-scale training, with some reaching million-token contexts.Evo2's billion-parameter training on 9 trillion nucleotides required several months on thousands of GPUs.Source: §3.1 · p. 3
  • Sequence-to-function models train on experimental tracks or add supervised post-training, which the authors describe as orthogonal to RAGenome's self-supervised sequence pretraining.The passage says supervised post-training on functional tracks could in principle also be applied to RAGenome.Source: §3.1 · p. 3

3.2 MSA-BASED METHODS

  • MSA-based methods model homologous genomic regions across species; GPN-MSA and GPN-Star use whole-genome alignments, while Gamba predicts evolutionary rate as an auxiliary task.GPN-Star uses an inferred phylogenetic tree to constrain attention and achieved state-of-the-art performance across several human variant-effect tasks, according to the passage.Source: §3.2 · p. 3

4 RAGENOME

RAGenome retrieves homologous sequences from whole-genome alignments and combines explicit evolutionary modeling with longer genomic contexts. Its retrieval mechanism, species-aware representations, attention architecture, and staged pretraining support this approach.

4.1 RETRIEVAL MECHANISM

  • RAGenome removes gap tokens, preserves retained tokens’ alignment columns as positional embeddings, and samples species across phylogenetic lineages within a fixed retrieval budget.The budget bounds memory use in cross-attention; the largest context setting uses 80,000 tokens to fit an NVIDIA H100 GPU.Source: §4.1 · p. 4

4.2 TAXONOMY EMBEDDINGS

  • Each species receives a shared embedding built by concatenating embeddings for its NCBI taxonomic lineage, allowing related species to share information through common taxonomy levels.The lineage spans superkingdom through species.Source: §4.2 · pp. 4–5

4.3 ARCHITECTURE

  • The Transformer alternates within-species sliding-window attention, whose cost scales linearly with sequence length, with cross-attention that passes retrieved-species information to the query sequence.Stacking local-attention layers expands the effective receptive field.Source: §4.3 · p. 5, Figure 3 · p. 6

4.4 PRETRAINING

  • Pretraining uses Homo sapiens as the query and 89 retrieved vertebrate species, masking contiguous spans of 3–6 nucleotides at selected non-gap positions.The model selects 15% of non-gap positions per query sequence and masks the same positions in species from the same clade.Source: §4.4 · p. 6
  • Training begins with 1,024-nucleotide segments from the top 5% most conserved regions, then expands coverage to the top 40% at the same context length.The first stage uses regions where retrieval signal is most reliable; the second adds sequence diversity and less-conserved regions.Source: §4.4 · p. 6
  • Context then increases to 4,096 nucleotides and finally the 13,312-nucleotide target length, with the latter covering the entire genome.The target length is a multiple of the 512-nucleotide local-attention window.Source: §4.4 · p. 6
  • Gene-finding performance rises sharply at context-extension transitions, while variant-effect prediction remains flat.The figure attributes the gene-finding pattern to the importance of longer context and the flat variant-effect pattern to local evolutionary conservation signals.Source: Figure 4 · p. 7

5 RESULTS

RAGenome is evaluated on gene finding and variant effect prediction, with pretraining stages that extend context improving long-range performance. Across benchmarks, it performs competitively on both tasks, though larger-scale models lead on gene finding.

5.1 DOWNSTREAM TASKS

  • Gene finding is 9-class nucleotide-level classification using frozen-model embeddings and a lightweight CNN, while variant effects are scored zero-shot by reference-versus-alternate log-likelihood ratios.Source: §5.1 · p. 7

5.2 SCALING RAGENOME IMPROVES LONG-RANGE CAPABILITIES

  • MCC rose from 0.47 to 0.52 at 4,096 nucleotides and from 0.54 to 0.57 at 13,312, while variant-effect performance stayed flat across pretraining stages.The authors attribute gene-finding gains to longer context rather than training compute alone.Source: §5.2 · pp. 7–8

5.3 COMPARISON WITH OTHER MODELS

  • MCC 0.60 puts RAGenome ahead of MSA-based and mid-scale gLMs on gene finding, but behind NT-MS (0.66) and Evo2 (0.81).Table 1 reports results at each model’s native context length.Source: Table 1 · p. 8, §5.3 · pp. 8–9
  • AUROC 0.87 on variant effect prediction trails GPN-MSA (0.90) and GPN-Star (0.92), but exceeds Evo2 (0.80) and NT-MS (0.57).Source: §5.3 · p. 9

6 LIMITATIONS

RAGenome's scope is constrained by its dependence on whole-genome alignments and by the species and model scale evaluated so far.

6 LIMITATIONS

  • RAGenome requires a whole-genome alignment at inference, so it cannot currently be applied where no alignment is available.Its behavior on query species other than Homo sapiens remains untested; applying it to a new species would likely require additional pretraining.Source: §6 · p. 9
  • The study evaluates one 168M-parameter RAGenome model and leaves scaling model size and training data to future work.Source: §6 · p. 9

7 CONCLUSION

RAGenome scales retrieval-based evolutionary modeling to long genomic contexts and shows strong gene-finding performance while retaining evolutionary modeling for variant prediction. The paper also reports its AI use, data ethics, and reproducibility resources.

7 CONCLUSION

  • RAGenome handles contexts up to 13,312 nucleotides and surpasses GPN-Star on gene finding, while outperforming much larger gLMs on variant effect prediction.The authors describe it as unifying explicit evolutionary modeling with long-range genomic understanding in an efficient architecture.Source: §7 · p. 10

AI USE STATEMENT

  • The authors used generative AI for drafting, editing, and suggesting paper structure, titles, and keywords, but not for developing models, methods, experiments, or analyses.They state that they reviewed AI-assisted work and manually verified suggested text against primary sources.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

  • The study uses publicly available genomic data and benchmarks that contain no individually identifiable data.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

A.1 SPECIES SELECTION UNDER THE BUDGET

RAGenome selects species from phylogenetic clades to keep retrieved tokens within a fixed budget while promoting clade diversity.

A.1 SPECIES SELECTION UNDER THE BUDGET

  • Each sample draws one random species per clade before drawing a second from any clade, until the aligned-token budget is reached.The query species' clade is excluded, and the clades are shuffled before sampling.Source: §A.1 · p. 13

A.2 TAXONOMY EMBEDDINGS

The learned taxonomy embeddings organize vertebrate species by taxonomic rank in a UMAP projection.

A.2 TAXONOMY EMBEDDINGS

  • Within Mammalia, species further separate by order rank, indicating that taxonomic information is encoded in the learned embeddings.Source: §A.2 · p. 13
  • Species belonging to the same class form well-separated clusters in the learned taxonomy embeddings.Figure 6 projects embeddings for the 100 vertebrate species in the WGA.Source: Figure 6 · p. 13, §A.2 · p. 13

A.3 SELECTION OF CONSERVED REGIONS

RAGenome’s pretraining selects increasingly less-conserved regions, while inference ablations show benefits from longer contexts and task-dependent retrieval budgets.

A.3 SELECTION OF CONSERVED REGIONS

  • Pretraining progresses from highly conserved, short segments to longer, less-conserved regions, using PhastCons-ranked windows and staged selection fractions.The selected fractions are 5% for stage 1 and 40% for stages 2 and 3; stage 4 uses all valid windows, with a random 0.1% sample of remaining windows included in selected stages.Source: §A.3 · p. 13

A.4 HYPERPARAMETERS

A.5 ABLATION ON INFERENCE CONTEXT LENGTH

  • Across checkpoints, performance improves monotonically with inference context length, and the model trained and evaluated at 13,312 nucleotides performs best.The comparison evaluates checkpoints at their training context length and at shorter lengths used in earlier training stages.Source: §A.5 · p. 14, Table 4 · p. 14

A.6 ABLATION ON RETRIEVAL BUDGET

  • Variant effect prediction saturates by B = 40,000, while gene finding improves slightly up to the full retrieval budget.Without retrieval, performance collapses; the authors suggest variant effects rely on local conservation signals, while gene finding benefits from additional evolutionary context.Source: §A.6 · p. 15, Figure 7 · p. 15
Loading 2610.11761v1…