Genomic Next-Token Predictors are In-Context Learners

1 Introduction

• The motivating gap is that most evidence for emergent ICL comes from human-language models, leaving its dependence on language-specific structure unresolved.The paper contrasts the possibility that language uniquely nurtures ICL with the hypothesis that it can arise in other sufficiently structured domains.Source: §1 · pp. 1–2
• Large-scale autoregressive genomic models make it possible to test whether ICL appears in a non-linguistic sequence domain without explicit meta-learning training.Evo2 is trained on next-nucleotide prediction at a scale comparable to mid-sized language models, while its few-shot reasoning potential had remained unexplored.Source: §1 · p. 2
• Both genomic and linguistic models show log-linear gains in pattern induction and extrapolation as demonstrations increase.The authors describe this as the first evidence of organically emergent ICL in genomic sequences.Source: §1 · p. 3
• Genomic models learn faster with more demonstrations, while linguistic models benefit more from scale; their strongest task types also differ.These patterns are specific to the experimental conditions and reasoning tasks studied.Source: §1 · p. 3
• Evo2 also exhibits robust few-shot learning on a real-world genomic promoter-classification task.This provides an additional validation beyond the symbolic reasoning tasks.Source: §1 · p. 3
• The study introduces a controlled framework that renders symbolic bitstring reasoning tasks in genomic and linguistic forms for direct comparison.Evo2 receives nucleotide encodings and Qwen3 receives digit encodings under matched few-shot prompting.Source: §1 · p. 3

2 Related Work and Broader Context

• ICL is presented as both a form of rapid adaptation from examples and a practical capability used in reasoning and synthetic-data generation.The paper connects it to case-based and analogical reasoning.Source: §2 · p. 3
• Meta-ICL uses an explicit objective to learn from input-output examples, unlike emergent ICL arising from next-token prediction on natural data.The authors distinguish the two partly because Meta-ICL can generalize poorly beyond its training domain, while emergent ICL is described as more flexible.Source: §2 · pp. 3–4
• The paper distinguishes its few-shot transformation learning from zero-shot prediction, noting that Evo2 performs poorly at one shot and improves meaningfully only with many shots.The comparison uses a baseline that guesses the most common output.Source: §2 · p. 4
• Prior genomic work largely lacked a suitable large autoregressive testbed, and some reported ICL-like behavior relied on explicit meta-training or token copying rather than latent-rule induction.Evo2's scale provides a testbed for studying organically emergent ICL in genomic sequences.Source: §2 · p. 4
• Claimed emergent ICL beyond language remains rare; the paper notes that prior non-linguistic examples often used explicit demonstrations or prompts that reduce the task to next-frame prediction.The discussion includes neuro-signal, multimodal, timeseries, and vision models.Source: §2 · p. 4

3.1 Experimental Desiderata

• Cross-domain tasks must work in both alphabets and avoid domain-specific semantics or vocabulary that would bias comparison through unnatural translation.The framework therefore excludes benchmarks tied to language or biological knowledge and tasks needing a richer vocabulary than four symbols.Source: §3.1 · p. 5

3.2 Experimental Setup

• Tasks ask models to infer a hidden transformation from examples and apply it to a query, using bitstrings as a shared symbolic substrate.The setup uses 8-bit inputs and a diverse set of single or composed transformations; per-trial digit and nucleotide mappings are randomized.Source: §3.2 · pp. 5–6
• The task set comprises 100 distinct transformations built from 30 primitives spanning bitwise, structural, positional, pattern, aggregate, and trivial operations.Candidates are programmatically checked for logical distinctness and manually inspected.Source: §3.2 · p. 6

3.3 Model Families and Selection Rationale

• The Qwen3 series ranges from 0.6B to 14B parameters, while Evo2 includes 1B, 7B, and 40B models, enabling measurement of ICL performance across model size and linguistic and genomic modalities.Source: §3.3 · p. 6

3.4 Evaluation Protocol

• Accuracy is estimated from sampled context sets and held-out queries, with a mode baseline that guesses the most frequent context output.The protocol uses eight Monte Carlo trials and estimates uncertainty by a two-stage cluster bootstrap.Source: §3.4 · p. 7
• At comparable model sizes, Evo2 outperforms Qwen3, and performance generally improves with log(shots), though smaller Qwen3 models are less consistent.The figure reports all models above the mode baseline; exact accuracies and error bars are in Appendix C.Source: Figure 2 · p. 8

4.1 Main Results

• Accuracy generally rises linearly with log(shots) in both model families; Evo2 gains monotonically, while Qwen3 has non-monotonic patches before improving from 32 to 128 shots.At 128 shots, Evo2 7B and 40B both surpass 40% accuracy, while Qwen3-14B approaches 35%.Source: §4.1 · p. 8
• At comparable sizes, Evo2 beats Qwen3 at higher shot counts, with the gap emerging around 8–16 shots and persisting at higher counts.Across families, performance is comparable at 1–4 shots.Source: §4.1 · p. 8

4.2 Analysis of ICL Capability Across Model Families and Sizes

• Qwen3 accuracy correlates positively with parameter count, while Evo2 jumps from 1B to 7B and improves little beyond that.The Qwen3 correlation is significant at shot counts of at least 16; the figure reports performance at 128 shots.Source: §4.2 · p. 9, Figure 3 · p. 9

4.3 ICL Sensitivity to Task Complexity: BitLoad Analysis

• At BitLoad 2, Evo begins to outperform Qwen; by BitLoad 4, Qwen falls below 20%, while Evo remains above 40% before nearing 20% at BitLoad 8.Both models achieve near-perfect accuracy on constant tasks and remain similar at BitLoad 1.Source: §4.3 · pp. 9–10, Figure 4 · p. 9
• BitLoad strongly correlates with task accuracy, but deviations within error bars suggest transformation depth and pretraining exposure may also matter.BitLoad counts input-bit positions whose perturbation changes the output.Source: §4.3 · pp. 9–10

4.4 Qualitative Analysis of Functional and Behavioral Differences

• Qwen’s exclusive strengths include right shifts, while Evo2’s include multi-bit transformations; they share 7 of their top 10 tasks.For flip_bits → right_half, Evo2 scores 87.5% and Qwen 25%; Qwen scores 100% and Evo2 50% on spread_last_bit → shift_right_zero.Source: §4.4 · p. 10
• Qwen’s strengths include shifts and global bitstring properties, while Evo2 performs better on full-bitstring operations; this comparison does not establish Evo2’s general ICL superiority.The authors note that the prompting setup may disadvantage Qwen and leave broader comparison to future work.Source: §4.4 · p. 10

Beyond the H1-H2 dichotomy

• Genomic ICL challenges explanations relying solely on language-specific patterns, while results across genomic and linguistic models support a compression-based account and show ICL in a non-pure-Transformer architecture.The discussion frames ICL as a modality-agnostic outcome of large-scale next-token prediction over structured data.Source: §5 · p. 11

Implications for frameworks explaining how ICL operates

• The symbolic tasks provide evidence for task learning, but their lack of pretrained semantic priors limits insight into task retrieval and leaves optimization- and induction-based accounts unresolved.The authors describe the results as most strongly supporting genuine task learning within ICL.Source: §5 · p. 11

Does Evo2 have an innate advantage on these tasks?

• The comparison may favor Evo2 because of its long repeated-nucleotide training sequences and architecture, while bitstring prompts can expose Qwen3’s prior knowledge.The authors caution that the results do not show Evo2 has superior ICL ability and call for better-controlled comparisons.Source: §5 · p. 11

Why not test the models on semantic tasks?

• Bitstring transformations enable cross-modality comparison without relying on semantic pretraining, but the authors limit their findings to the chosen reasoning tasks.The discussion also reports robust scaling ICL on a native genomic promoter-classification task, excluded from the main comparison to maintain parity.Source: §5 · pp. 11–12

Beyond the H1-H2 dichotomy

• Language-specific and modality-general explanations need not be mutually exclusive; shared statistical properties of natural data could contribute to ICL.The authors present the H1-H2 contrast to clarify the alternatives, not to imply only one can hold.Source: §5 · p. 12

6 Conclusion

• Genomic models show clear ICL on bitstring reasoning tasks, paralleling linguistic models and challenging the view that ICL is unique to human language.The authors report log-linear accuracy gains with increasing demonstrations across Evo2 model sizes.Source: §6 · p. 12
• Mechanistic interpretability of Evo2 could investigate how its activations and context aggregation relate to induction circuits studied in language models.The authors also suggest expanding evaluations beyond bitstrings, including codon-level encodings.Source: §6 · p. 12
• The conclusion motivates searching for ICL in other structured domains, including time series, system logs, physics simulations, chess games, and climate projections.Each is described as a patterned substrate that could support contextual reasoning.Source: §6 · p. 12

Broader Impact Statement

• Studying ICL beyond language broadens its empirical basis as a general computational phenomenon, with clearer mechanisms potentially informing model reliability, controllability, and usefulness.The broader-impact statement also connects ICL to interpreting model behavior and applications such as synthetic data generation.Source: Broader Impact Statement · p. 12

A.1 Table of Primitives

• Thirty unary primitives form the library used to construct functions in F.The table describes the primitives used via composition.Source: Table 1 · p. 23, §A.1 · p. 23

A.2 Table of Functions

• The evaluation suite contains 100 functions: 30 single primitives and 70 composed functions.The appendix lists the specific compositions used to construct the suite.Source: Table 2 · p. 24, §A.2 · p. 24

A.3 BitLoad of Functions

• BitLoad is tabulated for every primitive and composed function in F for use in §4.3.Source: Table 3 · p. 25

• For each function, model, and shot count, accuracy is averaged over context sets containing n examples.Source: B A Global Metric Over All Programs · p. 26
• The global benchmark score aggregates performance across all functions F.Source: B A Global Metric Over All Programs · p. 26
• Because exact evaluation is intractable, the authors estimate per-function accuracy by Monte Carlo.The passage points to §3.4 for the estimation procedure.Source: B A Global Metric Over All Programs · p. 26

• The full accuracy results support analysis of Evo2 and Qwen3 ICL capabilities and statistical testing.Source: C Full Accuracy Results · p. 26
• Table 4 reports accuracy ± standard error, in percentages, across model families and shot counts.Models are ordered by parameter count within each family, and bold values mark the best performance within a family.Source: Table 4 · p. 26

• The authors regress accuracy on log(n), interpreting α1 as ICL efficacy and α0 as baseline accuracy.All fitted α1 values are positive.Source: D Meta-Regression for ICL Efficacy… · p. 26
• Both model families have essentially constant α0 across model sizes, with intercepts around 0.12.The authors interpret the similar intercepts as indicating roughly similar prior knowledge on these tasks.Source: D Meta-Regression for ICL Efficacy… · p. 26, Figure 5 · p. 27
• Qwen3’s ICL efficacy more than doubles from 0.6B to 14B, while Evo2’s efficacy increases monotonically from 1B to 40B.Both families’ efficacy increases with parameter count, but the rise is more pronounced for Qwen3.Source: D Meta-Regression for ICL Efficacy… · pp. 26–27
• Evo2 40B significantly outperforms Qwen3 14B in ICL efficacy, while Qwen3 8B is needed to exceed Evo2 1B.In absolute terms, parameter-matched Evo2 models adapt faster than Qwen3 models.Source: D Meta-Regression for ICL Efficacy… · p. 27, Figure 5 · p. 27

E Demonstration of Evo2’s ICL Ability on a Genomic Task

• The task asks whether a 251-nucleotide human DNA sequence is a promoter, with positive examples restricted to non-TATA promoters to remove an obvious marker.Source: E Demonstration of Evo2’s ICL Ability… · p. 28
• Each prompt contains balanced labeled examples followed by a held-out sequence; buffer and class-label nucleotides are randomly reassigned each trial to avoid nucleotide-specific bias.Labels use a 24-nucleotide buffer followed by a 24-nucleotide label run.Source: E Demonstration of Evo2’s ICL Ability… · p. 28
• Evo2 chooses between class-label candidates by comparing perplexity on the final 24 label nucleotides, excluding the buffer.Source: E Demonstration of Evo2’s ICL Ability… · p. 28
• Accuracy rises from 48.2% at 1-shot to 63.6% at 16-shot and 68.6% at 256 shots; a saturated power law fits the 2–256-shot regime (R2 = 0.998).Permuting labels at 16 shots drops accuracy from 63.6% to random chance (50.2%).Source: E Demonstration of Evo2’s ICL Ability… · p. 28
• Performance on human_nontata_promoters improves monotonically with shot count.Figure 6 reports error bars of ± one standard error.Source: Figure 6 · p. 28

F Analysis of ICL with Program Synthesis Tasks: Bit-Diversity

• Few high-BitLoad tasks have high accuracy relative to medium-BitLoad tasks, so theoretical information bottleneck does not always track model difficulty.The authors suggest that some high-BitLoad tasks may be easy because of patterns encountered during pretraining.Source: F Analysis of ICL with Program… · p. 29, Figure 8 · p. 30, Figure 9 · p. 31
• Accuracy trends increase with shots within BitDiversity bins, while models generally perform better on low-entropy outputs.Figure 10 describes performance increasing as output entropy decreases.Source: F Analysis of ICL with Program… · p. 29, Figure 10 · p. 32
• About 25% of true targets are 0-BitDiversity; models overproduce such outputs at low shot counts, but their rate falls toward the expected prior as shots increase.The 1-shot baseline is partly explained by 0-BitDiversity outputs; higher-shot performance includes more correctly predicted non-0-BitDiversity cases.Source: F Analysis of ICL with Program… · p. 29, Figure 12 · p. 34
• Understandable mistakes are most concerning at low shot counts and occur especially on low-BitDiversity tasks; their effect vanishes by 16 shots.For Qwen3-4B, the rate falls from 16% at one shot to below 1% by eight shots and disappears at 32 shots.Source: F Analysis of ICL with Program… · p. 29, Figure 7 · p. 29
• Eight Monte Carlo trials have little aggregate impact but limit task-level trend analysis, and lucky prompts can inflate accuracy without genuine task understanding.The authors suggest increasing trial counts substantially or selectively for tasks of interest.Source: F Analysis of ICL with Program… · p. 29