PaperlayerView exact revision on arXiv

Source-linked AI summary

The Geometric Alignment Tax: Tokenization vs. Continuous Geometry in Scientific Foundation Models

Prashant C. Raju

arXiv:2604.04155v1cs.LGcs.ITq-bio.QMstat.ML

TL;DR

Scientific foundation models are evaluated by predictive accuracy, but these metrics do not establish whether their representations preserve continuous system geometry. Using controlled synthetic experiments, information-theoretic analysis, and evaluations of 14 biological models, the paper finds discrete tokenization produces major geometric instability and three recurring failure regimes.

  • Problem

    Predictive-accuracy metrics do not establish whether foundation-model representations preserve the continuous geometry of biological and physical systems.

    Source: §1 · p. 1

  • Method

    The paper combines controlled synthetic ablations, rate-distortion theory, MINE, and evaluation of 14 biological foundation models to analyze geometric alignment.

    Source: §1 · p. 2, §4.1 · p. 7

  • Results

    Under continuous objectives, architectures differ by 1.3×, whereas discrete tokenization separates them by 3,000× and yields three failure regimes.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    No evaluated model simultaneously achieves low distortion, high mutual information, and global coherence, exposing limits of predictive accuracy as a geometry-alignment measure.

    Source: §5 · p. 10

    The findings are scoped to AI for Science and models up to 15B parameters and 1M context length, leaving larger-scale mitigation unresolved.

    Source: §5 · p. 10

Abstract

from arXiv · show

Foundation models for biology and physics optimize predictive accuracy, but their internal representations systematically fail to preserve the continuous geometry of the systems they model. We identify the root cause: the Geometric Alignment Tax, an intrinsic cost of forcing continuous manifolds through discrete categorical bottlenecks. Controlled ablations on synthetic dynamical systems demonstrate that replacing cross-entropy with a continuous head on an identical encoder reduces geometric distortion by up to 8.5x, while learned codebooks exhibit a non-monotonic double bind where finer quantization worsens geometry despite improving reconstruction. Under continuous objectives, three architectures differ by 1.3x; under discrete tokenization, they diverge by 3,000x. Evaluating 14 biological foundation models with rate-distortion theory and MINE, we identify three failure regimes: Local-Global Decoupling, Representational Compression, and Geometric Vacuity. A controlled experiment confirms that Evo 2's reverse-complement robustness on real DNA reflects conserved sequence composition, not learned symmetry. No model achieves simultaneously low distortion, high mutual information, and global coherence.

1 Introduction

The paper identifies the Geometric Alignment Tax as the geometric distortion caused by forcing continuous systems through discrete tokenization, which predictive metrics fail to detect. Across synthetic and biological foundation models, it links this tax to symmetry failure, scaling limits, three failure regimes, and misleading reverse-complement robustness.

Core claim

  • Cross-entropy over discrete tokens is sufficient for symmetry failure because discretization, not the backbone architecture, distorts continuous physical manifolds.The Geometric Alignment Tax is presented as the intrinsic cost of forcing continuous manifolds through discrete categorical bottlenecks.Source: §1 · pp. 1–2

Motivation

  • Scaling parameters and context windows reduce visible discretization steps but do not restore continuity, because manifold fractures converge too slowly for practical geometric fidelity.The paper characterizes increased resolution as an illusion of smoothness: cumulative directional errors remain governed by slow scaling laws.Source: §1 · pp. 1–2

Core claim

  • 3,000× divergence separates architectures under discrete tokenization, versus only 1.3× under continuous objectives, showing tokenization dominates geometric stability.The discrete-to-continuous gap within one architecture exceeds the cross-architecture gap on synthetic dynamical systems and a biological mutation walk.Source: §1 · p. 2
  • Finer VQ quantization improves reconstruction but worsens geometric stability, with 1/log K distortion scaling implying exponentially more codes are needed for continuous performance.Boundary-crossing probability under perturbation increases as codebooks become finer, preserving the geometric tax despite higher resolution.Source: §1 · p. 2

Biological validation

  • Evo 2’s apparent reverse-complement robustness on real DNA reflects conserved k-mer composition rather than learned symmetry.A controlled four-condition experiment isolates per-sequence k-mer histogram matching as the source of the observed signal.Source: §1 · p. 2

Information-theoretic analysis

  • Across 14 biological foundation models, rate-distortion theory and MINE identify Local-Global Decoupling, Representational Compression, and Geometric Vacuity as distinct failure regimes.These regimes respectively describe failed global integration, mutual-information concentration that sacrifices fidelity, and smooth embeddings carrying less structure than random noise.Source: §1 · p. 2

2 Ground Truth: The Controlled Experiments

Controlled experiments show that continuous output objectives preserve geometry far better than discrete tokenization, while finer codebooks improve reconstruction but can worsen geometric distortion. Across domains, continuous architectures differ modestly, whereas tokenized biological models diverge by roughly 3,000× and exhibit fracture, absorption, or collapse.

2 Ground Truth: The Controlled Experiments

  • The controlled harness measures clean–perturbed RDM similarity, perturbation stability, feature split, sample split, and Lorenz dynamical fidelity under global 256-bin discretization.Experiments cover WAVEFORM, OSCILLATOR, and LORENZ using three small architectures; perturbations include 1%, 2%, 5%, and 10% value noise plus time reversal.Source: §2 · pp. 2–3

2.1 The Causal Proof: Continuous vs. Discrete

  • Under discrete cross-entropy, Lorenz dynamics remain preserved, but geometric stability differs only within the same order of magnitude across SmallBERT, SmallMamba, and SmallStripedHyena.LLE estimates are 0.036, 0.038, and 0.038 versus ground truth 0.037, while mean composite stability scores are 0.466, 0.400, and 0.470, respectively.Source: §2.1 · p. 3
  • 8.5× lower Lorenz distortion results when SmallStripedHyena replaces discrete cross-entropy with a continuous MSE head, while the identical encoder and protocol are retained.At 1% noise, distortion falls from 0.072 to 0.0085; across architectures, the discrete range 0.072–0.157 collapses to 0.0085–0.034.Source: §2.1 · p. 3, Figure 1 · p. 4
  • D = 0.105 at K = 1024 despite reconstruction MSE improving to 0.00014, demonstrating a non-monotonic VQ double bind in which finer codebooks worsen geometry.Distortion reaches a shallow optimum at K = 64 (D = 0.073), then rises; the observed scaling is D_proc ∝1/log K rather than the expected 1/K.Source: §2.1 · p. 3, Figure 1 · p. 4

2.2 Track A + Track B: The Smoking Gun

  • ∼3,000× separates tokenized biological models, versus 1.3× across continuous architectures, showing tokenization—not routing—drives the cross-domain geometric gap.Continuous oscillator embeddings are smooth across all architectures, while BRCA1 mutation walks produce three-orders-of-magnitude differences and distinct failure behaviors in genomic models.Source: §2.2 · pp. 4–5

3 Model Performance at Scale

At scale, ESM-2 and several tokenized models exhibit worsening geometric stability, while Caduceus remains nearly invariant. Longer context does not resolve Evo 2’s reverse-complement failure, whose apparent success on real DNA is explained by compositional statistics rather than learned symmetry.

3.1 Parameter Scale vs. Stability

  • ESM-2 stability declines from 0.463 (8M) to 0.391 (3B), then rebounds to 0.445 (15B), while Procrustes reduction identifies global drift rather than geometric recovery.The same monotonic decline and illusory 15B recovery appears on UniRef50, from 0.466 (8M) to 0.397 (3B) to 0.448 (15B).Source: §3.1 · p. 5
  • Caduceus remains nearly stable across scale (0.459, 0.449, 0.458), whereas NT v2 drops from 0.239 to 0.210 and SaProt degrades from 0.436 to 0.398.Caduceus also achieves near-perfect real-DNA reverse-complement preservation (RDM > 0.999), while ProtMamba’s low distortion is later shown to be informationally empty.Source: §3.1 · pp. 5–6
  • Procrustes reduction separates Brittle Glass (ρ = 0.7–1.8%) in ESM-2 8M–650M from Untethered Gel (ρ = 5.2–5.0%) in ESM-2 3B–15B.Under reversal, ESM-2 3B–15B reaches ρ = 19.7–20.1%, while Evo 2 frozen-head agreement falls from ∼85% under 1% SNP to ∼28% under synthetic RC.Source: §3.1 · p. 6

3.2 Context Length and the RC Dissociation

  • Evo 2’s 1% SNP RDM similarity rises only from 0.747 (8K) to 0.817 (1M), while real-chr22 similarity changes from 0.990 to 0.993.Frozen Head accuracy remains 0.988 (8K), 0.980 (262K), and 0.993 (1M), indicating effectively zero geometric gain from 128× more context.Source: §3.2 · p. 6
  • Synthetic-DNA RC similarity remains low at 0.139 (8K), 0.156 (262K), and 0.208 (1M), despite real-DNA values near 0.88.These results show that context scaling does not restore Evo 2’s A↔T / C↔G symmetry understanding.Source: §3.2 · p. 6
  • Dinucleotide shuffling recovers 97% of Evo 2’s real-random RC gap, whereas texture-matched Markov sequences recover 3%, showing that apparent RC success reflects per-sequence k-mer histograms.RCCR forces perfect pointwise RC consistency but increases population-level disparity by 91%, makes RC RDM similarity −0.036, and collapses SNP sensitivity by two orders of magnitude.Source: §3.2 · pp. 6–7, Figure 3 · p. 7

4 The Information Theory of the Tax

The section formalizes the Geometric Alignment Tax as a rate–distortion trade-off caused by discrete bottlenecks and quantifies its biological consequences with mutual information estimation. Across three failure regimes, no studied discrete-token model achieves low distortion, high mutual information, and global coherence simultaneously.

4.1 Rate–Distortion Framing

  • For a K-token vocabulary, capacity is log2 K bits per token and reconstruction distortion scales as K−2/dM, so halving error requires a 2^dM larger codebook.This high-rate quantization result is attributed to Gersho and Gray (1991) and Gray (1990).Source: §4.1 · p. 7
  • Geometric distortion differs from reconstruction error: Procrustes distortion follows Dproc ∝1/log K (R2 = 0.98, p < 0.001), while reconstruction MSE improves monotonically with K.The VQ double bind yields a shallow optimum at K=64 and rising distortion through K=1024 because finer cells increase perturbation boundary crossings.Source: §4.1 · pp. 7–8

4.2 The Three Pathologies of the Distortion Bound

  • Evo 2 exhibits Local-Global Decoupling: global mutual information exceeds local mutual information by only 14%, so 64× more context adds little beyond shallow composition.Its positive excess MI indicates biological signal, but the signal remains local and scale-invariant rather than encoding long-range structure.Source: Figure 4 · p. 8, §4.2 · p. 9
  • OpenFold’s Evoformer adds +2.3 to +2.5 nats of structural context over ESM-1b but incurs Procrustes disparities of 0.164, 0.162, and 0.149 at sequence lengths 100, 200, and 400.ProtMamba shows the opposite failure: negative excess MI at every length, chance-level probe accuracy (∼0.50), and perplexity 40.1 despite low geometric distortion.Source: §4.2 · pp. 8–9
  • No discrete-token model simultaneously achieves low distortion, high mutual information, and global coherence, because finite rate forces different compromises under the Gaussian rate–distortion bound.The tax appears as local geometric stability without global information, information-preserving compression with manifold warping, or smooth but informationally empty representations.Source: §4.2 · p. 9

5 Discussion

The discussion identifies discrete tokenization as the dominant source of geometric instability, with finer quantization worsening stability and no strategy escaping the observed scaling. It argues that scientific models require Physical Alignment—preserving continuous system invariants alongside predictive performance—while acknowledging important scope and empirical limitations.

Summary

  • Discrete tokenization dominates geometric instability, and finer quantization worsens stability once codebook boundaries become denser than the perturbation scale.The tax progresses with parameters, is invariant to context expansion, and follows a 1/log K distortion scaling that codebook refinement cannot overcome.Source: §5 · pp. 9–10
  • The three MINE failure regimes reflect different allocations of finite capacity: preserving local geometry sacrifices global coherence, preserving information sacrifices geometry, and preserving geometry sacrifices information.These regimes are presented as manifestations of one shared capacity constraint rather than independent pathologies.Source: §5 · p. 10

Implications

  • Current evaluation can miss severe geometric failure, motivating Physical Alignment and geometric stability auditing alongside predictive accuracy.Models may dominate benchmarks while lacking grounded global geometry, or pass geometric checks while encoding no biological signal.Source: §5 · p. 10

Limitations

  • The conclusions are scoped to AI for Science and models up to 15B parameters with 1M-context lengths, while biological continuous-objective benefits and larger-scale mitigation remain untested.The study also cautions that its MINE targets, synthetic-physics continuous-head ablations, vanilla CE setting, and one RCCR experiment limit generalization.Source: §5 · p. 10

Future directions

  • Future work should develop architectures that natively combine continuous geometric priors with high-fidelity discrete encoding and jointly optimize prediction with manifold preservation.The discussion presents geometric auditing, continuous-valued foundation models, and hybrid objectives as promising directions rather than prescribing a replacement.Source: §5 · p. 10

Code · Appendix · A Related Work

The paper provides publicly available reproducibility code and situates its contribution among methods for representation geometry, quantization, scientific foundation models, and symmetry enforcement. It distinguishes geometric fidelity from reconstruction, and frames discrete architectures as incurring an intrinsic Geometric Alignment Tax.

Code

  • All code for reproducing the experiments, benchmarks, and analyses is publicly available, with infrastructure and configuration details documented in Appendix J.Repository: https://github.com/prashantcraju/geometric-alignment-tax.Source: Code · p. 11

A Related Work

  • Unlike RSA, CKA, and linear probing, the Shesha framework tests whether perturbations preserve a source representation’s continuous geometry, supplemented by Procrustes analysis.Shesha separates local memorization from macroscopic geometric fidelity, while Procrustes distinguishes internal fracture from global manifold drift.Source: A Related Work · p. 15
  • Geometric distortion differs from reconstruction error: whereas high-rate quantization predicts K^-2/d reconstruction scaling, perturbation distortion follows a slower 1/log K pattern driven by Voronoi boundary crossings.The distinction separates encoding fidelity from representational stability under perturbation.Source: A Related Work · p. 15
  • Scientific foundation models span proteins, genomes, and chaotic time series, while long-context convolutions and continuous-time state-space models address friction between discrete tokens and continuous scientific ground truth.The paper formalizes and quantifies this architectural friction.Source: A Related Work · p. 15
  • Symmetry can be enforced architecturally through equivariance or post-hoc through consistency and smoothness regularization, but both approaches compensate for discrete models’ missing continuous structural prior.Examples include Caduceus, Frame Averaging, RCCR, Jacobian/Hessian penalties, and spectral constraints.Source: A Related Work · p. 15
  • Embedding-level RCCR achieves pointwise reverse-complement compliance but degrades population-level geometry, leaving architectural equivariance versus regularization unresolved.The experiment used DNABERT-2; the paper attributes the underlying tax to vanilla cross-entropy objectives.Source: A Related Work · p. 15

B Extraction Details and Evaluation Metrics · B.1 Geometric Stability Evaluation Metrics

The evaluation protocol standardizes embedding extraction and geometric comparison across models, then applies Shesha-based stability metrics to test whether latent manifolds preserve structure under perturbations. Metrics cover sample, feature, representational-topology, anchor, and composite stability.

B Extraction Details and Evaluation Metrics

  • All evaluated models use cosine distance for RDMs and mean-pooling over a center window unless otherwise noted.The extraction protocol applies these common settings across models.Source: Table 1 · p. 16

B.1 Geometric Stability Evaluation Metrics

  • The Shesha harness compares clean and perturbed embeddings over equivalent context windows, with optional stratification, bootstrapping, and 2,500-sample memory limits.It logs structured stability profiles for each perturbation sequence.Source: §B.1 · p. 16
  • Sample Split (SheshaSS) tests whether sample identity preserves pairwise proximity across perturbations using distance-matrix correlations.It evaluates Spearman correlations over vectorized upper-triangle distances for disjoint random sample subsets.Source: §B.1 · p. 16
  • Feature Split (SheshaFS) tests whether variance directions remain consistent across independent feature subspaces using correlations between restricted sample-distance matrices.Feature subsets are disjoint random dimension splits, with Spearman correlation computed over vectorized upper triangles.Source: §B.1 · p. 16
  • RDM Similarity (ρRDM) directly evaluates preservation of representational topology through Spearman correlations between clean and perturbed representational dissimilarity mappings.The metric evaluates off-diagonal correspondence between the two representations.Source: §B.1 · p. 16
  • Anchor Stability (Sheshaanchor) measures whether distance profiles from fixed anchor points remain consistent across random data splits.Anchor-to-subset distance matrices may be rank-normalized row-wise before Spearman correlation is computed.Source: §B.1 · pp. 16–17
  • The Composite Stability Score pools the individual stability results into a unified metric for comparing models across discrete topological spaces and target dimensions.It is intended to summarize stability results into a single comparison measure.Source: §B.1 · p. 17

C Track A + Track B Extended Methods and Details … C.3 Cross-Track Comparison

The dual-track test isolates tokenization as the key variable: architecture gaps remain small for continuous physics but expand dramatically for tokenized biological DNA. Across both tracks, the methods evaluate manifold continuity through interpolation or mutation walks using hidden-state embeddings and Lipschitz-based diagnostics.

C Track A + Track B Extended Methods and Details

  • The appendix fixes seed 320 across experiments and evaluates continuous-input control versus discrete-token genomic models with shared geometric probes.Track A provides the continuous-input control, while Track B applies the same probes to four discrete-token models.Source: C Track A + Track B Extended Methods… · p. 18

C.1 Track A: Synthetic Physics (Damped Harmonic Oscillator)

  • Track A uses 50,000 damped-oscillator trajectories for training and 2,000 held-out sequences, with 256 global discretization bins and 101-step interpolations between distinct dynamical regimes.The trajectories follow x(t) = A e−γt cos(ωt + ϕ), with phase sampled uniformly over [0, 2π].Source: §C.1 · p. 18
  • Track A trains three approximately 2M-parameter architectures identically with causal language modeling and cross-entropy on discretized inputs, isolating architecture in intermediate-representation geometry.All models use 20 epochs, batch size 64, AdamW with lr = 3 × 10^-4 and weight decay = 0.01.Source: §C.1 · p. 18, Table 2 · p. 18
  • Track A’s mean Lipschitz values are 65.3 for SmallBERT, 79.3 for SmallStripedHyena, and 84.6 for SmallMamba, a 1.3× spread with smooth PCA arcs.None of the three architectures shows staircase effects, teleportation, or fracturing; smoothness ratios are 0.425, 0.433, and 0.388, respectively.Source: §C.1 · p. 19

C.2 Track B: Biological DNA (BRCA1 Mutation Walk)

  • Track B constructs a 122-sequence BRCA1 mutation walk by introducing one pathogenic C61G substitution and 120 random SNPs into a 2 kb core region.Mutations are applied one at a time in a shared random order, with the pathogenic step retained as a biological landmark.Source: §C.2 · p. 19
  • Mean cosine Lipschitz values are 3.1×10^-3 for DNABERT-2, 2.6 × 10^-3 for Nucleotide Transformer, 1.0 × 10^-6 for Evo 2, and ∼1.0 × 10^-7 for Caduceus.The Transformers show 5 and 4 spikes, Evo 2 shows 2, and Caduceus’s 18 apparent spikes occur at the ∼10^-7 floating-point noise floor; no model marks pathogenic C61G as a spike.Source: §C.2 · p. 20

C.3 Cross-Track Comparison

  • Approximately 3,000× separates Track B’s highest and lowest mean cosine Lipschitz values, versus 1.3× across Track A architectures, implicating tokenization rather than routing or scale.Track B spans 3.1 × 10^-3 for DNABERT-2 to ∼1.0 × 10^-6 for Evo 2; comparing DNABERT-2 with Caduceus’s noise floor gives ∼26,000×.Source: §C.3 · p. 20

D Complete Ablation Battery … E.10 OpenFold

Across the ablation battery and model zoo, continuous ODE parameterization—not recurrence, regularization, filter depth, or output-head choice—best preserves geometry, while discrete tokenization produces distinct failure regimes across biological models. Scaling often worsens stability, with exceptions exposing either genuine ODE-based preservation or geometric vacuity and compression.

D.1 Variant A: Continuous MSE Head

  • 8.5×: SmallStripedHyena Lorenz Procrustes distortion falls from 0.072 to 0.0085 at 1% noise under continuous MSE, the study’s best condition.Across architectures, the distortion spread contracts from 0.072–0.157 under discrete CE to 0.0085–0.034 under continuous MSE.Source: §D.1 · p. 21

D.2 Variant B: Jacobian Norm Penalty

  • 0.467 → 0.491: SmallBERT’s Jacobian penalty improves composite stability as λ rises, while validation CE worsens from 0.81 to 1.07, yielding a stability–accuracy Pareto frontier.No λ simultaneously achieves low distortion and low CE because smoothing flattens hidden-state distinctions needed for prediction.Source: §D.2 · p. 21

D.3 Variant C: SmallMamba + Continuous Head (Positive Control); D.4 Variant D: LSTM Baseline (The Discrete Recurrence Test)

  • SmallMamba_Continuous maintains RDM similarity >0.999 across noise levels, whereas SmallBERT_Continuous and SmallStripedHyena_Continuous remain sensitive to time reversal.SmallMamba’s ODE prior therefore supplies architectural smoothness independent of the output head; LSTM instead fractures like the Transformer.Source: §D.3 · p. 22, §D.4 · p. 22

D.5 Variant E: Attention Layer Ratio Sweep (The Dilution Curve); D.6 Variant F: Hyena Filter Order Sweep (The ODE Depth Test)

  • Attention causes approximately linear geometric degradation: mean composite stability declines from ∼0.475 at 0/8 attention to ∼0.455 at 8/8, with no sharp threshold.Hyena filter order has little effect: Lorenz means range only from 0.471 to 0.477 across orders 1–8, supporting a binary attention-versus-continuous-convolution distinction.Source: §D.5 · p. 23, §D.6 · p. 23

D.7 Summary; D.1 Variant A: Continuous MSE Head; D.2 Variant B: Jacobian Norm Penalty; D.4 Variant D: LSTM Baseline (The Discrete Recurrence Test); D.6 Variant F: Hyena Filter Order Sweep (The ODE Depth Test)

  • Replacing discrete CE with continuous MSE eliminates manifold fracture, while regularization, recurrence, and filter depth cannot reproduce the continuous ODE benefit.The six ablations identify tokenization as the dominant causal bottleneck and continuous-time parameterization as the specific stability mechanism.Source: §D.1 · p. 21, §D.2 · p. 21, §D.4 · p. 22

E.1 ESM-2; E.2 Nucleotide Transformer v2; E.8 SaProt

  • ESM-2 and Nucleotide Transformer v2 stability declines with scale before apparent rebounds at 15B and 2.5B, while SaProt still degrades from 0.436 to 0.398.The ESM-2 synthetic-to-real composite-stability gap is ≤0.03, and NT scaling trends replicate on real chr22 DNA.Source: §E.1 · pp. 24–25, Table 4 · p. 24, §E.2 · p. 25

E.3 Caduceus; E.4 Evo 2

  • Caduceus is tax-exempt, preserving RC with RDM >0.999 across scales, whereas Evo 2 shows synthetic RC RDM 0.139–0.208 versus 0.873–0.889 on real DNA.Caduceus’s ODE prior anchors global topology; Evo 2’s real-DNA apparent success is a texture-dependent RC dissociation rather than robust symmetry.Source: §E.3 · pp. 27–28, Table 8 · p. 28, §E.4 · p. 29

E.5 HyenaDNA; E.6 DNABERT and DNABERT-2; E.7 GPN; E.9 ProtMamba; E.10 OpenFold

  • OpenFold’s stability rises from 0.369 at L=100 to 0.420 at L=800 but retains geometric warping of 0.15–0.16 despite high excess MI of +10.8 to +11.6 nats.ProtMamba instead has low Procrustes distortion and perplexity 40.1 but negative excess MI at every length, exemplifying Geometric Vacuity.Source: §E.9 · pp. 33–34, §E.10 · pp. 35–36, Table 23 · p. 36

E.11 Cross-Architecture Summary … F.1 Ghost Detection Protocol

Across architectures, tokenization dominates geometric stability: continuous representations vary little across model classes, whereas discrete models diverge sharply, with apparent rebounds reflecting manifold drift rather than recovery. The Ghost Detection protocol tests whether stable geometry is biologically grounded by combining self-alignment residuals with frozen-head utility under controlled context dilution.

E.11 Cross-Architecture Summary

  • A 1.3× cross-architecture spread under continuous objectives expands to three orders of magnitude in discrete Lipschitz constants, while Caduceus remains near-constant at 0.459–0.458 with perfect reverse-complement preservation.ESM-2, NT, and SaProt decline with scale; SaProt degrades like ESM-2 at matched sizes, showing that structural tokens do not escape the geometric tax.Source: §E.11 · pp. 37–38, Table 25 · p. 38
  • The 2.5B/15B composite rebound in NT-2.5B and ESM2-15B reflects global manifold drift, not genuine geometric recovery, according to Procrustes Ratio and Frozen Head tests.These models exhibit improved composites despite failing the complementary diagnostics of geometric alignment and downstream utility.Source: §E.11 · p. 38

E.12 Excluded models.

  • Evo 2 40B and AlphaGenome were excluded because their APIs exposed only final predictions, while Evo 2 40B’s partial activations were corrupted by near-zero FP8 artifacts.The analysis therefore covers Evo 2’s locally run 7B variants, with future direct weight access identified as a route for extension.Source: §E.12 · p. 38

F Ghost Detection and Phase Transition Extended Methods and Details

  • The Ghost Detection protocol tests whether apparent geometric stability is biologically grounded or instead reflects a self-consistent manifold decoupled from downstream utility through two complementary tests.The extended experiments use seed 320 and 5-fold stratified cross-validation where applicable.Source: §F.1 · p. 39, F Ghost Detection and Phase Transition… · p. 39

F.1 Ghost Detection Protocol

  • The Self-Procrustes Spin Test aligns clean and perturbed embedding matrices and diagnoses a Ghost when residuals rise with scale or remain high despite stable RDM similarity.Large embedding sets are uniformly subsampled with seed 320 to n = 5,000 for Evo 2 or n = 10,000 for ProtMamba and OpenFold.Source: §F.1 · p. 39
  • The Frozen Head Context Tax trains a frozen logistic-regression classifier while padding increases, flagging geometric instability when accuracy falls despite an unchanged local signal.Local pooling isolates signal-region geometry, whereas global pooling tests whether padding corrupts the full representation.Source: §F.1 · p. 39
  • The context-tax benchmark classifies DNA species from 1 kbp regions and protein species from 50 aa regions, using 200 sequences per class and 5-fold stratified cross-validation.DNA uses real human chr22 versus S. cerevisiae sequences; proteins use reviewed UniProt E. coli versus human proteins with random nucleotide or amino-acid padding.Source: §F.1 · p. 39
  • Embeddings are mean-pooled from model-specific hidden states, with Evo 2 capped at 120,000 tokens, ProtMamba taken from its final hidden layer, and OpenFold from its Evoformer trunk.Evo 2 uses local Vortex inference on A100 GPUs; OpenFold runs in single-sequence SoloSeq mode with a 1,022-residue maximum.Source: §F.1 · p. 40

F.2 Per-Model Ghost Detection Results … F.5 Procrustes Centering and Normalization

Across models, Procrustes and Frozen Head tests reveal architecture- and perturbation-dependent geometric regimes, while nonlinear probes distinguish inaccessible curvature from genuinely absent biological information. Centering, normalization, and optimal scaling isolate shape distortion from magnitude differences.

F.2 Per-Model Ghost Detection Results

  • Evo 2’s context scaling is geometrically neutral: SNP Procrustes ratios remain 0.963–0.964 across 8K–1M contexts, while Frozen Head accuracy changes by only ∆= +0.005.Its own next-token head likewise stays stable at 85.3%, 84.9%, and 85.2% agreement, with KL divergence of 0.0024–0.0027 nats; reverse complement falls to approximately 28–30%.Source: §F.2 · p. 40, §F.4 · p. 43, Table 33 · p. 43
  • ProtMamba remains at chance-level Frozen Head accuracy across sequence lengths, confirming Geometric Vacuity despite stable geometry and perplexity.Both local and global pooling remain near 50%, while nonlinear probes also stay at chance with an average MLP–linear gap below 0.01.Source: §F.2 · p. 40, Table 27 · p. 40, §F.3 · p. 41
  • OpenFold’s Frozen Head accuracy declines from 0.843 at L=100 to 0.773 at L=800, a 7.0 percentage-point drop despite increasingly stable orientation.Nonlinear probes provide only a modest intermediate-length advantage and also degrade at L=800, while global pooling produces statistically similar decline rates.Source: §F.2 · p. 41, Table 28 · p. 41, §F.3 · pp. 41–42

F.3 Nonlinear Probe Validation

  • Nonlinear validation separates curved encoding from vacuity: OpenFold retains biological signal under nonlinear readout, whereas ProtMamba remains uninformative under both linear and MLP probes.For OpenFold at L=100, MLP accuracy is 0.863 versus 0.843 linearly; at L=400, the gap is ∆= +0.053, but both degrade by L=800.Source: §F.3 · pp. 41–42

F.4 Phase Transition Proof: Brittle Glass vs. Untethered Gel

  • The phase-transition diagnostic uses Procrustes Ratio and Frozen Head agreement/KL: ratios near 1 indicate internal fracture, whereas ratios near 0 indicate coherent global drift.High KL despite preserved pairwise structure identifies Untethered Gel because the manifold shifts in absolute coordinates relative to the language-model head.Source: §F.4 · p. 42
  • Across architectures, the Brittle Glass–Untethered Gel transition appears in ESM-2, NT, and DNABERT; ESM-2 rises from 5.1% reduction at 8M to 26.2% at 15B, while DNABERT reaches 38–64%.Source: §F.4 · p. 43
  • Evo 2’s moderate Procrustes reduction of 22–27% is stable across context length, placing it between the more brittle and more globally drifting regimes.Its fixed-magnitude geometric tax is consistent with flat Frozen Head accuracy across context windows.Source: §F.4 · pp. 43–44

F.5 Procrustes Centering and Normalization

  • Procrustes preprocessing mean-centers embeddings, applies Frobenius normalization for rotation, and fits optimal scale so the ratio reflects shape distortion rather than overall magnitude.The optimal scale s* is reported to flag global gain or attenuation, especially under large perturbations.Source: §F.5 · p. 44

G RCCR Experiment Details

Embedding-level RCCR enforces perfect reverse-complement consistency in DNABERT-2, but does so by collapsing perturbation sensitivity rather than preserving population-level geometry. Although the composite score rises slightly, geometric-fidelity metrics degrade and Procrustes disparity increases.

G RCCR Experiment Details

  • Embedding-level RCCR fine-tunes DNABERT-2 (117M) for 10 epochs on 2,000 DNA sequences and evaluates baseline versus RCCR on 5,000 held-out sequences.The experiment uses LR 2 × 10−5, λ = 1.0, and the Shesha harness.Source: G RCCR Experiment Details · p. 45
  • RCCR reduces training loss 99.4% (1.56 × 10−3 to 9 × 10−6) and collapses the per-sequence RC cosine gap from 0.041 to 0.000.Every sequence maps to the same point as its reverse complement, producing perfect pointwise consistency.Source: G RCCR Experiment Details · p. 45, Figure 5 · p. 45
  • Procrustes disparity increases 91% (0.761 → 1.454), while RDM similarity and perturbation magnitude degrade uniformly despite a slight composite-score increase.The composite improvement is driven entirely by feature-split scores increasing from 0.66 to 0.96.Source: Figure 5 · p. 45, G RCCR Experiment Details · p. 46, Table 34 · p. 46

H MINE Extended Methods and Details … H.2 Results

The MINE appendix defines bias-controlled mutual-information estimation for biological embeddings and validates its stability across preprocessing, extraction, and convergence checks. Results compare excess MI across three biological failure regimes, with negative values indicating less ground-truth information than matched random embeddings.

H MINE Extended Methods and Details

  • MINE estimates use a Donsker–Varadhan bound, PCA-reduced embeddings, matched random baselines, repeated seeds, and excess-MI reporting to control dimensionality and finite-sample bias.Embeddings are projected to 50 principal components, and estimates average five seeded runs; negative excess MI indicates less structure than matched random noise.Source: H MINE Extended Methods and Details · p. 47

H.1 Biological Ground Truth

  • The biological ground truth comprises protein composition, length, charge, hydrophobicity, and species features, plus DNA GC and dinucleotide frequencies computed globally or within local windows.Protein experiments use 400 sequences, while DNA experiments use 200 human genomic regions of 8,192 base pairs.Source: §H.1 · p. 48

H.1.1 Model-Specific Embedding Extraction

  • Evo 2 embeddings are evaluated globally and locally, while ProtMamba, ESM-1b, and OpenFold embeddings are mean-pooled over defined signal regions for regime-specific comparisons.Evo 2 uses full-context pooling for global MI and three 128-bp windows for local MI; OpenFold also permits pre- and post-Evoformer comparison.Source: §H.1.1 · p. 48
  • A 64× context increase from 128 bp to 8,192 bp produces only a 14% excess-MI increase, consistent with Local–Global Decoupling and informational shallowness.The comparison tests whether local biological signal is washed out by full-context mean pooling.Source: §H.1.1 · p. 48

H.1.2 PCA Dimensionality Sensitivity

  • Across 30, 50, and 100 PCA components, ProtMamba stays at or below noise, ESM-1b and OpenFold remain positive, Evoformer shows ∼20% excess over ESM-1b, and Evo 2 retains its local–global gap.At 50 components, PCA retains more than 90% of variance for every tested model, while matched baselines absorb most dimensionality-dependent shifts.Source: §H.1.2 · pp. 48–49

H.1.3 Convergence Diagnostics

  • All models converge within 300 epochs, with final-tail run-to-run standard deviation below 0.15 nats; ProtMamba remains near the random baseline, while Evo 2 local traces stay ∼0.4 nats below global.These traces indicate ProtMamba’s noise-floor result is not optimization failure and that Evo 2’s local–global difference is stable across window positions.Source: §H.1.3 · p. 49

H.2 Results

  • Table 35 reports excess mutual information over matched random baselines across the three failure regimes, where negative values indicate less biological information than random embeddings.The table summarizes the regime-level MINE comparison without supplying individual cell values in the provided passage.Source: Table 35 · p. 49

I Texture Hypothesis Test (Evo 2 Reverse Complement Mechanism)

The Texture Hypothesis Test shows that Evo 2’s apparent reverse-complement robustness on real DNA is explained by per-sequence k-mer composition, not learned double-stranded symmetry. Dinucleotide shuffling preserves this behavior despite destroying positional structure, whereas population-level texture matching does not.

I Texture Hypothesis Test (Evo 2 Reverse Complement Mechanism)

  • Dinucleotide shuffling preserves each sequence’s compositional fingerprint while destroying positional structure, isolating composition from genomic architecture.It preserves exact per-sequence dinucleotide counts and lower-order frequencies while eliminating gene boundaries, repeat placement, regulatory motif locations, and long-range domains.Source: I Texture Hypothesis Test (Evo 2… · pp. 50–51
  • 97% of the real–random RC gap is recovered by dinucleotide-shuffled DNA, versus 3% by texture-matched Markov sequences, showing per-sequence k-mer composition is necessary and sufficient.RDM similarity is 0.858 vs. 0.873 for shuffled and real DNA, compared with 0.167 vs. 0.139 for Markov and random sequences; composite similarity is 0.464 vs. 0.469 and 0.292 vs. 0.287, respectively.Source: Table 36 · p. 51, I Texture Hypothesis Test (Evo 2… · p. 51
  • Across conditions, SNP sensitivity tracks k-mer disruption, while nearly identical real and shuffled response curves confirm that embedding structure is composition-driven rather than positional.Real chr22 and dinucleotide-shuffled sequences show nearly identical dose–response curves from 1% through 10% SNP perturbations.Source: I Texture Hypothesis Test (Evo 2… · p. 51
  • Markov generation matches population-level dinucleotide statistics but collapses individual sequence diversity, making RDM-based RC stability an artifact of representational collapse.Sequences drawn from one stationary transition matrix converge toward a common compositional mean, unlike heterogeneous real genomic regions with distinct k-mer fingerprints.Source: I Texture Hypothesis Test (Evo 2… · p. 51
  • The near-complete recovery from dinucleotide shuffling indicates dinucleotide composition dominates Evo 2’s RC behavior, with trinucleotide and higher-order texture contributing at most marginally.The shuffled control preserves exact per-sequence dinucleotide counts without explicitly preserving trinucleotide frequencies.Source: I Texture Hypothesis Test (Evo 2… · pp. 51–52
  • Evo 2’s apparent RC success is therefore consistent with a high-dimensional per-sequence k-mer histogram encoder, supporting Local–Global Decoupling rather than learned biophysical equivariance.Reverse complementation preserves k-mer counts, while the model captures local sequence statistics without integrating them into globally coherent biophysical geometry.Source: I Texture Hypothesis Test (Evo 2… · pp. 50, 52

J Reproducibility and Computational Infrastructure … J.4 Software Environment

The study documents reproducible statistical procedures, deterministic data handling, computational infrastructure, and software dependencies for geometric evaluations and phase-transition analyses.

J.1 Statistical Analysis and Sampling

  • Five-round stratified bootstrap resampling estimated variance for Sample Split, Feature Split, RDM Similarity, and Anchor Stability metrics.Resampling followed sequence-class proportions when labels were available and was otherwise uniform random.Source: §J.1 · p. 53
  • Pairwise RDM computations capped datasets at 2,500 samples, while Procrustes evaluations used 5,000 or 10,000 samples to manage O(n^2) memory costs.Intermediate results were deterministically cached, and NumPy generators used fixed seed 320.Source: §J.1 · p. 53

J.2 Track A + Track B

  • Track A and Track B used seed 320, deterministic BRCA1 sequence fallback data, archived embeddings and mutation metadata, and Jupyter notebooks.The BRCA1 wildtype sequence was fetched from the UCSC Genome Browser API when network access was available (Casper et al., 2025).Source: §J.2 · p. 53

J.3 Computational Infrastructure

  • Experiments ran across Google Colab Pro instances with NVIDIA A100 or T4 GPUs, while AlphaGenome scale tests used API-based gRPC inference without local GPUs.The reported hardware included A100 GPUs with 40 GB or 80 GB memory and T4 GPUs with 16 GB.Source: §J.3 · p. 53
  • Full-scale evaluations over 10,000 genomic sequences required approximately 15 minutes to 20 hours per architecture, depending on maximum context length.Source: §J.3 · p. 53

J.4 Software Environment

  • The software environment combined PyTorch 2.5–2.7, Transformers 4.x, mamba-ssm 2.3, shesha-geometry 0.1.x, SciPy, and scikit-learn.Evo 2 experiments were pinned to torch==2.7.1+cu128, and mamba-ssm used custom CUDA kernels compiled from source.Source: §J.4 · p. 53
  • Compatibility patches corrected embedding extraction issues in Nucleotide Transformer and DNABERT-2 caused by Transformers 4.x API changes.The patches modified tokenizer and hidden-state retrieval logic.Source: §J.4 · p. 53

Related papers

Loading 2604.04155v1…