PaperlayerView exact revision on arXiv

Source-linked AI summary

TESSERA v2: Scaling Pixel-wise Earth Foundation Models

Zhengpeng Feng, Sadiq Jaffer, Ira Shokar, Jovana Knezevic, James Ball, Pedro Sousa, Mark Elvers, Madeline Lisaius, Clement Atzberger, Robin Young, Aneesh Naik, Niall Robinson, David Coomes, Anil Madhavapeddy, Srinivasan Keshav

arXiv:2607.03949v2cs.CVcs.LG

TL;DR

TESSERA v2 asks how pixel-wise EO foundation models should scale and how pretraining compute should be allocated. It answers with a 395-run downstream-driven study, then uses the resulting rule to train large teachers and distil flexible students. The 44M-parameter student leads tested embedding products, while a 16-dimensional prefix retains 92% of full-dimensional performance at one-eighth the storage.

  • Problem

    How pixel-wise EO foundation models scale and how pretraining budgets should be allocated remains poorly understood, especially because pretraining loss may not reflect downstream utility.

  • Method

    The paper evaluates 395 fixed-family models on 15 downstream tasks, fits downstream-driven scaling laws, trains large teachers accordingly, and distils them into compact MATRYOSHKA students.

  • Results

    On 14 held-out datasets, the 44M-parameter TESSERA v2-2B-L student achieves the best composite score, 0.647, versus 0.614 for both OlmoEarth-L and TESSERA v1.

  • Takeaways & Limitations

    The supported recipe is to select models by downstream performance, scale encoder capacity and data together while holding the projector fixed, and distil for deployment flexibility.

  • Takeaways & Limitations

    The scaling laws apply only to pixel-wise Sentinel-1/2 encoders, one self-supervised objective, and one 15-task evaluation suite; broader climate and season generalisation remains unevaluated.

Abstract

from arXiv · show

Pixel-wise Earth-observation (EO) foundation models are now achieving state-of-the-art performance via generated spatial embeddings. However, how these models scale and how best to spend a pretraining budget remain poorly understood. We present the largest controlled scaling study for EO to date: 395 training runs within a fixed pixel-wise Barlow Twins family, each evaluated on 15 diverse downstream tasks. We find that pretraining loss barely predicts downstream performance (|Pearson r| < 0.2), so selecting models by loss wastes a large share of the compute. We also find that, as the training budget grows, the encoder and the data should grow together while the projector stays fixed, which gives a simple rule for allocating compute. Using this rule, we train a family of pixel-wise teachers (0.5B, 1B, and 2B) and distil the largest into compact students for embeddings-as-data deployment. In aggregate, our 44-million-parameter distilled student outperforms every open and proprietary embedding product we test, several of them an order of magnitude larger. These students produce Matryoshka representations that are inexpensive to serve: a 16-dimensional prefix keeps 92% of the full 128-dimensional performance at 1/8 of the storage. Together, these results give a concrete, empirically grounded recipe for scaling pixel-wise EO foundation models: train large encoders, select by downstream performance, and distil into flexible student models. We plan to release global 10 m annual embeddings covering 2017-2025 as version 2 of the TESSERA foundation-model embeddings product. All code is available at: https://github.com/ucam-eo/tessera

1 INTRODUCTION

TESSERA v2 addresses the cost and difficulty of preparing reusable EO representations by studying downstream-driven scaling and deploying distilled, storage-adaptive embeddings.

  • EO data require calibration, correction, harmonisation, and costly computation, while labels are scarce and region- or season-specific.
  • Irregular Sentinel-1/2 sampling and cloud occlusion make pixel time series difficult to represent without losing temporal dynamics.
  • An ideal embedding product should be analysis-ready, transferable, reproducible, economical, and adaptive in accuracy versus cost.
  • Across 395 runs and 15 tasks, pretraining loss was a weak selection target, while downstream-driven scaling increased encoder capacity and data together and held projector size essentially fixed.Loss-based selection required roughly 2–5× the compute to reach the same downstream score.
  • Across 29 tasks, distilled students outperform every compared open and proprietary embedding product, while largely suppressing Sentinel acquisition artefacts visible in v1.
  • TESSERA v2 trains large teachers, distils compact N/S/M/L students, and serves them as analysis-ready data through GEOTESSERA.The student family achieved the best composite score and mean rank on the 15-task ALPHAEARTH suite at deployment cost two orders of magnitude below the teacher’s.
  • A 16-dimensional MATRYOSHKA prefix retains approximately 92% of the 128-dimensional score at one-eighth the storage.

2 RELATED WORK

Prior EO work spans backbone models, precomputed embedding products, empirical scaling laws, and distillation, but existing systems generally lack flexible, storage-adaptive deployment.

  • Remote-sensing foundation models mainly pretrain spatial backbones on single-time or cloud-filtered patches, with some multi-sensor and multi-resolution variants.
  • TESSERA v1, ALPHAEARTH, ESD, and OLMOEARTH represent distinct precomputed or on-demand embedding-product approaches.Their differing sensor coverage, resolution, temporal scope, and inference modes define complementary trade-offs.
  • Existing embedding products typically impose one embedding dimension and storage budget, without coordinates that support truncation without retraining or loss.
  • TESSERA v2 adds four student sizes and MATRYOSHKA prefixes at dimensions 16, 32, 64, and 128 from one embedding without retraining.
  • EO scaling studies commonly fit power laws to pretraining loss, but that proxy can be noisy or non-monotone and existing studies confound architecture with capacity.
  • Knowledge distillation transfers representations from large teachers to compact students, while MATRYOSHKA learning creates nested multi-dimensional embeddings; their combination is rarely studied in EO.

3 DOWNSTREAM-DRIVEN SCALING LAWS

Within a fixed pixel-wise Sentinel-1/2 architecture, TESSERA evaluates 395 models across compute, encoder size, projector size, and data, selecting scaling laws by downstream performance.

  • Study design: The study fixes the architecture after sweeping ten structural axes, then varies encoder size, projector size, and training data.
  • Study design: 395 models are evaluated across 15 ALPHAEARTH tasks using a chance-adjusted composite downstream score.
  • Finding 1: Pretraining loss and downstream score are nearly independent, with Pearson r=−0.18 and Spearman ρ=−0.16 across runs.
  • Finding 1: 2–5× compute is required by loss-based selection to reach the same downstream score as downstream-driven selection.
  • Finding 2: Compute-optimal encoder vertices shift upward with budget, whereas projector vertices remain effectively fixed across encoder strata.
  • Scaling recipe: The downstream-optimal recipe is one oversized encoder trained on correspondingly more data, with the projector held near its optimum and deployment flexibility recovered through distillation.
  • Out-of-sample validation: Unseen 0.5B, 1B, and 2B teachers follow the extrapolated law and improve monotonically from 0.572 to 0.608, exceeding both reported baselines.

4 A PIXEL-WISE TEMPORAL TEACHER

TESSERA v2 uses a large temporal, multimodal pixel-wise teacher that preserves irregular annual observations during training and adapts inference to each pixel’s valid observations.

  • Compared with v1, v2 adds multi-scale temporal pretraining, adaptive full-observation inference, unified cross-modal Transformers, and scaling-law-guided MATRYOSHKA distillation.
  • Each input is a one-year Sentinel-1/2 time series for a 10 m pixel with a validity mask, preserving phenology despite clouds and irregular revisits.
  • The 2B teacher embeds each modality, adds sinusoidal day-of-year encoding, applies per-modality Transformers and attention pooling, then fuses modality tokens.
  • Training uses BARLOW TWINS on two temporally subsampled views, mix-up consistency, and random view lengths L∈{8,16}.Random sparse subsampling forces recovery of annual phenology from few observations.
  • The frozen teacher is distilled into compact students because its recurring global inference cost is unsuitable for most users.
  • At inference, each pixel’s k valid observations are packed into the smallest bucket B⋆≥k, with residual slots filled by midpoint resampling.This avoids discarding observations while retaining parallel fixed-length batches.

5 DISTILLING A DEPLOYABLE STUDENT FAMILY

TESSERA v2 distils a large teacher into compact students that support flexible, low-cost embeddings-as-data deployment. Across shared and held-out evaluations, the family delivers strong performance, graceful storage–accuracy trade-offs, and improved temporal stability.

  • Student family: The 2B teacher is distilled into four 128-dimensional students spanning provider-scale global inference to edge and on-device deployment.The L, M, S, and N students contain 44M, 21M, 7M, and 1M parameters, respectively.
  • MATRYOSHKA distillation: Distillation trains each prefix to reconstruct the frozen teacher embedding, allowing users to take any prefix directly after removing the training-only heads.The prefixes are rate–distortion codes rather than copies of the teacher’s first coordinates.
  • Nesting through distillation: Distillation is necessary for semantically ordered nested coordinates because self-supervised prefix losses are disrupted by the rotation symmetry of redundancy-reduction objectives.A fixed teacher target supplies information about coordinate ordering that prefix losses lack.
  • Benchmark results: On the 14 held-out datasets, TESSERA v2-2B-L achieves the best composite score, 0.647, versus 0.614 for OlmoEarth-L and TESSERA v1 and 0.590 for ALPHAEARTH.The 44M encoder is smaller than the compared ALPHAEARTH and OlmoEarth-L models.
  • Benchmark results: On the 15-task ALPHAEARTH suite, the composite score is 0.593 for TESSERA v2-2B-L, exceeding ALPHAEARTH at 0.560, TESSERA v1 at 0.541, and OlmoEarth-L at 0.512.Its mean rank is 2.0 at d=128, compared with 4.1 and 5.1 for the cited baselines.
  • Graceful degradation across students: A 16-dimensional prefix retains ∼92% of the d=128 composite at 1/8 of the storage, while d=32 and d=64 retain ∼97% and ∼99%.Every student lies on the score-versus-storage and score-versus-parameter Pareto frontier.
  • Held-out generalisation: Across label budgets of 1%, 30%, and 100%, TESSERA v2 beats the best non-v2 baseline on most held-out datasets, with the widest margins in the low-label regime.The held-out datasets cover vegetation and urban prediction groups.
  • Embedding quality: Compared with v1, v2 has fewer acquisition-geometry artefacts, lower consecutive-year cosine distances on stable land cover, and less persistent noise after temporary disturbances.These changes preserve geographic structure while improving perceptual quality and temporal stability.

6 DISCUSSION AND CONCLUSION

The study argues that EO scaling should be selected by downstream performance rather than pretraining loss, then uses the resulting allocation rule to build deployable embeddings. It reports broad benchmark gains and identifies empirical and geographic limits on the conclusions.

  • Discussion: Loss-based model selection in EO requires roughly 2–5× the compute to reach the same downstream score as downstream-driven selection.The study therefore recommends budgeting for downstream evaluation rather than trusting the pretraining loss.
  • Discussion: The downstream-driven allocation rule grows encoder capacity and matched training data together while keeping the projector essentially fixed.This rule is presented as the practical basis for scaling the models.
  • Limitations: The paper’s scaling laws apply only to pixel-wise Sentinel-1/2 encoders, one self-supervised objective, and one 15-task evaluation suite.The budget-allocation analysis is therefore specific to that setting.
  • Limitations: The expensive teacher requires substantial computation, and generalisation to under-represented climates and unseen seasons remains unevaluated.The benchmarks are drawn from well-studied regions.
  • Conclusion: TESSERA v2 combines a 2B pixel-wise teacher with a distilled MATRYOSHKA student family that leads both evaluation suites at two orders of magnitude lower inference cost.The conclusion frames train-large-then-distil as the paper’s central recipe.
  • Inference-time temporal windows: For OpenET, changing inference from an annual to a monthly window raises the TESSERA v2-2B-L score from 0.51 to 0.69.The result indicates that the temporal support matched to a target variable need not equal the annual product window.

F ARCHITECTURE SWEEP DETAILS

The architecture sweep varies model axes one at a time and evaluates candidates on three downstream tasks plus their arithmetic-mean composite. Most selected configurations maximize the composite, while several qualitative considerations determine exceptions.

  • Sweep design: Each architecture axis is varied with the others fixed, and candidates are evaluated on Canada crop type, ASTER GED, and LCMAP-LUC.All three metrics use a higher-is-better [0, 1] scale, with the composite equal to their arithmetic mean.
  • Selection: Seven of nine model axes select the highest-composite scheme-A candidate, while two axes are exceptions.The sweep therefore uses downstream performance alongside qualitative design considerations.
  • Pooling: GRU pooling scores 0.725 versus attention pooling’s 0.718, but attention is selected to preserve a simpler all-Transformer encoder.The decision prioritizes scalability and architectural simplicity despite the small composite difference.
  • Input modalities: Adding Landsat lowers downstream accuracy in every tested configuration, and neither interpolation choice removes the decline.Removing coarse thermal-infrared bands recovers part of the loss, but retained-Landsat variants remain below the selected alternative.
  • Cloud masking: OmniCloudMask and the default SCL mask produce almost indistinguishable downstream results despite SCL’s practical cloud-mask errors.The encoder appears to discount cloud-contaminated observations without explicit cleaner supervision.
  • Sentinel-1 fusion: Unified and separate Sentinel-1 orbit encoders score within noise, but the unified encoder is adopted because local generation visibly reduces tiling seams.The final choice reflects deployment behavior not captured by the composite.

G FULL SCALING-LAW SETUP

The scaling study evaluates 395 models across isoFLOP slices, fitting downstream-score optima rather than trusting pretraining loss. It finds that encoder capacity and data should grow together while the projector remains effectively fixed, then validates the rule with large teachers and distilled students.

  • Sweep setup: 395 models span encoder size, projector width, training data, and nine isoFLOP slices from 1.3×10^15 to 7.3×10^18 FLOPs.The grid includes encoder widths from 7–278 M and projector widths of 2048–16384.
  • Fitting: Within each isoFLOP slice, quadratic fits locate downstream-score optima, followed by log–log fits across slice vertices.Encoder and data laws use all nine slices, while the projector optimum is resolved only after downstream scores become sufficiently high.
  • Selection signal: Pretraining loss and downstream score are nearly independent across runs, with Pearson r=−0.18 and Spearman ρ=−0.16.Within fixed compute buckets, Spearman ρ has mean −0.08, median −0.13, and range [−0.25, +0.07].
  • Allocation rule: Doubling compute calls for approximately 1.28× the encoder and 1.55× the data, making the fitted frontier data-heavy.The reported relationship is D⋆∝N⋆^0.63/0.36, with exponent approximately 1.75.
  • Teacher validation: Teachers with 0.5B, 1B, and 2B encoders score 0.572, 0.596, and 0.608, while distillation costs ≤0.015 downstream score.The teachers follow the compute-optimal data prescription, and the 2B teacher uses approximately 14B d-pixels.
  • Selection cost: Loss-based selection needs roughly 2–5× the compute to reach the same downstream score as downstream-driven selection.The estimate comes from score gaps between loss- and score-selected runs and the score–compute slope bs=0.0886 per compute decade.
  • Projector role: A fixed projector is approximately width-invariant and is discarded at inference, shrinking from 25% to 8% of total parameters as the encoder grows.The teacher family therefore concentrates increasing model capacity in the encoder while retaining the projector as a training scaffold.
  • Distillation: After 0.6 B d-pixels, every prefix is within about 0.005 of its final cosine similarity and the composite reaches 0.589 of its eventual 0.593.The deployed student nevertheless uses the full 1.2 B d-pixels because the additional distillation is nearly free in wall-clock time.

M HELD-OUT GENERALISATION SUITE

The held-out suite adds 14 datasets not consulted during TESSERA v2 development, complementing the 15-task suite used during design. It tests generalisation, label efficiency, and per-task variability across vegetation and urban applications.

  • Suite construction: The held-out evaluation comprises 14 datasets spanning vegetation tasks such as tree species, crop mapping, and biomass, plus urban socioeconomic and environmental layers.These datasets were not consulted while building TESSERA v2.
  • Evaluation rationale: The held-out composite is treated as the headline because the 15-task ALPHAEARTH suite was available during architecture and checkpoint development.The authors use the ALPHAEARTH suite to describe in-distribution behavior and the held-out suite to assess generalisation.
  • Scoring: Held-out metrics are mapped to a common [0, 1] higher-is-better scale before averaging across datasets.Classification and segmentation use chance-anchored normalization, while regression uses max(0, R2).
  • Label efficiency: TESSERA v2 students lead on most held-out datasets at 1%, 30%, and 100% label budgets, with the widest margin at 1%.The comparison uses both a linear probe and a small approximately 2M-parameter CNN head.
  • Per-task variability: TESSERA v2 is best on four of six non-CITYREP held-out datasets, with its widest margins on PASTIS-R and Austrian Crop segmentation.A baseline narrowly leads on BioMassters and CropHarvest-Togo.

N ROBUSTNESS OF THE HELD-OUT RANKING

The held-out ranking is stress-tested against benchmark composition, scoring rules, and readout heads. TESSERA v2-2B-L remains first overall across these alternatives, although OlmoEarth-L leads the CITYREP subset under the original point probe.

  • Benchmark weighting: The held-out suite concentrates effective weight on CITYREP because several of its targets are spatially correlated within one benchmark family.The authors therefore remove CITYREP, alter aggregation, and replace the CNN head with the original point linear probe.
  • Subset robustness: Removing CITYREP leaves TESSERA v2-2B-L in first place, with its margin growing from +0.033 across 14 tasks to +0.052 on six vegetation tasks.On the eight CITYREP tasks alone it remains first, narrowly ahead of OlmoEarth-L.
  • Readout robustness: Under the point linear probe, OlmoEarth-L leads CITYREP 0.574 versus 0.562, but TESSERA v2 leads all 14 tasks 0.597 versus 0.573.The vegetation margin, 0.644 versus 0.573, outweighs the urban gap.
  • Scoring robustness: Alternative aggregation keeps TESSERA v2-2B-L first, scoring 0.685 with raw native metrics and 0.790 with per-task min-max normalization.The min-max scheme widens the gap because TESSERA v2-2B-L is top or near-top on many tasks.
  • Evaluation protocol: The shared evaluation reads per-pixel embeddings directly for embedding-as-data systems, while coarse-grid backbones use bilinear interpolation at the labelled point.This isolates pointwise representation quality without requiring an upsampling decoder.
  • Resolution caveat: Prithvi’s 30 m grid yields tokens summarizing roughly 480 m of ground, illustrating how coarse spatial resolution affects point-feature interpretation.Its concatenated point feature has dimension 768T for T∈{1,2,3} acquisitions.

P THE EVALUATION HEAD MATTERS: CITYREP UNDER A LINEAR PROBE

CITYREP rankings depend on the evaluation head: OlmoEarth-L narrowly leads under a linear probe, while TESSERA v2 leads when a small CNN can use spatial patches.

  • OlmoEarth-L and TESSERA v2-2B-L are the two strongest systems under the linear probe, with mean ranks of 1.50 and 1.75.
  • OlmoEarth-L leads socioeconomic and land-use targets, whereas TESSERA v2 leads road density and surface temperature and never ranks below second.
  • A small CNN head reverses the near-tie, giving TESSERA v2 the best CITYREP composite.The CNN reads each label’s 64 × 64 point-centred patch, unlike the benchmark’s linear probe on the point feature alone.
  • The reported distilled-student prefix table is a mechanism study rather than a deployed-model benchmark: its d=128 composite is 0.517 versus 0.593 for TESSERA v2-2B-L.Only the distilled-versus-naive gap is meaningful for that preliminary checkpoint.

R MATRYOSHKA FAILURE ANALYSIS

The failure analysis attributes naive MATRYOSHKA prefix behavior to unequal optimization pressure, while distillation supplies more robust nested representations. The scaling-law fits are empirically defined within a limited architectural family and evaluation suite.

  • R MATRYOSHKA FAILURE ANALYSIS: Naive MATRYOSHKA develops a per-coordinate scale staircase aligned with prefix-loss multiplicity, while distilled MATRYOSHKA remains flat.Naive early coordinates with m_j=4 are over-constrained, and the distortion weakens as multiplicity approaches one.
  • R MATRYOSHKA FAILURE ANALYSIS: The naive model’s gradient norm tracks prefix-loss multiplicity at approximately m_j-fold, whereas the distilled model stays near flat at 0.98–1.09.This supports unequal optimization pressure as the source of naive FIRST-d gains rather than learned semantic coordinate ordering.
  • R MATRYOSHKA FAILURE ANALYSIS: Student prefixes retain higher normalised effective rank and stronger alignment to the full teacher embedding than teacher coordinate-prefixes, especially at k=16.The difference vanishes at k=128, where both use the full coordinate set.
  • R MATRYOSHKA FAILURE ANALYSIS: Changing teacher target dimensionality changes the compression ratio but does not remove unequal self-supervised gradient pressure in naive prefix-BT.With d_T=512, prefix composites are 0.441/0.485/0.498/0.506, versus 0.454/0.493/0.510/0.517 with d_T=1024 for d=16/32/64/128.
  • S.1 SCOPE OF THE FINDINGS: The findings are limited to pixel-wise Sentinel-1/2 encoders pretrained with BARLOW TWINS at short sequence lengths L∈{8, 16}.The paper does not claim the same exponents for patch-level masked modeling, other modalities, different objectives, or other evaluation suites.
  • S.4 FITTED EXPONENTS: The scaling exponents are fitted from downstream-score optima within nine compute slices, with projector fitting restricted to mid-to-high compute slices.Bootstrap confidence intervals and leave-one-bucket-out stability are reported for the fitted exponents.

U DEPLOYMENT COST AND CHOOSING AN ARTEFACT

Deployment cost is dominated by encoder inference, so users can choose among student sizes and Matryoshka prefixes according to compute and storage constraints. The largest teacher is substantially more expensive than the deployable students.

  • U DEPLOYMENT COST AND CHOOSING AN ARTEFACT: Inference cost follows a near-linear power law in encoder parameter count across the student and teacher range.Figure 7 expresses the cost of one global annual 10 m Sentinel-1/2 pass in single-H100-year equivalents.
  • U DEPLOYMENT COST AND CHOOSING AN ARTEFACT: The 2 B teacher costs ≈100 GPUy for one global annual pass, versus 0.3, 0.9, and 2 GPUy for the 7 M, 21 M, and 44 M students.
  • U DEPLOYMENT COST AND CHOOSING AN ARTEFACT: The encoder forward pass accounts for ∼94% of embedding-as-data wall-clock time, making GPUy estimates a tight proxy for end-to-end cost.Data preparation contributes ∼6%, including temporal sampling, disk I/O, and normalisation/fusion/quantisation.
  • U DEPLOYMENT COST AND CHOOSING AN ARTEFACT: Recommendations range from 2B-S with 16/32 dimensions for edge deployment to the 2 B teacher for research or further-supervision use cases.Storage-constrained regional pipelines use S/M students at 32/64 dimensions, while balanced provider products use M/L at 64/128 dimensions.

V WHY NAIVE MATRYOSHKA SELF-SUPERVISION FAILS

Naive MATRYOSHKA-BARLOW TWINS imposes prefix losses whose unequal gradient multiplicities distort early coordinates without encoding useful semantic order. Distillation instead fixes the coordinate gauge against a teacher target, producing ordered compressed prefixes with stronger downstream performance.

  • Rotation symmetry: BARLOW TWINS permits orthogonal rotations, so its coordinate ordering is unspecified; prefix losses break this symmetry through gradient pressure rather than semantic information.Coordinates 1–16 receive four prefix-loss contributions, versus three, two, and one for later groups.
  • Representation pathology: Naive prefix training over-loads early coordinates, causing variance and marginal distortions that intensify with prefix-loss multiplicity.Observed standard deviations step down at d=16, 32, and 64; early naive dimensions are heavy-tailed and bimodal, unlike distilled coordinates.
  • Distillation remedy: Distillation supervises every prefix against one fixed teacher embedding, fixing the gauge, removing cross-prefix conflict, and preserving Gaussian-like coordinate regularity.Each prefix reconstructs the full target rather than merely matching the teacher’s corresponding coordinates.
  • Downstream consequences: Across 14 ALPHAEARTH tasks, distilled prefixes rank above naive prefixes at every size, while naive FIRST-D advantages reflect gradient multiplicity rather than ordered semantic content.The distilled FIRST-vs-RANDOM gap is at most 0.018, whereas naive gaps are 0.059 at d=16, 0.039 at d=32, and 0.018 at d=64.
  • Deployment implication: A 44 M distilled student scores 0.593 versus 0.527 for the same-size from-scratch encoder, a 0.066 advantage from distillation.The comparison uses identical architecture and the teacher’s data budget; the teacher itself scores 0.608.

W.2 COORDINATE-EXCHANGEABILITY TEST

The coordinate-exchangeability test shows that naive prefix losses create an apparent ordering without useful compression, whereas distillation produces genuinely informative low-dimensional prefixes. Student prefixes encode the full teacher embedding more effectively than the teacher’s own coordinate prefixes and support flexible storage–accuracy trade-offs.

  • Coordinate-exchangeability test: TESSERA v1 yields nearly identical FIRST-16, RANDOM-16, and LAST-16 performance, consistent with coordinate exchangeability.The protocol evaluates first, random, and last 16-coordinate subsets using the composite chance-adjusted downstream score.
  • Coordinate-exchangeability test: Naive MATRYOSHKA-BARLOW TWINS ranks FIRST-16 above RANDOM-16 above LAST-16, but its early-coordinate advantage reflects overconstraint rather than useful ordered compression.Its FIRST-16 score is 0.441, below the distilled RANDOM-16 score of 0.472.
  • Coordinate-exchangeability test: Distilled MATRYOSHKA achieves the strongest FIRST-16 score, 0.486, and a near-best full-dimensional score of 0.553.Its ordering signal comes from reconstructing a fixed teacher target rather than repeating self-supervised losses on early coordinates.
  • Prefix reconstruction: Student prefixes are better codes for the full teacher embedding than teacher coordinate-prefixes, with cosine similarity 0.842 versus 0.684 and R2 0.621 versus 0.358 at k=16.The gaps contract with dimensionality and vanish at k=128, where both use the full coordinate set.
  • Teacher-size ablation: Larger teachers improve fixed-size students and 16-dimensional retention, rising from 91.1% for the 500 M teacher to 92.1% for the 2 B teacher.This supports training a large teacher before flexible distillation.
  • Storage and deployment: A 16-dimensional prefix uses 1/8 of 128-dimensional storage while retaining approximately 92% of performance; d=64 nearly recovers the full composite at half the storage.With int8 quantisation, d=16 saves 32× versus fp32 128-D embeddings.
Loading 2607.03949v2…