PaperlayerView exact revision on arXiv

Source-linked AI summary

SpecMRL: Redesign Matryoshka Representation Learning within Parametric Spectrum

Zhongxin Chen, Songhao Wu, Heng Cui, Cong Li, Xiao Zhou, Rui Yan

arXiv:2610.26824v1cs.CL

TL;DR

MRL jointly optimizes nested prefixes for flexible representation sizes but does not explicitly ensure that leading dimensions provide compact encodings. SpecMRL replaces coordinate prefixes with prefixes in a parameter-derived spectral basis while otherwise retaining the MRL framework. It outperforms MRL at reduced dimensions, with stronger gains under greater compression, and reaches beyond MRL’s performance frontier using half the dimensions.

  • Problem

    MRL learns nested prefixes for multiple representation sizes but does not explicitly ensure that earlier dimensions concentrate information useful for compact readout.

    Source: §1 · p. 1, §2 · p. 3

  • Method

    SpecMRL derives an ordered spectral basis from model parameters and applies MRL prefix readout in that learned basis.

    Source: §1 · p. 1, §3.1 · p. 3

  • Results

    Up to 3.80 points under 16× compression, SpecMRL outperforms MRL at reduced dimensions and achieves its best performance with half the full dimension.

    Source: §1 · p. 2, §4.1.1 · p. 5

  • Takeaways & Limitations

    SpecMRL extends MRL’s performance frontier, with consistent reduced-dimension gains and negligible overhead across diverse experimental settings.

    Source: §6 · p. 11

    The degradation in the smallest-singular-value bands remains consistent across settings, and the authors identify disproportionate projection onto that spectral subspace as partially disrupting isotropy.

    Source: §4.3 · p. 8

Abstract

from arXiv · show

Matryoshka Representation Learning (MRL) is designed to support flexible representation sizes by jointly optimizing nested prefixes. While effective, this design lacks an explicit constraint to ensure that the leading portion provides a suitable compact encoding. To address this, we propose SpecMRL, which reorganizes the feature space to enable more effective prefix readout. Specifically, we extend MRL with a parameter-derived spectral basis organized with feature importance. SpecMRL otherwise follows standard MRL, with prefix readout performed in the learned spectral space. Experiments on MTEB show that SpecMRL outperforms MRL at all evaluated reduced dimensions, with larger gains under more aggressive compression. These results suggest that the learned spectral basis better prioritizes informative directions, enabling more effective representation compression. Notably, SpecMRL extends the performance frontier of MRL, surpassing full-dimensional MRL with only half the embedding dimensions.

1 INTRODUCTION

MRL supports multiple representation sizes through nested prefixes, but it does not explicitly order information to make leading dimensions effective compact representations. SpecMRL changes the readout basis and outperforms MRL at reduced dimensions, with larger gains under stronger compression.

1 INTRODUCTION

  • MRL learns nested prefixes in one training run, but provides no explicit constraint to order earlier dimensions for effective compact readout.The leading dimensions should ideally concentrate information useful for compact representations.Source: §1 · p. 1
  • SpecMRL derives a feature-importance-ordered spectral basis from WProj and applies standard MRL in that basis, changing only the representation basis.Figure 1 contrasts original-coordinate prefixes with spectral-basis readouts.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • A direct eigen-solver implementation poses computational and gradient-stability challenges; the proposed power-iteration alternative avoids training collapse while achieving throughput comparable to vanilla MRL.The passage identifies standard eigen-solver overhead and numerical instability as the challenges addressed.Source: §1 · p. 2
  • SpecMRL outperforms MRL at reduced dimensions, with improvements up to 3.80 points under 16× compression and full-capacity performance surpassed using half the dimensions.The introduction also reports that gains persist with larger backbones and longer training.Source: §1 · p. 2

2 BACKGROUND: MATRYOSHKA REPRESENTATION LEARNING (MRL)

MRL extends representation learning to nested dimensions, optimizing prefixes so one training run yields multiple output sizes. Its motivating feature ordering is not explicitly enforced, leaving information distribution across dimensions unspecified.

2 BACKGROUND: MATRYOSHKA REPRESENTATION LEARNING (MRL)

  • MRL applies contrastive learning at each nested dimension by retaining the first m entries of the representation, then aggregates the dimension-specific losses.The loss across dimensions is weighted.Source: §2 · p. 3
  • MRL’s nested structure motivates the Matryoshka metaphor, but the method does not explicitly ensure that more informative features occupy earlier dimensions.The resulting distribution of information across dimensions is unspecified.Source: §2 · p. 3
  • Prior work considers information across nested representations, mainly by adding cross-dimension constraints rather than directly ordering feature dimensions.The cited works are LI et al., 2025; Yang & Zhao, 2026; and Zhang et al., 2025.Source: §2 · p. 3

3 METHODOLOGY

SpecMRL replaces coordinate-prefix readout with prefixes in a parameter-derived ordered spectral basis while preserving the MRL framework. A streaming power-iteration solver updates the basis, addressing the cost and instability of direct eigendecomposition.

3.1 SPECMRL: PARAMETRIC SPECTRAL DESIGN FOR IMPROVED MRL

  • SpecMRL constructs an ordered basis from the spectrum of WProj’s Gram matrix and uses its leading m columns for nested readout.The spectral prefix replaces the original nested representation in each MRL loss term; the rest of the pipeline is unchanged.Source: §3.1 · p. 3
  • The learned parameters of WProj can be viewed as encoding embedding-space geometry, offering one possible explanation for SpecMRL’s performance gains.The authors present this as a conceptual interpretation, with further empirical observations in Appendix C.Source: §3.1 · p. 3

3.2 SPECMRL WITH EFFICIENT AND SCALABLE SPECTRAL SOLVER

  • Direct Gram-matrix eigendecomposition is computationally expensive and sensitive to ill-conditioned eigenvalue distributions.These challenges make the straightforward approach unsuitable in practice.Source: §3.2 · p. 4
  • A streaming power-iteration solver initializes from the ordered eigenvectors, then updates the basis with one matrix multiplication and QR orthogonalization.Gradients propagate through the Gram matrix and QR factorization while the previous basis is held fixed.Source: §3.2 · p. 4, Figure 2 · p. 4
  • The solver avoids training collapse across experimental settings with modest computational overhead; after training, inference requires only a D × D linear projection.The spectral basis is fixed after training and no longer needs recomputation from WProj.Source: §3.2 · p. 4

4 EXPERIMENT

Across MTEB evaluations, SpecMRL improves reduced-dimension performance over MRL, with larger gains under stronger compression, and reaches its best performance at half the full dimension. Ablations and scale tests examine the spectral ordering, its alternatives, efficiency, and robustness.

4.1.1 HOW DOES SPECMRL PERFORM RELATIVE TO TRADITIONAL MRL?

  • SpecMRL outperforms MRL across all reduced dimensions, with gains widening under stronger compression, and achieves its best performance at half the full dimension.The comparison averages performance across MTEB tasks; task-level results are reported separately.Source: Table 1 · p. 5, §4.1.1 · p. 5

4.1.2 WHAT MOTIVATES THE USE OF PARAMETRIC SPECTRUM FOR SPECMRL?

  • FeatMRL improves over MRL at some reduced dimensions but remains below SpecMRL at every evaluated dimension, while momentum smoothing does not close the gap.The tested alternatives derive the basis from input features rather than learned parameters.Source: §4.1.2 · pp. 5–6
  • Over-parameterizing WProj improves performance, with further gains from larger R diminishing; the experiments use R = 128D.The paper motivates this design as enabling finer-grained similarity modeling without directly applying MRL at dimension R.Source: §4.1.2 · p. 6

4.1.3 EFFICIENCY AND NUMERICAL STABILITY

  • The implementation avoids eigenvector-gradient dependence, mitigating sensitivity to ill-conditioned eigenvalues, while power iteration keeps training overhead affordable and near-MRL throughput.Once the basis is fixed, inference adds a D × D matrix multiplication; retrieval speedup reaches 1.86×.Source: §4.1.3 · pp. 6–7

4.2.1 SPECTRUM CONTRIBUTES BEYOND GRAM-MATRIX OVER-PARAMETERIZATION.

  • With the same learnable factor, the Gram-based variant improves only modestly over MRL and degrades under stronger compression, unlike SpecMRL’s consistent reduced-dimension gains.Learnable orthogonality also beats MRL but remains below SpecMRL, highlighting the value of ordering directions by learned importance.Source: §4.2.1 · p. 7, §4.2.2 · p. 7

4.3 VALIDATE SPECMRL AT LARGER DIMENSIONS AND TRAINING SCALES

  • On Qwen3-4B, SpecMRL’s relative gains follow trends similar to those on Qwen3-0.6B; performance generally improves as retained spectral bands increase, including at unseen dimensions.Retaining the final spectral bands unexpectedly degrades performance, consistently across settings; analysis attributes this to disproportionate projection that partially disrupts isotropy.Source: Table 6 · p. 8, §4.3 · p. 8
  • In an over-training regime, SpecMRL retains its advantage over MRL throughout training and reaches a higher performance ceiling.The authors describe larger-model and longer-horizon validation as a potential direction, while noting that larger computational budgets remain to be tested.Source: §4.3 · pp. 8–9

4.4 COMPARISON OF SPECMRL WITH EXISTING IMPROVEMENTS TO MRL

  • SpecMRL achieves the best overall performance among the compared MRL variants and preserves its low-dimensional advantage across training configurations, including sparse supervision.With sparse supervision, the gain over MRL is 7.29 points at 64 dimensions; encoding 100,000 passages takes 313.99 s versus 312.85 s for MRL, a 0.36% overhead.Source: §4.4 · p. 9, §4.5 · p. 9, §4.6 · p. 9

5 RELATED WORK

Prior work improves low-dimensional readouts through auxiliary objectives or lightweight adaptation, while SpecMRL’s related alternative learns an orthogonal basis without explicit spectral ordering.

5 RELATED WORK

  • ESE and MIPIC supervise low-dimensional prefixes using targets derived from full-dimensional embeddings, while MIC uses geometric regularization to reduce prefix–residual redundancy.ESE uses PCA-based compression targets; MIPIC aligns token-level relations across dimensions.Source: §5 · p. 10
  • Matryoshka-Adaptor and SMEC use lightweight adapters for frozen models, with SMEC adding sequential training and adaptive dimension selection.Source: §5 · p. 10
  • AdaWidth learns an orthogonal rotation of frozen embeddings without explicit spectral ordering, and is conceptually related to the learnable-basis ablation that underperforms SpecMRL.Source: §5 · p. 10

6 CONCLUSION

SpecMRL redesigns MRL by reorganizing representations with a parameter-derived spectrum. Across diverse settings, it improves reduced-dimension performance with negligible overhead and extends MRL’s performance frontier.

6 CONCLUSION

  • SpecMRL uses an ordered orthogonal learned spectrum to reorganize representations for nested truncation.Source: §6 · p. 11
  • Across diverse experimental settings, SpecMRL consistently outperforms MRL at reduced dimensions with negligible overhead and further extends its performance frontier.Source: §6 · p. 11
  • The findings identify readout space as an important learning target in nested representation.Source: §6 · p. 11

A IMPLEMENTATION DETAILS

The appendix describes the experimental setup, baseline implementations, and Gram-matrix ablation used to assess SpecMRL and related alternatives.

A.1 GENERAL SETUP

  • Experiments use Qwen3-0.6B-Base, Qwen2-0.5B, and Llama-3.2-1B backbones, with Qwen3-4B-Base added for scaling experiments.Training uses a 1,152,000-example subset by default and the full dataset for long-run experiments.Source: §A.1 · p. 15
  • Evaluation uses English MTEB’s 56 datasets across seven task categories, with a 15-dataset subset for Figures 3 and 4.SpecMRL’s basis is fixed during evaluation after one orthogonal iteration in FP32; embeddings are L2-normalized for non-classification tasks.Source: §A.1 · p. 15

A.2 IMPLEMENTATION OF ESE AND MIPIC

  • ESE applies its learn-to-compress objective to final-layer sentence embeddings, while MIPIC applies SIA and Linear CKA alignment to final-layer token representations.ESE supervises the 128-dimensional prefix; MIPIC aligns token-importance distributions and relational structures.Source: §A.2 · pp. 15–16

A.3 IMPLEMENTATION OF THE GRAM-MATRIX ABLATION

  • The Gram-matrix ablation applies a prefix of SpecMRL’s WProj directly to truncated backbone embeddings, without spectral decomposition, then computes their cosine similarity.The lifted variant preserves inner products, norms, and cosine similarities.Source: §A.3 · p. 16

B COMPARISON WITH A LEARNABLE LINEAR READOUT

The comparison tests whether a generic jointly trained linear readout can reproduce SpecMRL’s gains. It underperforms SpecMRL at every evaluated dimension, with the gap widening as dimensions decrease.

B COMPARISON WITH A LEARNABLE LINEAR READOUT

C PARAMETER SPACE REFLECTS EMBEDDING GEOMETRY

The authors treat WProj as a parameter-space proxy for embedding geometry and report stronger alignment of SpecMRL’s spectral readouts with dominant sentence-embedding subspaces than MRL’s fixed coordinates.

C PARAMETER SPACE REFLECTS EMBEDDING GEOMETRY

  • WProj’s columns are interpreted as embedding-space vectors whose organization reflects the geometry of learned embeddings.The analysis compares readout subspaces with principal directions extracted from calibration embeddings; random k-dimensional subspaces have expected overlap k/D.Source: C PARAMETER SPACE REFLECTS EMBEDDING… · p. 16
  • SpecMRL’s spectral readout subspaces align more strongly with dominant sentence-embedding subspaces than MRL’s fixed coordinate basis.The authors suggest this closer alignment may help preserve informative structure under dimensional readout and partially explain SpecMRL’s improved low-dimensional performance.Source: Table 12 · p. 17, C PARAMETER SPACE REFLECTS EMBEDDING… · p. 17
  • The power-iteration readout basis has over 99.9% subspace overlap with the ordered eigenbasis for every evaluated k.The comparison is made at the final checkpoint.Source: C PARAMETER SPACE REFLECTS EMBEDDING… · p. 17

D ENERGY CONCENTRATION IN THE TAILING SPECTRAL SUBSPACE

The analysis finds disproportionate energy in the final spectral band and discusses how this pattern departs from the expected ordering, while noting that its cause is unclear.

D ENERGY CONCENTRATION IN THE TAILING SPECTRAL SUBSPACE

  • The analysis partitions the spectrum into 20 eigenvalue-ordered bands and measures their average energy fraction over calibration embeddings.The calibration sample comprises N = 106 passages from MS MARCO; each projection is onto a spectral direction.Source: D ENERGY CONCENTRATION IN THE TAILING… · p. 17
  • The final spectral band contains 21.72% of energy despite spanning only 5% of spectral directions.The authors associate this disproportionate concentration with increased anisotropy and departure from the expected geometry, but say its cause remains unclear.Source: D ENERGY CONCENTRATION IN THE TAILING… · p. 17
  • The expected pattern is smoothly decaying energy across ordered bands, with the first band carrying the largest share.The authors relate this pattern to earlier dimensions capturing more informative semantic directions and smoothness maintaining isotropy.Source: D ENERGY CONCENTRATION IN THE TAILING… · p. 17

E COMPUTATIONAL COMPLEXITY

The computational analysis estimates SpecMRL’s added forward cost and reports that it remains below 1% of backbone forward FLOPs for Qwen3-0.6B.

E COMPUTATIONAL COMPLEXITY

  • The additional computations include Gram matrix construction, orthogonal basis updates, and projection of a batch of Bproj embeddings.The basis update uses one matrix multiplication and one QR factorization.Source: E COMPUTATIONAL COMPLEXITY · pp. 17–18
  • The backbone-cost comparison uses P for the number of linear weights and T for tokens processed per device.The authors follow the approximation in Kaplan et al. (2020).Source: E COMPUTATIONAL COMPLEXITY · p. 18
  • For Qwen3-0.6B, SpecMRL’s additional forward cost is below 1% of backbone forward FLOPs.The authors attribute the low overhead to their power-iteration design.Source: E COMPUTATIONAL COMPLEXITY · p. 18

Related papers

Loading 2610.26824v1…