PaperlayerView exact revision on arXiv

Source-linked AI summary

Your Transformer Is Secretly Elastic

Walter Simoncini, Dheeraj Varghese, Gertjan J. Burghouts, Cees G. M. Snoek

arXiv:2610.20588v1cs.LGcs.AI

TL;DR

Fixed-size transformer families do not fit fluctuating workloads well, and retraining or fine-tuning elastic models can be costly. This paper jointly optimizes individual neuron and head rankings with differentiable soft pruning across sparsity targets, without updating pre-trained weights. Across 27 vision and text encoders, it improves the performance-compute trade-off over evaluated baselines, especially under deep pruning.

  • Problem

    Transformer deployments face varying computational requirements, but model families offer few fixed sizes and many elastic approaches require training from scratch or substantial fine-tuning.

    Source: §1 · p. 2

  • Method

    The method uses differentiable soft pruning to jointly optimize individual MLP-neuron and attention-head importance scores across sparsity targets while keeping pre-trained weights frozen.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    Across 27 vision and text encoders, the method improves the performance-compute trade-off over evaluated baselines, with the largest gains under deep pruning.

    Source: §5 · p. 10

  • Takeaways & Limitations

    The learned ranking supports inference-time budget changes in one model, with measured speedups and low budget-switching overhead for fluctuating workloads.

    Source: §5 · p. 10

    Deep pruning remains challenging for the evaluated decoder-based text embedders; diffusion-based image generation and autoregressive language modelling remain unevaluated.

    Source: §5 · p. 10

Abstract

from arXiv · show

Scaling up transformers has produced models with broad capabilities, deployed across heterogeneous hardware and fluctuating workloads. Yet, model families offer only a few fixed sizes (e.g., S, B, L), forcing users to commit to a model based on hardware and latency constraints. This rigidity is often suboptimal under varying query loads. We introduce a retraining-free method that elastifies pre-trained transformers into a single model that adapts on-the-fly across a continuum of computational budgets. Our key contribution is a differentiable relaxation of the functional unit-ranking problem: unlike prior work, which optimizes a single importance weight per MLP block or attention head, we rank every individual MLP neuron and attention head jointly, on a common scale, by minimizing the model's loss under soft pruning across many sparsity targets, all at once. Across 27 vision and text transformer encoders ranging from 22M to 8B parameters, our method yields smooth degradation up to 60% sparsity and outperforms one-shot baselines under deep pruning. For example, pruning DINOv3 ViT-H+/16 to 50% sparsity removes 427M parameters while reducing ImageNet-1k linear-probing accuracy by only 0.8%, without the need for weight correction or retraining. At 60% sparsity, a pruned AugReg ViT-B/16 model achieves up to $1.80\times$ higher throughput and $1.26\times$ lower single-query latency on an A100 GPU. Elastic models are produced in minutes, with construction time scaling sub-linearly with parameter count.

1 Introduction

Fixed transformer sizes fit fluctuating workloads poorly, while producing separate pruned models for each budget adds storage and switching costs. This work makes pre-trained transformers elastic by jointly optimizing individual neuron and head rankings, improving the performance-compute trade-off especially under deep pruning.

1 Introduction

  • A single elastic model can adapt its computational budget at inference time, unlike fixed-size model families or separate fixed-budget pruned models.Serving k pruned models requires producing, storing, and switching between them as demand changes; retrieval may also require separate indices.Source: §1 · p. 2
  • The method jointly optimizes an individual importance score for every MLP neuron and attention head, allowing units to change order within MLP blocks.Prior elastic ranking optimized one scaling factor per MLP block and head, preserving the initial within-block neuron ordering.Source: §1 · p. 2
  • Soft masks make loss differentiable with respect to unit scores, enabling joint ranking optimization across multiple sparsity targets while pre-trained weights remain frozen.The resulting ranking supports hard pruning at intermediate budgets without further optimization.Source: §1 · p. 2
  • Across 27 vision and text encoders, the method improves the performance-compute trade-off over evaluated baselines, with the largest gains under deep pruning.Evaluations span image classification, semantic segmentation, text embedding, and retrieval.Source: §1 · p. 2
  • At 50% sparsity, pruning DINOv3 ViT-H+/16 removes 427M parameters with a 0.8% drop in ImageNet-1k linear probing accuracy, without weight correction or retraining.At 60% sparsity, AugReg ViT-B/16 reaches up to 1.80× dense throughput on an A100.Source: §1 · p. 2

2 Related Works

Related work covers pruning methods that remove weights or structural units, elastic inference approaches, and differentiable pruning. This paper adopts structured pruning and builds on retraining-free elastic ranking while addressing the fixed within-block neuron ordering of prior work.

2 Related Works

  • Structured pruning removes functional units such as MLP neurons or attention heads and is adopted to enable inference acceleration without specialized hardware.Pruning also includes unstructured removal of individual weights and semi-structured regular sparsity patterns.Source: §2 · p. 2
  • Unit importance criteria include weight magnitude, second-order saliency, Wanda, and magnitude-augmented SNIP; the method initializes scores with a Hessian-diagonal approximation of second-order saliency.Wanda scales weight magnitudes by input activation norms, while magnitude-augmented SNIP combines gradient-based loss sensitivity with a magnitude term.Source: §2 · p. 3
  • Token pruning and merging reduce sequence length, whereas this method shrinks the network while retaining all tokens, making the approaches complementary.Keeping all tokens preserves dense patch-level features.Source: §2 · p. 3
  • Differentiable pruning replaces hard keep/remove decisions with continuous masks, but prior approaches generally produce one sparse network per target and typically train weights with masks.This method instead learns one ranking across targets while keeping pre-trained weights frozen.Source: §2 · p. 3
  • Transformer elastic-inference approaches include early exit, variable depth, and nested widths or dimensions, and typically require training from scratch or substantial fine-tuning.Simoncini et al. (2025) made pre-trained ViTs elastic without retraining, but kept within-block neuron order fixed; this work uses gradient-based optimization to optimize per-neuron scores.Source: §2 · p. 3

3 Method

The method initializes importance scores for MLP neurons and attention heads, then jointly optimizes their ranking with differentiable soft pruning across sparsity targets. The resulting single ranking supports inference-time pruning without retraining or updating pre-trained weights.

3 Method

  • A ranking of MLP neurons and attention heads lets one pre-trained transformer be pruned to different computational budgets at test time without retraining.Removing units from the bottom of the ranking realizes a target sparsity.Source: §3 · p. 3

3.1 Preliminary: Importance Score Estimation

  • Initial unit scores use an empirical-Fisher estimate of the diagonal Hessian, derived from calibration samples and a modality-specific loss.Vision uses the DINO objective and text uses symmetrized in-batch InfoNCE; per-weight scores are averaged over each neuron's projections or a head's value-projection slice. The resulting scores may miss unit interactions and lack comparability across blocks and unit types.Source: §3.1 · p. 4
  • Unlike prior block-and-head scaling that preserves within-block neuron order, the method jointly optimizes all unit scores on a common scale across sparsity targets.This gives the ranking freedom to reorder individual neurons and heads in response to pruning interactions.Source: §3.1 · p. 4

3.2 Differentiable soft pruning

  • Differentiable soft pruning replaces hard ranking truncation with soft masks, allowing gradient descent to optimize scores while keeping model weights frozen.The objective minimizes masked-model loss over a set of sparsity targets, using the native loss where applicable or cosine distance between pruned and dense embeddings otherwise.Source: §3.2 · pp. 4–5
  • Median-centering log-scores for neurons and heads aligns their medians at zero and produces one combined ranking without changing within-type rankings.The median is used because it is less sensitive to outliers than the mean.Source: §3.2 · p. 4
  • For each sparsity target, sigmoid masks relax the hard keep/remove cutoff; the cutoff is recomputed as scores and the ranking change.Scores above the threshold receive values closer to one and those below it values closer to zero; temperature controls transition sharpness.Source: §3.2 · pp. 4–5
  • The optimization averages loss over 14 targets from 5% to 70% sparsity, giving units near different cutoffs gradient signal across budgets.The final scores define the ranking for inference-time pruning, while optimization decides where sparsity goes across blocks and unit types.Source: §3.2 · p. 5

4 Experiments

The experiments compare the method with pruning baselines across vision and text tasks, model sizes, fixed-index deployment, and design ablations. The method generally retains performance better as pruning deepens, while model family and initialization affect the results.

4 Experiments

  • Experiments span ImageNet-1k k-NN, Pascal VOC Hummingbird mIoU, and 41-task MTEB, evaluating pruning methods without weight correction.Unless otherwise specified, the evaluation uses six sparsity targets from 10% to 60%.Source: §4 · p. 5, §4.1 · p. 5

4.1 Comparison with Pruning Methods

  • Across classification, segmentation, and text embedding, our method matches baselines under shallow pruning and leads under deeper pruning, with the largest gains at moderate-to-deep sparsity.The comparison covers ImageNet-1k k-NN accuracy, Pascal VOC Hummingbird mIoU, and mean MTEB score.Source: Figure 2 · p. 6, §4.1 · p. 6

4.2 Effect of Model Size

  • Larger vision models tolerate pruning better, with the advantage widening for DINOv3 but narrowing on DeiT-III; text robustness varies more by encoder family than size.At 50% sparsity, DINOv3 ViT-H+/16 loses 1.8%, compared with 26.1% for ViT-B/16; Qwen3 losses show no consistent size trend.Source: Figure 3 · p. 7, §4.2 · p. 7

4.3 Elastic Deployment

  • With a fixed dense index, our method preserves image-query accuracy and text retrieval quality deeper than baselines, although some backbones still incur large losses.For text retrieval, it outperforms both baselines on every tested backbone at 50% and 60% sparsity.Source: Figure 4 · p. 8, §4.3 · p. 8
  • At 60% sparsity, AugReg ViT-B/16 reaches up to 1.80× the dense model’s throughput and a 1.26× speedup in single-query latency on an A100.Switching between budgets costs a mean of 0.46 ms, excluding initial compilation.Source: §4.3 · p. 8

4.4 Ablations

  • 59.7% ImageNet-1k top-1 k-NN accuracy at 50% sparsity with per-unit scores, versus 29.4% with sublayer-level scores under equal compute.Accuracy increases monotonically with optimization granularity; the SnapViT parametrization reaches 34.6% under the same optimization protocol.Source: Table 1 · p. 9, §4.4 · p. 9
  • At 20% sparsity, optimization lifts Wanda from 7.2% to 77.8% accuracy and random initialization from 58.2% to 76.8%.Initialization matters more at 60% sparsity, where DINO leads Wanda by 12.6%.Source: §4.4 · p. 9, Table 2
  • 0.4% to 3.6% gains over single-target optimization, from 20% to 60% sparsity, when optimizing jointly across 14 targets.Joint optimization outperforms separate rankings at all three evaluated sparsities.Source: §4.4 · p. 9, Table 3

5 Conclusion

The method makes pre-trained transformer encoders elastic through differentiable soft pruning, jointly optimizing neuron and head scores across sparsity targets without weight updates. It improves the performance-compute trade-off across 27 vision and text encoders, though deep pruning remains challenging for evaluated decoder-based text embedders.

5 Conclusion

  • Across 27 vision and text encoders, the method improves the performance-compute trade-off over evaluated baselines, with the largest gains under deep pruning.It jointly optimizes individual neuron and attention-head scores across sparsity targets without weight updates; the resulting ranking supports pruning to the deepest optimization target without further optimization.Source: §5 · p. 10
  • Deep pruning remains challenging for the evaluated decoder-based text embedders.The authors leave diffusion-based image generation, autoregressive language modelling, and elastic post-training using learned rankings for future work.Source: §5 · p. 10

A.1 Datasets

The appendix evaluates image classification, semantic segmentation, text embedding, and retrieval using specified datasets and metrics. Table 4 summarizes the image-classification datasets and their characteristics.

Image classification

  • Image classification uses ImageNet-1k and nine transfer datasets, reporting test-split top-1 accuracy except for ImageNet-1k, which uses validation accuracy.The transfer datasets cover natural images, textures, and remote sensing, and span coarse- and fine-grained recognition; Table 4 summarizes class counts, split sizes, and native image resolutions.Source: §A.1 · p. 17, Table 4 · p. 17

Semantic segmentation

  • Semantic segmentation is evaluated on the Pascal VOC 2012 validation split using mIoU.Source: §A.1 · p. 17

Text embedding and retrieval

  • Text embedding uses MTEB(eng, v2), covering 41 English-language tasks, and reports the mean score across tasks.Source: §A.1 · p. 17
  • Elastic retrieval with asymmetric document and query encoders is assessed by mean nDCG@10 across the 15 BEIR datasets.Source: §A.1 · p. 17

A.2.1 How does ranking optimization change allocations?

At 50% sparsity, ranking optimization changes both where retained capacity is allocated and which units are selected within blocks. The cross-block changes differ by backbone, while within-block reordering occurs across all three examined models.

A.2.1 How does ranking optimization change allocations?

  • Across AugReg, DINOv3, and SigLIPv2 ViT-B/16 at 50% sparsity, optimized rankings select different units within blocks, even when the number retained changes little.The within-block reordering is most visible for MLP neurons.Source: §A.2.1 · p. 17
  • Cross-block allocation changes are backbone-specific: DINOv3 restores capacity to its final MLP block, while SigLIPv2 preserves aggressive pruning of later MLP blocks.AugReg’s allocation changes only modestly.Source: §A.2.1 · p. 17

A.2.2 Additional Backbones and Model Scaling

Additional backbone evaluations show that the method retains advantages under deep pruning across DeiT and large vision transformers, while larger AugReg models tolerate more pruning. Linear-probing results include small losses for DINOv3 at 50% sparsity, and the appendix provides further model-scaling curves.

Additional backbones and model scaling

  • At 50% sparsity, ranking optimization changes both cross-block allocations and within-block unit selection for AugReg, DINOv3, and SigLIPv2 ViT-B/16.Newly retained units appear even in blocks where allocations remain nearly unchanged.Source: Figure 5 · p. 18

Original DeiT

  • On DeiT-B/16, the method retains higher k-NN and linear-probing accuracy than baselines under deep pruning, although several baselines are competitive under shallow pruning.Figure 6 shows the comparison on ImageNet-1k and the transfer average.Source: §A.2.2 · pp. 17–18, Figure 6 · p. 19

Linear probing

  • At 50% sparsity, DINOv3 ViT-H+/16 loses only 0.8% ImageNet-1k accuracy and 3.3% transfer-average accuracy while removing 427M parameters.It retains 84.4% ImageNet-1k accuracy and 85.4% on the transfer average.Source: §A.2.2 · p. 18
  • On AugReg ViT-B/16, linear probes retain more accuracy under deep pruning than the corresponding k-NN evaluations.Source: Figure 7 · p. 19
  • For DINOv3 ViT-H+/16, ImageNet-1k linear accuracy is essentially flat through 50% sparsity; the figure also compares large SigLIPv2 ViT-G/16 models.Source: Figure 8 · p. 20

Model scaling

  • Larger AugReg models tolerate more pruning, and the appendix supplies performance-versus-compute curves for additional model families.The listed curves cover AugReg, DeiT-III, DINOv3, and DINO WebSSL, with comparisons of SigLIPv2 ViT-B/16 and ViT-G/16.Source: §A.2.2 · p. 18, Figure 9 · p. 21

A.2.3 Transfer Datasets

Transfer-task performance varies by model and task granularity: pruning is especially damaging on fine-grained tasks, while the method matches or exceeds baselines across many moderate and deep pruning settings.

A.2.3 Transfer Datasets

  • For AugReg ViT-B/16, the method matches or outperforms baselines on every listed transfer task except EuroSAT, while pruning impacts vary by task.Pets performance is approximately constant except for a dip at 60% sparsity; Aircraft is approximately constant, likely because dense-model performance is poor. The authors note that Pets stability may reflect ImageNet calibration-data alignment.Source: §A.2.3 · p. 21
  • From 40% sparsity, the method matches or exceeds every baseline on every dataset for DINOv3 ViT-H+/16; for SigLIPv2 ViT-G/16, this holds from 30%.At 50% sparsity, DINOv3 stays within 2.0% of dense accuracy on ImageNet-1k, CIFAR-10, EuroSAT, and Pets, while Flowers 102 improves by 7.7%.Source: §A.2.3 · p. 21
  • At 60% sparsity, SigLIPv2 loses 17.0% on ImageNet-1k and over 30% on Aircraft, Flowers 102, and Food 101, but at most 9.1% on EuroSAT, RESISC45, and Pets.DINOv3's fine-grained tasks degrade most at 60% sparsity: CIFAR-100 loses 16.4%, compared with 5.6% on ImageNet-1k.Source: §A.2.3 · p. 21
  • At 60% sparsity, ViT-H/14 removes approximately 378M parameters, losing 10.4% on ImageNet-1k and 2.73% on the transfer average.The figure reports top-1 k-NN accuracy for DeiT-III models on ImageNet-1k and transfer datasets.Source: Figure 10 · p. 22

A.2.4 In-Context Semantic Segmentation

On Pascal VOC 2012 semantic segmentation, elastically pruned AugReg and DINOv3 preserve patch-level representations more effectively than the evaluated baselines under deep pruning.

A.2.4 In-Context Semantic Segmentation

  • At 50% sparsity, AugReg outperforms OPTIN, the runner-up, by 25.1% mIoU.The comparison is on Pascal VOC 2012.Source: §A.2.4 · p. 22
  • At 40% sparsity and 141.5 GFLOPs, DINOv3 retains 70.3% mIoU, exceeding runner-up VBP by 39.8% while using 3.1 fewer GFLOPs.The results are on Pascal VOC 2012.Source: §A.2.4 · p. 22
  • Across both backbones, the method preserves patch-level representations more effectively than the evaluated baselines under deep pruning.Source: §A.2.4 · p. 22

A.2.5 Text Representations

Elastic pruning extends to text encoders, with relatively small losses under shallow pruning and stronger retrieval performance than baselines at higher sparsities. Performance and pruning tolerance vary by encoder family and task.

A.2.5 Text Representations

  • At 60% sparsity, BGE-L loses 19.6 retrieval nDCG@10 points versus 5.3 on non-retrieval; the method leads nDCG@10 on every encoder at 50% and 60%.SnapViT remains competitive under shallow pruning, and at 40% sparsity the method leads on all but E5-Mistral 7B.Source: §A.2.5 · p. 22
  • At 20% sparsity, no model loses more than 3.3 MTEB points, and E5 models lose under 1.0; at 60%, BGE and E5 lose 5.1–10.0 points versus 11.5–19.5 for E5-Mistral and Qwen3.Table 5 covers nine text encoders across the BGE, E5, E5-Mistral, and Qwen3 families.Source: §A.2.5 · p. 22, Table 5 · p. 23
  • At 60% sparsity, DINOv3 ViT-H+/16 removes 512M parameters, with k-NN accuracy reductions of 5.6% on ImageNet-1k and 6.4% on the transfer average.The figure reports smoother degradation from the method on both larger DINOv3 models.Source: Figure 11 · p. 24
  • At 60% sparsity, Qwen3 4B loses more than 0.6B on retrieval and non-retrieval tasks, with drops of 26.2 versus 22.2 and 17.3 versus 16.7 points, respectively.Larger text encoders do not consistently retain more of their dense performance; E5-L loses less than E5-B on retrieval but has a similar non-retrieval drop.Source: §A.2.5 · p. 24
  • Elastic pruning extends the compute-performance frontier: Qwen3 8B and 4B remain on it up to 10% and 20% sparsity, while BGE-S reaches below 5 GFLOPs without collapse.Dense scaling benefits vary across families, and E5-L is always Pareto-dominated by dense E5-B.Source: §A.2.5 · p. 24
  • At 4.3B parameters, the pruned E5-Mistral 7B model reaches 49.9 mean nDCG@10 versus 24.7 for pruning-only baselines; against fine-tuned models, it leads down to 5.7B parameters.At 4.3B parameters, it trails the fine-tuned comparison by 5.4 points.Source: §A.2.5 · p. 24
  • At 50% sparsity, SigLIPv2 ViT-G/16 removes 573M parameters, reducing k-NN accuracy by 10.5 percentage points on ImageNet-1k and 12.9 on the transfer average.The figure reports smoother degradation from the method on both SigLIPv2 models.Source: Figure 12 · p. 25

A.2.6 Elastic Text Retrieval

This section reports supplementary evaluations of elastic pruning for text encoders, including text retrieval performance across sparsity targets and comparisons with corrected baselines. It also covers calibration diversity, ranking-optimization settings, speed, construction time, and the procedures used to score, prune, and measure models.

A.2.6 Elastic Text Retrieval

  • At 50% and 60% sparsity, our method outperforms both baselines on every text backbone in elastic retrieval, with the document index encoded once by the dense model.It also leads both baselines at every evaluated sparsity on BGE-S and E5-B, and from 40% onward on the other backbones except E5-Mistral 7B, where SnapViT leads by up to 1.8 points at up to 40% sparsity.Source: §A.2.6 · p. 25

A.2.7 Weight Correction

  • At 40%–60% sparsity, our uncorrected method outperforms corrected SlimGPT on seven of nine text encoders; corrected SlimGPT leads on E5-Mistral 7B and Qwen3 4B by at most 1.5 points.On AugReg ViT-B/16, correction improves our method by up to 8.8% at 60% sparsity, surpassing corrected SnapViT by 6.5%; on DINOv3 ViT-H+/16, the uncorrected method leads corrected SnapViT by 18.2% at 60%. Correction requires separate weights per sparsity target.Source: §A.2.7 · pp. 25–26

A.3.1 Backbones

  • The evaluation covers 18 vision encoders from 22M to 7B parameters and nine text encoders from 33M to 8B parameters, including bidirectional and decoder-based embedders.For text initial scores, the method uses symmetrized in-batch InfoNCE with model-specific pooling and 1000 MS MARCO query-passage pairs truncated to 256 tokens.Source: §A.3.1 · p. 26, §A.3.2 · p. 27

A.3.3 Differentiable Soft Pruning

  • Differentiable soft pruning optimizes unit log-scores over 14 sparsity targets from 5% to 70%, averaging per-target losses; text uses learning rate 0.2 and sigmoid temperature 0.1.Calibration uses up to 1000 MS MARCO passages for text, and objectives use native losses where possible or cosine distance between pruned and dense features for text and selected vision encoders.Source: §A.3.3 · p. 28

A.3.4 Theoretical FLOPs of functional units

  • Functional-unit FLOP accounting includes prunable MLP projections and attention projections and operations; for text, sequence length is set to 256, and grouped-query attention uses an amortized per-query-head ranking cost.For text initial importance scores, symmetrized in-batch InfoNCE uses each model’s pooled embeddings and 1000 MS MARCO pairs; dense retrieval evaluation encodes the document index once with the dense model and queries with pruned counterparts.Source: §A.2.6 · p. 25, §A.3.2 · p. 27, §A.3.4 · p. 29

A.4 Does pruning translate to speedups?

  • At 60% sparsity, compiled models reach up to 1.80× dense throughput; with reduce-overhead compilation, batch-1 latency improves 1.26× for A = 64 and 1.09× for unconstrained pruning.In eager execution, unconstrained pruning at 20% sparsity reaches 0.70× dense throughput, while block pruning is consistently faster; tiling can trade accuracy, with blocks of 8 and 64 trailing unconstrained pruning by 3.4 and 3.2 points on ImageNet-1k at 60% sparsity.Source: §A.4 · p. 31

A.5 Runtimes

  • Producing all six evaluated sparsity targets takes 8.7 minutes for ViT-S/16 and 40.7 minutes for ViT-H/14, a 4.7× increase for a 29× larger model.For ViT-H/14, OPTIN and VBP require 34.6 and 53.7 hours respectively, while SparseGPT, SnapViT, FPTP, and our method operate in the same minutes regime.Source: §A.5 · p. 31

A.7.1 Task-aligned Calibration

  • Task-aligned calibration improves results in 25 of 27 settings, with gains of 26.6% on Flowers 102 at 40% sparsity and 20.8% on Pets at 60%.Using native ranking losses also improves ImageNet-1k accuracy at every evaluated sparsity: gains rise from 0.9% to 8.9% for AugReg and from 1.4% to 10.1% for SigLIPv2 between 20% and 60% sparsity; transfer results generally improve, with some dataset-specific regressions.Source: §A.7.1 · p. 33, §A.7.2 · p. 33

A.7.4 Unified importance ranking

  • Unified ranking improves ImageNet-1k accuracy by up to 8.7% over cross-block thresholding on DINOv3 at 50% sparsity, but trails it by up to 2.5% on SigLIPv2 transfer average at 40%.Cross-block thresholding improves on per-block thresholding by up to 34.9% on DINOv3 ImageNet-1k at 40% sparsity; unified ranking avoids a hand-crafted neuron–head allocation schedule.Source: §A.7.4 · p. 34, Table 23 · p. 46, Table 24 · p. 47

A.7.5 Elastic interpolation

  • Across 11 evaluated sparsity targets, accuracy decreases monotonically, and every interpolated target scores within 0.1 points of or above the linear interpolation between neighboring anchors.The ranking was optimized at seven anchors from 10% to 70% and evaluated every 5% from 10% to 60%.Source: §A.7.5 · p. 34, Table 25 · p. 47

A.7.6 Ranking optimization temperature

  • Mean accuracy peaks at sigmoid temperature τ = 0.1, decreases steadily above 0.2, and approaches the unoptimized baseline at τ = 5.0.The results indicate a tradeoff: the sigmoid must approximate hard pruning while remaining smooth enough to propagate gradients to units near the threshold.Source: §A.7.6 · p. 34, Table 26 · p. 48

A.7.7 Run-to-Run Variability

  • Across three seeds, standard deviation is at most 0.65% on ImageNet-1k at 60% sparsity and 1.81% on transfer average at 40%; variability is otherwise below 0.56%.Transfer accuracy shows greater variability at 40% sparsity.Source: §A.7.7 · pp. 34–35, Table 27 · p. 48
  • At 60% sparsity, retrieval loses 19.6 points for BGE-L and 22.1 for Qwen3 8B, while non-retrieval tasks lose 5.3 points for BGE-L and 4.3 for E5-L.At 20% sparsity, E5-B slightly exceeds its dense retrieval score.Source: Table 8 · p. 38, Table 9 · p. 39
  • At 50% and 60% sparsity, the method outperforms both baselines on all nine text encoders; at 4.3B parameters, it scores 49.9 versus 24.7 for dropping 16 MLP sublayers.For E5-Mistral 7B, the method’s mean score is higher than the fine-tuned baseline down to 5.7B parameters; at 4.3B, it scores 49.9 versus 55.3 without weight updates.Source: Table 6 · p. 36, Table 7 · p. 37, Table 10 · p. 41

A.7.3 Calibration Set Size

  • With fixed optimization compute, 9600 unique calibration samples improve accuracy by up to 14.3 points over 64 samples at 50% sparsity, but trail the 2000-sample default by 2.0 points at 60%.Beyond the default 2000 samples, expanding calibration data does not consistently improve performance.Source: §A.7.3 · p. 33, Table 21 · p. 46

Related papers

Loading 2610.20588v1…