PaperlayerView exact revision on arXiv

Source-linked AI summary

FastE: Readout-Triggered Token Compression for LLM Embedding Inference

Jinsong Shu, Jinyong Wen, Baokun Wang, Zhongle Xie, Lidan Shou, Weiqiang Wang, Gang Chen

arXiv:2609.08407v1cs.AI

TL;DR

Long serialized prefixes make final-readout LLM embedding inference costly, motivating a method that exploits the greater compressibility of prefix states at deeper layers. FastE uses alignment-triggered, readout-guided prefix compression without retraining, retaining embedding quality while reducing computation across text and cross-modal evaluations.

  • Problem

    Long serialized prefixes make final-readout LLM embedding inference computationally expensive, while fixed-depth or modality-specific reduction methods provide limited flexibility.

  • Method

    FastE is a training-free method that triggers compression using a shared threshold on batch-mean readout–prefix alignment and retains states ranked by readout attention.

  • Results

    Across text and cross-modal evaluations, FastE supports quality–efficiency trade-offs; on NarrativeQA, it retains 99.53% of Full Forward nDCG@10 with 40.11% decoder-backbone FLOPs reduction.

  • Takeaways & Limitations

    FastE provides a plug-and-play route to reducing final-readout embedding inference cost without retraining or custom sparse-attention kernels.

  • Takeaways & Limitations

    FastE targets final-readout models, while final-layer pooling backbones require a pooling-aware compression signal and broader video evaluation remains future work.

Abstract

from arXiv · show

In this study, we identify depth-dependent prefix redundancy in final-readout LLM embedding models, notably across representative backbones including Qwen3-Embedding and Qwen3-VL-Embedding. We find that removing prefix states is substantially more damaging in shallow layers than at greater depth, showing that prefix states become increasingly compressible as the prefix and readout states propagate through the network. To this end, we introduce FastE, a training-free, plug-and-play method. FastE uses a shared fixed threshold on batch-mean readout-prefix alignment as a lightweight online heuristic for selecting when compression occurs, and ranks prefix states by the attention scores they receive from the readout position to determine which states are retained in subsequent layers. Our evaluations demonstrate FastE's ability to substantially reduce computational costs: on NarrativeQA with Qwen3-Embedding-0.6B, it reduces decoder-backbone FLOPs by 40.11% while retaining 99.53% of Full Forward nDCG@10. Across five text embedding benchmarks, two backbone scales, and three cross-modal retrieval tasks, the quality-efficiency trade-off is directly customizable through the maximum removal ratio without retraining. We believe FastE offers practical value for scalable embedding generation in retrieval, indexing, clustering, and multimodal representation systems.

Introduction

Long serialized prefixes make final-readout embedding inference expensive, but prefix states become more compressible at greater depth. FastE uses this depth effect to trigger and perform training-free prefix compression while preserving embedding quality.

  • Motivation: 450 million inputs with mean valid serialized-prefix length 732 tokens require approximately 6 hours on 300 L20 GPUs.This motivates reducing computation by filtering prefix states during later transformer blocks.
  • Depth-dependent redundancy: At 50% prefix-state removal, performance drop decreases from 35.4–94.9% at layer 4 to 0–5.5% at layer 24 across three evaluated settings.The depth at which removal becomes tolerable varies across task–backbone settings.
  • Depth-dependent redundancy: Readout–prefix alignment generally increases with depth, providing a lightweight heuristic for identifying when compression is better tolerated.The paper does not treat alignment as causal proof of redundancy.
  • FastE: FastE separates compression into deciding when to begin and which prefix states to retain.It is designed as a training-free method for final-readout embedding inference.
  • FastE: FastE uses a shared threshold on batch-mean readout–prefix alignment and readout-guided attention ranking to select and retain prefix states.The method performs one-shot compression while preserving the readout position, causal order, and original position identifiers.
  • Evaluation: On NarrativeQA with Qwen3-Embedding-0.6B, FastE retains 99.53% of Full Forward nDCG@10 while reducing decoder-backbone FLOPs by 40.11%.The reported result also includes a 1.363× measured end-to-end speedup.

Related Work

Prior work reduces sequence computation through pruning, merging, or cache compression, but FastE addresses final-readout embedding inference with a different operating regime.

  • Final-readout embedding models: Final-readout embedding models derive sequence embeddings from the final-layer state of an EOS or dedicated readout token.The readout token's causal attention covers the serialized prefix.
  • Sequence reduction during representation construction: FastE accelerates one-pass sequence-embedding construction by filtering prefix states, reducing attention and MLP computation in subsequent transformer blocks.This differs from autoregressive KV-cache compression, which selects past states for future autoregressive queries.

Method

FastE compresses serialized prefixes online after depth-dependent redundancy emerges, using readout–prefix alignment to trigger one-shot filtering and readout attention to select retained states. The shortened sequence preserves causal order and the readout position while reducing later transformer computation.

  • Sequence representation: FastE operates on unified hidden states after multimodal and textual inputs are serialized into a common sequence.The model preserves the readout token while considering serialized-prefix states for compression.
  • Readout-triggered start depth: FastE preserves the full prefix during warm-up and uses batch-mean readout–prefix alignment to select the first compression layer.The trigger is the first layer whose batch-mean alignment reaches θ; if none does, filtering occurs before the final block.
  • Readout-guided prefix-state filtering: At the trigger layer, FastE ranks prefix states by attention received from the readout position and retains the highest-scoring states within the configured budget.The readout position is excluded from the competition, while the retained prefix states preserve original causal order and rotary position IDs.
  • Efficiency: Shortening sequences from N_i + 1 to R_i + 1 before the trigger layer reduces computation in all remaining transformer blocks.The reported backbone FLOPs exclude controller costs, including alignment monitoring and triggering operations.

Experiments

FastE is evaluated across text, industrial, cross-modal, and cross-architecture settings, with matched prefix-state budgets and efficiency measurements that include online overhead. Results show strong quality retention, adaptive compression benefits, and a configurable quality–efficiency trade-off.

  • FastE limits mean retention loss on Qwen3-Embedding-0.6B to 0.02% at 30% removal and 4.15% at 70%, versus 2.53% and 14.55% for transferred baselines.
  • FastE remains near Full Forward through 50% removal, while at 70% it achieves the highest mean quality retention among compared methods.NarrativeQA and ArXiv P2P distinguish FastE most clearly from transferred baselines, whereas IMDb is less sensitive.
  • At 50% removal, cross-modal tasks retain at least 98.99% Recall@10 and average 99.53% retention; at 70%, average retention falls to 96.34%.The 70% setting is identified as aggressive for cross-modal retrieval, especially because NIGHTS declines from 0.9930 to 0.9190.
  • On E5-Mistral-7B-Instruct, FastE retains or slightly improves all three retrieval metrics while achieving 1.189× total end-to-end speedup.
  • At 70% removal, FastE reduces Qwen3-Embedding-0.6B FLOPs by 51.19% with 1.934× GPU-forward and 1.533× end-to-end speedups at 95.12% retention.For the 4B backbone, the corresponding 70% FLOPs reduction is 59.15% with 2.309×/2.070× speedups.
  • Ablation Studies: The adaptive trigger outperforms fixed start depths: Dynamic ¯g reaches 0.43179 nDCG@10 and improves FLOPs reduction over L12 from 46.78% to 51.19%.Dynamic ¯g triggers at mean layer 10.48, while fixed L9 filters prematurely and reduces nDCG@10 to 0.40351.
  • Ablation Studies: Readout-guided ranking achieves 0.43179 nDCG@10, exceeding mean-attention, random, and position-only ranking at 0.42464, 0.1426, and 0.05006.
  • Validation selects lw = 8 and θ = 0.60 because this configuration provides the largest FLOPs reduction among settings retaining at least 95% of Full Forward validation nDCG@10.

Limitations

FastE is scoped to final-readout LLM embedding models and does not directly transfer to final-layer pooling backbones without a pooling-aware compression signal.

  • FastE targets final-readout LLM embedding models; final-layer pooling backbones would require a pooling-aware compression signal.

Conclusion

FastE reduces the cost of final-readout LLM embedding inference by triggering and performing prefix compression without retraining. It retains near-Full-Forward quality while reducing decoder-backbone computation and supports textual and visual prefixes.

  • FastE retains 99.53% of Full Forward nDCG@10 while reducing decoder-backbone FLOPs by 40.11% on NarrativeQA with Qwen3-Embedding-0.6B.The same setting achieves a 1.363× end-to-end speedup.
  • FastE triggers compression with batch-mean readout–prefix alignment and ranks retained prefix states using readout attention.Later layers then process a shortened sequence.
  • Results across text tasks and three cross-modal retrieval tasks support the same final-readout interface for textual and visual prefixes.The method requires neither retraining nor custom sparse-attention kernels.

Supplementary Material for FastE: Readout-Triggered Token Compression for

The supplementary inference procedure keeps the full prefix during warm-up, triggers one-shot compression at the first qualifying threshold crossing, and guarantees compression before the final block if needed.

  • FastE retains the full prefix during warm-up and until the first threshold crossing, then compresses the sequence once.If no crossing occurs earlier, a final-layer fallback forces compression before the final transformer block.
  • The fallback guarantees a defined compression path when a batch never crosses the alignment threshold.It was not activated in the reported Qwen3-Embedding-0.6B NarrativeQA threshold sweep or E5-Mistral transfer run.
  • Algorithm 1 summarizes FastE inference for one batch.
  • For each input, the algorithm computes a retained-state budget from the maximum removal ratio before iterating through decoder layers.
  • At the trigger, FastE ranks prefix states by attention received from the readout position and selects the top states for retention.The retained states preserve their original rotary position identifiers and causal order.
  • After compression, the algorithm continues layer processing and returns normalized final-layer readout states as sequence embeddings.

B.1 Additional Evaluation Details

Additional evaluations examine task protocols, baseline fairness, timing, threshold and batch-size sensitivity, efficiency records, and representation-space evidence for depth-dependent compression.

  • B.1 Additional Evaluation Details: IMDb uses logistic-regression probe accuracy, while ArxivClusteringP2P.v2 uses MiniBatchKMeans mean V-measure; cross-modal tasks cover three retrieval settings.
  • B.1 Additional Evaluation Details: All baselines share FastE’s serialized prefix, truncation, batching protocol, target removal ratios, and preservation of readout and special tokens.ToMe instead performs layer-wise bipartite soft matching with cosine-similar attention keys.
  • B.1 Additional Evaluation Details: Timing includes alignment checks and trigger-layer scoring, while end-to-end timing additionally includes tokenization, data transfer, compression, and embedding output.
  • B.2 Joint Warm-Up and Threshold Sensitivity: 51.19% decoder-backbone FLOPs reduction accompanies 95.12% Full Forward nDCG@10 retention for the validation-selected configuration (lw, θ) = (8, 0.60).Lower thresholds generally trigger earlier and reduce more FLOPs, whereas θ = 0.70 favors quality retention.
  • B.2 Joint Warm-Up and Threshold Sensitivity: Across corpus batch sizes 1–16, nDCG@10 remains within [0.43139, 0.43637], while mean trigger layer varies by only 0.14 layers.The passage supports stability over tested length-sorted batches, not invariance to arbitrary batch composition.
  • B.1 Additional Evaluation Details: NarrativeQA efficiency records report Full Forward once per backbone, paired-run speedups, mean GPU-forward and end-to-end times, and peak allocated memory.Corpus batch-size sensitivity uses fixed query batch size and observed per-layer active sequence shapes for FLOPs.
  • B.4 Representation-Space Discussion: Figure 5 shows that alignment, readout–prefix similarity, and prefix-state commonality all increase toward deeper layers.The agreement between alignment and readout–prefix similarity indicates the trend is not solely caused by forming a raw prefix mean.
  • B.4 Representation-Space Discussion: Increasingly homogeneous prefix states and readout absorption provide a plausible explanation for why individual prefix states become less marginally useful at greater depth.This representation-space evidence complements, rather than replaces, controlled compression interventions and alignment–distortion analysis.

B.5 Cross-Architecture Transfer on E5-Mistral

FastE transfers to E5-Mistral through its official EOS-based embedding interface and reduces candidate-document sequence length while preserving retrieval quality. Supplementary analyses also report FLOPs accounting and broader industrial-task evaluation context.

  • B.5 Cross-Architecture Transfer on E5-Mistral: FastE uses E5-Mistral’s final valid EOS state as the sequence embedding and applies compression only to candidate documents.Query embeddings remain Full Forward and are shared across conditions.
  • B.5 Cross-Architecture Transfer on E5-Mistral: 69.97% realized prefix-state removal reduces mean candidate-document sequence length from 4,096 to 1,230 hidden states.Compression triggers at layers 20–22 with mean trigger layer 20.77 and no last-layer fallback.
  • B.5 Cross-Architecture Transfer on E5-Mistral: All three retrieval metrics remain at least as high as Full Forward, while mean corpus GPU-forward and end-to-end speedups reach 1.493× and 1.369×.Mean total E2E speedup including shared query encoding is 1.189×.
  • B.5 Cross-Architecture Transfer on E5-Mistral: The industrial evaluation uses a Qwen2.5-0.5B-Instruct user-embedding model with fixed warm-up and threshold settings and several maximum removal ratios.
  • B.7 Decoder-Backbone FLOPs Accounting: Decoder-backbone FLOPs accounting includes dense attention and MLP projections but excludes embeddings, normalization, RoPE, pooling, alignment evaluation, trigger scoring, and Top-K selection.
  • B.7 Decoder-Backbone FLOPs Accounting: FastE uses 40.85% of Full Forward decoder-backbone FLOPs, corresponding to an approximately 2.45× FLOPs-implied speedup for Qwen3-Embedding-4B.The shape passes record deterministic padded shapes and do not repeat quality or timed inference.
  • B.5 Cross-Architecture Transfer on E5-Mistral: The industrial-task results are summarized across 39 anonymized prediction tasks, with macro averages weighting tasks equally and retention measured relative to Full.

B.8 Alignment–Distortion Diagnostic

Alignment generally rises with depth as compression-induced distortion falls, but the relationship is not perfectly synchronous and alignment is only a heuristic for compression readiness. Backbone-specific threshold settings are therefore used in transfer experiments.

  • ρ = −0.942 pooled Spearman association links increasing batch-mean alignment with decreasing compression-induced distortion across monitored batch–layer pairs.
  • Mean distortion falls from 0.0240 at L10 to 0.00092 at L17 while alignment changes more slowly in intermediate layers.This pattern is consistent with continued information integration after alignment begins to plateau.
  • ρ = −0.190 partial Spearman correlation after controlling for layer and log sequence length supports alignment as a depth-wise heuristic, not a per-batch accuracy oracle.The 95% bootstrap confidence interval is [−0.248, −0.125].
  • Qwen3-VL uses θ = 0.70 because lower thresholds trigger earlier and substantially reduce Recall@10 on both visual probes.At θ ∈[0.55, 0.65], mean triggers occur at layers 9.4–12.4; θ = 0.70 delays the mean trigger to L18.3 and substantially recovers quality.
  • E5-Mistral uses a lower backbone-local threshold because its EOS readout has a different raw alignment scale.The reported setting is θ = 0.40, while the supplied passage describes the threshold sweep as post-hoc rather than heldout validation selection.

C.2 Additional E5-Mistral Results

The section reports additional E5-Mistral transfer results on Core17 and two LegalBench groups using fixed compression settings, with retention measured separately for each task. Corpus E2E speedup refers specifically to candidate-document encoding.

  • Additional E5-Mistral transfer evaluations cover Core17 and two LegalBench groups under fixed lw = 8, θ = 0.40, and rmax = 0.70 settings.
  • Retention is computed relative to Full Forward within each task, without forming a cross-task average.
  • Corpus E2E speedup measures candidate-document encoding.
Loading 2609.08407v1…