PaperlayerView exact revision on arXiv

Source-linked AI summary

Lychee-Reranker: Fast but Not Late Interaction for Compressed Document Reranking

Xinping Zhao, Jiaxin Xu, Ziqi Dai, Xin Zhang, Huiyao Chen, Shouzheng Huang, Xianhao Xiong, Danyu Tang, Xinshuo Hu, Guohong Fu, Meishan Zhang, Baotian Hu

arXiv:2606.22807v5cs.CL

TL;DR

Existing rerankers often recompute passage representations for each query, coupling their online computation and limiting flexible compute allocation. Lychee-Reranker pre-encodes passages and uses a decoder for expressive relevance modeling; experiments report competitive reranking with superior efficiency across multi-domain, multilingual, and long-horizon memory settings.

  • Problem

    Joint query–passage processing couples online computation and requires passage representations to be recomputed per query, while existing paradigms lack flexible inference-time compute allocation.

    Source: §1 · pp. 1–2

  • Method

    Lychee-Reranker pre-encodes passages, compresses their representations with Matryoshka embedding pooling, and uses decoder self-attention and cross-attention for query-conditioned relevance modeling.

    Source: §1 · p. 2

  • Results

    Lychee-Reranker reports strong reranking performance with superior efficiency across multi-domain, multilingual, and long-horizon memory retrieval settings.

    Source: §1 · p. 2, §5.2 · pp. 6–7

  • Takeaways & Limitations

    A cascaded strategy screens many candidates at high compression and examines promising ones at lower compression, improving efficiency with negligible performance loss.

    Source: §1 · p. 2

Abstract

from arXiv · show

As retrieval systems scale, effective and efficient reranking becomes increasingly important. However, most existing encoder- and decoder-based rerankers jointly process every query--passage pair, tightly coupling their online computation and limiting deployment efficiency and flexibility. We present Lychee-Reranker, a fast but not late-interaction (FBNL) reranker that decouples query and passage computation while retaining expressive relevance modeling. Built on an encoder--decoder architecture, Lychee-Reranker pre-encodes passages using Matryoshka embedding pooling, while its decoder models system and user instructions together with query intent; cross-attention then captures fine-grained relevance between the resulting query context and passage representations. Together, these designs offer four key advantages: (i) efficiency from offline passage encoding, (ii) expressiveness from cross-attention, (iii) compactness from Matryoshka embedding pooling, and (iv) test-time compute through an adjustable compute budget. We instantiate Lychee-Reranker in three sizes, Nano, Small, and Large, with 0.27B, 1B, and 4B activated parameters, respectively. Extensive experiments on BEIR, MIRACL, and LMEB demonstrate strong reranking performance with superior efficiency. On BEIR and MIRACL, our models achieve competitive performance in multi-domain and multilingual reranking, on par with strong industrial rerankers such as the Qwen3/BGE-Reranker series. On LMEB-Dialogue, a compact embedding model paired with our Nano reranker, which has only 0.27B activated parameters, remains competitive with 7--12B embedding models. Lychee-Reranker was previously known as KaLM-Reranker-V1. Data and models are available at https://huggingface.co/collections/KaLM-Embedding/lychee-kalm-reranker-and-jev. The code is available at https://github.com/HITsz-TMG/Lychee-Reranker.

1 INTRODUCTION

Lychee-Reranker decouples query and passage computation to combine reusable offline passage representations with expressive relevance modeling. Its experiments report competitive reranking across several settings, with efficiency gains and adjustable compute.

1 INTRODUCTION

  • Existing rerankers couple query and passage processing, requiring passage representations to be recomputed for each query, while late interaction limits scoring to predefined token-level similarities.Both paradigms also lack a mechanism for flexibly allocating inference-time computation to progressively refine relevance modeling.Source: §1 · pp. 1–2
  • Lychee-Reranker pre-encodes reusable passage representations and uses decoder self-attention and cross-attention over instructions, queries, and passage representations for fine-grained relevance modeling.Matryoshka embedding pooling compresses passage representations; changing its compression ratio supports flexible test-time compute.Source: §1 · p. 2
  • About 8x more efficient, Lychee-Reranker-Nano substantially outperforms gte-reranker-base on BEIR and slightly outperforms it on MIRACL.Across MEP compression ratios from 2x to 128x, performance degrades gradually, with only a very small drop between 2x and 8x.Source: §1 · p. 2, Figure 1 · p. 2
  • A compact embedding model paired with the 0.27B Nano reranker remains competitive with 7–12B embedding models on LMEB-Dialogue.The series also uses a cascaded strategy that screens many candidates at high compression and examines promising ones at lower compression, improving efficiency with negligible performance loss.Source: §1 · p. 2

2 RELATED WORK

Embedding retrieval scales through separate encoding but offers limited query–document interaction, while rerankers provide deeper relevance modeling. Existing reranking approaches trade off interaction depth and efficiency; Lychee-Reranker introduces FBNL to combine reusable passage representations with fine-grained scoring.

2 RELATED WORK

  • Bi-encoder embedding models match separately encoded queries and documents through vector similarity, limiting fine-grained interaction and motivating reranking.DPR established dual-encoder passage retrieval as a scalable first-stage retriever for open-domain QA.Source: §2 · p. 3
  • Full-interaction rerankers jointly process query–passage pairs, whereas late-interaction methods encode separately but rely largely on similarity aggregation between token representations.The two approaches illustrate the recurring trade-off between relevance modeling and efficiency.Source: §2 · p. 3
  • FBNL uses reusable encoder-produced passage representations while the decoder performs fine-grained query–passage relevance modeling.This is presented as a new reranking paradigm offering a more favorable efficiency–effectiveness trade-off.Source: §2 · p. 3

3 MODEL ARCHITECTURE

Lychee-Reranker reuses document representations to reduce query–document computation while retaining fine-grained interaction through decoder cross-attention. Its prompt templates keep document encoding reusable and make scoring instruction- and query-aware.

3.1 ARCHITECTURE

  • Built from T5Gemma2, Lychee-Reranker has encoder–decoder scales of 270M–270M, 1B–1B, and 4B–4B.The models initialize from T5Gemma2 to inherit pretrained language-modeling capabilities.Source: §3.1 · p. 3
  • FBNL reuses document representations for serving efficiency while decoder cross-attention preserves rich query–document interaction, unlike conventional and late-interaction rerankers.Figure 2 depicts the encoder producing compressed reusable passage representations and the decoder computing relevance scores via merged attention.Source: §3.1 · p. 3, Figure 2 · p. 4
  • The decoder attends to encoded documents using system and task instructions plus the query, then scores relevance from the next-token likelihoods of “yes” and “no”.The relevance score is defined using the logits for those two label tokens.Source: §3.1 · p. 4, §3.2 · p. 5

3.2 PROMPT TEMPLATES

  • The encoder prompt takes only the document, while the decoder prompt combines instruction and query with the reusable representation to compute the reranking score.Because T5Gemma2 does not support a system role, the system prompt is included in the user message.Source: §3.2 · pp. 4–5, Table 2 · p. 5

4 MODEL TRAINING

Lychee-Reranker training combines binary-label supervision with Matryoshka Embedding Pooling across compression ratios, followed by knowledge distillation and model-soup checkpoint averaging.

4.1 TRAINING OBJECTIVE

  • The SFT objective uses the negative log-likelihood of the ground-truth yes/no relevance label.Relevant documents use yes labels and irrelevant documents use no labels.Source: §4.1 · p. 5
  • MEP mean-pools consecutive groups of at most r passage tokens, producing compressed representations while aiming to preserve reranking effectiveness.The compressed representation has sequence length ⌈n/r⌉.Source: §4.1 · p. 5
  • Training jointly optimizes SFT losses across multiple compression ratios, encouraging representations to remain effective at different compression levels.The example ratios are 2, 8, 32, and 128; r = 1 denotes the uncompressed representation.Source: §4.1 · pp. 5–6

4.2 MULTI-STAGE TRAINING

  • The pipeline first fine-tunes on hard-labeled supervised reranking data, then distills fine-grained soft relevance signals from Qwen3-Reranker-8B.The soft signals are intended to capture nuanced differences between positive passages and hard negatives.Source: §4.2 · p. 6
  • The final model averages Stage 2 checkpoints trained with different LoRA configurations after fine-tuning and knowledge distillation.The training pipeline proceeds through fine-tuning, distillation, and model soup.Source: §4.2 · p. 6

5 EVALUATION

Lychee-Reranker performs competitively across BEIR and MIRACL while reducing estimated online computation. The evaluation also examines compression, selective test-time compute, and progressive training as ways to balance reranking quality and efficiency.

5.1 EXPERIMENTAL SETTINGS

  • Unless otherwise specified, experiments use a default compression ratio of r = 4; the main-results comparison covers BEIR and MIRACL.Tables 3 and 4 report reranking performance and efficiency against competitive baselines.Source: §5.1 · p. 6, §5.2 · p. 6, Table 3 · p. 7

5.2 MAIN RESULTS

  • 13/13, 12/13, and 13/13 BEIR tasks place Nano, Small, and Large, respectively, among the best or second-best results in their size groups.Lychee-Reranker-Small also outperforms Qwen3-Reranker-0.6B at an estimated online computation cost of 4.6x versus 28.2x.Source: §5.2 · pp. 6–7
  • 11 of 18 MIRACL languages rank Lychee-Reranker-Large first in its parameter group; it also outperforms Qwen3-Reranker-8B at 29.1x versus 359.8x online computation cost.The six listed languages with reported website usage rates total approximately 74.2% among websites with known content languages.Source: §5.2 · p. 7
  • On BEIR, Lychee-Reranker achieves performance comparable to Qwen3-Reranker with 3.9M training instances, versus 19M for the conventional Qwen3-Reranker series.The authors report that FBNL combines expressive relevance modeling with lower serving costs.Source: §5.2 · p. 8

5.3.1 EFFECTIVENESS OF MATRYOSHKA RERANKING

  • At r = 128, all variants retain at least 92% of their average nDCG@10 at r = 2 on BEIR and MIRACL; the authors recommend r = 4 to r = 16 for deployment.Performance decreases as compression increases, but only modestly from r = 2 to r = 16.Source: §5.3.1 · p. 8
  • At r = 128 on BEIR, Large retains 96.1% of its average nDCG@10 at r = 2, versus 95.1% for Small and 93.8% for Nano.The authors suggest larger rerankers may suit higher compression ratios, partly because their larger hidden dimensions may preserve more information per compressed passage token.Source: §5.3.1 · p. 8

5.3.2 POTENTIAL OF TEST-TIME COMPUTE SCALING

  • On BEIR, Small achieves average nDCG@10 of 61.11 with selective rescoring versus 61.17 for the baseline, while estimated relative serving cost falls from 8.65x to 2.75x.The strategy first reranks 100 candidates at r = 32 or r = 64, then rescores the top 20 at r = 2; some settings outperform the baseline on individual tasks.Source: §5.3.2 · pp. 8–9, Table 5 · p. 9

5.3.3 ABLATION STUDY ON MULTI-STAGE TRAINING

  • Across all three model sizes and seven compression ratios, performance improves consistently from Stage 1 through Stage 3 on BEIR and MIRACL.For Nano at r = 128 relative to r = 2, retention rises from 92.88% to 93.80% on BEIR and from 90.93% to 92.15% on MIRACL.Source: §5.3.3 · p. 9, Table 6 · p. 9

6 CONCLUSION

Lychee-Reranker uses offline passage encoding and Matryoshka embedding pooling within an encoder–decoder design. Experiments report competitive performance with lower online computation, alongside robustness to compression and benefits from test-time compute scaling.

6 CONCLUSION

  • Lychee-Reranker combines reusable offline passage representations with decoder self- and cross-attention for relevance modeling, while Matryoshka embedding pooling compresses passages.The design is intended to balance serving efficiency, expressive relevance modeling, compact representations, and adjustable test-time compute.Source: §6 · p. 10
  • Experiments on BEIR, MIRACL, and LMEB find competitive performance against strong industrial rerankers with significantly reduced online computation costs.The paper also reports compression robustness, effective multi-stage training, and value from test-time compute scaling.Source: §6 · p. 10

AI USE STATEMENT

The appendix analyzes online serving complexity and caching overhead for Lychee-Reranker, including how compression affects computational cost and storage. The block also includes the authors’ AI-use disclosure.

AI USE STATEMENT

  • The authors used AI tools to polish the writing and provide feedback on experimental design, then reviewed and verified suggestions and take responsibility for the final paper.Source: AI USE STATEMENT · p. 10

A.1 TIME COMPLEXITY

  • Lychee-Reranker reuses cached passage representations online, applying decoder computation and merged self- and cross-attention to query context and compressed passage representations.Passage encoding occurs offline and is excluded from online serving complexity because each passage representation is reused across queries.Source: §A.1 · pp. 17–18, Table 7 · p. 17
  • Estimated computation-cost reductions range from 16.6x at passage length n = 256 to 203.4x at n = 4096; at r = 4 and r = 8, reductions are 18.5x and 33.3x.The estimates vary passage length or compression ratio under fixed representative settings; at r = 2, the reduction is nearly 10x.Source: §A.1 · p. 18, Figure 5 · p. 18

A.2 SPACE COMPLEXITY

  • With compression ratio r, the online candidate-buffer space is O(K⌈n/r⌉d), reducing storage nearly linearly as r increases while retaining information for relevance modeling.The uncompressed online buffer requires O(Knd) space.Source: §A.2 · pp. 18–19

B TRAINING DATA

Lychee-Reranker is fine-tuned on multilingual, multi-domain retrieval data with hard negatives and teacher scores. The resulting training set contains approximately 3.9 million instances.

B TRAINING DATA

  • Training data spans web search, question answering, duplicate-question retrieval, and fact verification, drawn primarily from the KaLM embedding fine-tuning dataset and a selected BGE-M3 subset.Source: B TRAINING DATA · p. 19
  • For queries with fewer than 16 hard negatives, the authors retrieve 100 candidates and sample 16 negatives from ranks 10–50; otherwise, they retain existing hard negatives.Each resulting instance contains one query, one positive passage, and at least 16 hard negatives.Source: B TRAINING DATA · p. 19
  • Qwen3-Reranker-8B supplies teacher scores defined as yes-minus-no logits, whose sigmoid gives the soft teacher label.Source: B TRAINING DATA · p. 19
  • The final preprocessed training set contains approximately 3.9 million instances; BGE-Reranker-Large is evaluated with a maximum combined query-and-passage length of 512 tokens.Source: B TRAINING DATA · p. 19, Table 8 · p. 19

C EXPERIMENTAL SETUP

Experiments cover heterogeneous English retrieval, 18-language multilingual retrieval, and six long-horizon dialogue-memory tasks, using a shared retrieve-then-rerank evaluation pipeline. The section also specifies comparison baselines, model training, and task-specific instructions.

C EXPERIMENTAL SETUP

  • BEIR covers 13 English tasks and MIRACL covers 18 languages; LMEB adds six dialogue-memory tasks, with all rerankers scoring the same top-100 candidates using nDCG@10.Touche2020 and TRECCOVID are excluded because their test sets contain only 49 and 50 queries, respectively.Source: C EXPERIMENTAL SETUP · pp. 19–20
  • Comparisons span open-source encoder- and decoder-based rerankers with 0.3B–8B parameters; Lychee-Reranker uses r = 4 unless otherwise specified.The evaluated model specifications and task instructions are summarized in Tables 9 and 11–13.Source: C EXPERIMENTAL SETUP · p. 20, Table 9 · p. 20, E TASK INSTRUCTIONS · p. 21

D IMPLEMENTATION DETAILS

  • Models start from T5Gemma2 checkpoints and use LoRA; training includes three stages, Qwen3-Reranker-8B distillation, and model-soup averaging of two second-stage variants.MEP trains at R = {2, 8, 32, 128} with equal loss weights; all models use three stages, and the third stage adds no optimization.Source: D IMPLEMENTATION DETAILS · pp. 20–21, Table 10 · p. 21

F PERFORMANCE SCALING WITH TEST-TIME COMPUTE

Test-time compute gains depend on the dataset and model size: more compute generally improves performance with diminishing returns, while compression affects smaller models more. Model size and compression should therefore be chosen jointly for the application and compute budget.

F PERFORMANCE SCALING WITH TEST-TIME COMPUTE

  • At comparable cost, Large at r = 32 costs 46.6x versus Small at r = 2 at 62.3x and scores 62.18 versus 61.17 average nDCG@10 on BEIR, but 73.26 versus 74.34 on MIRACL.The larger, more compressed model improves on BEIR but not MIRACL.Source: F PERFORMANCE SCALING WITH TEST-TIME… · p. 22
  • For a fixed model size, reducing the compression ratio generally improves performance, but gains diminish; r = 4 or r = 8 often approaches r = 2 performance at lower cost.As curves flatten, switching to a larger model can yield greater gains than spending more compute on the same model.Source: F PERFORMANCE SCALING WITH TEST-TIME… · p. 22
  • On BEIR, increasing r from 2 to 128 lowers average nDCG@10 by 3.64 points for Nano (58.75 to 55.11) and 2.46 points for Large (63.69 to 61.23).The results indicate that smaller models are more sensitive to compression.Source: F PERFORMANCE SCALING WITH TEST-TIME… · p. 22

G RESULTS ON LMEB

On LMEB dialogue-memory retrieval, the paper reports gains from reranking and a competitive result from a compact embedding-plus-reranker pair. Results also show that memory reranking remains challenging, while compression degrades discrimination more noticeably at higher ratios and for smaller models.

G RESULTS ON LMEB

  • On LMEB, Nano, Small, and Large improve over the first-stage retriever by 10.27, 12.85, and 14.91 nDCG@10 points, versus 4.76, 7.29, and 9.75 points on BEIR.The larger gains on LMEB are reported across the three model sizes.Source: G RESULTS ON LMEB · pp. 22–23
  • Across six dialogue-memory tasks, no reranker achieves the best result on every task; Qwen3-Reranker-8B scores 78.93 nDCG@10 on LongMemEval, below Lychee-Reranker-Large’s 89.54.The paper identifies this as evidence that memory retrieval needs further improvement.Source: G RESULTS ON LMEB · p. 23
  • On TMD, Qwen3-Reranker-4B scores 58.79 nDCG@10, compared with 16.82 for the first-stage retriever; TMD contains many temporal queries involving relative or absolute time expressions.The paper says these queries challenge embedding models, while rerankers handle them more effectively.Source: G RESULTS ON LMEB · p. 23
  • On six LMEB dialogue-memory tasks, KaLM-Embedding-V2.5 (0.5B) with Lychee-Reranker-Nano (0.27B activated parameters) scores 61.07 average nDCG@10, versus 59.30 for bge-multilingual-gemma (9B) and 59.86 for Nemotron-3-Embed-8B-BF16.The paper presents this as an advantage of retrieve-then-rerank over simply increasing embedding-model size.Source: G RESULTS ON LMEB · pp. 23–24, Table 15 · p. 24

H UNDERSTANDING RERANKING PERFORMANCE DEGRADATION UNDER COMPRESSION

I ABLATION STUDY ON MATRYOSHKA EMBEDDING POOLING

The MEP ablation finds that training with MEP improves compression robustness and peak performance at low compression ratios. A qualitative case study illustrates how the decoder attends to query-relevant evidence in compact passage representations.

I ABLATION STUDY ON MATRYOSHKA EMBEDDING POOLING

  • Training Nano with MEP improves robustness as compression increases and yields higher peak performance at low compression ratios than training without MEP.Without MEP, BEIR Stage 1 average nDCG@10 falls 15.01 points, from 53.58 to 38.57, as r rises from 2 to 4; with MEP, performance declines much more slowly.Source: I ABLATION STUDY ON MATRYOSHKA… · p. 26

J CASE STUDY

  • The case study illustrates that MEP compresses precomputed passage representations eightfold while retaining evidence the decoder can attend to through cross-attention.The authors describe these visualizations as qualitative evidence that compact representations retain information accessible to the decoder.Source: J CASE STUDY · p. 31
  • At r = 8, attention in two FiQA examples overlaps query-relevant spans, including commuting tax guidance and advice about a deceased co-signer’s loan.The visualization averages the six largest layer-wise attention weights at each compressed position and maps them to the original passage chunks.Source: Figure 10 · p. 31, J CASE STUDY · p. 31

Related papers

Loading 2606.22807v5…