PaperlayerView exact revision on arXiv

Source-linked AI summary

KaLM-Reranker-V1: Fast but Not Late Interaction for Compressed Document Reranking

Xinping Zhao, Jiaxin Xu, Ziqi Dai, Xin Zhang, Shouzheng Huang, Danyu Tang, Xinshuo Hu, Meishan Zhang, Baotian Hu, Min Zhang

arXiv:2606.22807v2cs.CL

TL;DR

Existing rerankers couple query and passage computation, while late interaction sacrifices fine-grained interaction. KaLM-Reranker-V1 decouples passage encoding while retaining cross-attention, achieving competitive reranking with improved efficiency across benchmarks.

  • Problem

    Existing rerankers recompute passage representations for each query, while late-interaction methods improve efficiency but limit fine-grained query–passage interaction.

  • Method

    KaLM-Reranker-V1 pre-encodes passages offline, uses decoder cross-attention for relevance modeling, and applies Matryoshka pooling to reduce representation storage.

  • Results

    KaLM-Reranker-V1 delivers competitive reranking across BEIR, MIRACL, and LMEB, with Nano competitive against 7–12B embedding models using 0.27B activated parameters.

  • Takeaways & Limitations

    The FBNL design combines reusable passage representations with richer relevance modeling than late interaction while supporting efficiency–effectiveness trade-offs.

  • Takeaways & Limitations

    KaLM-Reranker-V1 remains relatively weak on the Chinese subset of MIRACL, suggesting the foundation model’s Chinese capability is a bottleneck.

Abstract

from arXiv · show

As retrieval systems scale, high-quality reranking becomes increasingly important. However, most existing rerankers, whether encoder-based or decoder-based, jointly encode the query and passage, tightly coupling their computation and limiting deployment efficiency as well as flexibility. We present KaLM-Reranker-V1, a fast but not late-interaction (FBNL) reranker that decouples query and passage computation while retaining expressive relevance modeling. Built on an encoder-decoder architecture, KaLM-Reranker-V1 uses the encoder to pre-encode passages with Matryoshka embedding pooling, while the decoder models the system instruction, user instruction, and query intent; cross-attention then captures relevance between the query context and passage representations. This design makes KaLM-Reranker-V1 efficient through decoupled passage encoding, yet not late interaction, by preserving rich relevance modeling through cross-attention. We instantiate KaLM-Reranker-V1 in three sizes, Nano, Small, and Large, with 0.27B, 1B, and 4B activated parameters, respectively. Extensive experiments on BEIR, MIRACL, and LMEB demonstrate that KaLM-Reranker-V1 achieves strong reranking performance with superior efficiency. On BEIR, KaLM-Reranker-V1 achieves state-of-the-art performance, on par with strong industrial models such as the Qwen3-Reranker series; on MIRACL, despite not being extensively trained on multilingual data, KaLM-Reranker-V1 still shows excellent reranking performance. Moreover, on LMEB, reranking models demonstrate a clear advantage, with even the 0.27B Nano model remaining competitive with 7-12B embedding models.

1 Introduction

KaLM-Reranker-V1 addresses the efficiency–expressiveness trade-off in reranking by decoupling passage and query computation while retaining richer relevance modeling than late interaction. Its compressed model family supports offline passage encoding and remains competitive across retrieval benchmarks and model scales.

  • Motivation: Reranking follows candidate retrieval and fine-grained relevance modeling, making reranking quality central to end-to-end systems.Ranked passages are subsequently provided to LLMs or processed through context engineering.
  • Motivation: Joint query–passage rerankers require passage representations to be recomputed for each query, increasing online computation and hindering offline precomputation.Late interaction improves inference efficiency by decoupling encodings but limits fine-grained query–passage interaction.
  • Method: KaLM-Reranker-V1 decouples query and passage computation through an encoder–decoder design while preserving expressive relevance modeling.The encoder represents passages and supports offline document encoding; the decoder provides richer relevance modeling than late interaction.
  • Method: Matryoshka embedding pooling reduces storage and serving costs while preserving reranking quality.Passage representations are precomputed offline to reduce online serving costs.
  • Results: 0.27B, 1B, and 4B activated parameters define the Nano, Small, and Large variants, respectively.The three sizes target diverse performance and serving-load requirements; on LMEB, Nano remains competitive with 7–12B embedding models while using only 0.27B activated parameters.

2 Related Work

Related work spans efficient embedding-based retrieval and deeper-interaction reranking models. Embedding methods map text to vectors for similarity search, while rerankers perform more detailed query–document relevance modeling.

  • Embedding Models: Embedding models map text into continuous vectors for efficient similarity search, evolving from GloVe word vectors to transformer-based sentence embedders such as Sentence-BERT.DPR demonstrated that dual-encoder passage retrieval can serve as a retrieval approach.
  • Reranking Models: Early neural rerankers concatenate the query and passage and apply full self-attention to estimate relevance.This deeper interaction addresses the limited interaction of embedding retrieval.
  • Reranking Models: Generative and LLM-based rerankers include RankGPT, RankVicuna, BGE, Qwen3-Reranker, and Jina-Reranker.These models are presented as later developments in reranking research.

3 Model Architecture

KaLM-Reranker-V1 uses a fast-but-not-late-interaction architecture that reuses compressed passage representations while retaining query–document interaction through decoder cross-attention. Built on T5Gemma2, it supports three encoder–decoder sizes and computes relevance from the likelihood of generating “yes” versus “no.”

  • FBNL architecture: Unlike late-interaction models, KaLM-Reranker-V1 preserves rich query–document interaction inside the decoder rather than postponing interaction until after separate encoding.This design targets scalable reranking without sacrificing expressive relevance modeling.
  • FBNL architecture: KaLM-Reranker-V1 decouples passage encoding from query scoring: the encoder produces reusable compressed representations, while the decoder models instructions and queries through cross-attention.Matryoshka embedding pooling compresses encoder outputs, and merged attention enables fine-grained relevance scoring.
  • Scoring mechanism: The encoder maps each candidate document p to Hp = Enc(p) ∈ Rn×d, which the decoder attends to alongside instruction and query inputs.The decoder forms queries from its input and keys and values from the concatenation of decoder input and encoder output.
  • Scoring mechanism: The relevance score compares the first-position likelihoods of generating “yes” and “no” for the candidate document.The two corresponding vocabulary logits are denoted zyes and zno.

4 Model Training

KaLM-Reranker-V1 is trained with supervised relevance classification, compression-aware passage representation learning, and teacher-guided distillation. A progressive three-stage pipeline moves from general reranking to task-specific adaptation and nuanced relevance learning.

  • Supervised Fine-Tuning: SFT trains KaLM-Reranker-V1 to predict yes/no relevance labels for an instruction, query, and candidate document.The objective uses the logits of the two label tokens, with yes assigned to relevant documents and no to irrelevant documents.
  • Compression-Aware Training: Matryoshka Embedding Pooling compresses passage representations along the sequence dimension to reduce storage overhead while preserving reranking effectiveness.Full passage representations Hp ∈ R^n×d can be costly to store across large corpora.
  • Compression-Aware Training: Training optimizes reranking across compression ratios R = {1, 2, 4, 8, 16, 32}, enabling flexible memory–quality trade-offs.The ratio r = 1 corresponds to the vanilla representation, and λr is set to 1 by default.
  • Knowledge Distillation: Knowledge distillation uses a stronger teacher’s soft labels to train student relevance scores with binary cross-entropy.Soft labels retain fine-grained relevance signals and mitigate potential false hard negatives.
  • Multi-Stage Training: The series follows a progressive three-stage pipeline: instruction-free general reranking learning, task-specific adaptation, and nuanced relevance learning.The first stage establishes domain-agnostic reranking ability across diverse retrieval scenarios before task-specific adaptation.

5 Complexity Analysis

KaLM-Reranker-V1 reduces online reranking cost by reusing compressed, pre-encoded passage representations while retaining decoder-side relevance modeling through merged attention. This decoupling yields large deployment efficiency gains, with additional cache memory controlled by Matryoshka Embedding Pooling.

  • Online Complexity: Conventional rerankers jointly encode each of K query–passage pairs online, whereas KaLM-Reranker-V1 reuses cached passage representations and runs only its decoder during serving.The model’s passage encoder is used only for offline pre-encoding, while query tokens attend to query and cached passage tokens through merged self-attention and cross-attention.
  • Efficiency Gains: 16.6× efficiency gain is achieved at n = 256, increasing to 203.4× when n = 4096.These gains are reported while varying passage length n over {256, 512, 1024, 2048, 4096} under fixed representative deployment settings.
  • Efficiency Gains: Nearly 10× efficiency gain is achieved even with a low compression ratio of r = 2.The compression-ratio experiment varies r over {2, 4, 8, 16, 32} while fixing the other deployment parameters to common values.
  • Memory Complexity: ⌈n/r⌉ cached tokens per passage reduce the online candidate buffer from O(Knd) before compression to O(K⌈n/r⌉d) with MEP.Larger compression ratios reduce cached passage representations nearly linearly while preserving decoder-side fine-grained relevance modeling.

6 Evaluation

KaLM-Reranker-V1 delivers strong, efficient reranking across BEIR and MIRACL, while its FBNL design preserves expressive relevance modeling. Performance remains relatively robust under moderate Matryoshka compression, although larger compression ratios and Chinese MIRACL expose limitations.

  • BEIR and MIRACL evaluation: On BEIR, Nano, Small, and Large achieve the best or second-best results on 9/13, 8/13, and 11/13 tasks, respectively.The series performs on par with strong Qwen, BGE, and Jina rerankers while retaining superior efficiency.
  • BEIR and MIRACL evaluation: Despite limited multilingual training, KaLM-Reranker-V1 shows competitive reranking ability on MIRACL, while Chinese performance remains relatively weak.The Chinese limitation suggests that the foundation model’s Chinese capability remains a bottleneck.
  • BEIR and MIRACL evaluation: FBNL achieves BEIR performance comparable to Qwen3-Reranker despite training on 3.7M rather than 19M high-quality data, while providing substantially higher efficiency.KaLM-Reranker-V1 uses the FBNL paradigm, whereas Qwen3-Reranker follows vanilla reranking.
  • Compression analysis: Reranking performance decreases as compression increases, but degradation is relatively mild from r = 2 to r = 16.Aggressive pooling discards useful information, whereas moderate compression preserves most reranking effectiveness.
  • Compression analysis: Increasing compression from r = 16 to r = 32 causes a substantially larger performance drop, so practical deployment is recommended at r = 2 to r = 8.This recommendation provides a favorable effectiveness–efficiency trade-off; larger models are also more robust to compression.

7 Conclusion

KaLM-Reranker-V1 is an efficient fast-but-not-late-interaction reranker family that decouples passage encoding from decoder-side relevance modeling while preserving expressive attention-based relevance modeling.

  • KaLM-Reranker-V1 introduces a family of efficient rerankers built on the fast but not late-interaction (FBNL) paradigm.
  • Decoupled passage encoding enables offline passage pre-encoding for more efficient deployment.
  • The decoder preserves expressive relevance modeling through merged self-attention and cross-attention over query context and pre-encoded passage representations.

A Training Data

KaLM-Reranker-V1 is fine-tuned on large-scale retrieval-specific data to improve reranking robustness and generalization. The data combines public embedding datasets with mined hard negatives for constructing reranking instances.

  • The model is fine-tuned on retrieval-specific datasets to develop its reranking capability.
  • The training data is large-scale, multilingual, and multi-domain, covering web search, question answering, duplicate-question retrieval, and fact verification.
  • The main public sources are the retrieval subset of KaLM embedding fine-tuning data and a selected subset of BGE-M3 training data.
  • Because many retrieval datasets provide only query-positive pairs, the authors mine hard negatives to construct reranking training instances.

B Implementation Details · C Instruction Templates

KaLM-Reranker-V1 uses LoRA-fine-tuned T5Gemma2 encoder–decoder backbones across Nano, Small, and Large variants, with progressive multi-stage training differing by model size. Evaluation instructions are organized for BEIR, MIRACL, and LMEB, with BEIR instructions detailed in Table 9.

  • B Implementation Details: KaLM-Reranker-V1 initializes from T5Gemma2 encoder–decoder backbones and fine-tunes both encoder and decoder parameters with LoRA.LoRA targets q_proj, k_proj, v_proj, and out_proj.
  • B Implementation Details: The Nano, Small, and Large variants use t5gemma-2-270m-270m, t5gemma-2-1b-1b, and t5gemma-2-4b-4b checkpoints, respectively.
  • B Implementation Details: Nano and Small complete all three progressive training stages, whereas Large completes only the first two stages.Each stage lasts one epoch.
  • B Implementation Details: The first, second, and third training stages use learning rates of 1 × e−4, 2 × e−4, and 5 × e−5, respectively.
  • C Instruction Templates: Tables 9, 10, and 11 summarize evaluation instructions for BEIR, MIRACL, and LMEB, respectively.
  • C Instruction Templates: Table 8 is identified as the source of training hyperparameters for the KaLM-Reranker-V1 series.
  • C Instruction Templates: Table 9 presents the task instructions used for evaluation on the BEIR benchmark.The benchmark is attributed to Thakur et al., 2021.

D Performance Scaling with Cost

Performance improves with greater computation but exhibits diminishing returns, while larger models do not reliably outperform smaller ones at comparable cost. Compression affects smaller models more strongly, with r = 4 often offering a favorable cost–performance trade-off.

  • Model size and cost: At comparable computational cost, switching to a larger model rarely brings performance gains.Increasing model capacity alone is not always cost-effective because gains must be balanced against higher serving cost.
  • Computation and compression: As computation cost increases, reranking performance generally improves with diminishing marginal gains across BEIR and MIRACL.The passage characterizes this as diminishing returns from additional online computation.
  • Computation and compression: r = 4 often provides a favorable trade-off, achieving performance close to r = 2 at substantially lower cost.This pattern is reported across both BEIR and MIRACL.
  • Model size and compression: Smaller models show larger performance changes as r varies, with Nano fluctuating the most.The passage suggests smaller models need more informative passage representations, whereas larger models better tolerate compression.

E Results on LMEB

On LMEB’s six dialogue memory-retrieval tasks, reranking substantially improves first-stage retrieval, especially for complex temporal queries, and a compact retrieve-then-rerank pipeline can outperform scaling embedding models alone. However, no reranker is best across every task, leaving memory retrieval challenging.

  • Reranking improves memory retrieval: 12.35 nDCG@10 points: KaLM-Reranker-V1-Small improves the first-stage retriever on LMEB, compared with a 6.23-point gain on BEIR.The larger LMEB gain highlights the value of reranking for long-horizon memory retrieval.
  • Remaining challenges: No reranker achieves the best result on all six dialogue memory-retrieval tasks, and every model has weaknesses on some datasets.The results indicate that memory retrieval still needs further optimization.
  • Rerankers help complex queries: 58.79 on TMD: Qwen3-Reranker-4B outperforms the first-stage retriever’s 16.82 on temporal and time-based queries.TMD includes relative and absolute time expressions that are difficult for embedding models, while rerankers handle them better.
Loading 2606.22807v2…