PaperlayerView exact revision on arXiv

Source-linked AI summary

TurkColBERT: A Benchmark of Dense and Late-Interaction Models for Turkish Information Retrieval

Özay Ezerceli, Mahmoud El Hussieni, Selva Taş, Reyhan Bayraktar, Fatma Betül Terzioğlu, Yusuf Çelebi, Yağız Asker

arXiv:2511.16528v1cs.CLcs.AIcs.IR

TL;DR

Turkish IR lacks systematic comparisons of dense and late-interaction retrieval, despite Turkish’s morphological complexity. TurkColBERT adapts pretrained encoders for retrieval and evaluates them across five Turkish BEIR datasets, finding that late-interaction models outperform dense encoders across the benchmarks.

  • Problem

    Turkish’s morphological complexity and limited systematic evaluation leave dense and late-interaction retrieval models without a consistent Turkish-specific comparison.

    Source: §1 · p. 2

  • Method

    The paper fine-tunes pretrained encoders on Turkish NLI/STS tasks, adapts them into ColBERT-style retrievers, and evaluates them across Turkish BEIR datasets.

    Source: §1 · p. 2

  • Results

    Late-interaction architectures outperform dense encoders across five Turkish BEIR datasets; ColmmBERT-base-TR reaches 70.0% Recall@10 on SciFact-TR.

    Source: §4 · pp. 6–7

  • Takeaways & Limitations

    ColmmBERT-small-TR achieves 97.5% of the full model’s performance on SciFact-TR while requiring less than half the computational resources.

    Source: §4 · p. 7

    The study uses translated benchmarks and datasets of at most 50K documents, which may not fully reflect real-world Turkish retrieval; large-scale production scalability needs further evaluation.

    Source: §5 · p. 7

Abstract

from arXiv · show

Neural information retrieval systems excel in high-resource languages but remain underexplored for morphologically rich, lower-resource languages such as Turkish. Dense bi-encoders currently dominate Turkish IR, yet late-interaction models -- which retain token-level representations for fine-grained matching -- have not been systematically evaluated. We introduce TurkColBERT, the first comprehensive benchmark comparing dense encoders and late-interaction models for Turkish retrieval. Our two-stage adaptation pipeline fine-tunes English and multilingual encoders on Turkish NLI/STS tasks, then converts them into ColBERT-style retrievers using PyLate trained on MS MARCO-TR. We evaluate 10 models across five Turkish BEIR datasets covering scientific, financial, and argumentative domains. Results show strong parameter efficiency: the 1.0M-parameter colbert-hash-nano-tr is 600$\times$ smaller than the 600M turkish-e5-large dense encoder while preserving over 71\% of its average mAP. Late-interaction models that are 3--5$\times$ smaller than dense encoders significantly outperform them; ColmmBERT-base-TR yields up to +13.8\% mAP on domain-specific tasks. For production-readiness, we compare indexing algorithms: MUVERA+Rerank is 3.33$\times$ faster than PLAID and offers +1.7\% relative mAP gain. This enables low-latency retrieval, with ColmmBERT-base-TR achieving 0.54 ms query times under MUVERA. We release all checkpoints, configs, and evaluation scripts. Limitations include reliance on moderately sized datasets ($\leq$50K documents) and translated benchmarks, which may not fully reflect real-world Turkish retrieval conditions; larger-scale MUVERA evaluations remain necessary.

1 Introduction

Turkish IR lacks systematic evaluation of late-interaction retrieval alongside dense encoders, despite the language’s morphological complexity. TurkColBERT adapts pretrained encoders and benchmarks both approaches across Turkish retrieval datasets.

1 Introduction

  • Turkish IR lacks systematic baselines comparing dense and late-interaction models, including modern late-interaction systems adapted to Turkish.Prior Turkish late-interaction work provides limited understanding across retrieval contexts and lacks uniform training protocols.Source: §1 · p. 2
  • Multilingual and English-centric models have not been tested in a Turkish-specific late-interaction setup using a consistent, reproducible pipeline.Source: §1 · p. 2
  • The paper adapts multilingual and English pretrained encoders in two phases: Turkish NLI/STS fine-tuning, followed by PyLate retrieval training on MS MARCO-TR.Source: §1 · p. 2
  • Evaluation compares the models with dense baselines across five Turkish BEIR collections and tests PLAID, MUVERA, and MUVERA with reranking.The authors release models, training configurations, and evaluation code.Source: §1 · p. 2

2 Literature Review

Dense and late-interaction retrieval offer different efficiency and representation trade-offs, while Turkish-specific evidence on their relative effectiveness and indexing remains limited. TurkColBERT addresses this gap with a controlled comparison and Turkish evaluation of multi-vector indexing.

2 Literature Review

  • Dense encoders use one vector per query and document for efficient approximate search, but this bottleneck can lose fine-grained semantic information.Source: §2 · p. 2
  • Late-interaction models preserve contextualized token embeddings for query-document scoring, while PLAID and MUVERA target their storage and latency costs.PLAID prunes candidates and compresses residuals; MUVERA encodes multi-vectors into fixed-dimensional representations.Source: §2 · pp. 2–3
  • Multilingual models’ effectiveness can fall on agglutinative Turkish because high-resource languages dominate pretraining data and limit exposure to Turkish morphology.Source: §2 · p. 3
  • Turkish-specific adaptations include mmBERT, Turkish BERT variants, and the dense TurkEmbed4Retrieval model, but no systematic dense-versus-late-interaction benchmark exists.Source: §2 · p. 3
  • The benchmark evaluates dense and late-interaction models under controlled training protocols and brings MUVERA indexing to Turkish, where both paradigms and indexing methods lacked systematic assessment.Source: §2 · p. 3

3 TurkColBERT: Benchmark for Turkish Information Retrieval Task

TurkColBERT adapts pretrained encoders for Turkish retrieval through semantic fine-tuning, PyLate late-interaction training, and MUVERA indexing. Its evaluation protocol compares architectures and indexing configurations across five Turkish BEIR datasets.

Stage 1: Semantic Fine-Tuning on All-NLI-TR & STSb-TR

  • Stage 1 uses Turkish NLI and STS fine-tuning to strengthen sentence-level semantics before retrieval-specific training.The pipeline initializes encoders, mean-pools final-layer token representations, and trains with MultipleNegativesRankingLoss wrapped in MatryoshkaLoss.Source: §3 · p. 3
  • mmBERT-small reaches 0.78 Spearman correlation on STSb-TR and 93% triplet accuracy on AllNLI-TR, improving over its pretrained baseline by +22% and +26%, respectively.These semantically improved checkpoints then start Stage 2 retrieval adaptation.Source: §3 · p. 4

Stage 2: Late-Interaction Adaptation via PyLate on MS MARCO-TR

  • Stage 2 uses PyLate to adapt semantically fine-tuned encoders into ColBERT-style retrievers, trained with contrastive triplet loss on MS MARCO-TR query-positive-negative triples.The ColBERT module retains per-token embeddings and uses MaxSim scoring; its training margin is 0.2.Source: §3 · p. 4

Stage 3: MUVERA Integration

  • MUVERA hashes token embeddings into partitions, sketches them with sparse projections, and aggregates partition representations into fixed-dimensional encodings for retrieval.The evaluated configurations include 128D, 512D, 1024D, and 2048D encodings; documents use nearest-neighbor imputation for empty partitions.Source: §3 · pp. 4–5

Final Stage: Comprehensive Evaluation on Turkish BEIR Benchmarks

  • The evaluation compares dense bi-encoders and late-interaction models across five Turkish BEIR datasets, with retrieval metrics spanning ranking quality and relevance.Models range from 0.2M to 600M parameters and include mmBERT, Ettin, BERT-Hash, and TurkEmbed4Retrieval architectures.Source: §3 · p. 5

MUVERA Indexing Ablation Study

  • The indexing ablation compares PLAID, MUVERA, and MUVERA with exact MaxSim reranking on four strong late-interaction models across all five datasets.The protocol measures retrieval metrics, indexing time, and per-query latency; Table 1 provides dataset statistics.Source: §3 · p. 5, Table 1 · p. 5

4 Results and Discussion

Across five Turkish retrieval datasets, late-interaction models lead overall, but the strongest model varies by task. Results also show trade-offs in model size and retrieval speed.

Model Comparison Across Architectures

  • Table 2 categorizes evaluated models by architecture, with late-interaction variants using token-level ColBERT representations.The reported evaluation metrics include mAP, P@10, and R@10.Source: §4 · p. 5, Table 2 · p. 6

MUVERA Indexing Ablation Study

  • On SciFact-TR, higher MUVERA encoding dimensions retrieve faster but slightly lower NDCG@100; 128D MUVERA+Rerank recovers near-PLAID quality with 4–5× speedup.Figure 1 compares dimensions from 128D to 2048D.Source: Figure 1 · p. 6

Model Comparison Across Architectures

  • The task difficulty varies sharply: SciFact-TR is easiest, with multiple models above 70% R@10, while Scidocs-TR peaks at 10.4%.The reported disparities often exceed 500% across metrics.Source: §4 · p. 6

Performance Across Benchmarks

  • ColmmBERT-base-TR leads mAP on SciFact-TR (56.8%), Fiqa-TR (19.5%), and NFCorpus-TR (11.5%); on SciFact-TR, its 70.0% Recall@10 exceeds TurkEmbed4Retrieval (60.5%) and turkish-e5-large (63.3%).Table 3 reports mAP, Precision@10, and Recall@10 across five Turkish retrieval datasets.Source: §4 · pp. 5, 7
  • ColmmBERT-small-TR has 140 million parameters and reaches 55.4% mAP and 70.3% Recall@10 on SciFact-TR, about 97.5% of the base model's performance.It requires less than half the computational resources of the 310-million-parameter base model.Source: §4 · p. 7

MUVERA Indexing Ablation Study

  • With TurkEmbed4Retrieval, MUVERA+RERANK reaches 0.5253 NDCG@100 versus PLAID's 0.3257, while reducing latency from 73.6ms to 35.2ms.The passage describes the approach as rapid candidate generation followed by detailed rescoring.Source: §4 · p. 7

Performance Across Benchmarks

  • Dense turkish-e5-large leads mAP on Arguana-TR at 17.9%, whereas on Scidocs-TR ColmmBERT-base-TR reaches 6.8% mAP against 2.2% for turkish-e5-large.The Scidocs-TR difference is reported as a 209% improvement.Source: §4 · p. 7
  • Colbert-hash-nano-tr retains 63.7% of the base model's SciFact-TR mAP with 1.0M parameters, while 0.2M-parameter colbert-femto-tr scores 19.0% mAP.The passage describes colbert-femto-tr's result as below production viability.Source: §4 · p. 7

5 Conclusion

TurkColBERT finds that late-interaction models outperform dense encoders across Turkish BEIR datasets, with strong parameter efficiency and low-latency MUVERA retrieval. The study is bounded by moderately sized datasets and translated benchmarks.

5 Conclusion

  • Up to 87% improvement in mAP on domain-specific tasks is reported for ColmmBERT-base-TR over strong dense baselines.The 1.0M-parameter colbert-hash-nano-tr retains over 71% of turkish-e5-large’s average mAP despite being 600 times smaller.Source: §5 · p. 7
  • 3.33x faster than PLAID, MUVERA+Rerank maintains 90–95% retrieval quality, with ColmmBERT-base-TR latency as low as 0.54 ms.Source: §5 · p. 7
  • Evaluation is limited to datasets of at most 50K documents and translated benchmarks, which may not reflect real-world Turkish retrieval.Further evaluation is needed to assess MUVERA scalability on large-scale production systems.Source: §5 · p. 7
Loading 2511.16528v1…