PaperlayerView exact revision on arXiv

Source-linked AI summary

MoganColBERT-TR: A Late-Interaction Multi-Vector Retrieval Model for Turkish

Furkan Yilmaz, Habibe Aleyna Tasdemir, Muhammed Faruk Gozay

arXiv:2608.26344v1cs.CL

TL;DR

Single-vector retrieval compresses variable-length documents into fixed-size points, while cross-encoders are too expensive at corpus scale; Turkish morphology further motivates token-level matching. MoganColBERT-TR adapts a Turkish embedding encoder through one-epoch cross-encoder distillation and achieves second place in clean zero-shot evaluation, with its main gap on long-query ArguAna-TR.

  • Problem

    Single-vector retrieval creates a severe information bottleneck, while cross-encoders require a full pass per query-document pair and are unusable at corpus scale.

  • Method

    MoganColBERT-TR adapts the embedding model’s encoder to late-interaction retrieval using cross-encoder score distillation over positives and mined negatives in a single training phase.

  • Results

    37.36 overall score: MoganColBERT-TR finishes second among five models in clean zero-shot evaluation, outperforming ColmmBERT-base-TR by +3.05 overall and LFM2.5-ColBERT-350M by +12.30.

  • Takeaways & Limitations

    Language fit of the encoder, rather than parameter count, is identified as decisive, while the largest deficit against the leader occurs on long-query ArguAna-TR.

  • Takeaways & Limitations

    The study reports results from one epoch and one seed, and does not empirically validate its major design decisions through ablations.

Abstract

from arXiv · show

We previously reported a ModernBERT encoder trained from scratch for Turkish (MoganBERT-TR) and a single-vector embedding model built on top of it (MoganBERT-embed). This work introduces the third model in that lineage: MoganColBERT-TR, a multi-vector retrieval model that, instead of compressing a query or a document into a single vector, represents it at the token level through a 768->128 projection and scores it with MaxSim late interaction. The model is not trained from scratch: the embedding model's encoder is taken as the starting point and adapted to the ColBERT objective with a single-epoch distillation phase. Training data is produced from two sources - title-to-passage pairs carved out of our own pretraining corpus in the character domain and at sentence boundaries, and two Turkish question-based retrieval sets - and is distilled from the soft scores of a cross-encoder teacher (bge-reranker-v2-m3) over one positive and seven mined negatives. We show that in hard negative mining, rank-based skipping alone is insufficient and must be combined with a group mask and a cosine ceiling. Evaluation is carried out with the official pipeline of TurkColBERT, a benchmark built for Turkish late-interaction retrieval (PLAID index, exact MaxSim), on five Turkish BEIR datasets; none of them appears in our training pool, so all five results are clean zero-shot. With 148.9M parameters, MoganColBERT-TR reaches an overall score of 37.36 (35.53 nDCG@100, 31.81 nDCG@10) averaged over the five datasets and finishes second among the five models compared: it outperforms the twice-as-large ColmmBERT-base-TR on four of five datasets and by +3.05 overall, and the benchmark's largest model by +12.30. The gap to the leading model (mLateOn) is concentrated on ArguAna-TR, the dataset with by far the longest queries.

1 Introduction

MoganColBERT-TR applies token-level late interaction to Turkish retrieval, targeting the compression limits of single-vector models while retaining precomputable document representations. It is a Turkish monolingual model in the MoganBERT lineage, with hard-negative filtering and corpus-derived training data as central contributions.

  • Motivation: Single-vector retrieval compresses each variable-length document into one fixed-size point, while cross-encoders require a query–document forward pass at corpus scale.ColBERT addresses this middle ground by encoding query and document separately but matching token representations during late interaction.
  • Motivation: 95% of Turkish word types split into multiple pieces, motivating token-level late interaction that keeps subword signals separate until scoring.Turkish agglutination concentrates inflectional and derivational information within words, making subword representation important for retrieval.
  • Contribution: MoganColBERT-TR is the third model in the MoganBERT lineage and a multi-vector retrieval model for Turkish.It follows MoganBERT-TR and MoganBERT-embed rather than introducing an unrelated model family.
  • Contribution: The model contributes a Turkish ColBERT trained from the authors’ corpus, a three-way hard-negative filter, and a decoding-free sentence-boundary data pipeline.The filter combines rank-based skipping with a group mask and cosine ceiling because skipping alone is insufficient under the stated corpus conditions.

2 Background: MoganBERT-TR and the Single-Vector Embedding Model

The work builds on a Turkish ModernBERT encoder and a single-vector embedding model whose training addresses representation anisotropy and retrieval alignment. MoganColBERT-TR starts from that embedding-model encoder to move beyond single-point document compression.

  • MoganBERT-TR: 237.3B tokens per branch were used to pretrain the ModernBERT-based Turkish encoder with a two-stage CLM→MLM curriculum.The encoder has 22 layers, hidden size 768, and 149.4M parameters.
  • MoganBERT-TR: 78.41 ± 0.32 on the 8-task TrGLUE average was achieved by the final Turkish encoder, ahead of TabiBERT at 77.83 and ModernBERT-TR at 77.64.This is the best score among the Turkish ModernBERTs compared in the supplied passage.
  • MoganBERT-embed: cosraw = 0.9841 and zero-shot IR = 0.2361 characterize the raw pretrained model’s anisotropic retrieval starting point.The passage describes random sentence representations as pointing in nearly the same direction, burying discriminative signal in cosine similarity.
  • MoganBERT-embed: 0.2361 → 0.5927 in zero-shot IR accompanied the embedding model’s geometry changes, alongside cosraw 0.9841 → 0.0851 and effective rank 117.9 → 157.2.The supplied passage attributes these changes to the single-vector embedding model’s training phases.
  • Transition to MoganColBERT-TR: MoganColBERT-TR initializes from MoganBERT-embed rather than a raw encoder because its retrieval-aligned geometry is presented as a better starting point for ColBERT projection.The encoder is subsequently updated, so the two models are independent afterwards.

3 Related Work

Related work positions late interaction between single-vector retrieval and cross-encoders, then distinguishes multilingual and monolingual approaches. Turkish evaluation requires a dedicated benchmark because standard embedding runners assume one vector per document.

  • From single-vector retrieval to late interaction: ColBERT keeps token-level representations and defers interaction to MaxSim, preserving precomputable document representations while retaining finer matching detail.This places late interaction between bi-encoders and cross-encoders in the retrieval design space.
  • Late-interaction systems: ColBERTv2 combines cross-encoder distillation with centroid-based residual compression, while PLAID reduces serving latency through centroid pruning.The supplied related-work passage describes this as the production-oriented development of late interaction.
  • Multilingual and monolingual late interaction: MoganColBERT-TR follows the monolingual lineage exemplified by JaColBERT and is presented as its first Turkish instance.Its encoder is a ModernBERT trained from scratch on an entirely Turkish corpus rather than fine-tuned from a foreign model.
  • Turkish-language setting: Turkish agglutination makes subword fragmentation a first-order retrieval design variable because one word can carry information distributed across several English words.This motivates attention to how tokenizer pieces are formed for late-interaction matching.
  • Turkish retrieval and evaluation: TurkColBERT evaluates Turkish versions of five BEIR datasets with PLAID and MUVERA paths because MTEB’s retrieval runner assumes one vector per document.It serves as the measurement ground for dense and late-interaction models across a broad parameter range.
  • Hard-negative mining and distillation: Passage splitting creates a Turkish-specific false-negative source: another passage from the same document can rank highly for the same title and survive naive first-k skipping.This motivates explicit filtering beyond rank-based skipping.

4 Model

MoganColBERT-TR replaces mean pooling with token-level projected representations and scores query–document pairs using asymmetric MaxSim. Query augmentation supplies additional matching slots, while fixed document length reflects the training data’s long-span construction.

  • Architecture: 768 → 128 is the bias-free token projection that replaces mean pooling in the MoganBERT-embed encoder.The dimensionality follows the established ColBERTv2 setting and trades index size against representational capacity.
  • Architecture: Each document is represented as a matrix of L_d token vectors, and the same encoder and projection are used for queries.The model therefore retains multiple vectors instead of compressing each input into one pooled embedding.
  • MaxSim scoring: MaxSim sums, over query tokens, the maximum similarity to any document token.The asymmetry lets every query token seek a document counterpart without penalizing unmatched document tokens.
  • Query expansion and length choices: 32 tokens is the fixed query length used for [MASK]-based query augmentation.Masked positions create additional context-generated matching slots, but the ceiling constrains paragraph-length queries.
  • Document length: 512 tokens is the document length setting, chosen to preserve long training spans and cover the benchmark document lengths.The training bands contain 250–1000 and 1000–2500-token passages, making a much shorter cap inconsistent with the data-generation rationale.

5 Training Data

Training data combines Turkish title-to-passage and question-based retrieval pairs, with careful passage construction and three complementary hard-negative filters before teacher distillation.

  • The training pool combines title→passage pairs from the pretraining corpus with two Turkish question-based retrieval sets.
  • Bodies exceeding 512 tokens are split in the character domain at sentence boundaries without decoding, preserving token limits and verbatim substrings.
  • At most 2 passages are taken per document to limit noisy positives and prevent long documents from skewing the training pool.
  • Seven negatives per query are mined from the MoganBERT-embed space among the top 100 candidates, using rank skipping, a group mask, and a cosine ceiling.
  • The group mask removes passages from the query’s source document, while the cosine ceiling catches relevant documents missed by fixed rank skipping.
  • Each training instance contains one positive and seven mined negatives, whose teacher scores are stored as soft targets for distillation.

6 Training

Training adapts the embedding encoder to late-interaction retrieval by distilling a cross-encoder’s score distribution into the student’s MaxSim distribution.

  • The student’s MaxSim score distribution over eight documents is aligned with the teacher’s distribution using KL-divergence distillation.
  • Training uses distillation rather than a contrastive objective, accepting the additional cost of the teacher phase.
  • AdamW uses separate learning rates of 1 × 10−5 for the encoder and 1 × 10−4 for the projection.
  • A two-step smoke test with the real Trainer, collator, and loss validates the data pipeline, tokenizer identifiers, and projection parameter groups.

7 Evaluation Protocol

Evaluation uses TurkColBERT’s official late-interaction pipeline on five Turkish BEIR datasets, with exact MaxSim indexing and no training-set overlap.

  • The study uses TurkColBERT because MTEB and the earlier TabiBench protocol assume single-vector retrieval and cannot score multi-vector models.
  • TurkColBERT evaluates five Turkish BEIR datasets spanning claim verification, argument retrieval, financial answers, citation prediction, and domain-intensive retrieval.
  • None of the five benchmark datasets appears in the training pool, making all reported results clean zero-shot evaluations.
  • PLAID with exact MaxSim is used to measure representational quality without introducing approximate-search effects.
  • The official document_length=300 setting is shorter than the model’s 512-token training window, so results are treated as a lower bound.
  • Metrics include nDCG@10, nDCG@100, Recall@100, and mAP, with @100 reflecting deep candidate-list retrieval.

8 Results

MoganColBERT-TR ranks second overall across five datasets, outperforming larger Turkish late-interaction models except on the biomedical NFCorpus-TR task.

  • 8.1 Overall result: MoganColBERT-TR finishes second on all four averaged metrics and outperforms the other three compared late-interaction models.
  • 8.1 Overall result: +3.05 overall separates MoganColBERT-TR from the twice-as-large ColmmBERT-base-TR, while LFM2.5-ColBERT-350M trails by 12.30 points overall.
  • 8.1 Overall result: +4.59 overall separates MoganColBERT-TR from the nearest model by size, ColmmBERT-small-TR.
  • 8.1 Overall result: The leading mLateOn model remains ahead by 5.27 points overall and 6.61 on Recall@100.
  • 8.2 Per-dataset results: MoganColBERT-TR ranks second on four datasets and fourth on one.
  • 8.2 Per-dataset results: Against ColmmBERT-base-TR, nDCG@10 gains are largest on SciFact-TR (+7.24) and ArguAna-TR (+5.97), where queries are full propositions.
  • 8.2 Per-dataset results: NFCorpus-TR is the sole loss: MoganColBERT-TR trails both Turkish ColmmBERT models in a biomedical domain dominated by specialist terminology.

9 Discussion

The discussion argues that Turkish language and tokenizer fit may matter more than parameter count, while query and document length settings constrain evaluation. MoganColBERT-TR’s largest gap to the leader appears on long-query ArguAna, and deployment favors two-stage retrieval as multi-vector indexes grow.

  • Model scale and language fit: MoganColBERT-TR’s ranking does not track parameter count: it is second-smallest yet second-placed, while the largest model is last.The authors hypothesize that Turkish tokenizer fit feeds late interaction directly, but call for an encoder-versus-tokenizer ablation.
  • Gap to the leader: ArguAna has the largest nDCG@10 deficit against mLateOn, ranging to 8.46, and its Recall@100 gap is 8.89 points.ArguAna contains the longest queries, which are full argument paragraphs rather than keyword phrases.
  • Evaluation constraints: The reported evaluation uses document_length=300 although the model was trained with 512, so the results should be read as a lower bound.The authors retained the shorter official setting for comparability.
  • Training efficiency: Single-epoch distillation produced a competitive model without ColBERTv2’s multi-round index-guided negative refresh, provided the negative pool was sufficiently clean.The discussion identifies the group mask as important for preventing same-document false negatives.
  • Deployment: Because multi-vector indexes can be tens of times larger than dense indexes, the recommended deployment is single-vector candidate retrieval followed by MoganColBERT-TR reranking.Standalone PLAID retrieval remains possible, but compression or pruning may become necessary as corpora grow.

10 Limitations

The reported evidence is bounded by limited training and evaluation variation, unmeasured design choices, a 32-token query ceiling, and domain and teacher coverage constraints.

  • Experimental variation: Training used one epoch and one seed, so reported differences were not separated from within-run noise.
  • Unvalidated choices: No ablations measured the effects of distillation, initialization, passage cap, cosine ceiling, or query length.These design decisions are justified but empirically unvalidated.
  • Sequence lengths: The 32-token query ceiling is a known source of loss on long-query tasks and was not measured against alternatives.
  • Sequence lengths: The official evaluation uses document_length=300, but no measurement was made at the 512-token training window.
  • Evaluation scope: All evaluations use five datasets from one benchmark, whose translation quality and qrels decisions may introduce shared systematic bias.No independent Turkish retrieval collection was used for validation.
  • Search regime: Only PLAID exact MaxSim was run, leaving the model’s speed-quality curve under approximate search unknown.
  • Domain coverage: NFCorpus-TR indicates weak biomedical coverage in the pretraining corpus, limiting domain coverage independently of the architecture.
  • Distillation: Using one cross-encoder teacher may transfer that teacher’s Turkish biases to the student through distillation.

11 Conclusion

MoganColBERT-TR is a Turkish late-interaction model initialized from a dense embedding encoder and trained with token-level distillation and cleaned hard negatives. In clean zero-shot TurkColBERT evaluation, it ranks second while outperforming substantially larger competitors, motivating longer-query and biomedical-data extensions.

  • Contribution: MoganColBERT-TR is the first Turkish late-interaction retrieval model derived from a ModernBERT encoder trained from scratch on an entirely Turkish corpus.
  • Method: The model uses a 768→128 token-level projection, single-epoch cross-encoder distillation, and hard-negative cleaning with group masking and a cosine ceiling.Training data is produced from the pretraining corpus without decoding and split at sentence boundaries.
  • Results: 37.36 overall: MoganColBERT-TR finishes second among five models in clean zero-shot TurkColBERT evaluation.The benchmark uses its official pipeline across five Turkish BEIR datasets.
  • Results: +3.05 overall: MoganColBERT-TR outperforms the twice-as-large ColmmBERT-base-TR on four of five datasets.
  • Results: +12.30: MoganColBERT-TR outperforms the benchmark’s largest model, LFM2.5-ColBERT-350M.
  • Implications: Raising the query-length ceiling and adding Turkish biomedical text are direct, measurable next steps under the paper’s evaluation protocol.
Loading 2608.26344v1…