PaperlayerView exact revision on arXiv

Source-linked AI summary

Mecellem Models: Turkish Models Trained from Scratch and Continually Pre-trained for the Legal Domain

Özgür Uğur, Mahmut Göksu, Mahmut Çimen, Musa Yılmaz, Esra Şavirdi, Alp Talha Demir, Rumeysa Güllüce, İclal Çetin, Ömer Can Sağbaş

arXiv:2601.16018v1cs.CL

TL;DR

Turkish legal language modeling remains challenging because English-centric pretraining and Turkish legal text’s terminology, morphology, and long-context demands limit existing models. Mecellem addresses this with scratch-pretrained encoders and curriculum-based continual pretraining for decoders, achieving top Turkish retrieval rankings and a 36.2% perplexity reduction on legal text.

  • Problem

    Turkish legal language modeling lacks robust evidence for adapting models to formal, terminology-intensive, morphologically rich text requiring precise contextual understanding.

  • Method

    Mecellem develops scratch-pretrained ModernBERT encoders with downstream-aware checkpoint selection and applies four-phase continual pretraining to Qwen3 decoders on Turkish legal data.

  • Results

    Mecellem’s encoders rank first and second on the Turkish retrieval leaderboard, while continual pretraining reduces perplexity on Turkish legal text by 36.2%.

  • Takeaways & Limitations

    The models provide practical tools for Turkish legal information retrieval and advance Turkish legal NLP through complementary encoder pretraining and decoder adaptation.

  • Takeaways & Limitations

    The decoder-to-encoder conversion uses single-stage supervised training rather than the official Qwen3 Embedding pipeline, contributing to a performance gap.

Abstract

from arXiv · show

This paper presents Mecellem models, a framework for developing specialized language models for the Turkish legal domain through domain adaptation strategies. We make two contributions: (1)Encoder Model Pre-trained from Scratch: ModernBERT-based bidirectional encoders pre-trained on a Turkish-dominant corpus of 112.7 billion tokens. We implement a checkpoint selection strategy that evaluates downstream retrieval performance throughout training, revealing that optimal checkpoints achieve best retrieval scores before pre-training loss reaches its minimum. Our encoder models achieve top-3 rankings on the Turkish retrieval leaderboard, with smaller models (155M parameters) achieving comparable performance to larger reference models (307M-567M parameters). Our approach achieves 92.36% production efficiency compared to state-of-the-art models (embeddinggemma-300m: 100.00%, BAAI/bge-m3: 99.54%, newmindai/bge-m3-stsb: 94.38%), ranking fourth overall despite requiring less computational resources. SOTA models rely on multi-stage, computationally intensive training pipelines, making our single-stage pre-training followed by efficient post-training approach a cost-effective alternative; (2)Decoder Model with Continual Pre-training (CPT): Qwen3-1.7B and Qwen3-4B models adapted to Turkish legal domain through controlled curriculum learning. Four-phase CPT with optimal sample ratios enables gradual transition from general language knowledge to specialized legal terminology and long-context reasoning. This approach achieves 36.2% perplexity reduction on Turkish legal text, demonstrating domain adaptation gains.

1. Introduction

Mecellem addresses the shortage of Turkish legal language resources with two complementary approaches: Turkish-dominant pretraining from scratch for ModernBERT encoders and continual pretraining for Qwen3 decoder models. The framework targets legal text’s specialized terminology, complex syntax, and domain-specific reasoning requirements.

  • Motivation: Turkish legal language resources remain insufficient for AI-powered legal tools despite growing demand for legal information systems, document analysis, and RAG applications.The gap is particularly consequential because Turkish legal applications must address Turkey’s unique legal framework.
  • Motivation: Legal adaptation requires accounting for specialized terminology, complex syntactic structures, domain-specific reasoning, corpus composition, training objectives, and evaluation metrics.These requirements reflect the distinct challenges of legal text processing compared with general-domain NLP.
  • Framework: Mecellem introduces two complementary approaches for developing Turkish legal language models.The framework combines encoder pretraining from scratch with decoder continual pretraining.
  • Encoder models: 112.7 billion tokens train ModernBERT-based bidirectional encoders from scratch on a carefully curated Turkish-dominant corpus.The encoder models include 155M base and 403M large parameter variants and use Masked Language Modeling as the primary objective.
  • Decoder models: Qwen3-1.7B and Qwen3-4B decoder models undergo continual pretraining on predominantly Turkish legal and official texts.Ablation studies determine initialization strategies and dataset sample ratios while balancing domain adaptation with preservation of general language capabilities.

2. Related Work

Related work establishes legal NLP benchmarks and domain-specific pre-training while highlighting gaps in Turkish legal modeling. It also motivates CPT, ModernBERT-based encoders, curriculum learning, and methods for converting decoder models into embeddings.

  • Legal NLP and benchmarks: Legal NLP research spans document classification, case retrieval, and summarization, with MLEB evaluating legal embeddings across document types, jurisdictions, and tasks.LEGAL-BERT demonstrated domain-specific pre-training for English legal text, while Turkish work has mainly used traditional machine learning and transformer-based classification.
  • Domain adaptation: Domain adaptation aligns general-purpose models with domain-specific linguistic and conceptual characteristics, but Turkish law’s formal and morphologically rich language challenges standard fine-tuning and training from scratch.The paper therefore uses CPT on Turkish-dominant legal datasets to preserve general knowledge while learning legal terminology, long-context reasoning, and domain-specific structures.
  • Encoder architectures: ModernBERT extends masked-language-model context to 8,192 tokens from BERT’s 512-token constraint and targets NER, semantic embedding generation, and classification.Rotary Positional Embeddings facilitate this longer-context architecture.
  • Decoder-based embeddings: Qwen3 Embedding converts autoregressive decoder models into embeddings through multi-stage training combining 150M synthetic pre-training samples with 7M labeled and 12M synthetic fine-tuning samples.The pipeline uses LLM-generated multilingual, multi-domain query-document pairs.
  • Training strategies: Curriculum learning progressively moves from simpler examples to intensive, normative, and long-context legal documents, improving training stability and supporting gradual domain adaptation.CPT must also address catastrophic forgetting through approaches such as EWC, replay, and orthogonal subspace constraints.

3. Methodology

The methodology combines a Turkish-dominant legal corpus, curriculum-based training, VLM-powered document extraction, and Turkish-specific quality filtering. It also uses tokenizer analysis to support morphology-aware modeling and reports extraction optimizations for large-scale processing.

  • Corpus and Training: Training follows four curriculum phases progressing from general texts to legal content, intensive long-context material, and mixed-complexity domain refinement.This arrangement is designed to transition gradually toward specialized legal language and reasoning.
  • Document Extraction: 56% throughput improvement increased extraction speed from 7–9 images per second to 11–14 images per second after upgrading from Flash Attention v2 to v3.The optimization supported processing tens of millions of pages within practical time constraints.
  • Document Extraction: 99.0% (±0.1%) natural completion remained stable, while only 1.0% of requests reached the 3,072-token generation limit.The configured output length was sufficient for most pages, minimizing truncation-related information loss.
  • Turkish Quality Filtering: 50% lemma diversity was selected because increasing the threshold caused remaining samples to fall from 8.7M at 50–50 to 1.6M at 50–65.Suffix-entropy thresholds were then refined between 70% and 90% using Zemberek-based morphological analyses.
  • Corpus and Training: The corpus combines Turkish legal text with general web data and totals approximately 112.7 billion tokens for ModernBERT-based encoders trained entirely from scratch.Sources include court decisions, academic legal publications, official gazette content, and limited multilingual and technical material.
  • Tokenizer Analysis: Turkish pre-trained models showed higher pure token counts of 6,000–12,000 and Turkish token coverage of 14,000–20,000 than multilingual mmBERT-base.These measures correlated with improved general and legal-task performance, motivating selection of Llama pre-tokenization with BPE.

URL-filtered web corpus with academic and legal

The pre-training corpus progressed through increasingly large and refined Turkish web, academic, and legal datasets, reaching ∼112.7B tokens. Additional quality and URL filtering, OCR-enriched theses, and page packing expanded domain coverage and improved sequence efficiency, while even minimal truncation harmed legal retrieval performance.

  • Corpus progression: ∼101.5B tokens formed a URL-filtered web corpus incorporating native Turkish academic articles and legal documents from YÖKTEZ, Yargıtay, and Danıştay.The filtering removed unwanted sources while aggregating previously used web data.
  • Corpus progression: ∼109.3B tokens expanded the web-and-legal foundation with a substantially larger OCR-derived collection of YÖKTEZ academic theses.This version increased academic-domain coverage beyond the initial corpus.
  • Corpus progression: ∼112.7B tokens constituted the final corpus after additional web-source quality and URL filtering plus page packing for YÖKTEZ documents.Page packing increased effective sequence length during training.

4. Experiments

Experiments show that downstream retrieval quality in Turkish legal models depends on checkpoint, data, sequence length, and post-training choices rather than pre-training loss alone. Continual pre-training improves legal-domain adaptation, with larger models reducing the need for curriculum structure.

  • Encoder models: 56.43 MTEB Score and 46.42 Legal Score make the ModernBERT-large v2 checkpoint best after post-training, despite non-monotonic MLM-loss and retrieval progression.The 403M-parameter v2 model achieves 56.43 overall versus 55.76 overall for the 155M base model at v5.
  • Encoder models: 55.67 MTEB Score and 46.36 Legal Score show that checkpoint 4 with a BGE-M3 guide combines domain-specific pretraining from scratch with effective post-training.GISTEmbed with guide models improves 4.6% overall and 15.9% legal performance over the InfoNCE baseline through false-negative filtering.
  • Encoder models: 8.5% degradation in legal performance for seq_len=256 relative to seq_len=2048 shows that long-context understanding is essential for legal retrieval.The degradation reaches 23.6% in regulation retrieval and 17.8% in case law retrieval, despite equivalent MS MARCO-TR validation scores.
  • Decoder models: 46.6% improvement in EnvL, 46.4% in FL, and 44.2% in CCML show the largest Qwen3-1.7B perplexity gains in complex legal domains.Qwen3-4B achieves lower absolute PPL values across all legal subdomains, including 4.884 versus 5.24 for Qwen3-1.7B Phase 4.
  • Decoder models: 43.1% overall improvement for four-phase Qwen3-1.7B CPT and 36.2% overall improvement for single-phase Qwen3-4B CPT demonstrate effective legal-domain adaptation.Qwen3-1.7B perplexity decreased monotonically across CPT phases, while Qwen3-4B maintained stable single-phase training because of its higher capacity.

5. Discussion

The discussion finds that Turkish legal domain adaptation improves retrieval and language-model performance, while parameter-efficient architectures and staged training provide strong alternatives to larger or more complex systems. It also identifies sequence length, false-negative filtering, morphology-based filtering, and decoder-to-encoder conversion as important design constraints.

  • Domain Adaptation Effectiveness: 36.2% perplexity reduction was achieved by Mecellem-Qwen3-4B-TR, while Mursit-Base-TR-Retrieval improved legal-domain performance by 17.5% over checkpoint 2.These results support the effectiveness of continual pre-training on Turkish legal text.
  • Encoder Model Parameter Efficiency: 155M-parameter Mursit-Base-TR-Retrieval achieved 55.86 MTEB Score and 47.52 Legal Score, ranking 2nd on the Turkish retrieval leaderboard.The 403M-parameter Mursit-Large-TR-Retrieval ranked 1st with 56.43 MTEB Score and 46.42 Legal Score.
  • Curriculum Learning Benefits: Qwen3-1.7B achieved comparable results to single-phase Qwen3-4B despite 2.4× fewer parameters through four-phase curriculum learning.The curriculum staged training from general language toward specialized content.
  • False-Negative Filtering Critical: 8.9% overall improvement was achieved by GISTEmbed with a guide model over standard InfoNCE, including 15.9% legal improvement.The result demonstrates the importance of preventing false-negative learning during contrastive training.
  • Sequence Length Configuration: 8.5% degradation in legal-domain performance occurred with seq_len=256 versus seq_len=2048, including a 23.6% relative decrease in regulation retrieval.Sequence length selected from training-data distribution alone was insufficient for specialized-domain evaluation.
  • Turkish Quality Filtering Impact: 0.6% perplexity reduction occurred for Qwen3-1.7B after Turkish morphology-based filtering, whereas Qwen3-4B showed a 1.0% increase.The filtering used suffix entropy > 75% and lemma diversity > 50%, with effects depending on model scale.
  • Decoder-to-Encoder Conversion: Mursit-Embed-Qwen3-4B-TR ranked 9th despite 4B parameters, while the 155M Mursit-Base-TR-Retrieval ranked 2nd and achieved 4.1% higher MTEB Score.The discussion attributes the conversion shortfall to unavailable large-scale synthetic data, multi-stage training, and model-merging infrastructure.

6. Conclusion

The work shows that Turkish legal language models can be developed through complementary encoder pre-training and controlled decoder continual pre-training strategies. The resulting models achieve competitive Turkish legal retrieval performance and provide practical tools for legal information retrieval.

  • Contributions: The framework combines encoder models pre-trained from scratch with downstream-aware checkpoint selection and decoder models adapted through controlled continual pre-training.The conclusion presents these as complementary strategies for developing Turkish legal language models.
  • Encoder models: Trainable MLM models support embedding tasks effectively when training quality is evaluated through downstream performance rather than pre-training loss alone.This finding motivates checkpoint selection based on downstream retrieval behavior.
  • Decoder models: Systematic ablations of initialization strategies and dataset compositions enable controlled decoder domain adaptation while preserving general capabilities.The conclusion identifies these ablations as the basis for adapting decoder models.
  • Impact: The resulting encoder models rank among the top positions on the Turkish retrieval leaderboard and advance Turkish legal NLP for legal information retrieval applications.The paper positions these results as both a research contribution and a practical resource.
  • Future work: Future work will explore advanced decoder-to-encoder conversion techniques and practical retrieval-augmented generation applications.These directions extend the framework toward additional model-conversion and application capabilities.
  • Availability: All models are released as open-source on HuggingFace under NewmindAI.The release makes the models available for reuse.

Appendix A. Results Reproduction · Appendix A.1. MLM Benchmark Results · Appendix A.2. Evaluation Results

Appendix A documents reproducible MLM and evaluation benchmarks through dedicated repositories containing code, configurations, and instructions. The MLM benchmark covers Turkish datasets with 80-10-10 masking, while the evaluation benchmark covers all 17 models on MTEB-Turkish.

  • Appendix A.1. MLM Benchmark Results: MLM accuracy scores in Table 13 are reproducible through the repository’s benchmark code and evaluation configurations.The repository also provides instructions for reproducing results across all evaluated models.
  • Appendix A.1. MLM Benchmark Results: The MLM reproduction procedure evaluates models on Turkish datasets using an 80-10-10 masking strategy.
  • Appendix A. Results Reproduction: The MLM benchmark repository is located at github.com/newmindai/mecellem-models/benchmark/mlm.
  • Appendix A.2. Evaluation Results: Evaluation results in Table 22 are reproducible through a repository containing the necessary code, evaluation configurations, and instructions.
  • Appendix A.2. Evaluation Results: The evaluation reproduction covers all 17 models assessed on the MTEB-Turkish benchmark.
  • Appendix A. Results Reproduction: The embedding-model reproduction repository is located at github.com/newmindai/mecellem-models/benchmark/embedding_model.

Appendix A.3. Post-Training Benchmark Results

Post-training with contrastive learning on MS MARCO-TR improves Turkish legal retrieval performance, with Mursit-Base-TR-Retrieval leading Legal Score and Mursit-Large-TR-Retrieval leading MTEB Score. Mursit models also show competitive or superior performance to TabiBERT across non-retrieval tasks, led overall by Mursit-Large.

  • Post-training methodology: Contrastive learning on MS MARCO-TR was used to fine-tune all models, improving retrieval performance on Turkish legal tasks.The evaluation covers retrieval models across all tasks and follows the methodology described in Section 4.3.
  • Retrieval benchmark: 47.52 Legal Score was achieved by Mursit-Base-TR-Retrieval, the highest among all evaluated models.Legal Score is the weighted average of Contracts, Regulation, and Caselaw scores.
  • Retrieval benchmark: 56.43 MTEB Score was achieved by Mursit-Large-TR-Retrieval, the highest among all evaluated models.MTEB Score is the mean performance across task types.
  • Non-retrieval benchmarks: Mursit models achieved competitive or superior performance to TabiBERT across all evaluated non-retrieval task types.The tasks include classification, named entity recognition, question answering, semantic textual similarity, natural language inference, and part-of-speech tagging.
  • Non-retrieval benchmarks: Mursit-Large showed the strongest overall performance across the non-retrieval task types.Performance was averaged across datasets within each task type, with Overall representing the mean across all tasks.

Appendix A.3.1. Production Deployment Efficiency

The appendix introduces a composite Production Efficiency metric balancing retrieval performance, legal specialization, and computational requirements. Among evaluated models, Mursit-Base-TR-Retrieval ranks fourth with 92.36 despite using fewer parameters than larger top-performing models.

  • Metric definition: Higher Production Efficiency scores indicate better trade-offs between retrieval performance and deployment costs.The metric balances retrieval performance, legal domain specialization, and computational resource requirements.
  • Model comparison: 94.38 is achieved by newmindai/bge-m3-stsb with 567M parameters, ranking third overall.Scores are normalized across 25 state-of-the-art embedding models for fair comparison.
  • Model comparison: 92.36 is achieved by newmindai/Mursit-Base-TR-Retrieval with 155M parameters, ranking fourth overall.Its score ranks fourth despite using fewer parameters than embeddinggemma-300m (307M) and bge-m3 (567M).
  • Model comparison: 91.26 is achieved by newmindai/Mursit-Large-TR-Retrieval with 403M parameters, ranking fifth overall.Smaller models can benefit from reduced memory requirements and faster indexing speeds; sequence length, embedding dimension, and parameter count influence production efficiency.

Appendix B. Reward Model Architecture Details

The Muhakim reward model uses prompt-conditioned gating to combine response-dependent rewards across five legal quality dimensions. Its three-stage training pipeline is designed to produce stable, interpretable, context-aware reward signals.

  • Reward Model Architecture: Prompt-conditioned gating adjusts the priorities of response-dependent reward components according to the legal domain or question type.This addresses fixed-weight evaluation, which assigns the same importance across contexts.
  • Reward Model Architecture: The model combines gating coefficients α_i(prompt) with response-dependent reward scores r_i(response) in a weighted score formulation.The design preserves per-dimension scores for interpreting which legal-text quality aspects drive the final assessment.
  • Training Pipeline: The training pipeline has three stages, beginning with multi-objective supervision over five legal quality dimensions using labeled scores.Specialized heads learn each dimension independently before subsequent preference learning.

Appendix C. Turkish Quality Filtering Threshold Selection

The appendix selects SE ≥75% and LD ≥50% as optimal Turkish morphology-based filtering thresholds. This configuration balances morphological variation and corpus diversity while retaining 31% of the original corpus for robust Turkish legal language modeling.

  • Threshold selection: SE ≥75% and LD ≥50% were selected as the optimal suffix entropy and lemma diversity thresholds.The evaluation systematically compared threshold combinations across the corpus for Turkish morphology-based quality filtering.
  • Threshold selection: 85-99% data reduction under SE ≥80% and LD ≥70% excessively limits corpus diversity and may degrade performance on rare morphological patterns.These more restrictive thresholds were contrasted with less restrictive configurations that retain more tokens.
  • Threshold selection: 31% of the original corpus is retained by SE ≥75% and LD ≥50%, preserving comprehensive morphological coverage for Turkish legal language modeling.The selected configuration provides a balanced trade-off between morphological variation requirements and corpus diversity.

Appendix D. Data Preprocessing and Corpus Construction · Appendix E. Decontamination Module

The appendices describe a conservative, reproducible preprocessing pipeline that produces a legally compliant corpus through cleaning, filtering, OCR normalization, deduplication, and curriculum assignment. A complementary decontamination module removes unsafe, low-quality, sensitive, high-risk, and linguistically uncertain content.

  • Appendix D. Data Preprocessing and Corpus Construction: 0.95 cosine similarity: SemHash filters dense BGE-M3 document embeddings to remove near-duplicate legal templates.Semantic deduplication uses dense document embeddings generated by the BGE-M3 encoder.
  • Appendix D. Data Preprocessing and Corpus Construction: GPU-accelerated cuDF workflows execute all preprocessing stages in batch-parallel form on MareNostrum 5.The approach prioritizes legal compliance, semantic diversity, and training stability over raw corpus size.
  • Appendix D. Data Preprocessing and Corpus Construction: The pipeline uses conservative and fully reproducible transformations to construct a legally compliant, high-quality dataset.The stated priorities are legal compliance, semantic diversity, and training stability.
  • Appendix D. Data Preprocessing and Corpus Construction: 132.6M to 72.2M documents: the full preprocessing pipeline reduces corpus size while preserving linguistically meaningful content for large-scale pretraining.The pipeline combines cleaning, filtering, OCR normalization, exact and semantic deduplication, and curriculum assignment.
  • Appendix E. Decontamination Module: The decontamination module removes profanity and filters low-quality content with C4, FineWeb, and Gopher filters.These stages extend the preprocessing pipeline described in Appendix D.
  • Appendix E. Decontamination Module: Sensitive personal information—including emails, phone numbers, URLs, national identifiers, and financial numbers—is anonymized.The module also filters high-risk domains and normalizes OCR artifacts.
  • Appendix E. Decontamination Module: GlotLID language confidence filtering removes content that fails the module’s language-quality checks.The decontamination module applies this filtering alongside profanity removal, quality filtering, anonymization, domain filtering, and OCR normalization.
Loading 2601.16018v1…