PaperlayerView exact revision on arXiv

Source-linked AI summary

Morpheus: A Morphology-Aware Neural Tokenizer and Word Embedder for Turkish

Tolga Şakar

arXiv:2606.18717v1cs.CLcs.AI

TL;DR

Turkish NLP needs tokenization and word representations that respect morphemes without sacrificing faithful text reconstruction. Morpheus jointly provides a lossless morphology-aware tokenizer and word embedder, achieving the strongest reversible-tokenizer results while excelling on lexical retrieval but trailing contextual encoders on context-dependent tasks.

  • Problem

    Turkish NLP tools only partially support morpheme-centered tokenization and word representation, with some morphology-aligned tokenizers unable to decode faithfully.

  • Method

    Morpheus uses one neural morpheme-boundary model with differentiable soft segmentation for training and exact surface-preserving morphemes at inference, while producing a word embedding.

  • Results

    Among reversible tokenizers, Morpheus achieves the lowest BPC (1.425) and leads lexical retrieval and same-root verification, while contextual encoders remain ahead on NER and inflection probing.

  • Takeaways & Limitations

    Morpheus is a complementary, cheap, morphology-aware lexical encoder rather than a drop-in replacement for contextual encoders, while also supporting generation-valid reversible tokenization.

  • Takeaways & Limitations

    Morpheus emits more tokens per word, making raw generation approximately 1.6× slower than BPE and favoring subwords for latency-bound generation.

Abstract

from arXiv · show

Turkish is agglutinative: meaning is carried by morphemes, yet the subword tokenizers that drive modern language models split words by corpus statistics, fragmenting semantically loaded suffixes and -- in the case of WordPiece and rule-based analyzers -- failing to decode their output back to the original text. This paper presents \textbf{Morpheus}, a neural morpheme-boundary model for Turkish that is at once a lossless, morphology-aware tokenizer and a word-embedding producer. A differentiable Poisson-binomial dynamic program turns per-character boundary probabilities into soft morpheme memberships during training and exact segments at inference, with no string normalization, so $\mathrm{decode}(\mathrm{encode}(w)) = w$ holds by construction. Because the model is neural, the same forward pass that tokenizes also emits a structured word embedding. Among reversible tokenizers -- the only ones valid for generation -- Morpheus attains the lowest bits-per-character ($1.425$), roughly doubles the gold morphological alignment of the subword family (MorphScore macro-F1 $0.61$ vs.\ ${\sim}0.32$), and uses ${\sim}19\%$ less GPU memory than 64K-vocabulary subword tokenizers. As an embedder, frozen Morpheus vectors lead on lexical retrieval (root-family MAP $0.85$) and same-root verification (ROC-AUC $1.00$), surpassing the multilingual retriever BGE-M3 and BERTurk; on context- and inflection-dependent tasks (NER, case/number probing) the heavier contextual encoders remain ahead -- a trade-off we attribute to Morpheus's root-centric geometry. Code: https://github.com/lonewolf-rd/TurkishMorpheus; model: https://huggingface.co/lonewolflab/Morpheus-TR-50K; interactive demo: https://huggingface.co/spaces/lonewolflab/morpheus-tr-demo.

1 Introduction

Turkish concentrates meaning in productive morpheme chains, exposing limitations in frequency-based tokenization and implicit word representations. Morpheus addresses both gaps as a lossless, morphology-aware neural tokenizer that also produces structured word embeddings.

  • Motivation: Turkish encodes most semantic content in derivational and inflectional suffix chains, making morphemes—not words or frequency fragments—the meaningful units.A single root can generate hundreds of surface forms through ordered morphemes.
  • Tokenization problem: BPE, WordPiece, and Unigram segment Turkish words by corpus statistics, causing poor morphological alignment and, for several tokenizers, irreversible decoding.Existing tokenizers address reversibility, morphological alignment, or low fertility only partially, rather than all three together.
  • Representation problem: Contextual encoders such as BERTurk provide strong embeddings but are heavyweight, tied to lossy subword vocabularies, and encode morphology only implicitly.The introduction identifies a missing representation that places morphologically related forms together by construction.
  • Morpheus: Morpheus is a neural morpheme-boundary model that jointly provides a lossless, morphology-aware Turkish tokenizer and a structured word-embedding producer.Its training combines Morfessor boundary supervision with skip-gram negative sampling, root-family contrastive learning, and masked language modeling.
  • Contributions: Morpheus achieves losslessness without inference-time normalization through differentiable Poisson-binomial soft segmentation that unifies training and inference.The paper evaluates the tokenizer with reversibility, MorphScore, SIGMORPHON, surface fidelity, and language-modeling BPC, and evaluates embeddings against BERTurk and BGE-M3.

2 Related Work

Related work frames Turkish tokenization as a tension between scalable frequency-driven subwords and morphology-aware alternatives. Morpheus builds on this literature by combining lossless morphology-derived tokenization with direct word-embedding production and evaluating both against established Turkish and multilingual representations.

  • Subword tokenization and its limits for Turkish: BPE, WordPiece, and Unigram are de facto interfaces for language models, but frequency-driven tokenization is not morphology-neutral for Turkish.The passage situates these methods as standard subword approaches and highlights their limitations for morphologically rich languages.
  • Morphology-aware and linguistically informed tokenizers: Morfessor provides an unsupervised morphemelike segmentation baseline, while Zemberek explicitly analyzes Turkish morphology but is dictionary-bound.Morpheus uses Morfessor as its boundary teacher.
  • Morphology-aware and linguistically informed tokenizers: Recent Turkish-specific tokenizers improve linguistic alignment through hybrid root/affix segmentation, phonological normalization, subword fallback, or syllable-based modeling.TurkishTokenizer reports strong Turkish-token and purity rates with competitive STS and TurBLiMP results, while Gulgonul targets resource-light retrieval.
  • Evaluation standards for Turkish tokenization: The TR-MMLU work introduces Turkish-token (%TR) and pure-token (%Pure) metrics, and Morpheus complements this protocol with exact reversibility and gold morphological evaluation.The cited work argues that linguistic token alignment correlates with downstream performance more strongly than raw token purity.
  • Turkish word representations and the tokenizer–embedding gap: BERTurk and multilingual encoders such as BGE-M3 provide established Turkish representation baselines, whereas Morpheus jointly tokenizes losslessly and emits a morphology-derived embedding.The paper evaluates this embedding directly against BERTurk and BGE-M3.

3 Methodology

Morpheus is trained on a diverse, cached Turkish corpus and jointly learns morphology-aware boundaries and word embeddings through a differentiable segmentation pipeline. Its architecture combines character-level contextual encoding, relative-position boundary detection, Poisson–binomial soft segmentation, and multi-objective training.

  • Data and preprocessing: The training corpus combines a multi-register author collection with cleaned Turkish Wikipedia, covering informal, academic, journalistic, and encyclopedic Turkish.The web sources were cleaned using documented extraction, HTML/URL stripping, Unicode normalization, and deduplication procedures.
  • Data and preprocessing: Sentences are split 95/5 with a fixed seed and pre-tokenized into cached tensors containing character ids, case flags, Morfessor boundary labels, and word ids.Character sequences are padded to max word len = 32; Morpheus caches up to 900K training and 100K validation sentences.
  • Model architecture: A character encoder using case embeddings, kernel widths 2–6, and 3 self-attention layers produces d = 320 contextual character states, followed by 4 RoPE attention layers that predict inter-character boundary probabilities.RoPE injects relative offsets so learned boundary patterns can generalize across roots of different lengths.
  • Differentiable segmentation: A differentiable Poisson–binomial dynamic program converts boundary probabilities into a soft segment-membership matrix with O(LS) cost and preserves differentiable gradient flow.As p_i approaches {0, 1}, memberships converge to one-hot assignments, recovering exact hard segmentation.
  • Joint tokenization and embedding: Attention-pooled segment vectors are averaged and passed through a residual two-layer FFN with LayerNorm to produce e_w ∈ R^320 from the same forward pass as tokenization.The total loss combines boundary supervision, skip-gram negative sampling, root-identity InfoNCE, and masked-language-model objectives.

4 Results

Morpheus combines stable training with lossless, morphology-aware Turkish tokenization, achieving strong morphological alignment and efficient language-modeling behavior. Its embeddings excel at root-level lexical tasks but trail contextual encoders on inflection- and context-dependent tasks.

  • Training dynamics: Morpheus training is stable: train and validation losses decrease together, boundary metrics plateau early, and the four objectives converge jointly.The auxiliary boundary loss drops fastest, while skip-gram, contrastive, and MLM losses continue shaping embedding geometry.
  • Reversibility: 58.2% of WordPiece words and 95.4% of TurkishTokenizer words round-trip correctly, whereas Morpheus and the subword family are reversible.WordPiece strips Turkish diacritics, while TurkishTokenizer’s canonical re-harmonization can rewrite surface forms such as saatlerde.
  • Morphological alignment: 66% root accuracy and 38% →38% len-to-exact fidelity make Morpheus the strongest reversible option, with MorphScore macro-F1 0.61 versus ∼0.32 for the subword family.TurkishTokenizer places boundaries well but drops from 78% len to 10% exact because canonical suffix rewriting corrupts surface strings.
  • Language modeling and efficiency: 1.425 validation BPC is Morpheus’s lowest among reversible tokenizers, while generation uses ∼19% less peak GPU memory than 64K-vocabulary subword tokenizers.Morpheus reaches 83.5% frequency-weighted purity and 91.8% Turkish-token rate on TR-MMLU, with 1.73 tokens per word.
  • Embedding evaluation: Root-family retrieval reaches MAP 0.85 and same-root verification ROC-AUC 1.00, surpassing BGE-M3 and BERTurk despite Morpheus’s 320-dimensional embeddings.Contextual encoders instead lead on number probing, case probing, and WikiANN NER, where Morpheus scores 0.59, 0.22, and macro-F1 0.48 respectively.

5 Discussion

Morpheus’s discussion centers on one neural morphology signal serving both lossless tokenization and structured word embedding. Its root-centric design yields strong lexical performance and reversible generation, while trading away inflection-sensitive context understanding, fertility, and throughput.

  • One signal, two roles: A single neural morpheme-boundary model supports both lossless tokenization and word embedding through differentiable Poisson-binomial segmentation.The same morphological signal places boundaries and shapes the pooled embedding.
  • Lossless-versus-lossy is the decisive axis: Among reversible tokenizers, Morpheus simultaneously achieves the lowest BPC, highest frequency-weighted token purity, strongest morphological alignment, and lower memory.WordPiece and TurkishTokenizer lead isolated raw-BPC or gold-morphology metrics but are disqualified for generation by irreversibility.
  • A root-centric embedding, by design: Morpheus leads lexical retrieval and deduplication but underperforms on number/case probing and NER.Root-identity contrastive training pulls inflections together, while static pooled vectors lack the sentence context required by NER.
  • A root-centric embedding, by design: Morpheus is suited to a cheap, morphology-aware lexical index, while BGE-M3 or BERTurk can serve a dense semantic index.This division follows Morpheus’s strength at root matching and contextual encoders’ advantage on context-dependent tasks.
  • What you trade: ∼1.73 vs. ∼1.5 tokens/word marks Morpheus’s higher fertility, alongside a heavier artifact and lower raw character throughput.For latency-bound generation, the discussion retains subword tokenizers as preferable.

6 Limitations and Trade-offs

Morpheus’s limitations are trade-offs tied to concrete gains: longer sequences and slower generation accompany reversible, morphology-aligned tokenization, while root-centric static embeddings and Turkish-focused supervision constrain broader contextual and morphological coverage.

  • Fertility for quality and faithfulness: ∼1.73 tokens per word and ∼1.6× slower generation than BPE trade throughput for 1.425 BPC, morpheme alignment, lossless decoding, and ∼19% lower GPU memory.The slowdown reflects token count rather than slow encode/decode, which remain competitive.
  • A neural artifact for OOV generalization: Model-based segmentation handles any Turkish word, including nonce and rare agglutinative forms, but requires a PyTorch checkpoint instead of a small vocabulary.Unlike fixed BPE/WordPiece tables, it has no vocabulary cap.
  • A root-centric embedding: strength and limit are the same design: MAP 0.85 and ROC-AUC 1.00 on root-focused evaluations contrast with weaker number/case probing and NER performance against contextual encoders.The contrastive objective concentrates a root’s inflections, making the same root-centric geometry both strength and limitation.
  • A root-centric embedding: strength and limit are the same design: A static 320-dimensional embedding versus 768/1024-dimensional contextual encoders makes Morpheus complementary to, rather than a replacement for, contextual models.Its lower dimensionality and static representation align with its root-centric design.
  • Scope: Turkish-specific design and inflection-focused gold sets leave derivational families and long, rare agglutinative chains comparatively under-probed.The boundary detector occasionally merges adjacent suffixes in long, rare chains.

7 Conclusion

The conclusion presents Morpheus as a lossless, morphology-aware Turkish tokenizer and word embedder built around differentiable Poisson–binomial segmentation. It reports strong reversibility, tokenization, memory, and lexical-retrieval results, while identifying fertility, generation speed, and context-dependent understanding as trade-offs.

  • Novelty and mechanism: Morpheus uses differentiable Poisson–binomial segmentation to train boundaries end-to-end, recover exact inference segments, and guarantee decode(encode(w)) = w.The mechanism only groups characters and emits a word embedding in the same forward pass.
  • Measured success: BPC 1.425, token purity 83.5%, MorphScore macro-F1 0.61, 100% reversibility, and ∼19% lower GPU memory establish Morpheus as the strongest faithful tokenizer.These results are restricted to tokenizers whose ids decode to faithful Turkish and are therefore valid for generation.
  • Measured success: MAP 0.85 and ROC-AUC 1.00 lead lexical retrieval and same-root verification, ahead of BGE-M3 (0.80/0.98) and BERTurk (0.49/0.70).The conclusion attributes this advantage to Morpheus’s root-centric embedding geometry.
  • Trade-offs and where to use it: Fertility 1.73 vs. ∼1.5 tokens/word and ∼1.6× slower generation accompany a root-centric embedding that trails contextual encoders on NER and number/case probing.Morpheus is a neural artifact rather than a lookup table, adding a concrete deployment cost.
  • Trade-offs and where to use it: Morpheus is recommended for Turkish NLU, sequence-labeling, and lexical or keyword indexing, while subword tokenization remains preferable for latency-bound raw generation.The conclusion frames Morpheus as a single lossless, morphology-aware, embedding-producing alternative to lossy or morphology-blind systems.
Loading 2606.18717v1…