Source-linked AI summary
Diffutron: A Masked Diffusion Language Model for Turkish Language
Şuayp Talha Kocabay, Talha Rüzgar Akkuş
TL;DR
Masked diffusion models remain insufficiently studied for morphologically rich Turkish. Diffutron adapts a multilingual encoder through LoRA-based continual pre-training and progressive instruction tuning, achieving competitive benchmark performance with 307 million parameters against much larger autoregressive baselines. The results support masked diffusion with multi-stage tuning as a resource-efficient approach to non-autoregressive Turkish generation.
Problem
Diffusion language models have largely focused on English, leaving their effectiveness for morphologically rich Turkish insufficiently understood.
Method
Diffutron uses LoRA-based continual pre-training of a multilingual encoder followed by progressive instruction tuning on general and task-specific instruction sets.
Results
307 million parameters yield competitive benchmark performance against roughly 2B-parameter autoregressive baselines.
Takeaways & Limitations
Masked diffusion modeling combined with multi-stage tuning offers a viable resource-efficient path for non-autoregressive text generation in Turkish.
Takeaways & Limitations
The model is limited by the multilingual backbone, scarce native Turkish instruction data, a 256-token context window, and constrained evaluation coverage.
Abstract
from arXiv · showhide
Masked Diffusion Language Models (MDLMs) have emerged as a compelling non-autoregressive alternative to standard large language models; however, their application to morphologically rich languages remains limited. In this paper, we introduce $\textit{Diffutron}$, a masked diffusion language model specifically designed for Turkish. Our approach leverages a resource-efficient training pipeline, starting with LoRA-based continual pre-training of a multilingual encoder on a large-scale corpus. To enable generative capabilities, we employ a progressive instruction-tuning strategy, sequentially adapting the model on general and task-specific instruction sets. Experimental results across comprehensive benchmarks demonstrate that, despite its compact size, our model achieves competitive performance compared to existing multi-billion-parameter baselines. These findings validate the effectiveness of masked diffusion modeling combined with multi-stage tuning for non-autoregressive text generation in Turkish.
1 Introduction
Diffutron addresses the limited study of masked diffusion models for morphologically rich Turkish through a lightweight, multi-stage adaptation pipeline. Despite its compact size, it achieves competitive benchmark performance against much larger autoregressive models.
- Research on diffusion language models has largely focused on English, leaving their effectiveness for morphologically rich languages such as Turkish insufficiently understood.
- Diffutron is a lightweight, parameter-efficient masked diffusion language model tailored specifically to Turkish.
- The training pipeline begins with LoRA-based continual pre-training of the multilingual jhu-clsp/mmBERT-base encoder on a large-scale corpus.
- Progressive instruction tuning sequentially adapts the model using general and task-specific instruction sets to improve instruction-following capabilities.
- 307 million parameters enable Diffutron to achieve competitive performance against roughly 2B-parameter autoregressive baselines.
2 Related Works
Prior work establishes autoregressive models as the dominant paradigm while motivating masked diffusion as a non-autoregressive alternative. LoRA is used to adapt multilingual representations while reducing the risk of catastrophic forgetting.
- Autoregressive Transformers dominate language modeling but generate tokens sequentially, creating bottlenecks in generation speed, planning, and inference.
- Early diffusion language models used continuous latent spaces and rounding to recover tokens, creating challenges in mapping continuous states back to discrete text.
- Discrete masked diffusion models operate directly on token states through transition matrices and align conceptually with masked language modeling.
- Multilingual adaptation risks catastrophic forgetting, while full-parameter fine-tuning can disrupt pre-trained feature spaces.
- LoRA serves as a regularization mechanism that preserves robust cross-lingual representations while adapting the model to a target language.
3 Preliminaries
Masked diffusion language modeling treats generation as discrete diffusion: text is progressively masked, then reconstructed through iterative denoising. This enables global sentence refinement using bidirectional visible context.
- MDLMs generate tokens in parallel through iterative refinement rather than sequential autoregressive decoding.
- The forward process gradually corrupts clean text x0 into xT across T timesteps, with <mask> serving as the absorbing noise state.
- At each transition, tokens either remain unchanged or are replaced by <mask> with probability βt through transition matrix Qt.
- By timestep T, the sequence is effectively composed entirely of <mask> tokens.
- The reverse process predicts original tokens from the noisy state and iteratively samples less-masked states from a fully masked sequence.
- Reverse generation refines the entire sentence globally, with masked-token predictions attending bidirectionally to visible context.
4 Continual Pre-training
Continual pre-training adapts a multilingual encoder to Turkish using a curated, length-filtered corpus and LoRA across attention and MLP modules. The reported loss trajectory decreases steadily without training instability.
- Continual pre-training aligns the multilingual encoder’s latent space with Turkish while preserving semantic reasoning capabilities from initial pre-training.
- The corpus combines Havadis, Temiz-OSCAR, and Turkish Wikipedia to balance web, news, and encyclopedic content within architectural context constraints.
- Sequences longer than 512 tokens were filtered, producing approximately 406,000 encyclopedic sequences and a sampled pool of 1.6 million web and news sequences.
- LoRA targets all attention and MLP linear modules to capture Turkish agglutinative complexity while preserving the frozen backbone’s cross-lingual reasoning.
- The masked language modeling objective is trained with memory-efficient optimization and a cosine learning-rate schedule.
- The pretraining loss decreases steadily, and the reported trajectory indicates that high-rank adaptation models the target distribution without training instability.
5 Instruction Fine-Tuning
Diffutron uses two sequential supervised fine-tuning stages: broad Turkish instruction following first, followed by specialized instruction adaptation. Training loss declined in both stages, with the second stage converging more smoothly and slightly more slowly.
- 5 Instruction Fine-Tuning: The fine-tuning pipeline uses two stages: general instruction tuning followed by specialization for Turkish instruction tasks.Both stages use AdamW with a learning rate of 1 × 10−4.
- 5.1 First Stage: The first stage fine-tunes on diverse Turkish instruction-response pairs to improve general instruction understanding and response coherence.Training runs for 20 epochs with a relatively small batch size intended to stabilize gradient updates.
- 5.1 First Stage: The first-stage configuration is summarized in Table 2.The supplied table passage provides its configuration label, while the accompanying description specifies 20 epochs and a small batch size.
- 5.1 First Stage: Figure 3 shows a consistent loss decline, with rapid early learning followed by a later plateau as general instruction-following patterns are captured.This stage provides the foundation for subsequent specialized fine-tuning.
- 5.2 Second Stage: The second stage fine-tunes on InstrucTurca, whose specialized and nuanced Turkish instructions target more complex or domain-specific tasks.Training uses a significantly larger batch size, two A100 GPUs, and 8 epochs.
- 5.2 Second Stage: Figure 4 shows steadily decreasing loss during the second stage, with smoother and slightly slower convergence than in the first stage.The absence of evaluation was intentional to maximize exposure to training data and adaptation to new instruction patterns.
6 Evaluation
Diffutron is evaluated through intrinsic language-modeling analysis and selected Turkish NLP benchmarks. Continued pre-training lowers perplexity, while the 307-million-parameter model achieves a higher average benchmark score than the larger listed baselines.
- 6 Evaluation: The evaluation combines intrinsic language-modeling analysis with comparative downstream testing across Turkish NLP benchmarks.The downstream evaluation uses selected parts of CETVEL because the full suite was computationally expensive and time-consuming.
- 6.1 Language Modeling Analysis: Perplexity is measured on the Bilkent Turkish Writings Dataset to assess fluency and adaptation to Turkish linguistic structures.Evaluation uses a maximum sequence length of 512 and an MLM probability of 0.15.
- 6.1 Language Modeling Analysis: 3.42 to 2.75: continued pre-training reduces perplexity from jhu-clsp/mmBERT-base to DiffutronLM-0.3B-Base.The reported decrease is interpreted as improved predictive capability and reduced uncertainty relative to the target language distribution.
- 6.2 Downstream Task Performance: 34.68 average score: Diffutron (2nd Stage) surpasses Kumru-2B (34.09) and TURNA (33.19) despite having 307 million parameters.The progression from the first to second stage also shows consistent improvements on semantic tasks such as News Classification and STS_TR.
Limitations
Diffutron is constrained by limitations in Turkish NLP resources, available computation, context length, and evaluation coverage. These constraints affect its representational quality, cultural and linguistic nuance, long-form applicability, and benchmark comprehensiveness.
- A lack of modern native Turkish encoder-only foundation models required using a multilingual backbone, potentially limiting representational quality.
- Scarce high-quality native Turkish instruction datasets may limit the model’s capture of complex cultural and linguistic nuances.Existing resources often rely on translated or synthetic data.
- The inherited 256-token context window restricts applicability to long-form generation and summarization tasks.
- Computational constraints limited evaluation to a representative subset of the CETVEL benchmark rather than its full suite.
Conclusion
Diffutron demonstrates that a compact masked diffusion model can provide robust Turkish linguistic capabilities while using far fewer parameters than larger autoregressive baselines. The authors also release the models and pre-training dataset to support further research.
- Diffutron shows that masked diffusion modeling can provide robust linguistic capabilities for Turkish with parameter efficiency.The conclusion frames this as an alternative to the traditional autoregressive framework for morphologically rich languages.
- The authors published Diffutron models and the pre-training dataset on Hugging Face for further research and development.The released models include Base, 1st-Stage, and Instruct variants, alongside the pre-training corpus.
- The work challenges the view that massive scale is the only path to competence in complex languages and highlights diffusion architectures as viable alternatives.
A Generation Examples
The qualitative examples use two LLaDAGeneratorConfig settings to demonstrate generation across varying lengths. The outputs cover factual answers, comparisons, explanations, and short creative writing in Turkish.
- Two LLaDAGeneratorConfig settings are used to demonstrate Diffutron’s qualitative performance across varying generation lengths.Table 5 is identified as containing the generation configurations and common parameters for both settings.
- For real-time weather requests, the response states that the language model lacks real-time information access and recommends external weather sources.
- The model also generates a short story and discusses benefits of reading, illustrating both creative and explanatory instruction-following outputs.