PaperlayerView exact revision on arXiv

Source-linked AI summary

Seed-X: Building Strong Multilingual Translation LLM with 7B Parameters

Shanbo Cheng, Yu Bao, Qian Cao, Luyang Huang, Liyan Kang, Zhicheng Liu, Yu Lu, Wenhao Zhu, Jingwen Chen, Zhichao Huang, Tao Li, Yifu Li, Huiying Lin, Sitong Liu, Ningxin Peng, Shuaijie She, Lu Xu, Nuo Xu, Sen Yang, Runsheng Yu, Yiming Yu, Liehao Zou, Hang Li, Lu Lu, Yuxuan Wang, Yonghui Wu

arXiv:2507.13618v4cs.CLcs.AI

TL;DR

LLM translation still struggles with idioms, slang, and natural culturally appropriate expression, while open-source models face a performance gap against proprietary systems. Seed-X combines multilingual pretraining with chain-of-thought instruction tuning and reinforcement learning; its 7B models achieve translation quality comparable to or better than leading systems in automatic and human evaluations.

  • Problem

    LLM translation remains challenging for idioms, slang, and culturally natural expression, and a performance gap persists between proprietary and open-source models.

    Source: §1 · p. 2

  • Method

    Seed-X pretrains a 7B open-source model on monolingual and bilingual data across 28 languages, then uses chain-of-thought instruction tuning and reinforcement learning for translation.

    Source: §1 · p. 2

  • Results

    With 7B parameters, Seed-X achieves translation quality comparable to or better than leading LLMs and commercial translation systems in automatic and human evaluations.

    Source: §7 · p. 13

  • Takeaways & Limitations

    The publicly available Seed-X weights are intended as an accessible translation tool to support translation research.

    Source: §7 · p. 13

    The limited model size entails trade-offs: adding languages degrades some core capabilities, and stage II parallel training significantly reduces English reasoning performance.

    Source: §5.4.2 · p. 12

Abstract

from arXiv · show

Multilingual translation stands as a challenging task for large language models (LLMs) to handle intricate language patterns and stilted translations that arise in automated translations. In this paper, we introduce Seed-X, a family of open-source LLMs comprising instruct and reasoning models, pushing the limits of translation capability with 7B parameter size. The base model is pre-trained on a diverse, high-quality dataset encompassing both monolingual and bilingual content across 28 languages, harnessing the full potential of multilingual data. The instruct model is then finetuned to translate by Chain-of-Thought (CoT) reasoning and further enhanced through reinforcement learning (RL) to achieve better generalization across diverse language pairs. Seed-X achieves performance comparable to leading closed-source models, including Gemini-2.5 and GPT-4o, across 28 languages, and significantly outperforms larger open-source models in both automatic metrics and human evaluations. We share the best practices through our optimization process, and make the parameter public available for advancing translation research and applications.

1 Introduction

Seed-X is a 7B-parameter open-source translation model family designed to address difficult multilingual translation and the performance gap between proprietary and open-source systems. It combines multilingual pretraining with reasoning-based instruction tuning and reinforcement learning, and reports competitive results against both larger open models and leading closed systems.

1 Introduction

  • The paper identifies unresolved challenges in idioms, slang, and internet buzzwords, alongside a persistent performance gap between proprietary and open-source models.Natural, culturally appropriate translation requires more than word-for-word mapping.Source: §1 · p. 2
  • Seed-X comprises 7B-parameter instruct and RL models plus a reward model, pretrained from scratch on monolingual and parallel data across 28 languages.The instruct model learns chain-of-thought translation for high-resource languages before reinforcement learning extends it across a broader language spectrum.Source: §1 · p. 2
  • Seed-X outperforms Google-Translator and translation-specialized peers, including Gemma3-27B, Llama4-Scout-17B-16E, and Qwen3-235B-A22B, while human evaluation matches or exceeds GPT-4o, Claude-3.5, DeepSeek-R1, and Gemini-2.5-Pro.Figure 3 covers human evaluation from 0–4 for Chinese/English translation to seven languages.Source: §1 · p. 2, Figure 3 · p. 2
  • The paper contributes a challenging English/Chinese-to-X test set covering idioms, proverbs, poetry, internet slang, and abbreviations, with annotated translation keys.It also presents a detailed training recipe and insights into multilingual knowledge transfer.Source: §1 · pp. 2–3

2 Pre-training

Seed-X pretraining combines monolingual and bilingual data through progressive multilingual training stages. The evaluation reports strong multilingual comprehension and cultural knowledge, with a relative weakness in mathematical reasoning.

2 Pre-training

  • The pretraining dataset combines monolingual and bilingual corpora to support multilingual modeling and cross-language semantic alignment.The paper evaluates multilingual capabilities after stage S2, emphasizing knowledge and comprehension rather than mathematical or logical reasoning assessments.Source: §2 · p. 3, §2.4 · p. 4

2.1 Monolingual Data

  • Monolingual data totals 6 trillion tokens; high-, medium-, and low-quality documents are respectively retained, paraphrased, or excluded.The pipeline also tags documents and balances topic distribution while reducing redundancy.Source: §2.1 · p. 3, Table 1 · p. 3

2.2 Bilingual Data

  • Bilingual data is built iteratively from a 200B-token seed set, using translation, rewriting, filtering, and increasingly capable models to refine quality.The process begins with an early translation model and expands the data with pseudo-parallel translations of monolingual material.Source: §2.2 · p. 3

2.3 Pre-training Stage

  • Pretraining moves from dominant-language monolingual data, to multilingual data, then to filtered parallel-only training; scaling laws guide stage hyperparameters.Chinese and English are the primary languages in the general stage; the model uses the Mistral-7B architecture with an expanded vocabulary.Source: §2.3 · p. 4

2.4 Pre-training Evaluation

  • Seed-X shows strong multilingual comprehension and cultural and region-specific knowledge, but relatively lower mathematical reasoning on MMMLU.Evaluation includes regional and general knowledge benchmarks and cross-lingual comprehension tasks; MMMLU uses a 10% sample and reports averages for supported languages.Source: §2.4 · p. 4, Table 2 · p. 4

3 Post-training

Seed-X post-training combines translation-focused supervised fine-tuning with preference learning, targeting strong high-resource translation and generalization to low-resource languages. Linguist-annotated reasoning data and two reward approaches support this training.

3.1 Supervised Fine-tuning

  • 236K translation-only instruction instances combine FLORES dev data with manually annotated general and business-domain pairs, then undergo quality filtering and rejection sampling.Rejection sampling particularly refines low-resource-language data, where manual annotations are scarce.Source: §3.1 · pp. 4–5
  • Linguists annotate sentence meaning, difficult linguistic elements, target-language conventions, and potential translation pitfalls to create Chain-of-Thought training data.Specific prompts guide the model to translate through this reasoning process.Source: §3.1 · p. 5

3.2 MT-oriented Preference Learning

  • Preference learning uses human-preference rewards for high-resource pairs and a reference-free dual-based reward when preference data are unavailable.The human-preference set contains approximately 20k pairs; the dual-based approach scores a translation by comparing the source with a reconstructed source after back-translation.Source: §3.2 · p. 5
  • The paper avoids automatic metric rewards because they only partially reflect translation quality and may miss subtle linguistic features and contextual appropriateness.PPO optimizes the model; the critic is initialized from the reward model to support training stability.Source: §3.2 · p. 5

4 Translation Evaluation

Seed-X is evaluated on multilingual public benchmarks and a challenging curated test set, using automatic metrics and expert assessment. Results and ablations examine its performance and the effects of training-data and prompt choices.

4.1 Settings

  • Evaluation uses FLORES-200 and WMT-25, plus a Chinese/English-to-seven-language challenge set spanning domains, styles, idioms, slang, and classical literature.Automatic evaluation uses XCOMET-XL and BLEURT, alongside expert scoring of accuracy, fluency, and idiomaticity on a 0–4 scale.Source: §4.1 · p. 6, §4.2 · p. 6

4.2 Main Results

  • Seed-X is comparable to Tier 1 and substantially outperforms Tiers 2 and 3 on automatic metrics; human evaluation places it first on EN⇒XX and second on ZH⇒XX.Table 4 covers English-centric, Chinese-centric, and XX⇒XX performance; Figure 3 evaluates seven language pairs.Source: §4.2 · p. 6, Table 4 · p. 7

4.3 Ablation

  • Arc-mm rises 25.17→30.18→32.98 and Trans. 6.97→8.19→9.41 from raw to filtered to paraphrased data in a 1.3B-model experiment.The reported results associate filtering and paraphrasing with lower training loss and improved multilingual ARC performance.Source: §4.3 · pp. 6–8
  • Reasoning and translation abilities decline as unique pretraining tags decrease, supporting the importance of dataset diversity.The diversity analysis is identified as Table 5b.Source: §4.3 · p. 8
  • Iterative LLM-based filtering identifies noisy parallel pairs better than rule-based methods, and subsequent revision improves data quality and translation performance.The study evaluates revised data at different scales and uses updated filter and rewriter models across rounds.Source: §4.3 · p. 8
  • Omitting the source language from prompts does not impair performance and improves robustness to mixed-language inputs; prompt diversity and CoT also improve translation quality.The source-language and prompt-diversity findings are reported in Tables 6 and 3, respectively.Source: §4.3 · p. 8
  • Language tags improve translation quality over other delimiters, while natural-language prompt delimiters add low-information text compared with language-aware tags.The delimiter comparison is reported in Table 7.Source: §4.3 · p. 8, Table 7 · p. 9

5 Empirical Insights

The empirical analyses examine how monolingual and parallel data, translation reasoning, and training strategy shape Seed-X’s capabilities. They also report knowledge-transfer gains alongside capability trade-offs and an overfitting risk.

5.1 Monolingual Data Shapes Core Language Capabilities

  • With 200B monolingual data, factual accuracy rises from 59.1% to 67.7%; parallel-only models can follow diverse instructions, while parallel-derived reasoning remains limited without code and mathematical data.The authors present monolingual data as a practical alternative when acquiring large-scale parallel data is challenging.Source: §5.1 · p. 9

5.2 Parallel Data Quality and Usage Matter

  • Simple word-mapping pairs can harm translation quality, while high-quality parallel data supports continued training on pure parallel data with optimized learning-rate scheduling.Over-training on pure parallel data may overfit, so the authors monitor translation performance and stop training in time.Source: §5.2 · p. 10

5.3 Translation Entails Thinking

  • Linguist-annotated Chain-of-Thought asks the model to consider source context, domain terminology, and idiomatic target-language expression, and improves translation accuracy.The example addresses a phrase whose contextual meaning differs substantially from its literal meaning.Source: §5.3 · p. 10

5.4.1 Progressive Training Strategy with Language Prioritization

  • The training strategy prioritizes English as the primary language and uses multilingual-dominant followed by parallel-only stages to reduce the risk of secondary languages being overlooked.The strategy is motivated by prior work describing multilingual inputs as converted into English for task processing.Source: §5.4.1 · p. 11

5.4.2 Knowledge Transfer from Primary to Secondary Languages

  • As parallel-data training proceeds, English-distant languages improve on knowledge examination, while Chinese understanding gains substantially in Stage II; core capabilities and English reasoning also degrade.The authors attribute the English-distant-language growth to transfer from core languages and the Chinese Stage II gains primarily to the high proportion of Chinese-English parallel data.Source: §5.4.2 · p. 12

5.5 Overfitting Risks in Multilingual Instruction Data

  • Multi-parallel data substantially impairs instruction-finetuning performance, which the authors attribute to overfitting risk from its simplified learning patterns.They avoid indiscriminately increasing directional coverage with this data and instead rely on the model’s generalization to uncovered directions.Source: §5.5 · p. 12

6 Related Work

The related-work discussion situates Seed-X among specialized language models and translation-oriented LLMs, emphasizing both model scale and evaluation breadth. It also frames multilingual balance and cross-lingual transfer as central concerns.

Specialization of LLMs

  • Specialized models can be pretrained from scratch or adapted from open-source models; the latter is cheaper but may be constrained by base-model capabilities.The authors train from scratch to establish a multilingual pretraining foundation for translation.Source: §6 · p. 12

Translation-oriented LLMs

  • Prior translation-oriented models remain behind GPT-4 in automatic evaluation, while the paper reports progress over ultra-large models on automatic metrics and human evaluation.The paper also introduces a challenging MT benchmark focused on conciseness, accuracy, and elegance.Source: §6 · pp. 12–13

Related work

  • The paper presents multilingual balancing as important for translation, particularly for low-resource languages, and explores transfer from primary to secondary languages.The discussion frames this as an efficient approach to multilingual translation capability.Source: §6 · p. 13

7 Conclusion

Seed-X is an open-source translation model family designed for bidirectional translation across 28 languages. The paper presents its public weights and training practices as resources for translation research and applications.

7 Conclusion

  • Seed-X combines pre-training, instruction tuning, and reinforcement learning, with publicly available weights intended to support translation research and applications.The family supports bidirectional translation across 28 languages and has 7B parameters.Source: §7 · p. 13

A Supported Languages

  • Table 12 lists the languages supported by Seed-X.The supplied passage identifies the table but does not specify its entries.Source: Table 12 · p. 18

B Implementation Details

The implementation uses a Mistral-7B-based Transformer, an expanded multilingual tokenizer, tuned training schedules, and beam-search decoding. Figure 6 compares SFT and PPO performance across evaluated Flores-200 translation directions.

Architecture

  • The 7B model follows Mistral-7B, with 32 decoder layers, 32 attention heads, 4,096-dimensional embeddings, and a 14,336-unit feed-forward network.It uses layer normalization, RoPE, and a 2,048-token maximum sequence length.Source: B Implementation Details · p. 18

Vocabulary

  • The tokenizer expands the vocabulary from 32,000 to 65,269 tokens, raising multilingual compression from 3.17 to 3.74 characters per token.The passage says the expansion is intended to improve handling of multilingual data and translation nuances.Source: B Implementation Details · p. 18

Training

  • Pretraining uses 2M-token batches and a 3e-4 learning rate; supervised fine-tuning uses batches of 64 sentences and a 3e-6 learning rate.Pretraining warms up for 2,000 steps before cosine decay to 10% of peak learning rate.Source: B Implementation Details · p. 18

Inference

  • Beam search with beam size 4 slightly outperforms greedy decoding, at the expense of inference speed.Source: B Implementation Details · p. 18
  • Figure 6 compares Seed-X-SFT and Seed-X-PPO across evaluated translation directions on Flores-200.Source: Figure 6 · p. 19

C Translation Performance Across Individual post-training stages

On Flores-200, the paper reports that Seed-X already performs strongly after supervised fine-tuning and that PPO further improves its translation quality.

C Translation Performance Across Individual post-training stages

  • PPO further improves Seed-X over SFT across translation directions, raising the model’s performance ceiling as measured by BLEURT.The paper describes SFT as already achieving strong translation quality.Source: C Translation Performance Across… · p. 19

D Human Evaluation

Human evaluations report strong Seed-X-PPO results across 14 language directions, with a particularly clear advantage in English-to-other-language directions. On the Chinese-English task, Seed-X outperforms most evaluated models on EN⇒ZH, trailing Deepseek-R1.

D Human Evaluation

  • Across 14 directions, Seed-X-PPO earned 6 first places and one second place, and had the highest overall average human-evaluation score.The paper also reports a clear advantage in EN⇒XX directions.Source: D Human Evaluation · p. 19, Table 13 · p. 20
  • On EN⇒ZH, Seed-X outperformed most models, trailing only Deepseek-R1 in the Chinese-English evaluation.Source: D Human Evaluation · p. 19
  • Figure 7 uses strict deductions for major and minor translation errors, so its final Chinese-English scores are negative; compare the models’ average translation ability within the arc against Seed-X.Source: Figure 7 · p. 20

E Case Study

The case studies use representative translations to illustrate Seed-X’s handling of complex expressions across internet buzzwords, Chinese slang, and Classical Chinese.

E Case Study

  • The comparative cases are presented as evidence of Seed-X’s contextual awareness and ability to handle complex translations.Table 14 compares translations across scenarios, with red text marking errors in Google Translate outputs.Source: E Case Study · p. 20, Table 14 · p. 22
  • For internet buzzwords, Seed-X renders expressions such as “绝绝子” and “YYDS” as natural praise, while Google’s version misses the intended meaning.The example also interprets “加油瘦” as referring to the speaker.Source: E Case Study · p. 21
  • For Chinese slang, Seed-X interprets “水分” as hype and “说一车” as giving many responses rather than translating them literally.Other cases gloss “撞南墙” as learning the hard way and “泰裤辣” as “awesome.”Source: E Case Study · p. 21
  • For Classical Chinese, Seed-X renders a proverb as “A good example always speaks for itself,” while Google translates its imagery literally.A separate example translates a passage about prolonged rain, howling wind, and turbulent waves.Source: E Case Study · pp. 21–22

Related papers

Loading 2507.13618v4…