PaperlayerView exact revision on arXiv

Source-linked AI summary

KORMo: Korean Open Reasoning Model for Everyone

Minjun Kim, Hyeonseok Lim, Hangyeol Yoo, Inho Won, Seungwoo Song, Minkyung Cho, Junhun Yuk, Changsu Choi, Dongjae Shin, Huige Lee, Hoyun Song, Alice Oh, Kyungtae Lim

arXiv:2510.09426v1cs.CL

TL;DR

Fully open model development remains largely focused on English, and non-English efforts face data and curation challenges as well as incomplete disclosure that limits reproducibility. This paper builds and fully documents a Korean-English model trained predominantly on synthetic data. It reports competitive English reasoning and stronger Korean instruction-following and logical coherence, while noting that Korean long-context retrieval remains unsettled.

  • Problem

    Fully open model development remains predominantly English-focused, while non-English work faces data and curation challenges and open-weight releases often omit the training pipeline needed for reproducibility.

    Source: §1 · pp. 1–2

  • Method

    The authors train a Korean-English model from scratch using a 150-billion-token curriculum with 68.73% synthetic data in the Korean corpus, and release the model and training pipeline.

    Source: §1 · pp. 2–3

  • Results

    KORMo maintains competitive English reasoning while achieving superior Korean instruction-following and logical coherence, with stable convergence and multilingual generalization reported across evaluations.

    Source: §8 · p. 33

  • Takeaways & Limitations

    The authors conclude that carefully balanced and stylistically diverse synthetic data can serve as a scalable, sustainable resource for non-English fully open models and reproducible multilingual research.

    Source: §8 · p. 33

    Stable information retrieval in long Korean contexts is not yet fully established, with tokenizer instability, synthetic-data imbalance, and dataset domain bias identified as possible factors.

    Source: §5.1 · p. 22

Abstract

from arXiv · show

This work presents the first large-scale investigation into constructing a fully open bilingual large language model (LLM) for a non-English language, specifically Korean, trained predominantly on synthetic data. We introduce KORMo-10B, a 10.8B-parameter model trained from scratch on a Korean-English corpus in which 68.74% of the Korean portion is synthetic. Through systematic experimentation, we demonstrate that synthetic data, when carefully curated with balanced linguistic coverage and diverse instruction styles, does not cause instability or degradation during large-scale pretraining. Furthermore, the model achieves performance comparable to that of contemporary open-weight multilingual baselines across a wide range of reasoning, knowledge, and instruction-following benchmarks. Our experiments reveal two key findings: (1) synthetic data can reliably sustain long-horizon pretraining without model collapse, and (2) bilingual instruction tuning enables near-native reasoning and discourse coherence in Korean. By fully releasing all components including data, code, training recipes, and logs, this work establishes a transparent framework for developing synthetic data-driven fully open models (FOMs) in low-resource settings and sets a reproducible precedent for future multilingual LLM research.

1 Introduction

Fully open model development remains concentrated on English, while non-English efforts face data, curation, and design challenges; synthetic data offers a potential resource but requires validation. This work investigates a Korean-English fully open model built predominantly with synthetic data and reports its feasibility alongside release of the full training pipeline.

1 Introduction

  • Open-weight releases often disclose only final parameters, leaving training data, preprocessing, code, hyperparameters, and logs undisclosed and weakening reproducibility and fair comparison.The paper positions full training-pipeline disclosure as a way to support reproducible research and derivative-model development.Source: §1 · pp. 1–2
  • Fully open models remain predominantly English-focused, while non-English development faces limited corpora, licensing issues, costly curation, and tokenizer and language-composition ambiguities.The authors argue that releasing a foundational model for a specific language can lower research barriers and accelerate ecosystem development.Source: §1 · p. 2
  • Synthetic data may help address data constraints, but concerns about model collapse and possible bias make quantitative validation of stability and bias important.Tokenizer and language-mixture choices also affect compression efficiency, training cost, and downstream generalization, particularly for non-Latin and morphologically rich languages.Source: §1 · p. 2
  • The study builds a Korean-English model from scratch using a 150-billion-token curriculum, with synthetic data comprising 68.73% of the Korean corpus.Synthetic data was produced using Qwen and open-source model families, and the curriculum covered pretraining, supervised fine-tuning, and preference learning.Source: §1 · p. 2
  • A 1B-model, 60B-token proxy study compares fully synthetic with fully non-synthetic data while testing stabilization methods, curricula, tokenizers, and language-mixture ratios.After stability and efficiency are confirmed, the work proceeds to the 10.8B-scale model and evaluates knowledge, reasoning, reading comprehension, common sense, and mathematics benchmarks.Source: §1 · pp. 2–3
  • The paper reports feasibility of a non-English fully open model with majority synthetic data and releases a 10.8B Korean-English model, data, code, recipes, and logs.It also examines tokenizer settings, language mixture ratios, and training curricula for stability, efficiency, and generalization.Source: §1 · p. 3

2 Exploring Training Design Choices

The study uses proxy experiments to examine synthetic-data training and core design choices before scaling up. Across those experiments, Intra-doc masking and Pre-LN were selected, while NTP outperformed MTP overall in the 1B-scale comparison.

2 Exploring Training Design Choices

  • The proxy study examined synthetic-data performance and stability alongside tokenizer composition, with exploratory results informing the final KORMo design.The final design includes Pre-LN and RMSNorm, GQA and MQA, SwiGLU, RoPE, and document packing.Source: §2 · p. 3

2.1 Experiment Settings

  • The 1B proxy compared 60B-token UltraFineWeb and Nemotron-CC English runs, using relative trends for scale-up and a mostly 5-shot MCQA suite.TruthfulQA used 0-shot evaluation; the authors note that broader comparisons across synthetic sources would be ideal but were constrained by data-construction and training costs.Source: §2.1 · p. 4

2.2 Normalization Methods

  • 43.38% vs. 41.28%: Pre-LN outperformed MixLN and was adopted as the default normalization method.With Pre-LN, synthetic-data training showed no performance degradation or increase in loss spikes compared with non-synthetic training.Source: Table 2 · p. 5, §2.2 · p. 5

2.3 Attention Masking Method

  • 44.48% vs. 43.38%: Intra-doc masking outperformed standard causal masking and was adopted for the final model.Under the same masking setting, synthetic-data training performed better with almost no difference in learning curve or loss volatility; this experiment used English only.Source: §2.3 · pp. 5–6, Table 3 · p. 6

2.4 Multi-Token Prediction

  • 41.35 vs. 43.38 (approx. –2.0%p): MTP had lower average performance than NTP under identical pretraining conditions.MTP nevertheless outperformed NTP on BoolQ (+2.3%p), OBQA (+4.6%p), TFQA_mc1 (+5.8%p), and TFQA_mc2 (+8.9%p), while NTP led on ARC-E, PIQA, and RACE.Source: Table 4 · p. 6, §2.4 · pp. 6–7
  • Under small-model and limited-data conditions, the authors retained NTP and conducted no additional MTP experiments on augmented-data models.They identify the 1B model's capacity and the 60B-token corpus as constraints on MTP's observed performance.Source: §2.4 · p. 7

3 Proposed Tokenizer based on Mixture of Datasets

The section studies bilingual tokenizer design, balancing compression efficiency with downstream performance. Its findings show that synthetic–crawl mixtures and vocabulary choices affect English and Korean differently, and identify EPK-125K as the final tokenizer.

3.1 Experiments Settings for Building Tokenizer from Scratch

  • The study trains byte-level BPE tokenizers on 20 GB samples and evaluates compression with BPT alongside downstream performance on Korean and English benchmarks.The evaluation includes web-crawled and synthetic RLHF data; higher BPT indicates more efficient compression.Source: §3.1 · p. 7, Table 5 · p. 8

3.2 Impact of Synthetic Data on Tokenizer Training

  • Compression improved on average as synthetic data increased in both languages, but comparable Korean compression required around 80% crawled data versus approximately 60% synthetic data in English.The authors suggest the distributional gap between synthetic and crawled data is more pronounced in Korean.Source: Figure 1 · p. 8, Figure 2 · p. 8, §3.2 · pp. 8–9

3 Proposed Tokenizer based on Mixture of Datasets

  • The tokenizer candidates encode language-specific synthetic–crawled ratios, an all-synthetic option, and an option adding code data, each at 125K and 196K vocabulary sizes.The design also considers compression, domain weaknesses, and downstream performance.Source: §3 · p. 7, §3.2 · p. 9, Table 6 · p. 9

3.3 Compression Comparison in Bilingual Settings with Commercial Tokenizers

  • Proposed tokenizers matched commercial models in English compression and substantially outperformed them in Korean; larger vocabularies generally improved compression by approximately 2–7%.The authors note that larger vocabularies also trade off against model parameters and memory use.Source: Figure 3 · p. 9, §3.3 · pp. 9–10

3.4 Downstream Model Performance with Diverse Tokenizers

  • EK 125K achieved the best overall English downstream performance, while EPK 125K ranked highest in Korean; vocabulary expansion often reduced English performance.The English comparison also reports a few exceptions, including EK-with-Crawl.Source: Table 7 · p. 10, Table 8 · p. 10, §3.4 · p. 10

3.5 Safety of KORMo Tokenizers

  • EPK-125K was selected as the final tokenizer after proposed candidates showed comparatively fewer harmful tokens; crawled data tended to increase their number.English-centric models had minimal Korean vocabulary coverage, limiting detection of harmful Korean tokens under the study’s counting criteria.Source: Table 9 · p. 11, §3.5 · pp. 11–12, Figure 4 · p. 11

4 Pretraining Phase

This section introduces the approach to collecting and generating pretraining data and constructing training stages that account for data quality and difficulty.

4 Pretraining Phase

  • The section frames pretraining data collection, synthetic generation, and training-stage construction around data quality and difficulty.The passage introduces the approach without reporting procedures or results.Source: §4 · p. 12

4.1 Pretraining Datasets

KORMo’s pretraining corpus combines directly collected and synthetic data from general, specialized, and auxiliary sources. The section describes Korean data scarcity, synthetic augmentation strategies, and filtering choices, including deduplication and quality-classifier results.

4.1 Pretraining Datasets

  • Pretraining data came from direct collection and synthetic generation, spanning general web text, domain-specific materials, and auxiliary datasets.The English sources include filtered public corpora and datasets for mathematics, coding, and reasoning.Source: §4.1 · p. 12, Table 10 · p. 12, §4.1.1 · p. 13

4.1.1 Public Data Collection

  • Korean data came from institutional and community resources and recent Common Crawl dumps; the study notes that publicly available Korean pretraining resources are extremely limited.The crawl-language filter used fastText LID-176 with a 0.8 threshold; only four 2025 subsets were used due to time constraints.Source: §4.1.1 · p. 13

4.1.2 Synthetic Data Generation

  • To transfer knowledge from English seed corpora while addressing Korean-specific contexts, the authors generated five augmentation datasets using varied sources and prompt strategies.The designs vary across translations, custom Korean prompts, and paired English queries with Korean reasoning traces and answers.Source: §4.1.2 · pp. 13–14
  • In a proxy evaluation using 4B tokens per augmented dataset, Kosmopedia performed strongest, and most other augmentations also outperformed standard Korean web corpora.The synthetic data drew on five distinct English sources.Source: §4.1.2 · p. 14

4.1.3 Data Filtering

  • The filtering pipeline combines heuristic rules, deduplication, and quality filtering; its Korean quality classifiers were trained on 300K positive and 300K negative samples per version.The selected Version 2 classifier was applied to refine the Korean pretraining corpus.Source: §4.1.3 · pp. 14, 16, 17
  • Stricter Old-both deduplication consistently outperformed Document deduplication across evaluation metrics, despite removing approximately 70% of the Korean corpus.The authors retained Old-both because its performance gains were clear and consistent.Source: Table 11 · p. 15, §4.1.3 · p. 15
  • Version 2’s random web negatives outperformed score-0 negatives, while versions with curated high-quality positives outperformed versions with more synthetic data.The authors nevertheless considered synthetic data necessary to reach the trillions of tokens required for large-scale pretraining, given the scarcity of high-quality Korean web data.Source: Table 12 · p. 16, §4.1.3 · p. 17

4.2 Pretraining

KORMo’s pretraining design combines a searched learning rate with a two-stage curriculum and a Korean share above 5%. Performance rose in both languages across stages, though Korean evaluation fluctuated more and additional pretraining beyond 2.8T tokens was judged less cost-effective than later training.

4.2.1 Optimal Learning Rate Search

  • 7e-4 produced the lowest overall loss across the search steps and was selected for final training as a stable, efficient setting.High rates such as 3e-3 oscillated, while 1e-4 converged stably but too slowly.Source: §4.2.1 · p. 17, Figure 6 · p. 18

4.2.2 Proposed Language Ratio and Pre-training Stages

  • The Korean data proportion was set above 5% to support stable bilingual learning given English–Korean structural differences.The section frames language ratio as a determinant of bilingual model performance.Source: §4.2.2 · p. 18
  • KORMo used two pretraining stages: roughly 1T tokens of primarily lower-quality web data, followed by 1.8T tokens of higher-quality web, synthetic, and low-difficulty reasoning data.The second stage comprised 1.7T English and 0.1T Korean tokens.Source: §4.2.2 · pp. 18–19
  • Training used a Pre-LN decoder, intra-document masking, next-token prediction, and 4,096-token sequences, with reasoning samples left unpacked to preserve connected chains.The optimizer was reinitialized at each stage, with a warm-up of 0.03% of total steps.Source: §4.2.2 · p. 19
  • Stage 2 raised Korean average evaluation above 0.55, after both languages improved sharply in Stage 1 and more gradually in Stage 2.Korean’s curve oscillated more, attributed to its lower corpus share and fewer evaluation benchmarks; gains beyond 2.8T tokens were judged less cost-effective than allocating compute to mid- or post-training.Source: §4.2.2 · p. 20

4.3 Impact of Augmented Data Diversity

A Stage 2 run using Korean synthetic data from a single synthesizer degraded in both languages and lost the Stage 1 performance peaks. The authors attribute the failure to inadequate diversity in synthetic models, seeds, and prompts.

4.3 Impact of Augmented Data Diversity

  • The single-synthesizer Stage 2 run degraded in both languages, with English fluctuating between 0.55–0.63 and Korean falling toward early Stage 1 levels.The authors interpret the abrupt, irreversible decline as model collapse.Source: §4.3 · p. 20, Figure 8 · p. 20
  • Using one model to generate all Korean synthetic data may repeat its biases and errors, while limited seeds and prompts narrow the data distribution and risk overfitting.The authors connect narrow synthetic coverage with loss of broad knowledge, language understanding, and reasoning abilities.Source: §4.3 · pp. 20–21
  • The authors conclude that large-scale bilingual training needs diverse augmented data, not merely a large quantity, to avoid severe performance degradation.The failure comparison contrasts a single-synthesizer dataset with data generated using multiple synthesizers.Source: §4.3 · pp. 20–21

5 Mid-training

KORMo’s mid-training extends context and adds reasoning supervision, with the stages producing different strengths. English retrieval remained strong at long lengths, while Korean retrieval weakened beyond 13K tokens; benchmark results show foundational gains from Stage 2 and specialized gains from Long and Reason stages.

5.1 Long Context Training

  • English Needle In A Haystack accuracy was 99.04% for KORMo-10B-longcontext and 98.97% for YarnX4-extended, while Korean accuracy was 69.04% and 63.15%, respectively.Korean accuracy fell below 60% beyond 21K tokens; YarnX4 showed degradation between 82K and 118K tokens.Source: Figure 9 · p. 22, §5.1 · p. 22
  • Korean long-context retrieval was not yet stable, with tokenizer instability, synthetic-data imbalance, and dataset domain bias suggested as contributing factors.The authors call for Korean-focused improvements to data design and augmentation.Source: §5.1 · p. 22

5.2 Reasoning Context Training

  • Reasoning Context Training follows long-context extension and uses large-scale reasoning traces to elicit reasoning and support internalization of advanced inference patterns.Korean reasoning data reused Ko-Reasoning and was supplemented with translated Nemotron samples because large-scale public Korean reasoning datasets were unavailable.Source: §5.2 · pp. 22–23
  • Reasoning samples were filtered by requiring two models both to answer incorrectly and label a sample high-difficulty; chat and tool-calling data were not difficulty-filtered because ground-truth labels were unavailable.The Korean dataset totaled approximately 2.8B tokens, and the <think> token was excluded from loss computation to avoid interference with its later SFT role as a reasoning trigger.Source: §5.2 · pp. 23–25

5.2.1 Experiment over Mid-training

  • Stage 2 improved English aggregate metrics by +4.45 points (60.15→64.60), while Midtrain(Reason) added +1.55 points over Stage 2 on reasoning benchmarks (58.79→60.34).Long reached high scores on several commonsense and multiple-choice tasks, but showed slight drops on some reasoning-intensive tasks.Source: Table 16 · p. 25, Table 17 · p. 25, §5.2.1 · p. 25
  • 54.12 average Korean benchmark performance was reached by Midtrain(Reasoning), following Stage 2’s largest gain of +6.88 points over Stage 1 (43.43→50.31).Long modestly improved the average and helped context-dependent tasks; Reasoning improved high-difficulty reasoning and knowledge-intensive tasks.Source: Table 18 · p. 26, §5.2.1 · p. 26

6 Post-training

Post-training comprises supervised fine-tuning and preference learning. SFT uses filtered data and distinct reasoning-enhanced and hybrid variants, while preference datasets were prepared but not yet used for training.

6 Post-training

  • Post-training comprises supervised fine-tuning and a preference-learning fine-tuning component.SFT aims to improve faithful instruction understanding and execution through Base SFT and Instruction-Following SFT phases.Source: §6 · p. 26, §6.1 · p. 27

6.1 Supervised Fine-tuning

  • SFT data was deduplicated, difficulty-balanced, and capped at 16,384 tokens, with added multi-turn, tool-calling, and linguistically diverse examples.Korean samples were balanced between reasoning-required and reasoning-not-required categories; English samples were balanced across easy, medium, and hard levels.Source: §6.1 · pp. 27–28
  • The Reasoning-Enhanced variant trained on reasoning traces for all queries, while the Hybrid variant paired easy samples with non-reasoning mode and difficult samples with reasoning mode.The Hybrid variant maintained a 1:1 balance between modes.Source: §6.1 · p. 28
  • Instruction-Following SFT targeted multi-turn coherence, instruction compliance, and consistent output formatting.Korean queries were translated and re-instructed, and the data included model-generated multi-turn interactions and Korean reasoning paths.Source: §6.1 · p. 28

6.2 Optimization on Preference Learning

  • APO and ORPO preference datasets were constructed, but training and evaluation were deferred because of GPU constraints.The APO dataset used 100K prompts across chat, STEM, code, and math, with smaller-model responses treated as rejected and larger-model responses as chosen.Source: §6.2 · p. 28

7 Experiments

The experiments evaluate KORMo before reinforcement learning, using broad English and Korean benchmarks and matched instruction-tuned comparisons. KORMo-10B is competitive in English reasoning and instruction following, with Korean general reasoning and practical QA strengths but weaker specialist knowledge and fine-grained lexical discrimination.

7.4 Evaluation of Reinforcement Learning Models

  • The experiments report pre-RL results; strengthening reasoning through reinforcement learning is deferred to subsequent work.The proposed next step is to add mathematics and logical-reasoning data generated with the APO and GRPO frameworks.Source: §7.1 · p. 29, §7.4 · p. 33

7.1 Experiment settings

  • More than 26 English and Korean benchmarks covered reasoning, knowledge, and domain-specific tasks, generally using five-shot prompts and accuracy.Base models were compared before instruction tuning; instruction-following evaluation used models with the same tuning procedure and GPT-4o as judge.Source: §7.1 · p. 29, §7.2 · p. 29

7.2 Base Model Evaluation

  • KORMo’s general reasoning was strong across English and Korean, but it lagged on specialist knowledge and fine-grained lexical-semantic discrimination.English margins behind larger models were 2–7%; Korean weaknesses included KOBALT and the KMMLU suite, while KR-Clinical-QA was a relative strength.Source: §7.2 · pp. 30–32
  • 64.2 on English benchmarks put KORMo-10B near OLMo2-13B’s 65.3, while its Korean score of 58.2 was 4–8% below several multilingual models.The authors note the result came from 2.9T pretraining tokens; the table reports accuracy and training-token counts.Source: §7.2 · p. 30, Table 21 · p. 31
  • KORMo’s English–Korean average-score gap was around 6 points, while Korean-optimized KANANA’s was about 1 point but with relatively limited English performance.KORMo scored 1.84 points higher in English and 2.79 points lower in Korean than KANANA-1.5; its Korean data ratio was estimated at 5.6%, versus KANANA’s reported 10%.Source: §7.2 · p. 31

7.3 Evaluation of SFT Models

  • 8.61 was KORMo-10B’s highest average MT-Bench score, compared with 8.56 for Gemma3-12B and 8.50 for Qwen3-8B.The comparison uses instruction-tuned models before reinforcement learning; evaluations were automatically scored using GPT-4o.Source: §7.3 · p. 32, Table 22 · p. 32
  • KORMo scored 8.75 on Korean instruction benchmarks versus 8.32 on English MT-Bench, which the authors attribute to its larger Korean instruction-data volume.The instruction-following comparisons cover KO-MT-Bench and Logickor alongside MT-Bench.Source: §7.3 · p. 32
  • Instruction-only evaluation found Korean conversational and logical-inference robustness, with the authors reporting an 8.6 MT-Bench score without reinforcement learning.The planned RL stage is intended to extend reasoning chains and reinforce multi-hop consistency.Source: §7.3 · pp. 32–33, §7.4 · p. 33

8 Conclusion

KORMo is a fully open Korean–English model built primarily with synthetic data. Across pretraining and instruction tuning, it shows stable convergence and competitive multilingual reasoning, while its benchmark results indicate remaining room in fine-grained expert knowledge.

8 Conclusion

  • The paper reports that carefully balanced, stylistically diverse synthetic corpora can support stable convergence and multilingual generalization across pretraining and instruction tuning.KORMo is presented as a fully open bilingual Korean–English LLM developed primarily with synthetic data.Source: §8 · p. 33
  • KORMo combines competitive English reasoning with stronger Korean instruction following and logical coherence, while task-specific results show room to improve fine-grained expert knowledge.The conclusion also identifies balanced bilingual compression as improving token efficiency and stability.Source: §8 · p. 33
  • KORMo publicly releases its data and training process, enabling the community to extend and audit components of the LLM lifecycle.Future work includes reasoning-oriented reinforcement learning, generalization beyond Korean–English, and standardized evaluation suites.Source: §8 · p. 33

Related papers

Loading 2510.09426v1…