Pushing on Multilingual Reasoning Models with Language-Mixed Chain-of-Thought

• The work targets a gap left by reasoning pipelines that overwhelmingly focus on English and Chinese rather than language-specific reasoning in mid-resource languages.Translated corpora can introduce artifacts and weak coverage of natural colloquial expressions.Source: §1 · p. 1
• KO-REAson-35B outperforms comparable closed-data reasoning models on average across nine tasks.It ranks first on five tasks and second on the remaining four.Source: §1 · p. 2
• YI-SANG contains 5.79M Korean prompts and 3.7M long reasoning traces, plus a 260k high-yield subset distilled through extensive ablations.The dataset supports training models from 4B to 35B across multiple model families.Source: §1 · p. 2
• Language-Mixed CoT switches between English as an anchor and Korean as the target during reasoning, outperforming monolingual CoT baselines.The scheme is designed to combine English reasoning capability with Korean semantic preservation.Source: §1 · p. 2
• The released KO-REAson series spans 4B–35B models and is reported to surpass comparable closed systems on nine benchmarks.The models and datasets are released under the Apache-2.0 license.Source: §1 · p. 2

• Successful cold-start online RL generally requires strong base models, reliable reward or process models, and large-scale high-quality data, increasing cost and brittleness.These requirements concentrate progress in high-resource languages such as English and Chinese.Source: §2 · p. 3
• Translated OpenThoughts-114K fine-tuning improves MATH performance while degrading HAE-RAE Bench performance.The result exposes a trade-off between mathematical reasoning and Korean cultural competence.Source: Table 1 · p. 3, §2 · p. 3
• Mid-resource language reasoning remains understudied because weaker base models and limited high-quality data make direct replication of high-resource pipelines difficult.Existing approaches include English training, translated datasets, and objectives that bring non-English representations closer to English.Source: §2 · p. 3

3.1 TRAINING DETAILS

• YI-SANG is reported as larger than existing Korean fine-tuning datasets and pretraining corpora, with 6.77B tokens.Source: Figure 2 · p. 3
• The study evaluates instruction-tuned Gemma, A.X, Kanana, Llama, KONI-Llama, and Qwen models spanning approximately 4B–35B parameters.Ablations use Gemma-3-4B and Kanana-1.5-8B, while broader evaluation covers nine models.Source: §3.1 · pp. 3–4
• Training runs use at least 50,000 data points and generally five epochs, except A.X-3.1-35B, which uses three epochs because of compute constraints.Source: §3.1 · p. 4

3.2 EVALUATION DETAILS

• Held-in benchmarks support monitoring and ablations, while held-out benchmarks are evaluated only after final training to reduce inadvertent benchmark overfitting.Held-in tests include MCLM, KMMLU-Redux, HAE-RAE Bench, and ClinicalQA; held-out tests include KMMLU-Hard, KMMLU-Pro, KSM, CLIcK, and KoBALT-700.Source: §3.2 · p. 4
• Main experiments use temperature 0.7, top p 0.9, and up to 32,768 tokens, with three independent trials reported as mean ± standard error.Boxed answers are validated with math-verify 2, and invalid outputs are marked incorrect.Source: §3.2 · p. 4

4 LANGUAGE-MIXED CHAIN-OF-THOUGHT

• English-only reasoning can introduce translation noise and topic drift, whereas Korean-only reasoning reduces reasoning capability and may cause distributional drift in English-pretrained bases.Source: §4 · p. 4
• Language-Mixed CoT outperforms English- or Korean-only CoT for both Gemma-3-4B and Kanana-1.5-8B across the reported evaluations.The comparison covers HAE-RAE Bench and KMMLU-Redux among the evaluated metrics.Source: Table 2 · p. 5, §4 · p. 5
• Language-Mixed CoT lets models code-switch during reasoning, using English for logical scaffolding while preserving Korean terms and quotations.The method filters generated traces to retain a Korean-character ratio between 5% and 20%.Source: §4 · p. 5

5.1 INSTRUCTION COLLECTION

• The instruction pipeline collects native Korean prompts from 54 public Q&A and community sites, excluding legally restricted, low-volume, obfuscated, and near-duplicate sources.Data from sites permitting crawling but restricting redistribution is used for training and analysis but not redistributed.Source: §5.1 · p. 5
• Prompts are kept verbatim to preserve typos, abbreviations, mixed script, and internet style, with only light length and Korean-character filters.Source: §5.1 · pp. 5–6
• YI-SANG supplements web data with translated OpenThought prompts because the web-sourced mix lacks competition-level problems associated with reasoning ability.Earlier translation attempts with GPT-4o-mini, Qwen2.5-72B-Instruct, and Gemini-2.0-Flash produced training instabilities.Source: §5.1 · p. 6

5.2 RESPONSE GENERATION

• Qwen3-32B with language-mixed CoT provides the strongest teacher setting, while explicit reasoning outperforms short-CoT alternatives and either augmentation remains beneficial.Option and Style augmentation are comparable on Gemma-3-4B, while Option is stronger on Kanana-1.5-8B.Source: §5.2 · p. 7, Figure 4 · p. 7
• Qwen3-32B regenerates all targets from prompts alone using Language-Mixed CoT, producing 3.7M long-reasoning trajectories after filtering and Korean-ratio enforcement.The approach avoids relying on web-collected answers as an oracle.Source: §5.3 · p. 7, §5.2 · p. 7

5.3 DATASET COMPOSITION

• OpenThought and Exams provide the largest category gains, followed by Code and Science, whereas Medical improves ClinicalQA but harms other benchmarks.The final mixture therefore uses OpenThought, Code, Exams, and Science, totaling about 1.8M instances.Source: §5.3 · pp. 7–8, Table 3 · p. 8
• The finalization process removes repeated responses, multiple reasoning blocks, non-Korean final answers, and traces exceeding 16k tokens after investigating loss spikes.A one-epoch proxy-model shakedown is repeated until the loss curve stabilizes.Source: §5.3 · p. 8
• 13-gram decontamination removes about 0.7% of trajectories, leaving a 260k-instance YI-SANG-HQ corpus composed of OpenThought, Code, Science, and Exams examples.Source: §5.3 · p. 8

6 RESULTS

• Across nine benchmarks, KO-REAson-35B achieves the best performance on five tasks, ranks second on the remaining four, and records the highest overall average among comparable reasoning models.It trails GPT-OSS-20B on AIME2024 and KSM but ranks second on both benchmarks.Source: §6 · p. 8
• Competition-level mathematics remains a relative weakness: KO-REAson-35B trails GPT-OSS-20B on AIME2024 and KSM while ranking second on both.The paper attributes this to only about 60k translated competition-style OpenThought items versus much larger competition-focused datasets in prior work.Source: §6 · pp. 8–9
• YI-SANG-HQ improves performance consistently across nine models spanning 4B–35B parameters and six model families, especially on math-intensive, Korean-specific, and general-knowledge benchmarks.Only two cases show degradation, and both drops are smaller than two points.Source: §6 · p. 9, Table 5 · p. 9
• KO-REAson-12B outperforms its Gemma3-12B base on English AIME-2025 and GPQA and on Korean KAIO-2 and HAERAE-Vision evaluations.The transfer is strongest on reasoning-heavy tasks and limited on shallow factoid-style evaluations.Source: §6 · p. 10

7 CONCLUSION

• YI-SANG provides 5.9M native-authored Korean prompts for reasoning-model training, with its high-yield subset YI-SANG-HQ delivering consistent gains across six model families.Source: §7 · p. 10

A.1 ORIGIN

• The dataset name references Yi Sang’s mathematically inflected Korean literature and the Korean noun meaning “the most complete state.”Source: §A.1 · p. 16

A.2 PROMPTS

• The paper keeps the generation prompt simple because longer instructions may constrain stylistic diversity, and releases YI-SANG and YI-SANG-HQ under the MIT License.Source: §A.2 · p. 16, §A.3 · p. 16

A.4 ADDITIONAL ABLATIONS

• The same distillation procedure benefits students using either DeepSeek-R1-32B or Qwen3-32B, while student performance remains bounded by teacher capability.Source: §A.4 · p. 16, Table 7 · p. 17
• Doubling the Medical subset from 50k to 100k produces negative performance effects, supporting its exclusion from the final mixture.Source: §A.4 · p. 17, Table 8 · p. 17
• Scaling the Daily subset consistently reduces overall performance, so Daily is also excluded from the final training composition.Source: §A.4 · p. 17, Table 9 · p. 17
• MCLM-Ko rises from 55.0 to 67.4 after adding 500k English OpenThought samples, but Korean-specific benchmarks decline, so the larger mixture is excluded.Source: Table 10 · p. 17, §A.4 · p. 18

A.5 ABLATION DETAILS

• Style and option augmentation show no single consistently superior method in the compared results.Source: Table 11 · p. 18
• Qwen3-32B long chain-of-thought supervision performs best among the compared teacher and response-format choices, with the clearest caps on MCLM.Source: Table 12 · p. 18

B.1 MODELS

• The model set includes multilingual, Korean-focused, and bilingual families such as Gemma-3, Qwen-2.5, A.X-3.1, Kanana-1.5, Llama-3.1, and KONI-Llama-3.1.Source: §B.1 · pp. 18–19

B.2 HYPERPARAMETERS

• Training uses bfloat16 AdamW optimization with a 2 × 10^-5 learning rate, 10% warmup, FSDP, Liger kernels, and FlashAttention-2; final YI-SANG-HQ runs use batch size 512.Source: §B.2 · p. 19

B.3 PACKING

• Although packing substantially improves speed, it causes measurable drops on general-knowledge and reasoning benchmarks, so reported models omit packing.Source: §B.3 · p. 19, Table 13 · p. 19

C.1 PROMPTS

• The evaluation prompt instructs models to present final answers inside boxed notation, as illustrated by Figure 7.Source: §C.1 · p. 19, Figure 7 · p. 19

C.2 PROCESSING DETAILS

• Only the first boxed answer after hidden reasoning is evaluated; boxed strings inside the think section and later visible answers are ignored.Source: §C.2 · p. 20
• Responses without a parsable post-think boxed answer are marked incorrect, while interrupted generations are retried once under the same decoding settings.Source: §C.2 · p. 20

• Table 14 reports consistent YI-SANG-HQ benefits across nine 4B–35B models and the AIME24, AIME25, and GPQA benchmarks.Source: Table 14 · p. 20
• Every model improves on English benchmarks including AIME2024/2025 and GPQA, although gains do not yet match similarly scaled state-of-the-art systems.Source: §D.1 · p. 20
• The authors link transfer to universal math and science knowledge and to Language-Mixed CoT preserving alignment with models’ original English distribution.Source: §D.1 · p. 20

D.2 CROSS-MODAL GAINS ON VISUAL LANGUAGE BENCHMARKS

• KO-REAson-12B gains on reasoning-oriented HAERAE-Vision and KAIO-2, while K-Viscuit shows little to no benefit.The model has a visual encoder but receives only YI-SANG-HQ text reasoning training.Source: §D.2 · p. 21, Figure 8 · p. 21

D.3 IMPORTANCE OF HELD-IN BENCHMARKS AS PRACTICAL PROXIES.

• Held-in benchmarks show larger gains than held-out benchmarks, with gains generally smaller at higher baseline scores.Figure 9 compares baseline scores with post-training improvements for held-in and held-out benchmark pairs.Source: §D.3 · p. 21, Figure 9 · p. 21
• Despite mild contamination risk from using held-in benchmarks during subset refinement, YI-SANG-HQ models improve consistently on unseen held-out targets.The authors retain held-in and held-out distinctions as essential for fair evaluation.Source: §D.3 · p. 21