PaperlayerView exact revision on arXiv

Source-linked AI summary

K-EXAONE 2.0 Technical Report

Eunbi Choi, Kibong Choi, Sehyun Chun, Seokhee Hong, Junwon Hwang, Hyojin Jeon, Ahra Jo, Hyunjik Jo, Yeonsik Jo, Minhyeok Jung, Doyoung Kim, Heegyu Kim, Joonkee Kim, Seonghwan Kim, Soyeon Kim, Sunkyoung Kim, Yireun Kim, Yongil Kim, Byungoh Ko, Changhun Lee, Dohaeng Lee, Haeju Lee, Jinsik Lee, Kyungmin Lee, Minwoo Lee, Wonkee Lee, Sangha Park, Sungjune Park, Kwangrok Ryoo, Kijung Seo, Minju Seo, Yongwoo Song, Sejong Yang, Heuiyeen Yeen, Stanley Jungkyu Choi, Yemuk Choi, Yongchan Chun, Jiwon Ham, Dasol Hong, Sujeong Im, Kijeong Jeon, Gerrard Jeongwon Jo, Hyeongjun Jo, Yujin Jo, Jiyeon Jung, Naeun Kang, Daeseong Kim, Euisoon Kim, Hayeon Kim, Hyosang Kim, Myoungshin Kim, Unsol Kim, Youchul Kim, Chaeeun Lee, ChaeYoon Lee, Edward Hwayoung Lee, Honglak Lee, Hwansoo Lee, Minkyung Lee, Sangeun Lee, Solji Lim, Woohyung Lim, Chanwoo Moon, Jueun Mun, Jimin Park, Seojeong Park, Yongmin Park, Hyerin Seo, Donghyeon Shin, Donghyun Son, Eunyong Son, Kaehyun Um, Sihoon Yang, Chang En Yea, Sihyuk Yi, Kyungjae Yoo, Chansik Yoon

arXiv:2608.04505v1cs.CL

TL;DR

The report addresses how to advance a frontier-scale foundation model toward reliable practical capabilities without discarding the prior model’s accumulated investment. It upcycles K-EXAONE into a larger model and finds more than 10% average improvement across 24 benchmarks, with strongest gains in coding, agentic coding, and long-context understanding.

  • Problem

    The report addresses the need for a frontier-scale foundation model whose increased capacity translates into reliable capabilities under practical deployment conditions.

  • Method

    The authors upcycle K-EXAONE by expanding model depth and expert capacity, then apply continued pre-training and domain-specific preference optimization.

  • Results

    More than 10% average improvement over K-EXAONE was observed across 24 benchmarks, with largest gains in coding, agentic coding, and long-context understanding.

  • Takeaways & Limitations

    Released under Apache 2.0, K-EXAONE 2.0 can be independently evaluated, deployed, adapted, and built upon by the wider AI ecosystem.

  • Takeaways & Limitations

    Problematic content may remain in training data, so the models may occasionally generate inappropriate or undesirable responses.

Abstract

from arXiv · show

This technical report presents K-EXAONE 2.0, an open-weight multilingual foundation model developed by LG AI Research as a step in our effort toward global frontier-scale foundation models. Rather than training from scratch, we upcycle K-EXAONE and expand its architecture, yielding a Mixture-of-Experts (MoE) model with 750B total parameters and approximately 37B activated per token---more than three times the capacity of its predecessor. K-EXAONE 2.0 supports context lengths of up to 256K tokens and expands multilingual coverage from six to ten languages. Its training pipeline combines continual pre-training, difficulty-focused mid-training, and post-training to strengthen reasoning, agentic coding, multilingual capability, and safety grounded in Korean sociocultural contexts. Across nine evaluation categories selected to reflect the conditions of practical use, K-EXAONE 2.0 improves over K-EXAONE and remains competitive with open-weight models, showing its largest gains in agentic coding and long-context understanding and its clearest strengths in long-context retrieval and safety. Released under the Apache 2.0 license, K-EXAONE 2.0 enables the wider AI ecosystem to evaluate, deploy, adapt, and build upon it, while marking the beginning---rather than the endpoint---of our challenge toward the global frontier.

1 Introduction · 2 Modeling

K-EXAONE 2.0 upcycles K-EXAONE into a 750B-parameter MoE model with 37B activated parameters, extending capacity while retaining prior training investment. Its modeling innovations target practical capability, training stability, and faster speculative decoding through difficulty-focused training, architectural expansion, and MTP/DSpark drafting.

  • 1 Introduction: K-EXAONE 2.0 upcycles K-EXAONE rather than training from scratch, expanding depth and width before continued pre-training.The expansion increases layers from 48 to 78 and experts from 128 to 256 while preserving accumulated parameters, computation, data, and training practices.
  • 1 Introduction: Across nine practical-use evaluation categories, K-EXAONE 2.0 consistently improves over K-EXAONE, with largest gains in agentic coding and long-context understanding.The evaluation also identifies long-context retrieval and safety as the model’s clearest strengths over comparable open-weight models.
  • 2 Modeling: 750B total parameters and 37B activated parameters scale K-EXAONE 2.0 to 3.2× its predecessor while retaining resource-efficient MoE inference.Total parameters rise from 236B to 750B, while activated parameters increase from 23B to 37B.
  • 2.1 Model Configurations: The 78-layer architecture combines two dense layers with 76 fine-grained sparse MoE layers, with Clamped SwiGLU applied to the final 16 MoE layers.MTP and DSpark are supported as independently applicable modules.
  • 2.1 Model Configurations: One shared expert and the top-8 routed experts activate per token, while inherited routing and attention choices support stable, cost-efficient long-context modeling.The inherited design includes sigmoid-based scoring, sequence-level load balancing, dropless routing, hybrid attention, QK Norm, and SWA-only RoPE.
  • 2.2 Model Upcycling: Random rotation noise differentiates duplicated experts during width up-scaling, preventing tied copies from receiving identical gradients and leaving added capacity unused.Each expert is duplicated together with its router row, and the perturbation breaks symmetry in a norm-preserving way.
  • 2.2 Model Upcycling: 6862 peak SwiGLU activation without clamping is reduced to 48.96 with the clamp at the last layer, improving low-precision training and serving stability.The clamp uses threshold 7.0, bounds the branches before combination, and leaves routing decisions unchanged.
  • 2.3 Inference Optimization via Speculative Decoding: DSpark outperforms MTP at the same draft budget γ = 7, increasing acceptance length by 32∼66% and end-to-end speedup from 1.27∼1.77× to 1.81∼2.57×.DSpark uses a five-layer semi-autoregressive drafter with block size 7; its block-diffusion backbone drafts in one forward pass and sequentially refines left to right.

3 Continual Pre-training

Continual pre-training scales K-EXAONE 2.0 through depth upscaling, healing, and additional data, while improving knowledge acquisition, Korean cultural understanding, multilingual coverage, and Korean-subset loss. The process combines selective synthetic-data strategies with targeted Korean and multilingual data construction.

  • Scaling the pre-training process: Depth upscaling is followed by a healing stage and 8T additional training tokens to stabilize and extend continual pre-training.The recipe largely preserves K-EXAONE’s data composition, quality filtering, and synthetic-data generation methods.
  • Synthetic data strategies: Active Reading achieves the largest average improvement over the initial checkpoint, primarily through ARC-C gains, while textbook-style generation performs best on MMLU and GSM8K.Active Reading and textbook-style variants each use 40B tokens, including the same 30B baseline data and 10B synthetic Wikipedia tokens.
  • Korean data construction: Institution-sourced Korean data performs better on Korean culture and history benchmarks, whereas open-source Korean data achieves higher average scores on knowledge-intensive benchmarks.The results indicate that general knowledge transfers across languages, while country-specific historical and cultural knowledge benefits from high-quality local data.
  • Multilingual expansion: Adding French, Italian, Polish, and Portuguese expands K-EXAONE’s coverage from six to ten languages, using filtered FineWeb2 corpora to strengthen language capabilities and cross-lingual transfer.The new language datasets are constructed through an in-house filtering pipeline.
  • Training-loss trends: Korean-subset loss decreases consistently across the K-EXAONE lineage, plausibly reflecting increased capacity and compute alongside greater volume, coverage, and diversity of newly constructed Korean data.Thinking-augmented and latent-thought data construction may contribute additionally, but their effects are not isolable.

4 Mid-training

Mid-training progressively extends the context window from 8K to 256K tokens through two stages while targeting reasoning, long-context understanding, and agentic capabilities. K-EXAONE 2.0 achieves perfect long-context retrieval up to 256K tokens, and Mid-Stage 2 yields larger HLE gains than direct post-training after Mid-Stage 1.

  • Mid-training stages: Mid-training uses two sequential stages, training 400B tokens at 64K context before an additional 400B tokens with contexts exceeding 64K.The overall progression extends the context window from 8K to 64K and then to 256K tokens, while targeting reasoning, long-context understanding, and agentic capabilities.
  • Data composition: Long-context adaptation uses complete documents, large code repositories, and extended tool-use and interaction trajectories to support realistic agent workflows.These workflows require maintaining state and integrating information across extended histories, multiple files, tool outputs, and intermediate reasoning steps.
  • Long-context verification: Perfect retrieval scores are maintained across evaluated needle positions and context lengths of up to 256K tokens on the NIAH test.NIAH measures recovery of information inserted at varying positions within increasingly long contexts.
  • Effect of mid-training stages: 4.71 points is the HLE improvement from Mid-Stage 2 over Mid-Stage 1, exceeding the 3.15-point gain from applying SFT directly after Mid-Stage 1.The comparison uses the small-scale model and further trains both checkpoints on the same 23B token subset of the K-EXAONE SFT dataset.

5 Post-training

Post-training combines broad supervised fine-tuning with targeted preference learning to strengthen reasoning, agentic capabilities, and safety. The pipeline preserves expert routing and thinking behavior while using diverse, filtered, and verifiable supervision.

  • Post-training: Post-training has two stages: supervised fine-tuning aggregates 350B tokens across reasoning, knowledge, instruction following, and agentic systems, while preference learning targets reasoning, agents, and safety.SFT uses broad domain coverage; preference learning focuses on a smaller set of capabilities.
  • Supervised Fine-Tuning: During SFT, the router is frozen and thinking and non-thinking modes are jointly trained; agent training preserves thinking behavior across agentic supervision.This preserves expert specialization and routing patterns while supporting deliberate reasoning and direct responses.
  • Supervised Fine-Tuning: Reasoning SFT spans mathematics, code, long-context understanding, knowledge-intensive tasks, and general conversations to improve instruction interpretation, decomposition, information integration, and response generation.The reasoning mixture is organized into five primary domains: mathematics, code, long-context, knowledge, and general chat.
  • Supervised Fine-Tuning: Reasoning supervision is filtered for validity, consistency, formatting, repetition, and synthetic artifacts, with answer- or execution-based verification and benchmark decontamination.The pipeline removes contradictory, unsupported, leaked, malformed, or substantially overlapping examples.
  • Supervised Fine-Tuning: Agentic SFT standardizes tool calling and covers general tools, Korean public APIs, agentic search, and coding agents, while balanced depth- and breadth-oriented queries improve both search capabilities.Preserved thinking retains reasoning blocks across multi-turn agentic conversations; hybrid queries combine broad discovery with deep verification.
  • Preference Learning: Multi-task preference optimization covers mathematics, coding, knowledge, agentic, and chat tasks, using domain-specific criteria and rewards, with verifiable signals for mathematics and coding.Rejected responses are penalized for undesirable patterns, while preferred responses are selected using task-appropriate criteria.

6 Evaluation

K-EXAONE 2.0 is evaluated across nine benchmark categories, with the strongest reported gains in agentic coding, tool use, instruction following, long-context understanding, and multilingual mathematical reasoning. It maintains strong performance in world knowledge, mathematics, Korean, multilinguality, and safety-related evaluation.

  • Reasoning Abilities: On mathematical reasoning, K-EXAONE 2.0 scores 92.3 on AIME 2026, 78.4 on HMMT FEB 2026, and 78.6 on IMO-ANSWERBENCH.The IMO-ANSWERBENCH score improves by 2.3 points over K-EXAONE.
  • Agentic Abilities: K-EXAONE 2.0 improves substantially over K-EXAONE in agentic coding, scoring 68.2 on SWE-BENCH VERIFIED and 43.8 on TERMINAL-BENCH 2.1.These scores increase by 18.8 and 13.5 points, respectively.
  • General Abilities: OPENAI-MRCR rises from 52.3 to 94.4, while AA-LCR and KO-LONGBENCH reach 56.2 and 89.6, respectively.The results indicate improved retrieval and reasoning over long inputs across English.
  • General Abilities: K-EXAONE 2.0 scores 92.4 on IFEVAL and 72.6 on IFBENCH, improving over K-EXAONE by 2.7 and 5.3 points, respectively.The evaluation spans instruction following, alongside other general capability categories.
  • Korean and Multilingual Abilities: Across Korean benchmarks, K-EXAONE 2.0 scores 69.1 on KMMLU-PRO, 84.2 on CLICK, and 91.1 on HRM8K-KSM.These results reflect Korean professional knowledge, cultural and linguistic understanding, and mathematical reasoning.
  • Korean and Multilingual Abilities: Across nine non-English languages, K-EXAONE 2.0 scores 86.6 on both MMMLU and GLOBALMMLU-LITE, and 71.3 on POLYMATH.The POLYMATH score improves by 13.9 points over K-EXAONE.

7 Limitations

K-EXAONE 2.0 may generate inappropriate, biased, incorrect, false, or contradictory responses because outputs rely on learned token probabilities and training data. LG AI Research seeks to reduce these risks, while prohibiting malicious use intended to induce inappropriate outputs.

  • Limitations: K-EXAONE 2.0 may generate inappropriate responses containing personal, harmful, or other inappropriate information.Problematic content may remain in training data despite efforts to exclude it.
  • Limitations: K-EXAONE 2.0 may produce biased responses associated with age, gender, race, and other attributes.Its responses rely heavily on training-data statistics, which can also produce semantically or syntactically incorrect sentences.
  • Limitations: K-EXAONE 2.0 responses may be false or contradictory because the models do not reflect the latest information.Users are prohibited from malicious activities, including entering illegal information, that may induce inappropriate outputs violating LG AI’s ethical principles.

8 Deployment

Deployment requires consulting the Appendix’s Section B for K-EXAONE 2.0 license information. This information is essential for legally using the language model.

  • Section B in the Appendix provides license information for using K-EXAONE 2.0 models.
  • Understanding the license information is essential for the legal utilization of the language model.
  • Deployment should therefore account for the licensing information documented in Appendix Section B.

9 Conclusion · A Contributors · B Model License

K-EXAONE 2.0 scales K-EXAONE to 750B total parameters with approximately 37B activated per token and improves practical-deployment capabilities, especially coding and long-context understanding. Released under Apache 2.0, it supports broader evaluation and use while positioning future work toward trillion-parameter-scale models.

  • 9 Conclusion: 750B total parameters and approximately 37B activated per token result from upcycling K-EXAONE and expanding its depth and expert capacity.The model retains hybrid attention supporting context lengths of up to 256K tokens.
  • 9 Conclusion: Continual pre-training, difficulty-focused mid-training, and post-training direct expanded capacity toward practical deployment capabilities.The training pipeline targeted knowledge, Korean-language proficiency, multilingual foundations, advanced reasoning, long-context understanding, and repository-level coding.
  • 9 Conclusion: More than 10% average improvement over K-EXAONE appears across 24 benchmarks spanning nine practical-deployment categories.The largest gains were in coding, agentic coding, and long-context understanding, including approximately 30% improvements on three benchmarks.
  • 9 Conclusion: Apache 2.0 licensing enables the wider AI ecosystem to evaluate, deploy, adapt, and build upon K-EXAONE 2.0.The release also permits examination of reported capabilities and limitations across broader applications and deployment environments.
  • 9 Conclusion: The next phase targets models at and beyond the trillion-parameter scale, making K-EXAONE 2.0 a beginning rather than an endpoint.This direction builds on the project’s established technical and operational capabilities.
  • A Contributors: All authors are listed in alphabetical order by last name, with separate rosters for core contributors and contributors.The contributor sections enumerate the individuals involved in each category.
  • B Model License: Copyright © 2026 LG AI Research, and the software is licensed under the Apache License, Version 2.0.Use is subject to the License’s terms and conditions.
  • B Model License: The software is distributed on an “AS IS” BASIS without warranties, subject to applicable law and the License’s permissions and limitations.The license text directs users to its specific governing terms.

C Evaluation Setup Details · D Preserved Thinking

The evaluation setup follows benchmark-specific official protocols with deterministic patches, defined agents, sampling limits, resource constraints, and scoring procedures. In agentic scenarios, preserved thinking retains prior reasoning across turns and interleaves reasoning with tool calls within each turn.

  • C Evaluation Setup Details: Official benchmark setups are followed unless otherwise specified, with internal evaluation settings documented for reproducibility.This establishes the general evaluation policy for the reported benchmarks.
  • C Evaluation Setup Details: Claw-Eval uses commit d3f02d4, with missing mock_today fields patched to make date-dependent grading deterministic across runs.The patch supplies the reference date required by affected graders.
  • C Evaluation Setup Details: Terminal-Bench 2.1 uses Terminus 2 with temperature 1.0, top-p 0.95, 32,768 generated tokens per step, a 256K context window, and a 4-hour timeout.Each task runs with 8 CPU cores, 16GB of memory, and 20GB of disk.
  • C Evaluation Setup Details: SWE-Bench Verified uses mini-SWE-agent with temperature 1.0, top-p 0.95, 32,768 generated tokens per step, a 256K context window, and a 4-hour timeout.Each instance runs with 4 CPU cores, 8GB of memory, and 10GB of disk.
  • C Evaluation Setup Details: OpenAI-MRCR follows its official protocol, averages 2-needle, 4-needle, and 8-needle scores within bins, evaluates up to 128K tokens, and reports the macro-average across bins.Scores use the difflib.SequenceMatcher ratio, and the provided alphanumeric hash must be prepended.
  • C Evaluation Setup Details: ROK-Fortress uses a multi-model judging ensemble of Kimi K2 Thinking, o4-mini, and GPT-5.2 across four culturally varied English/Korean categories.The categories combine original or translated language with US context or Korean cultural adaptation.
  • C Evaluation Setup Details: The ROK-Fortress final safety score averages TRS across four categories, subtracts that average from 100, and thereby makes higher scores indicate better safety.This inversion converts the original risk-oriented TRS into a safety-oriented measure.
  • D Preserved Thinking: Preserved thinking retains every prior-turn reasoning block throughout the conversation and interleaves within-turn reasoning with tool calls conditioned on observed results.After a final answer, these blocks are typically dropped, requiring intermediate conclusions to be re-derived at the next turn.

E Multilingual · F Safety

K-EXAONE 2.0 expands multilingual support from six to ten languages while maintaining balanced performance across languages and reporting bidirectional translation results for English and nine additional languages. Its safety program broadens multilingual and adversarial coverage, recognizes the limits of known-risk red teaming, and adds domain-expert risk discovery grounded in Korea-augmented universal standards.

  • E Multilingual: 10 supported languages mark an expansion from K-EXAONE’s six, adding French, Italian, Polish, and Portuguese.The original six were Korean, English, Spanish, German, Japanese, and Vietnamese.
  • E Multilingual: Balanced multilingual capability results from performance gains distributed evenly across languages, without pronounced degradation.K-EXAONE 2.0 achieves higher average performance comparable to K-EXAONE in the reported comparisons.
  • E Multilingual: WMT24++ evaluates English and nine additional languages in both English-to-target and target-to-English translation directions.Detailed results are presented in Table 9.
  • F Safety: K-AUT defines safety risks across four domains and 226 detailed risk areas, combining universal human values with Korea-specific sensitivities.The taxonomy is intended to avoid becoming meaningful only within Korea and draws on UN declarations and internationally recognized human-rights norms.
  • F Safety: Four newly supported languages are incorporated into safety training alongside expanded defenses against previously reported attacks and multi-turn adversarial testing.The additional languages are French, Italian, Polish, and Portuguese.
  • F Safety: Known-method red teaming assesses whether enumerated risks can be elicited but is comparatively weak at surfacing risks not previously considered or defined.This limitation identifies a structural blind spot in established attack-method evaluation.
  • F Safety: Field experts from specialized professional domains are added to risk discovery to identify hazards that general-purpose safety frameworks may miss.The goal is broader protection against both societal harms and domain-specific risks.

F.1 Expanding the Locus of Risk Discovery · F.2 Consultation Process with the Safety Teacher Advisory Council · F.3 K-AUT-V2 : Expanded Korea-Augmented Universal Taxonomy

K-AUT risk discovery expands beyond technical experts through a 46-member Safety Teacher Advisory Council and a four-week review loop. The resulting K-AUT-V2 grows from 226 to 296 risk areas while strengthening protections, Korea-specific coverage, frontier-risk coverage, and behavioral-policy guidance.

  • F.1 Expanding the Locus of Risk Discovery: A 46-member Safety Teacher Advisory Council extends risk discovery from technical experts to value and field experts.The council comprises teachers who completed UNESCO GCED training and currently teach it in their classrooms.
  • F.1 Expanding the Locus of Risk Discovery: The council combines sensitivity to universal values with deep familiarity with the Korean context when judging model risks.Its members consider human rights, diversity, inclusion, and Korean context in classroom judgment.
  • F.1 Expanding the Locus of Risk Discovery: Recruitment draws from several thousand GCED-certified teachers, with more than four hundred applicants and selection across six diversity dimensions.The dimensions include gender, age, school level, teaching experience, subject area, and region.
  • F.1 Expanding the Locus of Risk Discovery: 35 members (76%) have more than ten years of classroom experience, while school levels are distributed evenly across elementary, middle, and high schools.The composition targets both maturity of judgment and breadth of perspective across student developmental stages.
  • F.2 Consultation Process with the Safety Teacher Advisory Council: The program operates as a four-week closed loop in which the council red-teams the model, proposes criteria and risks, and reviews curated revisions.Adopted changes are incorporated into K-AUT and applied to model training and evaluation.
  • F.3 K-AUT-V2 : Expanded Korea-Augmented Universal Taxonomy: K-AUT-V2 strengthens protection for structurally vulnerable parties and expands Korea-specific geopolitical, historical, and frontier-risk coverage.Examples include minors, subordinate parties, victims, the North Korean nuclear issue, constitutional order, historical revisionism, diaspora identity, and AI goal misalignment.
  • F.3 K-AUT-V2 : Expanded Korea-Augmented Universal Taxonomy: K-AUT functions as a comprehensive behavioral policy specifying what models should refuse, permitted assistance levels, and how refusals should be delivered.Its intended role extends beyond a list of prohibitions.

F.4 Making the Judgments Trustworthy

Trustworthy judgments are built through expert selection, deliberate resolution of disagreement, and verification that revised criteria produce intended model changes. Iterative refinement of these criteria is associated with consistently higher Safe Rates across KGC-SAFETY dimensions than previous K-EXAONE models.

  • Making the Judgments Trustworthy: Trustworthiness is managed through expert selection, disagreement resolution, and verification that decisions lead to actual model changes.These stages address who is asked, how divergent opinions are resolved, and whether resulting decisions affect the model.
  • Selection: APCEIU provides a pre-qualified expert pool, while the authors directly apply the stated balancing criteria during final selection.This avoids ad hoc recruitment and delegated responsibility for balancing.
  • Handling disagreement: Divergent teacher assessments are treated as evidence of sensitivity; multiple reviews and examination of proposed revisions guide the final determination without averaging or discarding minority views.Teachers identify what is risky and why, while the authors decide which grounds become final criteria and retain responsibility.
  • Verification: Revised criteria are converted into training data and re-evaluated with human and model-based evaluation to verify intended behavioral changes.Human evaluation better reflects genuine preferences but is slow and costly, whereas model-based evaluation offers speed and scalability with transfer risk.
  • Conclusion and Analysis: Iterative refinement of evaluation criteria helps mitigate safety vulnerabilities and yields consistently higher Safe Rates across all evaluated KGC-SAFETY dimensions than previous K-EXAONE models.The results support a robust framework for social safety and reliability in the Korean context.
Loading 2608.04505v1…