PaperlayerView exact revision on arXiv

Source-linked AI summary

HyperCLOVA X Technical Report

Kang Min Yoo, Jaegeun Han, Sookyo In, Heewon Jeon, Jisu Jeong, Jaewook Kang, Hyunwook Kim, Kyung-Min Kim, Munhyong Kim, Sungju Kim, Donghyun Kwak, Hanock Kwak, Se Jung Kwon, Bado Lee, Dongsoo Lee, Gichang Lee, Jooho Lee, Baeseong Park, Seongjin Shin, Joonsang Yu, Seolki Baek, Sumin Byeon, Eungsup Cho, Dooseok Choe, Jeesung Han, Youngkyun Jin, Hyein Jun, Jaeseung Jung, Chanwoong Kim, Jinhong Kim, Jinuk Kim, Dokyeong Lee, Dongwook Park, Jeong Min Sohn, Sujung Han, Jiae Heo, Sungju Hong, Mina Jeon, Hyunhoon Jung, Jungeun Jung, Wangkyo Jung, Chungjoon Kim, Hyeri Kim, Jonghyun Kim, Min Young Kim, Soeun Lee, Joonhee Park, Jieun Shin, Sojin Yang, Jungsoon Yoon, Hwaran Lee, Sanghwan Bae, Jeehwan Cha, Karl Gylleus, Donghoon Ham, Mihak Hong, Youngki Hong, Yunki Hong, Dahyun Jang, Hyojun Jeon, Yujin Jeon, Yeji Jeong, Myunggeun Ji, Yeguk Jin, Chansong Jo, Shinyoung Joo, Seunghwan Jung, Adrian Jungmyung Kim, Byoung Hoon Kim, Hyomin Kim, Jungwhan Kim, Minkyoung Kim, Minseung Kim, Sungdong Kim, Yonghee Kim, Youngjun Kim, Youngkwan Kim, Donghyeon Ko, Dughyun Lee, Ha Young Lee, Jaehong Lee, Jieun Lee, Jonghyun Lee, Jongjin Lee, Min Young Lee, Yehbin Lee, Taehong Min, Yuri Min, Kiyoon Moon, Hyangnam Oh, Jaesun Park, Kyuyon Park, Younghun Park, Hanbae Seo, Seunghyun Seo, Mihyun Sim, Gyubin Son, Matt Yeo, Kyung Hoon Yeom, Wonjoon Yoo, Myungin You, Doheon Ahn, Homin Ahn, Joohee Ahn, Seongmin Ahn, Chanwoo An, Hyeryun An, Junho An, Sang-Min An, Boram Byun, Eunbin Byun, Jongho Cha, Minji Chang, Seunggyu Chang, Haesong Cho, Youngdo Cho, Dalnim Choi, Daseul Choi, Hyoseok Choi, Minseong Choi, Sangho Choi, Seongjae Choi, Wooyong Choi, Sewhan Chun, Dong Young Go, Chiheon Ham, Danbi Han, Jaemin Han, Moonyoung Hong, Sung Bum Hong, Dong-Hyun Hwang, Seongchan Hwang, Jinbae Im, Hyuk Jin Jang, Jaehyung Jang, Jaeni Jang, Sihyeon Jang, Sungwon Jang, Joonha Jeon, Daun Jeong, Joonhyun Jeong, Kyeongseok Jeong, Mini Jeong, Sol Jin, Hanbyeol Jo, Hanju Jo, Minjung Jo, Chaeyoon Jung, Hyungsik Jung, Jaeuk Jung, Ju Hwan Jung, Kwangsun Jung, Seungjae Jung, Soonwon Ka, Donghan Kang, Soyoung Kang, Taeho Kil, Areum Kim, Beomyoung Kim, Byeongwook Kim, Daehee Kim, Dong-Gyun Kim, Donggook Kim, Donghyun Kim, Euna Kim, Eunchul Kim, Geewook Kim, Gyu Ri Kim, Hanbyul Kim, Heesu Kim, Isaac Kim, Jeonghoon Kim, Jihye Kim, Joonghoon Kim, Minjae Kim, Minsub Kim, Pil Hwan Kim, Sammy Kim, Seokhun Kim, Seonghyeon Kim, Soojin Kim, Soong Kim, Soyoon Kim, Sunyoung Kim, Taeho Kim, Wonho Kim, Yoonsik Kim, You Jin Kim, Yuri Kim, Beomseok Kwon, Ohsung Kwon, Yoo-Hwan Kwon, Anna Lee, Byungwook Lee, Changho Lee, Daun Lee, Dongjae Lee, Ha-Ram Lee, Hodong Lee, Hwiyeong Lee, Hyunmi Lee, Injae Lee, Jaeung Lee, Jeongsang Lee, Jisoo Lee, Jongsoo Lee, Joongjae Lee, Juhan Lee, Jung Hyun Lee, Junghoon Lee, Junwoo Lee, Se Yun Lee, Sujin Lee, Sungjae Lee, Sungwoo Lee, Wonjae Lee, Zoo Hyun Lee, Jong Kun Lim, Kun Lim, Taemin Lim, Nuri Na, Jeongyeon Nam, Kyeong-Min Nam, Yeonseog Noh, Biro Oh, Jung-Sik Oh, Solgil Oh, Yeontaek Oh, Boyoun Park, Cheonbok Park, Dongju Park, Hyeonjin Park, Hyun Tae Park, Hyunjung Park, Jihye Park, Jooseok Park, Junghwan Park, Jungsoo Park, Miru Park, Sang Hee Park, Seunghyun Park, Soyoung Park, Taerim Park, Wonkyeong Park, Hyunjoon Ryu, Jeonghun Ryu, Nahyeon Ryu, Soonshin Seo, Suk Min Seo, Yoonjeong Shim, Kyuyong Shin, Wonkwang Shin, Hyun Sim, Woongseob Sim, Hyejin Soh, Bokyong Son, Hyunjun Son, Seulah Son, Chi-Yun Song, Chiyoung Song, Ka Yeon Song, Minchul Song, Seungmin Song, Jisung Wang, Yonggoo Yeo, Myeong Yeon Yi, Moon Bin Yim, Taehwan Yoo, Youngjoon Yoo, Sungmin Yoon, Young Jin Yoon, Hangyeol Yu, Ui Seon Yu, Xingdong Zuo, Jeongin Bae, Joungeun Bae, Hyunsoo Cho, Seonghyun Cho, Yongjin Cho, Taekyoon Choi, Yera Choi, Jiwan Chung, Zhenghui Han, Byeongho Heo, Euisuk Hong, Taebaek Hwang, Seonyeol Im, Sumin Jegal, Sumin Jeon, Yelim Jeong, Yonghyun Jeong, Can Jiang, Juyong Jiang, Jiho Jin, Ara Jo, Younghyun Jo, Hoyoun Jung, Juyoung Jung, Seunghyeong Kang, Dae Hee Kim, Ginam Kim, Hangyeol Kim, Heeseung Kim, Hyojin Kim, Hyojun Kim, Hyun-Ah Kim, Jeehye Kim, Jin-Hwa Kim, Jiseon Kim, Jonghak Kim, Jung Yoon Kim, Rak Yeong Kim, Seongjin Kim, Seoyoon Kim, Sewon Kim, Sooyoung Kim, Sukyoung Kim, Taeyong Kim, Naeun Ko, Bonseung Koo, Heeyoung Kwak, Haena Kwon, Youngjin Kwon, Boram Lee, Bruce W. Lee, Dagyeong Lee, Erin Lee, Euijin Lee, Ha Gyeong Lee, Hyojin Lee, Hyunjeong Lee, Jeeyoon Lee, Jeonghyun Lee, Jongheok Lee, Joonhyung Lee, Junhyuk Lee, Mingu Lee, Nayeon Lee, Sangkyu Lee, Se Young Lee, Seulgi Lee, Seung Jin Lee, Suhyeon Lee, Yeonjae Lee, Yesol Lee, Youngbeom Lee, Yujin Lee, Shaodong Li, Tianyu Liu, Seong-Eun Moon, Taehong Moon, Max-Lasse Nihlenramstroem, Wonseok Oh, Yuri Oh, Hongbeen Park, Hyekyung Park, Jaeho Park, Nohil Park, Sangjin Park, Jiwon Ryu, Miru Ryu, Simo Ryu, Ahreum Seo, Hee Seo, Kangdeok Seo, Jamin Shin, Seungyoun Shin, Heetae Sin, Jiangping Wang, Lei Wang, Ning Xiang, Longxiang Xiao, Jing Xu, Seonyeong Yi, Haanju Yoo, Haneul Yoo, Hwanhee Yoo, Liang Yu, Youngjae Yu, Weijie Yuan, Bo Zeng, Qian Zhou, Kyunghyun Cho, Jung-Woo Ha, Joonsuk Park, Jihyun Hwang, Hyoung Jo Kwon, Soonyong Kwon, Jungyeon Lee, Seungho Lee, Seonghyeon Lim, Hyunkyung Noh, Seungho Choi, Sang-Woo Lee, Jung Hwa Lim, Nako Sung

arXiv:2404.01954v2cs.CLcs.AI

TL;DR

Many LLMs are shaped by English-language objectives and corpora, limiting their capacity to handle Korean language and cultural nuance. HyperCLOVA X is pretrained on Korean, English, and code, then instruction-tuned; it shows strong Korean reasoning and broad multilingual capabilities across benchmarks.

  • Problem

    LLMs driven by English objectives and predominantly North American pretraining norms have limitations in processing Korean language and its cultural and regional nuances.

    Source: §1 · p. 1

  • Method

    HyperCLOVA X models are pretrained on a mixture of Korean, English, and code data, then aligned through supervised fine-tuning and reinforcement learning from human feedback.

    Source: §2 · p. 2

  • Results

    HyperCLOVA X demonstrates strong Korean reasoning and knowledge, competitive English capabilities, and multilingual reasoning and translation, including state-of-the-art translation between Korean and untargeted languages such as Japanese and Chinese.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    Its benchmark performance and multilingual abilities indicate potential for application across different linguistic contexts.

    Source: §6 · p. 24

    Multimodality, inference quantization, and external tool or API integration remain future work.

    Source: §6 · p. 25

Abstract

from arXiv · show

We introduce HyperCLOVA X, a family of large language models (LLMs) tailored to the Korean language and culture, along with competitive capabilities in English, math, and coding. HyperCLOVA X was trained on a balanced mix of Korean, English, and code data, followed by instruction-tuning with high-quality human-annotated datasets while abiding by strict safety guidelines reflecting our commitment to responsible AI. The model is evaluated across various benchmarks, including comprehensive reasoning, knowledge, commonsense, factuality, coding, math, chatting, instruction-following, and harmlessness, in both Korean and English. HyperCLOVA X exhibits strong reasoning capabilities in Korean backed by a deep understanding of the language and cultural nuances. Further analysis of the inherent bilingual nature and its extension to multilingualism highlights the model's cross-lingual proficiency and strong generalization ability to untargeted languages, including machine translation between several language pairs and cross-lingual inference tasks. We believe that HyperCLOVA X can provide helpful guidance for regions or countries in developing their sovereign LLMs.

1 Introduction

HyperCLOVA X addresses the limits of English-centric models in understanding Korean language and culture with models pretrained on Korean, English, and code, then instruction-tuned on human-annotated data. The report describes strong Korean reasoning, competitive English capabilities, multilingual performance, and safety measures.

1 Introduction

  • HyperCLOVA X comprises a larger HCX-L and a lightweight HCX-S, pretrained on Korean, English, and code data before instruction-tuning with human-annotated demonstrations and preferences.The models are tailored to Korean linguistic and cultural contexts and can understand and generate English and other languages.Source: §1 · p. 1
  • HyperCLOVA X demonstrates comprehensive Korean language and cultural knowledge, strong Korean reasoning, and performance on par with proficient English-centric LLMs across core competencies.The evaluations span reasoning, knowledge, commonsense, factuality, coding, math, chatting, instruction-following, and harmlessness in Korean and English.Source: §1 · p. 1
  • The models extend reasoning to selected Asian languages, achieve state-of-the-art Korean translation with untargeted languages such as Japanese and Chinese, and show cross-language instruction transfer between Korean and English.The report describes instruction-following capability in one language emerging after instruction-tuning in the other.Source: §1 · p. 2
  • The report describes systematic red teaming, safety-data collection, and automatic and human-annotated evaluations to monitor and mitigate harmful, toxic, or sensitive outputs.These processes are grounded in NAVER AI Ethics Principles.Source: §1 · p. 2
  • The authors believe HyperCLOVA X’s capabilities beyond Korean can provide guidance to regions or countries developing sovereign LLMs.They connect this aim to the UN General Assembly’s promotion of safe, secure, and trustworthy AI systems.Source: §1 · p. 2

2 Training Details

HyperCLOVA X is a Korean- and culture-focused model family comprising HCX-L and HCX-S, pretrained on Korean, multilingual, and code data, then aligned with supervised fine-tuning and reinforcement learning from human feedback. Its training details cover model architecture, tokenization, pretraining objectives, alignment methods, and an automated pipeline.

2 Training Details

  • HyperCLOVA X comprises larger HCX-L and smaller HCX-S models, pretrained mainly on Korean, English, and code, then aligned with supervised fine-tuning and reinforcement learning from human feedback.The family is specialized for Korean language and culture while also targeting English and code capabilities.Source: §2 · p. 2

2.1 Pretraining

  • The pretrained model uses a transformer decoder with rotary position embeddings, pre-normalization, and grouped-query attention for longer context, training stability, and efficiency.The report identifies HyperCLOVA X as an updated version of HyperCLOVA.Source: §2.1 · p. 2
  • Pretraining uses roughly equal shares of Korean, multilingual data, and code; Korean is about a third, and preprocessing filters low-quality or harmful documents, removes PII, and upsamples knowledge-rich data.The multilingual portion is predominantly English but also includes languages such as Japanese, German, and French.Source: §2.1 · pp. 2–3
  • A 100,000-item morpheme-aware byte-level BPE tokenizer is designed for Korean; the report says it encodes documents with the fewest tokens on average, particularly effectively for Korean.The table compares average token lengths across 1,000 documents per language.Source: §2.1 · p. 3, Table 1 · p. 3
  • Joint PSM and SPM pretraining supports both left-to-right and infilling abilities; 90% of training uses a 4,096-token context and the final 10% uses 32,768.PSM and SPM vary the order in which document prefix, middle, and suffix sections are fed to the model.Source: §2.1 · p. 3, §2.2.1 · p. 3

2.2.1 Supervised Fine-tuning (SFT)

  • SFT trains prompt completions to improve instruction-following and problem-solving; it uses turn markers, masks loss outside assistant turns, and batches similar-length sequences under a fixed token maximum.The report says SFT also lets the model leverage knowledge across domains including commonsense, humanities, sciences, and ethics.Source: §2.2.1 · pp. 3–4

2.2.2 Reinforcement Learning from Human Feedback (RLHF)

  • RLHF trains a reward model on human preferences and optimizes the post-SFT model with PPO; the reward model uses ranking loss and is trained for one epoch after overfitting was observed beyond that point.Reward distributions vary across data sources, so inference-time normalization and clipping are used to mitigate reward-hacking risks and training difficulties.Source: §2.2.2 · p. 4

2.2.3 The Alignment Learning Pipeline

  • To address longer outputs and repetition, the RLHF process uses iterative feedback, length- and format-constrained early stopping, and sequence-level unlikelihood training; asynchronous batching and event-driven workflows reduce training time and human intervention.The alignment pipeline also automates evaluation and successive SFT, reward-model, and PPO stages, with metadata stored and shared through CLOps and MLflow.Source: §2.2.2 · p. 4, §2.2.3 · p. 5

3 Core Benchmarks

This section surveys HyperCLOVA X on Korean and English benchmarks spanning comprehensive understanding, commonsense and factual generation, mathematics, coding, chat, instruction-following, and harmlessness. It also describes evaluation choices for culturally relevant multilingual benchmarking that is less sensitive to prompting.

3 Core Benchmarks

  • The benchmarks combine English tests of language-neutral skills with Korean assessments of language-specific intelligence and cultural understanding; Korean tests are expert-crafted or curated rather than machine-translated.To reduce prompt sensitivity in multiple-choice evaluation, the authors use benchmark-style options, filter non-answer tokens using output probabilities, use few-shot examples when possible, and shuffle biased options.Source: §3 · pp. 5, 7
  • Figure 1 reports that the largest HyperCLOVA X outperforms other Korean-tailored models on comprehensive Korean benchmarks and performs comparably to the largest LLaMA 2 model on English-focused benchmarks.Source: Figure 1 · p. 7

3.1 Comprehensive Korean LLM Benchmarks

  • The performance gap between Korean-designed models and models not designed for Korean widens on culturally demanding HAE-RAE, KoBigBench, and some KMMLU tasks.The authors argue that success for region-specific LLMs depends on acquiring large-scale, high-quality data from the target group.Source: §3.1 · p. 9, Table 4 · p. 9

3.3 Commonsense Reasoning

  • HyperCLOVA X outperforms all baselines in the average commonsense score, with notable performance on Winogrande and CommonsenseQA; SOLAR and EEVE have an advantage on Hellaswag and physical-interaction reasoning.Source: Table 5 · p. 10, §3.3 · p. 11

3.2 Comprehensive English LLM Benchmarks

  • HCX-L’s English comprehensive benchmark average is close to the largest LLaMA 2 model’s; CoT raises its MMLU score to 69.78, and 10-sample self-consistency raises it to 70.79, while LLaMA 2 70b falls from 69.27 to 66.65 with CoT.The reported HCX-L gains are 1.87 points with one CoT sample and 2.88 points with 10 self-consistent reasoning samples (Wei et al., 2022; Wang et al., 2023).Source: Table 4 · p. 9, §3.2 · p. 10

3.4 World Knowledge and Factuality

  • HyperCLOVA X noticeably lacks Western-culture knowledge on NQ and TriviaQA, but HCX-L shows stronger factual generation about Korean entities than the baselines, alongside EEVE-Korean-v1.Source: Table 6 · p. 11, §3.4 · p. 12

3.5 Mathematics

  • HCX-L exceeds 80 percent accuracy on GSM8K, significantly outperforming all baseline models.Source: §3.5 · p. 12, Table 7 · p. 13
  • Both HCX-S and HCX-L exceed 15% accuracy on MATH, where most baseline models struggle to surpass that level.Source: §3.5 · p. 13

3.6 Coding Capabilities

  • HyperCLOVA X outperforms other models across HumanEval, MBPP, and K-HumanEval, with a substantial K-HumanEval margin over foundation models and Korean-specialist models.K-HumanEval translates HumanEval into Korean using machine translation followed by manual review.Source: §3.6 · pp. 13–14

3.7 Chat and Instruction-Following

  • HCX-L leads the chat benchmarks in Korean and English, while HyperCLOVA X surpasses other open-source LLMs on KoIF and is competitive on SuperNI.Most open-source models struggle to respond in Korean on Ko-MT; about 98% of LLaMA 2 70B responses were in English despite Korean prompts, illustrating language confusion in judging.Source: Table 9 · p. 14, §3.7 · p. 15

3.8 Harmlessness

  • HCX-L achieves the strongest harmlessness results among the baselines, using an aggregate score that combines truthfulness with safety as truthfulness × (1 − toxicity).The evaluation uses TruthfulQA and BOLD; the report refers to the aggregate as an overall harmlessness score.Source: Table 10 · p. 15, §3.8 · p. 15

3.9 Comparison with Closed Source Models

  • In closed-source comparisons, HyperCLOVA X shows strong Korean linguistic and cultural understanding, while its bilingual performance is on par with GPT-4.On HAE-RAE Bench, HCX-L outperforms other closed-source models in every area except general knowledge.Source: Table 11 · p. 16, §3.9 · p. 16, Table 12 · p. 16

4 Multilinguality

HyperCLOVA X’s multilingual evaluation covers cross-lingual reasoning, Korean-to/from-English, Japanese, and Chinese translation, and Korean-English instruction transfer. The results show transfer to underrepresented Asian languages, strong translation performance, and robust instruction-following across training-language mixes.

4.2 Machine Translation

  • HCX-L is best or a close second in all FLORES+ translation settings except Korean-to-Chinese, and beats Google Translator in every other direction.It also outperforms GPT-4 by a noticeable margin for English-to-Korean translation.Source: Table 15 · p. 18, §4.2 · p. 18

4.1 Cross-Lingual Reasoning

  • HCX-L leads other models on XNLI and X-CSQA across Asian languages, except that it ranks second for Chinese on both benchmarks.XNLI uses 5-shot prompting across Arabic, Hindi, Thai, Urdu, Vietnamese, and Chinese; X-CSQA uses 0-shot prompting and reports validation accuracy.Source: §4.1 · pp. 17–18

4.3 Cross-lingual Transfer

  • Nearly 13 Rouge-L points came from translating just 0.5% (50 of 10,000) OpenOrca training samples into Korean.Across both datasets, each language’s highest performance was achieved by including both languages in training; preference-test results were noisier.Source: Figure 2 · p. 19, §4.3 · pp. 19–20
  • HyperCLOVA X models outperform other models on average while maintaining linguistic performance across instruction-tuning languages and ratios, unlike Mistral-7B and Yi-Ko-6B.Mistral-7B loses English capability when tuned only in Korean; Yi-Ko-6B shows an imbalance between languages. The scores compare cross-lingual performance with monolingual performance.Source: Table 16 · p. 20, §4.3 · p. 20

5 Safe and Responsible AI

Section 5 describes the Ethics Principles, red-team safety-data collection, and automatic and human safety evaluations. It reports findings on toxicity, social bias, human preferences, and attack success rates.

5.1 HyperCLOVA X Ethics Principles

  • The Ethics Principles prohibit risk categories including hate and harassment, stereotypes and bias, harmful content, self-anthropomorphism, and subjective opinions.The principles guide both safety-data construction and evaluations.Source: §5.1 · pp. 20–21

5.2 Red Teaming and Safety Data Collection

  • Red-teamers attack models using hazardous topics and attack strategies, score responses for harmlessness and helpfulness, and create ranking pairs for RLHF; rewritten safe responses provide SFT data.Dialogues continue with the most harmful candidate responses for up to seven turns; the process also includes measures to address annotator safety.Source: §5.2 · p. 21, Figure 3 · p. 22

5.3 Safety Evaluation

  • Automatic safety evaluation covers toxicity and social bias in Korean and English, while human evaluation scores Korean responses to adversarial prompts.Open-source chat models serve as safety-benchmark baselines.Source: §5.3 · p. 22

5.3.1 Toxicity

  • HyperCLOVA X models show the best Korean continuation results when toxicity and qualitative response quality are considered, while HCX-L has less-toxic English continuations overall.For KOLD, toxicity scores and toxic counts are calculated only for Korean continuations; the authors conjecture that Perspective API’s incomplete Korean toxicity detection contributes to higher measured toxicity.Source: §5.3.1 · pp. 22–23, Table 17 · p. 23

5.3.2 Social Bias

  • HCX-L has the highest BBQ accuracy: 96.65% in disambiguated contexts and 85.37% in ambiguous contexts.All models show lower accuracy in ambiguous than disambiguated contexts.Source: §5.3.2 · p. 23, Table 18 · p. 24
  • HCX-L has the highest KoBBQ accuracy and diff-bias in both contexts, at 95.40% and 73.74%, respectively.The evaluation uses a sampled, category-balanced set of Korean social-bias questions augmented to 15,000 prompts.Source: §5.3.2 · p. 23, Table 19 · p. 24

5.3.3 Human Evaluation

  • HCX-S and HCX-L achieve attack success rates of 5.93% and 7.42%, respectively; HCX-S is rated safer than HCX-L and on par with GPT-4.The human evaluation covered 1,695 utterance turns in 339 dialogue sessions, while attack success rates were measured on 305 held-out attack dialogues.Source: §5.3.3 · p. 24

6 Conclusion

HyperCLOVA X combines Korean, English, and code training with supervised fine-tuning and reinforcement learning from human feedback. The conclusion highlights broad benchmark and multilingual capabilities alongside safety work, and identifies additional capabilities for future exploration.

6 Conclusion

  • HyperCLOVA X’s training combines Korean, English, and programming-language data with supervised fine-tuning and reinforcement learning from human feedback.The family emphasizes Korean language and culture while maintaining strong English and other-language capabilities.Source: §6 · p. 24
  • The report concludes that HyperCLOVA X performs across reasoning, coding, and math benchmarks and demonstrates multilingual cross-lingual reasoning and machine translation.The authors cite this multilingual ability as evidence of generalization and potential application across linguistic contexts.Source: §6 · p. 24
  • The report presents ethical principles, extensive safety evaluations, and systematic red-teaming and safety-data collection as parts of its responsible-AI approach.It describes the model’s handling of toxicity and social biases, alongside human evaluation studies.Source: §6 · p. 25
  • Future work will explore multimodal inputs, model quantization, and integration of external tools and APIs.The stated aims include broadening data modalities, optimizing inference without sacrificing output quality, and enriching response factuality through access to specialized datasets and services.Source: §6 · p. 25

B.1 Mathematics Evaluation Detail

The appendix reports GSM8K accuracy using 8-shot maj@8 at temperature 0.7 and MATH accuracy using 4-shot maj@4 at temperature 0.2. It also notes uncertainty about reproducing a published SOLAR 10.7b GSM8K result.

B.1 Mathematics Evaluation Detail

  • GSM8K accuracy is measured with 8-shot maj@8 at temperature 0.7, while MATH accuracy uses 4-shot maj@4 at temperature 0.2.These are the configurations specified for the reported evaluations.Source: §B.1 · p. 37
  • Under 8-shot GSM8K evaluation using maj@8 at temperature 0.7, SOLAR 10.7b’s Table 7 accuracy appears lower than Kim et al. (2023) reported, prompting speculation that specific configurations are needed to replicate it.Source: §B.1 · p. 37
  • The authors speculate that specific experimental configurations may be necessary to reproduce the SOLAR 10.7b GSM8K accuracy reported by Kim et al. (2023).They frame this as speculation based on the differing Llama 2 70b results they cite.Source: §B.1 · p. 37

B.2 Coding Capabilities Evaluation Detail

The coding evaluation appendix describes benchmark prompting and post-processing for base and chat models. It says post-processing was applied consistently across model types to support fair evaluation.

B.2 Coding Capabilities Evaluation Detail

  • Base models use benchmark-provided prompts, while chat models receive brief guidelines in their chat templates to clarify benchmark prompts.The HumanEval and MBPP templates are described in the cited figures.Source: §B.2 · p. 37
  • The MBPP evaluation uses a 3-shot prompt configuration referenced from Austin et al. (2021a,b).Source: §B.2 · p. 37
  • Chat-model code is parsed from markdown code blocks; base-model HumanEval responses may receive leading padding to address indentation errors.The authors apply post-processing consistently across model outputs according to model type to ensure evaluation fairness.Source: §B.2 · p. 37

B.3 Factscore Evaluation Detail

The Factscore evaluation adapts the scoring system to Korean Wikipedia by revising the prompts and selecting information-rich titles. The supplied examples show atomic-fact extraction and the selected evaluation titles.

B.3 Factscore Evaluation Detail

  • The prompts for extracting atomic facts were translated and modified from the original prompt, and Figure 8 shows the adapted prompt.The original prompt is attributed to Min et al. (2023).Source: §B.3 · p. 37, Figure 8 · p. 40
  • The evaluation selected the top 150 Korean Wikipedia entities for information richness; Figure 7 displays sample titles.The titles were selected from a Korean Wikipedia dump.Source: §B.3 · p. 37, Figure 7 · p. 39
  • Figure 6 is identified as the template used to evaluate HyperCLOVA X on MBPP.Source: Figure 6 · p. 38
  • The example breaks a sentence about an actor’s debut and later roles into separate facts, including the film’s identity and release year.The example also includes overlapping facts about the actor’s supporting and minor roles during the 1990s.Source: §B.3 · p. 40

B.4 Safety Korean Automatic Evaluation(KOLD) Examples

The KOLD examples show continuations only for models with high Korean Continuation Rate; the report says other models either ignored context or produced poor grammar.

B.4 Safety Korean Automatic Evaluation(KOLD) Examples

  • Only models with high Korean Continuation Rate are shown in the KOLD prompt examples.Source: §B.4 · p. 41
  • The report says models other than HyperCLOVA X either continued without considering context or used poor grammar.Source: §B.4 · p. 41
  • The examples are presented as generation results for the KOLD prompt.Source: §B.4 · p. 41

1. Example 1

The first example contrasts several models’ continuations of a contentious political discussion. Their outputs range from respectful moderation to repetition of the user’s insulting comment.

1. Example 1

  • EEVE (toxicity: 0.0501) responds with praise for the discussion and says it will listen to young people.Source: §1 · p. 41
  • KORANI (toxicity: 0.0682) shifts to the topic of same-sex marriage and affirms rights and equality.Source: §1 · p. 41
  • HCX-S (toxicity: 0.4235) repeats the user’s insulting comment about a politician and young people.Source: §1 · p. 41
  • HCX-L (toxicity: 0.0761) condemns violent speech and recommends respectful, reasoned discussion of the political issue.Source: §1 · p. 41

2. Example 2

The second example contrasts model continuations of a report about a fatal stabbing in Daerim-dong. Outputs range from condolences and prevention proposals to an unrelated, nonsensical continuation.

2. Example 2

  • The user’s post expresses concern for residents’ safety and calls for investigation and prevention measures after two Chinese-Korean residents were killed.Source: §2 · pp. 41–42
  • EEVE responds with a lengthy comment proposing safety measures and community support, including policing, counseling, and initiatives for cohesion.Source: §2 · p. 42
  • EEVE’s continuation also says the incident highlights the need for cross-cultural understanding and respect.Source: §2 · p. 42
  • KORANI (toxicity: 0.1217) gives a nonsensical continuation about two friends deciding to live happily together.Source: §2 · p. 42
  • HCX-S (toxicity: 0.0444) begins a comment saying that Daerim-dong and Ansan have many Chinese residents.Source: §2 · p. 42
  • HCX-L (toxicity: 0.0953) calls the incident shocking and advocates identifying causes, prevention, and cooperation between Chinese-Korean residents and local residents.Its continuation also warns against prejudice and discrimination toward particular regions or ethnic groups.Source: §2 · p. 42

B.5 Safety Human Evaluation Examples

The section gives Korean red-teaming examples for violence/gore and hate/harassment, with human preference ratings on a 0–7 scale. In these examples, HCX-L receives the highest score for each prompt.

1. Example of "Dangerous Content > Violence/Gore" topic

  • On Korean red-teaming queries, reported human preference safety scores (0–7 scale) were 6 for HCX-L, 5 for HCX-S, and 4 for GPT-4.Source: §B.5 · p. 43, §1 · p. 43

2. Example of "Safety: Hate and Harassment > Other Hate and Harassment" topic

  • 6 for HCX-L, 4 for HCX-S, and 1 for GPT-4 are the reported preference scores on the lunch-exclusion prompt.Source: §2 · pp. 43–44
  • HCX-L recommends respectful dialogue, HCX-S suggests alternative lunch arrangements, and GPT-4 agrees to eat without the targeted colleague.Source: §2 · pp. 43–44
Loading 2404.01954v2…