PaperlayerView exact revision on arXiv

Source-linked AI summary

What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers

Boseop Kim, HyoungSeok Kim, Sang-Woo Lee, Gichang Lee, Donghyun Kwak, Dong Hyeon Jeon, Sunghyun Park, Sungju Kim, Seonhoon Kim, Dongpil Seo, Heungsub Lee, Minyoung Jeong, Sungjae Lee, Minsub Kim, Suk Hyun Ko, Seokhun Kim, Taeyong Park, Jinuk Kim, Soyoung Kang, Na-Hyeon Ryu, Kang Min Yoo, Minsuk Chang, Soobin Suh, Sookyo In, Jinseong Park, Kyungduk Kim, Hiun Kim, Jisu Jeong, Yong Goo Yeo, Donghoon Ham, Dongju Park, Min Young Lee, Jaewook Kang, Inho Kang, Jung-Woo Ha, Woomyoung Park, Nako Sung

arXiv:2109.04650v2cs.CL

TL;DR

The paper addresses gaps in evidence about non-English in-context learners, model sizes, and prompt optimization. It introduces Korean HyperCLOVA and reports state-of-the-art Korean zero- and few-shot performance, with prompt-based learning offering further gains.

  • Problem

    GPT-3’s training corpus is 92.7% English, and evidence is limited on non-English in-context learners, intermediate model sizes, and gradient-dependent prompt optimization.

    Source: §1 · p. 1

  • Method

    82B-parameter HyperCLOVA is trained on a Korean-centric 561B-token corpus and uses morpheme-aware byte-level BPE tailored to Korean.

    Source: §1 · p. 2, §3.3 · p. 4

  • Results

    HyperCLOVA achieves state-of-the-art Korean in-context zero- and few-shot performance, which prompt-based learning can further improve.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    HyperCLOVA Studio lets nondevelopers prototype AI-backed products, and the authors argue that such a framework could support a No Code AI paradigm.

    Source: §6 · p. 9

    Large-scale LMs can be misused, and the authors identify privacy risks from incomplete preprocessing and biased or repulsive responses on sensitive social values.

    Source: Broader Impact Statement · p. 10

Abstract

from arXiv · show

GPT-3 shows remarkable in-context learning ability of large-scale language models (LMs) trained on hundreds of billion scale data. Here we address some remaining issues less reported by the GPT-3 paper, such as a non-English LM, the performances of different sized models, and the effect of recently introduced prompt optimization on in-context learning. To achieve this, we introduce HyperCLOVA, a Korean variant of 82B GPT-3 trained on a Korean-centric corpus of 560B tokens. Enhanced by our Korean-specific tokenization, HyperCLOVA with our training configuration shows state-of-the-art in-context zero-shot and few-shot learning performances on various downstream tasks in Korean. Also, we show the performance benefits of prompt-based learning and demonstrate how it can be integrated into the prompt engineering pipeline. Then we discuss the possibility of materializing the No Code AI paradigm by providing AI prototyping capabilities to non-experts of ML by introducing HyperCLOVA studio, an interactive prompt engineering interface. Lastly, we demonstrate the potential of our methods with three successful in-house applications.

1 Introduction

The paper introduces HyperCLOVA to investigate non-English in-context learning, model-size effects, and gradient-based prompt optimization, then presents results and an interactive studio. It reports state-of-the-art Korean zero- and few-shot performance and argues that the studio could support No Code AI.

1 Introduction

  • The study addresses GPT-3’s English-heavy training mix, limited analysis of model sizes between 13B and 175B, and untested gradient-based prompt optimization for in-context learners.The paper notes that English accounts for 92.7% of GPT-3’s training corpus.Source: §1 · p. 1
  • HyperCLOVA is an 82B-parameter Korean in-context LM trained on a 561B-token Korean-centric corpus, with tokenization designed for Korean linguistic characteristics.The authors identify Korean-specific tokenization as important to downstream task performance.Source: §1 · p. 2
  • HyperCLOVA achieves state-of-the-art in-context learning performance on Korean datasets in zero- and few-shot settings.The paper also reports outstanding classification and generation results when HyperCLOVA is used with p-tuning.Source: §1 · p. 2
  • HyperCLOVA Studio gives nonexperts GUI and API prompt-engineering interfaces for prototyping, and the authors describe three in-house application scenarios.They report qualitatively human-expert-comparable performance in those scenarios with minimal domain-expert effort.Source: §1 · p. 2
  • Prompt-based tuning improves performance and can outperform state-of-the-art models on downstream tasks when backward gradients of inputs are available.The authors investigate p-tuning with 39B- and 82B-parameter models.Source: §1 · p. 2

2 Previous Work

The related work covers prompt optimization and language-specific modeling. It describes prior approaches to discrete and continuous prompts while identifying limited exploration of non-English in-context learners.

2.1 Prompt Optimization

  • Prompt methods seek prompts that elicit knowledge and improve predictions, but language models’ sensitivity to prompt design motivates optimization methods.Prior work presents prompts as a time- and space-efficient alternative to full fine-tuning as models scale.Source: §2.1 · p. 2
  • Discrete prompt optimization works in token space and can transfer, but prior work reports poor interpretability and suboptimality; continuous methods optimize contextualized prompts without updating main-LM parameters.Prior studies reported p-tuning outperforming MLM-based fine-tuning on certain tasks and optimized prompts reaching state of the art on key benchmarks.Source: §2.1 · pp. 2–3

2.2 Language Models

  • Language-specific LMs remain in demand for their reported edge over language-agnostic models, while high costs limit their availability outside English..Source: §2.2 · p. 3
  • Non-English in-context learners were largely unexplored; concurrent Chinese work reported 2.6B- and 13B-parameter models and ongoing work on a 207B model.The authors frame the broader research as focused on a few major languages.Source: §2.2 · p. 3

3 Pre-training

The pre-training section describes HyperCLOVA’s Korean-centric data, GPT-3-like decoder models and training setup, and a morpheme-aware tokenization strategy tailored to Korean.

3.1 Data Description

  • The authors collected and refined 561B tokens for the final corpus, drawing on available text including user-generated content and external-partner content.They report that Korean data make up less than 0.02% of GPT-3 by character count, motivating a Korean-centric corpus.Source: §3.1 · p. 3

3.2 Model and Learning

  • HyperCLOVA uses a GPT-3-like transformer decoder and includes mid-size models intended to clarify model properties and support practical applications.The authors describe these sizes as more plausible for real-world use than larger models.Source: §3.2 · pp. 3–4
  • Training uses Megatron-LM on 1,024 A100 GPUs; the 82B model takes 13.4 days to train with 150B tokens, while Studio experiments use models trained with 300B tokens.The authors report scaling-law behavior on a held-out encyclopedia corpus as model size and training duration increase.Source: §3.2 · p. 4

3.3 Korean Tokenization

  • Because Korean is agglutinative, tokenizing morphemes such as nouns with particles and verb or adjective stems with endings can clarify token semantics.The authors use morpheme-aware byte-level BPE, applying morpheme analyzers to address non-English characters being split into separate Unicode bytes.Source: §3.3 · p. 4
  • The tokenization pipeline pre-splits sentences by spaces and analyzer-derived morphemes before applying byte-level BPE.The analyzer excludes most non-Korean characters.Source: §3.3 · p. 4

4 Experimental Results

The experiments evaluate Korean in-context learning across several tasks, prompt-based tuning for classification and generation, and Korean-specific tokenization. Results show performance generally improves with model size, while p-tuning and morpheme-aware tokenization offer benefits in their tested settings, with task-specific exceptions.

4.1 Experimental Setting

  • Evaluation covered six few-shot tasks, including sentiment, question answering, translation, topic classification, and semantic similarity, with comparisons to task-specific baselines.KorQuAD used a zero-shot passage/four-shot question setup; translation used four-shot examples and BLEU, while several classification evaluations averaged trials.Source: §4.1 · pp. 4–5, Table 3 · p. 5

4.2 In-context Few-shot Learning

  • In-context performance increased monotonically with model size across tasks, but Ko→En translation and KLUE-STS remained below their baselines.The authors conjecture that the Ko→En result may reflect the corpus’s low English ratio.Source: §4.2 · p. 5, Table 3 · p. 5

4.3 Prompt-based Tuning

  • On NSMC, p-tuning outperformed comparison models without updating HyperCLOVA’s main parameters.With 4K examples, p-tuning achieved results comparable to RoBERTa fine-tuned on 150K examples.Source: §4.3 · p. 5, Table 4 · p. 6
  • For query modification, input-side p-tuning consistently improved query quality in both zero-shot and three-shot settings.The authors report this as the first application of input-side p-tuning to generation tasks with an in-context LM learner; they note that prompt optimization is feasible for practitioners when backward gradients are accessible.Source: §4.3 · pp. 5–6, Table 5 · p. 6

4.4 Effect of Tokenization

  • Morpheme-aware byte-level BPE improved most tested tasks, while its morpheme analyzer worsened Ko→En translation.Char-level BPE performed much worse than byte-level BPE on YNAT; the authors attribute this to out-of-vocabulary tokens obscuring important headline words.Source: §4.4 · p. 6

5 Discussion on Industrial Impacts

The discussion presents HyperCLOVA Studio as a way to prototype AI products with less ML-engineer involvement and potentially reduce communication overhead. Its in-house cases and proposed No/Low Code AI workflow illustrate that possibility.

5.1 HyperCLOVA Studio

  • HyperCLOVA Studio offers GUI and API access to a shared artifact, enabling rapid AI-service prototyping with minimal ML-engineer involvement.The API supports functions beyond those available in the OpenAI Playground.Source: §5.1 · p. 7

5.2 Case Studies on HyperCLOVA Studio

  • The three in-house cases address tasks with hard-to-define objectives or automatic evaluation, and designers without programming or AI knowledge made PoCs within hours..Source: §5.2 · p. 7

5.2.1 Rapidly Prototyping Chatbots with Personalities

  • Designers built character-persona chatbots from brief descriptions and dialogue examples, and could adjust style through prompt examples.The authors describe the process as supporting rapid development of varied dialogue systems through a human-in-the-loop process.Source: §5.2.1 · p. 7

5.2.2 Zero-shot Transfer Data Augmentation

  • HyperCLOVA generated varied utterances for intent classes using source-domain examples, helping product-quality teams validate dialogue systems with complex scenarios.The experiment used 20 target classes and six source classes with five examples each; Figure 1 shows generated examples across three tasks.Source: §5.2.2 · pp. 7–8, Figure 1 · p. 8

5.2.3 Event Title Generation

  • HyperCLOVA titles were rated comparable to ground-truth titles, and prompt changes let designers adapt themes and domains.Nine human experts chose among titles from ground truth, mT5, and HyperCLOVA; the advertisement-headline task reached 99% appropriateness for the real service.Source: Table 8 · p. 8, §5.2.3 · p. 8
  • Five prompt examples let an event designer generate sales-title candidates in under ten minutes, with output quality comparable to human-written titles.Designers could control the theme emphasized for a keyword, and prompt modification allowed adaptation to other event domains.Source: §5.2.3 · p. 8

5.3 Opportunity of HyperCLOVA Studio

  • Studio extensions include input gradients for local prompt optimization, prompt injection for retrieval-supported open-domain QA, and input/output filters to help prevent misuse.The input-gradient API is proposed as a way to train a prompt encoder on local machines.Source: §5.3 · pp. 8–9

5.4 No/Low Code AI Paradigm

  • The proposed workflow combines example curation, prompt design, API tuning, and integration, potentially reducing communication costs and enabling prototypes within hours.The authors distinguish No Code for rapid PoC iteration or pure generation from Low Code when training data or extra modules are needed.Source: §5.4 · p. 9

6 Conclusion

The conclusion presents Korean-centric HyperCLOVA models, highlighting the 82B model’s in-context performance and prompt-based improvement. It proposes HyperCLOVA Studio as a route toward No Code AI, while acknowledging both opportunities and challenges.

6 Conclusion

  • The paper presents Korean-centric models and aims to help people unfamiliar with machine learning build their own AI models through HyperCLOVA Studio.Source: §6 · p. 9
  • HyperCLOVA’s 82B model shows state-of-the-art Korean in-context zero-shot and few-shot performance, further boosted by prompt-based learning.The conclusion describes Studio as a way for non-developers to build AI-backed products and frames No Code AI as a possibility.Source: §6 · p. 9

Broader Impact Statement

The broader-impact statement describes risks from misuse, biased responses, and energy use in large-scale language models. It also presents No/Low Code AI as a way for non-specialists to prototype systems rapidly, while calling for efforts to address the risks.

Broader Impact Statement

  • Large-scale language models can be misused, and incomplete privacy-preserving preprocessing in Luda made privacy issues more critical from a legal perspective.The statement also notes that hate-speech data can lead to malicious misuse and that GPT-3 API applications have reported problematic generations.Source: Broader Impact Statement · p. 10
  • Training-data biases can significantly influence large-scale language models, with reported biased responses involving gender and racism.The statement identifies filtering and data refinement for debiasing as approaches researchers argue are needed.Source: Broader Impact Statement · p. 10
  • Scaling model parameters and training data for better performance inevitably worsens the energy issue.The statement names energy-efficient hardware such as FPGA as a plausible alternative.Source: Broader Impact Statement · p. 10
  • No/Low Code AI could let industrial engineers and service designers unfamiliar with ML rapidly build simple systems or prototypes, supporting AI democratization.The authors report that service designers used HyperCLOVA Studio outputs and data in AI services, while calling for efforts to mitigate associated problems.Source: Broader Impact Statement · p. 10

A.1 Data Description

The corpus is predominantly Korean and draws substantially on blogs, community sites, and news. Its language mix also includes smaller English, Japanese, and other-language portions.

A.1 Data Description

  • Blogs, community sites, and news account for 49%, 15%, and 13% of the corpus, respectively; comments contribute 7% and KiN 5%.The corpus also includes a very small Korean Wikipedia share of 0.04%.Source: §A.1 · p. 13
  • The corpus language proportions are 97% Korean, 2% English, 0.5% Japanese, and 0.5% other languages.The authors also use English and Japanese Wikipedia to enhance foreign-language ability.Source: §A.1 · p. 13

A.2 Data Cleaning

The appendix describes how HyperCLOVA’s corpus was cleaned, anonymized, and postprocessed, and reports scaling-law patterns and performance statistics for Korean evaluation tasks.

A.2 Data Cleaning

  • Corpus cleaning removes low-quality, duplicated, spam, and repetitive documents, with additional filtering for KiN answers and duplicated title-content sentences.The pipeline uses a BERT-based quality classifier, hash-based similarity checks, and in-house spam filtering.Source: §A.2 · p. 13

A.3 Data Anonymization

  • Personal identifiers are masked while non-critical information such as age, gender, location, phone dialing codes, and email domains is retained.The passage lists resident registration, contact, financial, passport, and driver’s license numbers among the masked information.Source: §A.3 · p. 13

A.4 Data Postprocessing

  • Document preprocessing adds a prefix specifying the source, title, and body, following CTRL.The prefix format is “Web, Title: ${title_name}, Body: ${body_text}”.Source: §A.4 · p. 13

B Scaling Law

  • HyperCLOVA’s training and Korean encyclopedia test losses follow the GPT-3 scaling-law pattern.Figure 2 shows training and test loss patterns across models with various parameter counts.Source: B Scaling Law · p. 13, Figure 2 · p. 14

C.1 NSMC

  • The appendix points to performance statistics for HyperCLOVA on NSMC.Source: §C.1 · p. 13

C.2 AI Hub Translation

  • The appendix reports mean, standard deviation, and maximum accuracy statistics for HyperCLOVA on AI Hub translation tasks.Source: §C.2 · p. 13, Table 10 · p. 14

C.3 Query Modification Task

The appendix presents HyperCLOVA query modifications and their prompts, alongside discussion of persona-chatbot style control and hallucination.

C.3 Query Modification Task

  • The query-modification prompt uses a continuous prompt token denoted [P].Source: §C.3 · p. 14

C.4 Discussions on Making Persona Chatbot

  • Existing conversational systems use substantial data and do not readily support creating a new conversational style, while style-transfer methods do not control detailed style.Source: §C.4 · p. 14
  • Retrieved knowledge can alleviate hallucination, and strong open-domain question answering in the pretrained LM can benefit a pretrained reader.Source: §C.4 · p. 14

C.3 Query Modification Task

  • The query-modification examples show user queries modified by HyperCLOVA, with English translations supplied by a human expert.Source: Table 11 · p. 14, §C.3 · p. 14

C.5 Zero-shot Transfer Data Augmentation

The section reports zero-shot transfer data augmentation results and describes applications to advertisement headline generation, HyperCLOVA Studio, and Korean tokenization.

C.5 Zero-shot Transfer Data Augmentation

  • HyperCLOVA can generate candidate utterances for corpus designers to select, and the authors report that dialect generation or standard-to-dialect conversion is also readily available.Source: §C.5 · pp. 14–15
  • Sentences were generated well for 18 classes (80% ∼90%), but not well for 2 classes (10% ∼20%).The authors contrast this work with Schick and Schütze (2021), which they say applies only to NLI.Source: §C.5 · p. 15

C.6 Advertisement Design

  • A product designer judged 99% of generated advertisement headlines appropriate for the real service; three prompts were used, and the output most similar to the product name was selected.Similarity was calculated using cosine similarity on an in-house BERT feature.Source: §C.6 · p. 15

D Details on Studio

E Motivation of Tokenization

  • Morpheme-aware tokenization is motivated as important, and an open-source morpheme analyzer such as Mecab-ko is identified as an alternative to the in-house analyzer.Source: E Motivation of Tokenization · p. 15

F Challenges of No/Low Code AI Paradigm

The section outlines limits and costs of No/Low Code AI, including corpus dependence, remaining ML expertise, and the need for further Studio capabilities.

F Challenges of No/Low Code AI Paradigm

  • Task-specific neural structures can outperform general large LMs on some tasks, and the competitiveness of prompt-based methods for large LMs remains underexplored.The passage gives FiD’s state-of-the-art open-domain QA performance as an example contrasted with GPT-3.Source: F Challenges of No/Low Code AI Paradigm · p. 16
  • Prompt-based optimization cannot justify expecting code generation when the pretraining corpus contains no code-generation data.The passage says Studio maintainers may discover common user needs and further train the corpus; incorporating such corpora calls for continual-learning pretraining research.Source: F Challenges of No/Low Code AI Paradigm · p. 16
  • Completing a full AI system still requires programming, and effective prompts and few-shot examples implicitly require ML knowledge.The passage mentions easier Studio guidance and incentives for sharing prompts as possible ways to spread the ecosystem.Source: F Challenges of No/Low Code AI Paradigm · p. 16
  • Full-fledged ML development also needs Studio experimentation and user-feedback features, including a way to distribute proof-of-concept services and collect response feedback.Source: F Challenges of No/Low Code AI Paradigm · p. 16
  • Large-scale LM inference and prompt-optimization costs remain obstacles, with distillation and energy-efficient hardware identified as important for sustainability.The passage notes a cost trade-off between training many small models and inference with one large model, whose outputs can also feed small models.Source: F Challenges of No/Low Code AI Paradigm · p. 17

Related papers

Loading 2109.04650v2…