PaperlayerView exact revision on arXiv

Source-linked AI summary

KLUE: Korean Language Understanding Evaluation

Sungjoon Park, Jihyung Moon, Sungdong Kim, Won Ik Cho, Jiyoon Han, Jangwon Park, Chisung Song, Junseong Kim, Yongsook Song, Taehwan Oh, Joohong Lee, Juhyun Oh, Sungwon Lyu, Younghoon Jeong, Inkwon Lee, Sangwoo Seo, Dongjun Lee, Hyunwoo Kim, Myeonghwa Lee, Seongbo Jang, Seungwon Do, Sunkyoung Kim, Kyungtae Lim, Jongwon Lee, Kyumin Park, Jamin Shin, Seonghyun Kim, Lucy Park, Alice Oh, Jung-Woo Ha, Kyunghyun Cho

arXiv:2105.09680v4cs.CL

TL;DR

Korean NLU lacked a unified, accessible benchmark that reflected the language’s diverse tasks and characteristics. KLUE constructs eight tasks from scratch with carefully designed, ethically informed protocols and releases baselines and Korean pretrained models. The benchmark reports stronger performance from Korean-specific models overall, while its construction and documentation support reproducible Korean NLP research.

  • Problem

    Korean NLU lacks a unified benchmark, limiting standardized evaluation of progress in the language.

  • Method

    KLUE constructs eight Korean NLU tasks from diverse corpora with tailored annotation protocols, evaluation metrics, and released pretrained-model baselines.

  • Results

    Korean-specific language models generally outperform multilingual models across KLUE, with different models performing best on different tasks.

  • Takeaways & Limitations

    KLUE provides an accessible, ethically considered benchmark and documented construction process for advancing Korean NLP research.

  • Takeaways & Limitations

    KorNLI relies on machine translation with post-editing but lacks human validation to quantify translation errors, including unnatural Korean sentences.

Abstract

from arXiv · show

We introduce Korean Language Understanding Evaluation (KLUE) benchmark. KLUE is a collection of 8 Korean natural language understanding (NLU) tasks, including Topic Classification, SemanticTextual Similarity, Natural Language Inference, Named Entity Recognition, Relation Extraction, Dependency Parsing, Machine Reading Comprehension, and Dialogue State Tracking. We build all of the tasks from scratch from diverse source corpora while respecting copyrights, to ensure accessibility for anyone without any restrictions. With ethical considerations in mind, we carefully design annotation protocols. Along with the benchmark tasks and data, we provide suitable evaluation metrics and fine-tuning recipes for pretrained language models for each task. We furthermore release the pretrained language models (PLM), KLUE-BERT and KLUE-RoBERTa, to help reproducing baseline models on KLUE and thereby facilitate future research. We make a few interesting observations from the preliminary experiments using the proposed KLUE benchmark suite, already demonstrating the usefulness of this new benchmark suite. First, we find KLUE-RoBERTa-large outperforms other baselines, including multilingual PLMs and existing open-source Korean PLMs. Second, we see minimal degradation in performance even when we replace personally identifiable information from the pretraining corpus, suggesting that privacy and NLU capability are not at odds with each other. Lastly, we find that using BPE tokenization in combination with morpheme-level pre-tokenization is effective in tasks involving morpheme-level tagging, detection and generation. In addition to accelerating Korean NLP research, our comprehensive documentation on creating KLUE will facilitate creating similar resources for other languages in the future. KLUE is available at https://klue-benchmark.com.

1 Introduction

KLUE addresses the lack of a unified Korean NLU benchmark by constructing an accessible, ethically considered suite spanning diverse tasks, domains, and linguistic phenomena.

  • Motivation: KLUE fills the gap of a unified Korean NLU benchmark by building the suite from scratch rather than adapting existing tasks or corpora.The construction process covers corpus selection, task identification, annotation design, and validation.
  • Coverage: The benchmark spans diverse domains and styles, including news, encyclopedic text, reviews, smart-home queries, task-oriented dialogue, and formal and colloquial language.This breadth supports analysis across multiple domains and writing styles.
  • Accessibility: KLUE uses freely copyable, redistributable, remixable, and transformable resources so the benchmark is accessible without restriction.Its source corpora are selected to permit derivative work, redistribution, and commercial use.
  • Task selection: Eight tasks cover diverse aspects of Korean NLU, including classification, similarity, inference, entity and relation analysis, parsing, reading comprehension, and dialogue tracking.The selected tasks are intended to maximize coverage while minimizing redundancy.
  • Annotation: Annotation guidelines are iteratively refined to improve accuracy and reflect Korean linguistic properties, including agglutination and task-specific tagging conventions.Examples with poor annotator agreement are often filtered out, and guidelines address Korean NER, POS tagging, and dependency parsing.
  • Ethics: KLUE mitigates ethical risks by filtering or excluding noisy, toxic, socially biased, and personally identifiable content during corpus preparation and annotation.The benchmark explicitly incorporates ethical considerations into data selection and annotation protocols.

2 Source Corpora

KLUE constructs its source corpora from scratch using accessibility, quality, diversity, and ethical criteria. The selected resources support redistribution, derivatives, and broad use while documenting corpus characteristics and addressing problematic content.

  • Corpus construction: KLUE builds its benchmark from scratch rather than assembling existing datasets, documenting corpus-selection decisions and protocols.The process follows datasheets and data statements to describe how and why corpora were selected.
  • Selection criteria: Source selection prioritizes unrestricted accessibility, contemporary Korean, quality, domain diversity, and suitability for annotation.The final set also balances formal and colloquial language and avoids narrow-domain concentration.
  • Accessibility and licensing: KLUE permits non-commercial and commercial use, redistribution with attribution, and derivative works to support broad and continued reuse.The source corpus is released under CC BY-SA, while selected resources are drawn from permissive licenses, copyright-exempt materials, or explicit contracts.
  • Selected corpora: The source collection spans formal and colloquial Korean across news, encyclopedia, government, smart-home, review, and other domains.Examples include News Headlines, Wikipedia, Wikinews, Policy News, Wikitree, ParaKQC, and Airbnb Reviews.
  • Documentation: KLUE documents each corpus’s collection mechanism, timeframe, domain, style, license, and background.Table 2 identifies source-corpus legal status, license categories, volume ranges, and the corpora ultimately used for KLUE.
  • Corpus concerns: Wikitree is included for its scale and permissive distribution despite concerns about low quality, clickbait, advertising, and social bias.KLUE addresses these concerns through additional inspection, excludes Wikitree headlines from Topic Classification and article contents from Machine Reading Comprehension, and discards problematic sentences via annotation.

3 KLUE Benchmark

KLUE provides eight Korean NLU benchmarks built from diverse corpora with documented construction, annotation, metrics, and ethical safeguards. The task-specific resources emphasize accessibility, Korean linguistic characteristics, and annotation reliability while identifying remaining artifacts and scope gaps.

  • KLUE documents source selection, annotation, dataset splitting, metric design, and ethical review across eight Korean NLU benchmarks.
  • Semantic Textual Similarity: KLUE-STS is a freely accessible, human-annotated Korean semantic similarity benchmark spanning multiple domains and styles.
  • Limitations: The NLI construction has known artifacts: sentence length and lexical patterns, including negation, are associated with particular classes.
  • Natural Language Inference: KLUE-NLI uses naturally occurring Korean premises and worker-written hypotheses, producing more reliable labels than existing translated or English datasets.Validation reports an 18% higher unanimous-gold rate than SNLI and MNLI, while human majority agreement reaches 98%.
  • Other Tasks: KLUE also addresses Korean NER and relation extraction gaps by providing a freely accessible, broad-domain NER benchmark and a large-scale Korean RE dataset.The NER design requires morphological and contextual cues, while RE targets relationships between entities.

1. Collect Candidate Sentences

Candidate sentences for KLUE-RE are drawn from multiple Korean corpora and filtered for entity coverage and ethical concerns.

  • Candidate sentences are sampled from WIKIPEDIA, WIKITREE, and POLICY to cover diverse named entities and relational facts.
  • Sentences containing social bias or hate speech are filtered using a classifier trained on a Korean hate-speech dataset.

2. Define Relation Schema

KLUE-RE defines a relation schema adapted from TAC-KBP to fit Korean corpus characteristics and organizes 30 relation classes.

  • The schema adapts TAC-KBP by defining entity types and restricting subjects to PER or ORG.
  • Objects may be PER, ORG, LOC, DAT, POH, or NOH under the schema.
  • The schema contains 30 relation classes comprising 18 person-related relations, 11 organization-related relations, and no_relation.
  • Rare relations are removed, related organization classes are merged, and Korean regional hierarchy labels are integrated into place_of.
  • The schema includes product relations for organizations and persons, plus a colleague relation for people who work together or share a group.

3. Detect Entities

KLUE-RE detects entities automatically with Korean NER models and then forms entity pairs from the detected entity set.

  • Two Korean NER models are built by fine-tuning ELECTRA on separate existing Korean NER resources.
  • The source-resource entity types are modified to match the entity types defined in the KLUE-RE schema.
  • Two distinct approaches are used to select entity pairs from a sentence’s entity set: KB-based sampling and uniform sampling.

4. Select Entity Pairs

KLUE-RE selects entity pairs through knowledge-base-based or uniform sampling, then obtains relation labels from crowdsourced annotation.

  • KB-based sampling considers entity pairs appearing in a triplet pool gathered from a Korean knowledge base, Wikipedia, and Namuwiki infoboxes.
  • Uniform sampling randomly selects two entities from a sentence, producing realistic pairs that are often irrelevant and labeled no_relation.
  • Workers recruited through DeepNatural annotate each entity pair with a relation label while focusing on current context and avoiding external knowledge.

5. Annotate Relations

KLUE develops relation extraction and dependency parsing resources alongside diverse machine reading comprehension data, using Korean-specific annotation, formats, and evaluation choices.

  • 32,470 training, 7,765 development, and 7,766 test examples comprise KLUE-RE, with unseen test entities and uniform sampling for development and test.
  • KLUE-RE combines micro F1 on meaningful relations with AUPRC over all classes to assess fine-grained and holistic extraction quality.
  • Dependency parsing represents each word with HEAD and DEPREL information and is evaluated using UAS and LAS.
  • KLUE-DP contains 14,500 balanced formal-news and informal-review sentences, split into 10,000 training, 2,000 development, and 2,500 test sentences.
  • KLUE-MRC includes multi-domain passages and paraphrase, multi-sentence reasoning, and unanswerable question types designed to reduce shortcut reasoning.

1. Common Guidelines

KLUE’s common annotation guidelines target natural, unambiguous questions grounded in passages while reducing lexical shortcuts and answer artifacts.

  • Workers are instructed to write questions that resemble natural web-search queries while preserving generalizability to open-domain question answering.
  • Questions must retain grammatical components because Korean pronoun dropping can create ambiguity when locating answers.
  • Questions should avoid copying passage phrases to mitigate high word overlap between questions and passages.
  • Questions must be answerable from the passage alone and cannot rely on external or world knowledge.
  • Questions should require understanding of the whole question rather than a small cue, and answers should be unique within the passage.
  • Guidelines prohibit questions whose answer is simply the passage’s main topic or title, preventing frequency-based answer artifacts.

2. Type-Specific Guidelines

KLUE-MRC uses three question types and targeted construction rules to test paraphrase understanding, multi-sentence reasoning, and answerability rather than surface matching.

  • 2.1. Question Paraphrasing (Type 1): Paraphrase questions evaluate semantic understanding by reducing lexical overlap with passage sentences.
  • 2.1. Question Paraphrasing (Type 1): Paraphrase construction requires syntactic or lexical variation, with more than half of the question words not overlapping the corresponding passage sentence.
  • Type 2: Multi-sentence reasoning questions require aggregating information from at least two passage sentences to infer an answer span.
  • Type 2: Their construction identifies shared properties across statements, selects an answer entity, and generates a question around it.
  • Type 3: Unanswerable questions include passage-relevant entities and plausible distractors but no correct answer within the passage.
  • 29,313 KLUE-MRC examples span 22,343 documents and 23,717 passages, with 17,554 training, 5,841 development, and 5,918 test examples.
  • KLUE-MRC is reported as more challenging than KorQuAD 1.0 even after matching training-set sizes.
  • KLUE-MRC’s lexical overlap is nearly 10%p lower than KorQuAD’s 70%, while Type 2 questions reach 68% overlap.

1. Defining Task Schema

The WoS task schema organizes task-oriented dialogue across five domains and enriches slots with attributes that support precise annotation.

  • The schema covers hotel, restaurant, attraction, taxi, and metro domains, each with domain-specific slots.
  • Informable slots constrain user goals, whereas requestable slots provide information users may ask for without narrowing the goal.
  • Slots may be marked as boolean, required, booking-related, or requestable only after booking confirmation.

2. Creating Knowledge Base

KLUE constructs dialogue knowledge bases and goal instructions to support realistic, multi-domain task-oriented conversations. System-side workers annotate dialogue states and respond using knowledge-base information when needed.

  • Knowledge-base construction: KLUE constructs domain-specific knowledge bases with predefined realization candidates, combining manually created instances, web-collected names, and dynamically generated taxi instances.Hotels and restaurants use virtual instances; attractions and metro use real names, while taxi instances are generated during collection.
  • Goal instructions: Goal instructions specify user goals, slot values, and persona context, guiding workers to generate varied user utterances.The instructions are written in natural language and include corresponding slot values and user context.
  • Goal instructions: Multi-domain scenarios include domain transitions, such as moving from hotel booking to transportation, making dialogue state tracking more challenging.Shared slots and changing domains require tracking goals across multiple conversational contexts.
  • Goal instructions: Templates realize goal instructions with diverse domain-specific sentences, subgoals, slot values, lexical entailment, and coreference patterns.The templates are designed to cover varied dialogue scenarios and naturally express relationships across turns.
  • System-side annotation: System-side workers infer slot values, clarify ambiguous utterances, annotate dialogue states, and generate responses using the knowledge base.Workers may request missing slot values, provide information, or ask for more details when search results are numerous.
  • System-side annotation: A graphical interface with dropdown candidates supports system workers by presenting likely slot values from goal instructions and domain knowledge.The interface limits displayed options to probable candidates so workers can annotate efficiently.

4. Dataset Construction

KLUE’s dataset construction combines trained worker procedures, scalable self-dialog collection, diverse Korean pretraining resources, and challenging evaluation splits. Wizard-of-Seoul provides a large multi-domain dialogue dataset with counterfactual goals and unseen knowledge-base instances.

  • Dialogue collection: Self-dialog lets one worker control both user and system roles, reducing collection cost while supporting varied dialogue flows and personal speaking styles.The authors use pilot studies, trustworthy-worker selection, and explicit turn switching to reduce annotation errors and encourage realistic miscommunications.
  • Wizard-of-Seoul: Wizard-of-Seoul contains 10,000 dialogues and 146,692 turns across five domains.Its dev and test sets include 294 and 361 counterfactual-goal dialogues, respectively, and all splits include domain transitions.
  • Wizard-of-Seoul: Splitting evaluation data by counterfactual goals and unseen knowledge-base instances causes a performance drop, making Wizard-of-Seoul more challenging.The authors report this comparison in Table 27.
  • Dialogue collection: The Korean task-oriented dialogue setting builds on earlier WOZ, asynchronous, self-dialog, simulator-based, and counterfactual-data collection approaches.These prior methods address the cost, coordination, or robustness limitations of conventional WOZ collection.
  • Wizard-of-Seoul: WoS is introduced as the first large-scale Korean multi-domain task-oriented dialogue dataset simulating Seoul tourists and travel agents.Its collection uses Self-dialog, interface support, and counterfactual goals to promote efficient, natural, and challenging conversations.
  • Language-model pretraining: KLUE pretrains Korean language models using multiple configuration choices and combines five public corpora into an approximately 62GB corpus.The configurations vary corpus, preprocessing, tokenization, and other training settings.
  • Language-model pretraining: Morpheme-based subword tokenization pre-tokenizes Korean text into morphemes before applying BPE, while inference uses only the BPE model for speed and usability.The method uses Mecab-ko for morpheme segmentation and a 32k vocabulary.

5 Fine-tuning Language Models

KLUE fine-tunes pretrained language models with task-specific heads for classification, pairwise prediction, span extraction, tagging, parsing, and multi-turn slot-value prediction. Experiments compare Korean and multilingual models and analyze pseudonymization and tokenization choices.

  • Task formulations: KLUE organizes its eight tasks into four fine-tuning types: single-sentence classification, sentence-pair classification or regression, multi-sentence slot-value prediction, and token-level tagging.The categories cover TC/RE, STS/NLI, DST, and NER/DP/MRC, respectively.
  • Task formulations: Single-sentence classifiers map the [CLS] hidden state to predefined labels and minimize cross-entropy loss.KLUE-RE additionally marks subject and object entities with special tokens.
  • Task formulations: STS predicts a real-valued similarity score with mean squared error, while NLI predicts one of three classes with cross-entropy loss.Sentence pairs are concatenated with a separator token.
  • Task formulations: WoS uses a TRADE-style encoder-decoder with a pretrained-language-model utterance encoder, a state generator, and an expanded slot-gate classifier.The model jointly minimizes state-generator and slot-gate cross-entropy losses and adds yes/no slot-gate labels.
  • Evaluation results: KLUE-RoBERTaLARGE outperforms all tested baseline models on every task except KLUE-NER.Monolingual Korean models generally outperform similarly sized multilingual models, except that XLM-RLARGE is comparable on KLUE-NER character-level F1.
  • Model analysis: Pseudonymization produces only minimal performance degradation on a subset of tasks.The authors interpret minimal pseudonymization as balancing task performance with private-information leakage risk.
  • Model analysis: Morpheme-consistent tokenization improves performance especially for KLUE-NER, KLUE-MRC, and WoS, while differences are not evident for classification or word-level tagging.The authors associate the gains with tagging, detection, and generation involving morpheme-level processing.

6 Ethical Considerations

KLUE addresses ethical and legal risks through licensing choices, dataset filtering, annotation guidance, and pseudonymization. The benchmark datasets are openly released, while the pretraining corpora remain unreleased because their copyright status cannot be guaranteed.

  • Ethical principles: KLUE’s ethical design addresses harmful social bias, hate speech, personally identifiable information, copyright, and accessibility concerns.The paper describes these mechanisms throughout benchmark and model construction.
  • Copyright and access: KLUE uses source texts that are copyright-free, openly licensed, or provided under agreements permitting release and reuse.The source corpus is released under CC BY-SA, with selected licenses including CC0, CC BY, CC BY-SA, and KOGL Type 1.
  • Copyright and access: Task-specific datasets are released under CC BY-SA, allowing copying, redistribution, remixing, transformation, and commercial or non-commercial derivative use under the same license.This licensing approach is intended to facilitate future NLP research and development.
  • Copyright and access: The pretraining corpus may contain copyrighted work despite using publicly available text, so it is used for training but not publicly released.The authors state that sufficiently large Korean pretraining resources are otherwise unavailable.
  • Bias and toxicity: Task-specific datasets reduce toxic content through automatic filtering, annotator instructions, and manual inspection.Hate speech and gender-biased sentences are screened before annotation, while annotators identify additional problematic instances.
  • Bias and toxicity: Pretraining corpora are used as-is because manual inspection is infeasible at scale and reliable Korean toxicity detection tools are limited.The authors also argue that exposure to toxic content may be needed for models to recognize such issues.
  • Privacy: KLUE pseudonymizes detectable PII in pretraining corpora and manually removes or pseudonymizes PII in task-specific resources.For simulated DST dialogues, database entries are pseudonymized with the Faker library.
  • Motivation: Existing Korean datasets often lack unified coverage and may have restrictive or inaccessible licensing, motivating KLUE’s integrated and openly reusable benchmark.KLUE is designed to broaden domain, style, and linguistic coverage while addressing redistribution and transformation barriers.

8 Discussion

KLUE is openly distributed to support Korean NLP research and future benchmark development. The authors discuss limitations involving overall performance aggregation, expected saturation, and incomplete explanations of model behavior.

  • Open Access: KLUE is distributed under CC BY-SA, allowing copying, redistribution, commercial use, and derivative benchmark development.The authors argue these permissions support broader participation and future dataset improvement.
  • Measuring Overall Performance: KLUE evaluates each task separately because its heterogeneous metrics and granularities make a single averaged score misleading.The authors leave a unified overall-performance measure for future work.
  • Rapid Saturation of KLUE: The authors expect KLUE to saturate rapidly rather than artificially filtering easy examples to enlarge improvement headroom.They instead rely on open licensing to support collectively developed, more challenging tasks after saturation.
  • Analysis of Korean Language Models: Many performance patterns among Korean language models remain understudied and difficult to explain precisely.Future investigations are intended to examine interactions among models, corpora, Korean linguistic properties, and training mechanisms.
  • Facilitating Korean NLP Research: KLUE and its accompanying Korean language models are intended to facilitate future Korean NLP research through openly documented benchmark and model development.The paper presents this as a precedent for creating and sharing similar resources more broadly.

Contribution

The contribution section records the distributed project work behind KLUE, spanning dataset construction, modeling, corpus collection, ethical review, management, and manuscript preparation.

  • Project Management: Project managers Sungjoon Park and Jihyung Moon coordinated overall progress, resources, datasets, models, and ethical concerns.Their roles also included IRB preparation, article licensing, and manuscript responsibilities.
  • Dataset and Modeling: Dataset and modeling responsibilities were distributed across task leads for TC, STS, NLI, RE, MRC, DST, DP, and NER.Contributors handled construction, fine-tuning, modeling, preprocessing, and task-specific leadership across the benchmark.
  • Corpus and Research Support: Additional contributors collected and processed source corpora, built fine-tuning infrastructure, and supported annotation and ethical-review activities.Named contributions included work on WIKINEWS, MODU, NEWSCRAWL, PETITION, AIRBNB, POLICY, IRB preparation, and the fine-tuning pipeline.
  • Advising and Sponsorship: Advisors and sponsors provided project guidance, funding, computing resources, news articles, dataset feedback, and manuscript support.The paper credits Alice Oh, Jung-Woo Ha, and Kyunghyun Cho with these forms of support.
  • Manuscript: The manuscript states that all participants contributed to its preparation.This complements the task-specific and management contributions listed throughout the section.

A Dev Set Results

The paper reports dev-set performances for its pretrained language models and other baselines on KLUE. Submission frequency is limited to help prevent early benchmark saturation.

  • Dev Set Results: Table 36 reports performances of the authors’ pretrained language models and other baselines on the KLUE dev set.The table uses the same notations as Table 32.
  • Evaluation Protocol: Models may be submitted once per day, while dev-set results provide a reference for future work and local testing.The models used for the dev set are the same as those used for the test set.
Loading 2105.09680v4…