PaperlayerView exact revision on arXiv

Source-linked AI summary

KULTURE Bench: A Benchmark for Assessing Language Model in Korean Cultural Context

Xiaonan Wang, Jinyoung Yeo, Joon-Ho Lim, Hansaem Kim

arXiv:2412.07251v1cs.CLcs.AI

TL;DR

Fluent, plausible LLM responses make evaluation difficult, while existing multilingual approaches can rely on translated English benchmarks. KULTURE Bench assesses Korean cultural comprehension through news, idioms, and poetry; models perform best on modern texts and news but remain less proficient with idioms and poetry.

  • Problem

    Fluent and plausible LLM responses complicate assessment of models’ knowledge and reasoning.

    Source: §1 · p. 1

  • Method

    KULTURE Bench combines 3,584 instances across idiom, poetry, and cultural-news datasets to evaluate Korean cultural understanding at word, sentence, and paragraph levels.

    Source: §1 · p. 1, §3.1 · p. 2

  • Results

    Models perform best on modern texts and news but lack proficiency in idioms and poetry.

    Source: §6 · p. 8

  • Takeaways & Limitations

    Without sufficient cultural knowledge, chain-of-thought techniques cannot be effectively utilized, and inappropriate reasoning lengths can affect accuracy.

    Source: §6 · p. 8

    The study needs to evaluate a broader range of models for a more comprehensive assessment.

    Source: §6 · p. 8

Abstract

from arXiv · show

Large language models have exhibited significant enhancements in performance across various tasks. However, the complexity of their evaluation increases as these models generate more fluent and coherent content. Current multilingual benchmarks often use translated English versions, which may incorporate Western cultural biases that do not accurately assess other languages and cultures. To address this research gap, we introduce KULTURE Bench, an evaluation framework specifically designed for Korean culture that features datasets of cultural news, idioms, and poetry. It is designed to assess language models' cultural comprehension and reasoning capabilities at the word, sentence, and paragraph levels. Using the KULTURE Bench, we assessed the capabilities of models trained with different language corpora and analyzed the results comprehensively. The results show that there is still significant room for improvement in the models' understanding of texts related to the deeper aspects of Korean culture.

1 Introduction

Fluent, plausible LLM responses make knowledge and reasoning harder to assess, while translated English benchmarks can carry Western cultural bias and Korean-focused research remains scarce. KULTURE Bench addresses this gap with three Korean-culture datasets and shows that even the strongest models struggle with idioms and poetry.

1 Introduction

  • Fluent, plausible responses have made assessing LLM knowledge and reasoning more difficult.The introduction notes that many benchmarks, including GLUE and MMLU, are primarily in English.Source: §1 · p. 1
  • Translated English benchmarks can retain Western cultural bias, and research on comparable Korean culture-focused evaluation remains relatively scarce.The introduction situates this gap among culturally specific benchmarks developed for India, Japan, and China.Source: §1 · p. 1
  • KULTURE Bench comprises 3,584 instances across three datasets, assessing Korean cultural comprehension and reasoning at word, sentence, and paragraph levels.It includes idioms and poetry alongside cultural news.Source: §1 · p. 1
  • Clova X performs best overall, yet reaches only 69.12% accuracy on idioms; GPT-4 records the best poetry result at 45.7%.The authors interpret these results as evidence that models still struggle with texts carrying deep cultural and historical context.Source: §1 · p. 1

2 Related Work

Language-specific evaluation is increasingly important as multilingual models advance. Existing Korean culture benchmarks largely use question-and-answer formats, while KULTURE Bench also incorporates news, idioms, and poetry.

2.1 Language Model

  • As multilingual models advance, evaluating language-specific capabilities becomes increasingly important, motivating benchmarks tailored to multilingual proficiency.The section cites models from China, Korea, and Japan alongside English-language models.Source: §2.1 · p. 2

2.2 Korean Cultural Benchmarks

  • Prior Korean-focused resources include benchmarks for bias, vocabulary, history, general knowledge, reading comprehension, exam questions, and social norms.The section names the Korean Bias Benchmark for Question Answering, HAE-RAE Bench, CLIcK, and KorNAT.Source: §2.2 · p. 2
  • Existing Korean culture datasets typically test culturally relevant content through question-and-answer formats, whereas KULTURE Bench adds news, idioms, and poetry.The authors describe these enduring cultural elements as requiring deeper understanding and contextual integration.Source: §2.2 · p. 2

3 KULTURE Bench

KULTURE Bench combines three Korean-culture datasets spanning word-, sentence-, and paragraph-level tasks. Its idiom dataset targets meanings that depend on cultural stories, using news contexts and similarity-selected distractors.

3.1 Task Overview

  • KULTURE Bench contains 3,584 instances: KorID tests idioms at word level, KorPD poem lines at sentence level, and KorCND cultural-news comprehension at paragraph level.The datasets contain 1,631, 453, and 1,500 instances, respectively.Source: §3.1 · p. 2

3.2.1 The characteristics of four-character idioms

  • Many four-character idioms have story-derived metaphorical meanings that cannot be recovered from their literal words alone.The example “함흥차사” refers to delayed news or responses, drawing on a Joseon-era story.Source: §3.2.1 · p. 3

3.2.2 Construction of KorID dataset

  • KorID’s vocabulary was digitized and manually corrected from idiom dictionaries, then matched against Korean news corpora to identify 1,631 instances.The vocabulary contained 5,372 unique idioms; the news corpus covered 2009–2022.Source: §3.2.2 · p. 3
  • KorID uses KorBERT cosine similarity to select four distractors from similarity bands, then manually checks samples for alternative valid answers.All 50 reviewed samples had a uniquely selective Golden Answer, supporting exclusion of candidates with similarity scores above 0.9.Source: §3.2.2 · p. 4

3.3 KorPD: A Korean Poem Dataset for Cloze Test

Poetry’s ambiguous, intertwined semantics complicate automatic evaluation. KorPD builds a cloze task from Korean textbook poems, using semantically similar lines as distractors.

3.3.1 The characteristics of poems

  • Poetic semantics can be ambiguous and intertwined, complicating automatic analysis and evaluation of models’ understanding of poetic meaning.Poetry is also characterized by intense emotion, stylistic expression, and abstractly conveyed themes.Source: §3.3.1 · p. 4

3.3.2 Construction of the KorPD dataset

  • KorPD digitizes and manually reviews textbook poems, selecting 91 poems that reflect contemporary educational and cultural contexts in Korea.The collection spans a broad range of themes and styles.Source: §3.3.2 · p. 4
  • KorPD extracts up to five unique lines per poem, yielding 453 items, and selects distractor lines using KorBERT representations and cosine-similarity bands.The gold and negative answers are randomized; each item includes the golden answer and one line selected from each of four similarity intervals.Source: §3.3.2 · p. 4

3.4 KorCND: A Korean Culture News Dataset for Headline Matching

KorCND targets the cultural and contextual inferences needed to understand Korean news headlines. It combines culturally relevant news articles with semantically similar negative choices to test headline matching.

3.4.1 The characteristics of news headline

  • Korean cultural headlines require interpreting context and cultural references beyond their literal words.The Chuseok example combines a festival reference, cultural activities, and figures from Korean arts.Source: §3.4.1 · p. 5

3.4.2 Construction of the KorCND dataset

  • The dataset curates 1,500 Korean-culture articles from the 2021 NIKL Newspaper Corpus after keyword filtering and manual review.The source corpus contains 978,342 articles from 34 news outlets; selected titles and full texts were organized for analysis and checked for entry errors.Source: §3.4.2 · p. 5
  • KorBERT [CLS] embeddings represent headline meaning, and cosine similarity is used to identify semantically close negative choices.Similarity ranges from [0.5, 0.6] through [0.81, 0.9] group candidates by semantic closeness.Source: §3.4.2 · pp. 5–6
  • One negative headline is randomly selected from each similarity interval, then answer order is randomized.The selected alternatives are intended to resemble the Golden Answer closely enough to test discrimination.Source: §3.4.2 · pp. 5–6

4 Experimental Setup

The experiments compare models trained primarily on English, Korean, and Chinese corpora. They use zero-shot prompts, a separate chain-of-thought condition, accuracy scoring, and explicit answer-validation rules.

4.1 Evaluated Models

  • The evaluated models were GPT-4 and GPT-3.5 Turbo (English), Clova X (Korean), and Tiangong (Chinese).Tiangong was included to explore possible spillover from the historical influence of Chinese Hanja on Korean.Source: §4.1 · p. 6

4.2 Prompt Types

  • The first experiment used zero-shot prompts; the second tested three chain-of-thought lengths on KorID with GPT-3.5 Turbo, Clova X, and Tiangong.The requested reasoning lengths were 1–2, 3–4, or 5–6 sentences.Source: §4.2 · p. 6

4.3 Evaluation Metric

  • Performance was measured as the proportion of samples answered correctly.Higher accuracy indicates better performance under the paper's metric.Source: §4.3 · p. 6

4.4 Response Validation Criteria

  • A response counted as an answer only if it matched an option, clearly stated an option, or placed an option in square brackets.Responses meeting none of these criteria were classified as out-of-option answers.Source: §4.4 · p. 6

5 Results

Clova X leads on KorID and KorCND, while GPT-4 leads on KorPD; all models do better on news than on idioms and poetry. Chain-of-thought effects vary with model and reasoning length, and human participants outperform the models across datasets.

5.1 Main Results

  • Clova X ranks first on KorID (69.12%) and KorCND (92.77%), while GPT-4 leads KorPD (45.7%).Table 2 reports accuracy across the three datasets.Source: §5.1 · p. 6, Table 2 · p. 7
  • All models perform relatively well on news but substantially worse on texts rich in cultural elements, including poetry and idioms.The authors suggest scarcity of such training data and the texts' cultural complexity as possible reasons.Source: §5.1 · pp. 6–7, Figure 1 · p. 7

Does Chain of Thought Enhance Understanding and Reasoning in Culture Texts?

  • Chain of thought improves Tiangong and Clova X accuracy, but GPT-3.5 Turbo shows no notable improvement and sometimes declines.Tiangong rises from 26.92% to 35.83% with short reasoning, then falls to 33.70% and 32.88% with medium and long reasoning; Clova X rises from 69.12% to 71.28%, then falls to 70.69% and 67.80%.Source: Does Chain of Thought Enhance… · pp. 7–8, Table 3 · p. 7, Figure 2 · p. 8
  • Among sampled correct responses, the reasoning was judged correct in 96% for Clova X, 74% for Tiangong, and 46% for GPT-3.5 Turbo.The authors also report that 90% of idioms correctly reasoned by Tiangong remain in active use in contemporary Chinese society.Source: Does Chain of Thought Enhance… · p. 7

How Do Model Performances Compare to Human Level Proficiency?

  • Human participants outperform LLMs across datasets, especially on cultural news and poetry; humans with Korean Studies expertise excel on poetry and idioms.LLMs perform well on structured news-headline matching but struggle more with cultural and linguistic context.Source: How Do Model Performances Compare to… · p. 8, Figure 3 · p. 8

What Types of Errors Occur in Model Responses to Korean Cultural Texts?

  • The response-error analysis groups errors into semantic understanding, cultural knowledge, grammar or vocabulary, logic, and context interpretation.The authors sampled 20 error responses per model across three datasets, totaling 240 responses across all models.Source: What Types of Errors Occur in Model… · p. 8

6 Conclusion and Limitations

KULTURE Bench shows that models handle modern texts and news better than idioms and poetry, while humans still outperform them. The authors also identify limits in model coverage, Korean-only CoT experiments, and the small human sample.

6 Conclusion and Limitations

  • Models perform best on modern texts and news but lack proficiency in idioms and poetry; human capabilities also surpass model performance.The authors additionally report that CoT is ineffective without sufficient cultural knowledge, and inappropriate reasoning lengths can affect accuracy.Source: §6 · p. 8
  • KULTURE Bench evaluates Korean cultural knowledge and linguistic phenomena across word-to-paragraph levels using news, textbooks, and dictionaries.Source: §6 · p. 8
  • The evaluation needs more models, CoT experiments beyond Korean cultural reasoning, and a larger human sample.Source: §6 · p. 8

A Examples from KULTURE Bench

The examples illustrate three task formats: selecting idioms for blanks in news articles, completing lines in classical poems, and choosing culturally appropriate news headlines.

A Examples from KULTURE Bench

B Prompt Types

The appendix presents example tasks from the three datasets and identifies prompts used in the first evaluation round and the second, Chain of Thought round.

B Prompt Types

  • The prompt figures cover a first-round evaluation and a second-round Chain of Thought experiment.Source: B Prompt Types · p. 10, Figure 4 · p. 10, Figure 5 · p. 11
  • The KorID example tests filling a news-article blank with an idiom selected for its meaning and context.Source: Table 4 · p. 10
  • The KorPD example tests choosing a missing poem line using the poem’s meaning, emotional expression, and rhythm.Source: Table 5 · p. 10
  • The KorCND example tests selecting a headline that captures the main idea and cultural elements of a news article.Source: Table 6 · p. 11

C Categories of Errors in Model Responses

The error analysis distinguishes failures of meaning, cultural knowledge, language form, logic, and broader contextual interpretation. Examples show that a response may appear plausible while missing a text’s intended content or cultural and emotional significance.

C Categories of Errors in Model Responses

  • Semantic understanding errors misread a text’s meanings, nuances, intent, or theme, producing coherent but contextually or factually incorrect responses.One example describes a headline that focuses on literary-market economics rather than the article’s account of a poet’s life and achievements.Source: C Categories of Errors in Model… · pp. 11–12
  • Lack of cultural background knowledge can lead to culturally inappropriate choices, as when a model misreads 삼고초려 as careful deliberation rather than devotedly attracting talent.Source: C Categories of Errors in Model… · p. 12
  • Grammatical or lexical errors include incorrect syntax or spelling; one response chose the right idiom but was marked wrong for not outputting correct Korean.Source: C Categories of Errors in Model… · p. 12
  • Logical errors involve inconsistent or contradictory responses, including a case where the model identified the correct option but chose a contradictory answer.Source: C Categories of Errors in Model… · pp. 12–13
  • Insufficient context interpretation misses broader cultural nuance; a poem answer was grammatically complete but overlooked its emotional and cultural background.Source: C Categories of Errors in Model… · p. 13

Related papers

Loading 2412.07251v1…