PaperlayerView exact revision on arXiv

Source-linked AI summary

CNeo-Bench: Diagnosing Large Language Models on Chinese Neologisms

Kaiyan Zhao, Zhongtao Miao, Zheyong Xie, Shaosheng Cao, Yoshimasa Tsuruoka

arXiv:2608.28053v1cs.CL

TL;DR

LLMs face a distinctive challenge in understanding Chinese neologisms formed through diverse linguistic mechanisms. The paper introduces CNEO-BENCH and a two-tier evaluation framework, finding persistent difficulty, a recognition-manipulation gap, and incomplete few-shot recovery. The results support mechanism-aware evaluation beyond meaning description alone.

  • Problem

    Chinese neologisms use diverse formation mechanisms, creating a need to assess whether LLMs can describe them and operate on their underlying source forms.

  • Method

    The paper introduces CNEO-BENCH, a 4,759-item benchmark organized by linguistic mechanism, with definition generation and category-specific diagnostic tasks.

  • Results

    Across 18 LLMs, Chinese neologisms remain an open challenge, with a systematic recognition-manipulation gap and only partial recovery from few-shot examples.

  • Takeaways & Limitations

    CNEO-BENCH provides a mechanism-aware lens for evaluating Chinese-language LLMs beyond whether they can describe meaning.

  • Takeaways & Limitations

    Tier 1 semantic adequacy judgments may mark definitions correct without requiring reproduction of the neologism’s source form.

Abstract

from arXiv · show

Chinese neologisms exploit diverse and unique linguistic mechanisms, such as phonetic substitution (e.g., 886 for ``bye-bye'') and visual character decomposition that are rare in other languages. We introduce CNeo-Bench, a benchmark of 4,759 such neologisms with reference definitions, organized into five top-level categories and nine subcategories by the linguistic mechanism behind each expression. CNeo-Bench is paired with a two-tier evaluation framework that separates whether a model can describe a neologism from whether it can operate on its underlying mechanism. Evaluating 18 LLMs, we find that Chinese neologisms remain an open challenge; most models fall below 40\% on definition generation, and on several subcategories a systematic recognition-manipulation gap emerges: models describe neologisms correctly but, in source-form restoration tasks, substitute a semantic equivalent (paraphrase) for the source form rather than producing the source form itself. A few-shot analysis on 1,058 hard items shows that in-context examples can solve many difficult cases, but leave a noticeable portion of errors remaining, indicating challenges beyond prompting alone can address.

1 Introduction

Chinese neologisms challenge LLMs because they use diverse mechanisms beyond ordinary lexical and semantic innovation. CNeo-BENCH evaluates both whether models can describe these expressions and whether they can operate on their source forms.

  • Motivation: Chinese neologisms exploit mechanisms including phonetic substitution, visual character decomposition, and cross-script abbreviation.Examples include 886 for “bye-bye,” 彳亍 for 行, and yyds for 永远的神.
  • Benchmark: CNEO-BENCH contains 4,759 Chinese neologisms with reference definitions, organized into five top-level categories and nine mechanism-based subcategories.The benchmark is paired with tasks designed around the linguistic operations that produce the expressions.
  • Evaluation framework: Its two-tier framework separates definition generation from category-specific tasks that test operation on a neologism’s source form.A model may describe a visually decomposed expression correctly while failing to restore its source form.
  • Findings: 67.74% was the strongest definition-generation score, achieved by Kimi-K2.5 across evaluations of 18 LLMs.The paper reports that Chinese neologisms remain an open challenge even for frontier models.
  • Findings: A systematic recognition-manipulation gap appears when models describe neologisms correctly but copy them or produce semantic equivalents instead of source forms.The gap is reported on subcategories requiring open-ended source-form restoration.
  • Findings: Few-shot examples recover many difficult cases, but a noticeable residual of errors remains.This result comes from analysis of hard items rather than the full benchmark.

2 CNEO-BENCH

CNEO-BENCH organizes Chinese neologisms by formation mechanism and evaluates them through definition generation and mechanism-specific diagnostic tasks. Its data collection combines licensed sources, filtering, annotation, and manually verified task construction.

  • Taxonomy: The taxonomy has five top-level categories and nine subcategories covering abbreviations, homophony, lexical innovation, stylistic or pragmatic expressions, and foreign borrowings.Subcategories distinguish, for example, Pinyin abbreviations, number homophones, semantic neologisms, formulaic expressions, and foreign-source forms.
  • Data collection: The final dataset contains 4,759 entries paired with reference definitions after deduplication, machine verification, and a RedNote popularity filter.The filter required at least five returned posts and removed 158 entries with negligible usage.
  • Data collection: Moegirlpedia contributes 1,393 entries, Wikiversity 1,024, and other web sources 2,342 entries.These correspond to 29.3%, 21.5%, and 49.2% of the final dataset, respectively.
  • Annotation: Three annotators assign each entry one of nine subcategories, achieving 78% raw agreement and Fleiss’ κ=0.69.The annotation statistics are reported for the nine-category scheme.
  • Evaluation: Tier 1 asks models to generate a zero-shot definition scored against references by an LLM-based judge.This task is applied uniformly across all entries and subcategories.
  • Evaluation: Tier 2 uses open-ended restoration for three subcategories and multiple-choice diagnostics for the remaining six.Restoration is scored by exact string match, while other tasks test cloze completion, semantic discrimination, scenario matching, or foreign-source identification.
  • Evaluation: LLM-generated example sentences and scenarios support several Tier 2 tasks, with manual verification and 492 original source examples used where available.The multiple-choice tasks also require manually checked distractors.

3 Experiments

The experiments evaluate 18 LLMs on CNEO-BENCH using definition generation and category-specific diagnostics, revealing uneven performance across linguistic mechanisms and task formats.

  • Experimental Setup: 18 LLMs are evaluated on CNEO-BENCH using greedy decoding, Chinese prompts, and one generation per instance.GPT, DeepSeek, and Kimi models are accessed through official APIs; other models run on a single H800.
  • Evaluation: Tier 1 definition generation is judged automatically for semantic adequacy, while Tier 2 uses option matching or exact source-form string matching.Tier 1 uses a Qwen3-235B-A22B-Instruct-2507 judge; Tier 2 scoring is deterministic.
  • Tier 1 Results: 67.74% overall accuracy is achieved by Kimi-K2.5 on definition generation, while most open-source models remain below 40%.Chinese neologisms remain challenging even for frontier models, and scale alone does not eliminate the gap within model families.
  • Tier 1 Results: 50.89% overall accuracy is achieved by GPT-5.1, below every evaluated Chinese-focused frontier model, with especially large gaps on Pinyin Abbr. and Unconv. Chars.GPT-5.1 scores 47.40% versus Kimi-K2.5's 76.56% on Pinyin Abbr., and 21.74% versus 82.61% on Unconv. Chars.
  • Tier 1 Results: Subcategory difficulty is uneven: English Abbr. is easier than Pinyin Abbr., while Unconv. Chars. has the widest spread across models.Most open-source models score below 20% on Unconv. Chars., whereas Kimi-K2.5 reaches 82.61%.
  • Tier 2 Results: 81.94% overall accuracy is achieved by Kimi-K2.5 on Tier 2, and every model exceeds its own Tier 1 accuracy, although some small models approach random guessing on Pinyin Abbr.English Abbr. and Semantic Neo. are near-ceiling for many mid-to-frontier models, while open-ended Chinese Homo., Number Homo., and Unconv. Chars. remain difficult for small models.

4 Analysis

The analysis examines the Tier 1–Tier 2 relationship and few-shot recovery, identifying restoration failures despite correct descriptions and partial recovery on hard items.

  • Analysis Design: The analysis combines contingency analysis of Tier 1 and Tier 2 outcomes with a few-shot mitigation experiment on hard samples.The paper also reports a human-performance analysis in the appendix.
  • 4.1 Recognition-Manipulation Gap: The largest Tier 2–Tier 1 divergences occur in Chinese Homo., Number Homo., and Unconv. Chars., the open-ended restoration subcategories.Most other subcategories show positive Tier 2–Tier 1 gaps associated with task-format advantages.
  • 4.1 Recognition-Manipulation Gap: RM Gap is quantified as B/(A + B), the fraction of items correctly described in Tier 1 but not restored in Tier 2.A, B, C, and D represent the four combinations of Tier 1 and Tier 2 success or failure.
  • 4.1 Recognition-Manipulation Gap: Tier 2 successes in Cell C often reflect inference from usage context rather than evidence of memorized neologism knowledge.This interpretation applies especially to Chinese Homo. and Number Homo., where Tier 2 includes example context.
  • 4.2 Hard Samples and Few-Shot Recovery: 1,058 hard samples, representing 22.23% of the benchmark, are defined as items all three selected models miss on zero-shot Tier 1.Few-shot prompting uses 1 to 3 manually filtered natural usage sentences as exemplars, with recovery reported under 1-, 2-, and 3-shot settings.

5 Related Work

Prior neologism research has focused largely on English identification and evaluation, while Chinese work has mainly addressed translation challenges rather than mechanism-specific operations.

  • Prior Neologism Research: Early neologism research primarily studies identification in English text, and recent LLM benchmarks continue to focus largely on English neologisms.Existing efforts do not engage substantially with mechanisms specific to how neologisms are produced.
  • Chinese Neologism Research: Chinese neologism evaluation has emerged mainly in machine translation, where cultural specificity and rapid evolution create translation challenges.This differs from evaluating the linguistic mechanisms underlying Chinese neologism formation.

6 Conclusion

CNEO-BENCH evaluates Chinese neologism understanding through linguistic mechanisms and separate description and manipulation abilities, revealing persistent challenges and a cross-model recognition-manipulation gap.

  • CNEO-BENCH contains 4,759 Chinese neologisms with reference definitions, organized by linguistic mechanism within a two-tier evaluation framework.
  • The benchmark separates describing a neologism in Tier 1 from operating on its underlying form in Tier 2.
  • Evaluation across models identifies a systematic recognition-manipulation gap: models describe neologisms correctly in Tier 1 but fail in Tier 2.
  • Few-shot prompting cannot fully resolve Chinese neologism understanding, leaving errors beyond what prompting alone addresses.

Limitations

The paper identifies evaluation, data-coverage, and scope limitations that constrain interpretation of some benchmark results and leave mitigation for future work.

  • Tier 1 correctness is judged by an LLM judge on semantic adequacy rather than explicit reproduction of a neologism’s source form.
  • Number Homo. (n = 29) and Unconv. Chars. (n = 23) contain few items, so fine-grained comparisons are indicative rather than definitive.
  • The study diagnoses the recognition-manipulation gap through evaluation but leaves mitigation strategies to future work.

Ethical Statements

CNEO-BENCH is built from publicly available, research-permitted sources and excludes personally identifiable information.

  • The benchmark uses publicly available sources, including Moegirlpedia, Wikiversity, and other open web content indexed by general-purpose search.
  • The authors verified source licensing and included only content permitting research use with attribution.
  • The benchmark contains no personally identifiable information and limits public-figure neologisms to their originating public-domain context.

A.1 Detailed Taxonomy

The taxonomy groups Chinese neologisms by the linguistic mechanisms that generate them, distinguishing abbreviation, phonetic, lexical, visual-pragmatic, and foreign-source pathways.

  • Alphabetic Abbreviations: Alphabetic Abbreviations split into Pinyin Abbr. and English Abbr. because identical letter-string surfaces require different decoding chains.
  • Homophonic Expressions: Homophonic Expressions use digit sequences or phonetically similar characters to encode target Chinese phrases.
  • Lexical Innovations: Lexical Innovations distinguish Lexical Neo., which introduces newly coined meanings, from Semantic Neo., which shifts an existing word’s meaning.
  • Stylistic and Pragmatic Expressions: Unconv. Chars. visually decomposes standard characters, while formulaic expressions reuse fixed phrases or sentences for pragmatic effects.
  • Stylistic and Pragmatic Expressions: These stylistic and pragmatic forms are convention-based and non-compositional, so readers must recognize specific forms rather than apply a general rule.
  • Foreign Borrowings: Foreign Borrowings depend on non-Chinese sources through paths including phonetic reinterpretation and direct lexical adoption.

A.2 Annotation

The annotation process labels Chinese neologisms by formation mechanism using a nine-subcategory taxonomy, with adjudication for uncertain or disputed cases.

  • Annotation procedure: Three annotators are native Mandarin speakers familiar with contemporary Chinese internet culture and trained using a written rubric derived from the taxonomy definitions.The annotation instructions were delivered in Chinese and provided with an English translation.
  • Agreement and adjudication: 78% raw three-way agreement corresponded to Fleiss’ κ=0.69 before adjudication resolved disagreements through the same rubric.A senior reviewer assigned the final label for the remaining 22% with at least one disagreement.
  • Annotation procedure: Annotators assign one of nine mechanism-based subcategory labels, or mark an item “uncertain” when it fits multiple or no categories.A senior annotator reviews and finalizes uncertain assignments.

A.3 Human Evaluation of LLM-as-Judge

The human evaluation validates the LLM judge against stratified human judgments while documenting its scoring setup, benchmark context, and known error patterns. Humans substantially outperform the strongest evaluated model on Tier 1 definition generation, but judge disagreements and lenient edge cases remain.

  • LLM-judge validation: 1,000 Tier 1 model outputs were sampled across all nine subcategories and all 18 models, with small categories exhaustively included.The validation sample was stratified to ensure coverage of formation mechanisms.
  • LLM-judge validation: Three annotators independently judged semantic correctness using the LLM judge’s criterion, and majority vote supplied the final human label.The judging criterion assessed whether a generated definition described the neologism’s meaning and origin.
  • Agreement results: 95% agreement with human majority labels on 1,000 items produced Cohen’s κ = 0.74 for the LLM judge.The remaining disagreements primarily involved borderline definitions, including approximate meanings that omitted specific source-form references.
  • Benchmark tasks: The benchmark evaluates definition generation separately from mechanism-specific operations, including open-ended source-form restoration and semantic discrimination.Figure 4 reports that English-based abbreviations are substantially easier than Pinyin-based abbreviations for all evaluated models.
  • Human performance: Human evaluators achieved 85.63% overall Tier 1 accuracy versus 67.74% for Kimi-K2.5, the strongest evaluated model.The human evaluation sampled 100 items per subcategory when available and evaluated smaller subcategories in full.
  • Human error patterns: Human errors included recognition failures on unfamiliar isolated neologisms and omissions of source language or borrowing path for Foreign Borrowings.For example, defining “哈基米” as “cat” without identifying its Japanese origin was judged incorrect.
  • Restoration failures: In Cell B cases, the dominant Tier 2 failure is explanation or paraphrase: models describe the neologism correctly but replace its source form with a semantic equivalent.This pattern occurs across all three open-ended subcategories.
  • Judge prompt: The judge prompt scores semantic adequacy with a binary 0/1 criterion and marks empty outputs as incorrect.Definitions that partially cover the meaning without affecting core understanding may still receive a positive score, whereas obvious misunderstandings receive 0.
Loading 2608.28053v1…