Source-linked AI summary
CLUE: A Chinese Language Understanding Evaluation Benchmark
Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, Yin Tian, Qianqian Dong, Weitang Liu, Bo Shi, Yiming Cui, Junyi Li, Jun Zeng, Rongzhao Wang, Weijian Xie, Yanting Li, Yina Patterson, Zuoyu Tian, Yiwen Zhang, He Zhou, Shaoweihua Liu, Zhe Zhao, Qipeng Zhao, Cong Yue, Xinrui Zhang, Zhengliang Yang, Kyle Richardson, Zhenzhong Lan
TL;DR
Chinese NLU lacked a comprehensive benchmark comparable to English resources, despite Chinese linguistic differences and the need to assess Chinese models. CLUE addresses this gap with nine Chinese tasks, supplementary pre-training and diagnostic datasets, and evaluation infrastructure. Results show varied task difficulty, generally better performance from larger or better-pretrained models, and Chinese NLU models approximately 10% below estimated human performance.
Problem
Chinese lacked a general, comprehensive NLU evaluation benchmark, while existing benchmarks were mostly English and Chinese differs linguistically from English.
Method
CLUE assembles nine Chinese NLU tasks and releases large-scale pre-training data, a diagnostic dataset, model evaluations, a toolkit, and a leaderboard.
Results
Across tasks, larger models, more pre-training data, and whole-word masking generally improve results, while task difficulty varies across models and evaluations.
Takeaways & Limitations
CLUE provides a shared basis for evaluating Chinese NLU and indicates substantial remaining room for improvement.
Takeaways & Limitations
Chinese NLU models remain approximately 10% below estimated human performance, and the current task set is presented as a first step in an open-ended benchmark.
Abstract
from arXiv · showhide
The advent of natural language understanding (NLU) benchmarks for English, such as GLUE and SuperGLUE allows new NLU models to be evaluated across a diverse set of tasks. These comprehensive benchmarks have facilitated a broad range of research and applications in natural language processing (NLP). The problem, however, is that most such benchmarks are limited to English, which has made it difficult to replicate many of the successes in English NLU for other languages. To help remedy this issue, we introduce the first large-scale Chinese Language Understanding Evaluation (CLUE) benchmark. CLUE is an open-ended, community-driven project that brings together 9 tasks spanning several well-established single-sentence/sentence-pair classification tasks, as well as machine reading comprehension, all on original Chinese text. To establish results on these tasks, we report scores using an exhaustive set of current state-of-the-art pre-trained Chinese models (9 in total). We also introduce a number of supplementary datasets and additional tools to help facilitate further progress on Chinese NLU. Our benchmark is released at https://www.CLUEbenchmarks.com
1 Introduction
CLUE addresses the lack of a comprehensive Chinese NLU benchmark by combining nine tasks with supporting datasets, tools, and model evaluations. It also provides large-scale Chinese pre-training data and a linguistically motivated diagnostic dataset.
- Existing NLU benchmarks are mostly English, while Chinese lacks a general evaluation benchmark despite linguistic differences from English.
- CLUE combines nine Chinese NLU tasks spanning semantic similarity, inference, classification, and machine reading comprehension.Two of the tasks were created by the authors.
- The benchmark evaluates popular Chinese pre-trained models and finds task difficulty differences reflected in model accuracies and human–machine comparisons.
- CLUE releases a pre-training corpus exceeding 214 GB of raw text and roughly 76 billion Chinese words.
- The project also provides a linguist-created diagnostic dataset, toolkit, and online leaderboard for analyzing and comparing Chinese NLU models.
2 Related Work
Prior evaluation work standardized comparisons through shared datasets and pipelines, but Chinese NLU still lacked an equivalent diverse and challenging benchmark.
- Earlier research evaluated word and sentence representations with different intrinsic and downstream datasets and pipelines.
- SentEval, GLUE, and SuperGLUE improved consistency by standardizing evaluation datasets, pipelines, and more difficult NLU tasks.
- Chinese NLU research still needed a standard evaluation pipeline and a diverse set of difficult language-understanding tasks.
3 CLUE Overview
CLUE is organized as a Chinese NLU evaluation ecosystem: diverse tasks, pre-training and diagnostic data, reproducibility tools, and a public ranking system.
- CLUE contains nine Chinese language-understanding tasks, a large raw pre-training dataset, a hand-crafted diagnostic dataset, and ranking and toolkit infrastructure.
- The nine tasks were selected for diversity, processing simplicity, moderate difficulty, practical usefulness, and Chinese-specific characteristics.
- The project solicited task proposals from the Chinese NLP community while addressing the relative scarcity of diverse public Chinese NLU datasets.
- The released pre-training corpus contains 214 GB of raw text and around 76 billion Chinese words.
- A linguistically motivated diagnostic dataset measures models on nine linguistic and logic phenomena.
- CLUE supports submitted-result scoring, certification of reproducible open-source models, and quick validation through the PyCLUE TensorFlow toolkit.
4 Tasks
CLUE covers Chinese single-sentence, sentence-pair, and machine-reading tasks with varied labels, formats, domains, and reasoning demands.
- CLUE includes nine Chinese NLU tasks spanning single-sentence classification, sentence-pair classification, and machine reading comprehension.
- 4.1 Single Sentence Tasks: TNEWS classifies 73,360 Chinese news titles into 15 categories, while IFLYTEK assigns app descriptions to 119 categories.
- 4.1 Single Sentence Tasks: Task selection includes binary coreference resolution in CLUEWSC2020, with 1,838 linguist-annotated questions from contemporary Chinese literature.
- 4.2 Sentence Pair Tasks: Sentence-pair tasks include semantic similarity prediction in AFQMC and abstract-keyword validity judgment in CSL.
- 4.2 Sentence Pair Tasks: OCNLI contains 56k inference pairs across five genres, with Chinese premises and hypotheses written by language-major university students.
- 4.3 Machine Reading Comprehension: CMRC 2018 asks models to extract answer spans from Wikipedia contexts in response to human-annotated questions.
- 4.3 Machine Reading Comprehension: ChID tests Chinese idiom cloze completion, while C3 uses free-form multiple-choice questions over dialogue and mixed-genre documents.
5 Pre-Training Dataset
The paper assembles large Chinese pre-training resources to address the smaller scale of existing public Chinese datasets. The collection combines multiple corpora and provides data intended for direct pre-training.
- Motivation: Chinese Wikipedia contains only around 1.1 GB of raw text, motivating collection of a larger clean crawled corpus.The paper contrasts the Chinese resource scale with larger English pre-training datasets.
- Corpus Overview: 214 GB of raw Chinese text and roughly 76 billion words are collected across three corpora.The corpora are CLUECorpus2020-small, CLUECorpus2020, and CLUEOSCAR.
- CLUECorpus2020-small: CLUECorpus2020-small contains 14 GB of Chinese text from news, WebText, Wikipedia, and comments.Its sources include self-media news, Chinese question-answer websites, Chinese Wikipedia, and e-commerce comments.
- CLUECorpus2020: CLUECorpus2020 provides 100 GB of Common Crawl Chinese text in a pre-training format that can be used without additional preprocessing.The dataset contains around 29K separate files.
- CLUEOSCAR: CLUEOSCAR starts from 250 GB of Chinese OSCAR raw corpus and is filtered to produce 100 GB of Chinese corpus.The source corpus is obtained from Common Crawl through language classification and filtering.
6 Experiments
The experiments fine-tune representative Chinese pre-trained Transformer models separately on CLUE tasks and compare their performance with human results. Larger models, more pre-training data, and whole word masking generally improve results, while task difficulty and model scale remain important.
- Experimental Setup: Baseline models add task-specific output layers to pre-trained Transformers and are fine-tuned separately for each CLUE task.Single-sentence and sentence-pair tasks use pooled representations, while reading-comprehension tasks use task-specific heads.
- Models: Nine publicly available Chinese pre-trained models are evaluated, including BERT, ALBERT, ERNIE, XLNet, and RoBERTa variants.The evaluated set includes models trained on both Chinese Wikipedia and the CLUE pre-training corpus.
- Benchmark Results: Larger models, more pre-training data, and whole word masking are associated with better CLUE results.RoBERTa-wwm-ext-large and ALBERT-xxlarge perform best overall, particularly on machine reading tasks such as C3.
- Benchmark Results: ALBERT-tiny has about 1/20 of BERT-base’s parameters, with performance drops ranging from almost 0 to roughly 12 percentage points on sentence tasks.On machine reading tasks, its performance drop reaches up to 40 percentage points compared with BERT-base on ChID.
- Benchmark Results: XLNet-mid performs poorly on token-level Chinese span-extraction reading comprehension, including CMRC 2018.The result is reported for a model using SentencePiece, a tokenizer developed in English-language work.
- Human Comparison: The best machine result remains roughly 11 points below human performance on average.The paper presents this gap as room for further improvement in models and methods.
7 Diagnostic Dataset for CLUE
CLUE adds a linguist-authored Chinese diagnostic dataset to test inference over linguistically important phenomena. Initial evaluations show substantial difficulty, especially for monotonicity and some lexical-semantic cases.
- Dataset Construction: The diagnostic dataset frames 514 hand-crafted Chinese sentence pairs as natural language inference items.Each hypothesis must be classified as entailed, contradicted, or neutral with respect to its premise.
- Dataset Construction: The diagnostics target Chinese-specific phenomena alongside anaphora resolution and monotonicity reasoning.The dataset was hand-crafted rather than translated from English diagnostics because some English items depend on English or Western culture.
- Evaluation: The highest accuracy among BERT-base, XLNet-mid, and RoBERTa-wwm-ext-large is only about 61%.Each model is fine-tuned on OCNLI before evaluation on the diagnostic dataset.
- Error Analysis: Monotonicity is the hardest diagnostic category, and BERT also struggles with comparatives.The paper reports this pattern in its breakdown of results across the nine linguistic categories.
- Error Analysis: Two of three models appear to ignore the final character distinguishing the Chinese words for “sad” and “ugly.”The authors suggest this may reflect a simple lexical-overlap heuristic.
8 Conclusions and Future Work
The paper presents CLUE as a Chinese NLU benchmark with nine tasks, a diagnostic dataset, an online leaderboard, and a pre-training corpus. Its analysis finds that state-of-the-art models remain about 10% below estimated human performance on Chinese NLU.
- CLUE combines nine Chinese NLU tasks, a linguistically motivated diagnostic dataset, an online leaderboard, and a clean crawled corpus for Chinese-model pre-training.
- The diagnostic analysis shows that current state-of-the-art models have limited ability to handle some Chinese linguistic phenomena.
- ∼10% below estimated human performance, Chinese NLU models still have considerable room for improvement on the benchmark.
- CLUE is intended as an open-ended, community-driven first step toward more comprehensive Chinese NLU evaluation.
A Dataset Samples
The appendix provides development-set examples for CLUE datasets, while noting that some displayed examples are shortened and complete datasets will be available through related papers and released links.
- Table 5 compiles examples from each CLUE dataset for reference.
- Some examples are intercepted because their sentences are too long.
- Complete datasets can be consulted in related papers, with download links planned for the final paper version.
B.1 Hyperparameters for pre-training
This appendix section documents pre-training and fine-tuning references for the experiments, including a small-model convergence observation and a learning-rate example for CSL.
- Pre-training hyperparameters are listed in Table 6 for reference, although most models were not trained by the authors.
- Fine-tuning hyperparameters for the experiments are listed in Table 7.
- For CSL, negative samples replace one real keyword with a fake one.
- 5e-6 is an example of the very small learning rate at which some larger CSL models converge during fine-tuning.
- ALBERT-tiny requires more fine-tuning epochs to converge than the other models in the IFLYTEK experiments.
D Dataset Filtering
The appendix describes dataset filtering and related table conventions. IFLYTEK and TNEWS examples are filtered with four-fold cross-validation using ALBERT-tiny to remove easy examples.
- D Dataset Filtering: Four-fold cross-validation filters IFLYTEK and TNEWS by fine-tuning ALBERT-tiny on three folds and selecting easy examples in the remaining fold.
- A Dataset Samples: Table 5 presents development-set examples from CLUE tasks, including Chinese model inputs and expected outputs.
- B.1 Hyperparameters for pre-training: Table 6 defines pre-training table abbreviations including WWM, ext, and ∼BERT.
- D Dataset Filtering: RoBERTa-large uses a shorter 256-token pre-training sequence length than the 512-token length used for the other listed models.
- D Dataset Filtering: The shorter RoBERTa-large sequence length may reduce performance on datasets whose text cannot be effectively shortened, such as C3.