PaperlayerView exact revision on arXiv

Source-linked AI summary

OCNLI: Original Chinese Natural Language Inference

Hai Hu, Kyle Richardson, Liang Xu, Lu Li, Sandra Kuebler, Lawrence S. Moss

arXiv:2010.05444v1cs.CL

TL;DR

NLI progress has been concentrated in English because reliable datasets for other languages are scarce. The paper introduces OCNLI, a large-scale Chinese corpus built from original Chinese sources by linguistically trained native speakers, and finds that pretrained models remain about 12 points below human performance, making the resource challenging for Chinese NLU.

  • Problem

    Reliable large-scale NLI datasets are scarce outside English, while translation-based extensions can introduce quality and language-specific problems.

  • Method

    OCNLI uses approximately 56,000 original Chinese premise-hypothesis pairs elicited and annotated by native speakers with linguistic expertise, following MNLI-style procedures and diverse hypothesis strategies.

  • Results

    The strongest RoBERTa model achieves approximately 78% accuracy versus approximately 90% for expert humans, a roughly 12-point gap on OCNLI.

  • Takeaways & Limitations

    OCNLI provides a challenging human-elicited Chinese NLI resource for advancing Chinese natural language understanding.

  • Takeaways & Limitations

    The work monitors hypothesis-only and related biases but leaves systematic filtering, bias reduction, and stress testing for future work.

Abstract

from arXiv · show

Despite the tremendous recent progress on natural language inference (NLI), driven largely by large-scale investment in new datasets (e.g., SNLI, MNLI) and advances in modeling, most progress has been limited to English due to a lack of reliable datasets for most of the world's languages. In this paper, we present the first large-scale NLI dataset (consisting of ~56,000 annotated sentence pairs) for Chinese called the Original Chinese Natural Language Inference dataset (OCNLI). Unlike recent attempts at extending NLI to other languages, our dataset does not rely on any automatic translation or non-expert annotation. Instead, we elicit annotations from native speakers specializing in linguistics. We follow closely the annotation protocol used for MNLI, but create new strategies for eliciting diverse hypotheses. We establish several baseline results on our dataset using state-of-the-art pre-trained models for Chinese, and find even the best performing models to be far outpaced by human performance (~12% absolute performance gap), making it a challenging new resource that we hope will help to accelerate progress in Chinese NLU. To the best of our knowledge, this is the first human-elicited MNLI-style corpus for a non-English language.

1 Introduction

OCNLI addresses the shortage of reliable Chinese NLI data with a large-scale, human-elicited corpus and baseline evaluations. Its strongest model remains substantially below expert human performance, highlighting the dataset’s difficulty.

  • 1 Introduction: The corpus fills a gap left by English-centered NLI resources and translation-based extensions to other languages.The paper identifies automatic translation and limited non-English resources as shortcomings motivating OCNLI.
  • 1 Introduction: OCNLI provides approximately 56,000 Chinese premise-hypothesis pairs created from original Chinese sources by linguistically trained native speakers.The dataset follows MNLI-style collection across five written and spoken genres and reports approximately 98% annotator agreement under a three-vote consensus.
  • 1 Introduction: OCNLI establishes baselines with standard Chinese NLI models and pretrained Chinese transformers, including RoBERTa.The paper uses these models to assess performance on the new dataset and its difficulty.
  • 1 Introduction: Approximately 78% model accuracy versus approximately 90% expert human accuracy leaves a roughly 12-point gap on the test data.The results indicate that current pretrained models remain well behind human performance on OCNLI.

2 Related Work

Prior NLI research is dominated by English datasets and deep learning systems, while non-English resources often rely on translated data. The paper situates OCNLI against translation quality, translationese, cultural-transfer concerns, dataset biases, and the limited availability of human-elicited non-English corpora.

  • Data: English datasets such as SNLI and MNLI have driven NLI research and supported transfer learning, question answering, and model probing beyond NLI.These corpora use human-elicited hypotheses and labels from predefined premises, a procedure also adopted by OCNLI.
  • Data: Most non-English NLI efforts have focused on translating English resources, with few large-scale human-elicited alternatives.The paper notes that existing Portuguese, Japanese, and Persian datasets do not provide human-elicited sentence pairs in the same way.
  • Data: Translated NLI data can contain incorrect or unintelligible sentences, unnatural target-language choices, and source-language cultural context.The cited XNLI examples illustrate poor translation quality and translationese, including copied lexical and syntactic patterns.
  • Systems: Recent NLI systems predominantly use deep learning, with transformer models achieving strong English and Chinese benchmark results.The related work discusses high MNLI performance and reports a Chinese RoBERTa result of 81.2% on XNLI.
  • Biases: NLI datasets exhibit annotator-associated biases, including hypothesis-only cues, motivating bias-reduction and probing research.The paper states that it monitors such biases but leaves systematic filtering, bias reduction, and stress testing for future work.

3 Creating OCNLI

OCNLI adapts MNLI-style collection to Chinese using multi-genre premises, expert native annotators, and multi-hypothesis strategies designed to produce more diverse and challenging inferences. Verification results indicate generally high agreement, while the data also exposes limitations in translated Chinese NLI resources and residual hypothesis-only bias.

  • Data collection: OCNLI uses Chinese premises from five genres and follows MNLI-style premise selection, expert hypothesis elicitation, and three-way NLI labeling.The genres include government documents, news, literature, television talk shows, and telephone conversations.
  • Hypothesis Generation: Multi-hypothesis elicitation asks writers for three entailments, neutrals, and contradictions per premise, creating easy, medium, and hard examples.The method is intended to push annotators beyond the simplest inference strategies used in standard single-hypothesis collection.
  • Hypothesis Generation: MULTI is more challenging than SINGLE but has slightly higher agreement, while it also introduces more hypothesis-only bias from cues such as negators.The authors therefore add encouragement and constraint variants to control this bias, and note that lexical-overlap heuristics remain outside the paper’s probing scope.
  • Hypothesis Generation: MULTIENCOURAGE promotes diverse, high-quality hypotheses, while MULTICONSTRAINT limits each set to one contradiction containing a negator and penalizes violations.Both variants target the use of simple lexical cues, especially negation, in contradiction hypotheses.
  • Data Verification: 97.90% agreement on the medium and hard portions suggests high overall data quality, although agreement is lower for MULTICONSTRAINT.The authors interpret the lower constrained-setting agreement as evidence that producing many hypotheses under explicit constraints can be difficult.
  • Data Verification: Only 67% of Chinese XNLI development pairs matched the authors’ labels, and 11.6% contained more than 10 Roman characters, motivating exclusion of XNLI dev data.The comparison supports concerns about using translated data as a Chinese NLI validation set.

4 Experimental Investigation of OCNLI

Experiments evaluate Chinese NLI baselines, pretrained transformers, translated-data comparisons, annotation artifacts, and subset difficulty. RoBERTa performs strongly against other models but remains substantially below humans, while original human-annotated data outperforms much larger translated data.

  • Baseline Results: Pretrained BERT and RoBERTa outperform non-transformer baselines by approximately 20 percentage points over ESIM.The baseline comparison uses CBOW, biLSTM, ESIM, BERT, and RoBERTa on OCNLI, with mean accuracy reported over five runs.
  • How Difficult is OCNLI?: 78.2% accuracy for RoBERTa remains about 12 points below the 90.3% human score, demonstrating substantial task difficulty.BERT is about 18 points behind humans on OCNLI, compared with roughly 8 points on MNLI; RoBERTa reaches only 61.3% on the CLUE diagnostic task.
  • Annotation Artifacts: Hypothesis-only models achieve about 66% on OCNLI, revealing annotation artifacts involving polarity items, negators, “only,” and “at least.”MULTICONSTRAINT contains no negators among the identified indicators, while “only” correlates with contradictions.
  • Comparison with XNLI: XNLI contains eight times more examples than OCNLI, yet models trained on XNLI perform much worse when tested on OCNLI.The comparison targets whether translated MNLI data can substitute for original Chinese NLI data.
  • Comparison with XNLI: Combining OCNLI with XNLI improves over XNLI alone but remains below the smaller OCNLI-only model except on diagnostics.The combined training set contains 50k OCNLI examples and 392k XNLI examples.
  • Learning Curves: OCNLI reaches its highest CLUE diagnostic performance at 30,000 examples and matches full-XNLI performance at around 25,000 examples.The XNLI learning curve continues improving at 50,000 examples, supporting the value of reliable human annotation.

5 Conclusion

OCNLI is a large-scale, non-translated Chinese NLI corpus built from expert human annotation. Its challenging results and public release are intended to support further Chinese NLU research.

  • OCNLI is a 56,000-pair, non-translated Chinese NLI corpus created by language-study students from five genres using an enhanced MNLI protocol.
  • BERT and RoBERTa results show that current pre-trained transformers remain challenging to use on OCNLI, with the best model about 12 percentage points below human performance.
  • The corpus is publicly available and is intended to accelerate Chinese NLU, including future model probing and sentence representation learning.

B Relabeling Results for Different Portions

Labeler agreement remains comparable to established English NLI datasets across portions of the multi-hypothesis variants. The results support the feasibility and reliability of generating multiple hypotheses per premise.

  • Medium and hard portions of MULTI, MULTIENCOURAGE, and MULTICONSTRAINT generally have lower agreement but remain comparable to SNLI and MNLI.
  • Producing three hypotheses for each label is a feasible and reliable annotation strategy across the evaluated portions.

C Relabeling Results for XNLI Development Set

Relabeling reveals substantial disagreement between annotators and the XNLI development labels. The results also identify Chinese text artifacts that limit XNLI's suitability for Chinese NLI evaluation.

  • 67% of the 200 XNLI pairs received the same label from the study annotators and the XNLI development set.
  • 8.5% of XNLI pairs were judged irrelevant by the annotator majority, while 867 examples contained more than 10 Roman alphabet characters.
  • The low agreement and frequent Roman-alphabet content prevent XNLI from serving as proper evaluation data for Chinese NLI.

D Model Details and Hyper-parameters

The experiments compare conventional neural NLI systems with Chinese BERT and RoBERTa models. The transformer models are fine-tuned using settings selected on the OCNLI development data.

  • The evaluated systems include CBOW, biLSTM, ESIM, Chinese BERT, and Chinese RoBERTa.
  • Chinese BERT is a 12-layer transformer with hidden size 768, pretrained on 0.4 billion Chinese Wikipedia tokens.
  • Chinese RoBERTa uses whole-word masking and extended data, with 24 layers, hidden size 1024, and 5.4 billion pretraining tokens.
  • BERT and RoBERTa are fine-tuned for three epochs with learning rate 2e-5 and batch size 32, using hyperparameters tuned on the authors' development sets.

E Determining Human Baselines

OCNLI human baselines were estimated from independent undergraduate and linguistics-PhD annotators using a controlled labeling procedure. Undergraduate performance was selected as the unbiased baseline, while PhD familiarity may introduce bias.

  • Undergraduate performance was used as OCNLI’s human baseline because it followed the unbiased evaluation procedure.Five undergraduates received instructions and 20 OCNLI training examples before labeling.
  • Five linguistics PhDs showed higher five-label agreement and similar accuracy to undergraduates against OCNLI.test gold labels.Their scores may be biased because they were already familiar with NLI.
  • Human OCNLI performance was similar to MNLI, which reached 92.0% and 92.8% for matched and mismatched conditions, respectively.

F More Examples from OCNLI

This section points readers to additional OCNLI examples and a human-score table, with the available text identifying the table and its annotator-count context.

  • Table 13 presents additional OCNLI sentence pairs, while Table 12 reports human scores for the dataset.
  • The available table text lists 300 annotated pairs for undergraduates and 300 for linguistics PhDs.

G Filtering training data

To match MNLI’s premise separation, the authors removed training pairs whose premises appeared in evaluation, creating OCNLI.train.small. Models lost 1.5–2.5% performance, likely because the filtered set retained only 60% of the original training data.

  • Filtering removed about 20,000 training pairs whose premises also appeared in development or test data, leaving OCNLI.train.small at 60% of OCNLI.train.
  • Models trained on OCNLI.train.small lost 1.5–2.5% performance relative to training on OCNLI.train.The authors describe this decrease as moderate and expected, attributing it more likely to reduced training data than premise-overlap removal.
  • Both training sets were planned for public release and inclusion in the OCNLI leaderboard.
Loading 2010.05444v1…