PaperlayerView exact revision on arXiv

Source-linked AI summary

COLD: A Benchmark for Chinese Offensive Language Detection

Jiawen Deng, Jingyan Zhou, Hao Sun, Chujie Zheng, Fei Mi, Helen Meng, Minlie Huang

arXiv:2201.06025v2cs.CLcs.AI

TL;DR

Chinese offensive language detection lacks reliable datasets and detectors despite its importance for social platforms and language-model deployment. The paper introduces the COLD benchmark with COLDATASET and COLDETECTOR, then evaluates Chinese generative models and identifies inputs that more easily trigger offensive outputs. The results show that non-offensive inputs, especially anti-bias language, can induce safety problems comparable to offensive inputs.

  • Problem

    Chinese offensive language detection remains underexplored because reliable labeled datasets and detectors are scarce, despite the task's importance for social platforms and language-model deployment.

  • Method

    The paper introduces COLD, combining the COLDATASET with a baseline COLDETECTOR and using the detector to evaluate offensive generation in popular Chinese language models.

  • Results

    Chinese generative models show varying offensive-generation risks, and both offensive and non-offensive inputs can trigger offensive outputs, with anti-bias contents among the easier triggers.

  • Takeaways & Limitations

    Non-offensive inputs, particularly anti-bias language, warrant the same safety attention as offensive inputs in Chinese language-model evaluation.

  • Takeaways & Limitations

    The dataset focuses on sentence-level offensiveness without contextual information, so context-sensitive expressions such as dialogue require further exploration.

Abstract

from arXiv · show

Offensive language detection is increasingly crucial for maintaining a civilized social media platform and deploying pre-trained language models. However, this task in Chinese is still under exploration due to the scarcity of reliable datasets. To this end, we propose a benchmark --COLD for Chinese offensive language analysis, including a Chinese Offensive Language Dataset --COLDATASET and a baseline detector --COLDETECTOR which is trained on the dataset. We show that the COLD benchmark contributes to Chinese offensive language detection which is challenging for existing resources. We then deploy the COLDETECTOR and conduct detailed analyses on popular Chinese pre-trained language models. We first analyze the offensiveness of existing generative models and show that these models inevitably expose varying degrees of offensive issues. Furthermore, we investigate the factors that influence the offensive generations, and we find that anti-bias contents and keywords referring to certain groups or revealing negative attitudes trigger offensive outputs easier.

1 Introduction

Chinese offensive language research lacks reliable open resources, while generative models can produce offensive content from both offensive and non-offensive inputs. COLD addresses these gaps with a dataset, baseline detector, and analyses of offensive generation triggers.

  • Existing Chinese offensive language detection lacks labeled datasets and reliable detectors, limiting study of Chinese language-model safety.
  • COLD provides COLDETECTOR, a baseline detector based on pretrained Chinese BERT and fine-tuned on COLDATASET.
  • The benchmark evaluates popular Chinese generative models and finds that both offensive and non-offensive inputs can trigger offensive outputs.
  • Anti-bias contents, target-group keywords, and negative-attitude words more easily trigger offensive outputs than other inputs.
  • COLDATASET is a publicly available Chinese offensive language dataset containing 37,480 sentences across race, gender, and region.

2 Related Work

The paper treats offensive, toxic, and hate speech interchangeably and reviews automatic detection and language-model safety analysis. It highlights the scarcity of reliable Chinese resources and motivates COLD as a benchmark for Chinese offensive language detection.

  • Definitions: The paper uses offensive language as an umbrella term for targeted offenses toward individuals or groups, including rudeness, insults, threats, and profanity.The paper does not distinguish offensive language, toxic language, and hate speech.
  • Detection research: Automatic offensive-language detection is important for detoxifying online communities and supporting safer deployment of large-scale language models.Prior efforts include topic analysis and keyword-based detection, alongside methods enabled by deep learning and pre-trained models.
  • Language-model analysis: Previous language-model analyses use templates such as fill-in-the-blank cloze statements to examine safety issues including social bias.These studies investigate relational knowledge presented in training data and stored in pre-trained language models.
  • Chinese resources: Reliable resources for Chinese offensive-language detection remain scarce despite the effectiveness of data-driven detection and safety-evaluation methods.The paper compares COLDATASET with other related Chinese datasets in Table 1.
  • Benchmark motivation: This work aims to provide resources and establish a benchmark supporting the development of Chinese offensive-language detection.The stated resources-and-benchmark goal addresses the limited Chinese research infrastructure described in the related work.

3 Dataset Construction

COLDATASET is a Chinese offensive-language dataset built from real-world social-media data, covering racial, gender, and regional topics with binary and fine-grained annotations. Its collection combines keyword querying, subtopic crawling, model-in-the-loop filtering, and multi-annotator quality control.

  • Dataset overview: COLDATASET contains 37,480 Chinese comments covering race, gender, and region, with fine-grained test labels for attacking individuals, attacking groups, anti-bias, and other non-offensive content.
  • Data sources: The data combine keyword querying with crawling related subtopics to capture both high-density topic examples and broader user expressions.
  • Model-in-the-loop collection: Model-in-the-loop collection uses an initialized classifier to rank unlabeled candidates, followed by stratified sample checking and manual labeling.
  • Annotation quality: Test-set annotation assigns each sample to three annotators, using majority voting and reporting Fleiss’ κ of 0.819 for binary labels and 0.757 for four-class labels.
  • Data analysis: Topic distributions are relatively even, about 10% of data lack topic keywords, and overlaps such as race–gender and region–gender make the dataset more diverse.

4 Offensive Language Detection

The study evaluates Chinese offensive-language detectors using COLDATASET and existing resources, finding that the proposed COLDETECTOR performs best overall but remains vulnerable to implicit and anti-bias content. Comparisons also show that translated data and keyword-based methods transfer poorly to the original Chinese task.

  • Detection methods: COLDETECTOR uses a pretrained Chinese BERT encoder with a linear prediction layer and is fine-tuned on COLDATASET.
  • Fine-grained performance: COLDETECTOR performs strongly on attacking individuals and groups, with 79.51% and 85.49% accuracy, but reaches only 38.32% on anti-bias data.
  • Error analysis: Acknowledgment-followed-by-denial expressions can cause COLDETECTOR to focus on an initial offensive-looking clause and miss the subsequent anti-bias statement.
  • Limitations: The detector still performs worse on covert offensiveness and anti-bias samples, and semi-automatically collected training data introduce unavoidable noise.
  • Existing resources: Existing resources alone struggle on COLDATASET: keyword matching obtains 54% accuracy, while translated-data detection drops from 91% on its own test set to 60% on COLDATASET.

5 Evaluation of Generative LMs

The study evaluates popular Chinese generative language models using COLDATASET prompts and COLDETECTOR, finding model- and prompt-dependent offensive-generation risks. Anti-bias content, target-group keywords, and negative-attitude words are especially associated with offensive outputs.

  • Evaluation setup: COLDATASET sentences serve as prompts, while COLDETECTOR measures generated-content offensiveness using offensive rate.Offensive rate is the proportion of offensive generations among all generations; lower values indicate safer models.
  • Evaluated models: The evaluation covers CPM, CDialGPT, and EVA, including the specified CPM-Generate, CPM-Generate-distill, CDialGPT-Base, and CDialGPT-Large variants.The models differ in architecture, parameter scale, and training data.
  • RQ1: Offensiveness of Different Models: 32.14% overall offensive rate was observed for CPM-Generate, rising to 43.43% under offensive prompts.CPM-Generate had the greatest offensive exposure, while CDialGPT and EVA were described as safer than CPM models.
  • RQ2: Offensiveness Triggered by Different Prompts: Both offensive and non-offensive prompts can induce offensive generations across the evaluated models.The automatic-evaluation table explicitly reports risks from both prompt types, while human and automatic evaluations are presented separately.
  • RQ2: Offensiveness Triggered by Different Prompts: Anti-bias inputs show especially high risk, partly because they contain target-group keywords and negative-attitude words associated with more toxic outputs.The analysis reports model sensitivity to particular groups and negative attitudes, with anti-bias content combining these triggering factors.

6 Conclusion

The paper presents COLDATASET and COLDETECTOR for Chinese offensive-language analysis and uses them to evaluate generative models. It concludes that non-offensive inputs, especially anti-bias language, can also pose substantial safety risks.

  • Contributions: COLDATASET provides a dataset for Chinese offensive-language analysis, and COLDETECTOR effectively detects offensive content on the proposed data.The detector also serves as a benchmark for evaluating language-model offensiveness.
  • Findings: The evaluated Chinese language models show different degrees of risk when generating offensive content.The benchmark is used to compare offensiveness across popular generative models.
  • Findings: Non-offensive inputs can induce safety problems similarly to offensive inputs, while anti-bias language is often overlooked despite comparable hazards.The conclusion identifies anti-bias language as a particularly underexamined non-offensive input type.
  • Implications: The benchmark is intended to provide a basis for Chinese safety research and support further study of inputs that induce unsafe generation.The authors connect this research direction with safer deployment of language models.

Ethical Considerations

The ethical discussion identifies annotation uncertainty, narrow topic coverage, incomplete offensiveness detection, and possible misuse as important boundaries of the benchmark. The authors state that the resources are intended for safer language-model deployment despite these risks.

  • Data risks: Manual subjectivity and semi-automatic, incomplete training-data annotation may introduce mislabeled data.The authors suggest optional re-annotation of semi-automatically labeled training data when needed.
  • Data risks: The dataset covers only common race, gender, and region topics and uses a simple annotation schema.The authors call for broader topic coverage and a more fine-grained taxonomy.
  • Detector scope: The benchmark detector cannot detect all forms of offensiveness because of data-coverage and neural-network training limitations.This constrains the scope of conclusions drawn from detector outputs.
  • Data use: The dataset consists of content from publicly available social platforms, and the authors state that its contents do not represent their views or opinions.The paper says it follows protocols for using the data sources.
  • Misuse risks: The resources and analyses could be misused to create more offensive language or make people uncomfortable.The authors nevertheless judge the benchmark's value to outweigh these risks for building safer online communities and language models.

Limitations

The paper's main limitations are the absence of contextual information and the restricted baseline-model comparison. These choices limit evaluation of context-sensitive offensiveness and the strength of model comparisons.

  • Limitation of contextual information: COLDATASET studies offensiveness mainly at the sentence level and excludes contextual information.The authors identify context-sensitive scenarios such as dialogue as requiring further exploration.
  • Limitation of baseline models: The baseline experiments use BERT-base-Chinese as the backbone for COLDETECTOR, TJIGDET, and PSELFDET.The authors note that broader backbone coverage, including mBART and XLM-Roberta, would enable a more solid comparison.

A Data Statement

COLDATASET is a Chinese offensive-language resource collected from Zhihu and Weibo, with topic coverage, annotation procedures, and stakeholder uses documented in a data statement. Its collection combines rule-based crawling and post-processing with an iteratively optimized classifier to identify candidate data.

  • Dataset: COLDATASET contains 37,480 sentences covering racial, gender, and regional bias, with 32,157 semi-automatically labeled training examples and 5,323 manually labeled test examples.The test set uses four fine-grained categories: Attack Individual, Attack Groups, Anti-Bias, and Other Non-Offen.
  • Speaker: The dataset is collected from Zhihu and Weibo, whose users serve as the speakers generating the data.
  • Annotator and curator: Seventeen trained native Chinese workers label the data, while the authors define the scope and taxonomy, design guidelines, train annotators, and control quality.
  • NLP system: Rule-based crawling and post-processing are aided by an iteratively optimized classifier that selects candidate data for further manual annotation.
  • Stakeholders: COLDATASET supports researchers, social-platform managers optimizing offensive-language filters, and language-model developers evaluating offensiveness for safer deployment.
  • Collection: Collected keywords target racial, gender, and regional topics, while samples are restricted to 5–200 tokens, cleaned, and deduplicated before model-in-the-loop selection.

B.3 Model-in-the-loop Collection

The model-in-the-loop procedure iteratively expands and improves the dataset by using a classifier to select examples for annotation. Annotation distinguishes offensive targets and separates anti-bias from other non-offensive content, while prompt-based self-detection remains sensitive to token-pair and prompt choices.

  • Model-in-the-loop procedure: The collection starts with 500 manually labeled sentences and iteratively trains a classifier, selects candidate training and test data, and updates both datasets.The classifier uses a 12-layer BERT model during data collection.
  • Dataset expansion: The dataset expands over six rounds: both training and test data grow in the first five, while only training data grows in the sixth.
  • Early rounds: The classifier reaches only 58% accuracy in the first round, with predictions concentrated between 0.2 and 0.5 and a tendency to label samples Non-Offensive.Samples from this interval are selected for annotation to improve the classifier.
  • Later rounds: After the third round, performance stabilizes and the accuracy of predicted high-scoring samples increases, so more high-scoring data are selected for efficient collection.
  • Annotation guidelines: Annotation first labels offensiveness, then distinguishes Attack Individuals or Attack Groups, while non-offensive samples are divided into Anti-Bias or Other Non-Offensive.
  • Annotation guidelines: Anti-Bias is separated because it counters offensiveness and is considered beneficial for harmonious communication, making it a distinct object of study.
  • Prompt-based self-detection: Prompt-based self-detection uses Chinese BERT scores for candidate words, and different word pairs and prompts significantly affect detection results.
  • Additional data: Translated Jigsaw data are also evaluated as a source for this task, using 59k English examples translated into Chinese.

D.3 Keyword Matching

Keyword matching performs unsatisfactorily on COLDATASET because sensitive words occur in both offensive and non-offensive content. Some strongly offensive keywords also appear in anti-bias samples, limiting their reliability as offensive indicators.

  • Performance: Keyword matching performs unsatisfactorily on COLDATASET because sensitive words can occur in both offensive and non-offensive sentences.
  • Why matching fails: Most sensitive words occur in both offensive and non-offensive samples, while some strongly offensive words also appear in anti-bias content.
  • Why matching fails: Although some keywords appear only in offensive samples, the paper expects them to occur in non-offensive samples as dataset scale and coverage increase.

E Details of Evaluation

The evaluation examines offensive generations using COLDETECTOR across target-group keywords, prompt templates, and generation lengths. It also identifies a scope boundary: the sentence-level detector is insufficient for context-sensitive dialogue behaviors.

  • Generation setup: The generation study constructs 927 prompts from 103 target-group keywords and 9 templates, producing 20 CPM-Generate responses per prompt with a 200-token maximum.
  • Generation factors: The study analyzes how offensive-generation ratios vary across target-group keywords, prompt templates, and maximum generation lengths.
  • Detection findings: COLDETECTOR discovers offensive generations, including cases triggered by both offensive and non-offensive input content.
  • Limitation: COLDETECTOR focuses on sentence-level offensive language and is insufficient for context-sensitive dialogue cases such as offending-user and toxicity-agreement behaviors.
Loading 2201.06025v2…