Source-linked AI summary
A Holistic Approach to Undesired Content Detection in the Real World
Todor Markov, Chong Zhang, Sandhini Agarwal, Tyna Eloundou, Teddy Lee, Steven Adler, Angela Jiang, Lilian Weng
TL;DR
Real-world content moderation needs robust handling of ambiguous taxonomies, inconsistent labels, distribution shift, and rare undesired content. The paper presents a holistic approach combining taxonomy and labeling design, data quality control, active learning, and robustness methods. It reports that active learning captures rare undesired samples more effectively and concludes that multiple coordinated methods are needed for practical systems.
Problem
Real-world moderation is limited by taxonomy ambiguity, subjective labeling, production-data distribution shift, and the rarity of some undesired categories.
Method
The paper combines taxonomy and labeling design, data quality control, active learning, model-training methods, and robustness techniques for real-world moderation.
Results
Active learning captures up to 22× more rare undesired samples and can improve rare-category model performance by up to 10×.
Takeaways & Limitations
Practical moderation systems require coordinated taxonomies, reliable data pipelines, active learning for rare content, and additional methods such as curated synthetic data and semi-supervised learning.
Takeaways & Limitations
Only about 5% of training samples are non-English, so non-English performance has not been rigorously evaluated or optimized.
Abstract
from arXiv · showhide
We present a holistic approach to building a robust and useful natural language classification system for real-world content moderation. The success of such a system relies on a chain of carefully designed and executed steps, including the design of content taxonomies and labeling instructions, data quality control, an active learning pipeline to capture rare events, and a variety of methods to make the model robust and to avoid overfitting. Our moderation system is trained to detect a broad set of categories of undesired content, including sexual content, hateful content, violence, self-harm, and harassment. This approach generalizes to a wide range of different content taxonomies and can be used to create high-quality content classifiers that outperform off-the-shelf models.
1 Introduction
Real-world content moderation requires broad taxonomies, careful labeling and quality control, and methods for handling distribution shift and rare undesired content. The paper proposes a holistic system combining these elements with active learning and publicly available data.
- Data sources: Public data requires care because taxonomy and distribution differences can limit its usefulness for the target moderation problem.The paper combines public data with production-oriented approaches rather than treating public datasets as universally sufficient.
- Scope: The system covers sexual content, hateful content, violence, self-harm, harassment, and severe subcategories.The taxonomy is intended to support a more comprehensive detection system across varied use cases.
- Challenges: Real-world moderation must address taxonomy ambiguity, subjective labeling decisions, production traffic distribution shift, and rare undesired categories.Observed examples included self-harm in 0.04% of sampled prompts and hateful content involving threats in 0.017%.
- Data quality: Detailed labeling instructions, calibration, and quality control are needed to produce consistent training data.Imprecise instructions can increase subjective judgments and inconsistent labels, while poor quality metrics can hurt model performance.
- Active learning: Active learning can capture up to 22× more undesired samples for rare events and improve rare-category model performance by up to 10×.The approach targets the cold-start problem by discovering undesired samples in production traffic.
2 Taxonomy
The taxonomy organizes undesired content into broad categories and severity-based subcategories while recognizing that context, use case, users, and regions affect what is considered undesired.
- Scope: A universal taxonomy is challenging because sentence meaning and judgments about undesired content depend on context.The dataset was designed for broad applicability but should be critically evaluated before use in particular contexts.
- Granularity: Severity subcategories distinguish harmful cases from neutral or contextualized content within the same broad category.Examples include S3, H2, and V2 as severe subcategories, while S0, H0.a, H0.b, and V0 are not considered undesired in specified contexts.
- Granularity: Subcategorization captures differences in potential repercussions and can reduce labeling disagreement and corner cases.The paper contrasts calls for violence with less severe derogatory statements as an example of differing consequences.
- Top-level categories: The taxonomy uses top-level categories for sexual, hateful, violent, self-harm, and harassing content.The model predicts eight selected categories, including five top-level categories and three severe subcategories.
- Category boundaries: The sexual-content definition maps one category boundary to activities that could be illegal in real life today, with legality mapped most closely to the USA.Nonerotic or contextualized sexual content, including medical or sex education material, is excluded from undesired sexual content.
3 Methods
The system combines iterative production-data collection, active learning, labeling-quality interventions, synthetic data, domain adaptation, and model probing to build robust real-world moderation models.
- Production data and active learning: Production data is collected iteratively through random selection, model-based scoring, active-learning subsampling, and metadata-based reweighting.The final mixture is updated as data distributions and target categories change.
- Data quality: Detailed labeling definitions, calibration sessions, targeted audits, and cross-model disagreement checks improve data correctness and consistency.The audit strategy accounts for category imbalance, while flagged samples are reviewed when disagreement suggests possible mislabeling.
- Synthetic data: Synthetic data supports cold-start initialization, rare-category augmentation, and mitigation of counterfactual demographic bias.Zero-shot prompts initialize categories with little public data, few-shot prompts augment rare categories, and contrastive templates separate identity terms from hateful descriptions.
- Synthetic data: Mixing large amounts of noisy synthetic data into a large high-quality training set hurt model performance.The authors suspect that synthetic-data noise confuses the model and lowers learning efficiency in this setting.
- Domain adaptation: Wasserstein Distance Guided Domain Adversarial Training encourages domain-invariant representations when public datasets differ from production traffic.The method uses a feature extractor, classification head, and domain critic within a minimax objective alongside classification loss.
- Model probing: Key-token probing identifies overfitting, including inflated hate predictions associated with “#” and racial tokens such as black.The desired distinction is between hateful references to people and benign references such as “I hate black cats!”
4 Experiment Results
The experiments evaluate the moderation model across public and production-related settings, comparing baselines, active learning, and domain-adversarial training. Results show stronger rare-content discovery with active learning and context-dependent benefits from domain adaptation.
- The model uses a lightweight transformer decoder with eight independent MLP heads, one per category, to reduce interference between categories.Each head has shape [dmodel, 256, 1].
- The model is evaluated on public-data test sets and several external undesired-content benchmarks, while production test traffic remains unavailable for privacy and legal reasons.
- 4.3 Active Learning Experiments: About 40% of actively selected samples triggered at least one undesired label, compared with 3.4% of randomly selected samples.The authors report that active learning captured undesired content more than 10 times more effectively across all categories.
- 4.3 Active Learning Experiments: Active learning produced greater AUPRC improvement than random sampling across all categories after three iterations.The comparison uses the same validation set at each model iteration.
5 Related Work
Prior work has developed definitions and detectors for toxic, hateful, offensive, and abusive language, but category overlap and limited comprehensive coverage complicate dataset sharing and real-world moderation. This work situates its approach alongside active learning and red-teaming methods for discovering difficult examples and model failures.
- Overlapping but nonidentical definitions of hatefulness, toxicity, offensiveness, and abusiveness create obstacles for sharing datasets across projects.
- Research has increasingly examined rigorous taxonomies, but detailed subcategorizations and broad real-world deployment remain limited.
- Existing content-detection research often targets limited categories or specific use cases such as toxic-comment moderation.
- Training-data standards remain underdeveloped, while synthetic data offers lower collection costs but may not be diverse enough for real-world distributions.
- Active learning selects informative samples across domains, while red-teaming iteratively exposes model weaknesses through human or adversarial data collection.
6 Future Work and Limitations
The paper identifies unresolved limitations involving demographic bias, data augmentation, multilingual evaluation, red-teaming scalability, and active-learning strategy design. Future work targets broader robustness, coverage, and operational efficiency.
- Bias and Fairness: The models may produce higher hate predictions for inputs containing “gay” and higher sexual predictions for inputs containing “her.”Balanced synthetic templates were tried but did not fully eliminate the issue.
- Data Augmentation: Data augmentation remains a planned direction for improving lexicon robustness and model generalizability under changing real-world data distributions.
- Better Multilingual Support: Only about 5% of training samples are non-English, and non-English performance has not yet been rigorously evaluated or optimized.Improved multilingual classification may require more non-English data and changes to tokenization or model architecture.
- Red-teaming at scale: Internal red-teaming is conducted for each model version, but this approach is not scalable.The authors plan a production-like pipeline and specialized interface to improve red-teamers’ efficiency.
- More Active Learning Experiments: The current active-learning strategy is simple and does not explore diversity sampling because of computational restrictions.Future experiments will compare uncertainty, diversity, and combined sampling strategies.
7 Broader Impacts
Content moderation classifiers can support safer language-model development by operationalizing policies, filtering datasets, and evaluating models, but they depend on normative choices and may introduce bias. The paper therefore emphasizes contextual evaluation, ethical data practices, and complementary mitigation strategies.
- Classifiers can operationalize policies on model inputs and outputs, filter datasets for training, and support language-model evaluation.
- Classifier assumptions and normative decisions can make tools vulnerable or inappropriate for particular use cases and text types.
- Problematic biases may produce disproportionate false positives for groups frequently targeted by hate.
- The taxonomy was designed for generalizable viewpoints but reflects a largely US-centric dataset context and should be adapted to deployment contexts.
- The taxonomy should be critically evaluated for each context, used with other mitigation strategies, and updated as use cases and deployment settings change.
- Annotation disagreement can reflect meaningful uncertainty and minority perspectives rather than merely labeling error.
- Annotators were vetted, warned about risks, given mental-health resources, and allowed to opt out of sensitive labeling tasks.
8 Conclusion
The paper concludes that robust real-world undesired-content detection requires coordinated taxonomy, data, labeling, active-learning, and modeling choices. It presents content detectors as one means of controlling generative-model outputs while anticipating continued refinement.
- High-quality real-world content detection requires multiple methods spanning taxonomy design, data pipelines, active learning, labeling operations, synthetic data, and semi-supervised learning.
- Active learning substantially improves performance when target content occurs rarely, while curated synthetic data and semi-supervised learning can provide further gains.
- Content detection models demonstrate one way to control and guide increasingly prevalent generative language-model outputs.
A Experiment Details
The evaluation maps the model’s taxonomy to labels in several external datasets, with some mappings only approximate. The datasets include hate, offensive, violence, toxic, and erotic-content evaluations.
- Evaluation taxonomy mapping: Some taxonomy mappings are approximate rather than exact.Perspective’s threat label excludes graphic violence, so it does not perfectly match the model’s violence category.
- Evaluation datasets: The evaluations cover hate and offensive tasks, the Stormfront hate-speech dataset, Reddit erotic-content data, and Jigsaw toxic-comments data.The cited evaluation resources include 2,970 hate-test samples, 860 offensive-test samples, 478 Stormfront samples, and 5,000 Reddit examples.
- Evaluation taxonomy mapping: Table 6 maps taxonomies from different APIs into labels used by various evaluation datasets.The table documents how model categories correspond to benchmark labels.