PaperlayerView exact revision on arXiv

Source-linked AI summary

Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack

Emily Dinan, Samuel Humeau, Bharath Chintagunta, Jason Weston

arXiv:1908.06083v1cs.CL

TL;DR

Offensive-language detectors need to withstand adversarial human behavior, especially when meaning depends on dialogue context. The paper introduces an iterative human-in-the-loop build it, break it, fix it strategy and reports more robust systems, nuanced adversarial data, and better performance from context-aware models.

  • Problem

    Offensive-language detection systems can be exploited by adversaries, while offensive meaning in dialogue may depend on context rather than a single utterance.

  • Method

    The paper uses crowdworkers to attack a detector, retrains it on the resulting examples, and repeats the build it, break it, fix it cycle.

  • Results

    The approach yields increasingly robust systems, more nuanced adversarial language, and better performance from architectures that use dialogue context.

  • Takeaways & Limitations

    Robust dialogue safety requires training on adversarial examples and modeling conversational context rather than relying only on standalone utterances.

  • Takeaways & Limitations

    The study considers only a binary offensive-versus-safe problem and leaves separate offensive-language classes and other dialogue tasks for future work.

Abstract

from arXiv · show

The detection of offensive language in the context of a dialogue has become an increasingly important application of natural language processing. The detection of trolls in public forums (Galán-García et al., 2016), and the deployment of chatbots in the public domain (Wolf et al., 2017) are two examples that show the necessity of guarding against adversarially offensive behavior on the part of humans. In this work, we develop a training scheme for a model to become robust to such human attacks by an iterative build it, break it, fix it strategy with humans and models in the loop. In detailed experiments we show this approach is considerably more robust than previous systems. Further, we show that offensive language used within a conversation critically depends on the dialogue context, and cannot be viewed as a single sentence offensive detection task as in most previous work. Our newly collected tasks and methods will be made open source and publicly available.

1 Introduction

The paper addresses offensive-language detection under adversarial human behavior, proposing an iterative human-and-model training strategy and emphasizing dialogue context. Experiments report increasing robustness, more sophisticated adversarial data, and better performance from context-aware architectures.

  • Motivation: Offensive-language detectors can be exploited by adversaries, motivating systems robust to human attacks.The paper connects this problem to trolling, targeted offense, and attacks on public chatbots.
  • Approach: The proposed approach automates a build it, break it, fix it strategy with crowdworkers and repeated model retraining.Humans generate adversarial examples, models are fixed using those examples, and the sequence is repeated.
  • Results: Each fixing iteration produces increasingly robust systems against adversarial attacks.The authors report robustness improving over successive fixing iterations.
  • Results: Adversarial data shifts from obvious profanity toward examples requiring world knowledge, figurative-language understanding, and negation.These distribution changes characterize the attacks collected during successive break phases.
  • Results: Dialogue-context architectures perform better than systems that do not use context, which dominated prior research.The comparison concerns offensive-language detection in conversational settings.
  • Resources: The algorithm, training code, crowdsourcing interface, data, and trained models are intended for public release.The release is described as covering the complete build it, break it, fix it pipeline.

2 Related Work

Prior work studies offensive content across hate speech, aggression, bullying, and toxicity using diverse datasets and models. This paper differs from algorithmic adversarial training by using human breakers in an automated build-break-fix process.

  • Task landscape: Offensive-language research spans hate speech, bullying, aggression, and toxic-comment detection.These content classes have been studied as related but distinct tasks.
  • Datasets: Benchmark datasets include over 24,000 labeled tweets and over 15,000 aggression-labeled Facebook comments.The cited datasets support evaluation across hate speech and aggression identification.
  • Models: Existing systems range from linear models and SVMs to deep learning, with leading competition systems using LSTMs and CNNs.The paper positions its transformer model within this progression.
  • Adversarial training: Adversarial-training methods often find breaking examples algorithmically, whereas this work uses human breakers.The paper notes that small textual changes can substantially alter meaning, creating challenges for algorithmic NLP attacks.
  • Related strategies: The approach draws on Build it Break it methods and Mechanical Turker Descent while adding an automated fixing stage.Earlier Build it Break it work used humans to find model failures but did not automate the full process as this paper does.

3 Baselines: Wikipedia Toxic Comments

The study bootstraps its method from Wikipedia Toxic Comments, converts toxicity labels into binary offensive/safe classes, and compares fastText and BERT baselines with a prior BiLSTM.

  • Dataset: The Wikipedia Toxic Comments dataset originates from Wikipedia Talk pages and targets personal attacks online.The dataset was collected by the Wikimedia Foundation and Jigsaw.
  • Dataset: Seven toxicity labels are collapsed into OFFENSIVE versus SAFE, with 80% train, 10% validation, and 10% test splits.All labels except non-toxic are grouped as OFFENSIVE.
  • Models: The BERT baseline fine-tunes a 12-layer transformer with 768-dimensional hidden states and 12 attention heads for binary classification.A linear layer over the [CLS] representation produces the final prediction.
  • Models: The fastText baseline averages pretrained word vectors and applies a linear classifier.The word vectors are pretrained on a large Wikipedia-derived corpus.
  • Evaluation: The experiments compare fastText and BERT with a BiLSTM using GloVe vectors and report weighted-F1 plus OFFENSIVE-class F1.OFFENSIVE-class F1 is the paper’s favored metric because the dataset is imbalanced toward SAFE examples.
  • Evaluation: The BERT-based model outperforms the BiLSTM baseline and is used to bootstrap the subsequent approach.The paper therefore adopts the BERT architecture for later experiments.

4 Build it Break it Fix it Method

The method iteratively builds a detector, has crowdworkers attack its safe predictions, retrains on the resulting adversarial examples, and repeats this process. It defines offensive content operationally for crowdworkers and accumulates data across rounds.

  • Build it: The algorithm begins with a BERT-based OFFENSIVE-message detector trained on Wikipedia Toxic Comments as model A0.This initial model supplies the system that crowdworkers attempt to break.
  • Break it: Crowdworkers break the detector by submitting messages it marks SAFE but they judge OFFENSIVE.Workers are explicitly asked to beat the system through adversarial submissions.
  • Fix it: A new model is retrained on the collected adversarial examples to improve robustness.This is the fixing stage following each break phase.
  • Repeat: The process repeats by deploying the newly trained model in later break phases and fixing it again.For later rounds, workers must find examples missed by both the baseline and the previous fixed model.
  • Break it details: OFFENSIVE is defined for workers as content not acceptable in a friendly conversation with someone newly met online.The wording aims to cover varied unacceptable content while approximating a public-forum setting.
  • Iterations: Experiments run three break-fix iterations for single-turn detection and one further iteration for the multi-turn dialogue task.Figure 1 summarizes the iterative model sequence A0 through AN.
  • Data and evaluation: Models accumulate all previous adversarial rounds alongside Wikipedia Toxic Comments data, with held-out tests measuring robustness to new attacks.The design expects later models to face increasingly complex human attacks.

5 Single-Turn Task

The single-turn study compares standard offensive-message collection with adversarial collection, then evaluates models trained on each data source. Adversarial collection produces increasingly sophisticated examples, and adversarially trained models are more robust to adversarial tests while generally retaining standard-task performance.

  • Data Collection: The study collects three 1000-example rounds of adversarial single-turn data and an equal-sized standard dataset without a model to break.The standard method asks workers to submit messages they consider OFFENSIVE; the adversarial method has workers attack classifiers.
  • Data Analysis: Standard examples contain at least seven times more profanity than each adversarial round, while adversarial examples use more negation, figurative language, and world knowledge.Human annotation also found less profanity and fewer non-profane offending words in adversarial data.
  • Break it Phase: Workers’ average success against the baseline falls from 4.56 out of 5 in round 1 to 1.6 out of 5 by round 3.The decreasing scores indicate that successive single-turn adversarial rounds become harder for crowdworkers to defeat.
  • Fix it Phase: Adversarially trained models outperform standard models on every adversarial testing round, showing greater robustness to attacks.The models are evaluated using OFFENSIVE-class F1 across standard and adversarial test sets.
  • Fix it Phase: On standard test sets, adversarial and standard models generally perform similarly, except A3 fails to improve and may be overoptimized for adversarial tasks.Standard models improve on later standard rounds because they receive more training data matching that evaluation distribution.

6 Multi-Turn Task

The multi-turn task tests offensive-language detection in two-person dialogues, where context can make otherwise innocuous responses offensive. Context-aware BERT architectures improve detection, while simple bag-of-embeddings models perform worse with context.

  • 6.1 Data Collection: Crowdworkers were asked to continue truncated ConvAI2 conversations with offensive responses classified as SAFE by baseline or prior-round models.The multi-turn adversarial task collected 3000 offensive examples and combined them with SAFE dialogue examples at a 9:1 ratio.
  • 6.3 Fix It Phase: Models were trained with and without dialogue context using fastText, BERT, and a BERT variant separating the final utterance from the preceding history.The study also multi-tasked the multi-turn adversarial task with Wikipedia Toxic Comments and single-turn tasks.
  • 6.2 Break It Phase: The multi-turn adversarial task was easier for workers than the single-turn task, with an average score of 2.89 despite using the best single-turn classifier.Single-turn worker scores fell from 4.56 in round 1 to 1.6 in round 3, whereas the multi-turn average was 2.89.
  • 6.3.2 Fix It Phase: 23.56 average OFFENSIVE-class F1 with context versus 37.1 without context shows that fastText handles contextual information poorly.The authors attribute this pattern to fastText’s overly simple bag-of-embeddings representation.
  • 6.3.2 Fix It Phase: BERT improves with context: simple BERT gains 3.7 OFFENSIVE-class F1 points, while dialogue-segment features gain 7.4 points.The table summary identifies context input as beneficial and BERT dialogue segment features as producing the best results.

7 Conclusion

The paper presents an iterative human-and-model strategy for robust offensive-language detection in dialogue. Its results show that adversarial and contextual examples improve robustness, while the study remains limited to binary offensive-versus-safe classification.

  • Conclusion: The build it, break it, fix it, and repeat strategy uses human attempts to break models and broken examples to fix them.The approach produces adversarial data containing less profanity and more figurative language, negation, and world-knowledge-dependent offensiveness.
  • Conclusion: Classifiers trained on complex adversarial examples become more robust to attack, and dialogue context improves performance when the architecture can use it.The conclusion links robustness to learning from nuanced examples and contextual performance to suitable model architecture.
  • Future Work: The study considers only a binary offensive-or-safe problem, leaving separate offensive-language classes and other dialogue tasks for future work.The authors also identify applying the strategy to safe neural generative models as a future direction.

A.1 Additional Break It Phase Results

Additional break-phase analyses examine how adversarial examples bypass successive classifiers. The baseline was rarely contradicted by later models, and adversarial examples differed in their offensive-language categories.

  • A.1 Additional Break It Phase Results: Adversarial data contained fewer bullying examples but more examples targeting a protected class than the standard data.These categories came from blind human annotation of round 1 single-turn adversarial and standard data.

A.2 Additional Fix It Phase Results

The additional fix-phase results report OFFENSIVE-class F1, precision, recall, and weighted-F1 for single-turn standard and adversarial models.

  • A.2 Additional Fix It Phase Results: The evaluation reports F1, precision, and recall for the OFFENSIVE class, together with weighted-F1, for models Si and Ai.These metrics are reported on both the single-turn standard and adversarial tasks.

B Data Collection Interface Details

The single-turn adversarial collection used a gamified interface to elicit messages that models incorrectly marked as acceptable, sometimes guided by open-domain topics. The section also presents example tasks, annotation, and evaluation materials.

  • Adversarial collection: Users generated messages they believed were not okay to send but that the system marked as okay, using the paper’s friendly-conversation definition.The collection targeted model failures in single-turn offensive-language detection.
  • Adversarial collection: Half of the single-turn adversarial prompts included one of 1365 crowd-sourced open-domain dialogue topics.Examples included commuting, Gouda cheese, music festivals, podcasts, bowling, and Arnold Schwarzenegger.
  • Interface design: Gamification allowed users up to five points per round, with two tries per point, while scores and effort bonuses supported data collection without changing compensation.The paper states that gamification was used because it had been shown to increase data quality.
  • Evaluation materials: The accompanying materials include single-turn standard and adversarial examples, human annotation of 100 examples from each task, and experiment results.The results table reports offensive-class F1, precision, recall, and weighted F1.
  • Interface design: Figure 2 shows the user interface used for the single-turn adversarial collection.The paper directs readers to the example chat interface image.
Loading 1908.06083v1…