Source-linked AI summary
Deceiving Google's Perspective API Built for Detecting Toxic Comments
Hossein Hosseini, Sreeram Kannan, Baosen Zhang, Radha Poovendran
TL;DR
Online harassment motivates scalable automated toxicity detection, but Perspective may be vulnerable to adversarial text changes. The paper demonstrates attacks that preserve abusive meaning while lowering toxicity scores, alongside false alarms and possible poisoning exposure.
Problem
Manual and crowd-based moderation are inefficient and not scalable, motivating reliable automated detection of abusive or toxic content.
Method
The paper tests adversarial perturbations of toxic words in Perspective’s demonstration phrases, including punctuation insertion and misspellings.
Results
The attack consistently reduces toxicity scores to benign-phrase levels, while the system also produces false alarms and shows possible poisoning vulnerability.
Takeaways & Limitations
Adversarial examples seriously undermine the usability of toxic-comment detectors, motivating countermeasures.
Abstract
from arXiv · showhide
Social media platforms provide an environment where people can freely engage in discussions. Unfortunately, they also enable several problems, such as online harassment. Recently, Google and Jigsaw started a project called Perspective, which uses machine learning to automatically detect toxic language. A demonstration website has been also launched, which allows anyone to type a phrase in the interface and instantaneously see the toxicity score [1]. In this paper, we propose an attack on the Perspective toxic detection system based on the adversarial examples. We show that an adversary can subtly modify a highly toxic phrase in a way that the system assigns significantly lower toxicity score to it. We apply the attack on the sample phrases provided in the Perspective website and show that we can consistently reduce the toxicity scores to the level of the non-toxic phrases. The existence of such adversarial examples is very harmful for toxic detection systems and seriously undermines their usability.
I. INTRODUCTION
Online platforms facilitate discussion but also enable harassment, while manual and crowd-based moderation do not scale. The paper examines adversarial examples that can substantially reduce Perspective’s toxicity scores for abusive text.
- Social platforms support information sharing and discussion but also expose users to fake news and online harassment.
- Crowdsourcing, disabling comments, and manual moderation are inefficient and difficult to scale for real-time abuse detection.
- Google and Jigsaw’s Perspective uses machine learning to automatically detect online insults, harassment, and abusive speech.
- Perspective is available through an API and a demonstration website that returns toxicity scores for entered phrases.
- The paper shows that misspellings and punctuation inserted into abusive words can deceive Perspective and lower toxicity scores while preserving abusive content.
A. Brief Description of Google’s Perspective API
Perspective is a Google- and Jigsaw-associated API that scores input-text toxicity for online discussion applications. Its model was trained from publisher comments rated by human panels, and its scores can be integrated in real time.
- Perspective is an API created by Jigsaw and Google’s Counter Abuse Technology team within Conversation-AI.
- The API uses machine learning models to score input-text toxicity, defined as a rude, disrespectful, or unreasonable comment likely to make someone leave a discussion.
- Google and Jigsaw trained the model using millions of publisher comments rated by panels of ten people from “very toxic” to “very healthy.”
- Publishers can receive scores in real time and use them to display toxicity ratings or flag abusive comments for human moderators.
B. Adversarial Examples for Learning Systems
Adversarial examples are slightly perturbed inputs that alter machine-learning predictions, often without human notice. In toxic-text detection, the goal is to preserve abusive meaning while reducing the classifier’s toxicity score.
- Machine-learning models optimized for clean, benign data remain vulnerable to adversarial attacks.
- Adversarial examples change a model’s prediction score through slight, often human-unnoticeable input perturbations.
- Prior work applied adversarial examples to image, music, and malware classification, while this paper targets a real-world text classifier.
- For toxicity scoring, an adversarial example preserves highly abusive language while receiving a significantly lower model toxicity score.
III. THE PROPOSED ATTACKS
The attack modifies toxic phrases with small textual errors and evaluates the effects on Perspective’s sample phrases. These perturbations can lower toxicity scores, transfer across phrases, and expose additional false-alarm and poisoning risks.
- III. THE PROPOSED ATTACKS: The attack repeatedly perturbs toxic words in Perspective demonstration phrases until the model produces a very low toxicity score.
- III. THE PROPOSED ATTACKS: Table I compares original phrases and toxicity scores with adversarially modified phrases and their corresponding scores.
- III. THE PROPOSED ATTACKS: Inserted dots, spaces, or misspellings—including duplicated or swapped letters—consistently reduce toxic-word scores to benign-phrase levels.
- III. THE PROPOSED ATTACKS: Adversarial perturbations transfer across phrases, allowing a dictionary of effective word modifications to simplify future attacks.
- III. THE PROPOSED ATTACKS: Perspective assigns high toxicity scores to apparently benign, negated phrases and fails to capture their intended semantics.
- III. THE PROPOSED ATTACKS: 34% toxicity score is assigned to most misspelled and random words, although the system is somewhat robust to randomly modified toxic words.
- III. THE PROPOSED ATTACKS: User feedback can expose the interface to poisoning attacks if labels are modified so the model assigns low toxicity to selected phrases.
IV. OPEN PROBLEMS IN DEFENSE METHODS
Perspective’s adversarial-example vulnerability motivates defenses, but each reviewed approach has practical limitations. Proposed measures include adversarial training, spell checking, and temporarily blocking repeatedly suspicious users.
- The system’s vulnerability to adversarial examples makes improving toxic-detection robustness a challenging problem.The paper describes scoring semantic toxicity as clearly difficult after demonstrating Perspective’s vulnerability.
- Adversarial Training: Adversarial training adds modified toxic-word variants to training data so the model retains the original label.The approach may improve robustness, but covering all variants of every word is impractical.
- Spell checking: Spell checking before toxicity detection can identify many adversarial examples but may increase false alarms.
- Blocking suspicious users for a period of time: Temporarily blocking users after repeated threshold failures can make repeated evasion attempts less frequent.The adversary must try different error patterns to evade detection, allowing repeated failures to trigger a temporary block.
V. CONCLUSION
The paper shows that Google’s Perspective API can be deceived by slight perturbations that preserve abusive phrases’ intended meaning, and that it also assigns high toxicity to benign phrases. The authors provide detailed examples and identify countermeasure development as future work.
- Slight perturbations can make abusive phrases receive very low toxicity scores while preserving their intended meaning.
- Perspective also has a high false alarm rate when assigning high toxicity to benign phrases.
- The paper provides detailed examples for the studied cases and leaves countermeasure development for future work.The disclaimer states that phrases in Tables I and II come from Perspective website examples and do not represent the authors’ or sponsors’ views.