PaperlayerView exact revision on arXiv

Source-linked AI summary

Why Should Adversarial Perturbations be Imperceptible? Rethink the Research Paradigm in Adversarial NLP

Yangyi Chen, Hongcheng Gao, Ganqu Cui, Fanchao Qi, Longtao Huang, Zhiyuan Liu, Maosong Sun

arXiv:2210.10683v1cs.CLcs.CRcs.LG

TL;DR

The paper argues that mixing adversarial samples’ different roles obscures security-oriented problem definitions and goals. It builds Advbench, reframes attacks around real-world attackers’ goals, and proposes a heuristic method; experiments report practical advantages and difficulty for the tested defense.

  • Problem

    Security-oriented adversarial NLP lacks clear problem definitions because prior work mixes the distinct roles of adversarial samples.

    Source: §1 · p. 1

  • Method

    The paper releases Advbench, reformalizes textual attacks around real-world attackers’ goals, and proposes a simple heuristic-rule attack method.

    Source: §1 · p. 2

  • Results

    Experiments on Advbench report the proposed method’s superiority in attack performance, efficiency, and validity, while the tested SOTA defense cannot handle it.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that the SoadNLP research paradigm may need adjustment to better address real-world adversarial challenges.

    Source: §7 · p. 9

    Experiments use BERT-base as the only victim model; broader tests across model architectures and training paradigms are left for future work.

    Source: Limitation · p. 9

Abstract

from arXiv · show

Textual adversarial samples play important roles in multiple subfields of NLP research, including security, evaluation, explainability, and data augmentation. However, most work mixes all these roles, obscuring the problem definitions and research goals of the security role that aims to reveal the practical concerns of NLP models. In this paper, we rethink the research paradigm of textual adversarial samples in security scenarios. We discuss the deficiencies in previous work and propose our suggestions that the research on the Security-oriented adversarial NLP (SoadNLP) should: (1) evaluate their methods on security tasks to demonstrate the real-world concerns; (2) consider real-world attackers' goals, instead of developing impractical methods. To this end, we first collect, process, and release a security datasets collection Advbench. Then, we reformalize the task and adjust the emphasis on different goals in SoadNLP. Next, we propose a simple method based on heuristic rules that can easily fulfill the actual adversarial goals to simulate real-world attack methods. We conduct experiments on both the attack and the defense sides on Advbench. Experimental results show that our method has higher practical value, indicating that the research paradigm in SoadNLP may start from our new benchmark. All the code and data of Advbench can be obtained at \url{https://github.com/thunlp/Advbench}.

1 Introduction

The introduction argues that mixing the distinct roles of adversarial samples obscures security-oriented research goals and produces definitions misaligned with real-world cases. It proposes Advbench, a reframed attacker-goal task, and a heuristic attack method, then reports experiments supporting the method’s practical value.

1 Introduction

  • Mixing adversarial samples’ distinct roles yields inconsistent problem definitions and research goals for real-world security scenarios.The paper argues that different roles require different priorities and experimental settings.Source: §1 · p. 1
  • Prior work often evaluates on non-security tasks and overlooks real attackers’ goals, sometimes imposing imperceptibility constraints or inefficient methods.The introduction gives sentiment analysis and natural language inference as examples of irrelevant evaluation tasks, and cites high query counts and long running times as practical concerns.Source: §1 · pp. 1–2
  • Experiments on Advbench report the proposed method’s superiority in attack performance, efficiency, and validity, while the tested SOTA defense cannot handle it.The authors suggest the SoadNLP research paradigm may start from their benchmark.Source: §1 · p. 2
  • Advbench collects, processes, and releases security datasets to address the shortage of security-relevant evaluation benchmarks.The collection is assembled from datasets corresponding to five security tasks.Source: §1 · p. 2
  • The paper reformalizes textual adversarial attacks around real-world attackers’ goals and adjusts the emphasis placed on different adversarial goals.The stated attacker goals include bypassing detection while conveying offensive meaning.Source: §1 · p. 2
  • The authors propose a simple heuristic-rule attack method intended to fulfill actual attackers’ goals and simulate real-world attacks.The method is presented as facilitating future research on both attack and defense.Source: §1 · p. 2

2 Advbench Construction

Advbench addresses the shortage of security-relevant adversarial NLP evaluations with a collection of processed datasets spanning five security tasks.

2.1 Motivation

  • Most surveyed adversarial NLP evaluations use non-security tasks, motivating a security-task dataset collection.Source: §2.1 · p. 2

2.2 Tasks

  • Advbench covers five security tasks and releases processed balanced and unbalanced datasets after removing duplicate, missing, and unusual values.Source: §2.2 · p. 2

2.2.1 Misinformation

  • LUN labels satire, hoaxes, and propaganda as untrusted against trusted Gigaword articles; SATNews comprises satirical and verified news.Source: §2.2.1 · pp. 2–3

2.2.2 Disinformation

  • Amazon-LB uses Amazon Luxury Beauty review verification labels; CGFake contrasts original human-created and computer-generated fake reviews.Source: §2.2.2 · p. 3

2.2.3 Toxic

  • HSOL combines hate and offensive tweets as “hate” versus “non-hate,” while Jigsaw2018 contains toxic and non-toxic Wikipedia comments.Source: §2.2.3 · p. 3

2.2.4 Spam

  • Enron combines legitimate and spam email collections; SpamAssassin merges easy-ham and hard-ham into ham before equal train/test splitting.Source: §2.2.4 · p. 3

2.2.5 Sensitive Information

  • EDENCE restores Enron parsing-tree samples to text and maps TREC LEGAL labels; FAS uses expert-annotated TREC LEGAL labels.Source: §2.2.5 · pp. 3–4

3 Task Formalization

The section reframes security-oriented textual attacks around practical black-box settings and real attackers’ goals, prioritizing meaning preservation and efficiency over minor perturbations and sample quality. It proposes ROCKET, a heuristic-rule attack intended to simulate real-world strategies.

3.2 Formalization

  • Attack formalization starts from correctly classified text x and perturbs it into x* to fool the classifier.Source: §3.2 · p. 4
  • Practical attacks should use a decision-based setting, since real security tasks may expose only victim-model decisions, not confidence scores or gradients.Source: §3.2 · p. 4
  • Because attackers need to bypass detection while conveying malicious meaning, the paper treats imperceptibility constraints and sample quality as secondary.Source: §3.2 · p. 4
  • The paper argues attacks should reduce both runtime and victim-model query counts, since existing substitution and paraphrase methods may require hundreds of queries.Source: §3.2 · p. 4
  • Rather than assume expert attackers using external knowledge or NLP models, the paper also targets simple heuristic strategies used by ordinary attackers.Source: §3.2 · pp. 4–5
  • The proposed goal priorities emphasize adversarial-meaning preservation and computation efficiency, while down-weighting minor perturbations and sample quality when goals trade off.Source: §3.2 · p. 5, Table 3 · p. 5

3.3 Our Method

  • ROCKET combines heuristic perturbation rules with black-box search; its rules are selected for commonness, efficiency, ease of implementation, and independence from external knowledge.Source: §3.3 · p. 5, Table 4 · p. 5
  • The search first adds task-specific distracting words, then iteratively samples words and perturbation actions, querying the victim until success or a threshold.Source: §3.3 · p. 5

4 Experiments

The experiments evaluate attacks on Advbench under decision-only access and compare attack success, efficiency, validity, and secondary perturbation-quality metrics. ROCKET performs strongly overall, while the defense evaluation finds that DeepWordBug and ROCKET outperform word-level attacks.

4.1 Experimental Settings

  • The experiments use BERT-base and prioritize attack success rate, query efficiency, and validity over perturbation degree and quality.Validity is human-rated; query efficiency is average victim-model queries, perturbation degree uses Levenstein distance, and quality uses perplexity and grammar-error changes.Source: §4.1 · p. 6

4.2 Baseline Methods

  • The comparison implements five representative attacks—TextFooler, PWWS, BERT-Attack, SememePSO, and DeepWordBug—in a black-box setting exposing only model decisions.The implementations use OpenAttack.Source: §4.2 · p. 6

4.3 Experimental Results

  • Prior attacks struggle on Misinformation and Spam in the decision-based setting, and often require hundreds of victim-model queries to craft a successful sample.The authors attribute the difficulty to the setting and distinct security-task features; they also warn that attack performance is overestimated when validity is considered.Source: §4.3 · p. 6
  • ROCKET shows overall superiority in attack performance and efficiency on Advbench, including stronger performance under query limits on Amazon-LB, HSOL, and EDENCE.The authors describe this restricted-query result as more consistent with real-world situations.Source: §4.3 · p. 7
  • ROCKET achieves overall great validity versus baselines, while character-level attacks preserve meaning best and BERT-Attack seriously damages it.Validity was assessed by human evaluation; Table 6 gives the score upper bound as 2.Source: Table 6 · p. 6, §4.3 · p. 7
  • DeepWordBug performs well on perturbation degree and grammaticality, whereas ROCKET's first-priority strength comes with more disruptive perturbations and grammar errors.ROCKET nevertheless receives more fluent perplexity scores, which the authors suspect may reflect informal text in language-model pretraining data.Source: §4.3 · p. 7

4.4 Evaluation on the Defense Side

  • On the defense side, DeepWordBug and ROCKET consistently outperform word-level attacks, indicating that defenses still need improvement against real-world harmfulness.The defense-side experiment details are in Appendix E.Source: §4.4 · p. 7

5 Related Work

Related work groups textual attacks by perturbation granularity and describes several defense strategies. The section also places Advbench within the broader scope of security NLP.

5.1 Adversarial Attack

  • Character-level attacks swap, delete, or insert characters and are described as common in real-world attacks because they need little external knowledge and are easy to implement.Word-level methods search for substitutions and adversarial samples, often relying on external knowledge bases and being inefficient; sentence-level methods use paraphrasing models.Source: §5.1 · p. 7
  • Other attack lines include multigranularity and token-level methods, as well as universal adversarial triggers.Source: §5.1 · p. 7

5.2 Adversarial Defense

  • The section groups defenses into training-data augmentation, adversarial training, preprocessing, robust representation learning, and certified robustness.Source: §5.2 · pp. 7–8

5.3 Security NLP

  • Security NLP also covers privacy attacks, backdoors, data poisoning, and outlier detection; Advbench may support some research in these areas.Source: §5.3 · p. 8

6 Discussion

The discussion distinguishes security-focused attacks from adversarial samples used for other purposes and argues for clearer goals and settings. It also identifies a gap between NLP and security-community research emphases.

Research on Adversarial Attack

  • Synonym-substitution attacks may be unrealistic for real-world attackers yet remain useful for robustness evaluation, explaining model behavior, and adversarial training.Source: §6 · p. 8
  • For security concerns, attackers may access only model decisions; for other purposes, methods can instead be granted model parameters, making confidence-only access assumptions less significant there.The authors leave refinement of the problem definition to future research and discussion.Source: §6 · p. 8

Research on Adversarial Defense

  • The authors distinguish defense for out-of-distribution robustness from resistance to malicious attacks, and recommend studying these functions separately.They suggest security-side defenses consider attacks used by real-world attackers, such as ROCKET.Source: §6 · p. 8

Research on Security NLP

  • The pilot survey finds a gap between NLP research emphasizing method novelty and security-community work centered on actual scenarios; the authors recommend stating goals and using reasonable problem definitions.Source: §6 · p. 8

7 Conclusion

The conclusion presents Advbench, a reformalized security-attack task, and a simple real-world-oriented attack method, with experiments on attack and defense. It argues that the SoadNLP paradigm may need adjustment and leaves other roles of adversarial samples for future work.

7 Conclusion

  • The paper combines Advbench, a reconsidered task definition, and a simple method aimed at real-world attackers' goals, then evaluates on both attack and defense sides.Source: §7 · p. 8
  • Experiments show the method's superiority on first-priority adversarial goals, and the overall results suggest the SoadNLP research paradigm may need adjustment for real-world challenges.Source: §7 · p. 9
  • Future work will consider other roles of textual adversarial samples to make the overall account more complete.Source: §7 · p. 9

Ethical Consideration

  • The authors present the benchmark and attack method as a baseline to facilitate future attack- and defense-side research on real-world security issues.Source: Ethical Consideration · p. 9
  • The paper frames its refinements as making security-oriented problem definitions more standardized and attack situations better simulated.Source: Ethical Consideration · p. 9

Limitation

The experiments use BERT-base as their sole victim model, while the paper also surveys the tasks and datasets used in prior adversarial-attack evaluations.

Limitation

  • Experiments use BERT-base as the sole victim model; testing other architectures and training paradigms is left for future work.The authors cite a benchmark of up to ten datasets and limited computing resources as reasons for this scope.Source: Limitation · p. 9

A Survey on Previous Work

B Task Description

The section describes five security detection tasks and defines adversarial validity in terms of preserving task-specific harmful meaning. It also describes real-world perturbation strategies and reports that Figure 2 concerns attack success under query limits.

B Task Description

B.1 Misinformation

  • Misinformation detection targets deceptive or unverified information, including rumors, misreports, and satirical news.Source: §B.1 · p. 13

B.2 Disinformation

  • Disinformation detection targets deliberate fabrication, including comments that reverse facts or generated nonexistent information.Source: §B.2 · p. 13

B.3 Toxic

  • Toxic detection targets abusive online content, including cyberbullying, cyberterrorism, sexism, racism, and hate speech.Source: §B.3 · p. 13

B.4 Spam

  • Spam classification targets scams, harassment, advertising, and promotions in email, SMS, and chat messages.Source: §B.4 · p. 13

B.5 Sensitive Information

  • Sensitive-information detection targets company, public-institution, and individual information that could be exposed in text.Examples include intellectual property, product progress, trading and strategic information, and private information.Source: §B.5 · pp. 13–14

C Definition of Validity

  • Validity means preserving the task-specific adversarial meaning, even when perturbations affect unrelated content.Advbench examples include exaggerated or satirical content, fabricated comments, abusive language, spam, and sensitive information embedded in common comments.Source: C Definition of Validity · p. 14

D Real-world Adversarial Attack

  • Real-world cases use character substitutions, deletions, insertions, or distracting words that people can understand but detection systems may misclassify.The authors employ these strategies in their simple attack method.Source: D Real-world Adversarial Attack · p. 14

E.2 Secondary Priority Metrics

The section identifies secondary metrics for perturbation and grammaticality and describes how the defense-side evaluation measures detection and restoration.

E.2 Secondary Priority Metrics

  • Table 9 reports secondary-priority metrics concerning perturbation degree and grammaticality.Source: §E.2 · p. 14

E.3 Evaluation on the Defense Side

  • The defense evaluation uses a word-substitution detector based on frequency differences, with word frequencies from training data and tuning on validation data.At inference, the detector identifies and restores adversarial samples; results are reported in Table 10.Source: §E.3 · p. 14
  • For N successful adversarial samples, detection and restoration rates are calculated from the numbers detected and successfully restored, respectively.Source: §E.3 · p. 14

F Experimental Details

The experimental details specify input truncation, task-specific heuristic insertions, and a human evaluation of adversarial-meaning validity.

F Experimental Details

  • Sentences are truncated to 480 words, and attack hyperparameters are set empirically based on attack performance and preservation of adversarial meaning.Source: F Experimental Details · pp. 14, 16
  • Table 11 lists ROCKET hyperparameters by task, while Table 9 covers secondary metrics and Table 10 reports defense-side results.Source: Table 11 · p. 15, Table 9 · p. 15, Table 10 · p. 15
  • Figure 1 presents real-world adversarial cases with modified content highlighted in red, and Figure 2 covers success rates under query limits.Source: Figure 1 · p. 16, Figure 2 · p. 16
  • ROCKET inserts task-specific distracting text, including “Reuters,” “up,” “peace,” “>,” or “any.”The choices reflect the authors’ stated intuitions about text commonly appearing in each task and are intended not to affect the original content’s nature or validity.Source: F Experimental Details · p. 16

G Human Evaluation Details

  • The human evaluation samples 100 original–adversarial pairs per attack method for each of two tasks and uses three annotators to score validity from 0 to 2.The tasks are disinformation and toxic detection; voting produces sample labels, and scores are averaged across the 100 samples for each attack method.Source: G Human Evaluation Details · p. 16

Related papers

Loading 2210.10683v1…