Source-linked AI summary
Breaking BERT: Understanding its Vulnerabilities for Named Entity Recognition through Adversarial Attack
Anne Dirkson, Suzan Verberne, Wessel Kraaij
TL;DR
BERT models are widely used for NER, but their robustness to variation in entities and entity contexts remains insufficiently understood. The paper develops adaptive adversarial attacks for NER and finds substantial vulnerability, especially to local context changes and rare or emergent entities. It also identifies grammatical errors in some adversarial examples as a limitation of the attack construction.
Problem
The paper asks how vulnerable general and domain-specific BERT models are to adversarial variation in NER, whether domain-specific training affects vulnerability, and which variations are most damaging.
Source: §1 · p. 1Method
The authors adapt adversarial attacks to NER by replacing entity-context words with synonyms or entities with alternatives of the same type, targeting changes that fool the model.
Source: §1 · pp. 1–2, §3.1.2 · p. 3Results
BERT models can be fooled by single context-word substitutions and are even more vulnerable when entities are replaced by less frequent entities of the same type.
Source: §7 · p. 8Takeaways & Limitations
Rare or emergent entities remain a problem for both generic and domain-specific NER models, motivating research into zero- or few-shot learning and entity masking.
Source: §7 · p. 8Some adversarial examples contain grammatical errors, including part-of-speech changes and singular-plural inconsistencies.
Source: §6 · p. 8
Abstract
from arXiv · showhide
Both generic and domain-specific BERT models are widely used for natural language processing (NLP) tasks. In this paper we investigate the vulnerability of BERT models to variation in input data for Named Entity Recognition (NER) through adversarial attack. Experimental results show that BERT models are vulnerable to variation in the entity context with 20.2 to 45.0% of entities predicted completely wrong and another 29.3 to 53.3% of entities predicted wrong partially. BERT models seem most vulnerable to changes in the local context of entities and often a single change is sufficient to fool the model. The domain-specific BERT model trained from scratch (SciBERT) is more vulnerable than the original BERT model or the domain-specific model that retains the BERT vocabulary (BioBERT). We also find that BERT models are particularly vulnerable to emergent entities. Our results chart the vulnerabilities of BERT models for NER and emphasize the importance of further research into uncovering and reducing these weaknesses.
1 Introduction
This paper investigates how adversarial variation in entities and their contexts exposes vulnerabilities in general and domain-specific BERT models for NER.
1 Introduction
- The attacks replace context words with synonyms or entities with alternatives of the same type, selecting changes that fool the model or reduce its correct-output score.Context substitutions exclude entity words, while entity replacements preserve the entity type.Source: §1 · pp. 1–2
- The study evaluates BERT vulnerability to adversarial attacks on general and domain-specific NER.It asks how vulnerable the models are, whether domain-specific training changes vulnerability, and which variations are most damaging.Source: §1 · p. 1
- NER attacks are challenging because labels are predicted per word, sentences may contain multiple entities, and entities may span multiple words.The method therefore constrains context changes to non-entity words and entity substitutions to entities of the same type.Source: §1 · pp. 1–2
- The experiments assume a black-box setting, requiring no knowledge of the data, parameters, or model architecture.The authors state that the approach can also extend to other neural architectures and languages with an appropriate synonym-selection language model.Source: §1 · p. 2
- The authors adapt adversarial attacks to sequence labeling and evaluate both general and domain-specific BERT models for NER.They also release code for follow-up research.Source: §1 · p. 2
2 Related Work
Prior adversarial-attack work largely targets classification or entailment, while NER has received limited attention; this paper extends the setting with adaptive attacks and broader model coverage.
2 Related Work
- Earlier token-level black-box attacks mainly target classification and textual entailment, commonly using synonym substitutions and semantic-similarity constraints.Word-embedding synonym selection can confuse antonyms with synonyms, motivating additional constraints.Source: §2 · p. 2
- The paper provides code and links to benchmark data and annotation guidelines for follow-up research.The code is released under a BSD-3 Clause license.Source: §2 · p. 2
- Before this work, Araujo et al. (2020) provided the only study attacking BERT models for NER, using spelling errors and entity synonyms on biomedical models.They reported substantially worse biomedical NER performance under these perturbations.Source: §2 · p. 2
- This paper selects perturbations by word importance rather than random changes, targeting severe stress conditions.Importance is based on the output difference caused by removing a word.Source: §2 · p. 2
- Unlike prior work, the study replaces entities with same-type entities and alters entity context, while testing original and biomedical BERT models across general and biomedical NER.These design choices broaden both the attack types and model domains examined.Source: §2 · p. 2
3 Methods
The method constructs adaptive NER adversarial examples through context synonym replacement or same-type entity replacement, measuring complete and partial label failures.
3 Methods
- The paper uses two attacks: synonym replacement in entity context and replacement of an entity with another entity of the same type.Each attack is designed to fool NER models while preserving relevant sentence constraints.Source: §3 · p. 2, §3.1.3 · p. 4
3.1 Aim of the attacks
- An attack succeeds when a correctly labelled entity receives a changed label, while partial success means at least half of an entity is mislabelled.A change from an I-tag to a B-tag of the same entity type is excluded as a full success.Source: §3.1 · p. 3, §3.1.1 · p. 3
3.1.1 Metrics for evaluation of the attacks
- Context-attack difficulty is measured by the average percentage of out-of-mention words perturbed before success.The method also reports complete and partial success rates.Source: §3.1.1 · p. 3
3.1.2 Entity Context Attack
- Context words are ranked by how much deleting them changes prediction scores for the entity’s correct labels.For multiword entities, importance is summed across tokens, while stop words and other entities are excluded.Source: §3.1.2 · p. 3
- Candidate synonyms must exceed a similarity threshold and preserve part of speech and sentence-level semantic similarity.The final synonym is chosen to change labels when possible, otherwise to reduce correct-label scores most; the method iterates until success or exhaustion.Source: §3.1.2 · pp. 3–4
3.1.3 Entity Attack
- The entity attack samples same-type replacements, filters semantically dissimilar candidates, and selects a successful replacement with highest sentence similarity.If no candidate changes the predicted label, the attack is unsuccessful.Source: §3.1.3 · p. 4
4 Experiments
Experiments evaluate attacks on general and biomedical NER datasets using general BERT plus BioBERT and SciBERT, with automatic and human assessments of adversarial examples.
4.1 Data
- The evaluation covers CoNLL-2003 and W-NUT 2017 for general NER, plus BC5CDR and NCBI disease for biomedical NER.W-NUT 2017 focuses on unusual, previously unseen entities in online discussions.Source: §4.1 · p. 4
4.2 Target models
- The target models include three base-cased BERT runs and, for biomedical datasets, BioBERT and SciBERT trained with three seeds.Training uses Hugging Face implementations, learning rate 5 × 10^-5, and three or four epochs.Source: §4.2 · pp. 4–5
4.3 Evaluation of the adversarial attacks
- Automatic evaluation samples 500 eligible test sentences per dataset, except NCBI disease, which contributes 487 eligible sentences.Eligible sentences contain at least one entity and one verb.Source: §4.3 · p. 5
- Attacks are generated only for entities initially predicted correctly, and models are evaluated on how often those entities become mislabelled.The semantic similarity threshold is set to ϵ = 0.8.Source: §4.3 · p. 5
- Human evaluation rates grammaticality on a five-point scale for original and adversarial CoNLL-2003 and BC5CDR sentences.Four linguistically trained PhD student annotators score 100 original and 100 adversarial sentences from each attack type per dataset.Source: §4.3 · p. 5
5 Results
BERT models are highly vulnerable to both context and entity adversarial attacks in NER, with local synonym changes and entity replacements often sufficient to induce errors. Importance ranking substantially improves context attacks, while human evaluation shows context attacks reduce grammaticality but generally remain acceptable.
5.1 Entity Context Attack
- 87.3% of CoNLL entities and 93.8% of W-NUT entities were partially or fully misclassified after context attacks.More than 75% of successful context attacks required only one word change.Source: §5.1 · pp. 5–6
- SciBERT was more vulnerable than BERT, with complete-fooling increases of 6.2 percentage points and partial-fooling increases of 9.7 and 7.4 percentage points.BioBERT and SciBERT were also frequently fooled by a single synonym replacement, at 65–75% and 68–76% of cases respectively.Source: §5.1 · p. 6
- Successful attacks were associated with longer sentences, more replaceable words, shorter entities, and rare synonyms replacing common words.Entity attacks also targeted entities less frequent in training and development data, except on CoNLL, and models showed sensitivity to capitalization changes.Source: §5.1 · p. 6, §5.3 · p. 7
- Context attacks were strongest near entities, with BERT models most vulnerable to changes one or two words to the left or right.Long-distance context sometimes mattered, especially in list-like sentences, while SciBERT was more vulnerable to local-context changes than BioBERT or original BERT.Source: §5.1 · p. 6, Figure 1 · p. 6
5.2 Evaluating the necessity of importance ranking
- Removing word-importance ranking reduced average full success from 37.3% to 9.5% and partial success from 52.8% to 20.1%.The number of perturbed words also fell by 6.9 percentage points, indicating that ranking is central to attack effectiveness.Source: §5.2 · p. 6, Table 5 · p. 6
5.3 Entity Attack
- 97.5% of initially correct CoNLL entities and 89.2% of W-NUT entities were fooled by entity attacks, with domain-specific success rates above 99%.The attacks replaced entities with other entities of the same type while preserving the surrounding context.Source: §5.3 · p. 7, Table 6 · p. 7, Table 7 · p. 7
5.4 Results of human evaluation
- Entity attacks did not significantly change grammaticality, whereas context attacks did, although adversarial sentences remained above 3 on average and lost less than one point.Because inter-annotator agreement was limited, grammaticality was reported separately for each annotator.Source: §5.4 · p. 7, Table 8 · p. 8
6 Discussion and limitations
The discussion identifies semantic, lexical, grammatical, and interpretive limits of the adversarial examples and cautions against overgeneralizing successful attacks.
6 Discussion and limitations
- Synonym-based adversarial examples are susceptible to word-sense ambiguity, including replacements that are unrelated to the intended meaning.For example, “surfed” may be replaced by “paddled” rather than a contextually appropriate synonym.Source: §6 · p. 8
- Some synonym replacements introduce foreign words or cross-lingual mismatches because the embedding space contains multilingual associations.Examples include replacing “number” with “nombre” and interpreting French “vie” as “existence.”Source: §6 · p. 8
- Adversarial sentences can contain grammatical errors, especially part-of-speech mismatches and singular–plural inconsistencies that the POS filter misses.The authors suggest removing non-English vocabulary and improving POS-tag selection as mitigation directions.Source: §6 · p. 8
- Entity replacements can violate semantic consistency for broad types such as Location, whereas replacements for Disease, Chemical, and Person are often semantically close.The paper contrasts replacing “U.S.” with “Tavildara” against medically or personally related substitutions.Source: §6 · p. 8
- Successful attacks do not always isolate a general weakness because attention is context-dependent and multiple changed words may interact.A successful perturbation may therefore be specific to its context or to the combination of changes.Source: §6 · p. 8
7 Conclusions
The paper concludes that black-box token-level attacks expose substantial vulnerabilities in BERT-based NER models. It highlights rare entities, unseen context words, and SciBERT’s robustness–domain-specificity trade-off as priorities for future work.
7 Conclusions
- BERT models can be fooled by replacing a single context word with a synonym and are even more vulnerable when entities are replaced by less frequent entities of the same type.These findings come from black-box attacks on NER tasks.Source: §7 · p. 8
- Rare or emergent entities remain problematic for both generic and domain-specific NER models, motivating research on zero- or few-shot learning and entity masking during fine-tuning.The paper presents these as possible directions rather than established solutions.Source: §7 · p. 8
- BERT models also appear vulnerable to rarely seen or unseen words in entity contexts, suggesting adversarial training examples with infrequent words or alternative pre-training schemes as research avenues.Curriculum learning is offered as one possible alternative pre-training direction.Source: §7 · p. 8
- SciBERT is more vulnerable to entity-context changes than BioBERT or BERT, possibly because its biomedical vocabulary increases sensitivity to substitutions with common English terms.The authors frame this as a potential trade-off between robustness and domain specificity.Source: §7 · p. 8
- The study is intended as a step toward understanding token-level vulnerabilities in self-attentive NER models and encouraging work to mitigate them.The authors explicitly invite further research building on their analysis.Source: §7 · p. 8