NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentation

Problem

• Task-sensitive augmentation is needed because linguistic changes can benefit one NLP task while being less relevant or effective for another.For example, changing “happy” to “very happy” is described as more relevant for sentiment analysis than summarization.Source: §1 · p. 2
• Long-tailed language distributions make rare phenomena difficult to represent and evaluate, motivating researcher contributions that encode diverse domain and cultural knowledge.The paper argues that a collection built from the “wisdom-of-researchers” can identify and operationalize a wider range of linguistic variation.Source: §1 · p. 2

Framework motivation

• NL-Augmenter is a Python-based participatory framework that collects task-specific and task-independent augmentations for diverse natural-language tasks.Transformations are tied to data formats and task types, while the framework broadens transformations beyond strict equivalence to include quasi-paraphrases, noise, sociolinguistic variation, style, and syntax changes.Source: §1 · pp. 2–3
• NL-Augmenter provides filters that create input subpopulations based on features such as input complexity, input size, and toxicity, enabling dataset splitting and evaluation on subsets with specific linguistic properties.Source: §1 · p. 3

Evaluation

• The authors apply selected transformations and filters to datasets to examine how different perturbation types affect natural-language models.The paper presents the robustness analysis as an empirical demonstration of the framework’s efficacy.Source: §1 · pp. 2–3

2.2 Wisdom-of-Researchers

• NL-Augmenter extends participatory benchmark practice by collecting executable transformations from researchers through open collaboration.The approach follows projects such as Masakhane, GEM, BIG-Bench, DynaBench, and SyntaxGym while targeting broader task and model coverage.Source: §2.1 · pp. 3–4, §2.2 · p. 4

2.1 Evolving Participatory Benchmarks

• Unlike tools focused mainly on English or narrow syntactic evaluation, NL-Augmenter supports transformations and filters across many languages, tasks, and model types.Its contributions need not be syntactically or linguistically motivated, and each contribution can specify supported languages.Source: §2.3 · p. 4, §2.1 · p. 4

2.3 Robustness Evaluation Tools

• NL-Augmenter differs from manually constructed contrast sets by reducing replication costs through reusable programmatic perturbations.Gardner et al. (2020) are described as producing higher-quality manual perturbations, but with greater task-specific annotation and replication costs.Source: §2.3 · p. 4
• Compared with existing robustness libraries, NL-Augmenter emphasizes an open-source collaborative transformation collection with a broad taxonomy and more than 117 transformations.Robustness Gym depends on existing transformation libraries, while Polyjuice offers less fine-grained control over challenge types; NL-Augmenter also plans worst-case evaluation beyond its current average-case use.Source: §2.3 · p. 4

3 NL-Augmenter

• NL-Augmenter provides a crowd-sourced suite for applying transformations and filters to training, development, and evaluation data across NLP tasks.Filters support robustness and bias analysis on selected subpopulations, while transformations support data augmentation.Source: §3 · pp. 4–5

3.1 Participatory Workshop on GitHub

• Participants contributed Python implementations through a GitHub workshop with shared interfaces, starter code, evaluation scripts, and review criteria requiring style conformity, JSON test cases, novelty, and specificity.The review process used test cases to clarify each transformation’s capabilities and generation scope.Source: §3.1 · p. 5, Figure 2 · p. 5

3.2 Transformations and filters

• The repository contains 117 transformations and 23 filters, each linked to detailed descriptions and implementations in the appendix and repository.Transformations modify examples, whereas filters return boolean decisions for constructing feature-based dataset subsets.Source: §3.2 · p. 5, Table 2 · p. 7

3.3 Tags for the classification of perturbations

• Tags classify perturbations by general properties, output properties, and processing properties to support navigation and analysis of their performance characteristics.General tags include task, language, and linguistic level; output tags cover similarity and preservation qualities; processing tags cover algorithms, implementation, resources, and complexity.Source: §3.3 · pp. 5, 7

3.4 Tag retrieval and assignment

• Metadata tags are gathered automatically from contributor implementations, mapped into property groups, and supplemented through manual assignment and author review.Required in-code metadata describes supported languages, applicable tasks, and other characteristics.Source: §3.4 · pp. 7–9

4.1 Experiment

• The study perturbed random samples of 20% of validation sets for SST-2, IMDB, QQP, and MNLI, reporting transformation rates and score variation against original data.For paired datasets, only the first question in QQP and the premise in MNLI were perturbed.Source: §4.1 · p. 9, §4.2 · p. 9

4.2 Results and Analysis

• Transformations designed for robustness testing caused mean performance drops of 9% to 13.7%, while 34 augmentation transformations caused similar drops of 4% to 13%.These results motivate drawing on paraphrasing literature to improve robustness testing.Source: §4.2 · p. 9, Table 6 · p. 10
• The task a transformation targets did not necessarily match the task on which it most challenged model robustness.The experiment reports task-type robustness scores using the task-type grouping in Table 7.Source: §4.2 · p. 9, Table 7 · p. 10
• Character-level and morphological transformations produced larger performance drops than lexical or syntactic changes across all four models.For roberta-large-mnli, character-level changes lowered performance by over 31% on average, morphological changes by 28%, and lexical changes by 4.4%.Source: §4.2 · p. 9, Table 8 · p. 10
• The visual_attack_letters transformation reduced accuracy from 94% to 56% for roberta-base-SST-2, from 92% to 69% for bert-base-uncased-QQP, and from 91% to 47% for roberta-large-mnli.The authors connect this vulnerability to possible OCR-like inputs and limited exposure to character variation.Source: §4.2 · p. 9
• Transformations marked as highly meaning preserving produced larger average drops than those marked as possibly meaning changing, based on 22 versus 20 transformations.The authors note that noisy transformation logic may contribute and question whether models rely on surface-form shortcuts.Source: §4.2 · pp. 9, 12, Table 10 · p. 11
• Preserving grammaticality did not correlate with robustness: always-preserved transformations caused average drops of 10.6%, 8.1%, and 4.6% on SST-2, MNLI, and QQP models.The grapheme_to_phoneme transformation produced drops of 13%, 20%, and 13%, respectively.Source: Table 11 · p. 11, §4.2 · p. 12
• Transformations tagged as impairing readability or naturalness generally caused large performance drops across all tasks and models.The largest effects were associated with transformations tagged as “always imparing” the input.Source: Table 12 · p. 11, Table 13 · p. 11, §4.2 · p. 12

• The robustness analysis tested only some perturbations across four datasets, so its broad results remain shallow and require transformation-specific studies.The authors also note that many tags were hand-annotated, and inconsistent annotations may explain surprising findings.Source: §5 · p. 13
• The current evaluation shows perturbations can reveal model weaknesses, but it does not establish that they mitigate those weaknesses.Source: §5 · p. 13

Dilution of Contributions

• The framework’s broad collection creates a credit-dilution risk for individual contributions and may discourage use of related work, despite easier switching among transformations.The authors address this through contributor data cards, co-authorship, citation guidance, and encouragement to try other solutions.Source: §5 · p. 13

Participatory Setup

• NL-Augmenter’s flexible participatory interface supports contributions ranging from protected-attribute filters to neural style-transfer transformations.The low participation barrier is intended to broaden augmentation and evaluation strategies.Source: §5 · p. 13

• NL-Augmenter provides more than 117 transformations and more than 23 documented and tested filters for robustness testing and data augmentation.The authors report that open participation enabled broad coverage across languages, tasks, transformations, and filters.Source: §6 · p. 14
• The authors identify future priorities as expanding augmentation coverage, addressing current shortcomings, and studying robustness effects through larger-scale data augmentation experiments.Source: §6 · p. 14

• Researchers and developers from different NLP niches organized NL-Augmenter, with a steering committee coordinating direction and broader contributors providing implementations and feedback.Source: §7 · p. 14
• Participants’ submissions underwent first-phase review, second-phase cross-review by three peers, and a subsequent meta-review by organizers.Source: §7 · p. 14
• The evaluation engine and robustness analysis were developed and conducted by designated contributors, while separate contributors created the project webpages.Source: §7 · p. 14

A Review criteria for submission evaluation

• Review criteria guided contributors toward a consistent style, JSON test cases, novelty, and specificity when submitting transformations and filters.The criteria were presented to both participants and reviewers.Source: A Review criteria for submission… · p. 21, Figure 3 · p. 22

B Transformations

• Lexical and stylistic transformations include abbreviations, hashtag addition, adjective antonym substitution, British-American spelling conversion, and double-negation perturbations.Source: §B.1 · p. 23, §B.2 · p. 23, §B.3 · p. 23

B.6 Auxiliary Negation Removal

• Negation and keyboard-oriented transformations remove auxiliary negation or introduce character errors through keyboard-neighbor substitutions, including an Indian-language variant.The casing transformation is restricted to cased models or languages.Source: §B.6 · p. 23, §B.10 · pp. 23–24, §B.11 · p. 24

B Transformations

• Several transformations are explicitly scoped by implementation constraints, including low coverage for auxiliary-negation removal, casing dependence, external libraries, and manually specified behavior.The catalog directs readers to implementation requirements and documented limitations.Source: B Transformations · p. 22, §B.6 · p. 23, §B.12 · p. 24

B.8 BackTranslation

• Backtranslation generates paraphrases for English text and for contexts around named entities, with the latter also assigning new entity tags.The named-entity variant is motivated by reported benefits for named entity tagging.Source: §B.8 · p. 23, §B.9 · p. 23

B.13 Change Date Format

• Other cataloged transformations alter dates, person names, paired name references, Chinese synonyms or antonyms, and Chinese characters using pinyin-based substitutions.Source: §B.13 · p. 24, §B.14 · p. 24, §B.15 · p. 24

B.18 Chinese Person Named Entities and Gender Perturbation

• Chinese name perturbation swaps Chinese names and may also swap gender, supporting bias detection and inference of implicit gender information.Source: §B.18 · pp. 24–25

B.19 Chinese (Simplified & Traditional) Perturbation

• Chinese character perturbation converts Simplified and Traditional Chinese, regional variants, Japanese Kanji, and character- or phrase-level forms.Source: §B.19 · p. 25

B.20 City Names Transformation

• City-name transformation replaces populous, well-known Spanish and English cities with less populous, less-known cities to reveal demographic biases in named entity recognition.Source: §B.20 · p. 25

B.21 Close Homophones Swap

• Close-homophone swapping tests robustness to phonetic attacks by replacing randomly selected words with similar-sounding or similar-looking alternatives.The authors caution that the resulting choices may not reflect the natural distribution of intentional or accidental swaps.Source: §B.21 · p. 25

B.22 Color Transformation

• Color transformation randomly replaces mentioned colors with alternatives from the W3C’s 147 extended color keywords.Source: §B.22 · p. 25

B.23 Concatenate Two Random Sentences (Bilingual)

• Bilingual sentence concatenation joins a sentence with a previously occurring sentence, increasing training-data context and length diversity for text tasks.Prior work reported stronger low-resource machine translation models using this method (Nguyen et al., 2021).Source: §B.23 · p. 25

B.24 Concatenate Two Random Sentences (Monolingual)

• The monolingual concatenation transformation is presented as the counterpart to bilingual random-sentence concatenation.Source: §B.24 · p. 25

B.25 Concept2Sentence

• Concept2Sentence extracts keyword concepts and uses a CommonGen-trained BART model to generate a related sentence whose label remains invariant.Its scenic-description style may diverge from the original syntax, and a small keyword subset may not fully describe the source.Source: §B.25 · p. 26

B.29 Country/State Abbreviation

• Country and state abbreviation transformation replaces full names with common abbreviations, but ambiguous abbreviations can lose information when context provides insufficient signals.“MH” may denote County Meath in Ireland or Maharashtra in India.Source: §B.29 · p. 26

B.30 Decontextualisation of the main Event

• Decontextualisation uses semantic role labeling to remove contextual arguments, producing a minimal event description for entailment pairs.Source: §B.30 · p. 27

B.31 Diacritic Removal

• Diacritic removal replaces accented characters with unaccented versions, modeling cases where non-native or inexperienced speakers omit accents.Source: §B.31 · p. 27

B.32 Disability/Differently Abled Transformation

• Disability-related transformation replaces outdated disability references with more respectful, inclusive alternatives that avoid negative connotations.Source: §B.32 · p. 27

B.33 Discourse Marker Substitution

• Discourse-marker substitution replaces a marker with another associated with the same semantic class, using PDTB 2.0 corpus associations with probability at least 0.5.Source: §B.33 · p. 27

B.34 Diverse Paraphrase Generation Using SubModular Optimization and Diverse Beam Search

• Diverse paraphrase generation combines backtranslation models with DiPS, Diverse Beam Search, Beam Search, or random candidate selection to produce multiple paraphrases.DiPS uses submodular optimization to emphasize paraphrase diversity.Source: §B.34 · p. 27

B.35 Dislexia Words Swap

• Dyslexia-word swapping perturbs sentences by replacing words with aberrations likely to occur in dyslexic writing contexts.Source: §B.35 · p. 27

B.36 Emoji Icon Transformation

• Emoji icon transformation converts emojis into equivalent keyboard forms and converts those keyboard forms back into emojis.Source: §B.36 · p. 27

B.37–B.42

• Emojify, inflectional variation, mention replacement, filler insertion, style transfer, and French conjugation substitution modify wording, morphology, entities, discourse markers, style, or verb forms.Inflectional variation preserves polysemous-word sense through POS-constrained reinflection, while mention replacement swaps entities within the same type for NER augmentation.Source: §B.37 · p. 27, §B.38 · p. 28, §B.39 · p. 28

B.43–B.47

• Name, pronoun, gender, and location transformations diversify demographic representation, neutralize gendered language, swap gendered terms, or append geographic knowledge.The name changer samples country-gender pairs from 42,812 names across 141 countries, addressing limited cultural and gender coverage in common augmentations; neopronoun substitution targets trans-erasure.Source: §B.43 · p. 28, §B.44 · p. 29, §B.45 · p. 29

B.48–B.53

• German gender swapping, grapheme-to-phoneme substitution, greetings and farewells, hashtagification, abbreviations, and leet conversion perturb language form for robustness or social-media-style inputs.The German method uses 2,226 common names, while phoneme substitution targets speech systems and leet conversion replaces letters with visually similar symbols.Source: §B.48 · p. 29, §B.49 · p. 29, §B.50 · p. 29

B.54–B.61

• Lexical counterfactuals, longer entity names, lost-in-translation, mixed-language perturbation, transliteration, and MR replacement alter semantics, entity spans, language, script, or meaning representations.NER-specific methods expand location or person mentions while updating labels; MR replacement aligns substituted values with sentence tokens but may produce ungrammatical segments.Source: §B.54 · p. 30, §B.55 · p. 30, §B.56 · p. 30

B.62–B.65

• Multilingual back translation, dictionary code-switching, lexicon perturbation, and causal negation or strengthening generate multilingual data or controlled changes to causal meaning.Back translation paraphrases through a pivot language across 100 languages; lexicon perturbation supports 100 languages and is 160x faster than its model-based counterpart.Source: §B.62 · pp. 30–31, §B.63 · p. 31, §B.64 · p. 31

B.66–B.67

• The transformation checks noun-compound paraphrases from Hendrickx et al. (2013), resulting in low coverage, while some outputs sound odd and may not fit well in context.Source: §B.67 · p. 31

B.68–B.74

• Number conversion, OCR errors, noun definitions, Pig Latin, Pinyin transcription, SRL argument exchange, and punctuation changes provide numerical, visual, cipher, linguistic, semantic, or structural perturbations.OCR perturbation supports English, French, Spanish, and German; SRL argument exchange applies deterministic meaning-preserving rules proportional to identified arguments.Source: §B.68 · p. 31, §B.69 · p. 31, §B.70 · p. 32

B.75–B.77

• ProtAugment diverse paraphrasing and question-question paraphrasing use neural generation to create structurally diverse outputs for paraphrase, QA, question generation, and related tasks.ProtAugment uses BART trained on three paraphrase datasets and Diverse Beam Search; question paraphrasing uses a T5 model fine-tuned on Quora Question Pairs.Source: §B.75 · p. 32, §B.77 · p. 32

B.78 Question in CAPS

• Question in CAPS upper-cases a question-answering context and adds upper-cased versions of acceptable original answers.Source: §B.78 · pp. 32–33

B.79 Random Word Deletion

• Random Word Deletion removes each word with probability p, default 0.25, using whitespace tokenization; it is English-oriented and may change meaning.The transformation is destructive and is related to suggestions by Wei and Zou (2019) and word dropout by Goldberg (2017).Source: §B.79 · p. 33

B.80 Random Upper-Case Transformation

• Random Upper-Case Transformation independently upper-cases characters with default probability 0.1, supports languages with case distinctions, but is ineffective after lowercasing tokenization.It resembles Wei and Zou (2019), with a possible extreme-value-theory improvement from Jalalzai et al. (2020).Source: §B.80 · p. 33

B.81 Double Context QA

• Double Context QA repeats a question-answering context and is intended not to change the result.Source: §B.81 · p. 33

B.82 Replace Abbreviations and Acronyms

• Replace Abbreviations and Acronyms expands or shortens English abbreviations using a manually collected business-communication list, but context-dependent or missing entries can alter meaning.The transformation was first introduced by Regina et al. (2020).Source: §B.82 · p. 33

B.83 Replace Financial Amounts

• Replace Financial Amounts converts monetary values to another currency while changing their amount, formatting, and currency, as introduced by Regina et al. (2020).Source: §B.83 · p. 33

B.84 Replace Numerical Values

• Replace Numerical Values detects English numerical values and substitutes random values with the same cardinality, a transformation first introduced by Mille et al. (2021).Source: §B.84 · p. 33

B.85 Replace Spelling; B.86 Replace nouns with hyponyms or hypernyms

• Replace Spelling replaces common misspellings with erroneous variants using Deorowicz and Ciura (2005)'s corpora, with default probability p=0.2, while noun replacement uses related hyponyms or hypernyms when available.Not every noun has a hypernym or hyponym.Source: §B.85 · p. 33, §B.86 · p. 33

B.87 Sampled Sentence Additions

• Sentence additions sample up to 75 tokens from GPT-XL after a “paraphrase:” prompt appended to the input text.The transformation applies to sentences, paragraphs, and other English text sources.Source: §B.87 · p. 33
• Sampled additions may introduce harmful language, contradict earlier text, or change its meaning.Source: §B.87 · pp. 33–34

B.88 Sentence Reordering

• Sentence reordering randomly shuffles input sentences after partially decontextualizing them through coreference resolution.The perturbation applies to paragraphs, documents, and other text sources.Source: §B.88 · p. 34
• Sentence reordering requires inputs with more than one sentence and may fail when ellipsis or narrative structure prevents preserving discourse context.Source: §B.88 · p. 34

B.89 Emoji Addition for Sentiment Data

• The transformations modify sentiment, token order, surface form, and lexical choice, including emoji labeling, segment shuffling, ciphers, slang replacement, gender swapping, disfluencies, and style transfer.Positive and negative examples receive corresponding emojis, while other transformations alter coherent segments, characters, words, grammatical gender, speech fillers, or writing style.Source: §B.89 · p. 34, §B.90 · p. 34, §B.91 · p. 34

B.96 Subject Object Switch

• Semantic and paraphrasing transformations switch subjects and objects, compress sentences, convert QA questions to declarations, generate syntactically diverse paraphrases, and substitute same-label spans for sequence tagging.SOW enumerates feasible reorderings and REAP generates corresponding paraphrases, but the paraphraser supports only single-sentence inputs; subsequence substitution preserves the tag sequence.Source: §B.96 · p. 35, §B.97 · p. 35, §B.98 · p. 35

B.104 Change English Tense

• Other transformations change tense, inject lookup-table OCR errors or typos, fill words using masked-language-model recommendations, alter spaces, convert units, swap related nouns, and replace writing systems.Unit conversion preserves quantity format and accurate values, while writing-system replacement uses CJK Unified Ideographs as the source character set.Source: §B.104 · p. 35, §B.105 · p. 35, §B.106 · p. 36

B.110 Use Acronyms

• Acronym, visual-letter, calendar-name, whitespace, context-noise, yes-no-question, and Yoda transformations provide task-oriented variants while generally preserving or deliberately challenging input meaning and form.Acronym quality depends on a naively filtered Wikipedia list, leaving room for improvement; context noise prepends or appends randomly formed sentences to QA contexts.Source: §B.110 · p. 36, §B.111 · p. 36, §B.112 · p. 36

C Filters

• The filter collection outputs booleans for feature-based data selection, supporting targeted subpopulations by language, code-mixing, diacritics, encoding, British usage, gender representation, and group inequity.The group inequity filter compares major- and minor-group factor percentages, while the gender filter supports English, French, Polish, and Russian.Source: C Filters · p. 37, §C.1 · p. 37, §C.2 · p. 37

C.7 Keyword Filter

• Additional filters select examples by keywords, language confidence, length, named-entity count, numeric content, or quantitative-question structure to create task-specific evaluation splits.Language filtering uses langid.py confidence thresholds, while numeric and quantitative-question filters target numerical understanding in entailment, QA, and related analyses.Source: §C.7 · p. 38, §C.8 · p. 38, §C.9 · p. 38

C.12 Oscillatory Hallucinations Filter

• The oscillatory-hallucinations filter detects repeated bigram patterns by comparing source and target frequencies, and the polarity filter excludes augmented sentiment examples whose polarity changes.These filters respectively address hallucinations in generated outputs and label preservation during sentiment-data augmentation.Source: §C.12 · p. 38, §C.13 · p. 39

C.15 Question type filter; C.23 Yes/no question filter

• The question filters identify question categories from question words or answer entity types and select questions answerable with “yes” or “no,” with ambiguity limiting the latter.Question-type information supports question-answering development and performance assessment; the yes/no filter is rule-based.Source: §C.15 · p. 39, §C.23 · p. 39

C.16 Repetitions Filter

• The repetitions filter detects repeated text using simple heuristic rules, supporting identification of repetitions common in spoken-language data.An example contains the repeated phrase “I I want to sleep.”Source: §C.16 · p. 39

C.17 Phonetic Match Filter

• The phonetic-match filter compares phonetic-unit sets for input text and supplied keywords to detect homophone matches.The filter can match “trombno” with the keyword “trombone.”Source: §C.17 · p. 39

C.18 Special Casing Filter

• The special-casing filter selects texts that are entirely lowercased, uppercased, or title-cased, enabling splits containing unusual casing such as misspellings.It checks whether the input string has one of the three specified casing patterns.Source: §C.18 · p. 39

C.19 Speech-Tag Filter

• The speech-tag filter selects examples when counts of chosen part-of-speech tags satisfy predefined conditions, such as exceeding a threshold.The example meets conditions for containing one verb and two numbers.Source: §C.19 · p. 39

C.20 Token-Amount filter

• The token-amount filter selects texts containing specified keywords a required number of times.It can identify text containing two occurrences of “in.”Source: §C.20 · p. 39

C.21 Toxicity Filter

• The toxicity filter applies a pretrained toxicity detector, selecting texts whose toxicity scores match a threshold across seven available toxicity types.Any of the seven toxicity scores can serve as a filtering criterion.Source: §C.21 · p. 39

C.22 Universal Bias Filter

• The universal bias filter measures representation balance across twelve social categories, but its lexical seeds are currently English-only and require additions for other languages.The categories include religion, race, ethnicity, gender, sexual orientation, age, appearance, disability, experience, education, and economic status.Source: §C.22 · p. 39