PaperlayerView exact revision on arXiv

Source-linked AI summary

Text Data Augmentation: Towards better detection of spear-phishing emails

Mehdi Regina, Maxime Meyer, Sébastien Goutal

arXiv:2007.02033v2cs.CLcs.IRcs.LG

TL;DR

Data scarcity and the difficulty of producing intelligible, task-relevant text augmentations motivate a corpus- and task-agnostic framework combining several methods. The authors report augmentation benefits on BEC detection, including with a pretrained language model, while noting limits on preserving text characteristics.

  • Problem

    Text augmentation is difficult because transformations must account for language complexity and remain relevant to the NLP task, while annotated data can be costly to acquire.

    Source: §1 · pp. 1–2

  • Method

    The paper proposes a corpus- and task-agnostic English-text augmentation framework combining methods including BERT-based replacements and heuristics.

    Source: §1 · p. 2, Words replacements · p. 5

  • Results

    Augmentation significantly benefits CNN generalization on the imbalanced BEC dataset, with gains increasing as the per-sample cap rises, and has a positive impact with BERT as well.

    Source: §4.5 · p. 10, §4.6 · p. 11

  • Takeaways & Limitations

    The authors conclude that augmentation remains relevant in low-data regimes, including with recent pretrained language models such as BERT.

    Source: §4.6 · p. 11

    The framework cannot guarantee that transformations preserve data characteristics, and its validation module is not an oracle.

    Source: §5.3 · p. 13

Abstract

from arXiv · show

Text data augmentation, i.e., the creation of new textual data from an existing text, is challenging. Indeed, augmentation transformations should take into account language complexity while being relevant to the target Natural Language Processing (NLP) task (e.g., Machine Translation, Text Classification). Initially motivated by an application of Business Email Compromise (BEC) detection, we propose a corpus and task agnostic augmentation framework used as a service to augment English texts within our company. Our proposal combines different methods, utilizing BERT language model, multi-step back-translation and heuristics. We show that our augmentation framework improves performances on several text classification tasks using publicly available models and corpora as well as on a BEC detection task. We also provide a comprehensive argumentation about the limitations of our augmentation framework.

1 Introduction

Data scarcity makes generalization difficult, while text augmentation is challenging because transformations must remain intelligible, grammatical, and relevant to the NLP task. The paper introduces a corpus- and task-agnostic English augmentation framework, motivated in part by BEC detection.

1 Introduction

  • Data augmentation creates samples from existing data to increase diversity and reinforce characteristics such as invariances, addressing data scarcity in supervised learning.Annotated data acquisition is costly, and generalization concerns predictions on unseen data sampled from an unknown distribution.Source: §1 · p. 1
  • Text augmentation is difficult in NLP because discrete words, polysemy, and ambiguity make intelligible, grammatical, task-relevant transformations challenging.The passage contrasts this with augmentation in computer vision and audio, where simpler operations leverage continuity and task invariances [40].Source: §1 · p. 1
  • BEC emails are rare, difficult to detect and collect, and can cause substantial financial losses; the BEC corpus is also imbalanced because BEC examples are limited.An email text classifier trained on annotated BEC and benign email texts is one component of the detection system.Source: §1 · p. 2
  • The framework combines augmentation methods to serve English corpora across tasks, including BEC detection, sentiment analysis, and question classification.The authors describe it as usable without training or finetuning models.Source: §1 · p. 2

2 Related Work

Prior text-augmentation work includes word replacement, text generation, back-translation, and embedding-space noise. This paper situates its approach among these methods and argues for combining augmentation techniques.

2 Related Work

  • Word replacement methods use synonyms or embedding- and language-model-based substitutions [52] [25], while word dropout and swap may undermine readability or syntax [52].Preserving meaning may support label compatibility in supervised settings.Source: §2 · p. 3
  • Paraphrase generators offer control over text generation, but reported gaps in readability and relevance and complex architectures remain challenges [31].The cited approaches use a conditioned VAE or a generator-evaluator with supervised training and reinforcement-learning finetuning.Source: §2 · p. 3
  • Multi-step back-translation uses repeated machine-translation compressions and generations to create paraphrases; the paper implements its own version [9] [47] [56].The passage describes neural machine translation and Seq2Seq models as enabling this approach [55] [3] [28] [43].Source: §2 · p. 3
  • Embedding-space noise has been studied for augmentation [58], but the authors argue it may create unnatural representations because contextual embeddings are deterministic.They distinguish this from image-pixel noise, which can reflect sensor uncertainty or adversarial attacks [19].Source: §2 · p. 3
  • Combining augmentation methods is presented as a way to increase diversity; the proposed framework combines word replacements with back-translation [52].The related work also notes combinations of word-level and embedding-level augmentation.Source: §2 · p. 3

3 Method

The method generates candidate augmentations through configurable pipelines and filters them for similarity and redundancy. It combines token normalization, word-level transformations, and multi-step back-translation without requiring finetuning for its replacement methods.

3 Method

  • The framework combines augmentation methods to produce candidate texts intended to improve datasets and model generalization.Pipelines are configurable, with normalization, augmentation, and validation modules that may keep or discard each candidate.Source: §3 · p. 4

3.1 Normalization module

  • Normalization tokenizes and encodes input text, using spaCy to supply token features such as part of speech, named entities, and stems.These features are used by augmentation methods.Source: §3.1 · p. 4

3.2 Augmentation module

  • Each augmentation sequence applies k methods chosen one by one at random with replacement, transforming normalized text successively.The method set includes multi-step translation, BERT- and WordNet-based replacements, abbreviations, and heuristic misspellings.Source: §3.2 · p. 4

Words replacements:

  • Word replacements aim to preserve syntax and meaning through BERT/WordNet substitutions, context-safe abbreviation swaps, and heuristic misspellings, without finetuning.BERT replacements sample masked-word candidates and constrain them by part of speech and cosine similarity; the first method masks 10% of WordPiece tokens and uses 95% nucleus sampling.Source: Words replacements · pp. 4–5

Multiple steps of translation:

  • Multi-step back-translation cycles from English through intermediate languages; experiments use Google Translate and limit cycles to three edges.The selected English translation pairs are chosen for performance, and the example path is English → Spanish → Danish → English.Source: Multiple steps of translation · pp. 5–6

3.3 Validation module

  • Validation first classifies candidate texts as similar or dissimilar to the source, then detects redundancy among accepted candidates.The stated goal is to discard texts that are too dissimilar or redundant.Source: §3.3 · p. 6

Dissimilar text detection:

  • A logistic-regression text-pair classifier trained on paraphrase corpora filters dissimilar candidates; three complementary similarity metrics slightly improve on Sentence-BERT accuracy without added computation time.The training data include Quora Question Pairs, Microsoft Research Paraphrases Corpus, and an internal email-pair dataset.Source: Dissimilar text detection · p. 6

Redundant text detection:

  • Redundancy detection scores transformations, rescales scores by source length, and discards similar candidates below a threshold tuned for near-duplicate F1.The threshold is tuned on an internal labeled text-pairs dataset, among pairs classified as similar.Source: Redundant text detection · pp. 6–7

4 Experiments

The experiments evaluate the augmentation framework on public sentiment and question-classification datasets and a private, imbalanced BEC email dataset, using CNN classifiers and, for BEC, a fine-tuned BERT classifier.

4.1 Datasets

  • The BEC dataset contains anonymized suspicious and nonsuspicious emails, and only minority BEC typology samples are augmented to address class imbalance.Experiments also consider dataset size, text length, and task complexity as factors that may affect augmentation value.Source: §4.1 · pp. 7–8, Table 2 · p. 7

4.2 Models

  • The study uses CNNs for sentence classification on SST-2, TREC-6, and BEC, additionally fine-tunes BERT for BEC, and applies early stopping using development sets.SST-2 uses its provided development set; TREC-6 and BEC use stratified 90%/10% and 80%/20% splits of the training set, respectively.Source: §4.2 · p. 8

4.3 Data augmentation

  • Ten augmentation pipelines are generated per input, and the average number of retained augmentations per sample is recorded as the augmentation factor.The authors attribute differences in augmentation factor to average text length and expect greater augmentation potential for BEC.Source: §4.3 · p. 8, Table 3 · p. 9

4.4 Impact of data size

  • On SST-2 and TREC-6, merged original-plus-augmented training data contributes most in low-data regimes, while augmented-only training shows no significant performance drop.Figure 1 reports average results and standard-deviation bands across model-initialization and data-downsampling seeds.Source: §4.4 · p. 9, Figure 1 · p. 9

4.5 Impact of augmented data size on an imbalanced corpus

  • On BEC, merged data outperforms the other data types, with gains increasing strongly up to a cap of two augmentations per sample and more moderately thereafter.The slowdown may reflect harder-to-maintain diversity or other factors such as model capacity; sample analysis suggests diversity could be improved.Source: §4.5 · p. 10, Figure 2 · p. 10

4.6 Our use case with class imbalance

  • Augmentation positively affects BEC balanced accuracy for both CNN and BERT, with a larger contribution than class weighting for CNN and a smaller one for BERT.The authors conclude that augmentation remains relevant with pretrained language models in low-data regimes.Source: §4.6 · p. 11, Table 4 · p. 11

5 Further analysis

The framework’s corpus-agnostic design supports varied uses but can limit text quality and task-specific validity. Its diversity and validation analyses identify constraints that remain despite the framework’s safeguards.

5.1 Corpus agnostic framework

  • Corpus agnosticism supports varied company use cases but may degrade augmented-text quality by preventing context-specific methods such as model finetuning.The framework uses constrained word replacements to reduce inconsistent substitutions, though cosine similarity over BERT embeddings and preserving POS tags are imperfect constraints.Source: §5.1 · p. 11

5.2 Augmentations and diversity

  • Augmented texts add diversity but remain near their source samples, leaving neighborhood expansion while preserving validity as an open challenge.The authors propose adding translation paths and changing replacement sampling as future avenues for improving diversity.Source: §5.2 · p. 12

5.3 Augmentations validation

  • The validation module checks paraphrase compatibility and removes detected near-duplicates, but paraphrase detectors can fail on lexically overlapping sentences with different meanings.The authors assume paraphrasing provides sufficient control against poisoning in most contexts, not that validation is an oracle.Source: §5.3 · p. 13

5.4 Task influence

  • Valid paraphrases may still poison some task-specific data; back-translation can alter writer-style characteristics relevant to authorship attribution.The modular framework may address some contexts by omitting multi-step back-translation for authorship attribution.Source: §5.4 · p. 13

6 Closing remarks

The paper presents a corpus- and task-agnostic text augmentation framework and reports improved generalization in low-data contexts. Its analysis identifies future directions for improving augmented-text diversity and validity.

6 Closing remarks

  • The framework combines several augmentation methods and is designed for use across NLP projects without corpus- or task-specific constraints.The conclusion characterizes it as easily applicable to different projects.Source: §6 · p. 13
  • Experiments show significantly improved machine-learning generalization across different low-data-regime contexts.Source: §6 · p. 13
  • Future work targets augmented-text diversity and validity through adding methods such as natural language generation and improving replacements monitoring and validation.Source: §6 · p. 13

A Misspelling replacements heuristic

The appendix gives a token-selection probability for misspelling replacement and an example multistep translation graph. The example selects languages with stated translation-performance criteria and traces one English-centered cycle.

A Misspelling replacements heuristic

B Example of multistep translation graph

C Example of augmentation scoring

The appendix illustrates redundancy-detection scoring and states that its example score is rescaled by the number of spaCy tokens in the original text.

C Example of augmentation scoring

D BEC corpus labels

Table 7 describes the labels used for the BEC corpus.

D BEC corpus labels

E Redundancy among augmentations

An analysis found limited diversity in augmented texts compared with randomly selected original texts.

E Redundancy among augmentations

F Internal text pairs dataset labels

The internal text-pair labels distinguish similarity by meaning and lexical overlap, with labels 0–1 grouped as dissimilar and 2–3 as similar. The corpus combines manually anonymized professional emails with public email texts, for which new paired texts were handcrafted.

F Internal text pairs dataset labels

  • The four labels distinguish pairs by meaning and lexical overlap, from different-meaning, low-overlap pairs (0) to same-meaning, nearly identical pairs (3).Label 1 denotes different meaning with significant overlap; label 2 denotes the same meaning with low to moderate overlap.Source: F Internal text pairs dataset labels · p. 19
  • Labels 0 and 1 are grouped as dissimilar, while labels 2 and 3 are grouped as similar.Source: F Internal text pairs dataset labels · p. 19
  • The corpus pairs selected texts with handcrafted new texts, using manually anonymized professional emails and texts from public email corpora including Enron, DNC15, and Podesta16.Source: F Internal text pairs dataset labels · p. 19

Related papers

Loading 2007.02033v2…