Source-linked AI summary
Data Augmentation in Natural Language Processing: A Novel Text Generation Approach for Long and Short Text Classifiers
Markus Bayer, Marc-André Kaufhold, Björn Buchhold, Marcel Keller, Jörg Dallmeyer, Christian Reuter
TL;DR
NLP data augmentation is difficult when training data is scarce and text transformations must create novel linguistic patterns while preserving labels. The paper proposes generation-based augmentation for long and short texts and reports improvements across evaluated tasks, while noting dependence on the language model and substantial resource demands.
Problem
NLP applications often lack sufficient or affordable labeled training data, while textual transformations are difficult to define and can produce mixed results.
Method
The paper uses two text-generation augmentation methods: context-conditional generation for long texts and context-independent generation for short texts, with prefixes supporting label preservation and diversity.
Results
The method improved classification in low-data evaluations, with additive increases of up to 4.8% in average F1 and 4.5% in maximum F1, although gains were absent when classifiers already performed very well.
Takeaways & Limitations
Generation-based augmentation can provide useful improvements for both long- and short-text classification, particularly in small-data settings.
Takeaways & Limitations
Performance gains may not remain the same with other language models, and larger models such as GPT-3 require substantial resources that can limit their use.
Abstract
from arXiv · showhide
In many cases of machine learning, research suggests that the development of training data might have a higher relevance than the choice and modelling of classifiers themselves. Thus, data augmentation methods have been developed to improve classifiers by artificially created training data. In NLP, there is the challenge of establishing universal rules for text transformations which provide new linguistic patterns. In this paper, we present and evaluate a text generation method suitable to increase the performance of classifiers for long and short texts. We achieved promising improvements when evaluating short as well as long text tasks with the enhancement by our text generation method. Especially with regard to small data analytics, additive accuracy gains of up to 15.53% and 3.56% are achieved within a constructed low data regime, compared to the no augmentation baseline and another data augmentation technique. As the current track of these constructed regimes is not universally applicable, we also show major improvements in several real world low data tasks (up to +4.84 F1-score). Since we are evaluating the method from many perspectives (in total 11 datasets), we also observe situations where the method might not be suitable. We discuss implications and patterns for the successful application of our approach on different types of datasets.
1 Introduction
The paper addresses scarce and costly labeled data in NLP by proposing text-generation augmentation for long and short texts, with evaluations showing gains in low-data settings.
- Motivation: NLP data augmentation is difficult because automatic textual transformations must introduce useful variation while preserving label quality.The challenge is especially relevant when training data is scarce or labeling is costly.
- Approach: The proposed approach generates novel linguistic patterns rather than merely producing highly similar instances, targeting compatibility with pretrained models.The method is designed to provide high grammatical variety and substantially novel examples.
- Research questions: Two submethods address different inputs: context-conditional generation for long texts and context-independent generation for short texts.The paper uses a 280-character guideline to distinguish the two settings approximately.
- Contributions and results: Accuracy gains reach 15.53% versus no augmentation and 3.56% versus another augmentation technique in a constructed low-data regime.These are additive gains reported for the paper’s low-data evaluation.
- Contributions and results: The study evaluates the method across 11 datasets and also identifies situations where it may not be suitable.The evaluation includes both constructed low-data regimes and real-world low-data tasks.
2 Related Work
Related work frames NLP augmentation as a label-preservation problem and motivates generation methods that can create novel, coherent patterns for both short and long texts.
- Foundations of Data Augmentation: Data augmentation enlarges training data through label-preserving transformations, which modify examples while retaining their class information.The concept originated prominently in image augmentation but is applied here to textual data.
- NLP Data Augmentation: NLP transformations include swapping, deletion, spelling errors, paraphrasing, synonym replacement, and embedding-based methods, but their results are often mixed.The literature contains many transformation families rather than one universally reliable rule.
- Text Generation: Prior work has used GPT-2 for text generation, while surveys provide broader taxonomies of data augmentation techniques.The paper positions its method within existing generation-based and survey literature.
- Research Gap: A central hypothesis is that augmentation helps only when generated data contains task-relevant linguistic patterns not already encountered during pretraining.This motivates prioritizing novelty alongside label preservation.
- Research Gap: The paper targets three gaps: handling short and long texts, preserving labels and quality, and improving augmentation usefulness with pretrained classifiers.Its contribution combines novelty, coherence, and label preservation across both text-length settings.
- Research Gap: Unlike a prior classifier-based filter that can reduce diversity, the proposed method uses special finetuning, prefixes, and document-embedding filtering.The comparison emphasizes preserving both quality and diversity in generated data.
3 Concept and Implementation
The method finetunes a language model on class-specific data, generates prefix-conditioned examples, and filters semantically deviant instances using document embeddings.
- Conceptual Design: For long texts, generation uses contextual prefixes such as titles or opening words; for short texts, it uses numbered instance tokens.These prefixes are added during finetuning and reused during generation to support class-related variation.
- Conceptual Design: The pipeline finetunes a language model on class data, generates new instances, embeds generated and original examples, and retains instances near the class centroid.The algorithmic procedure produces a filtered augmentation set for classifier training.
- Conceptual Design: Document embeddings act as a final heuristic: generated instances judged too distant from the class training data are discarded.The filter uses embedding-space proximity as a proxy for semantic and syntactic relatedness.
- Implementation: GPT-2 with 355 million parameters is used because its generation capabilities suit small-data analytics, although the safety steps cannot guarantee correct labels completely.Implementation applies the three extensions to the pretrained model.
- Implementation: Figure 1 summarizes the ordered safety process: prefix addition, GPT-2 finetuning and generation, then BERT-embedding filtration.The figure presents the three stages as mechanisms for increasing the probability of label-preserving instances.
4 Evaluation
Across sentiment, news, and crisis-informatics tasks, the evaluation finds that GPT-2-based augmentation can improve classification in low-data settings, while benefits vary by dataset and method variant.
- The method generally improves classification over baselines, with gains strongest in low-data settings and across several real-world tasks.Results include improvements in sentiment, news, and crisis datasets, but not uniformly across every task.
- 15.53% and 3.56% are the largest additive gains over the baseline and EDA, respectively, in the low-data SST-2 evaluation.With more data, the reported gains decrease to 0.49% over the baseline and 1.22% over EDA.
- Ablations show that removing the numbering token lowers average accuracy by 5.42 points, while omitting filtering lowers it by 2.64 points.The authors therefore retain all augmentation steps for subsequent evaluations.
- Human inspection finds generated texts coherent, diverse, and usually class-relevant, while filtering mainly removes outputs with repeated words.Qualitative examples also show meaningful expansions, but occasional label changes or contextually incorrect reformulations remain possible.
- News-topic augmentation yields relative maximum-F1 increases above 4% for MC and layoff and 2% for M&A, with +0.64 for flood.Wildfire shows no improvement over baseline, although augmentation reduces result variability.
- Crisis-task augmentation improves Olteanu et al. datasets by 2.1%–3.8% on average and 1.5%–2.5% in best runs, but harms Dublin performance.F1 standard deviation decreases for every crisis task despite limited direct-score gains for Schulz et al. datasets.
5 Discussion and Conclusion
The paper presents GPT-2-based augmentation methods for long and short texts, using safety steps to preserve labels and evaluating them across 11 datasets. Results show substantial gains in low-data settings, alongside dataset- and model-dependent limitations.
- Method: The proposed methods combine GPT-2 generation with priming, finetuning, and BERT-embedding filtering to increase novelty while preserving label quality.The long-text variant conditions generation on instance context, whereas the short-text variant is context independent.
- Limitations: GPT-2 generation requires substantially more time than simple noise-based augmentation and is mainly limited to English in the evaluated implementation.Generating one long-text example took about 10–30 seconds, although other language models could reduce these constraints.
- Short-text classification: Up to 15.53 and 3.81 points of improvement were achieved for short-text tasks in constructed and real-world low-data regimes, respectively.The method was unsuitable for two broadly defined real-world tasks, where GPT-2 could not infer the right context from finetuning alone.
- Contributions: The study contributes augmentation methods, a basis for pretrained classifiers, and empirical insights across sentiment analysis, news classification, and crisis informatics.The evaluation covers 11 datasets and examines domain-specific applications of small-data analytics.
Appendices
The appendices document data availability, dataset composition, implementation details, and ethical constraints. They also describe the augmentation algorithms and qualitative properties of generated examples.
- Data availability: The second evaluation uses news datasets that are not publicly available because of publication restrictions by news agencies.The paper provides concise dataset descriptions but notes that reproducibility must be inferred from the public evaluations.
- Experimental setup: The experiments use English datasets, including crisis, sentiment, and news classification data, with GPT-2, Sentence-BERT filtering, and ULMFit classifiers.The appendix lists the relevant datasets, model implementations, and hardware used across evaluations.
- Ethics: The authors restrict social-media processing to textual content and labels, excluding user metadata such as names and locations.This was intended to respect privacy as much as possible.
- Augmentation algorithms: The short-text algorithm prefixes class instances with indexed start and end markers, finetunes a language model, generates examples, and filters them using document embeddings.The procedure generates n examples per training instance before embedding generated and original data for filtering.
- Augmentation algorithms: The long-text algorithm conditions generation on extracted context from each training instance before applying embedding-based filtering.Its pipeline uses a context-extraction function to form the generation prefix for each instance.
- Limitations: GPT-2 and other language models may contain gender, religious, or racial biases that can produce discriminatory classifier decisions.The appendix identifies bias propagation as a practical deployment risk.
- Generated data analysis: Generated examples can contain novel linguistic features, but increasing the amount of generated data also increases the number of duplicates.Qualitative inspection found truncations, word substitutions, highly diverse instances, and repetitions of originals.