Source-linked AI summary
On Data Augmentation for Extreme Multi-label Classification
Danqing Zhang, Tao Li, Haiyang Zhang, Bing Yin
TL;DR
Extreme multi-label classification has long-tail labels, and the impact of data augmentation on this task had not been studied. The paper evaluates GPT-2-generated label-invariant examples alongside rule-based augmentation and introduces LA-RoBERTa. Across the study, augmentation benefits both classifiers, with larger improvements for tail labels and different methods favored by data availability.
Problem
Data augmentation had not been studied for XMC, where even strong models perform poorly on tail labels, especially when training data is limited.
Source: §1 · p. 1, §5.2 · p. 8Method
The paper compares rule-based augmentation with GDA, which fine-tunes pretrained GPT-2 to generate label-invariant examples, and introduces a RoBERTa classifier with label attention.
Source: §1 · p. 2, §2.2 · p. 3Results
Both vanilla RoBERTa and LA-RoBERTa benefit from augmentation; GDA improves precision and ranking metrics, with larger improvements for tail labels, while synonym replacement performs well when data is scarce.
Source: §6 · p. 8Takeaways & Limitations
The study finds that augmentation benefits both evaluated classifiers, while the better-performing augmentation approach varies with training-data availability.
Source: §1 · p. 2, §6 · p. 8Generated text can still be artificial, including repeated or off-context phrases.
Source: §4.4 · p. 6
Abstract
from arXiv · showhide
In this paper, we focus on data augmentation for the extreme multi-label classification (XMC) problem. One of the most challenging issues of XMC is the long tail label distribution where even strong models suffer from insufficient supervision. To mitigate such label bias, we propose a simple and effective augmentation framework and a new state-of-the-art classifier. Our augmentation framework takes advantage of the pre-trained GPT-2 model to generate label-invariant perturbations of the input texts to augment the existing training data. As a result, it present substantial improvements over baseline models. Our contributions are two-factored: (1) we introduce a new state-of-the-art classifier that uses label attention with RoBERTa and combine it with our augmentation framework for further improvement; (2) we present a broad study on how effective are different augmentation methods in the XMC task.
1 Introduction
Extreme multi-label classification assigns inter-dependent labels across large, often imbalanced label sets, leaving tail labels poorly served even by strong models. The paper introduces GPT-2-based augmentation and a label-attention RoBERTa classifier, and studies augmentation across data regimes.
1 Introduction
- XMC assigns sets of inter-dependent labels, and real-world data can have extremely unbalanced label distributions that leave tail labels poorly served.The introduction notes that media-related labels cover about 60% of AmazonCat-13K.Source: §1 · p. 1
- The study compares rule-based and language-model augmentation on AmazonCat-13K, sampling training data from low-data to large-data settings.It evaluates vanilla RoBERTa and LA-RoBERTa as base models.Source: §1 · pp. 1–2
- GDA groups examples with matching label sets, fine-tunes pretrained GPT-2 to generate label-invariant perturbations, and augments training data, particularly for tail labels.Source: §1 · p. 2
- Augmented models outperform the baseline, including the new state-of-the-art baseline, in both overall precision and tail-label precision.The introduction reports this result for the evaluated augmented models.Source: §1 · p. 2
- LA-RoBERTa combines RoBERTa with label attention and outperforms the prior best model by 0.7 in top-5 precision.Source: §1 · p. 2
- GPT-2 augmentation performs better when training data is rich, while rule-based augmentation is relatively better when data is limited.Source: §1 · p. 2
2 Model
Section 2 presents rule-based and GPT-2-based augmentation for tail-label examples, introduces LA-RoBERTa, and describes weighted training on original and generated data. [p16, p17, p18, p36, p46, p47]
2.1 Generation
- The generation study focuses on producing examples for tail labels in XMC’s often unbalanced training data.The section compares rule-based and language-model-based augmentation.Source: §2.1 · p. 2, §2 · p. 2
2.1.1 Augmenting with rule-based system
- EDA makes perturbations through random replacement, insertion, deletion, and swapping; the WordNet-only variant keeps synonym swapping and replaces randomly chosen non-stop words.EDA uses WordNet for replacement, while its other edits modify text in place.Source: §2.1.1 · p. 2
2.1.2 Augmenting with pre-trained language model
- GDA groups training examples with identical label sets, fine-tunes GPT-2 to generate one text from another, and generates perturbations for tail labels.The generated examples augment the existing training data.Source: §2.1.2 · p. 3, Figure 2 · p. 3, Figure 3 · p. 3
2.2 Classification
- LA-RoBERTa builds multi-label attention on vanilla RoBERTa so each token embedding can have a different impact for each label.The attention design is adopted from AttentionXML.Source: §2.2 · p. 3
2.3 Training with Augmented Data
- Generated-data training uses a weighted loss: original examples receive λ=1, while λ for generated examples is selected by validation performance to account for possible noise and minor errors.The weight balances augmentation data against the original training data.Source: §2.3 · pp. 3–4
3 Evaluation
Evaluation covers performance across all labels and tail labels using precision and ranking metrics. P@5 and PSP@5 are the primary precision metrics, while nDCG@5 and PSnDCG@5 are the primary ranking metrics.
3 Evaluation
- The evaluation uses two metric sets to measure performance over all labels and tail labels.Source: §3 · p. 4
- All-label evaluation uses P@k and nDCG@k, with P@1, DCG@1, and nDCG@1 equal when k=1.The paper follows Bhatia et al. (2016) for these metrics; r(i) denotes the rank of the i-th label in the top-k labels.Source: §3 · p. 4
- Tail-label evaluation uses PSP@k for propensity precision and PSnDCG@k for normalized cumulative gain, following Jain et al. (2016).The propensity score pr(i) is calculated from the training dataset.Source: §3 · p. 4
- P@5 and PSP@5 are the primary precision metrics, while nDCG@5 and PSnDCG@5 are the primary ranking metrics across the experiments.The paper also identifies evaluation scripts for reference.Source: §3 · p. 4
4 Experiments
Experiments compare augmentation methods across training-data sizes on AmazonCat-13K, using RoBERTa and LA-RoBERTa as base classifiers.
4.1 Dataset
- The AmazonCat-13K setup uses 1%, 5%, 50%, and 100% training-data samples, with product titles and descriptions as input and a 500-word limit.The split contains 1,067,615 training examples, 118,624 validation examples, and 306,782 test examples.Source: §4.1 · p. 4
4.2 Generators
- Rule-based augmentation uses TextAttack EDA and WordNet, whereas neural augmentation uses small GPT-2; it keeps 5–200-word inputs and removes pairs with similarity over 0.95 to encourage diverse generation.Media-related labels cover about 60% of the data and are filtered out; examples with these labels are described as extremely diverse and often not semantically related.Source: §4.2 · pp. 4–5
4.3 Base Classifiers
- LA-RoBERTa beats the prior best by 0.7 P@5 and 1.0 nDCG@5, though AttentionXML has higher PSP@5 and PfastreXML remains stronger on tail labels.The authors attribute PfastreXML's tail-label advantage to its propensity-scored objective.Source: Table 1 · p. 5, §4.3 · p. 5, Table 2 · p. 5
4.4 Classifiers with Data Augmentation
- With ~1M examples, GDA improves RoBERTa by 1.0 P@5 and 0.9 nDCG@5; tail-label gains are 3.2 PSP@5 and 1.5 PSnDCG@5.The all-label and tail-label comparisons are reported separately.Source: §4.4 · p. 5, Table 4 · p. 6, Table 5 · p. 6
- At 1% training data, both rule-based augmentation and GDA beat the baseline, while rule-based methods improve more than GDA.The authors note that the small, highly biased training set makes GPT-2 fine-tuning challenging.Source: §4.4 · p. 6
- With full training data, LA-RoBERTa augmentation is on par overall but improves tail metrics by 0.2 PSP@5 and 0.1 PSnDCG@5.Rule-based systems do not improve precision over this baseline.Source: §4.4 · p. 6, Table 6 · p. 7, Table 7 · p. 7
5 Related Works
Prior XMC research develops neural classifiers and tail-label objectives, while NLP studies use several augmentation strategies; the effect of augmentation on XMC had not been studied.
5.1 Extreme Multi-label Classification
- XMC work moved from sparse linear models toward neural encoders, including CNN, BiLSTM with label attention, and pretrained transformers.Tail-label methods include PfastreXML's propensity-scored objective.Source: §5.1 · p. 7
5.2 Data Augmentation in NLP
- NLP augmentation includes rule-based methods such as EDA and sequence-generation approaches such as back translation, variational autoencoders, and conditional language models.The cited applications include relation extraction, spoken language understanding, and text classification.Source: §5.2 · p. 8
- The paper identifies a gap: despite augmentation studies in other NLP tasks, its impact on extreme multi-label classification had not been studied.Source: §5.2 · p. 8
6 Conclusion & Future Work
The paper presents a broad study of augmentation for XMC and reports benefits across strong RoBERTa-based classifiers, with larger gains for tail labels.
6 Conclusion & Future Work
- Both vanilla RoBERTa and LA-RoBERTa benefit from augmentation, and GDA improves precision and ranking metrics with larger gains for tail labels.Source: §6 · p. 8
- Rule-based synonym replacement performs well when training data is scarce.Source: §6 · p. 8
- Future directions include distinguishing erroneous from good generations and dynamically assigning instance-level loss weights to generated data.Source: §6 · p. 8