Source-linked AI summary
Balancing Methods for Multi-label Text Classification with Long-Tailed Class Distribution
Yi Huang, Buse Giledereli, Abdullatif Köksal, Arzucan Özgür, Elif Ozkirimli
TL;DR
Multi-label classification is complicated by long-tailed class frequencies and label co-occurrence. The paper applies balancing loss functions to NLP classification and compares them on Reuters-21578 and PubMed. Distribution-balanced loss outperforms other approaches across the datasets, with reported tail-label improvements.
Problem
Long-tailed label frequencies and co-occurrence can bias multi-label classification toward head labels, despite useful links between head and rare labels.
Source: §1 · p. 1Method
The paper applies and compares balancing loss functions for multi-label text classification, introducing distribution-balanced loss to NLP and proposing CB-NTR.
Source: §5 · p. 5Results
Takeaways & Limitations
The authors conclude that loss functions can effectively address class imbalance and label co-occurrence without additional information and can be used with neural-network models.
Source: §5 · p. 5
Abstract
from arXiv · showhide
Multi-label text classification is a challenging task because it requires capturing label dependencies. It becomes even more challenging when class distribution is long-tailed. Resampling and re-weighting are common approaches used for addressing the class imbalance problem, however, they are not effective when there is label dependency besides class imbalance because they result in oversampling of common labels. Here, we introduce the application of balancing loss functions for multi-label text classification. We perform experiments on a general domain dataset with 90 labels (Reuters-21578) and a domain-specific dataset from PubMed with 18211 labels. We find that a distribution-balanced loss function, which inherently addresses both the class imbalance and label linkage problems, outperforms commonly used loss functions. Distribution balancing methods have been successfully used in the image recognition field. Here, we show their effectiveness in natural language processing. Source code is available at https://github.com/Roche/BalancedLossNLP.
1 Introduction
Multi-label text classification assigns multiple labels from a fixed set to each text, but long-tailed class frequencies and label co-occurrence make the task difficult. This work introduces balancing loss functions to NLP and reports improved results on Reuters-21578 and PubMed.
1 Introduction
- Rare tail labels can be biased by co-occurrence with frequent head labels, even when their co-occurrence information is important for accurate modeling.In Reuters, half the labels occur in less than 5% of training data.Source: §1 · pp. 1–2
- Earlier NLP approaches resampled rare-label samples, used co-occurrence in initialization, or used multi-task architectures, but were unsuitable for imbalanced data or architecture-dependent.Source: §1 · p. 2
- Balancing losses address imbalance and co-occurrence in computer vision, while loss-function manipulation has also been explored in NLP as an architecture-agnostic approach.Source: §1 · p. 2
- On Reuters-21578 and PubMed, distribution-balancing methods outperform other loss functions on total metrics and significantly improve tail labels.The experiments cover a general-domain dataset and a biomedical dataset.Source: §1 · p. 2
2 Loss Functions
The section introduces BCE and three alternative balancing losses for multi-label classification: focal loss, class-balanced focal loss, and distribution-balanced loss. Distribution balancing addresses label co-occurrence as well as imbalance through rebalanced weighting and negative-tolerant regularization.
2 Loss Functions
- BCE is the commonly used multi-label loss, but it is vulnerable to imbalance because head classes or negative instances dominate.The balancing methods reweight BCE so rare instance-label pairs receive reasonable attention.Source: §2 · p. 2
2.1 Focal loss (FL)
- Focal loss modulates BCE with a tunable focusing parameter γ to give greater weight to hard-to-classify instances predicted with low probability.The multi-label focal loss is defined using the predicted class probabilities and ground-truth labels.Source: §2.1 · pp. 2–3
2.2 Class-balanced focal loss (CB)
- Class-balanced focal loss reweights focal loss using each label’s effective sample count, accounting for diminishing marginal benefits of additional data.The parameter β controls how quickly the effective number grows.Source: §2.2 · p. 3
2.3 Distribution-balanced loss (DB)
- Distribution-balanced loss combines rebalanced weighting with negative-tolerant regularization to reduce redundant co-occurrence information and downweight easy negative instances.Its rebalanced-FL weighting normalizes class- and instance-level resampling probabilities, then smooths the ratio into [α, α + 1].Source: §2.3 · p. 3
3 Experiments
The experiments use Reuters-21578 and PubMed to evaluate classification across datasets of different sizes, domains, and label distributions. They compare loss functions and an SVM baseline using micro- and macro-F1 across the full label set and frequency-based subsets.
3.1 Datasets
- Reuters-21578 contains 7,769 training and 3,019 test documents with 90 labels divided into head, medium, and tail subsets.The training set includes 1,000 documents for validation; the label subsets contain 30, 31, and 30 labels, respectively.Source: §3.1 · p. 3
- PubMed contributes 224,897 articles from 2020–2021 and 18,211 MeSH labels divided into head, medium, and tail subsets.Of the articles, 10,000 are used for validation and testing; the subsets contain 6,018, 5,581, and 6,612 labels, respectively.Source: §3.1 · pp. 3–4
3.2 Experimental Settings
- Evaluation compares loss functions and an SVM one-vs-rest baseline using best micro-F1 and macro-F1 for all labels and head, medium, and tail subsets.The metrics are reported separately for each dataset and label-frequency subset.Source: §3.2 · p. 4
4 Results
Distribution-balanced loss (DB) performs best overall across the reported class subsets, with especially strong medium- and tail-class results. Ablations associate DB’s gains with its focal, NTR, and rebalanced-weighting layers; errors still commonly involve similar or linked labels.
Dataset-level results
- Reuters shows micro-F1 around 90 versus macro-F1 around 50–60, while PubMed scores 60 versus around 15 on those metrics.On Reuters, BCE has total macro-F1 of 47 and tail F1 of 0; PubMed has more pronounced imbalance.Source: §4 · p. 4
Experiments with Reuters-21578 dataset
- 21.49 and 25.81 are DB’s tail micro-F1 gains over FL and CB, respectively, on Reuters-21578.The other listed loss functions also outperform BCE in medium and tail classes, while performing similarly to BCE on head classes.Source: §4 · p. 4
Experiments with PubMed dataset
- 41 and 24 are DB’s medium- and tail-class micro-F1 scores, respectively, on PubMed, where DB has the highest performance across classes.FL scores 26 for medium and 9 for tail micro-F1; the other loss functions outperform FL in those classes.Source: §4 · p. 4
Ablation Study
- Removing the NTR or focal layer reduces performance across subsets on both datasets; DB also has higher macro- and medium/tail F1 than NTR-FL.CB-NTR exceeds CB across class sets, while DB is close to or better than CB-NTR on medium and tail classes.Source: §4 · p. 5
Error Analysis
- The most common errors involve similar or linked labels; BCE makes significantly more errors on the listed PubMed class pairs than the other investigated losses.Reuters confusions include platinum/gold, yen/money-fx, and platinum/copper; PubMed confusions include Pandemics/Betacoronavirus and Pandemics/SARS-CoV-2.Source: §4 · p. 5
5 Conclusion
The paper introduces balancing loss functions, including DB, for multi-label text classification in NLP and proposes CB-NTR. Experiments report that DB outperforms other approaches while accounting for long-tailed distributions and label co-occurrence; the authors conclude that loss-based balancing is effective for this task.
Conclusion
- The work introduces balancing loss functions to NLP multi-label text classification and proposes CB-NTR as a novel loss function.It first introduces DB to NLP and designs CB-NTR.Source: §5 · p. 5
- DB outperforms other approaches by considering both long-tailed distribution and label co-occurrence, with reported robustness across Reuters and PubMed.The datasets cover 90 general-domain labels and 18,211 biomedical-domain labels.Source: §5 · p. 5
- The authors conclude that loss-based balancing addresses class imbalance and label co-occurrence without requiring additional information and can be used with neural network-based models.They suggest it may also be useful for other NLP tasks, including tagging, recognition, and reading comprehension.Source: §5 · p. 5
A.1 Experimental Settings
The experiments select each model’s threshold using validation micro-F1, then evaluate micro- and macro-F1 on test data. They compare several BCE-based loss variants and use BERT-based backbones for Reuters and PubMed.
Evaluation metrics
- Each dataset-method threshold is selected for the best validation micro-F1, and test performance is evaluated with micro-F1 and macro-F1.Source: §A.1 · p. 8
Loss function parameters
- The loss comparison includes BCE, FL (γ=2), CB (β=0.9), R-FL, NTR-FL, and DB.R-FL uses α=0.1 and β=10, with µ=0.9 for Reuters and 0.05 for PubMed; NTR-FL uses κ=0.05 and λ=2.Source: §A.1 · p. 8
Implementation Details
- Reuters uses bert-base-cased and PubMed uses biobert-base-cased-v1.1 with BertForSequenceClassification; both base models have 110 million parameters.Training sequences are truncated at length 512 and grouped in batches of 32.Source: §A.1 · p. 8
- Training uses AdamW with weight decay 0.01 and a searched learning rate; the SVM one-vs-rest baseline uses TF-IDF with a tuned linear kernel and hyper-plane shifting.Reuters experiments use one V100 GPU and PubMed experiments one A100 GPU.Source: §A.1 · p. 8
A.2 Additional Effectiveness Check
This check stratifies performance by the number of labels per instance: DB performs well across Reuters groups, while NTR-FL, CB-NTR, and DB lead across PubMed quantiles.
A.2 Additional Effectiveness Check
- On Reuters multi-label instances, DB beats other functions on micro-F1 and leads on macro-F1 for both instance groups.All functions perform similarly on single-label instances, while BCE drops more than DB on multi-label instances.Source: §A.2 · p. 8
- The PubMed figure pairs a log-scale label-distribution curve with a co-occurrence matrix colored by the quad root of conditional probability p(i|j).The matrix places class i in columns and class j in rows.Source: Figure 3 · p. 8
- In each PubMed label-count quantile, NTR-FL, CB-NTR, and DB outperform the other models on all metrics.Fewer than 0.1% of PubMed instances have a single label, so instances are divided into three quantiles by label count.Source: §A.2 · p. 8
- Table 3 reports micro- and macro-F1 for total label sets and instance groups across eight loss functions.It covers Reuters-21578 and PubMed, with groups based on the number of labels per instance.Source: Table 3 · p. 9