Source-linked AI summary
Large-scale Multi-label Text Classification - Revisiting Neural Networks
Jinseok Nam, Jungi Kim, Eneldo Loza Mencía, Iryna Gurevych, Johannes Fürnkranz
TL;DR
Large-scale multi-label text classification must handle label dependencies while remaining effective for high-dimensional, label-rich text collections. The paper replaces BP-MLL’s pairwise ranking loss with cross entropy and combines a simple neural network with modern training techniques. Experiments show that these models match or outperform state-of-the-art approaches, while the paper identifies scalability and optimization limitations of existing methods.
Problem
Large-scale text collections challenge methods that model label dependencies or joint label distributions, while independent binary relevance ignores those dependencies.
Method
The paper uses a single-hidden-layer neural network with cross entropy, ReLUs, Dropout, AdaGrad, and threshold-based label prediction.
Results
The proposed framework outperforms BP-MLL in predictive performance, computational complexity, and convergence speed, and performs as well as or better than state-of-the-art approaches.
Takeaways & Limitations
Simple neural networks with advanced training techniques are effective for large-scale multi-label text classification.
Takeaways & Limitations
Performance decreases for linear SVMs as label counts grow and label frequencies become skewed, while ranking-loss minimization becomes intractable at large scale.
Abstract
from arXiv · showhide
Neural networks have recently been proposed for multi-label classification because they are able to capture and model label dependencies in the output layer. In this work, we investigate limitations of BP-MLL, a neural network (NN) architecture that aims at minimizing pairwise ranking error. Instead, we propose to use a comparably simple NN approach with recently proposed learning techniques for large-scale multi-label text classification tasks. In particular, we show that BP-MLL's ranking loss minimization can be efficiently and effectively replaced with the commonly used cross entropy error function, and demonstrate that several advances in neural network training that have been developed in the realm of deep learning can be effectively employed in this setting. Our experimental results show that simple NN models equipped with advanced techniques such as rectified linear units, dropout, and AdaGrad perform as well as or even outperform state-of-the-art approaches on six large-scale textual datasets with diverse characteristics.
1 Introduction
Growing digital collections require automatic extraction of metadata from unstructured text because manual processing is impractical. The paper studies a simple neural-network approach for large-scale multi-label text classification using cross entropy and modern training techniques.
- Motivation: Manual extraction of metadata from rapidly growing document collections is impractical in time and cost.Target information ranges from titles and authors to descriptive keywords and categories.
- Problem setting: Multi-label classification learns to assign a suitable subset of categories to each text.Binary relevance treats labels independently, but the literature focuses on exploiting label dependencies.
- Approach: The proposed approach replaces BP-MLL’s pairwise ranking loss with cross entropy and uses ReLUs, Dropout, and AdaGrad.These modifications target more efficient and effective training.
- Design rationale: A single-layer neural network is retained because it scales better for large-scale tasks and may work well with high-dimensional textual representations.The paper presents empirical evidence that this simple model with advanced training techniques performs as well as or better than state-of-the-art approaches.
2 Multi-label Classification
Multi-label classification assigns multiple relevant labels to each instance, but independent binary relevance models ignore label dependencies. Existing approaches model dependencies in different ways, while large-scale, high-dimensional, label-rich text collections create important computational and performance challenges.
- Definition: Multi-label classification predicts a subset of relevant labels for each instance, equivalently represented by an L-dimensional binary target vector.A value yi = 1 marks label i as relevant, while yi = 0 marks it irrelevant.
- Independent modeling: Binary relevance trains L independent binary classifiers, so it ignores relationships among labels that often occur together.The presence of one label may suppress or increase the likelihood of another.
- Dependency-aware methods: Pairwise decomposition, label powerset, and classifier chains incorporate label dependencies through transformed subproblems, label combinations, or previous predictions.These methods extend beyond independent per-label prediction in different ways.
- Ranking approaches: RankSVM and BP-MLL formulate multi-label prediction around ranking relevant labels ahead of irrelevant labels.A threshold must calibrate the ranking into relevant and irrelevant predictions.
- Neural-network prediction: Figure 1 illustrates a neural network with one hidden layer and a threshold selected from output scores to maximize F1.The example marks relevant and irrelevant labels separately and compares candidate thresholds.
- Large-scale limitations: Linear SVMs are prominent for high-dimensional sparse text because such data often support linear separability.Their performance decreases as label counts grow and label frequencies become skewed.
3 Neural Networks for Multi-label Classification
The paper develops a single-hidden-layer neural-network framework for large-scale multi-label classification, replacing BP-MLL’s pairwise ranking objective with cross entropy and adding modern training techniques. The framework produces label scores and converts them into binary predictions through learned thresholding, while experiments and analysis examine convergence, efficiency, and scalability.
- Framework: The framework uses a single hidden layer to produce label scores, followed by a threshold predictor that converts scores into binary label assignments.The threshold predictor learns document-specific thresholds from training data using F1 score as a reference.
- Rank Loss and BP-MLL: BP-MLL minimizes errors from incorrectly ordered relevant and irrelevant label pairs, but its ranking-based objective is difficult to minimize because it is non-convex.The paper describes pairwise error as a relaxation of rank loss and motivates convex surrogate losses.
- Cross Entropy: Setting the weighting function w(y) to 1 makes the log loss equivalent to cross entropy with sigmoid outputs.The paper presents cross entropy as the commonly used neural-network classification objective in this setting.
- Efficiency and Scalability: Cross entropy is computationally more efficient than BP-MLL’s pairwise error because the PWE gradient becomes L times more expensive as the number of labels grows.The paper therefore identifies BP-MLL as scaling poorly with the number of unique labels.
- Recent Training Advances: The paper uses ReLUs, AdaGrad, and Dropout to improve neural-network training, with AdaGrad retaining comparatively larger learning rates for rare labels.Dropout randomly removes hidden-unit activations to regularize large networks and approximate an ensemble of smaller networks.
4 Experimental Setup
The study evaluates neural-network approaches for large-scale multi-label text classification using ranking and bipartition measures across six textual datasets. It compares BP-MLL and NN variants with BR-based SVMs, including a prepared German Education Index dataset.
- Evaluation Measures: Evaluation uses both bipartition measures, such as Precision, Recall, and F1, and ranking measures for assessing ordered label lists.Ranking measures evaluate the quality of ranked labels, whereas bipartition measures evaluate assigned label sets.
- Datasets: The six datasets represent diverse large-scale text-classification domains, including standard benchmarks and a newly prepared German Education Index dataset.The German Education Index data contain documents with abstracts and human-assigned index terms.
- Datasets: Table 1 summarizes each dataset by document count, vocabulary size, total label count, and average labels per instance.
- Algorithms: NNA is a single-hidden-layer neural network with ReLU units trained using SGD and AdaGrad, while NNAD adds Dropout under the same settings.
- Algorithms: The experiments compare the NN-based algorithms with BP-MLL and binary relevance using Liblinear SVMs as a state-of-the-art representative.BR variants tune the SVM penalty parameter using either bipartition or ranking measures, and BRR also uses the NN thresholding technique.
5 Results
Experiments across six large-scale datasets examine how training techniques affect convergence, overfitting, and overall multi-label classification performance. Cross-entropy single-hidden-layer networks generally outperform BP-MLL and perform as well as or better than competing methods across ranking and bipartition measures.
- Better Local Minima and Acceleration of Convergence Speed: AdaGrad produced lower rank loss than momentum, while ReLU hidden units outperformed tanh and sigmoid within each optimizer group.These combinations also converged faster into better weight regions, which is particularly useful for large-scale text classification.
- Decorrelating Hidden Units While Output Units Remain Correlated: Dropout eventually prevented overfitting on EUR-Lex despite initially slower convergence, whereas regular networks’ rank loss worsened with further updates.Overfitting was observed on all datasets except Reuters-21578 and RCV1-v2 under the reported settings.
- Limiting Small Learning Rates in BP-MLL: Cross entropy worked well with learning rate 0.1, whereas pairwise ranking error permitted only the smaller learning rate 0.01.The authors attribute this difference to flatter PWE gradients, oscillation near local minima, and slower parameter updates.
- Limiting Small Learning Rates in BP-MLL: Networks without Dropout converged faster than networks with Dropout on both rank loss and mean average precision at the same learning rates.The reported comparison also notes that networks without Dropout eventually overfit regardless of cost function.
- Comparison of Algorithms: NNA and NNAD performed much better than BP-MLL on rank loss and other ranking measures, while matching or exceeding other methods on ranking and bipartition measures.Table 2 summarizes average algorithm ranks across the six datasets and both measure groups.
- Comparison of Algorithms: Changing BP-MLL’s hidden units from tanh to ReLU alone did not significantly improve results, whereas replacing pairwise ranking error with cross entropy did.Because BP-MLLRAD and NNAD share the architecture but differ in cost function, the authors identify the cost-function replacement as the main improvement source.
6 Conclusion
The proposed neural-network framework outperforms BP-MLL in predictive performance, computational complexity, and convergence speed. ReLUs, AdaGrad, and Dropout contribute to fast convergence and better generalization.
- The framework outperforms BP-MLL in predictive performance, computational complexity, and convergence speed.
- ReLUs combined with AdaGrad support fast convergence, while Dropout helps prevent overfitting and improves generalization.
- The evaluation reports ranking and bipartition measures, but BP-MLL results on EUR-Lex are missing because the runs could not be completed promptly.