Source-linked AI summary
SGM: Sequence Generation Model for Multi-label Classification
Pengcheng Yang, Xu Sun, Wei Li, Shuming Ma, Wei Wu, Houfeng Wang
TL;DR
Multi-label classification must assign correlated labels while accounting for different text contributions to different predictions, challenges existing methods often overlook. The paper formulates the task as sequence generation with an attention-based LSTM decoder and global embedding. Experiments report substantial improvements over baselines, while analysis indicates label-correlation modeling and informative-word selection; performance remains difficult when samples have many labels.
Problem
Existing multi-label classification methods often overlook correlations among labels and differing contributions from text parts when predicting labels.
Method
The paper models multi-label classification as sequence generation using an encoder-decoder with attention, sequential LSTM label prediction, and global embedding.
Results
The proposed methods outperform baselines by a substantial margin and demonstrate label-correlation representation and automatic selection of informative words.
Takeaways & Limitations
SGM captures label correlations and selects informative words automatically when predicting different labels.
Takeaways & Limitations
Accurately predicting all true labels becomes intractable when a sample has many labels, and the proposed methods alleviate this only to some extent.
Abstract
from arXiv · showhide
Multi-label classification is an important yet challenging task in natural language processing. It is more complex than single-label classification in that the labels tend to be correlated. Existing methods tend to ignore the correlations between labels. Besides, different parts of the text can contribute differently for predicting different labels, which is not considered by existing models. In this paper, we propose to view the multi-label classification task as a sequence generation problem, and apply a sequence generation model with a novel decoder structure to solve it. Extensive experimental results show that our proposed methods outperform previous work by a substantial margin. Further analysis of experimental results demonstrates that the proposed methods not only capture the correlations between labels, but also select the most informative words automatically when predicting different labels.
1 Introduction
Multi-label classification assigns multiple labels to each text, but prior methods often neglect label correlations and differences in which text parts matter for each label. SGM reframes the task as sequence generation with an attention-based decoder designed to address both issues.
- Multi-label classification assigns multiple labels to each instance and supports applications including text categorization, tag recommendation, and information retrieval.
- Binary relevance ignores label correlations, while classifier chains model them but are computationally expensive for large datasets.
- Existing neural models either neglect label correlations or fail to consider differences in the text parts contributing to predictions.
- SGM views multi-label classification as sequence generation and uses an encoder-decoder architecture with attention.
- The LSTM decoder predicts labels sequentially from prior labels, while attention selects informative text parts for different label predictions.
- Extensive experiments report that the proposed methods outperform baselines by a large margin, with further analysis demonstrating effective correlation representation.
2 Proposed Method
The proposed model frames multi-label classification as sequence generation, using an encoder–decoder with attention and a masked softmax decoder. Its global embedding incorporates all label-probability information to reduce errors from previous predictions.
- Sequence Preparation: Label sequences are ordered by training-set frequency, with bos and eos symbols marking their beginning and end.The dataset statistics table summarizes total samples, label sets, words per sample, and labels per sample.
- Masked Decoding: A masked softmax outputs label probabilities while preventing labels predicted at earlier steps from being repeated.The mask assigns −∞ to previously predicted labels and 0 otherwise.
- Sequence Generation: The model consists of an encoder and decoder with attention, generating labels sequentially from encoded text.The decoder conditions each prediction on the previous decoder state, an input embedding, and an attention-derived context vector.
- Attention: Attention aggregates informative parts of the text into a time-step-specific context vector for label prediction.The encoder produces word-level hidden states, and attention assigns weights to words before forming the context vector.
- Decoder Motivation: The decoder uses the highest-probability previous label embedding, but this greedy input can create exposure bias after an incorrect prediction.Beam search is used during inference, although the paper states it cannot fundamentally eliminate exposure bias.
- Global Embedding: Global embedding combines the highest-probability label embedding with a probability-weighted average through a transform gate.The combination uses all possible labels’ probabilities, allowing the model to reduce damage from previous mispredictions.
3 Experiments
The experiments evaluate the proposed methods on two datasets by introducing datasets, metrics, implementation details, baselines, comparisons, and result analyses.
- The evaluation covers two datasets and includes dataset descriptions, evaluation metrics, experimental details, baseline comparisons, and analysis of results.
3.1 Datasets
The experiments use RCV1-V2, a large newswire dataset with multiple topics, and AAPD, a newly constructed dataset of computer-science paper abstracts and subjects.
- RCV1-V2 contains over 800,000 manually categorized Reuters newswire stories with 103 topics, and multiple topics may be assigned to each story.
- AAPD is a new large multi-label dataset built from abstracts and corresponding subjects of 55,840 computer-science papers.
- Each dataset is divided into training, validation, and test sets, with dataset statistics reported in Table 1.
3.2 Evaluation Metrics
The study uses hamming loss and micro-F1 as its main evaluation metrics, with micro-precision and micro-recall reported for additional analysis.
- Hamming loss and micro-F1 are the main evaluation metrics, while micro-precision and micro-recall support the analysis.
- Hamming loss measures the fraction of misclassified instance-label pairs, including missed relevant labels and predicted irrelevant labels.
- Micro-F1 is a weighted average of precision and recall calculated globally from total true positives, false negatives, and false positives.
3.3 Details
The experiments specify dataset-specific model configurations, optimization and regularization procedures, inference settings, and Table 2’s comparison conventions.
- RCV1-V2 uses a 50,000-word vocabulary, 500-word document truncation, beam size 5, 512-dimensional embeddings, and encoder and decoder hidden sizes of 256 and 512.
- AAPD uses 256-dimensional embeddings, encoder size 256, decoder size 512, a 30,000-word vocabulary, 500-word truncation, and beam size 9.
- Training minimizes cross-entropy with Adam using learning rate α = 0.001, β1 = 0.9, β2 = 0.999, and ϵ = 1 × 10−8.
- Table 2 compares methods on RCV1-V2 and AAPD using HL, P, R, and F1; “+” means higher is better and “-” means the opposite.
- Dropout and gradient clipping with maximum norm 10.0 are used, and the final model is selected by validation micro-F1.
3.4 Baselines
The study compares SGM with transformation-based and neural-network baselines, using established implementations and validation-based hyperparameter tuning.
- BR decomposes multi-label classification into independent binary problems, while CC models label correlations through a binary-classifier chain.
- LP transforms multi-label classification into multi-class classification over unique label combinations.
- CNN extracts text features with convolution kernels before sigmoid-based label prediction and multi-label soft-margin optimization.
- CNN-RNN combines convolutional and recurrent networks to capture global and local textual semantics and model label correlations.
- BR, CC, and LP use linear SVM base classifiers, while CNN and CNN-RNN hyperparameters are tuned with validation procedures.All baseline hyperparameters are tuned on the validation set using micro-F1.
3.5 Results
SGM is evaluated against all baselines on test sets, achieving strong results on RCV1-V2 and AAPD, with global embedding providing additional gains.
- RCV1-V2: 12.79% hamming-loss reduction and 2.33% micro-F1 improvement over BR are achieved by SGM with global embedding on RCV1-V2.
- RCV1-V2: 15.73% hamming-loss reduction and 2.69% micro-F1 improvement over CNN are achieved by SGM with global embedding on RCV1-V2.
- RCV1-V2: SGM without global embedding still outperforms all baselines on RCV1-V2.
- Global embedding: 7.41% hamming-loss reduction and 1.04% micro-F1 improvement are achieved by global embedding over SGM without global embedding on RCV1-V2.
- AAPD: SGM methods outperform all baselines on AAPD, while global embedding yields a 2.39% hamming-loss reduction and 1.57% micro-F1 improvement.
3.6 Analysis and Discussion
Analyses examine global embedding, decoder components, label-sequence length, attention, and generated sequences to explain SGM’s behavior and performance.
- Global embedding: Global embedding combines the original embedding with a weighted-average embedding through a transform gate to enrich decoder information.
- Global embedding: The proportion of weighted-average embedding is controlled by λ, with λ = 0 representing SGM without global embedding.
- Global embedding: The adaptive gate performs best across hamming loss and micro-F1, automatically selecting an appropriate λ according to the condition.
- Ablation study: Randomly shuffling label sequences causes a larger performance decline than removing the mask module, especially with global embedding.
- Label-sequence length: Performance deteriorates as label-sequence length increases, but SGM outperforms BR at every tested length and gains more as length grows.
- Attention analysis: Attention assigns larger weights to informative words that differ by predicted label, such as image and visual for CV and sentence and memory for CL.
- Case study: SGM captures correlated labels through sequential LSTM generation, while global embedding further improves generated label sequences.
4 Related Work
Related work organizes multi-label classification methods into problem transformation, algorithm adaptation, ensemble, and neural-network approaches.
- Problem transformation: Problem transformation methods convert multi-label learning into single-label or multi-class problems, including BR, LP, and CC.
- Algorithm adaptation: Algorithm adaptation methods modify learning algorithms to handle multi-label data directly, including decision trees, ranking methods, and maximum-entropy models.
- Ensemble methods: Ensemble methods partition labels into subsets or construct label co-occurrence graphs before training corresponding classifiers or detecting communities.
- Neural-network models: Neural-network approaches include fully connected, cross-entropy, clustering, CNN, and other architectures for capturing semantics or label correlations.
5 Conclusions and Future Work
The paper frames multi-label classification as sequence generation to model label correlations, and reports substantial gains over baselines. Its methods also automatically select informative words for different labels, while accurately predicting many labels remains difficult.
- The proposed methods view multi-label classification as sequence generation to model correlations between labels.
- The sequence generation methods outperform baselines by a substantial margin in extensive experiments.
- Further analysis shows that the methods capture label correlations and automatically select informative words for different labels.
- Accurately predicting all true labels becomes intractable when a sample has many labels.The proposed methods alleviate this difficulty to some extent, but more effective solutions remain future work.