Source-linked AI summary
Joint Embedding of Words and Labels for Text Classification
Guoyin Wang, Chunyuan Li, Wenlin Wang, Yizhe Zhang, Dinghan Shen, Xinyuan Zhang, Ricardo Henao, Lawrence Carin
TL;DR
Text classification traditionally uses word representations while label information enters mainly at classification, leaving label-aware representation learning limited. The paper proposes LEAM, which jointly embeds words and labels and uses their compatibility for attention; experiments report lower computational cost with better or comparable state-of-the-art performance and interpretable word highlighting.
Problem
Text classification needs effective text representations, but traditional pipelines use label information mainly in the final classification step rather than throughout representation learning.
Method
LEAM jointly embeds words and labels in one latent space and constructs text representations directly from word-label compatibility.
Results
LEAM achieves better or comparable state-of-the-art performance while requiring much lower computational cost across benchmark and clinical text tasks.
Takeaways & Limitations
The learned attention highlights informative words for downstream classification, supporting interpretation of predictions.
Abstract
from arXiv · showhide
Word embeddings are effective intermediate representations for capturing semantic regularities between words, when learning the representations of text sequences. We propose to view text classification as a label-word joint embedding problem: each label is embedded in the same space with the word vectors. We introduce an attention framework that measures the compatibility of embeddings between text sequences and labels. The attention is learned on a training set of labeled samples to ensure that, given a text sequence, the relevant words are weighted higher than the irrelevant ones. Our method maintains the interpretability of word embeddings, and enjoys a built-in ability to leverage alternative sources of information, in addition to input text sequences. Extensive results on the several large text datasets show that the proposed framework outperforms the state-of-the-art methods by a large margin, in terms of both accuracy and speed.
1 Introduction
The paper motivates using label information directly in text representations while retaining the efficiency and interpretability of simple models. It proposes jointly embedding words and labels so attention can focus on task-relevant words.
- Text classification requires representations that capture textual content for assigning one or more labels.
- Neural CNN and RNN models learn text representations, while attention captures important dependencies but increases model complexity.
- Word-embedding effectiveness strongly influences text-classification success, motivating preference for simple models when difficulty is similar across architectures.
- The paper seeks to combine dependency-aware representation learning with low computational cost by incorporating label information into text-sequence representations.
- LEAM jointly embeds words and labels, constructs representations from text-label compatibility, and uses attention to highlight informative words.
2 Related Work
Related work establishes attention as a powerful text-representation mechanism and shows that simple attention architectures can be efficient. This paper differs by applying attention between words and labels rather than only between word tokens.
- Label embeddings have supported image classification, multimodal learning, text recognition, and zero-shot learning by capturing label correlations.
- Attention-based encoder-decoder models have been applied successfully to sentiment classification, sentence-pair modeling, and summarization.
- Hierarchical, attention-over-attention, and multi-step mechanisms extend attention models for text representation.
- Simple attention architectures can achieve state-of-the-art performance with less computational time, without recurrence or convolutions.
- Unlike self-attention methods that compare word-token pairs, this work investigates attention between words and labels.
3 Preliminaries
The preliminaries define text classification as learning a function from token sequences to labels by minimizing empirical risk. They describe word embeddings and the standard three-stage pipeline that represents, composes, and classifies text.
- A text-classification example pairs a text sequence X with a label y, represented as one-hot or binary vectors for single-label or multi-label tasks.
- The learning objective is to learn f : X 7→Y by minimizing empirical risk over paired training data.
- Word embeddings map dictionary one-hot vectors from ∆D into a P-dimensional space, producing a matrix of token representations.
- The conventional pipeline maps X to word embeddings V, aggregates V into a fixed-length vector z, and maps z to a label y.
- Word embeddings are often pretrained and then refined jointly with composition and classification functions for task-specific representations.
4 Label-Embedding Attentive Model
LEAM incorporates label information throughout text representation learning by jointly embedding words and labels, then using label-word compatibility to weight word aggregation. It supports single- and multi-label classification while retaining a comparatively simple, efficient architecture.
- Framework: Label information is incorporated throughout the pipeline by embedding words and labels together, rather than using labels only in the final classifier.The framework treats labels as class-specific anchor points that influence word and sequence representations.
- Joint Embeddings: LEAM embeds words and labels in a shared Euclidean space, with one label embedding for each of K classes.The label embeddings are represented as C = [c1, · · ·, cK].
- Compatibility and Attention: Label-word compatibility is measured with cosine similarity, then extended over local phrases to capture consecutive-word structure and nonlinear compatibility.The local block of the compatibility matrix represents label-to-token compatibility for a phrase centered at each position.
- Compatibility and Attention: The model derives a label-based attention score and constructs the text representation as a weighted average of word embeddings.The attention score is pooled from compatibility values and used to weight the word vectors.
- Relation to Prior Work: Compared with PTE, LEAM uses label-attentive weighted averaging and nonlinear phrase-level interactions, while PTE averages word embeddings with linear word-label interactions.The PTE representation corresponds to a special LEAM case with r = 1 and uniform attention.
- Training: LEAM is trained end-to-end for both single-label and multi-label classification, with task-specific output functions and jointly learned embedding and network parameters.Single-label prediction uses one of K classes, while multi-label prediction allows any subset of the K target labels.
- Complexity: LEAM uses fewer compositional parameters than CNN and LSTM models, has complexity close to SWEM, and can be substantially cheaper than CNN or LSTM.Its computational complexity is smaller than CNN or LSTM by a factor of mh/K or h/K under the stated assumptions.
5 Experimental Results
Experiments on benchmark and clinical datasets evaluate LEAM's accuracy, efficiency, representation quality, interpretability, and partial-label performance. LEAM generally matches or exceeds competing methods while using fewer parameters and converging faster than Bi-BloSAN.
- Benchmark datasets: LEAM is evaluated on five standard document-classification datasets spanning topic and sentiment classification.The datasets include AGNews, Yelp Review Full, Yelp Review Polarity, DBPedia, and Yahoo! Answers Topic.
- Benchmark classification: LEAM outperforms state-of-the-art methods on Yahoo and DBPedia, while ranking second or third on the other datasets with similar accuracy to the top method.The paper attributes weaker relative gains partly to fewer classes and the lack of explicit corresponding word embeddings for label initialization on some datasets.
- Efficiency: LEAM uses fewer model parameters and converges significantly faster than Bi-BloSAN while retaining the simplicity and low cost of SWEM.Runtime is reported as wall-clock time for 1000 iterations.
- Partial supervision: LEAM consistently outperforms other methods across different proportions of labeled data.The comparison is presented in Figure 2(b).
- Hyper-parameter analysis: The window size r controls the attention span: larger values capture long-term dependencies, whereas smaller values enforce local dependencies.Topic classification generally favors larger r, sentiment classification permits smaller r, and r around 50 is suggested without finetuning.
- Representational ability: Learned label embeddings align with their corresponding classes and support interpretable attention over informative words.High on-diagonal and low off-diagonal cosine similarities, class-centered t-SNE placement, and highlighted task-relevant words provide the reported evidence.
- Clinical application: On clinical multi-label code prediction, LEAM achieves the best AUC and better F1 and P@5 values than all methods except CNN.The evaluation uses micro- and macro-averaged F1, AUC, and precision at n.
6 Conclusions
LEAM jointly embeds words and labels, using their compatibility to attend over documents for text classification. It achieves better or comparable state-of-the-art performance at lower computational cost, while producing interpretable attention highlights.
- LEAM embeds words and labels in a shared space and uses word-label compatibility to construct attended document representations.
- The framework was evaluated on several large standard datasets and a real clinical text application.
- LEAM requires much lower computational cost while achieving better if not comparable performance relative to state-of-the-art methods.
- The learned attention highlights the words most informative for the downstream classification task.