Source-linked AI summary
Weakly-Supervised Neural Text Classification
Yu Meng, Jiaming Shen, Chao Zhang, Jiawei Han
TL;DR
Neural text classification is constrained by the scarcity and expense of labeled training data, while existing weakly supervised methods have limited applicability to deep neural models or supervision types. WeSTClass addresses this gap by generating pseudo-documents from seed information and refining CNN- or RNN-based models through self-training on real unlabeled data. Experiments on three real-world datasets show strong performance without excessive training data and significant improvements over baselines.
Problem
Neural text classification often requires expensive, large-scale labeled datasets, while existing weakly supervised methods have limited support for deep neural models and supervision types.
Method
WeSTClass generates pseudo-documents from seed information for pre-training, then iteratively refines CNN- or RNN-based models using high-confidence predictions on real unlabeled documents.
Results
Across three real-world datasets, WeSTClass achieves inspiring text-classification performance without excessive training data and significantly outperforms baseline methods.
Takeaways & Limitations
WeSTClass flexibly incorporates class names, class-related keywords, or labeled documents and supports CNN- and RNN-based neural classifiers.
Takeaways & Limitations
The paper identifies combining different types of seed information as an important direction for future work.
Abstract
from arXiv · showhide
Deep neural networks are gaining increasing popularity for the classic text classification task, due to their strong expressive power and less requirement for feature engineering. Despite such attractiveness, neural text classification models suffer from the lack of training data in many real-world applications. Although many semi-supervised and weakly-supervised text classification models exist, they cannot be easily applied to deep neural models and meanwhile support limited supervision types. In this paper, we propose a weakly-supervised method that addresses the lack of training data in neural text classification. Our method consists of two modules: (1) a pseudo-document generator that leverages seed information to generate pseudo-labeled documents for model pre-training, and (2) a self-training module that bootstraps on real unlabeled data for model refinement. Our method has the flexibility to handle different types of weak supervision and can be easily integrated into existing deep neural models for text classification. We have performed extensive experiments on three real-world datasets from different domains. The results demonstrate that our proposed method achieves inspiring performance without requiring excessive training data and outperforms baseline methods significantly.
1 INTRODUCTION
Weakly supervised text classification addresses the label scarcity that limits neural text classifiers by using seed information instead of extensive labeled data. WeSTClass combines pseudo-document generation with self-training and performs strongly across three real-world datasets.
- Training deep neural text classifiers can consume millions of labeled documents, making data collection expensive in practical scenarios.
- Weak supervision can provide class keywords, a few labeled documents, or class names when extensive annotation is unavailable.
- WeSTClass generates pseudo-labeled documents from seed information for neural-model pre-training and flexibly supports multiple seed types.
- Its self-training module uses the initial model to iteratively exploit high-confidence predictions on real unlabeled documents for refinement.
- WeSTClass supports CNN- and RNN-based models and is designed to integrate with existing neural architectures.
- Across three real-world datasets, the method achieves strong performance without excessive training data and outperforms various baselines.
2 RELATED WORK
Prior weakly supervised text-classification methods largely use latent-variable or embedding-based approaches. Latent-variable methods incorporate seeds into probabilistic topic models, while embedding-based methods classify using semantic representations.
- Weakly supervised text-classification studies are grouped into latent-variable models and embedding-based models.
- 2.1 Latent Variable Models: Latent-variable methods extend topic models by incorporating seed information, priors, constraints, or category descriptions to induce category-aware topics.
- The paper distinguishes WeSTClass from these approaches as a weakly supervised neural model that can use seed information within deep classifiers.
- 2.2 Embedding-based Models: Embedding-based methods derive vector representations for documents and label names, then classify using semantic similarity or learned classifiers.
3 PRELIMINARIES
This section formulates weakly-supervised text classification and outlines WeSTClass, which uses seed information to generate pseudo documents and refine neural models with unlabeled data.
- Problem Formulation: Weakly-supervised text classification assigns class labels to documents using label names, class-related keywords, or a small set of labeled documents.
- Method Overview: The method targets neural text classification, where limited labeled data remains a bottleneck despite neural models’ expressive power and reduced feature-engineering requirements.
- Method Overview: WeSTClass combines a pseudo-document generator with self-training to initialize and iteratively refine neural models using weak supervision and unlabeled real documents.
- Method Overview: The framework models classes in a shared semantic space and supports multiple supervision sources while integrating with CNN-based or RNN-based neural models.
4 PSEUDO DOCUMENT GENERATION
The pseudo-document generator represents words and documents in a shared semantic space, models each class with a spherical distribution, and samples diverse class-correlated documents from it.
- Modeling Class Distribution: Word embeddings are learned with Skip-Gram and normalized onto a unit sphere, forming the joint semantic space for words and pseudo-document vectors.
- Modeling Class Distribution: Seed information is converted into class-related keywords using label-name embeddings, keyword similarities, or tf-idf keywords extracted from labeled documents.
- Modeling Class Distribution: Each class is modeled with a von Mises Fisher distribution whose mean direction represents semantic focus and whose concentration parameter controls dispersion.
- Generating Pseudo Documents: Each document term is sampled from a mixture of background and class-specific distributions, controlled by α, to produce class-correlated pseudo text.
- Generating Pseudo Documents: For each pseudo document, a vector is sampled from the class distribution and a vocabulary of the top-γ similar words is constructed.
- Generating Pseudo Documents: Sampling document vectors rather than fixing them to the class mean produces more semantically diversified pseudo documents for broad classes.
5 NEURAL MODELS WITH SELF-TRAINING
WeSTClass pre-trains CNN or RNN models on pseudo documents, then self-trains on real unlabeled documents using iterative high-confidence pseudo-label refinement.
- Neural Model Self-training: The self-training module first pre-trains a deep neural model on generated pseudo documents and then iteratively refines it using unlabeled real documents.
- Neural Model Pre-training: Naive one-hot labels can make the neural model overfit pseudo documents and perform poorly on real documents.
- Neural Model Pre-training: Because pseudo documents lack word-order information, the method creates soft pseudo labels by distributing background-distribution mass across classes.
- Neural Model Self-training: During self-training, current predictions generate pseudo labels, and model parameters are optimized with KL divergence until class assignments stabilize.
- Neural Model Self-training: In pre-training, pseudo labels supervise generated documents, whereas self-training pairs labels with every unlabeled real document and reflects current high-confidence predictions.
- Neural Model Architectures: CNN instantiations use convolution and max-over-time pooling, while HAN-based RNN instantiations use word- and sentence-level attention for document representations.
6 EXPERIMENTS
The evaluation measures weakly supervised text classification across three domains and compares WeSTClass against retrieval, topic-model, neural, and self-training baselines.
- Datasets: The experiments evaluate performance on New York Times news, AG’s News, and Yelp Review corpora from different domains.The AG’s News corpus contains 120,000 documents evenly distributed across four classes.
- Baselines: The comparison includes tf-idf retrieval, topic modeling, latent-variable approaches, neural models, and a no-self-training variant.
- Baselines: The full WeSTClass-CNN and WeSTClass-HAN systems enable comparison of the proposed framework with both CNN- and RNN-based neural architectures.
6.3 Experiment Settings
The experiments use three neural text-classification settings and evaluate WeSTClass under multiple weak-supervision sources across three datasets.
- Experiments use 100-dimensional Skip-Gram word embeddings trained separately on each dataset corpus.
- The framework is tested with CNN and HAN neural text-classification models, using SGD with batch size 256.
- Weak supervision consists of label surface names, three class-related keywords per class, or randomly sampled labeled documents.
6.4 Experiment Results
Across three datasets and weak-supervision sources, WeSTClass generally outperforms the baselines, with gains associated with both pseudo-document pre-training and self-training.
- Overall performance: WeSTClass achieves the overall best performance among baselines across three datasets and different weak-supervision sources, using macro-F1 and micro-F1.
- Overall performance: WeSTClass-CNN usually performs best, while WeSTClass-HAN performs slightly worse but still exceeds the other baselines.
- Overall performance: With label names or related keywords, both WeSTClass variants outperform their CNN and HAN counterparts trained with alternative pseudo-labeling approaches.
- Self-training effect: WeSTClass-CNN and WeSTClass-HAN outperform NoST-CNN and NoST-HAN, respectively, indicating the contribution of self-training.
- Self-training effect: Self-training generally improves performance after pre-training, but its effect is smallest when labeled documents are included during pre-training.Using pseudo documents together with labeled documents can yield better initialization, leaving less room for self-training gains.
- Labeled-document variation: As labeled documents per class vary, WeSTClass-based models, especially WeSTClass-CNN, remain stable while PTE, CNN, and HAN show performance drops and greater seed sensitivity.
6.5 Parameter Study
Parameter studies on AG’s News examine the background-word weight, pseudo-document quantity, and keyword-vocabulary size, revealing trade-offs in informativeness, generalization, and pre-training cost.
- The parameter study varies background-word weight α, pseudo-document amount β, and keyword-vocabulary size γ for WeSTClass with CNN and HAN models.
- Background-word weight: When α approaches 1, generated pseudo documents and pseudo-labels become uninformative; when α approaches 0, performance is slightly worse because overfitting becomes easier.
- Pseudo-document amount: Generating too few pseudo documents provides insufficient pre-training information, whereas generating too many unnecessarily lengthens pre-training.
- Pseudo-document amount: Generating 500 to 1000 pseudo documents per class balances pre-training quality against training time.
- Keyword vocabulary size: The keyword vocabulary size γ controls how many unique words appear frequently in generated pseudo documents, affecting pre-training generalization.
6.6 Case Study
Case studies examine sensitivity to seed-keyword selection and illustrate how self-training can correct an initially incorrect prediction on a real document.
- Self-training behavior: Figure 7 combines performance under different seed-keyword sets with class-prediction probabilities during self-training for one sample document.
- Seed-keyword sensitivity: The keyword-sensitivity study evaluates WeSTClass-CNN and WeSTClass-HAN under different seed-keyword sets on AG’s News.The study compares manually selected keywords with lists derived from average tf-idf rankings.
- Seed-keyword sensitivity: The seed-keyword lists are formed by ranking words according to their average tf-idf weights within each class.
- Self-training behavior: A sample AG’s News document is initially misclassified after pre-training, then examined during the self-training procedure.
7 DISCUSSIONS AND CONCLUSIONS
WeSTClass addresses neural text classification’s labeled-data bottleneck with pseudo-document generation and self-training on unlabeled data. It accommodates multiple weak-supervision sources and neural architectures, while experiments show significant baseline gains and robustness; integrating seed types remains future work.
- WeSTClass combines pseudo-document generation with self-training on real unlabeled data to refine neural text classifiers.The generator creates pseudo training data from seed information, while self-training bootstraps on unlabeled documents.
- The method supports class names, class-related keywords, and labeled documents as weak supervision.
- WeSTClass is generic enough to support both CNN and RNN neural models.
- The method significantly outperforms baseline methods and remains robust across hyperparameter settings and user-provided seed types.
- Different weak-supervision types are all highly helpful for neural-model performance, while integrating them could further improve WeSTClass.