Source-linked AI summary
Self-Taught Convolutional Neural Networks for Short Text Clustering
Jiaming Xu, Bo Xu, Peng Wang, Suncong Zheng, Guanhua Tian, Jun Zhao, Bo Xu
TL;DR
Short text clustering is challenged by sparse representations and limited textual information. STC2 addresses this with an unsupervised self-taught CNN framework that combines dimensionality reduction, word embeddings, and K-means, and the paper reports excellent performance across three short-text datasets.
Problem
Short text clustering is difficult because sparse texts often contain words occurring only once, while Bag-of-Words representations cannot preserve accurate textual information.
Method
STC2 embeds raw text features into compact binary codes, trains a CNN on word-embedding features to fit those codes, and applies K-means to the learned representations.
Results
The experiments on three short-text datasets report excellent performance in accuracy and normalized mutual information.
Takeaways & Limitations
STC2 provides a flexible unsupervised framework for learning non-biased deep representations without external tags, labels, or complicated NLP preprocessing.
Takeaways & Limitations
Existing short-text clustering methods may require solid NLP knowledge and still use high-dimensional representations.
Abstract
from arXiv · showhide
Short text clustering is a challenging problem due to its sparseness of text representation. Here we propose a flexible Self-Taught Convolutional neural network framework for Short Text Clustering (dubbed STC^2), which can flexibly and successfully incorporate more useful semantic features and learn non-biased deep text representation in an unsupervised manner. In our framework, the original raw text features are firstly embedded into compact binary codes by using one existing unsupervised dimensionality reduction methods. Then, word embeddings are explored and fed into convolutional neural networks to learn deep feature representations, meanwhile the output units are used to fit the pre-trained binary codes in the training process. Finally, we get the optimal clusters by employing K-means to cluster the learned representations. Extensive experimental results demonstrate that the proposed framework is effective, flexible and outperform several popular clustering methods when tested on three public short text datasets.
1. Introduction
Short text clustering is difficult because sparse, high-dimensional representations lose useful information, while existing neural approaches can be computationally costly or biased. STC2 combines unsupervised dimensionality reduction, word embeddings, CNN-based representation learning, and K-means clustering without external labels.
- Motivation and challenges: Short texts are sparse because most words occur only once, limiting TF-IDF and other high-dimensional representations.These representations may waste memory and computation time while failing to preserve accurate textual information.
- Motivation and challenges: Bag-of-Words methods are shallow and cannot preserve accurate representations of short texts.The paper identifies representation quality as a central limitation of directly training clustering methods on BoW features.
- Motivation and challenges: RNN representations are biased toward later words, whereas CNNs use convolutional filters to capture local features without that directional dominance.RecNNs also have high time complexity because they construct textual trees.
- Proposed framework: STC2 first embeds raw text features into compact binary codes, then trains a CNN on word-embedding matrices to learn deep representations that fit those codes.The framework uses the CNN output to fit pseudo labels produced by unsupervised dimensionality reduction.
- Proposed framework: K-means clusters texts using the learned deep feature representations, completing the STC2 pipeline.The framework combines a CNN, an unsupervised dimensionality-reduction function, and a K-means module.
- Contributions and evaluation: Experiments on three short-text datasets report excellent performance in both accuracy and normalized mutual information.The authors describe STC2 as flexible, non-biased, and usable without external tags, labels, or complicated NLP preprocessing.
2. Related Work
Related work addresses short-text sparsity through context enrichment, dimensionality reduction, and neural representations. However, existing approaches face limitations in semantic capture, computational cost, representation bias, or supervision requirements.
- Short text clustering: Short-text clustering methods enrich representations with external features or map original features into reduced spaces to address sparsity.Examples include Wikipedia and ontology features, as well as LSA, Laplacian Eigenmaps, and Locality Preserving Indexing.
- Short text clustering: Many clustering methods ignore word order and use shallow structures that cannot fully capture accurate semantic representations.
- Deep neural networks: Word embeddings support neural models for learning syntactic and semantic regularities, including Word2vec, GloVe, RecNN, and RNN.Word2vec uses single-layer architectures based on inner products between word vectors, while GloVe captures global information.
- Deep neural networks: Existing neural approaches have trade-offs: Para2vec is shallow and corpus-hungry, RecNN has high tree-construction complexity, and RNN representations are biased toward later words.
- Deep neural networks: Unsupervised representation learning includes Skip-thought encoders and methods that learn reconstruction-oriented codes, while this paper proposes a flexible self-taught CNN framework.The proposed framework couples semantic features and reports good performance on an unsupervised learning task.
3. Methodology
STC^2 combines unsupervised dimensionality reduction, a deep convolutional neural network, and K-means to cluster short texts from learned representations. The framework uses word embeddings and binary codes to guide unsupervised learning of deep semantic features.
- Framework overview: STC^2 comprises a deep CNN, an unsupervised dimensionality-reduction function, and a K-means clustering module.The framework is designed to learn deep feature representations before clustering the texts.
- Deep convolutional network: The input text is mapped to a matrix through word embeddings before passing through wide convolution, folding, and dynamic k-max pooling layers.These operations produce a deep representation from the embedded text.
- Deep convolutional network: Dynamic k-max pooling selects k highest values per row, with k varying by convolutional layer to extract higher-order and longer-range features.The layer-wise pooling parameter is computed from the topmost pooling parameter and the total number of convolutional layers.
- Unsupervised dimensionality reduction: Four dimensionality-reduction examples are considered: LSA, Laplacian Eigenmaps, Average Embedding, and Locality Preserving Indexing.LSA uses SVD, Laplacian Eigenmaps model manifold structure, Average Embedding averages word embeddings, and LPI extends LE to unseen texts.
- Unsupervised guidance: The framework converts reduced real-valued features into binary codes using the median of each dimension, then trains CNN outputs to fit those codes.The output vector is fitted through q logistic operations while optimizing the network parameters.
- Clustering: After training, K-means clusters the short texts using their learned deep semantic representations.The trained network first produces representations h, which are then supplied to K-means.
4. Experiments
Experiments evaluate STC2 and baseline clustering methods on three public short-text datasets using ACC, NMI, and complementary visualizations. Results show that STC2-LE and STC2-LPI consistently outperform the compared BoW- and RecNN-based approaches across all three datasets.
- Datasets and embeddings: Three public short-text datasets are evaluated, with dataset topics, preprocessing choices, and word-embedding coverage documented before clustering comparisons.SearchSnippets, StackOverflow, and Biomedical use different source corpora and preprocessing decisions; word vectors are trained from domain-relevant text or external corpora.
- Baselines: The study compares STC2 variants with K-means, Skip-thought, Recursive Neural Network, Paragraph Vector, dimensionality-reduction, and non-biased neural-network baselines.K-means uses TF or TF-IDF weighted keyword features, while Skip-thought provides several encoder configurations and K-means clusters the resulting representations.
- Results: RecNN (Top) can perform worse than K-means (TF-IDF), whereas averaging vectors across tree nodes can better represent sentence-level semantics.The paper attributes the weaker top-node result to a biased semantic representation and describes RecNN (Ave.) as potentially better for sentence-level semantics.
- Results: STC2-LE and STC2-LPI outperform BoW- and RecNN-based approaches across all three datasets, while STC2-AE and STC2-LSA show performance similar to several RecNN and Para2vec methods.The reported comparisons use ACC and NMI results from Tables 4 and 5.
- Non-biased models: STC2 generally outperforms bi-LSTM and bi-GRU clustering models, although those models both improve substantially over LPI and show no clear winner against each other.The comparison is reported for non-biased neural-network models in Tables 6 and 7.
- Feature weighting and visualization: TF-IDF improves K-means more than TF, while TF performs better than TF-IDF for Average Embedding; t-SNE shows clearer topic margins for STC2 variants.The visualization is presented as complementary to ACC and NMI, and STC2-AE, STC2-LSA, STC2-LE, and STC2-LPI show clearer topic separation than corresponding baselines.
5. Conclusions
The paper presents STC2 as a self-taught convolutional framework for learning deep representations for short-text clustering. It supports different dimensionality-reduction inputs, avoids external labels and complicated NLP preprocessing, and shows significantly better performance on three short-text datasets; selecting more effective semantic features remains future work.
- STC2 clusters short texts using deep feature representations learned through self-taught convolutional neural networks.
- The framework can incorporate traditional dimensionality-reduction approaches and operate without external tags or labels and complicated NLP preprocessing.
- Experiments on three short-text datasets show that the approach achieves significantly better performance.
- More research is needed to select and incorporate more effective semantic features into the framework.