Source-linked AI summary
An Analysis of Hierarchical Text Classification Using Word Embeddings
Roger A. Stein, Patricia A. Jaques, Joao F. Valiati
TL;DR
Many real-world classification problems involve numerous similar categories organized in hierarchies, but the effectiveness of improved text-classification techniques had not been established for HTC. This study experimentally evaluates word embeddings, classification models, and hierarchical strategies on HTC, finding that hierarchical approaches outperform flat approaches and that FastText performs especially strongly.
Problem
The study asks whether techniques that improve typical text classification can positively impact hierarchical text classification, which organizes many similar categories into taxonomies.
Method
The study experimentally evaluates GloVe, word2vec, and fastText embeddings with fastText, XGBoost, and CNN classifiers on RCV1, including LCPN and virtual-category strategies.
Results
Hierarchical models using LCPN with virtual categories surpassed flat approaches in all equivalent comparisons, while FastText was the outstanding classifier and also performed very well as an embedding generator.
Takeaways & Limitations
Flat measures are inadequate for HTC, and task-specific word embeddings combined with suitable classifiers are promising for hierarchical text classification.
Takeaways & Limitations
Comparisons with prior methods are limited because previous studies generally used different measures, datasets, or RCV1 label versions, and HTC lacks a widely used reference dataset.
Abstract
from arXiv · showhide
Efficient distributed numerical word representation models (word embeddings) combined with modern machine learning algorithms have recently yielded considerable improvement on automatic document classification tasks. However, the effectiveness of such techniques has not been assessed for the hierarchical text classification (HTC) yet. This study investigates the application of those models and algorithms on this specific problem by means of experimentation and analysis. We trained classification models with prominent machine learning algorithm implementations---fastText, XGBoost, SVM, and Keras' CNN---and noticeable word embeddings generation methods---GloVe, word2vec, and fastText---with publicly available data and evaluated them with measures specifically appropriate for the hierarchical context. FastText achieved an ${}_{LCA}F_1$ of 0.893 on a single-labeled version of the RCV1 dataset. An analysis indicates that using word embeddings and its flavors is a very promising approach for HTC.
1. Introduction
Hierarchical text classification addresses large, closely related document categories organized in taxonomies. This study tests whether modern word representations and classification algorithms improve HTC and whether hierarchical measures better assess effectiveness.
- Motivation: Large real-world classification problems often contain many similar categories organized into a hierarchy, creating challenges beyond ordinary text classification.HTC applies classification to repositories such as web directories, libraries, patents, and medical systems.
- Motivation: Efficient models are needed because HTC systems must exploit available training data and be induced within reasonable time while classifying closely related categories accurately.
- Recent techniques: Recent progress in text representation and machine learning includes word embeddings, softmax-based linear classifiers, scalable tree boosting, and neural-network variants.
- Study objectives: The study investigates whether GloVe, word2vec, fastText, XGBoost, and Keras’ CNN improve HTC through empirical experimentation and analysis.
- Study objectives: It also compares traditional flat F1 with hierarchical measures such as hF1 and lcaF1 to examine how evaluation should reflect the hierarchy.
2. Hierarchical Text Classification
HTC classifies documents within a hierarchy, whose structure can be incorporated through flat, global, or local strategies. Evaluation measures extend beyond ordinary precision, recall, and F1 to account for hierarchical relationships and misclassification distance.
- HTC concepts: Hierarchical classification uses a class hierarchy during learning or prediction when a multiclass task contains many categories.
- HTC strategies: Flat approaches ignore the hierarchy, global approaches train one hierarchy-aware classifier, and local approaches build classifiers from node-specific information.
- HTC strategies: Local strategies include LCN classifiers per child node, LCPN classifiers per parent node, and LCL classifiers per hierarchy level.
- HTC strategies: Top-down local prediction proceeds from upper levels toward candidate descendants until reaching a leaf or stopping under a non-mandatory leaf-node criterion.
- Evaluation: Traditional measures are inappropriate for HTC because they ignore parent-child and sibling relationships and treat nearby and distant errors similarly.
- Evaluation: LCA-based measures use the lowest common ancestor to calculate lcaP, lcaR, and lcaF1, avoiding excessive penalties associated with full ancestry-based measures.
3. Text Representation
Text representation converts documents into numerical forms for classification. The section contrasts high-dimensional bag-of-words representations with distributed representations built from lower-dimensional semantic vectors and word embeddings.
- Representation approaches: Document-term matrices represent corpus elements by rows and dictionary tokens by columns, while other approaches represent documents as fixed-size vectors or topic distributions.
- Bag of words: Bag-of-words represents text as a vector whose elements are word-specific weights, commonly derived from term frequency and inverse document frequency.
- Bag of words: BoW can be efficient for text mining but has very high dimensionality and ignores word order.
- Distributed representations: Distributed representations use an arbitrary, typically much smaller number of dimensions corresponding to semantic concepts.
- Word embeddings: Word2vec uses predictive models, GloVe uses count-based modeling, and fastText can also generate word embeddings.
- Word embeddings: Classification algorithms use word vectors directly or in combination, while the advantages and disadvantages of modern representations remain open.
4. Classification Models
The paper considers linear, tree-boosting, and neural classification models for text data. These models differ in how they transform features, combine predictors, and represent or process inputs.
- Classification models: Classification induces a model from training tuples and uses it to predict the class of previously unseen tuples.
- Linear classifiers: Linear classifiers assign class membership by comparing a weighted feature combination against a threshold and can use one-versus-all strategies for multiple classes.
- Linear classifiers: FastText averages word embeddings into document vectors and trains linear classifiers with hierarchical softmax in supervised mode.
- Gradient tree boosting: XGBoost implements gradient boosting by adding weak regression trees iteratively while combining parallelism, sparsity awareness, and cached data access.
- Neural networks: Neural networks consist of weighted units arranged as directed acyclic graphs, commonly with input, hidden, and output layers.
- Neural networks: CNN is a feed-forward neural-network specialization that uses convolutional operations and typically includes convolution, detection, and pooling stages.
5. Related Works
Prior HTC research spans Bayesian, SVM, neural-network, boosting, and other methods, but comparisons remain difficult because datasets and evaluation measures lack consensus. Related text-classification work suggests word embeddings and neural architectures are promising, while HTC-specific evidence remains limited.
- HTC research: TreeBoost.MH substantially surpassed its flat counterpart, especially for highly unbalanced classes, but remained inferior to SVM models under reservations about the comparison.The method was evaluated on Reuters-21578, RCV1, and ICCCFT datasets.
- Distributed text representations: Word embeddings improved related flat text-classification systems, including a reported 3.3% accuracy gain over TF-IDF and LDA-based approaches.The cited Gaussian-process approach modeled word-embedding distributions according to themes.
- Neural networks: CNNs with distributed representations produced competitive results across multiple NLP and text-classification tasks, including experiments using pre-trained word2vec embeddings and multiple channels.Earlier CNN work exceeded benchmark systems in all but semantic role labeling, where the margin was narrow.
- Discussion and considerations: HTC comparisons are impeded by the absence of a widely accepted reference dataset and evaluation measure, making cross-study and flat-versus-hierarchical comparisons problematic.The review characterizes direct flat-versus-hierarchical comparison as inadequate and inaccurate because the problems differ inherently.
- HTC research: HTC research has applied Bayesian models, SVM, neural networks, boosting, Rocchio, and kNN, without a consistent effectiveness increase or breakthrough over two decades.The surveyed methods are representative rather than exhaustive.
- Discussion and considerations: The reviewed literature indicates that appropriate word embeddings combined with suitable neural-network classifiers provide a promising direction for HTC.This conclusion connects advances in related NLP classification tasks to the present investigation.
6. Experiments and Analysis
The experiments combine RCV1 with word embeddings, several classifiers, and flat or hierarchical strategies to assess HTC performance. Results show that hierarchical evaluation and LCPN+VC materially change model comparisons, while fastText is the strongest studied classifier.
- Experimental setup: The study transforms publicly available HTC data and trains models using fastText, XGBoost, CNN, and baseline SVM classifiers with multiple word-embedding representations.The representations include pre-trained word2vec and GloVe vectors plus supervised fastText embeddings.
- Dataset: The data preparation converts XML to text, retains each document’s least frequent category for single-label multi-class experiments, and applies lowercasing and punctuation removal.The least common label is treated as the category that more specifically identifies the document.
- Results and analysis: 0.823 average lcaF1 versus 0.533 average flat F1 shows that flat evaluation can substantially understate performance in the hierarchical setting.XGBoost with word2vec and LCPN+VC illustrates the discrepancy: flat F1 ranks it highest at 0.716, while lcaF1 ranks it third at 0.870.
- Results and analysis: LCPN+VC consistently achieves higher lcaF1 than corresponding flat models, although the size of the improvement varies across classifiers.XGBoost shows a particularly pronounced increase, while the reason for classifier-specific differences remains unresolved.
- Results and analysis: FastText exceeds the other studied classifiers even under the flat strategy, while word2vec embeddings hold a slight advantage over GloVe’s.Supervised fastText embeddings also yield fair results for SVM and XGBoost despite being generated from relatively little data.
7. Conclusion
The study evaluates distributed text representations, modern classifiers, and hierarchical modeling strategies for HTC, finding that hierarchical approaches and task-specific methods are especially promising. Experiments on RCV1 also expose evaluation and scope boundaries that motivate further work.
- Approach: The study combines GloVe, word2vec, and fastText embeddings with fastText, XGBoost, and CNN models for HTC experiments.It also exploits the hierarchy through LCPN and virtual categories.
- Results: Hierarchical models using LCPN with a virtual category surpass flat approaches in all experimented equivalent comparisons.
- Evaluation: The RCV1 experiments confirm that flat measures are inadequate for HTC and report a strong correlation between hierarchical and LCA measures, presumably because the hierarchy is shallow.
- Results: FastText is the outstanding classifier and a very good word-embedding generator, with lcaF1 of 0.893 on single-labeled RCV1.The authors attribute much of its superiority to supervised estimation of class-oriented word embeddings.
- Future Work: Future work targets validation on larger medical hierarchies, additional text representations and sequence models, and training objectives approximating hF1 or lcaF1.The authors specifically propose PubMed or BioASQ experiments, further CNN and LSTM investigation, and differentiable hierarchical losses.