PaperlayerView exact revision on arXiv

Source-linked AI summary

Weakly-Supervised Hierarchical Text Classification

Yu Meng, Jiaming Shen, Chao Zhang, Jiawei Han

arXiv:1812.11270v1cs.CLcs.AIcs.LG

TL;DR

Hierarchical text classification requires neural models to cope with limited labeled data and to choose an appropriate level in the class hierarchy. WeSHClass uses weak supervision to generate pseudo documents, then self-trains a hierarchy-aware classifier with blocking. It outperforms supervised and weakly supervised baselines across three datasets from different domains.

  • Problem

    Neural models for hierarchical text classification face limited labeled training data and difficulty determining whether a document belongs at an internal or leaf level.

    Source: Introduction · p. 1

  • Method

    WeSHClass uses weak supervision to generate pseudo documents for pre-training, then self-trains a hierarchy-aware global classifier that uses blocking during recursive prediction.

    Source: Introduction · pp. 1–2, Global Classifier Self-Training · p. 4

  • Results

    WeSHClass outperforms various supervised and weakly supervised baselines on three datasets from different domains.

    Source: Quantitative Comparision · p. 6, Conclusions · p. 8

  • Takeaways & Limitations

    The reported results demonstrate the practical value of WeSHClass in real-world applications.

    Source: Conclusions · p. 8

    The formulation assumes a tree taxonomy and weak supervision sources supplied only for leaf classes, with internal-class sources aggregated from descendants.

    Source: Problem Formulation · p. 2

Abstract

from arXiv · show

Hierarchical text classification, which aims to classify text documents into a given hierarchy, is an important task in many real-world applications. Recently, deep neural models are gaining increasing popularity for text classification due to their expressive power and minimum requirement for feature engineering. However, applying deep neural networks for hierarchical text classification remains challenging, because they heavily rely on a large amount of training data and meanwhile cannot easily determine appropriate levels of documents in the hierarchical setting. In this paper, we propose a weakly-supervised neural method for hierarchical text classification. Our method does not require a large amount of training data but requires only easy-to-provide weak supervision signals such as a few class-related documents or keywords. Our method effectively leverages such weak supervision signals to generate pseudo documents for model pre-training, and then performs self-training on real unlabeled data to iteratively refine the model. During the training process, our model features a hierarchical neural structure, which mimics the given hierarchy and is capable of determining the proper levels for documents with a blocking mechanism. Experiments on three datasets from different domains demonstrate the efficacy of our method compared with a comprehensive set of baselines.

Introduction

Hierarchical text classification must handle class relationships and documents that may belong at intermediate levels, while neural models face data demands and difficulty choosing granularity. WeSHClass uses weak supervision, pseudo-document pre-training, and hierarchical self-training with blocking.

Introduction

  • Neural hierarchical classification faces both a shortage of costly labeled training documents and difficulty selecting the appropriate level for each document.Documents may fit intermediate classes rather than leaves, and existing deep neural networks lack simple ways to determine the best granularity.Source: Introduction · p. 1
  • WeSHClass pre-trains local classifiers at hierarchy nodes, aggregates them into a global classifier through self-training, and uses blocking to avoid forcing general documents to leaf nodes.The global classifier predicts recursively in a top-down manner.Source: Introduction · p. 2
  • WeSHClass uses a few class-related documents or keywords instead of requiring large amounts of training data, and supports classification types such as topics and sentiments.The paper describes these weak supervision sources as easy for users to provide.Source: Introduction · pp. 1–2
  • The method generates pseudo training documents from weak supervision sources to address the training-data bottleneck, followed by self-training.The generated documents serve as pseudo training data together with the subsequent self-training step.Source: Introduction · p. 2
  • The paper evaluates WeSHClass on three real-world datasets from different domains and reports case studies of its components.The introduction identifies this evaluation as evidence of the method’s effectiveness.Source: Introduction · p. 2

Problem Formulation

The task assigns documents to categories in a tree, allowing either internal or leaf labels. Weak supervision is supplied for leaf classes and propagated upward to internal classes.

Problem Formulation

  • Users provide weak supervision for each leaf class as either related keywords or a small set of labeled documents.The formulation contrasts this with traditional supervised classification, which relies on many labeled documents for each class.Source: Problem Formulation · p. 2
  • Weak supervision for internal classes is formed by aggregating the sources of their descendant leaf classes.The user supplies sources only for leaf classes before upward propagation through the taxonomy.Source: Problem Formulation · p. 2
  • The task assigns each document its most likely category in a tree, where the category may be internal or a leaf.The hierarchy is tree-structured, with each category having at most one parent and an arbitrary number of children; prediction is non-mandatory at leaves.Source: Problem Formulation · p. 2

Pseudo Document Generation

WeSHClass models class semantics from weak supervision and uses the resulting distributions to seed language-model generation of pseudo documents. The approach retrieves class-related keywords, fits spherical distributions, and generates class-specific text.

Pseudo Document Generation

  • Pseudo-document generation uses user-provided weak supervision to model class distributions and then generate class-specific documents with a language model for model pre-training.The section presents this process as a way to address the shortage of abundant labeled data.Source: Pseudo Document Generation · p. 2

Modeling Class Distribution

  • Word embeddings are normalized onto a unit sphere, and class semantics are modeled with mixtures of von Mises Fisher distributions.Leaf classes use one component, while an internal class uses one component per child class to represent its semantics as a mixture of child semantics.Source: Modeling Class Distribution · pp. 2–3
  • The method retrieves keywords from seed keywords or labeled documents, fits class distributions with EM, and uses retrieval to smooth supervision and reduce sensitivity to its sources.For labeled documents, representative keywords are extracted with tf-idf weighting; the number retrieved is the largest that avoids shared words across classes.Source: Modeling Class Distribution · p. 3

Language Model Based Document Generation

  • An LSTM language model generates each pseudo document by starting with the closest word to a sampled class-distribution embedding and recursively predicting subsequent words.The class-distribution starting word makes the generated document correlated to that class; mixture modeling also gives child-class semantics a chance to appear.Source: Language Model Based Document Generation · p. 3

The Hierarchical Classification Model

The model pre-trains local classifiers on generated pseudo data, ensembles them for iterative self-training, and uses blocking to stop general documents at internal categories.

The Hierarchical Classification Model

Local Classifier Pre-Training

  • Local classifiers are initialized on β pseudo documents per class, using noise-aware soft pseudo-labels and KL-divergence training to limit overfitting to generated data.The parameter α accounts for noise in pseudo documents.Source: Local Classifier Pre-Training · p. 3

Global Classifier Self-Training

  • At each taxonomy level, local classifiers are ensembled by multiplying path probabilities, allowing lower levels to correct higher-level misclassifications.Source: Global Classifier Self-Training · p. 4
  • The global classifier iteratively refines itself on unlabeled documents using high-confidence pseudo-labels, fine-tuning the ensemble until fewer than δ% of assignments change.Source: Global Classifier Self-Training · p. 4

Blocking Mechanism

  • Normalized entropy identifies general documents for blocking at an internal class; when γ = 1, no documents are blocked and all reach leaf classes.Source: Blocking Mechanism · p. 4

Inference

  • At inference, the trained model outputs class probabilities at every hierarchy level and applies the same blocking mechanism to select the document’s level.Source: Inference · p. 4

Algorithm Summary

  • Training proceeds top-down: generate pseudo documents and labels, pre-train local classifiers, self-train the ensemble, then block documents before advancing.Source: Algorithm Summary · p. 4

Experiment Settings

Experiments use three datasets from distinct domains, compare WeSHClass with supervised and weakly supervised baselines, and test keyword- or document-level weak supervision. Settings use limited seeds and specify the model, training, and blocking parameters.

Datasets and Evaluation Metrics

  • Three corpora cover news, arXiv abstracts, and Yelp reviews; classification performance is measured with Micro-F1 and Macro-F1.NYT has 13,081 articles across 5 super-categories and 25 sub-categories; arXiv has 230,105 abstracts across 53 sub-categories; Yelp has 50,000 reviews grouped into 3 super-categories.Source: Datasets and Evaluation Metrics · p. 5

Baselines

  • Baselines include hierarchical dataless and SVM methods, CNN, WeSTClass, and WeSHClass variants without global classification, movMF semantics, or self-training.Hier-Dataless is tested top-down and bottom-up, with and without bootstrapping; Hier-SVM trains local classifiers among sibling categories.Source: Baselines · p. 5

Parameter Settings

  • The method uses 100-dimensional Skip-Gram embeddings, 500 pseudo documents per class, and blocking thresholds of 0.9 for NYT and 1 for the other datasets.The pseudo-label parameter is 0.2 and the self-training stopping criterion is 0.1; local classifiers use a one-layer CNN trained with SGD.Source: Parameter Settings · pp. 5–6

Weak Supervision Settings

  • Weak supervision consists of three keywords per leaf class or randomly sampled labeled documents: 3 per class for NYT and arXiv, and 10 for Yelp.Document sampling is repeated 10 times, with average and standard deviation reported.Source: Weak Supervision Settings · p. 6

Quantitative Comparision

WeSHClass achieves the best overall performance among the baselines across all three datasets. Its results also support the contributions of the local-classifier ensemble, movMF class semantics, and self-training.

Quantitative Comparision

  • WeSHClass achieves the overall best performance across three datasets, with keyword supervision outperforming Hier-Dataless and WeSTClass and document supervision improving average performance and stability over supervised baselines.With extremely limited training documents, it is less sensitive to seed-document selection.Source: Quantitative Comparision · p. 6
  • No-global, No-vMF, and No-self-train ablations show that the local-classifier ensemble, movMF class semantics, and self-training each contribute to WeSHClass performance.The supplied passages state these component findings but do not provide numerical scores.Source: Quantitative Comparision · p. 6

Component-Wise Evaluation

Component studies examine pseudo-document quality, joint self-training of local classifiers, and blocking dynamics. The reported findings favor the proposed generation strategy and global-classifier training, while blocking decreases as model uncertainty falls.

Component-Wise Evaluation

  • The breakdown experiments use NYT with class-related keywords as supervision, with similar results reported for the other two datasets.The experiments investigate components of the proposed method.Source: Component-Wise Evaluation · p. 6

Pseudo Documents Generation

  • The movMF-plus-LSTM strategy generates higher-quality pseudo documents than bag-of-words generation or bag-of-words with reordering, and yields faster convergence and better final self-training performance.The reordering approach produces high-quality text early but poor text near the end; similar component-study results are reported on the other two datasets.Source: Pseudo Documents Generation · p. 6
  • Joint self-training of the ensemble improves accuracy at super-category, sub-category, and all-class levels; greedy classification leaves higher-level classifiers unchanged during lower-level decisions.The text states that higher-level misclassifications cannot be corrected under the greedy approach.Source: Global Classifier and Self-training · p. 6

Blocking During Self-training

  • During NYT self-training, average normalized entropy decreases and fewer documents are blocked, leaving more documents available for self-training.Figure 2 is identified as a component-wise evaluation on NYT.Source: Blocking During Self-training · p. 6, Figure 2 · p. 7

Related Work

Prior weakly supervised text-classification methods use keywords, limited labeled data, or unlabeled data, but are not specifically designed for hierarchical classification. Hierarchical approaches include local classifiers, hierarchy-aware losses, graph-CNNs, and methods for large label spaces.

Weakly-Supervised Text Classification

  • Weakly supervised methods include WeSTClass, which pre-trains on pseudo documents and self-trains on unlabeled data, plus keyword-guided topic, constraint-based, and predictive-embedding approaches.The paper states that these prior methods are not specifically designed for hierarchical classification.Source: Weakly-Supervised Text Classification · p. 7

Hierarchical Text Classification

  • Hierarchical methods include local SVMs, hierarchical loss, graph-CNNs, FastXML, and dataless classification, whose performance depends on similarity between Wikipedia and the target corpus.The text notes that these approaches rely heavily on training-data quantity and quality, while dataless classification is also affected by distant-supervision similarity.Source: Hierarchical Text Classification · pp. 7–8

Conclusions

WeSHClass uses weak supervision and a hierarchical network to leverage class taxonomies and generate pseudo documents. It outperforms supervised and weakly supervised baselines across three datasets from different domains.

Conclusions

  • WeSHClass outperforms various supervised and weakly-supervised baselines on three datasets from different domains.The authors cite these results as evidence of the method’s practical value in real-world applications.Source: Conclusions · p. 8
  • The hierarchical network and training method leverage different weak supervision sources to generate high-quality pseudo documents and use the class taxonomy for better performance than flat methods and greedy approaches.Source: Conclusions · p. 8
  • The authors identify studying effective weak supervision types and combining multiple sources as future work.Source: Conclusions · p. 8
Loading 1812.11270v1…