Source-linked AI summary
Incorporating Hierarchy into Text Encoder: a Contrastive Learning Approach for Hierarchical Text Classification
Zihan Wang, Peiyi Wang, Lianzhe Huang, Xin Sun, Houfeng Wang
TL;DR
Hierarchical text classification must handle structured label hierarchies, but existing methods keep hierarchy representations separate and constant across inputs. HGCLR transfers hierarchy information into a text encoder by constructing hierarchy-guided positive samples for contrastive learning, and experiments report consistent improvements on two datasets with comparable results on another.
Problem
Existing hierarchical text-classification methods separately encode text and label hierarchies, leaving the hierarchy representation unchanged across input texts.
Method
HGCLR uses label- and hierarchy-guided positive samples, a modified Graphormer, and contrastive learning to migrate taxonomy information into BERT encoding.
Results
HGCLR achieves consistent improvements on two distinct datasets and comparable results on another one.
Takeaways & Limitations
After training, the hierarchy-aware text encoder can represent hierarchy information without the constant label feature used by separate-model approaches.
Abstract
from arXiv · showhide
Hierarchical text classification is a challenging subtask of multi-label classification due to its complex label hierarchy. Existing methods encode text and label hierarchy separately and mix their representations for classification, where the hierarchy remains unchanged for all input text. Instead of modeling them separately, in this work, we propose Hierarchy-guided Contrastive Learning (HGCLR) to directly embed the hierarchy into a text encoder. During training, HGCLR constructs positive samples for input text under the guidance of the label hierarchy. By pulling together the input text and its positive sample, the text encoder can learn to generate the hierarchy-aware text representation independently. Therefore, after training, the HGCLR enhanced text encoder can dispense with the redundant hierarchy. Extensive experiments on three benchmark datasets verify the effectiveness of HGCLR.
1 Introduction
Hierarchical text classification must represent large, imbalanced, structured label hierarchies. HGCLR addresses redundancy in separate text–hierarchy modeling by injecting hierarchy guidance into the text encoder through contrastive learning.
- Hierarchical text classification assigns text to labels organized in a tree or directed acyclic graph, making large, imbalanced, structured hierarchies central to the task.
- Recent methods encode text and label hierarchies separately, then aggregate their representations into a mixed classification feature.
- Because the label hierarchy is unchanged across inputs, its constant representation may interact redundantly and less effectively with each text representation.
- HGCLR constructs positive samples by shortening input sequences while retaining label-relevant keywords, guided by both ground-truth labels and taxonomy hierarchy.
- HGCLR uses contrastive learning to obtain hierarchy-aware text representations, with a modified Graphormer encoding label-hierarchy features for sample construction.
- Experiments report improvements on three datasets, supporting the proposed approach for hierarchical text classification.
2 Related Work
Prior hierarchical text-classification methods vary in how they organize classifiers and encode label structure, while recent approaches increasingly use explicit structure encoders. Contrastive-learning work instead centers on constructing suitable positive pairs for representation learning.
- Hierarchical text-classification methods include local, global, and hybrid approaches that differ in whether classifiers are built per node, per level, or for the whole graph.
- Early global methods treated hierarchical classification as flat multi-label prediction, whereas later methods incorporated structure through regularization, reinforcement learning, capsules, or meta-learning.
- Recent work directly encodes holistic label structure and combines structural representations with text features through structure encoders, joint embeddings, information maximization, or adaptive fusion.
- In NLP contrastive learning, data augmentation such as back-translation, permutation, and masking is used to generate positive pairs with similar meaning.
3 Problem Definition
Hierarchical text classification predicts a subset of labels for an input sequence, with candidate labels organized as a directed acyclic graph. Valid predictions respect the hierarchy by including each selected label’s parent.
- Given an input sequence x and label set Y, hierarchical text classification predicts a subset y of candidate labels organized in a directed acyclic graph G.
- The graph’s nodes are labels and its edges encode their hierarchy; each selected non-root label must have its father label selected as well.
- Because each non-root label has one father, the taxonomic hierarchy can be treated as tree-like, while a valid label subset corresponds to one or more paths.
4 Methodology
HGCLR uses a Graphormer-based label-hierarchy encoder to guide positive-sample construction, then trains BERT with classification and contrastive objectives so hierarchy-aware representations can be used without the hierarchy at test time.
- Graph Encoder: HGCLR combines a BERT text encoder with a customized Graphormer that encodes label features and hierarchy structure.Label features combine learnable label embeddings with label-name embeddings; spatial and edge encodings modify graph self-attention.
- Positive Sample Generation: Token importance is estimated from attention between token embeddings and label features, producing label-guided probabilities for positive-sample generation.For multi-label inputs, probabilities across ground-truth labels are added before thresholding.
- Positive Sample Generation: HGCLR retains tokens whose sampling probabilities exceed threshold γ, while zero embeddings preserve the original positions in the shortened sequence.The resulting positive sample is passed through the same BERT encoder as the original input.
- Contrastive Objective: Contrastive learning pulls each original representation toward its positive counterpart and pushes it away from the other examples in the batch.The method applies a nonlinear projection and NT-Xent loss, with cosine similarity and temperature τ.
- Classification and Objective Function: The final objective combines classification losses for original and positive samples with a contrastive loss weighted by λ.During testing, only the text encoder and classification head are used.
5 Experiments
Experiments compare HGCLR with BERT-based and hierarchy-aware baselines across three datasets, then analyze its graph encoder, hierarchy representations, and positive-sample strategy. HGCLR improves over BERT on WOS and NYT, while component analyses support contributions from Graphormer, contrastive learning, and hierarchy-guided examples.
- 5.1 Experiment Setup: Experiments evaluate HGCLR on WOS, NYT, and RCV1-V2 using Macro-F1 and compare it with recent hierarchy-aware baselines implemented with BERT where applicable.WOS uses single-path labels, whereas NYT and RCV1-V2 use multi-path taxonomies.
- 5.2 Experimental Results: 1.5% Micro-F1 and 2.1% Macro-F1 improvements over BERT are reported on WOS, with HGCLR also outperforming HiMatch.The comparison is reported in the main results discussion.
- 5.2 Experimental Results: 2.3% Macro-F1 improvement over BERT is reported on NYT, while HGCLR slightly increases Micro-F1 and outperforms previous methods on both measurements.BERT already performs strongly on NYT and RCV1-V2 because it was pretrained on news corpora.
- 5.2 Experimental Results: On RCV1-V2, HGCLR achieves state-of-the-art Micro-F1, while HiMatch performs extremely well on Macro-F1; the dataset lacks label names for name embeddings.The text notes that baselines relying only on label-name initialization are less affected by this missing information.
- 5.3 Analysis: Graphormer outperforms GCN and GAT, and removing the graph encoder causes a significant performance drop on WOS.Graphormer’s global attention lets each node attend to all others, unlike GAT’s neighbor-only attention.
- 5.3 Analysis: Both hierarchy-guided positive-sample generation and contrastive learning contribute: generation alone improves BERT by around 1% on two measurements, while contrastive learning adds further gains.The contrastive objective regularizes the text representation.
- 5.3 Analysis: Figure 3 shows clustered same-parent label representations for HGCLR but scattered representations for BERT, supporting hierarchy-aware text encoding.Labels with the same father are expected to have more similar representations when hierarchy is injected into the text representation.
- 5.3.1 Effect of Hierarchy: Name embedding contributes most among Graphormer components, while spatial and edge encoding also contribute; edge encoding is least useful for this hierarchy.The paper attributes the limited edge-encoding effect to the lack of edge-feature information in HTC.
6 Conclusion
The paper presents HGCLR, which migrates taxonomy hierarchy information into BERT encoding through graph-encoder-guided positive examples and a modified Graphormer. It reports consistent improvements on two datasets, comparable results on another, and effectiveness for all designed components.
- HGCLR migrates taxonomy hierarchy information into BERT encoding through contrastive learning.
- The method constructs contrastive-learning positive examples under guidance from a graph encoder that learns label features from the taxonomy hierarchy.
- HGCLR modifies Graphormer to improve graph understanding for hierarchical text classification.
- HGCLR achieves consistent improvements on two distinct datasets and comparable results on another dataset.
- All designed components are empirically shown to be effective.
A Trick for Token Selection
The token-selection trick modifies token embeddings rather than tokens so selection probabilities can receive gradients. The positive counterpart is numerically either the original embedding or zero according to threshold γ.
- The method modifies token embeddings instead of tokens so the selection probabilities can acquire gradients.
- The positive counterpart of each token embedding uses the original embedding when its selection probability exceeds threshold γ, otherwise zero.
- Detach ignores the gradient of its input while preserving the intended numerical selection behavior.