PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM-Enhanced Dual-Branch Learning for Large-Scale Multi-Label Text Classification

Hui Ye, Jing Zhang, Xiulong Yang, Rajshekhar Sunderraman

arXiv:2609.12915v1cs.CLcs.AI

TL;DR

Large-scale multi-label classification must rank a small set of relevant labels from thousands or tens of thousands of candidates, while most representation-based systems center on one encoder or combine features in one ranker. DualMLC independently processes documents with autoregressive and bidirectional branches and fuses their label logits; it reports state-of-the-art results across three benchmarks, with the combined predictor outperforming either branch alone.

  • Problem

    Most representation-based approaches center prediction on a primary encoder or combine auxiliary features within one ranker, leaving heterogeneous language-model complementarity insufficiently explored.

    Source: §I · p. 1

  • Method

    DualMLC processes each document through independent autoregressive and bidirectional branches, then combines their label scores through late fusion in the shared label space.

    Source: §I · p. 1, B. Overview of DualMLC · p. 2

  • Results

    DualMLC achieves state-of-the-art results on three benchmarks, and branch ablations show the combined predictor outperforms either branch alone.

    Source: §I · p. 1

  • Takeaways & Limitations

    The findings support heterogeneous representation complementarity as a promising direction for prediction over large label vocabularies.

    Source: §V · p. 6

Abstract

from arXiv · show

Large-scale multi-label text classification assigns a small subset of relevant labels to each document from a vocabulary containing thousands or tens of thousands of candidate labels. Although pretrained language models have improved semantic text representations, most representation-based approaches center their prediction pipelines on a primary encoder or combine auxiliary features within a single ranker. The complementarity between heterogeneous language models therefore remains insufficiently explored. We propose DualMLC, a dual-branch framework that processes the same document through an autoregressive decoder-only language model and a bidirectional encoder. Each branch maintains its own representation pathway and independently estimates relevance scores over the shared label space. DualMLC combines the two score vectors through late logit fusion, allowing shared evidence to reinforce relevant labels and branch-specific evidence to compensate for limitations in the other branch's representation. DualMLC achieves state-of-the-art results on three widely used large-scale multi-label text classification benchmarks. Ablation results further confirm that integrating the heterogeneous predictors produces stronger rankings than either branch alone. The source code is publicly available at https://github.com/huiyegit/DualMLC.

I. INTRODUCTION

Large-scale multi-label classifiers often center on one encoder, leaving open whether heterogeneous encoders capture complementary evidence that improves rankings among similar labels. DualMLC uses independent heterogeneous branches and integrates their predictions; experiments report state-of-the-art results on three benchmarks and gains from branch integration.

I. INTRODUCTION

  • The central question is whether heterogeneous encoders capture complementary representations that improve the accuracy of highly ranked labels.For similar labels, small representation changes can alter ranking positions.Source: §I · p. 1
  • DualMLC learns independent text representations through two distinct modeling mechanisms and integrates their prediction evidence for large-scale classification.The framework preserves each branch's semantic evidence before combining predictions.Source: §I · p. 1
  • DualMLC achieves new state-of-the-art results on three benchmark datasets, and ablations verify the contribution of branch integration and key design choices.The branches also outperform either branch alone.Source: §I · p. 1

II. RELATED WORK

  • Earlier methods generally refine one encoder hierarchy or combine feature sources within a single ranking pipeline, whereas DualMLC independently supervises architecture-distinct encoders and integrates evidence in the common label space.Large-scale multi-label classification assigns each document a small subset from a vocabulary that may contain thousands or tens of thousands of labels.Source: §I · p. 1, §II · p. 2

III. METHOD

The task is to score labels for each document and rank the highest-scoring candidates. DualMLC uses parallel heterogeneous encoders that independently produce label logits, with the common label space serving as their fusion interface.

A. Task Definition

  • A document is represented by a multi-hot label vector, and the model learns a scoring function that assigns higher scores to relevant labels.For prediction depth k, the predicted set contains the k labels with the highest scores.Source: A. Task Definition · p. 2
  • Reliable ranking requires document representations that preserve semantic evidence across a large set of closely related labels.Source: A. Task Definition · p. 2

B. Overview of DualMLC

  • DualMLC processes the document through parallel heterogeneous encoders, each with an independent pathway that produces label logits.The branches remain independent until they map to the common label space, where their label-specific evidence can be fused without aligning hidden features.Source: B. Overview of DualMLC · p. 2

C. Heterogeneous Text Encoding

DualMLC uses separate tokenization and heterogeneous contextualization to preserve branch-specific representations. Its autoregressive branch uses low-rank adaptation, while both branches receive direct supervision and produce label logits.

C. Heterogeneous Text Encoding

  • The branches use separate tokenization and attention masks, respecting their different pretrained vocabularies and excluding padding during pooling.Source: C. Heterogeneous Text Encoding · p. 2
  • Causal attention forms context autoregressively, whereas bidirectional attention uses evidence from both directions, yielding different cues for the same label.The two context-formation mechanisms provide the representational diversity required by the dual-branch design.Source: C. Heterogeneous Text Encoding · p. 2
  • The autoregressive branch uses Qwen2.5-7B and low-rank adaptation, while the bidirectional branch uses BERTbase-uncased.The Qwen branch adapts attention projections; the BERT branch is fully fine-tuned.Source: C. Heterogeneous Text Encoding · pp. 2–3
  • Both heterogeneous branches independently produce label logits from the same input text, and both receive direct supervision during training.Source: Figure 1 · p. 3

D. Layer Aggregation and Document Representation

DualMLC aggregates upper-layer states before pooling, following each encoder's native sequence-summary convention. It also reduces the wider autoregressive representation with grouped mean reduction before classification.

D. Layer Aggregation and Document Representation

  • Averaging upper-layer states avoids relying on a single final layer, and branch-specific pooling avoids imposing the same sequence summary on different architectures.The passage describes uniform averaging as parameter-free.Source: D. Layer Aggregation and Document… · p. 3
  • Grouped mean reduction controls the wider autoregressive branch's label-head size while preserving feature scale; no reduction is applied to the bidirectional branch.The reduction lowers the autoregressive classifier from Ldq to Ldq/g weights, with comparability established after both branches map to L label logits.Source: D. Layer Aggregation and Document… · p. 3

E. Branch Specific Label Prediction

Each branch maps its own document representation into the shared label space with an independent classifier, making branch-specific label scores directly comparable.

E. Branch Specific Label Prediction

F. Logit Fusion and Inference

DualMLC combines branch logits in the common output space, preserving branch independence while using the fused scores to rank labels for inference.

F. Logit Fusion and Inference

G. Training Objective

Both branches receive supervision from the same multi-hot targets, while separate branch losses train the predictors without applying a loss to fused logits.

G. Training Objective

  • Both branches receive direct supervision from the same multi-hot target.Source: G. Training Objective · p. 4
  • Training optimizes separate parameter blocks: LoRA factors and the branch q classifier, plus the branch b encoder and classifier.Source: G. Training Objective · p. 4
  • Separate losses preserve each predictor's informativeness for separate evaluation and late fusion; the frozen autoregressive backbone and fixed α are excluded, and fused logits receive no loss.Source: G. Training Objective · p. 4

IV. EXPERIMENTS

The experiments use three multi-label benchmarks spanning legal, Wikipedia-tagging, and product-category data, and report precision at ranks one, three, and five.

1) Datasets:

  • The benchmarks are EURLex-4K, Wiki10-31K, and AmazonCat-13K, covering legal concepts, Wikipedia social tags, and product categories.Source: 1) Datasets · p. 4
  • The benchmarks differ in domain, label cardinality, and training scale.Source: 1) Datasets · p. 4
  • Evaluation reports P@1, P@3, and P@5, the fraction of the top k predictions that are ground-truth labels averaged over test instances.Source: 1) Datasets · p. 4

2) Baselines:

DualMLC is evaluated against sparse, embedding, label-tree, and neural text-encoder methods using two differently configured branches. It achieves the highest values across all nine dataset–metric combinations, with gains across ranking depths and datasets.

2) Baselines:

  • The baselines span sparse, embedding, and label-tree methods, alongside neural text encoders from convolutional and recurrent models to pretrained Transformers.The listed methods include AnnexML, DiSMEC, PfastreXML, Parabel, eXtremeText, Bonsai, XR-Linear, XML-CNN, AttentionXML, LightXML, APLC-XLNet, XR-Transformer, and MatchXML.Source: 2) Baselines · p. 4

3) Implementation Details:

  • The Qwen branch uses a frozen backbone adapted with LoRA, while BERT is fine-tuned end to end; both branches use 256-token inputs.Qwen uses mean pooling and dimensionality reduction, while BERT uses its classification token directly; the logits are weighted 0.6 for Qwen and 0.4 for BERT.Source: 3) Implementation Details · pp. 4–5

B. Main Results

  • DualMLC achieves the highest value for all nine dataset and metric combinations, with gains across ranking depths and benchmarks of different label vocabularies and training scales.Wiki10-31K has the largest evaluated label vocabulary, at 30,938 labels.Source: B. Main Results · p. 5

C. Ablation Study

The ablations find that the weaker standalone branch contributes evidence beyond the stronger branch, while hyperparameter tests examine fusion weights, pooling, layer aggregation, input length, and LoRA rank.

C. Ablation Study

  • DualMLC improves over the stronger standalone Qwen branch by 0.76% in P@1, 1.30% in P@3, and 1.00% in P@5, indicating added evidence from BERT.Qwen alone exceeds BERT by 1.28%, 1.82%, and 2.81% in P@1, P@3, and P@5, respectively; fusion gains at P@3 and P@5 also improve the ordering of additional relevant labels.Source: C. Ablation Study · p. 5

D. Hyper-parameter Analysis

  • Equal weighting gives the highest P@1 at 90.80%, while α = 0.6 gives the highest P@3 at 81.11% and P@5 at 71.94%.Performance declines more clearly as the Qwen weight rises to 0.8; α = 0.6 is motivated as a balanced setting.Source: D. Hyper-parameter Analysis · p. 5
  • With Qwen mean pooling fixed, BERT classification-token pooling improves P@1, P@3, and P@5 by 1.04%, 0.79%, and 0.94%, respectively.The pooling comparison is shown in Table V.Source: D. Hyper-parameter Analysis · p. 5, Table V · p. 5
  • Averaging the last four layers improves P@1, P@3, and P@5 over the final layer by 0.38%, 0.31%, and 0.58%; 256-token inputs outperform length 128 by 0.47%, 0.63%, and 0.51%.Length 512 raises P@5 by only 0.09% while lowering P@1 and P@3; averaging six layers gives no further gain.Source: D. Hyper-parameter Analysis · p. 5, Table VI · p. 5, Table VII · p. 6
  • LoRA rank 16 gives the best P@3 and ties for best P@1, while rank 8 exceeds it in P@5 by only 0.03%.Across ranks 4–32, P@1 varies by 0.15%, P@3 by 0.33%, and P@5 by 0.03%; the fixed scaling parameter means the comparison does not isolate rank capacity alone.Source: D. Hyper-parameter Analysis · p. 5, Table VIII · p. 6

E. Discussion

DualMLC improves consistently across three benchmarks and ranking depths, with its combined predictor outperforming either branch alone. The discussion attributes the benefit to complementary representations, while the conclusion points to hierarchical or clustered label indexing for future efficiency work.

E. Discussion

  • Ablation and sensitivity results indicate gains reflect complementary representations rather than capacity alone, with distinct contextual cues preserved before fusion in a common label space.Causal contextualization emphasizes progressively formed context and generative pretraining knowledge, while bidirectional contextualization evaluates tokens using evidence from both directions.Source: E. Discussion · p. 6

V. CONCLUSION

  • Across three benchmarks, DualMLC improves consistently across datasets and ranking depths, and its combined predictor outperforms either branch alone.Source: §V · p. 6
  • Future work will integrate DualMLC with hierarchical or clustered label indexing to improve efficiency on substantially larger label spaces.Source: §V · p. 6
Loading 2609.12915v1…