Source-linked AI summary
From Confusion to Clarity: Confusion-Aware Retrieval and Knowledge Injection for Text Classification
Manish Gupta, Chaitanya Giri, Jayasimha Talur
TL;DR
LLMs struggle with taxonomies containing many semantically similar labels because domain-specific distinctions are not captured by pre-training, and top-K retrieval does not resolve such confusions. The paper identifies confused label pairs, adds their partners to candidate sets, and injects targeted pairwise rules; across three benchmarks, it reports gains up to 10.0pp in Macro F1 and up to 11.5pp for smaller models through transfer.
Problem
LLMs struggle with semantically similar labels in domain-specific taxonomies, while top-K retrieval narrows candidates without helping distinguish confusable labels.
Method
The framework analyzes classification errors, augments retrieved candidates with confusion partners, and injects targeted pairwise disambiguation rules without fine-tuning.
Results
The method improves Macro F1 by up to 10.0pp on three benchmarks, while transferred knowledge improves smaller classifiers by up to 11.5pp.
Takeaways & Limitations
Confusion partners improve candidate coverage and pairwise rules help select among similar candidates, with knowledge transferring to smaller classifiers.
Takeaways & Limitations
The offline phase requires labeled data and must be rerun when taxonomies or confusion patterns change; generated rules are not automatically verified.
Abstract
from arXiv · showhide
Large language models (LLMs) struggle to classify text into taxonomies with many semantically similar labels, as the distinctions are domain-specific and not captured by pre-training. To handle large label spaces, a common approach retrieves top-$K$ candidate labels by embedding similarity and prompt the LLM to choose among them. However, top-$K$ retrieval reduces the number of candidates but does not help the model tell similar ones apart. When two similar labels both appear as candidates, the model lacks the signal to choose correctly between them. We propose a framework that (1) identifies which label pairs the model struggles to distinguish, (2) expands the candidate set to include confusable labels, and (3) generates targeted rules to differentiate between similar candidates. The framework requires no fine-tuning, and the generated rules transfer to smaller, cheaper models. On three benchmarks (WOS, Flipkart, LEDGAR), our approach improves Macro F1 by up to 10.0pp over retrieval baselines, with smaller models (2B--20B) gaining up to 11.5pp via cross-model transfer.
1 Introduction
Large-label taxonomy classification remains difficult for LLMs when semantically similar labels require domain-specific distinctions. The framework addresses this by identifying confused pairs, co-occurring candidates, and targeted disambiguation rules.
- Motivation: Hundreds of semantically similar taxonomy labels make industrial text classification error-prone even for large language models.Misclassification directly degrades search relevance, ticket routing, and compliance filtering.
- Motivation: Top-K retrieval narrows the label space but does not provide the signal needed to distinguish confusable candidates.When both similar labels appear, the LLM may still choose incorrectly.
- Approach: The framework identifies label pairs the model struggles with, ensures both appear during inference, and generates rules for differentiating them.These components target errors observed on training data rather than relying only on structural or generic signals.
- Contributions: Confusion-matrix analysis augments retrieval so systematically confused labels co-occur in the candidate set.This is one of the framework’s stated contributions.
- Contributions: A three-stage pipeline generates pairwise disambiguation rules from misclassified training examples.The rules supply targeted knowledge for choosing between similar candidates.
- Contributions: Across three datasets and seven models, pairwise knowledge improves F1 and transfers from large models to smaller classifiers without retraining.The reported model range is 2B–32B.
2 Related Work
Prior work narrows candidate labels, injects taxonomy structure, or optimizes prompts, but does not identify the specific label pairs responsible for classification errors. This work instead uses an error-informed strategy focused on post-retrieval disambiguation.
- LLMs for Text Classification: Zero-shot LLMs underperform fine-tuned models on domain-specific classification, motivating inference-time alternatives that avoid dedicated training.Fine-tuned models require infrastructure per taxonomy and may not generalize robustly to out-of-distribution inputs.
- LLMs for Text Classification: Retrieval-based methods narrow the candidate set, while other approaches inject knowledge-graph or taxonomy-expansion context.These methods primarily affect which labels reach the LLM.
- Positioning: The proposed approach is error-informed: it addresses failures that occur when similar labels co-occur after retrieval.Its focus complements methods that improve candidate selection or inject structural context.
- Prompt Optimization and Inference-Time Strategies: Prompt optimization and iterative pairwise comparison improve prompting generically but do not analyze which specific label pairs cause errors.Iterative comparison can require multiple LLM calls per query.
3 Methodology
The methodology combines confusion-aware retrieval with offline rule generation and inference-time knowledge injection. It uses observed classifier errors to add missing confusable labels and guide distinctions among retrieved candidates.
- 3.1 Confusion-Aware Retrieval: Top-K retrieval can miss taxonomy-near labels because general-purpose embeddings do not reflect taxonomy-specific similarity.Increasing K may recover labels but also introduce irrelevant candidates that degrade classification.
- 3.1 Confusion-Aware Retrieval: The method analyzes baseline training errors to rank directed confusion pairs and selects the smallest set covering at least τ% of errors.Each error increments a count from the true label to the predicted label.
- 3.1 Confusion-Aware Retrieval: For each retrieved label, the candidate set is expanded with its confusion partner so both labels of a confused pair are present.Co-occurrence alone does not resolve the distinction between them.
- 3.2 Knowledge Generation: Disambiguation rules are generated offline through per-example reasoning, consolidation of recurring signals, and bidirectional merging.The final knowledge base contains one rule per selected pair.
- 3.3 Inference: At inference, retrieved candidates are expanded, applicable rules are injected into the prompt, and the LLM selects a label in one call.Partners provide candidate coverage while rules guide the choice among candidates.
4 Experiments
Experiments evaluate component contributions, composition, transfer, and dataset coverage across three benchmarks and multiple model sizes. The setup uses hierarchical WOS and Flipkart taxonomies alongside flat LEDGAR.
- 4.1 Setup: The evaluation asks whether confusion partners and knowledge injection help independently, compose across datasets, transfer across models, and outperform larger K or per-label alternatives.These questions also assess the method’s mechanisms and practical trade-offs.
- 4.1 Setup: The study uses three public benchmarks: WOS, Flipkart, and LEDGAR.WOS and Flipkart are hierarchical, while LEDGAR is flat.
- 4.1 Setup: Main experiments use Qwen3-32B, while cross-model transfer applies Qwen3-235B-generated knowledge to models ranging from 2B to 20B.Qwen3-Embedding-8B provides retrieval embeddings.
- 4.1 Setup: Retrieval uses K=10 candidates, and confusion pairs cover at least 75% of training errors.All LLM calls use temperature 0.4.
- 4.1 Setup: Qwen3-235B is a mixture-of-experts model with non-deterministic routing at temperature 0, so standard deviation is reported.This is an experimental assumption affecting result variability.
5 Results
The framework improves classification across benchmarks and addresses complementary retrieval and selection errors. Its gains also transfer to smaller models and remain stronger than fine-tuning under several robustness conditions.
- The method achieves the best Macro F1 in every Table 1 column, with the largest margin at Flipkart L3.It improves by +11.2pp over zero-shot and +4.4pp over retrieval at K=20.
- Error decomposition: On Flipkart L3, confusion partners raise recall from 86% to 98.1%, while knowledge injection reduces LLM misses from 21.9% to 17.1%.Adding partners also increases LLM misses from 15.5% to 21.9%, showing why the two mechanisms are complementary.
- Confusion partners vs. larger K: At K=10, confusion partners achieve 74.2 F1 and 98.1% recall with 18 average candidates, outperforming plain retrieval at K=20.Plain retrieval reaches only 72.2 F1 and 92.4% recall at K=20.
- Pairwise vs. per-label knowledge format: Pairwise rules outperform per-label descriptions because they tailor the guidance to the exact candidate comparison.The pairwise format presents both labels and a rule specific to distinguishing them.
- Cross-model knowledge transfer: Small models gain +11.5pp on Flipkart L3 and Medium models gain +11.1pp over retrieval-only when using rules generated by Qwen3-235B.LEDGAR gains are +6.3pp for Small and +5.3pp for Medium models.
- Comparison with fine-tuning: Fine-tuned models outperform the framework on clean data but degrade more under label noise, data scarcity, and test-time perturbations.With 20% label replacement, the framework reaches 72.1 Macro F1 versus 60.0 and 61.0 for ModernBERT and RoBERTa; with 10% of training examples, it reaches 70.2 versus 23.2 and 31.8.
6 Conclusion
The paper concludes that converting classification errors into targeted pairwise rules, together with confusion-partner retrieval, improves accuracy without fine-tuning. The approach reaches gains on three benchmarks and transfers knowledge from large to smaller classifiers.
- The framework turns LLM classification errors into targeted disambiguation rules without fine-tuning.
- Confusion partners supply likely labels, while pairwise rules help the model choose between them.
- The approach improves Macro F1 by up to 10.0pp across three benchmarks, while transferred rules yield gains of up to 11.5pp for smaller classifiers.
Limitations
The framework is limited by its dependence on observed baseline confusions and by the lack of automatic verification for generated rules.
- The offline phase requires labeled data and only generates rules for confusion pairs observed in baseline errors.Changed taxonomies or confusion patterns may require rerunning the offline phase.
- Broader pair coverage lengthens inference prompts and increases cost, while multilingual inputs, larger taxonomies, and temporal drift remain untested.
- Generated rules are not automatically verified, despite Consolidation and Bidirectional Merge reducing noise from individual examples.In sampled human evaluation, 87.3% of rules met correctness and usefulness criteria; consequential applications still require domain-expert review.
A.1 Algorithms
The offline pipeline derives confusion partners and a knowledge base from baseline training errors, then the inference procedure adds these components to retrieval.
- The component ablation evaluates adding knowledge and combining it with confusion partners against the zero-shot baseline.
- Algorithm 1 takes baseline training predictions and a coverage threshold, producing a knowledge base K and PARTNER(·).
- The pipeline counts directed label-pair errors and selects the smallest pair set covering at least τ% of total errors.
- For each selected confusion direction, the LLM reasons over misclassified queries and consolidates those observations into a rule.
- Bidirectional rules are merged into one knowledge entry when both directions exist; otherwise, the available directional rule is retained.
A.2 Dataset Preprocessing
The Flipkart dataset is pruned to three semantic category levels and filtered for stratified splitting, yielding 17,847 samples across 351 labels.
- 17,847 samples across 351 labels remain after retaining the first three hierarchy levels and removing labels with fewer than three samples.Deeper levels are excluded because they represent brand or product names rather than semantic categories and more than double the unique-label count beyond level 3.
A.3 Fine-Tuning Baselines
The experiments compare fine-tuned and inference-time approaches across clean, perturbed, noisy, and data-scarce settings, while measuring coverage, cost, transfer, rule quality, and residual errors. The framework improves retrieval-based classification through confusion partners and knowledge injection, but incurs prompt-length costs and leaves errors from taxonomy inconsistency and genuine semantic overlap.
- A.3 Fine-Tuning Baselines: Fine-tuned models lead on clean, complete data at each dataset’s finest level.
- A.3 Fine-Tuning Baselines: Macro F1 increases monotonically as confusion-pair coverage rises from 10% to 90%, but broader coverage increases prompt size and inference cost.The paper uses 75% coverage as a cost-conscious operating point and allows applications to adjust the threshold according to accuracy and inference budgets.
- A.5 Knowledge Generation and Inference Cost: Inference latency rises from 379–429 ms for retrieval-only inference to 879–1,238 ms for the full approach.Offline generation processes 3,564 to 19,528 errors, produces 455 to 658 rules, and consumes 3.26M to 15.52M tokens once per taxonomy.
- A.6 Reusing a Fixed Knowledge Base: Fixed knowledge reuse improves Macro F1 by 6.6 to 10.3 points over retrieval alone across classifiers.Model-specific pair discovery performs better for three classifiers, while fixed reuse performs better for GPT-OSS-20B.
- A.12 Generated Knowledge Examples: 87.3% of generated rules are judged to correctly and usefully distinguish labels on average, while high-stakes deployment requires domain-expert approval.
- A.3 Fine-Tuning Baselines: Confusion partners increase recall but also increase the LLM miss rate, whereas knowledge injection lowers the miss rate at each retrieval setting.
- A.13 Error Analysis: Genuine semantic overlap accounts for 230 errors, or 31.2%, and remains for inherently multi-category items despite reducing Western→Fusion errors from 112 to 71.