PaperlayerView exact revision on arXiv

Source-linked AI summary

TaxCE : A Framework for Automated Taxonomy Construction and Evaluation at Scale

Sandeep Sricharan Mukku, Albert Aristotle Nanda, Rohit Pyati

arXiv:2608.30614v1cs.CL

TL;DR

Existing methods struggle to construct deep, corpus-grounded taxonomies from large, varied feedback and to evaluate them comprehensively. TaxCE progressively condenses raw text into standardized concepts and granular topics, builds and validates a hierarchy, and refines it using EEG metrics. Across three datasets, it outperforms baselines on exclusivity, exhaustivity, and granularity, with human evaluations also confirming stronger taxonomy quality, actionability, and navigability.

  • Problem

    Automated feedback taxonomy methods can produce shallow hierarchies, miss long-tail topics, or lack rigorous corpus-grounded evaluation.

  • Method

    TaxCE condenses raw feedback into actionable segments, deduplicated concepts, and granular topics, organizes them bottom-up, validates root-to-leaf paths, and iteratively refines the hierarchy using EEG metrics.

  • Results

    TaxCE achieves the highest scores across all three metrics on all three datasets, improving over the strongest baseline by 11.8, 20.5, and 15.7 percentage points in exclusivity, exhaustivity, and granularity respectively.

  • Takeaways & Limitations

    Results across three domains show consistent improvements over baselines on automatic and human evaluations.

  • Takeaways & Limitations

    Early-stage outputs can propagate downstream errors, EEG scores depend on the embedding model, very large-corpus extraction can be expensive, and fully non-English validation was not conducted.

Abstract

from arXiv · show

Organizing unstructured feedback text into hierarchical taxonomy is a fundamental challenge in NLP, particularly in domains where feedback arrives at massive scale in varied forms such as reviews, transcripts, and surveys. Existing approaches either produce shallow hierarchies, neglect long-tail topics, or lack rigorous evaluation frameworks. We present TaxCE, a fully automated framework that constructs multi-level hierarchical taxonomies from raw text through progressive condensation of corpus content into actionable segments, deduplicated semantic units, and granular topics with definitions, which are then organized bottom-up into a hierarchy with corpus-groundedness. We also introduce three corpus-grounded evaluation metrics, Exclusivity, Exhaustivity, and Granularity (EEG), and integrate them into a metrics-in-the-loop iterative refinement mechanism that diagnoses deficiencies and applies targeted corrections until convergence. Extensive experiments demonstrate that TaxCE consistently outperforms existing baselines spanning classical topic models, neural methods, and LLM-based approaches, with average improvements of 11.8, 20.5, and 15.7 percentage points in exclusivity, exhaustivity, and granularity respectively over the strongest baseline. Human evaluation further confirms superior taxonomy quality, actionability, and navigability.

1 Introduction

TaxCE addresses the difficulty of organizing massive, varied feedback corpora into actionable hierarchical taxonomies. It replaces costly, subjective manual construction and shallow automated approaches with an automated, corpus-grounded framework.

  • Large feedback corpora from reviews, transcripts, surveys, and support tickets are rich but unwieldy for decision-making.
  • Manual taxonomy construction requires experts to identify, scope, and hierarchically organize topics, making it expensive, subjective, and difficult to reproduce.
  • Existing automated approaches often produce shallow hierarchies that fail to represent multi-level, actionable sub-topics.
  • TaxCE converts raw feedback into validated, multi-level hierarchical taxonomies through concepts, granular topics, definitions, and bottom-up organization.
  • The framework requires no seed terms, labeled data, or domain ontology, uses minimal human supervision, and supports language-agnostic processing through translation during extraction.

2 Related work

Prior taxonomy-construction methods span classical, neural, and reinforcement-learning approaches but commonly require seeds, network structure, or limited term-level settings. Evaluation remains fragmented across structural, flat-topic, downstream, and human measures.

  • Classical methods use lexico-syntactic patterns, distributional similarity, or structured learning, with SemEval establishing term-level benchmarks.
  • Neural and reinforcement-learning methods improve taxonomy induction but may require seed taxonomies, network signals, or target term-level rather than topic-level hierarchies.
  • Existing evaluation uses downstream accuracy, structural measures, human ratings, or flat-topic coherence and diversity, often ignoring hierarchical structure.
  • TaxCE combines progressive knowledge condensation with root-to-leaf validation and EEG-driven refinement that loops back to grouping until convergence.

3 Problem statement

TaxCE formulates taxonomy construction as producing a granular, actionable hierarchy from an unstructured corpus without predefined schema, seed terms, or ontology. The taxonomy must meet minimum thresholds for exclusivity, exhaustivity, and granularity.

  • Given an unstructured corpus and specified information types, the goal is a multi-level taxonomy with granular, actionable leaves organized from coarse to specific.
  • The construction setting assumes no predefined schema, seed terms, or domain ontology.
  • The resulting taxonomy must satisfy minimum thresholds for exclusivity, exhaustivity, and granularity.
  • An actionable segment is a contiguous text span conveying a self-contained piece of information, paired with an intent label derived from document context.
  • A granular topic is a specific, actionable leaf-level topic derived from one or more standardized concepts and given a natural-language definition.

4 Methodology

TaxCE progressively condenses raw feedback into extracted intent-tuples, semantic groups, standardized concepts, granular topics, and a validated hierarchy. EEG metrics guide targeted iterative corrections until quality thresholds are met.

  • Framework Overview: TaxCE uses six stages: extraction, semantic grouping, concept generation and standardization, granular topic generation, hierarchical construction with validation, and iterative refinement.
  • Key Information Extraction: For each document, the framework extracts actionable segments and context-enriched intent labels, translating them into the target taxonomy language when needed.
  • Semantic Grouping: Intent-tuples are embedded after concatenating segments with intents, then organized into non-overlapping semantic groups using a grouping algorithm such as HDBSCAN.
  • Concept Generation: Within groups, an instruction-tuned LLM identifies distinct concepts and selects diverse, specific representative verbatims covering each concept’s scope.
  • Concept Standardization: Corpus-wide standardization merges semantically identical concepts and removes overlapping representative verbatims, producing atomic standardized concepts.
  • Hierarchy Construction: Standardized concepts are converted into specific, independent leaf topics, which are grouped bottom-up into parent topics until the target depth is reached.
  • Validation and Refinement: Root-to-leaf validation checks logical coherence and corpus support, while EEG-driven refinement adjusts grouping and standardization in response to deficient metrics.

5 Evaluation metrics

TaxCE evaluates taxonomy quality with three complementary, corpus-grounded metrics: Exclusivity, Exhaustivity, and Granularity. Together, they assess topic distinctiveness, corpus coverage, and specificity of leaf topics.

  • Metric design: The metrics are intrinsic, mathematically defined, and explicitly anchored to the input corpus rather than relying on extrinsic proxies or human judgment.Exhaustivity uses corpus-derived actionable segments, while granularity compares each segment with its best-matching leaf topic.
  • Exclusivity: Exclusivity measures semantic distinctiveness among leaf topics, with higher scores indicating less overlap.A score of 100 represents perfect distinctiveness, defined as zero average pairwise similarity.
  • Metric design: Leaf-level exclusivity is intended to extend to higher hierarchy levels because parent topics abstract over disjoint sets of distinct children.This follows the hierarchical construction in which each parent groups semantically related but distinct children.
  • Exhaustivity: Exhaustivity measures whether the taxonomy covers all actionable intents evidenced in the corpus.It uses actionable segments extracted from the corpus as a method-agnostic reference and counts a segment as covered when similarity to a leaf topic exceeds 0.65.
  • Granularity: Granularity measures how closely specific leaf topics represent the underlying corpus segments.It complements binary exhaustivity with a continuous similarity-based measure, distinguishing close representation from merely broad coverage.

6 Experiments

TaxCE is evaluated across three public feedback datasets against classical, neural, and LLM-based baselines using corpus-grounded metrics and human judgments. It achieves the strongest automatic and human-evaluated taxonomy quality, while ablations show that standardization and iterative refinement address complementary weaknesses.

  • Main results: TaxCE achieves the highest scores on exclusivity, exhaustivity, and granularity across all three datasets.Average improvements over LLM + Clustering are 11.8, 20.5, and 15.7 percentage points respectively.
  • Main results: Flat topic models capture prominent topics but score poorly on exhaustivity and granularity because they produce word distributions rather than actionable definitions.Their reported ranges are 76–83 for exclusivity, 28–47 for exhaustivity, and 36–53 for granularity.
  • Main results: LLM-based methods improve coverage or corpus grounding but retain overlap from undeduplicated or redundant topics.TaxCE addresses these issues through concept generation, standardization, and iterative refinement.
  • Ablation study: Removing concept standardization decreases exclusivity by 14.4 points, while removing iterative refinement decreases exhaustivity by 7.6 points.Removing both components produces the lowest scores, indicating complementary effects.
  • Human evaluation: TaxCE receives the highest human ratings for taxonomy quality, actionability, and navigability.Its actionability rating exceeds LLM + Clustering by 1.1 points, while BERTopic scores 1.9 on navigability.

7 Conclusion

The conclusion presents TaxCE as a progressive knowledge-condensation framework with corpus-grounded EEG evaluation and iterative refinement. Results across three domains show consistent gains over baselines, while the authors identify sensitivity, embedding dependence, depth selection, cost, and multilingual validation as boundaries.

  • Conclusion: TaxCE converts raw feedback into taxonomies through concept generation, standardization, hierarchical construction, and iterative refinement.Concepts act as intermediate representations between raw text and taxonomy nodes.
  • Conclusion: EEG provides corpus-grounded measures of exclusivity, exhaustivity, and granularity for evaluating taxonomy quality.The metrics are integrated into a refinement loop that applies targeted corrections.
  • Conclusion: TaxCE shows consistent improvements over baselines across three domains in both automatic and human evaluations.The conclusion describes this as confirmation of the framework’s core claims.
  • Limitations: The framework lacks an automatic mechanism for selecting taxonomy depth, and depth 3 worked best consistently only in the reported experiments.Other corpora may prefer different depths.
  • Limitations: KIE processing is expensive for very large corpora, representative sampling can miss long-tail topics, and fully non-English corpora were not validated.The experiments used primarily English data with only partial multilingual augmentation.

1. LLM-generated taxonomies may inherit biases

TaxCE’s LLM-based generation can inherit biases from its underlying language model, creating culturally skewed category names or underrepresenting minority viewpoints. The paper also identifies English-language focus and computational cost as important deployment boundaries.

  • 1. LLM-generated taxonomies may inherit biases: LLM-generated taxonomies may exhibit culturally skewed category names or underrepresentation of minority viewpoints.The paper recommends reviewing outputs for bias when taxonomies support complaint routing or content moderation.
  • 1. LLM-generated taxonomies may inherit biases: English-language focus, despite partial multilingual augmentation, may limit generalizability to other linguistic contexts.The authors encourage future validation across diverse languages.
  • 1. LLM-generated taxonomies may inherit biases: Multiple LLM inference calls create computational cost and associated environmental impact.TaxCE mitigates this through locally hosted models for high-volume extraction and concept-level downstream processing.

A Per-dataset ablation results

Ablations across three datasets show that concept standardization and iterative refinement each support taxonomy quality, while removing both produces the weakest results and a synergistic degradation.

  • A Per-dataset ablation results: Removing both standardization and iterative refinement yields the lowest scores across all three datasets.The combined degradation exceeds the sum of individual removals on exclusivity, including −18.9 on Flipkart versus −14.5 and −5.4 individually.
  • A Per-dataset ablation results: −15.3 exclusivity is lost on AskUbuntu without standardization, where overlapping technical topics create substantial cross-group redundancy.Exhaustivity marginally increases because no concepts are merged, but overlapping leaf topics remain.
  • A Per-dataset ablation results: −8.2 exhaustivity is observed on Flipkart without iterative refinement, as first-pass taxonomies miss long-tail product feedback.Removing refinement also causes Flipkart’s largest granularity drop, −4.9.

B Grouping algorithm analysis

HDBSCAN provides the strongest grouping performance in the comparison, combining the highest silhouette score with the best downstream EEG results averaged across all three datasets. Its density-based grouping also adapts to varying group densities in long-tailed feedback corpora.

  • B Grouping algorithm analysis: HDBSCAN achieves the highest silhouette score and best downstream EEG performance averaged across all three datasets.Its density-based approach automatically determines the number of groups and adapts to varying group densities.
  • B Grouping algorithm analysis: Spherical K-Means performs comparably when the number of groups is well-tuned via an a priori specification.The passage contrasts this tuning requirement with HDBSCAN’s automatic determination of group count.
  • B Grouping algorithm analysis: Agglomerative clustering produces uneven group sizes, including very large groups and many singletons, complicating downstream concept generation.HDBSCAN noise points are assigned to their nearest group through embedding similarity in post-processing.

C Hyperparameter analysis

Hyperparameter analysis identifies t_max = 5 and moderate standardization as the settings that best balance exclusivity, exhaustivity, and granularity. Extreme settings trade one metric against others or add redundancy and computational overhead.

  • C Hyperparameter analysis: t_max = 5 balances exclusivity, exhaustivity, and granularity because most groups contain 2–4 distinct intents.At t_max = 2, merging reduces exhaustivity and granularity; beyond t_max = 7, added concepts are predominantly near-duplicates.
  • C Hyperparameter analysis: At t_max = 2, exclusivity reaches 87.4 while exhaustivity falls to 61.7 and granularity to 64.2.Low t_max forces groups containing three or more distinct intents to merge.
  • C Hyperparameter analysis: At t_max = 10, exclusivity drops to 81.1 as fine-grained duplicates emerge after larger settings add computational overhead.The standardization step must merge many of these near-duplicate concepts.
  • C Hyperparameter analysis: Moderate standardization achieves the best overall balance between redundancy removal and preservation of distinct concepts.Strict standardization leaves exclusivity at 76.6, whereas aggressive standardization reaches exclusivity 89.5 but lowers exhaustivity to 68.2 and granularity to 67.1.
  • C Hyperparameter analysis: TaxCE dynamically shifts toward strict standardization when exhaustivity is low and toward aggressive standardization when exclusivity is low.Per-dataset results confirm this adaptive pattern across domains.

D Taxonomy depth analysis

Taxonomy depth primarily affects navigability rather than leaf-level EEG quality: depth 3 provides the clearest coarse-to-granular organization, while shallower or deeper structures add navigation costs. The section also examines model choices and iterative refinement behavior across datasets.

  • Depth effects: Leaf-level EEG metrics remain stable across depths because TaxCE constructs granular leaf topics bottom-up; depth changes intermediate navigational levels.At default depth h=3, L1 denotes coarse topics, L2 hinge topics, and L3 granular topics.
  • Depth effects: At h=2, 120+ root entry points for Flipkart make relevant topics difficult to locate without prior taxonomy knowledge.Each root contains approximately 15 leaf topics, but there is no higher-level organization.
  • Depth effects: At h=3, a second grouping creates about 18 Flipkart top-level categories and the highest human-rated navigability.This yields a coarse→hinge→granular navigation path.
  • Depth effects: At h=4 and h=5, additional abstraction levels add navigation overhead or overly broad categories without meaningful differentiation.The reported semantic gap between consecutive levels becomes too narrow to justify the extra navigation steps.
  • Dataset scale: Taxonomy size follows corpus diversity and scale: multilingual, product-diverse Flipkart is largest, while single-domain AskUbuntu is most compact.Flipkart averages about 15 L2-to-L3 children, compared with 5–7 for the other datasets.
  • Model selection: Qwen3-8B reaches F1 89.4 but runs at 15.8 docs/s, whereas 3–4B models provide F1 above 85 at 30–36 docs/s for corpora exceeding 1M documents.The 3–4B models are reported as the strongest cost-quality balance for high-volume KIE.
  • Model selection: Claude-3.5-Sonnet and Claude-3-Opus lead downstream EEG with less than 0.5-point differences, while open-source models trail by 2–4 points.The gap is primarily on exhaustivity and granularity.
  • Iterative refinement: Initial taxonomies consistently fail the exclusivity threshold, while CFPB and AskUbuntu converge after one correction and Flipkart requires a second coverage-focused iteration.Fewer than 5% of runs fail to converge within three iterations and are returned flagged for manual review.

G Stage-wise evaluation

Stage-wise evaluation shows strong extraction, improved concept distinctness after standardization, and complementary weaknesses in baseline methods. TaxCE’s condensation, deduplication, and refinement stages support stable EEG rankings and high-quality granular topics across datasets.

  • Evaluation setup: TaxCE evaluates Key Information Extraction, Semantic Grouping, and Concept Generation and Standardization as intermediate stages before final taxonomy construction.The stage-wise analysis tests whether intermediate outputs propagate quality to the final taxonomy.
  • Key Information Extraction: CFPB achieves extraction F1 90.0, while AskUbuntu scores 83.7 because technical jargon and code complicate actionable-segment extraction.Precision exceeds recall across datasets, indicating conservative extraction that prefers missing borderline segments to introducing noise.
  • Concept quality: Standardization improves concept distinctness by +0.7 on average while maintaining coherence.It merges redundant cross-group concepts without reducing whether representative verbatims describe the same concept.
  • Baseline comparison: Flat topic models achieve exclusivity 76–83 but score only 28–47 on exhaustivity and 36–53 on granularity, missing long-tail intents.Their word-distribution or centroid representations also limit actionable definitions.
  • Baseline comparison: LLM Zero-shot reaches exhaustivity 48–55 and granularity 53–57 but has exclusivity 66–70 because single-pass generation lacks corpus-grounded deduplication.LLM + Clustering raises exclusivity to 72–76 and exhaustivity to 55–60, but overlapping topics and missed intents remain.
  • TaxCE mechanisms: TaxCE addresses baseline weaknesses by condensing groups into atomic semantic units, standardizing across groups, and iteratively correcting remaining EEG gaps.These stages target context dilution, hallucinated or missed intents, and overlapping leaf topics.
  • Robustness: Across six embedding backbones, TaxCE ranks above LLM+Clustering and BERTopic, with EEG correlations of 0.89–0.97 and unchanged method rankings after domain tuning.For CFPB, FinBERT produces TaxCE scores of 89.4/82.8/80.6, a +2.3 average gain over the general-purpose baseline.
  • Human alignment: Exclusivity correlates most strongly with navigability and quality, while exhaustivity and granularity correlate with actionability.The reported Pearson correlations are approximately 0.72 and 0.78 for exclusivity, and 0.65–0.71 for exhaustivity and granularity.
Loading 2608.30614v1…