Source-linked AI summary
From Plausible Hierarchies to Useful Taxonomies: Evaluating Agentic Harnesses on Customer Feedback
Prabhath Chellingi, Raviraja G, Viraj Bagal
TL;DR
Generic checks assess taxonomy nodes and relations individually, although trustworthy counts and team ownership depend on properties of the whole tree. This paper introduces whole-tree metrics for taxonomy structure and record partitionability; generated trees pass the generic checks while exhibiting failures those checks miss.
Problem
Common taxonomy checks judge individual names, edges, or sibling pairs, although trusted volume counts and team ownership depend on properties spread across the tree.
Source: §1 · p. 1Method
Results
Across generated runs, generic checks passed despite whole-tree failures, with 97.7–100% of generated leaf names restating an ancestor.
Source: §1 · p. 1Takeaways & Limitations
Taxonomy evaluation needs to measure the whole tree and the records routed through it, as well as individual nodes.
Source: §1 · p. 1, §6 · p. 7The metrics have not been validated against an external usefulness signal such as team velocity.
Source: §7 · p. 7
Abstract
from arXiv · showhide
Taxonomies are the symbolic representations through which AI systems organize evidence, aggregate patterns, and answer questions over large document collections. Over customer feedback, the category tree decides how every record is counted and routed, which problems get seen, and which team owns them. Agentic harnesses now make it easy to generate a plausible-looking hierarchy, and such trees are checked today with generic, individually scoped checks: each name fits its description, sits under the right parent, and stays distinct from its siblings. We ask a more operational question: when is a generated hierarchy actually useful as a production taxonomy? We build six taxonomies over two proprietary feedback corpora (1,940 and 5,000 records): for each corpus, a production reference and two repeated runs of the same harness under identical inputs. All six pass every generic naming and structure check, and a deeper product-coverage check even prefers the generated trees. Yet in every generated tree at least 97.7% of leaf names merely restate an ancestor's name (13.9% and 2.9% in the references), and in one, three of every four records fall under multiple top-level categories. We introduce two families of whole-tree metrics: structural discriminators test whether a tree's shape was learned from the data or imposed by its generator; team partitionability tests whether branches split feedback into groups teams can own. Trees the generic checks rate as equally correct differ by 27 percentage points in cross-branch leakage, and only one of the two beats a random split. Surface plausibility is an insufficient measure of taxonomy quality: evaluation must measure the whole tree as well as each node.
1 Introduction
Taxonomy usefulness depends on properties spread across the tree, but common checks judge individual nodes, edges, or sibling pairs. The paper shows that these checks miss systematic whole-tree failures in generated taxonomies and introduces metrics that expose them.
1 Introduction
- A customer-feedback taxonomy supports trusted volume counts and team ownership through properties of the whole tree, including label distinctness and how cleanly branches divide feedback.Source: §1 · p. 1
- Generic checks inspect one node, edge, or sibling pair at a time, so their aggregate pass rates cannot reveal patterns that exist across the corpus.Examples include names built from the same template and levels that never branch.Source: §1 · p. 1
- Across two corpora of 1,940 and 5,000 records, six taxonomies comprise one production reference and two independent harness runs per corpus.All six pass every generic naming and structure check.Source: §1 · p. 1
- 97.7–100% of generated leaf names restate an ancestor, and one generated taxonomy’s branch boundaries perform worse than a random partition.The generic checks detect none of these whole-tree failures.Source: §1 · p. 1
- Structural discriminators and team partitionability separate trees that generic checks rate comparably: two harness runs differ by 27 percentage points in branch leakage and flip sign in co-occurrence modularity.The metrics are deterministic and judge-free.Source: §1 · p. 2
2 Related Work
Prior work constructs taxonomies from corpora, feedback, and language models, and evaluates their quality using structural or semantic criteria. This section identifies a gap: existing approaches do not assess whether taxonomy branches separate records into groups suitable for organizational use.
2.1 Statistical and Corpus-Based Taxonomy Construction
- Corpus-based methods include TaxoGen’s recursive clustering (Zhang et al. 2018), HiExpan’s seed-taxonomy expansion, and TaxoExpan’s graph-neural expansion.Topic models such as latent Dirichlet allocation (Blei, Ng, and Jordan 2009) and BERTopic also organize unstructured records.Source: §2.1 · p. 2
2.2 Taxonomies for Customer Feedback
- Customer-feedback taxonomies organize reviews, surveys, app-store reviews, and support interactions, but prior work evaluates classification accuracy or topic coherence rather than organizational fitness.A review found a large, heterogeneous label space organized around sentiment, intention, user experience, and topic (dos Santos, Groen, and Villela 2019).Source: §2.2 · p. 2
2.3 LLM and Agentic Taxonomy Construction
- LLM and agentic systems construct or expand taxonomies, and repeated-run variance in agentic benchmarks has been reported (Bjarnason, Silva, and Monperrus 2026).This paper examines whether such variance concentrates in corpus-level properties missed by generic evaluation.Source: §2.3 · p. 2
2.4 Taxonomy Quality Evaluation
- Taxonomy evaluation tools mostly inspect nodes, edges, or node pairs, leaving aggregate ancestor-label redundancy, generator-shaped branching, and record partitionability unexamined.Word-type/token ratios and graph modularity are established measures, but had not, to the authors’ knowledge, been applied to taxonomy artifacts.Source: §2.4 · pp. 2–3
3 Experimental Setup
The study evaluates six taxonomies built over two proprietary customer-feedback corpora, using generic LLM-judged checks alongside two metric families that do not use a language model. The new families assess tree structure and how records are partitioned among branches.
3.1 Corpora and Artifact Construction
- The study evaluates proprietary customer-feedback corpora of 1,940 and 5,000 records, each accompanied by product documentation, and assembles three taxonomies per corpus, including the product’s existing production taxonomy.The production taxonomies are reference points, not controlled arms: they ran on different infrastructure without a matched budget.Source: §3.1 · p. 3
3.2 Evaluation Protocol
- The four generic checks use an LLM judge on individual nodes, edges, or sibling pairs to assess naming and structure.They include name-description match, description completeness, hierarchy correctness, and sibling distinctness.Source: §3.2 · p. 3
- Product-term coverage uses a stronger judge to test whether taxonomy nodes represent canonical terms extracted from product documentation.It measures documented product coverage, not internal tree soundness.Source: §3.2 · p. 3
- Structural discriminators use artifact-only operations, while team partitionability routes classified records through each tree to compute branch-assignment and leaf co-occurrence structures.The two metric families use no language model.Source: §3.2 · p. 3
- Across six artifacts and two corpora, every generated artifact outscores its reference on product-term coverage in both corpora; every artifact also passes every check, including the generic checks.Source: Table 1 · p. 4
4 Existing Metrics: What They Test and Where They Break
All six artifacts pass the generic checks, and generated trees often score higher on product-term coverage than their production references. Yet examples and whole-tree measurements reveal repeated ancestor text and generator-imposed branching that these local checks miss.
4.1 Generic Checks in Common Use
- Every artifact clears every generic check, and generated runs often outscore references on product-term coverage.Generated coverage is 0.74–0.86 against 0.72 on the small sample and 0.90–0.91 against 0.80 on the large sample.Source: §4.1 · p. 4
4.2 What the Table Does Not Show
- In C-1’s illustrated path, every descendant below L3 copies its name verbatim before appending one new word, despite a 0.97 name-description score.The names are stored this way in the taxonomy artifact.Source: §4.2 · p. 4, Figure 1 · p. 4
- 97.7% of C-1’s 867 leaves and 100% of C-2’s 599 leaves restate an ancestor name; the same pattern recurs on the large sample.The two references show ancestor-restating rates of 13.9% and 2.9%.Source: §4.2 · p. 4, Figure 1 · p. 4
- All 158 of C-2’s L2 parents have exactly one L3 child, although its hierarchy-correctness score is 0.92.Individual parent-child edges can look defensible while the overall branching shape is clamped by the generation process.Source: §4.2 · p. 4
- Both repeated labels and branching caps are whole-tree failures that persist despite high generic-check scores and better product-term coverage than the reference.Source: §4.2 · p. 4
5 Two New Metric Families
The paper introduces two judge-free metric families to evaluate whole-tree structure and whether feedback branches form separable groups. Structural discriminators use the taxonomy artifact, while team partitionability also uses record classifications.
5 Two New Metric Families
- The two metric families are computed directly from taxonomy artifacts, with record classifications additionally required for team partitionability.The proposed measures do not require an LLM judge or a gold taxonomy.Source: §5 · p. 5
5.1 Structural Discriminators
- Structural discriminators test whether tree structure was learned from data or imposed by the generator, using label redundancy, lexical diversity, fan-out shape, and cross-branch reuse.The measures are computed directly from the taxonomy artifact; they do not require an LLM judge or gold taxonomy.Source: §5.1 · p. 5, §5 · p. 5
- Generated artifacts restate an ancestor name in 97.7–100% of leaves, versus 13.9% and 2.9% in the references.Their type-token ratio also collapses 2.5–4.4× on the small sample and 2.0–2.7× on the large sample.Source: §5.1 · p. 5, Table 2 · p. 5
5.2 Team Partitionability
- Team partitionability routes record classifications through the tree and measures branch leakage, co-occurrence modularity, and workload balance through dividability.Positive modularity means branches hold together more tightly than a random split; dividability is contextual workload-balance information, not a quality signal by itself.Source: §5.2 · p. 6
- C-1 and C-2 differ by 27 percentage points in leakage and a modularity sign flip, despite hierarchy-correctness scores within 0.08.C-2's L1 modularity is negative, while positive modularity means branches hold together more tightly than a random split.Source: Figure 3 · p. 6, §5.2 · p. 6, Table 3 · p. 7
6 Discussion
The paper argues that taxonomy failures and usefulness are whole-tree properties that generic node-level checks cannot assess. Its metrics expose generated-tree problems that coverage checks can reward rather than penalize.
6 Discussion
- Generic checks miss ensemble-level failures because they inspect individual nodes, edges, or sibling pairs rather than distributions across trees and record assignments.The proposed closed-form metrics require no judge or LLM call.Source: §6 · p. 7
- Generated artifacts combine zero reuse with 35–76% leakage, while the references have 48 and 73 shared nodes and about 20% leakage.Source: §6 · p. 7
- Coverage checks can reward minting a node for each documented term, while more compute did not improve leakage or diversity in the heaviest runs.Source: §6 · p. 7
- The worst-leakage tree also has the highest small-sample product-term coverage, and generated trees have higher large-sample coverage than the reference.Small-sample coverage is 0.86 versus 0.72; large-sample coverage is 0.90–0.91 versus 0.80.Source: §6 · p. 7
7 Limitations and Future Work
The evaluation uses one construction harness, and the metrics have scope and validation limits. The proprietary corpora and artifacts are not released, preventing independent recomputation of the reported tables.
7 Limitations and Future Work
- Harness diversity is untested, and the cap-suspect threshold can falsely flag genuinely narrow branching.Source: §7 · p. 7
- The metrics lack validation against an external usefulness signal such as team velocity, so the paper does not claim they suffice alone.Leakage, purity, and modularity depend on each artifact’s record classification.Source: §7 · p. 7
- The proprietary corpora, artifacts, and documentation are unavailable, so Tables 2 and 3 cannot be independently recomputed.The paper specifies the metrics and protocol for reimplementation.Source: §7 · p. 7
8 Conclusion
The conclusion is that taxonomy evaluation must assess the tree and routed records as a whole, not only individual nodes. Across all four tested runs on two corpora, the proposed metric families detect failures generic judging misses.
8 Conclusion
- All four tested harness runs on two real corpora exhibit the same failure modes despite passing every generic check and beating the reference on product-term coverage.Source: §8 · p. 7
- Structural discriminators and team partitionability detect failures that generic judging cannot see, deterministically and without an LLM judge.Source: §8 · p. 7
- The paper concludes that taxonomy quality metrics must assess the whole tree and the records routed through it.Source: §8 · p. 7