Hierarchy-Aware Supervised Uncertainty Estimation for Black-box LLM Taxonomic Reasoning

Motivation

• Black-box LLM confidence estimation remains difficult when hierarchical outputs span multiple semantic levels and abstention produces partial outputs.Reliable uncertainty quantification is especially important in high-stakes decision-support settings.Source: §1 · p. 1
• Long-tailed biodiversity taxonomies make rare and endangered species difficult to monitor, while qualitative abstentions lack numerical confidence for principled rejection thresholds.The motivating pipeline combines VLM image captioning, retrieval-augmented generation, and LLM reasoning across taxonomic ranks.Source: §1 · p. 1

Study focus

• The study investigates uncertainty estimation for hierarchical taxonomic reasoning using only externally accessible signals from black-box LLMs.This frames the work around uncertainty estimation rather than improving the upstream reasoning pipeline.Source: §1 · p. 1

Formulation

• The paper casts black-box LLM taxonomic uncertainty estimation as rank-level selective prediction under partial-path abstention.The formulation reflects variable-depth outputs in which the system may abstain at an uncertain rank.Source: §1 · p. 2

Method

• Lightweight supervised estimators use proxy-LLM features, hierarchy-aware supervision, and rank-aware output parameterizations to predict rank-wise correctness.These design choices align the estimator with the hierarchical structure of the outputs.Source: §1 · p. 2

Result

• Supervised estimators outperform a token-likelihood baseline for global-threshold selective prediction, with a rank-specific multi-head design performing best overall.The comparison is conducted on a realistic long-tailed arthropod pipeline.Source: §1 · p. 2

LLM uncertainty estimation

• White-box uncertainty methods use logits or hidden states, but these signals are unavailable for proprietary API models.Black-box alternatives commonly use self-verbalized confidence or consistency and semantic clustering across sampled responses.Source: §2 · p. 2
• Self-verbalized confidence may not faithfully reflect intrinsic uncertainty, while repeated-response consistency or semantic clustering can be prohibitively expensive.These constraints motivate uncertainty estimation methods that operate with externally accessible signals.Source: §2 · p. 2

Hierarchical taxonomic prediction

• Taxonomic reasoning complicates uncertainty estimation because predictions are hierarchical, long-tailed, and often partially specified through abstention.Prior hierarchical-classification work uses ancestor overlap and partial credit, and recent work extends this perspective to taxonomy-aware evaluation.Source: §2 · p. 2
• The paper complements upstream taxonomic-reasoning work by estimating uncertainty for hierarchical outputs with hierarchy-aware signals as auxiliary supervision.Its focus is uncertainty estimation rather than improving the upstream reasoning pipeline.Source: §2 · p. 2

Pipeline

• The pipeline maps target-LLM taxonomic outputs and tool-LLM proxy features to confidence scores for uncertainty estimation.Source: Figure 1 · p. 2, §3 · p. 2

Supervision labels

• Each instance receives binary correctness supervision and hierarchical accuracy based on taxonomic overlap, with constant kingdom and phylum ranks excluded.Source: §3.2 · p. 3

Proxy features

• Tool LLMs provide representation features from hidden states and distributional features based on entropy, negative log-probability, and token probability.Source: §3.3 · p. 3

Supervised estimators

• H3 uses five rank-specific binary heads plus an auxiliary regression head, whereas H1 and H2 use shared binary-head designs with progressively richer supervision.Source: §3.4 · p. 3

Evaluation protocol

• Rank-level evaluation treats each image–rank pair as one example with a correctness label and confidence score.Source: §3.6 · p. 4
• Global-threshold risk–coverage pools validation examples across ranks, while AUROC is reported as both pooled micro AUROC and per-rank-average macro AUROC.Source: §3.6 · p. 4

• Supervised estimators consistently and substantially outperform the mean-NLL baseline in micro AUROC across all three tool LLMs.Source: §4 · p. 4
• H3 achieves the highest mean micro AUROC overall, with a small margin over H1 for Qwen3 and clearer gains for Gemma 7B and GPT-OSS 20B.Source: §4 · p. 4
• Under a single global rejection threshold, supervised estimators achieve lower selective risk than the mean-NLL baseline across a broad coverage range.The largest gains occur at low-to-mid coverage.Source: §4 · p. 5
• H3 consistently yields the best overall risk–coverage trade-off, while H1 and H2 remain better than the baseline throughout.Source: §4 · p. 5
• Gains are less consistent under rank-balanced macro AUROC, although calibration results generally favour supervised estimators over the mean-NLL baseline.Source: §4 · p. 5

• Rank imbalance makes micro AUROC favor high-coverage coarse ranks, whereas macro AUROC weights ranks equally and exposes weaknesses at deeper ranks.Frequent abstentions reduce fine-rank training signal and increase variance.Source: §5 · p. 5
• H3 performs best overall for unified-threshold deployment, while H2 provides modest gains over H1 for most tool LLMs except GPT-OSS 20B.Source: §5 · p. 5
• Because H2 and H3 share auxiliary supervision, their pattern suggests rank-specific output heads matter more than the auxiliary hierarchical target alone.Source: §5 · p. 5
• The study is limited by sparse fine-rank supervision, a single taxonomic reasoning pipeline, and a global abstention threshold.Future work should test richer fine-rank coverage, wider distribution shifts, and rank-specific decision rules.Source: §5 · p. 5

• The study addresses UQ for black-box LLMs in hierarchical taxonomic reasoning with long-tailed data and partial-path abstention.Source: §6 · p. 5
• Lightweight supervised estimators use proxy features from a tool LLM and hierarchy-aware signals to predict rank-wise correctness.Source: §6 · p. 5
• Across tool LLMs, supervised estimators outperform an NLL baseline in micro discrimination and global-threshold selective prediction.Source: §6 · p. 5
• A rank-specific multi-head design performs best overall, highlighting estimator design for hierarchical outputs in unified-threshold deployment.Source: §6 · p. 5

• MLPs use Adam with learning rate 10^-3, weight decay 3 × 10^-4, batch size 128, and up to 200 epochs.Source: A Implementation details · p. 7
• Early stopping monitors validation micro AUROC with patience 5 and minimum improvement 10^-4.Source: A Implementation details · p. 7
• Hierarchical supervision uses λ = 0.1, while an L1 penalty of 10^-4 encourages sparse input-to-hidden weights.Source: A Implementation details · p. 7
• Feature selection is applied only to hidden-state representation features, combining the top-100 features from each listed selection method.Source: A Implementation details · p. 7

• Table 2 reports rank coverage and prediction accuracy before filtering.Source: Table 2 · p. 7

• Table 3 lists each tool LLM and its exact checkpoint identifier.Source: Table 3 · p. 7
• Tool LLMs extract representation, distribution, and hybrid feature types.Source: Figure 3 · p. 7

• Table 4 reports all-rank aggregated macro AUROC across tool LLMs.Results are reported as mean ± std over 5 random seeds.Source: Table 4 · p. 8
• Figure 4 reports per-rank AUROC before calibration across 5 random seeds.Horizontal bars indicate ±1 standard deviation.Source: Figure 4 · p. 8
• Table 5 reports calibration metrics before and after temperature scaling.Entries are mean with standard deviation across seeds, and arrows indicate whether lower or higher values are preferred.Source: Table 5 · p. 8