PaperlayerView exact revision on arXiv

Source-linked AI summary

TIDE: Tree-Induced Fine-Grained Diagnostic Evaluation from Model Behaviors

Jiayi Chen, Hui Li, Yujiong Shen, Qiyuan Peng, Yanxin Li, Chenhao Huang, Shaofan Liu, Guoqiang Zhang, Junjie Ye, Shihan Dou, Ming Zhang, Maxm Pan, Tao Gui, Qi Zhang, Xuanjing Huang

arXiv:2610.29985v1cs.AIcs.LG

TL;DR

Aggregate benchmark scores offer limited information about LLM capabilities, motivating finer-grained diagnosis. TIDE induces behavior-aligned decision trees from benchmark samples and model scores; across five models and four benchmarks, it finds capability contrasts invisible in aggregate results.

  • Problem

    Aggregate benchmark scores provide limited information about the underlying capabilities of large language models.

    Source: §1 · p. 1

  • Method

    TIDE builds hierarchical capability trees by having an LLM propose candidate splits and using behavior-based information gain to select them.

    Source: §1 · p. 2

  • Results

    Across five models and four benchmarks, TIDE produces consistent weakness rankings, transferable diagnostic trees, and capability findings invisible at the aggregate level.

    Source: §1 · p. 2

  • Takeaways & Limitations

    TIDE extends aggregate scoring with finer-grained diagnostic information, and behavior-supervised capability trees may serve as a building block for diagnosing frontier models.

    Source: §6 · p. 10

    Experiments focus on static, single-turn English benchmarks with deterministic scoring; multi-turn dialogue, agentic settings, and open-ended generation remain unvalidated.

    Source: Limitations · pp. 10–11

Abstract

from arXiv · show

Benchmark performance is the dominant standard for comparing large language models (LLMs), typically summarized by a single aggregate score, such as accuracy, pass rate, or rubric score. This aggregate score provides limited information about a model's underlying capabilities, yet understanding these capabilities is crucial for model development and evaluation. To this end, we propose Tree-Induced Diagnostic Evaluation (TIDE), which recasts diagnostic evaluation as behavior-driven decision-tree induction to perform fine-grained capability diagnosis. TIDE recursively builds a hierarchical capability tree from benchmark samples and model scores: at each node, an LLM judge proposes candidate splits and performs sample-to-group classification, and behavior-based information gain selects among them. We apply TIDE to five models across four benchmarks to demonstrate its effectiveness, and find fine-grained capability contrasts invisible at the aggregate level: Claude Opus 4.6 achieves 100% on microeconomic market reasoning where GPT-5.4 scores 83.3%, but trails GPT-5.4 on contract law reasoning with 77.8% versus 88.9%. We hope TIDE can be extended to past and future benchmarks so that the rich behavioral signal in evaluation results is fully leveraged for understanding model capabilities.

1. Introduction

Aggregate benchmark scores provide limited information about models’ underlying capabilities, motivating diagnosis that uses evaluation behavior to discover fine-grained capability structure. TIDE uses behavior-based split selection to build capability trees and reveals contrasts not visible in aggregate scores.

1. Introduction

  • Aggregate scores reveal little about which capabilities a model has mastered or where its weaknesses lie, even when models have nearly identical scores.The introduction argues that evaluation results themselves should supervise fine-grained capability structure.Source: §1 · p. 1
  • TIDE frames fine-grained diagnosis as supervised structure discovery over model behavior, with an LLM proposing splits and behavior-based information gain selecting among them.The LLM is the hypothesis generator; the behavior matrix supplies supervision.Source: §1 · p. 2
  • Across five models and four benchmarks, TIDE produces consistent weakness rankings, transferable diagnostic trees, and capability findings invisible at the aggregate level.Worst-leaf accuracy drops reach up to 37.91, and ablations identify behavior-based selection as the key diagnostic signal.Source: §1 · p. 2

2. Related Work

Prior fine-grained diagnosis often relies on external metadata, fixed categories, or semantic grouping rather than model behavior as supervision. TIDE adapts decision-tree induction so behavior selects among semantic split rules.

LLM Benchmarking

  • Benchmark leaderboards rank models by aggregate scores that provide little information about mastered capabilities or weaknesses.Benchmarking spans knowledge and reasoning, instruction following, and head-to-head comparison.Source: §2 · pp. 2–3

Fine-Grained Capability Diagnosis

  • Existing fine-grained methods use external metadata, capability attributes, topic groups, checklists, or flat lists, whose granularity may not match model capability boundaries.Metadata may be unavailable, and flat-list methods can conflate category frequency with weakness.Source: §2 · p. 3
  • Recent hierarchical methods build trees from semantic similarity rather than model behavior as the supervision signal.EvalTree clusters embeddings top-down, while SkillVerse derives dendrograms from LLM critiques.Source: §2 · p. 3

3. Problem Formulation

The formulation treats benchmark scores as supervision for discovering a hierarchy of sample subsets, rather than relying on pre-existing sample features. Its split objective rewards both within-model consistency and cross-model separability.

3. Problem Formulation

  • The behavior matrix contains deterministic judge scores for each model–sample pair and is the sole supervision signal; samples have no pre-existing feature representations.Scores abstract the benchmark’s rubric, accuracy, or pass-rate definition.Source: §3 · p. 3
  • A tree node represents a sample subset, internal nodes carry human-readable split rules, and each leaf is identified by the conjunction of rules along its path.The hierarchy matches nested variation in sample semantics and model behavior.Source: §3 · p. 3
  • A behavior-based split objective scores candidate partitions by whether their children improve both intra-model consistency and inter-model separability.It follows the decision-tree information-gain template and instantiates it with two complementary impurities.Source: §3 · p. 3
  • Intra-model gain favors children where each model behaves more consistently, while inter-model gain favors splits that expose subsets with more pronounced model contrasts.The unified criterion trades consistency against separability; for one evaluated model, it reduces to intra-model gain.Source: §3 · pp. 4–5

4. Tree-Induced Diagnostic Evaluation

TIDE recursively induces a capability tree from benchmark samples, selecting sample-based partitions with behavior-based scores and balance constraints. It extracts root-relative weaknesses for one model or discriminative slices across multiple models.

4.1. Overview

  • TIDE recursively builds a capability tree by proposing candidate splits, scoring them with behavior-based gain and a balance regularizer, then recursing on the highest-scoring feasible split.The process begins with the full benchmark sample set and stops when its criteria are met.Source: §4.1 · p. 5

4.2. LLM-Generated Candidate Partitions

  • At each node, an LLM proposes candidate rules from sample text and reference answers, then classifies samples into groups without consulting scores until partition scoring.The proposer sees sample content rather than model outputs.Source: §4.2 · p. 5

4.3. Behavior-Based Split Selection

  • Behavior-based information gain ranks candidate partitions, while a balance regularizer discourages highly uneven groups and feasibility requires at least nmin samples per group.Gain alone favors splits that leave some groups too sparse to represent meaningful capability slices.Source: §4.3 · pp. 5–6

4.4. Recursive Tree Construction and Stopping Criteria

  • Recursion stops at maximum depth, when a node is smaller than nmin, or when no feasible split reaches the minimum effective-gain threshold.Source: §4.4 · p. 6

4.5. Diagnostic Extraction from the Induced Tree

  • Single-model weakness nodes fall significantly below the root mean, while multi-model discriminative slices exceed the root cross-model gap; the deepest qualifying node per branch is reported.The diagnostic statistic is the within-node mean for one model and the cross-model score gap for multiple models.Source: §4.5 · p. 6

5. Experiment

TIDE evaluates diagnostic trees across benchmarks and model comparisons, using held-out stability, weak-leaf drops, discrimination gaps, and transfer accuracy as measures. It reveals capability-level contrasts that aggregate scores obscure, while ablations identify behavior-based split selection as a key diagnostic signal.

5. Experiment

  • The evaluation uses four benchmarks, 3,000 sampled items per benchmark split evenly into dev and test, and six frontier models; trees are built using dev data only.MMLU is additionally used for cross-benchmark transfer, and the baselines include two flat and two hierarchical methods.Source: §5 · p. 6
  • The four diagnostic axes are held-out weakness consistency, weak-leaf drop, pairwise model-discrimination gaps, and cross-benchmark transfer accuracy.Weakness consistency compares dev- and test-set severity rankings; weak-leaf drop compares aggregate accuracy with accuracy on extracted weak leaves.Source: §5 · pp. 6–7

5.1. Main Results

  • TIDE ranks among the top two methods on held-out weakness consistency in most settings, reaching 0.94–1.00 on BBH and DROP while producing the largest weak-leaf drop.The comparison uses dev-to-test weakness rankings and the gap between aggregate accuracy and weak-leaf accuracy.Source: Table 1 · p. 7, §5.1 · p. 8
  • Across both MMLU-to-MMLU-Pro transfer directions, every model has a positive worst-leaf drop, ranging from 1.61 to 11.74 on the target benchmark.The source tree is applied without observing target-side examples; Table 2 reports target aggregate accuracy, weak-leaf accuracy, and their drop.Source: §5.1 · p. 9, Table 2 · p. 9
  • +117% to +684% relative gap gains show that TIDE’s leaf-level gaps exceed aggregate gaps across all four benchmarks, including a 25.6 leaf-level gap against a 3.3 aggregate gap on BBQ.Figure 4 compares Qwen-Plus with DeepSeek-V3.2.Source: §5.1 · p. 8, Figure 4 · p. 9

5.2. Model Insights via TIDE

  • With aggregate accuracies of 85.2% and 86.2%, GPT-5.4 and Claude Opus 4.6 still show distinct MMLU-Pro capability contrasts, including a 19.1 gap on anatomical structure reasoning.GPT-5.4 scores 61.9% and Claude Opus 4.6 scores 81.0% on that leaf; the figure also compares their per-leaf accuracies.Source: Figure 3 · p. 8, §5.2 · p. 9
  • With only a 0.7 aggregate gap on DROP, GPT-5.4 improves over GPT-5.2 on person and location extraction but declines on four other capability leaves, with a 41.2 gap on large-number computation.Figure 3 presents per-leaf accuracy contrasts for the two model versions.Source: Figure 3 · p. 8, §5.2 · pp. 9–10

5.3. Ablation Analysis

  • TIDE achieves weakness consistency ρ = 0.94 and weak-leaf drop Δ = 13.59; random selection retains similar ρ but reduces Δ to 1.53.Removing the balance regularizer lowers ρ to 0.55; the α sweep and gain-component ablation indicate that both intra-model consistency and inter-model separability are necessary.Source: §5.3 · p. 10

6. Conclusion

TIDE reframes fine-grained diagnostic evaluation as supervised structure discovery over model behavior and builds capability trees using LLM-proposed splits selected by behavior-based information gain. Across five frontier models and four benchmarks, it produces consistent weakness rankings, transferable trees, and findings invisible to aggregate scores; ablations identify behavior-based selection as the key diagnostic signal.

  • Across five frontier models and four benchmarks, TIDE produces consistent weakness rankings, transferable diagnostic trees, and capability findings invisible at the aggregate level.Source: §6 · p. 10
  • TIDE uses an LLM judge to propose candidate split rules and behavior-based information gain to select among them.Source: §6 · p. 10
  • The ablation analysis identifies behavior-based selection as the key diagnostic signal.Source: §6 · p. 10

Limitations

TIDE’s practical limitations include repeated LLM-call costs and evaluation restricted to static, single-turn English benchmarks scored deterministically. Its algorithm recursively selects valid behavior-based splits and builds child trees until stopping conditions apply.

Limitations

  • Repeated LLM calls for split-rule generation and sample classification at every node incur non-trivial API cost.Parallel classification and caching mitigate cost, and an induced tree can be reused across additional models without re-induction.Source: Limitations · p. 10
  • Experiments use static, single-turn English benchmarks with deterministic scoring, leaving multi-turn dialogue, agentic, and open-ended generation settings unvalidated.The framework is benchmark-agnostic, and richer signals such as per-turn rewards or rubric-based scores are proposed as future extensions.Source: Limitations · p. 11

A. Algorithm

  • The recursive algorithm takes the current node’s samples, behavior matrix, depth, and hyperparameters, then selects a valid split and recursively builds child trees.Candidate gain combines intra-model and inter-model gains using α and 1 − α.Source: A. Algorithm · p. 15

B. Why Behavior, Not Error Frequency

Behavior-based information gain distinguishes model-specific weakness from subsets that are merely uniformly difficult, preventing error frequency alone from driving diagnostic splits.

  • When all models fail on the same samples, behavior-based gain stays near zero, so TIDE avoids descending into branches that reflect uniform difficulty rather than model-specific weakness.The gain measures behavior variation within and across models on candidate split children, not the number of errors in a subset.Source: B. Why Behavior, Not Error Frequency · p. 15

C.1. Baseline Implementations

The baselines use different strategies to construct weakness profiles, ranging from embedding-based trees and text comparisons to flat capability categories and critique-derived skill hierarchies. Their profiles are evaluated using held-out test instances under a shared development/test protocol.

Baseline setup

  • The shared protocol splits each dataset evenly, using the first half for development-set weakness identification and the second half for held-out testing.Source: §C.1 · p. 15

EvalTree

  • EvalTree (Zeng et al., 2025) recursively clusters instance embeddings into a capability tree, then ranks low-accuracy development-set nodes as weaknesses.Test accuracy is computed on instances linked to each identified node.Source: §C.1 · pp. 15–16

TextDiff

  • TextDiff (Zhong et al., 2022) contrasts failed and successful cases, assigns development instances to identified weakness capabilities, and ranks them by development accuracy.Each capability is evaluated on its correspondingly labeled test instances.Source: §C.1 · p. 16

QualEval

  • QualEval derives flat capability categories with an LLM, assigns instances subject to coverage constraints, and ranks all discovered capabilities by development accuracy.Its ranking spans every discovered capability rather than only a filtered weakness subset.Source: §C.1 · p. 16

SkillVerse

  • SkillVerse clusters critique-derived atomic skill judgments into a dendrogram and ranks below-threshold proficiency nodes as weaknesses.Proficiency is estimated from favorable critiques; test accuracy uses instances mapped to each node via embedding similarity.Source: §C.1 · p. 16

C.2. TIDE Implementation

TIDE’s implementation uses fixed default hyperparameters, confidence-interval tests for diagnostic extraction, and cached, parallelized LLM stages to construct trees.

C.2.1. Hyperparameter Settings

  • Unless otherwise stated, the implementation uses d_max=10, n_min=100, g_min=10^-3, C=10, α=0.5, and β=1.Source: §C.2.1 · p. 16
  • TIDE’s sample classifier assigns evaluation samples to groups under a specified splitting rule.Source: Figure 5 · p. 17
  • With tolerance τ = 0, a single-model node is weak if the upper confidence bound on s(Bv) is below s(Broot), while a multi-model node is discriminative if the lower confidence bound on g(Bv) exceeds g(Broot).Source: §C.2.1 · p. 17

C.2.1. Hyperparameter Settings

  • Tree construction proceeds breadth-first, while depth-first traversal was verified to yield the same tree and affect only compute scheduling.Source: §C.2.1 · p. 17

D.1. Models and Datasets

The evaluation covers four benchmarks and five frontier model families, using development-only tree construction and held-out testing; GPT-5.2 is the default TIDE backbone.

D.1. Models and Datasets

  • The four benchmarks span knowledge, reasoning, reading comprehension, and social-bias regimes: MMLU-Pro, BBH, DROP, and BBQ.Source: §D.1 · p. 17
  • Each benchmark contributes 3,000 items split evenly into development and held-out test sets; methods build trees or weakness profiles on development data only.Trees are reused across evaluated models, and HLE-Verified is reserved for single-model qualitative analysis.Source: §D.1 · p. 17
  • The five evaluated model families are Grok 4.20, Claude Opus 4.6, DeepSeek-V3.2, GPT-5.4, and Qwen-Plus.Source: §D.1 · p. 17
  • TIDE’s split-rule proposer and sample classifier use GPT-5.2 as the backbone unless otherwise stated.Source: §D.1 · p. 18

D.2. Additional Analysis

This appendix section provides additional TIDE analyses and materials, including single-model capability-slice variation, sample-assignment prompts, and task distinctions across domain questions, structured reasoning, exact-output formats, and translation-error classification. It also identifies tables and examples documenting diagnostic leaf paths, accuracies and model gaps, and a representative tree’s node descriptions.

D.2. Additional Analysis

  • For both Qwen3.6-Plus and Gemini-3.1-Pro on HLE-Verified, the weakest leaf falls well below aggregate accuracy, with wide leaf-level spreads and interpretable weak capability slices.For Qwen3.6-Plus, examples include references-heavy abstract physics and unit-conversion items.Source: §D.2 · p. 18

D.3. Case Studies

  • Representative root-to-leaf paths localize weaknesses hidden by aggregate scores; the appendix tables provide node descriptions and examples alongside paths and per-leaf accuracy.The single-model tables show tree nodes and corresponding paths and accuracy; multi-model tables also report per-leaf accuracy gaps.Source: §D.3 · p. 18

EvalTree Prompt: Capability Annotation; QualEval Prompt: Stage 1 – Capability Discovery; QualEval Prompt: Stage 2 – Relevance Scoring

TextDiff Prompt: Weakness Generation; TextDiff Prompt: Instance Annotation

SkillVerse Prompt: Stage 1 – Critique Generation; SkillVerse Prompt: Stage 2 – Atomic Judgment

TIDE Prompt: Splitting Rule Discovery

  • TIDE’s split-rule prompt asks for sample partitions based on inherent question properties that may correlate with performance, with clear commonality within groups and distinguishable differences between them.Suggested split dimensions include ability type, domain, cognitive level, strategy, complexity, knowledge source, answer format, and constraint structure.Source: TIDE Prompt: Splitting Rule Discovery · p. 25

TIDE Prompt: Sample Assignment

  • TIDE’s sample-assignment prompt classifies each evaluation sample into a group under a specified splitting rule.The appendix includes examples of rules and samples from the representative tree.Source: TIDE Prompt: Sample Assignment · p. 26
  • The prompt taxonomy also covers multiple-choice selection, exact-form artifacts, action choice, causal explanations, entity diagnosis, quantity computation, and validity evaluation.Sample prompts range from piano fingering and a multiple-choice language question to physics and scientific reasoning problems.Source: TIDE Prompt: Sample Assignment · p. 31
  • Table 5 lists TIDE leaf paths and their accuracies, while Table 6 describes nodes and gives an example sample from a representative multi-model tree.The captions specify each table’s scope but do not provide individual accuracy values in these passages.Source: Table 5 · p. 33, Table 6 · p. 34
  • TIDE distinguishes causal or intentionality judgments, truth/lie puzzles, Boolean evaluation, and sports plausibility as separate task types.The prompt taxonomy also includes examples of a causal question, a truth puzzle, Boolean expression evaluation, and a sports-plausibility judgment.Source: TIDE Prompt: Sample Assignment · p. 35
  • The assignment taxonomy separates bracket completion, alphabetical sorting, non-temporal questions, and temporal questions involving date or time interpretation.It illustrates bracket completion and word sorting with sample prompts.Source: TIDE Prompt: Sample Assignment · p. 35
  • Structured reasoning categories include fixed-order constraints, sequential swaps, table lookup or aggregation, and temporal feasibility.The examples include ranking objects, tracking exchanged partners, counting from a table, and finding an available time window.Source: TIDE Prompt: Sample Assignment · p. 36
  • Additional categories distinguish semantic or linguistic knowledge, state tracking, ordering constraints, similarity recommendation, table aggregation, and temporal feasibility.The taxonomy also names pronoun/coreference disambiguation, translation error classification, and ordering or state-tracking tasks.Source: TIDE Prompt: Sample Assignment · pp. 36–38
  • Table 7 lists every diagnostic-tree leaf, its root-to-leaf path, and the accuracy gap between two models.The supplied caption gives the table’s scope but does not provide leaf-level values.Source: Table 7 · p. 41

Related papers

Loading 2610.29985v1…