PaperlayerView exact revision on arXiv

Source-linked AI summary

Capturing LLM Capabilities via Evidence-Calibrated Query Clustering

Fangzhou Wu, Sandeep Silwal, Qiuyi Zhang

arXiv:2605.17110v2cs.AIcs.LG

TL;DR

Existing semantic taxonomies and embeddings often misalign query clusters with the latent capabilities needed for LLM evaluation. ECC calibrates semantic embeddings with limited posterior model comparisons, using Bradley-Terry capability profiles and soft responsibilities for mixed demands. It improves unseen-query ranking quality over human-labeled and embedding-only clustering and supports downstream query routing and sample-efficient model ranking.

  • Problem

    Semantic taxonomies and embeddings can misalign query groups with latent capability demands, while hard partitions poorly represent queries requiring mixed capabilities.

  • Method

    ECC jointly learns semantic centroids, Bradley-Terry capability profiles, and soft responsibilities by combining prior embeddings with limited pairwise model comparisons.

  • Results

    17.64 percentage points over human-labeled clustering and 18.02 percentage points over embedding-only clustering are ECC’s average unseen-query ranking-quality improvements.

  • Takeaways & Limitations

    ECC yields capability-aware clusters and improves downstream query routing and sample-efficient new-model ranking under limited comparison budgets.

  • Takeaways & Limitations

    The evaluation uses text-only standalone queries and does not assess multimodal inputs or conversational and historical context.

Abstract

from arXiv · show

Query clustering organizes queries into groups that reflect shared latent capability demands, enabling capability-aware LLM evaluation. Existing clustering methods, which primarily rely on semantic taxonomies or embeddings, often fail to capture such latent capability requirements due to a misalignment between surface-level semantics and actual model performance. We propose ECC, an algorithm that calibrates prior semantic embeddings using limited posterior model comparisons to bridge the gap between surface-level semantics and latent capability requirements. ECC characterizes each cluster through a capability profile parameterized by a Bradley-Terry model and uses trainable mixture weights to accommodate queries with mixed capability demands, jointly learning a flexible, capability-aware clustering structure that supports query-specific inference of LLM capabilities. Extensive quantitative and qualitative evaluations demonstrate that ECC significantly improves LLM capability ranking quality, outperforming human-labeled and embedding-based baselines by an average of 17.64 and 18.02 percentage points, respectively, and proves effective in downstream tasks such as query routing.

1 Introduction

Existing query clusters based on human taxonomies or embeddings can misrepresent the latent capabilities queries require. ECC addresses this gap by combining limited model comparisons with semantic embeddings to support capability-aware clustering and inference.

  • Motivation: Semantic clusters can group queries with very different capability demands or separate queries requiring the same underlying capabilities.These mismatches can limit or degrade capability estimates for unseen queries.
  • Motivation: Hard partitions derived from explicit skill labels do not naturally accommodate queries requiring mixtures of capabilities.Such approaches may also require evaluating each query across numerous models, creating substantial computational cost.
  • ECC: ECC integrates limited pairwise model comparisons with prior embeddings and jointly learns cluster centroids, capability profiles, and soft responsibilities.Each capability profile is parameterized by a Bradley-Terry model, while responsibilities quantify how strongly a query aligns with each cluster.
  • ECC: ECC uses a single pairwise model comparison alongside the query embedding to infer a query-specific, capability-aware ranking for unseen queries.This probe-informed mixture inference extends the learned soft cluster structure to new queries.
  • Results: 17.64 percentage points over human-labeled clustering and 18.02 percentage points over embedding-only clustering are the average ranking-quality improvements reported for unseen queries.ECC is also reported to improve routed-query response quality by 16.6% and new-model ranking quality by 16.3 percentage points under limited comparison budgets.

2 Methodology

ECC jointly calibrates semantic structure with posterior model-comparison evidence, representing clusters through capability profiles and queries through soft mixtures. Alternating optimization and probe-informed inference allow the learned structure to support capability-aware rankings.

  • Motivation: ECC addresses the gap created when posterior capability evidence is used only after embedding-based clustering.By incorporating model-comparison evidence during clustering, ECC allows capability information to influence cluster formation.
  • Cluster Representation: Each cluster combines a semantic centroid with a Bradley-Terry capability profile representing model strengths on associated queries.Together these components form the cluster-level system parameters.
  • Soft Clustering: Soft responsibilities let each query contribute to multiple clusters and combine cluster capability units into a query-specific representation.This design models queries that involve multiple capability demands rather than forcing a single hard assignment.
  • Optimization: ECC alternates between updating system parameters and updating query responsibilities until the learned structure stabilizes.Parameter updates include posterior-weighted semantic centroids and weighted Bradley-Terry capability profiles.
  • Optimization: ECC jointly optimizes semantic and comparison-based objectives with entropy-regularized soft responsibilities.The embedding loss measures semantic mismatch, while the comparison loss measures how well a cluster capability profile fits pairwise evidence; λ balances the two signals.
  • Inference: A single probe comparison calibrates responsibilities for an unseen query before ECC forms its weighted mixture ranking.The probe uses the same entropy-regularized update and computes comparison loss on that single comparison.
  • Convergence: ECC’s objective values are monotonically non-increasing and converge to a finite limit under the stated initialization assumption.This is the paper’s convergence guarantee for the alternating procedure.

3 Evaluation

ECC is evaluated across diverse benchmarks and inference settings, where comparison-calibrated clustering improves ranking quality and produces more coherent, separated capability partitions than semantic baselines. Qualitative analyses show that ECC splits semantically similar queries with divergent capability demands and merges semantically different queries with aligned demands.

  • Main Results: ECC improves average ranking quality gains by 17.64 pp over human-labeled clustering and 17.99 pp over embedding-only clustering across three datasets.Human-defined taxonomies provide only marginal gains over the global BT model and degrade quality on MMLU.
  • Clustering Signals: 18.02 pp is ECC’s average improvement over embedding-only clustering under Emb+Probe-infer at K=30, compared with 15.40 pp under Probe-infer.ECC remains stronger than embedding-only clustering even when inference uses embeddings alone.
  • Inference Signals: 19.06 pp and 5.64 pp are ECC’s Emb+Probe-infer gains over Emb-infer and Probe-infer, respectively.Combining query embeddings with one probe comparison provides the strongest inference signal for ECC.
  • Partition Structure: Comparison-based clustering yields higher within-cluster BT consistency, lower inter-cluster profile similarity, and larger cross-cluster ranking quality gaps.These measures indicate more coherent clusters internally and better separation across clusters.
  • Partition Structure: At K=20, ECC retains a coarse embedding scaffold while aligning more closely with comparison-only clustering, with ARI=0.108, NMI=0.221, and ARI=0.187, NMI=0.325, respectively.The concentrated overlaps reflect systematic repartitioning rather than arbitrary query reshuffling.
  • Qualitative Analysis: ECC separates queries within fixed embedding topics when capability demands diverge and merges semantically different queries when their capability demands align.Examples include separating planning layered onto mechanistic reasoning and merging reaction/biochemistry with materials/device-property queries requiring parameter-to-outcome causal modeling.
  • Applications: ECC is applied to guided optimal query routing and sample-efficient new model ranking as downstream capability-aware applications.The routing task selects the top-ranked model for each query, while the ranking task varies a limited budget of new-model comparisons.

4 Conclusion and Limitations

ECC calibrates semantic embeddings with limited model-comparison evidence to learn flexible, capability-aware clusters for query-specific LLM inference. Evaluations report improved ranking quality and downstream effectiveness, while the study is limited to text-only, standalone queries without multimodal or conversational context.

  • Conclusion: ECC calibrates prior semantic embeddings with limited posterior model comparisons to align clustering with latent capability requirements.Clusters use capability profiles and trainable responsibilities to support mixed capability demands.
  • Conclusion: ECC accommodates mixed capability demands through capability profiles and trainable responsibilities, enabling query-specific LLM capability inference.The paper validates this structure through quantitative and qualitative evaluations and downstream applications.
  • Limitations: The evaluation uses text-only queries and does not assess multimodal inputs or conversational and historical context.The authors identify context-aware extensions as future work.

A Experimental Details

The experiments span three multi-LLM benchmarks, compare semantic and comparison-based clustering baselines, and evaluate several inference signals and ranking-oriented metrics. Additional analyses examine ranking gains, partition agreement, and the relationship between clustering granularity and performance.

  • Benchmarks: The evaluation uses SPROUT, RouterBench, and Open LLM Leaderboard v2, covering diverse datasets including MMLU, MMLU-Pro, and MATH.The benchmarks contain responses from 13, 11, and 16 LLMs, respectively.
  • Baselines: Human-labeled and embedding-only clustering serve as semantic baselines, while Comparison-only isolates the contribution of posterior comparison signals.The baselines represent prior semantic clustering structures.
  • Inference Signals: Emb-infer, Probe-infer, and Emb+Probe-infer estimate rankings using embeddings, one probe comparison, or both signals.Emb+Probe-infer is the default inference setting.
  • Metrics: Ranking quality gains measure reductions in unseen-query pairwise BT loss relative to a global BT model without clustering.Larger gains indicate better alignment between the clustering structure and the underlying capability distribution.
  • Scaling Analysis: Figure 7 examines ranking quality gains across benchmarks, inference signals, and cluster counts K, with ECC consistently outperforming embedding-only clustering.With probe-based inference, gains rise rapidly at small K, saturate, and slightly decline at very large K because cluster-specific BT estimates become noisier.

B Additional Experimental Results

Additional experiments show that comparison-aware clustering produces more coherent and better-separated capability partitions than embedding-only clustering. ECC also maintains strong ranking quality across cluster granularities and outperforms P2L on unseen queries.

  • Cluster Consistency and Separation: Comparison-aware methods yield higher within-cluster BT profile consistency than embedding-only clustering across all three benchmarks.Embedding-only clusters mix heterogeneous capability demands, making cluster-specific profiles less stable across query subsets.
  • Cluster Consistency and Separation: Comparison-aware partitions achieve lower inter-cluster BT profile similarity and larger cross-cluster ranking quality gaps than embedding-only partitions.These measures indicate stronger separation in capability space.
  • Cluster Consistency and Separation: Posterior model-comparison evidence makes clusters more coherent within clusters and more separable across clusters, better reflecting query capability distributions.The results support incorporating comparison evidence during clustering rather than relying only on semantic signals.
  • Ranking Quality Across K: ECC remains the best-performing method as K varies from 2 to 100, while embedding-only inference gains little from increasing cluster granularity.Under Probe-infer and Emb+Probe-infer, gains rise rapidly at small K before saturating and slightly declining at very large K.
  • Comparison with P2L: ECC consistently achieves higher capability-aware ranking quality gains on unseen queries than P2L across all three benchmarks.P2L uses an LLM-based predictor to output query-conditioned Bradley-Terry coefficients from human-judged comparisons.

B.4 Cluster Overlap Analysis

Cluster-overlap analyses show that ECC preserves a coarse semantic scaffold while refining it according to latent capability demands. It splits semantically similar queries when their capability requirements differ and merges semantically distinct queries when those requirements align.

  • Partition Structure: ECC aligns more strongly with comparison-only clustering than with embedding-only clustering, with ARI=0.187, NMI=0.325 versus ARI=0.108, NMI=0.221.The overlap structure indicates that ECC retains capability-relevant organization captured by posterior comparisons while preserving part of the embedding scaffold.
  • Within-Embedding Splits: Within a fixed embedding topic, posterior comparisons separate queries requiring mechanistic reasoning from those adding constraint-aware design, procedure planning, or simulation.This refinement repartitions semantically close queries into more capability-aligned groups.
  • Cross-Embedding Merges: Across embedding clusters, ECC merges semantically different queries when their underlying capability demands align.Examples span biomedical or chemical decision-making and materials or process optimization despite their topical shift.
  • Capability Coherence: The combined split-and-merge pattern makes ECC clusters more capability coherent and better suited for capability inference.ECC splits embedding-aligned queries with divergent latent demands while merging embedding-separated queries with aligned demands.
  • Partition Structure: ECC preserves a coarse embedding-based organization while adjusting cluster boundaries using model-comparison signals.At K = 8, comparison-only clustering is visibly less aligned with embedding geometry and forms mixed clusters.

B.6 Ablation Studies

Ablations show that ECC benefits from moderate comparison evidence, balanced embedding and comparison signals, and accurate low-temperature inference probes. Its gains remain robust to probe selection, embedding backbones, tied comparisons, and distribution shifts.

  • Clustering Evidence and Hyperparameters: ECC ranking quality gains improve rapidly from 2 to 24 clustering comparisons before diminishing returns and a slight decline at the highest budgets.Moderate comparison evidence is more effective than either very sparse or excessive comparison budgets.
  • Clustering Evidence and Hyperparameters: ECC performs best at an intermediate λ, whereas over-emphasizing embeddings or model comparisons reduces capability-aware ranking quality.λ controls the relative weight between embeddings and model-comparison evidence, with larger values favoring embeddings.
  • Clustering Evidence and Hyperparameters: Using more clustering queries improves ranking quality for all methods, while ECC leads even when only 20% of queries are used.ECC is relatively robust to the fraction of queries allocated to clustering.
  • Clustering Evidence and Hyperparameters: ECC remains strongest in the low-temperature regime, while performance degrades rapidly as T increases because posterior signals become over-smoothed.Comparison-aware methods retain high performance across a broader range of small T values than embedding-only clustering.
  • Inference-Time Robustness: Increasing probe comparisons from 1 to 6 improves ECC ranking quality, while randomizing probe labels consistently degrades ECC and comparison-only performance.These results show that accurate posterior evidence at inference time is useful for query-specific capability estimation.
  • Robustness: ECC consistently leads across probe selections, embedding backbones, tied-comparison settings, and out-of-distribution queries.The robustness evaluations cover three benchmarks, alternative encoders, tie-aware comparisons, and MMLU-Pro-to-MATH distribution shift.

B.8 Robustness to Different Judges

ECC’s ranking gains remain consistent when evaluated with different judges. The method also supports query routing by combining a probe comparison with the query embedding to infer a query-specific ranking.

  • Judge Robustness: ECC consistently outperforms embedding-only clustering on MMLU-Pro under different evaluation judges.The cross-judge experiment uses a shared task appearing in SPROUT and Open LLM Leaderboard v2.
  • Judge Robustness: The judge-robustness evaluation tests whether ECC’s gains persist across different response-scoring settings.This directly examines evaluator choice as a source of variation in reported ranking quality gains.
  • Query Routing: For optimal query routing, ECC combines one probe comparison with the query embedding to infer a query-specific capability-aware ranking.The inferred ranking is used to route each query to the model expected to yield the highest response quality.

B.10 Sample-Efficient New Model Ranking

ECC improves sample-efficient ranking of a newly arrived model under limited comparison budgets, consistently outperforming embedding-only clustering across three benchmarks.

  • Evaluation setup: ECC ranks a held-out new model using a limited comparison budget, with one new-model comparison per selected query.Clusters are learned from pairwise comparisons among existing models before evaluating held-out new-model comparisons.
  • Results: Figure 20 evaluates ranking-quality gains for sample-efficient new-model ranking across three benchmarks under a limited comparison budget.The comparison is against embedding-only clustering.
  • Evaluation setup: 10 to 100 comparisons were tested for the new model, while old-model comparisons were limited to at most 3 per query.Results report median ranking-quality gains across new-model runs on three benchmarks.
  • Results: ECC consistently outperformed embedding-only clustering and achieved substantial ranking-quality gains under small comparison budgets.ECC was comparable to comparison-only clustering in this evaluation.

C Theoretical Proofs

The theoretical analysis shows that ECC’s alternating updates decrease its objective monotonically and converge to a finite limit, while its subproblems admit tractable optimization steps.

  • Convergence guarantee: ECC’s objective values are monotonically non-increasing and converge to a finite limit when the objective is finite at initialization.Boundedness below supplies the finite-limit conclusion.
  • Optimization procedure: ECC alternates between optimizing responsibilities and model parameters as coordinate descent on its objective.The responsibility update is convex and has a unique closed-form Boltzmann minimizer.
  • Optimization procedure: The responsibility update used in Algorithm 1 is the analytical minimizer and therefore does not increase the objective.The proof states L(P(t+1), Θ(t)) ≤ L(P(t), Θ(t)).
  • Optimization procedure: The cluster-center update has a closed-form minimizer when v_k ≠ 0, with c_k aligned to v_k.The supporting inequality uses Cauchy–Schwarz and normalized vectors.
  • Optimization procedure: Each cluster’s parameter update is convex in θ(k), so gradient-based optimization finds the global minimum for that subproblem.The convexity follows from the log-sum-exp form and nonnegative weighted sums.
  • Practical caveat: Tied comparisons can weaken capability signals because they provide less discriminative evidence and may reflect evaluator uncertainty or noise.The paper suggests stronger evaluators or stricter judge prompts to reduce tie frequency.

D.2 Evaluation Judges

ECC uses a designated LLM-based judge or fixed evaluation protocol for each benchmark to obtain consistent pairwise preference signals.

  • Evaluation judges: Each benchmark uses its designated LLM-based judge or fixed evaluation protocol to obtain consistent pairwise preference signals.Human judgments can vary across annotators in preference and reliability, introducing noise and systematic bias; large-scale collection is also more expensive.

D.3 Evaluation Efficiency of ECC

ECC reduces evaluation cost by learning capability-aware clusters from limited model comparisons rather than exhaustive response judging. Its lightweight optimization and constant-cost probing support practical query-specific LLM ranking.

  • Efficiency: ECC alternates between updating Bradley-Terry capability profiles and soft responsibilities using efficient CPU operations.The method is described as lightweight and not requiring dense compute such as GPU training.
  • Efficiency: 32 iterations on SPROUT and RouterBench and 29 on Open LLM Leaderboard show rapid convergence in the main K = 30 setting.These iteration counts are reported for ECC's main evaluation setting.
  • Evaluation cost: Around three pairwise comparisons per query already yield strong clustering performance, reducing the evidence needed for evaluation.ECC uses limited comparisons instead of judging responses from all models for every query.
  • Inference cost: One probe comparison for an unseen query adds O(1) overhead that does not scale with the number of candidate LLMs.The probe can be obtained with lightweight models because its signal is reported as robust to probing-model choice.
  • Practical relevance: Capability-aware clustering supports more fine-grained LLM ranking across queries with diverse capability requirements.The paper positions this capability-aware ranking as a methodological contribution with limited direct societal risk.

H Example Prompts Referenced in Cluster Overlap Analysis

The referenced examples show that ECC separates prompts within shared semantic regions according to capability demands, including mechanistic reasoning, constrained design, simulation, and process optimization. The examples span chemistry, biomedical design, materials, and polymer engineering.

  • H.1 Example Prompts in Cell ECC_1∩Emb_1: H.1 includes prompts about reaction mechanisms, selectivity, spectroscopy, quantum chemistry, and reactant concentration effects.Examples cover benzene bromination, alcohol-to-alkene conversion, molecular properties, and reaction yield.
  • H.1 Example Prompts in Cell ECC_1∩Emb_1: H.1 also includes simulation-oriented prompts about molecular interactions and phase transitions.The examples mention Monte Carlo and molecular dynamics simulations alongside chemistry questions.
  • H.2 Example Prompts in Cell ECC_10∩Emb_1: H.2 emphasizes constrained biomedical design, including selective drugs, antibacterial targeting, and preservation of healthy cells.Several prompts explicitly require targeting a disease or organism while avoiding effects on related or healthy cells.
  • H.2 Example Prompts in Cell ECC_10∩Emb_1: H.2 also contains quantitative and simulation-based prompts involving reaction rates, binding affinity, molecular dynamics, and Monte Carlo methods.These examples combine mechanistic questions with explicit equations, temperature effects, or computational procedures.
  • H.3 Example Prompts in Cell ECC_1∩Emb_10: H.3 focuses on materials and nanomaterials, linking size, shape, composition, doping, and computational methods to electronic or optical properties.Examples reference DFT, quantum chemistry, molecular dynamics, nanoribbons, quantum dots, liquid crystals, and related materials.
  • H.3 Example Prompts in Cell ECC_1∩Emb_10: H.3 includes prompts comparing computational predictions, experimental values, transport behavior, and magnetic or electronic properties.The examples address DFT accuracy, exchange-correlation functionals, graphene nanoribbons, perovskites, and metal-organic frameworks.
  • H.4 Example Prompts in Cell ECC_10∩Emb_10: H.4 emphasizes polymer synthesis and manufacturing optimization, including absorption, extrusion, printing conditions, and waste reduction.These prompts focus on changing processing parameters or equipment to improve production rate, quality, stability, or absorption capacity.
  • H.4 Example Prompts in Cell ECC_10∩Emb_10: H.4 also covers polymer characterization through DMA, GPC, and crystallinity-based analysis of mechanical properties.The examples request step-by-step DMA explanations and molecular-weight or structure-property analysis.
Loading 2605.17110v2…