Which LLM to pick? Online Active Model Selection for Large Language Models

1 Introduction

• Existing active model selection largely targets classification or assumes prompts are available upfront, leaving online LLM selection unaddressed.The paper states that prior work had not explored active model selection for LLMs in online settings.Source: §1 · pp. 1–2
• Selecting a suitable LLM is difficult because no single model consistently excels across domains or datasets, and effectiveness varies by context.The introduction notes that standard evaluation metrics alone cannot fully capture model quality or practical utility.Source: §1 · p. 1
• ONLINE LLM PICKER selects informative prompts from an incoming stream for annotation to identify the best candidate LLM.Its informativeness criterion combines variance across model responses to unannotated prompts with posterior uncertainty, without assumptions about the underlying LLMs.Source: §1 · p. 2
• ONLINE LLM PICKER identifies the best or near-best model while using up to 71.67% fewer annotations than competing baselines.The evaluation spans multiple generation tasks and model collections containing over 130 language models.Source: §1 · p. 2

2 Related Work

• Human-feedback evaluation relies on costly annotators, while LLM-as-a-judge methods offer scalability but introduce bias [Stiennon et al., 2022; Ouyang et al., 2022; Gu et al. (2025); Bai et al. (2024); Chiang et al. (2024); Dubois et al. (2025)].Evaluation also includes automatic metrics and benchmark-driven progress [Lin, 2004; Papineni et al., 2002; Lavie & Agarwal, 2007; Wang et al., 2019; Wang et al., 2020; Hendrycks et al., 2021; Srivastava et al., 2023].Source: §2 · p. 2
• Active testing reduces annotation cost through adaptive sampling, but focuses on evaluating a fixed model or small set of models [Berrada et al. (2025)].The paper's setting instead identifies the best LLM for an arbitrary data stream under limited annotations.Source: §2 · p. 2
• Prior active model-selection approaches often assume an upfront query pool or focus on classification, so they are not directly applicable to online LLM selection [Okanovic et al., 2024; Ashury-Tahan et al., 2024; Kay et al., 2025; Karimi et al., 2021; Liu et al., 2022; Li et al., 2024a].The paper introduces online active model selection for LLMs and presents strategies as baselines.Source: §2 · p. 3
• The paper introduces online active model selection for large language models and presents several strategies as baselines.It relegates discussion of related but less directly relevant frameworks to Appendix A.Source: §2 · p. 3

3.1 Problem Statement and Background

• With budget b ≪ T, the goal is to identify the model with minimum cumulative loss using annotations from only selected prompts.The returned model is evaluated by identification probability and annotation efficiency.Source: §3.1 · p. 3

3.2 The Algorithm

• At each time step, a Bernoulli query decision determines whether an annotation updates the posterior over which candidate model is best.When no annotation is queried, the reference remains hidden and the posterior is unchanged.Source: §3.2 · p. 3

3.2.1 Adaptive Learning Rate

• The adaptive learning rate uses an exponential moving average of observed loss variance because sparse queries and noisy examples can distort raw variance estimates.The decay rate β controls the moving average’s memory; its effective window is approximately (1 − β)^−1.Source: §3.2.1 · p. 4

3.2.2 Query Probability

• Query probability combines hypothetical loss variance across model responses with posterior entropy, increasing queries when disagreement or uncertainty is high.The entropy is normalized by log m, and a time-decaying lower bound guards against unstable importance-weighted estimates and skipped informative prompts.Source: §3.2.2 · pp. 5–6

4.1 Datasets and Models

• The dataset suite spans open-ended question answering, arithmetic and calculus, grammar correction, multi-turn conversation, long-form question answering, medical calculation, and finance.The model collections include open-source and proprietary LLMs, from a few million to several billion parameters.Source: §4.1 · pp. 6–7

4.2 Baselines

• Baselines query annotations probabilistically: Random uses fixed b/T, while disagreement, KL divergence, and entropy-based uncertainty guide adaptive query probabilities.Disagreement targets prompts with strong model disagreement; KL and uncertainty use divergence from posterior belief and entropy over pairwise losses, respectively.Source: §4.2 · pp. 7–8

4.5.1 Regret

• Up to 2.51× regret reduction versus the best competing baseline for sequential generation on unannotated prompts.The reduction occurs across the evaluated budget levels, including before the annotation budget is exhausted.Source: Figure 2 · p. 8, §4.5.1 · p. 9

4.3 Evaluation Protocol

• Each realization uses T i.i.d. test instances; methods are compared at matched average annotation counts after query-probability tuning.Results are averaged over independent realizations with fresh streams.Source: §4.3 · p. 8

4.4 Performance Metrics

• Evaluation measures regret on unannotated prompts, the fraction of realizations identifying the best model, and annotation efficiency for finding best or near-best models.Annotation efficiency relates the number of annotations used to identifying those models.Source: §4.4 · p. 8

4.5.2 Identification Probability

• Up to 71.67% annotation cost saved versus the best competing baseline when reaching maximum identification probability.Across datasets and annotation budgets, ONLINE LLM PICKER required the fewest annotations to reach that probability.Source: Figure 3 · p. 9, §4.5.2 · p. 9

4.5.3 Annotation Efficiency

• Up to 70.69% fewer annotations were required to identify a near-best model within the tested δ-vicinities.The comparison is against the best competing baseline and is reported separately by dataset and δ-vicinity.Source: §4.5.3 · p. 9

5 Discussions

• The paper introduces online model selection for LLMs with limited annotation evidence, a technically challenging setting given open-ended generative outputs.ONLINE LLM PICKER is presented as a method tailored to confidently identify the best model in this setting with annotation efficiency.Source: §5 · pp. 9–10

A Related Work

• Prior data-sampling approaches generally assume offline access to the full corpus or slowly changing data distributions, unlike adaptive decisions on streaming data.The discussion also distinguishes active testing that evaluates a fixed model or small model set from selecting the best LLM for an arbitrary stream under limited annotations.Source: A Related Work · p. 17

B Datasets and LLM collections

• More than 130 LLMs across 10 datasets are evaluated, using BERTScore for MT-Bench and ROUGE-L for the other datasets.Source: B Datasets and LLM collections · p. 17
• Scores span 0.02–0.97 across models and datasets, with ROUGE-L or BERTScore used depending on the task.Source: B Datasets and LLM collections · p. 17, Figure 4 · p. 18
• For several datasets, the authors generated LLM outputs themselves; the study includes instruction-tuned, dataset-fine-tuned, and general-purpose models.Source: B Datasets and LLM collections · p. 18

C Baselines

• Disagreement queries at a fixed probability but only when variance across experts’ average pairwise answer discrepancies exceeds a near-zero threshold.Source: C Baselines · p. 18
• The Kullback–Leibler baseline queries in proportion to the mismatch between its posterior belief and a loss-based predictive distribution.Source: C Baselines · p. 18
• The Uncertainty baseline queries more often when entropy of the per-model average-loss profile is higher.Source: C Baselines · p. 18

• The section presents additional results and comprehensive data from the experiments described in Section 4.Source: D Extended Results · p. 18
• Experiments are conducted separately for each dataset and budget level.Source: D Extended Results · p. 18
• Each dataset-budget combination uses 500 independent realizations.Source: D Extended Results · p. 18

D.1 Regret

• 2.51×, 2.47×, 2.44×, and 1.97× are the regret reduction factors on Comprehensive–Arithmetic–Problems, SQuAD, SQuAD v2, and FinQA, respectively.Source: Table 4 · p. 19, §D.1 · p. 19
• ONLINE LLM PICKER has consistently lower regret than baselines across datasets, compared against the strongest competing baseline at the same budget.Source: Table 4 · p. 19, §D.1 · p. 19

D.2 Identification Probability

• ONLINE LLM PICKER consistently requires fewer annotations than baselines to reach 100% identification probability.Source: §D.2 · p. 19

D.3 Robustness Analysis

• Table 5 reports the percentage of stream length annotated by each method before it first reaches 100% identification probability.Source: Table 5 · p. 20
• In 36 of 40 cases, ONLINE LLM PICKER has the lowest 95th-percentile accuracy gap; in the other four, it has the second-lowest gap.Source: §D.3 · p. 19, Table 6 · p. 21

D.4 Annotation Efficiency

• The section provides extended plots for annotation-efficiency results described in Table 1.Source: §D.4 · p. 21

E Ablation Study and Design Choice Analysis

• All versions of ONLINE LLM PICKER consistently outperform the baseline methods on the Comprehensive Arithmetic Problems dataset.The ablation describes the overall algorithm structure as effective.Source: E Ablation Study and Design Choice… · p. 21
• The largest improvements come from combining variance and entropy.The ablation identifies this component as having the strongest impact across the board.Source: E Ablation Study and Design Choice… · p. 21
• The full method reaches 100% identification probability at about 196 annotations, versus about 202 without adaptive learning rate.The gains from adaptive learning rate depend on the data distribution through a proxy of observed-annotation variance.Source: E Ablation Study and Design Choice… · p. 21
• General-purpose embeddings and LLM judges can produce noisy or misleading scores on specialized domains; investigating robust domain-appropriate metrics is beyond this work's scope.In the medical datasets, several open-source embedding models produced nearly constant similarity scores across sentences.Source: E Ablation Study and Design Choice… · p. 21
• Figure 5 presents annotation efficiency for ONLINE LLM PICKER and baselines across different δ values.The supplied caption gives the figure's scope but no specific values or winner.Source: Figure 5 · p. 22

F.1 Rationale for Metric Selection

• ROUGE and BERTScore were selected for stability, reproducibility, and computational feasibility.The authors contrast these established metrics with newer LLM-based evaluation frameworks.Source: §F.1 · p. 22
• LLM-based evaluators can vary with prompt phrasing and model biases, while embedding scores were noisier and less stable across streaming batches.Embedding scores clustered tightly even when model outputs differed meaningfully.Source: §F.1 · p. 22
• Pairwise comparisons across 54 models make LLM-based scoring computationally and financially impractical for large-scale or online evaluation.The passage attributes this constraint to the quadratic nature of the comparisons.Source: §F.1 · p. 22
• Established metrics support reproducibility without requiring proprietary or high-cost APIs.The authors describe the current metric choice as a robust and efficient baseline, while naming advanced LLM-based metrics as future work.Source: §F.1 · p. 22

F.2 Computational Overhead Analysis

• Figure 6 compares identification probability for full ONLINE LLM PICKER against its variance-entropy and adaptive-LR versions.The caption does not state the outcome of the comparison.Source: Figure 6 · p. 23
• For 54 models, each prompt requires 1,431 pairwise distances plus 54 oracle similarities, totaling 1,485 distance computations.Symmetry allows computing only the upper triangular part of the pairwise distance matrix, excluding its diagonal.Source: §F.2 · pp. 22–23
• Preprocessing pairwise similarities yields about 0.156 seconds per prompt; 1,000 prompts took 2 minutes and 36 seconds on the reported local machine.The authors describe this latency as low enough for real-time or interactive applications, including with dozens of models.Source: §F.2 · p. 23

F.3 Posterior Distribution

• The posterior distribution is updated using the cumulative loss estimate and adaptive learning rate.The passage introduces the posterior update but does not supply its displayed formula.Source: §F.3 · p. 23