PaperlayerView exact revision on arXiv

Source-linked AI summary

Good Rankings, Poor Choices? Rethinking Language-Only Vision-Language Model Selection

Hyuk Kwon, Jaebyeong Jeon, Hyeonseo Jang, Kibok Lee

arXiv:2610.34685v1cs.CVcs.LG

TL;DR

LOVM seeks to choose a target-task VLM using text, but ranking metrics do not directly assess the performance of the model selected. The paper adopts regret-based evaluation, proposes similarity-weighted source regret, and corrects two sources of target-performance leakage. TC-ESR achieves the lowest mean SR and ORNR among compared methods under the revised protocol.

  • Problem

    Ranking metrics such as R5 and Kendall’s rank correlation assess ranking quality but do not directly measure how much the selected model underperforms the best model on the target task.

    Source: §3.1 · p. 3

  • Method

    The paper uses SR and ORNR to evaluate selection regret and proposes TC-ESR, which weights source-task regret by textual similarity to the target along visual domain, label semantics, and label granularity.

    Source: §1 · p. 2

  • Results

    TC-ESR achieves the lowest mean SR and ORNR among compared methods across 22 target tasks and 43 VLMs.

    Source: §1 · p. 2

  • Takeaways & Limitations

    LOVM methods should be evaluated and designed around the performance of the selected model rather than the quality of the predicted ranking.

    Source: §6 · p. 9

    Min–max normalization is sensitive to a single unusually low-accuracy candidate, which can enlarge the denominator and reduce normalized regret despite an unchanged performance gap.

    Source: §3.3 · p. 4

Abstract

from arXiv · show

Language-Only Vision-Language Model Selection (LOVM) aims to identify the Vision-Language Model (VLM) with the highest zero-shot accuracy on a target task using only textual information. We revisit the existing LOVM evaluation protocol and argue that ranking metrics, such as top-5 recall and Kendall's rank correlation, do not directly assess the performance of the selected model. We therefore adopt simple regret (SR), defined as the performance gap between the best and selected models, and introduce Outlier-Robust Normalized Regret (ORNR) to account for the varying performance ranges across tasks while reducing sensitivity to outliers. Building on this revised evaluation protocol, we propose Target-Conditioned Expected Source Regret (TC-ESR), a training-free method that selects the model with the lowest similarity-weighted mean regret across source tasks. Across 22 target tasks and 43 VLMs, TC-ESR achieves the lowest mean SR and ORNR among all compared methods. Separately, we identify two sources of target performance leakage in the prior ranking-based evaluation protocol and provide corrections that remove both.

1 INTRODUCTION

LOVM seeks to select a VLM for a target task without target images or instance labels, but ranking metrics do not directly measure the selected model’s target performance. The paper introduces regret-based evaluation and target-conditioned source weighting, and corrects two sources of target-performance leakage.

1 INTRODUCTION

  • Prior ranking metrics can favor a method that selects a worse target-performing model, as illustrated by SWAB and TC-ESR on GTSRB.SWAB has higher R5 and τ5, while TC-ESR selects the best-performing model.Source: Figure 1 · p. 1, §1 · p. 2
  • LOVM uses textual task information and source-task model accuracies to select a VLM without target images or instance-level labels, avoiding labeled target-dataset construction.The need for target data is costly, and the best VLM can vary across tasks.Source: §1 · p. 1
  • The paper adopts SR for the gap between the selected and best models, and introduces ORNR to compare regret across tasks while reducing outlier sensitivity.ORNR uses a robust performance range because a single unusually weak candidate can make full-range normalization artificially small.Source: §1 · p. 2
  • Across 22 target tasks and 43 VLMs, TC-ESR achieves the lowest mean SR and ORNR among compared methods.It weights source-task regret by textual similarity to the target along visual domain, label semantics, and label granularity.Source: §1 · p. 2
  • The revised protocol addresses leakage by restricting feature selection to source tasks and excluding ImageNet-1K as an evaluation target when its accuracies are input features.Both problems arise when source and target tasks are not properly separated.Source: §1 · p. 2

2 PRELIMINARIES

LOVM uses textual descriptions of a target task and source-task model performance to estimate which candidate VLM will perform best. The section also outlines two prior LOVM methods, ModelGPT and SWAB.

2.1 PROBLEM SETUP

  • For evaluation, each task serves in turn as the target, with the remaining tasks serving as its source tasks.Selection estimates target-model performance by aggregating candidate performance across source tasks using source–target similarity.Source: §2.1 · p. 3

2.2 EXISTING LOVM METHODS

  • ModelGPT builds candidate-model features from textual proxies for target images and uses a learnable ranker; SWAB uses optimal transport and transfers rankings from relevant source classes.SWAB’s features capture modality and capability gaps.Source: §2.2 · p. 3

3 RETHINKING LOVM EVALUATION

Ranking metrics assess how well LOVM methods recover an ordering, but not the target-task performance of the model actually selected. The section revises evaluation with regret metrics and corrects two sources of target-performance leakage.

3.1 RANKING-BASED EVALUATION

  • Top-5 recall measures recovery of ground-truth top-five models, while Kendall’s τ5 measures their relative ordering; neither directly measures the selected model’s target-task performance.In practical selection, the top-ranked model is deployed, so these metrics do not quantify its gap from the best model.Source: §3.1 · p. 3

3.2 TARGET PERFORMANCE LEAKAGE

  • Feature selection using the target task and using ImageNet-1K as both an input-feature source and evaluation target leak target performance into prior evaluations.The corrections select features using source tasks only and exclude ImageNet-1K from target tasks; both changes lower ModelGPT and SWAB R5 and τ5.Source: §3.2 · p. 3

3.3 REGRET-BASED EVALUATION

  • Simple regret (SR) measures the absolute accuracy gap between the selected model and the best candidate, but equal SR can represent different fractions of task-specific accuracy ranges.The best candidate is the model with the highest performance on the task.Source: §3.3 · pp. 3–4, Figure 2 · p. 4
  • ORNR divides SR by a robust per-task range using the lower whisker rather than the minimum, reducing sensitivity to unusually weak candidates.Candidates below the lower fence are excluded only from the normalization denominator, not from the candidate set; ORNR can exceed one when selected accuracy falls below the lower whisker.Source: §3.3 · p. 4

4 METHOD

The method selects a VLM by aggregating simple regret across source tasks, with TC-ESR weighting those regrets according to text-derived source–target similarity. Similarity is estimated from visual domain, label semantics, and label granularity, while ESR provides an equal-weight baseline.

4.3 TARGET-CONDITIONED EXPECTED SOURCE REGRET

  • TC-ESR weights each model’s source-task simple regrets by target-conditioned task similarity and selects the model with the lowest weighted mean regret.It retains ESR’s modelwise source regret but replaces uniform averaging with similarity-based weights; the temperature is selected by leave-one-out validation over source tasks.Source: §4 · p. 5, Figure 3 · p. 5, §4.3 · p. 7

4.1 EXPECTED SOURCE REGRET

  • ESR selects the model with the lowest mean simple regret across available source tasks, equivalent to the highest average source accuracy.Because the best accuracy on each source task is constant across candidate models, minimizing mean regret and maximizing mean source accuracy yield the same selection.Source: §4.1 · p. 5

4.2 LANGUAGE-ONLY TASK SIMILARITY

  • Source–target similarity combines visual domain, label semantics, and label granularity, using text descriptions and class-name sets rather than images.Domain similarity uses exact membership in shared domain families; task metadata uses shared vocabularies for domain family and label scale type.Source: §4.2 · pp. 5–6
  • Label semantic similarity averages best-match class-name coverage in both directions, so a one-way subset match does not yield high similarity by itself.Class-name embeddings are produced by a fixed text encoder, and averaging the two directional matches handles class sets with different sizes or semantic scope.Source: §4.2 · p. 6
  • Label granularity similarity compares median pairwise class-name embedding distances only when the target and source labels are semantic-category taxonomies.The method sets the granularity component to zero otherwise and averages the similarity components applicable to the target with equal weights.Source: §4.2 · pp. 6–7

5 EXPERIMENTS

Across 22 target tasks, TC-ESR achieves lower mean SR and ORNR than ESR and prior LOVM methods by weighting source-task regret according to target similarity. Ablations and further analyses examine the similarity design, ranking-metric disagreement, and protocol corrections.

5.2 MAIN RESULTS

  • TC-ESR achieves lower mean SR and ORNR than ESR, prior LOVM methods, and the best fixed model, which requires unavailable target accuracies.The comparison uses the revised evaluation protocol across 22 target tasks.Source: §5.2 · p. 8, Table 1 · p. 8

5.3 ABLATION STUDIES

  • Removing any similarity axis degrades both metrics; removing label granularity causes the largest degradation, while all three axes perform best.Label granularity is applied to the 13 targets for which it is applicable.Source: §5.3 · p. 8, Table 2 · p. 8
  • Similarity-based source-weight assignment outperforms uniform weighting, while randomly permuting the same weights performs worse than uniform weighting.The permuted result is averaged over 10K permutations.Source: §5.3 · p. 8, Table 3 · p. 8
  • None of the alternative uses of task information outperforms ESR on both metrics, and restricting aggregation to the most similar sources performs worse than weighting all sources.Less similar sources also provide useful evidence, while simpler similarities do not improve over uniform weighting.Source: §5.3 · pp. 8–9, Table 4 · p. 8

5.4 FURTHER ANALYSIS

  • R5 and τ5 disagree with SR in 25.7% and 43.1% of comparisons, with mean SR losses of 3.32%p and 2.46%p, respectively.Across six methods, mean R5 has Kendall correlation −0.83 with mean SR when SR is negated so that higher is better.Source: §5.4 · p. 9, Table 5 · p. 9
  • Each protocol correction lowers R5 and τ5 for both ModelGPT and SWAB, and applying both corrections produces the largest decrease in all four cases.The corrections are source-only feature selection and excluding ImageNet-1K from evaluation targets.Source: §5.4 · p. 9, Table 6 · p. 9

6 CONCLUSION

The paper argues that LOVM should be evaluated and designed around the performance of the selected model, not predicted-ranking quality. It distinguishes LOVM from model-selection settings that use target images, labels, or multiple models at inference.

6 CONCLUSION

  • The conclusion reports that TC-ESR achieves the lowest mean SR and ORNR under the revised protocol by weighting source-task regret using text-derived similarity along three axes.The paper also identifies and corrects two sources of target-performance leakage in the prior protocol.Source: §6 · p. 9

A.1 MODEL SELECTION AND LOVM

  • Conventional selection and transferability estimators use labeled target examples, whereas LOVM uses textual target-task information and source-task performance without target images.LOVM selects one pretrained VLM for the whole target task.Source: §A.1 · p. 13

A.2 OTHER VLM SELECTION SETTINGS

  • MLL selects models separately by class and combines their predictions, while DCD uses unlabeled source and target images; neither matches LOVM’s single-model, no-target-image setting.Li et al. (2026) instead choose a vision encoder for a fixed LLM before multimodal training.Source: §A.2 · p. 13

A.3 TASK SIMILARITY FOR SOURCE WEIGHTING

TC-ESR estimates source–target task similarity from text and class names, then uses it to weight source-task regret. The evaluation uses the LOVM benchmark’s 23 tasks and 43 candidate VLMs, with ImageNet-1K retained as a source but excluded as a target.

A.3 TASK SIMILARITY FOR SOURCE WEIGHTING

  • Prior task-relationship approaches use images, labels, probe models, or observed transfer outcomes, while prior work identifies domain proximity, label semantics, and label granularity as transferability factors.These motivate the text-only similarity axes used by TC-ESR.Source: §A.3 · p. 13
  • TC-ESR estimates similarity along visual-domain, label-semantic, and label-granularity axes using task and domain descriptions and class-name sets.The resulting target-conditioned similarity weights source-task simple regrets and requires neither target images nor candidate-model inference.Source: §A.3 · p. 13

B.1 DATASETS

  • The benchmark contains 23 tasks; evaluation uses the 22 tasks other than ImageNet-1K as targets, while ImageNet-1K remains in each target’s source pool.The assumed task descriptions and class-name sets are used to construct TC-ESR metadata.Source: §B.1 · p. 14, Table 7 · p. 14

B.2 VLM ZOO

  • The candidate pool comprises 43 VLMs from OpenCLIP, BLIP, and BEiT-3 checkpoints, using zero-shot accuracies released by SWAB.Those accuracies serve as source performance for LOVM methods and target performance only for final evaluation.Source: §B.2 · p. 14

B.3 RANKING METRICS

The appendix defines ranking-quality diagnostics and describes a revised source-only protocol for ModelGPT and SWAB. The protocol also selects TC-ESR’s temperature using source-task validation, without target-task candidate performance.

B.3 RANKING METRICS

  • Top-5 Recall measures recovery of ground-truth top-five models, while τ5 measures ordering agreement among models shared by the predicted and ground-truth top-five sets.Both are supplementary ranking diagnostics; SR and ORNR are the primary measures of the rank-one selection.Source: §B.3 · pp. 14–15
  • For τ5, Kendall’s rank correlation is computed only among models appearing in both top-five sets and ranges from −1 to 1, with larger values indicating stronger agreement.Metrics are macro-averaged over the 22 evaluation targets.Source: §B.3 · p. 16

B.4 REVISED EVALUATION PROTOCOL

  • ModelGPT and SWAB select feature configurations by nested validation over source tasks, then fit the final ranker on the remaining source tasks without using target-task candidate performance.The feature configuration with the highest mean R5 + τ5 across validation tasks is selected.Source: §B.4 · p. 16, §B.3 · p. 16
  • TC-ESR remains training-free and uses the nested source-only validation procedure only to select its softmax temperature λ.ImageNet-1K is neither a target nor a validation task, but remains available as a source task.Source: §B.4 · p. 16

C ADDITIONAL IMPLEMENTATION DETAILS

The appendix specifies the experimental environment and baseline evaluation, then details how task metadata vocabularies are induced from anonymized task records.

C ADDITIONAL IMPLEMENTATION DETAILS

  • Experiments use a Ryzen 9 7950X CPU, one RTX 4090 GPU, PyTorch 2.10.0, and CUDA 12.8.ModelGPT and SWAB are evaluated over seeds 1–10; deterministic methods use seed 1.Source: C ADDITIONAL IMPLEMENTATION DETAILS · pp. 16–17
  • TC-ESR alternatives share its metadata, text encoder, and class-name template, with hyperparameters selected through source-only validation using mean ORNR.The alternatives vary source selection, similarity construction, or source weights.Source: C ADDITIONAL IMPLEMENTATION DETAILS · p. 17

D.1 VOCABULARY INDUCTION

  • Task metadata represents each task by a domain family and a label scale type, with shared category vocabularies and assignment criteria.For vocabulary induction, an LLM uses anonymized records containing domain and task descriptions and class names; it induces seven domain categories and six label-scale categories, then fixes both vocabularies before assignment.Source: D TASK METADATA CONSTRUCTION DETAILS · p. 17, §D.1 · p. 17, Table 9 · p. 17

D.2 TASK CATEGORY ASSIGNMENT

An LLM assigns fixed metadata categories from anonymized task records, while class-name embeddings supply semantic similarity and, where applicable, label granularity.

D.2 TASK CATEGORY ASSIGNMENT

  • The LLM assigns one category from each fixed vocabulary using only each anonymized task record; five runs and majority vote determine each task’s categories.All 23 tasks achieve a majority; five runs agree completely on domain family for 21 tasks and label scale for 20.Source: §D.2 · pp. 17–18, Table 10 · p. 18

D.3 CLASS NAME EMBEDDINGS AND LABEL GRANULARITY

  • Class names are encoded with all-mpnet-base-v2 using a fixed template and normalized for pairwise semantic similarity; median pairwise cosine distance supplies granularity where applicable.Granularity is computed for 14 of 23 tasks and 13 of 22 evaluation targets; fixed metadata determines whether that component applies.Source: §D.3 · p. 18

E.1 FURTHER ABLATION STUDIES

These appendix analyses test TC-ESR’s benchmark performance, component choices, task similarities, and robustness under alternative normalization and baseline-tuning conditions.

E.1 FURTHER ABLATION STUDIES

  • Exact domain-family matching beats embedding cosine similarity, bidirectional label coverage leads semantic aggregation, and median pairwise distance gives the lowest SR and ORNR for granularity.The authors relate domain matching to visual rather than label-semantic similarity, and the median’s advantage over the mean to reduced sensitivity to extreme distances.Source: §E.1 · p. 19, Table 11 · p. 19, Table 12 · p. 19
  • For source aggregation, SR preserves absolute performance gaps so similarity weights control source contributions, whereas ORNR normalizes each regret independently by its task’s performance range.The comparison concerns both the regret used for source aggregation and the regret used to select λ.Source: §E.1 · p. 19, Table 15 · p. 19
  • With Qwen3-4B-Instruct metadata construction, TC-ESR’s SR is 4.83%p versus 5.83%p for ESR and 7.11%p for ModelGPT; two alternative text encoders yield comparable SR and ORNR.The LLM substitution reruns vocabulary induction and category assignment; BGE-base-en-v1.5 and all-distilroberta-v1 are tested with encoder-specific templates.Source: §E.1 · p. 20, Table 16 · p. 20, Table 17 · p. 20

E.2 PER-TASK EXPERIMENTAL RESULTS

  • TC-ESR achieves the lowest mean SR and ORNR across 22 targets, while SWAB achieves the highest R5 and τ5.The table reports means under the revised protocol; SR is in percentage points and the other metrics are percentages.Source: Table 14 · p. 19, §E.2 · p. 20

E.3 TASK SIMILARITY ANALYSIS

  • Similarity scores reflect complementary domain, label-semantic, and granularity information, with high similarity reported for CLEVR pairs, EuroSAT–RESISC45, MNIST–SVHN, and Cars–Pets/Flowers.Figure 4’s rows are target tasks, columns are source tasks, and crosses mark excluded self-pairs; ImageNet-1K appears only as a source.Source: §E.3 · p. 20, Figure 4 · p. 21

E.4 ROBUSTNESS OF ORNR

  • TC-ESR has the lowest normalized regret under all 13 denominator choices; ORNR is less sensitive than min–max normalization to weak candidates but remains affected by substantial candidate-pool changes.With a single weakest candidate degraded, ORNR changes 74.0%–82.0% less and remains unchanged on 16 of 22 targets; removing multiple weak candidates reduces the advantage to 23.5%–47.7%.Source: §E.4 · pp. 21–22, Table 19 · p. 22, Table 20 · p. 22

E.5 BASELINES RETUNED FOR SELECTION QUALITY

  • Retuning ModelGPT and SWAB with SR or ORNR barely changes ModelGPT and degrades SWAB, leaving TC-ESR and ESR the best and second-best methods.This tests whether the baselines’ original R5 + τ5 feature-selection objective gives TC-ESR an advantage.Source: §E.5 · p. 22

F ROLE OF TARGET-CONDITIONED WEIGHTING

TC-ESR’s target-conditioned source weighting produces more target-specific selections than prior methods and ESR. Its advantage persists beyond the Retinopathy case study, though the SR margin narrows.

F ROLE OF TARGET-CONDITIONED WEIGHTING

  • TC-ESR selects eight distinct candidate models across targets, while ESR selects the best fixed model on 21 of 22 targets and prior methods largely favor one candidate.The best fixed model uses target accuracies, and target-specific selection is needed to achieve lower SR than that fixed choice.Source: F ROLE OF TARGET-CONDITIONED WEIGHTING · p. 23, Table 22 · p. 23
  • On Retinopathy, TC-ESR assigns PCam more than 99.99% of source weight and selects OpenAI RN50x16 at 2.22%p SR, versus 20.57%p for the best fixed model.Removing PCam raises TC-ESR’s SR to 20.57%p, while other methods’ selections remain unchanged.Source: F ROLE OF TARGET-CONDITIONED WEIGHTING · p. 23, Table 23 · p. 23
  • For targets with a single source sharing their domain family, TC-ESR’s effective number of sources is at most 1.01; it is at least 1.21 for other targets and 3.63–6.24 for object-centered targets.Shared-domain sources receive nearly all the weight, further differentiated by label semantics and, when applicable, granularity.Source: F ROLE OF TARGET-CONDITIONED WEIGHTING · p. 24, Table 24 · p. 24
  • Across the 21 targets excluding Retinopathy, TC-ESR remains the best LOVM method on SR and ORNR, with 4.78%p SR and 18.51% ORNR.Its SR is close to ModelGPT’s 5.08%p and the best fixed model’s 4.81%p, while its ORNR is below ModelGPT’s 22.06% and the best fixed model’s 21.19%.Source: F ROLE OF TARGET-CONDITIONED WEIGHTING · pp. 24–25, Table 25 · p. 24

Related papers

Loading 2610.34685v1…