PaperlayerView exact revision on arXiv

Source-linked AI summary

ModelLens: Finding the Best for Your Task from Myriads of Models

Rui Cai, Weijie Jacky Mo, Xiaofei Wen, Qiyao Ma, Wenhui Zhu, Xiwen Chen, Muhao Chen, Zhe Zhao

arXiv:2605.07075v1cs.LG

TL;DR

Choosing among vast, heterogeneous model and dataset ecosystems is difficult because existing methods rely on small candidate pools or costly target-dataset evaluation. ModelLens learns a metric-aware capability space from public model–dataset–metric interactions, surpassing these baselines and improving routing methods by 21–81% on average.

  • Problem

    Selecting suitable models across hundreds of thousands of heterogeneous models and newly emerging datasets remains difficult, while existing methods use small pools, costly candidate evaluation, or homogeneous metrics.

  • Method

    ModelLens learns a metric-aware ranking space from model–dataset–metric interactions to recommend unseen models for unseen datasets without running candidates on the target dataset.

  • Results

    21–81% average gains across 5 representative routing methods were obtained from ModelLens’s recommended Top-K pools, while ModelLens surpassed metadata-only and forward-pass-based baselines.

  • Takeaways & Limitations

    ModelLens provides scalable candidate pools for downstream routing and generalizes zero-shot to unseen models and datasets across text and vision-language tasks.

  • Takeaways & Limitations

    ModelLens assumes recommendation can rely on model and dataset descriptors plus evaluation context, without any target-dataset feature, gradient, or forward-pass signal.

Abstract

from arXiv · show

The open-source model ecosystem now contains hundreds of thousands of pretrained models, yet picking the best model for a new dataset is increasingly infeasible: new models and unbenchmarked datasets emerge continuously, leaving practitioners with no prior records on either side. Existing approaches handle only fragments of this in-the-wild setting: AutoML and transferability estimation select models from small predefined pools or require expensive per-model forward passes on the target dataset, while model routing presupposes a given candidate pool. We introduce ModelLens, a unified framework for model recommendation in the wild. Our key insight is that public leaderboard interactions, though scattered and noisy, collectively trace out an implicit atlas of model capabilities across heterogeneous evaluation settings, a signal rich enough to learn from directly. By learning a performance-aware latent space over model--dataset--metric tuples, ModelLens ranks unseen models on unseen datasets without running candidates on the target dataset. On a new benchmark of 1.62M evaluation records spanning 47K models and 9.6K datasets, ModelLens surpasses baselines that either rely on metadata alone or require running each candidate on the target dataset. Its recommended Top-K pools further improve multiple representative routing methods by up to 81% across diverse QA benchmarks. Case studies on recently released benchmarks further confirm generalization to both text and vision-language tasks.

1 Introduction

ModelLens addresses model recommendation in the wild, where the expanding model ecosystem and unseen datasets make conventional selection methods difficult to apply. It learns from noisy leaderboard interactions to rank unseen models on unseen datasets without evaluating candidates on the target dataset, and improves downstream routing.

  • Motivation: Hundreds of thousands of open-source models make choosing a model for a new task or dataset increasingly difficult.Platforms such as HuggingFace host models spanning diverse architectures, scales, and application domains.
  • Limitations of Existing Approaches: Existing AutoML, transferability estimation, and routing methods are ill-suited to large-scale, unseen, and heterogeneous model-selection settings.They rely on fixed candidate pools or require per-candidate forward passes or target-dataset evaluation, while commonly assuming homogeneous evaluation.
  • Key Insight: ModelLens treats fragmented leaderboard interactions as implicit supervision encoding alignment between model capabilities and dataset characteristics.Interaction-trained projections cluster models and datasets by modality and task type more naturally than spaces based on textual descriptions alone.
  • Method: ModelLens combines target-dataset and candidate-model descriptions with leaderboard interactions to output a performance-based candidate ranking.Its metric-aware framework supports unseen models and datasets without direct evaluation or finetuning, with recommendations usable for inference, in-context learning, fine-tuning, or routing.
  • Validation and Results: 1.62M evaluation records spanning 47K models and 9.6K datasets support validation across matrix completion, held-out datasets, and newly released models.Integrating ModelLens’s top-K outputs with routing methods yields gains of up to 81% on QA benchmarks, while case studies confirm transfer to text and vision-language tasks.

2 Related Works

Prior work addresses transferability estimation, automated model search, or model routing, but each assumes constraints such as target-dataset access or a fixed candidate pool. ModelLens complements routing by constructing task-specific candidate pools at the dataset level.

  • Transferability Estimation: Transferability estimation predicts target-task performance using single-pass statistics or learned feature–data interactions, but assumes access to the target dataset.Training-free methods use information-theoretic or likelihood-based statistics, while learning-based methods model feature representations and target data.
  • Automated Model Search: AutoML selects models and tunes hyperparameters through search or meta-learning over fixed, relatively small pools, limiting applicability beyond curated settings.Recent work extends this paradigm to pretrained model selection, but the predefined candidate-pool assumption remains restrictive.
  • Model Routing: Model routing chooses models for incoming queries from a fixed candidate pool, leaving construction of that pool from a large heterogeneous model space unresolved.ModelLens is complementary: it produces task-specific candidate pools at the dataset level for downstream routing.

3 ModelLens

ModelLens recommends models for unseen datasets by ranking candidates from heterogeneous metadata without running them on the target dataset. It combines structural model priors with context-dependent interactions and trains on relative rankings across dataset–task–metric groups.

  • Framework: ModelLens predicts candidate-model performance using model and dataset descriptors plus task and metric context, without target-dataset features, gradients, or forward passes.The framework supervises relative ordering within each dataset–task–metric group because absolute scores are incompatible across heterogeneous metrics.
  • Representations: Model representations combine learned IDs, tokenized names, frozen descriptions, model size, and architecture family, while datasets use frozen description embeddings.Task types and evaluation metrics are also represented as learned embeddings to adapt scores across evaluation protocols.
  • Score Composition: The compatibility score adds a structural prior for general competence to a residual interaction modeling dataset-specific specialization and metric-dependent behavior.Shared size and family embeddings support both pathways, while a learnable temperature controls ranking sharpness and preserves an interpretable decomposition.
  • Training Objectives: Training combines pairwise ranking, listwise likelihood, and pointwise regression objectives to capture local preferences, global rankings, and standardized performance signals.Pointwise targets are z-scored within each evaluation group, making performance comparable across heterogeneous metrics.

4 Experiments

Experiments show that ModelLens accurately recovers model–dataset interactions and generalizes to unseen datasets and models, outperforming feature-free baselines in both performance completion and cold-start settings. Its recommendations also improve transferability-based selection, routing across QA benchmarks, and task-specific model choice.

  • Performance Completion and Cold-start Generalization: ModelLens consistently outperforms all baselines in performance completion and cold-start generalization, with its largest advantage on unseen datasets.The results indicate that learned representations transfer beyond observed model–dataset pairs to entirely new datasets and models.
  • Transferability-based Model Selection: On eight vision benchmarks, ModelLens achieves the best average Kendall’s weighted τ without forward passes on the target dataset.Adding transferability features raises average τw to 0.609 and produces the best score on every dataset.
  • Routing with Recommended Model Pools: Replacing original pools with ModelLens-recommended pools consistently improves all six routers across all five QA datasets.The recommended alternatives are selected using dataset descriptions and evaluation metrics while matching inference cost and comparable model scale.
  • Ablations: The full listwise, pairwise, and pointwise objective achieves the best τw of 0.745, while removing listwise supervision causes the largest degradation to 0.632.Removing pairwise loss reduces τw to 0.703, and removing pointwise loss reduces it to 0.728; pointwise supervision primarily provides calibration.
  • Case Studies: On NGQA, the recommended top-1 model consistently outperforms GPT-4o-mini across binary classification, multi-label classification, and free-form text generation.The controlled model pool contains models under 20B parameters, matching the implied scale of the default GPT-4o-mini.
  • Case Studies: On RSVLM-QA, ModelLens recovers the exact empirical ranking with τ=ρ=1.00 and identifies Ovis2 as best at METEOR = 31.65.The evaluation ranks eight comparable-scale 7B–8B vision-language models, including three models not present in the original benchmark.

5 Conclusion · A Appendix · A.1 Appendix Overview

ModelLens addresses model recommendation in the wild with metric-aware ranking learned from large-scale model–dataset–metric interactions, generalizing zero-shot to unseen models and datasets. The appendix adds analyses, comparisons, dataset-construction details, and reproducibility information.

  • 5 Conclusion: ModelLens learns metric-aware rankings from model–dataset–metric interactions and generalizes zero-shot to unseen models and datasets.The benchmark contains 1.62M evaluation records spanning 47K models and 9.6K datasets.
  • 5 Conclusion: ModelLens surpasses metadata-only and forward-pass-based transferability approaches on the model recommendation benchmark.The supplied conclusion passage states this comparison but does not provide the corresponding numerical result.
  • A.1 Appendix Overview: The appendix provides additional details, analyses, and reproducibility information for ModelLens.Its supplementary material is organized into several topic areas.
  • A.1 Appendix Overview: The appendix visualizes interaction-trained and semantic-only model–dataset embedding spaces in Figures 4–5.These visualizations compare functional organization induced by performance interactions with organization based on textual similarity alone.
  • A.1 Appendix Overview: The appendix discusses model profiling, transferability estimation, model routing, AutoML, and matrix-completion baselines in Section A.5 and Table 5.It also reports additional baseline results.
  • A.1 Appendix Overview: The appendix documents dataset construction for the Model Recommendation in the Wild benchmark in Section A.3 and Table 6.It includes additional statistics and preprocessing details.
  • A.1 Appendix Overview: Dataset-construction details cover data sources, interaction normalization, metadata construction, and split design.These details support reproducibility of the benchmark and its preprocessing pipeline.

D. Implementation and Evaluation Details Section A.4 · H. Unseen-Family Generalization Section A.8

The paper documents its architectures, optimization, ranking losses, evaluation metrics, compute resources, baselines, routing pools, ablations, priors, and case studies. It also evaluates generalization when entire modern LLM families are held out until testing.

  • D. Implementation and Evaluation Details Section A.4: Implementation details cover architecture, embedding dimensions, optimization settings, ranking losses, batch construction, evaluation metrics, and compute resources.
  • D. Implementation and Evaluation Details Section A.4: Baseline details include feature-based transferability methods, feature-free methods, practitioner strawmen, and evaluation metrics.
  • D. Implementation and Evaluation Details Section A.4: Recommended replacement pools are provided for PopQA, HotpotQA, MuSiQue, and Bamboogle under comparable inference-scale constraints.The pools are accompanied by raw dataset descriptions used as semantic inputs.
  • D. Implementation and Evaluation Details Section A.4: Additional experiments analyze ablations, feature attribution, standardized advantage computation, learned size and family priors, and full case-study rankings.
  • H. Unseen-Family Generalization Section A.8: The unseen-family evaluation uses a strict family-level hold-out protocol that excludes entire modern LLM families from training until test time.
  • H. Unseen-Family Generalization Section A.8: This protocol assesses transferable model–dataset compatibility beyond family-specific memorization.

I. Limitations, Broader Impacts, Assets, and Reproducibility Section A.10 … A.2.5 Matrix Completion in Recommender Systems

The paper situates ModelLens among limitations, model-population analysis, transferability estimation, routing, AutoML, and matrix-completion approaches. It emphasizes interaction-based, inductive recommendation for cold-start models and datasets without target-dataset forward passes.

  • I. Limitations, Broader Impacts, Assets, and Reproducibility Section A.10: The section discusses limitations, broader impacts, data and asset licenses, and reproducibility information.
  • A.2.1 Model Profiling: Prior model-population research studies weight statistics, functional behavior, or repository structure for model prediction, similarity, clustering, lineage, and ecosystem analysis.Weight-space methods predict accuracy from parameter statistics; functional methods use probe responses but typically require multiple forward passes; Model Atlas represents repositories as transformation graphs.
  • A.2 Detailed Related Works and Baseline Comparison: ModelLens instead learns model–dataset interactions from leaderboard data to produce task-aware rankings without model-weight access or forward-pass evaluation.This supports dataset-level recommendation for open and closed models in rapidly evolving ecosystems containing tens of thousands of models.
  • A.2.2 Transferability Estimation and Model Selection: Transferability-estimation methods use target-dataset inference, making forward passes for every candidate increasingly costly as model ecosystems expand.Training-free methods extract feature–label statistics in a single pass, while learning-based approaches model interaction patterns but retain reliance on target-task inference.
  • A.2.3 Model Routing and Adaptive Inference: ModelLens supplies task-specific, dataset-level candidate sets that predefined-pool routing methods can pass to instance-level routers.Existing routing methods generally assume a relatively small candidate pool, whose quality affects downstream routing efficiency.
  • A.2.4 AutoML and Surrogate Modeling: Zero-shot AutoML methods use neural surrogates or language models to predict performance, but may struggle with the scale and modality heterogeneity of modern model hubs.The passage also notes structural priors from neural scaling laws and architectural family biases as part of the proposed direction.
  • A.2.5 Matrix Completion in Recommender Systems: Traditional matrix factorization captures latent model–dataset interactions but cannot generalize to newly released models or datasets because it relies on fixed identity embeddings.The recommendation problem is formulated as completing a sparse performance matrix Y ∈R^N×T.
  • A.2.5 Matrix Completion in Recommender Systems: ModelLens combines structural priors with residual interactions and uses ID-dropout to support memorization of seen models alongside inductive generalization.Its dual-pathway scoring function addresses the limitations of side-feature-heavy inductive methods such as IMC, while drawing inspiration from DropoutNet.

A.3 Dataset Construction Details · A.4 Implementation Details

ModelLens constructs its benchmark by aggregating noisy model–dataset performance records from multiple public sources, standardizing them into tuples, and evaluating cold-start generalization to unseen datasets and models. The framework uses pretrained dataset embeddings, model metadata, and a joint listwise–pairwise ranking objective implemented in PyTorch.

  • A.3 Dataset Construction Details: 1.64M raw interactions are extracted from HuggingFace repositories using YAML results, model-index metadata, and Markdown tables parsed with an LLM-based extractor.The three-tier pipeline covers diverse pipeline tags.
  • A.3 Dataset Construction Details: 147K evaluation interactions from 3,495 large language models across 43 benchmark datasets come from the Open LLM Leaderboard.
  • A.3 Dataset Construction Details: 10.8K interactions covering 5,443 models and 2,070 datasets are collected from publicly reported PapersWithCode results.
  • A.3 Dataset Construction Details: Each interaction is unified as (m, d, t, µ, v), representing model performance on a dataset under a task and metric.Deduplication, name normalization, and incomplete-entry filtering reduce the processed dataset to 1.62M interactions.
  • A.3 Dataset Construction Details: Dataset descriptions use pretrained text embeddings, while parameter size and model family provide structural model features.Dataset-level splits use stratified sampling, with held-out datasets for cold-start evaluation and temporally partitioned models for unseen-model evaluation.
  • A.3 Dataset Construction Details: The benchmark emphasizes scale, cross-domain and cross-modality heterogeneity, and noisy real-world reporting from model repositories.
  • A.4 Implementation Details: The framework is implemented in PyTorch and trained with a joint listwise–pairwise ranking objective.Experiments use the same backbone architecture, embedding configuration, and optimization settings unless otherwise specified.

A.4.1 Model Architecture and Embeddings

The full-feature ModelLens ranker combines frozen pretrained text embeddings with trainable hashed model-name embeddings and lightweight embeddings for discrete metadata.

  • Embedding configuration: 1536-dimensional text-embedding-3-small vectors encode model and dataset descriptions, are pre-computed and cached before training, and remain frozen during optimization.The configuration applies these embeddings to both model and dataset descriptions.
  • Embedding configuration: 512-dimensional trainable embeddings represent hashed model-name tokens.The model also represents model size bucket, model family, task ID, and dataset ID as discrete metadata features with lightweight learned embeddings.

A.4.2 Optimization and Training … A.8 Unseen-Family Generalization

ModelLens is trained with joint ranking objectives and evaluated through weighted ranking metrics, while its metadata-based recommendations scale without target-dataset model execution. The appendix details implementation, baselines, routing pools, feature attribution, and evaluation on unseen model families.

  • A.4.2 Optimization and Training: AdamW training uses a 1 × 10−3 learning rate, 1×10−4 weight decay, patience-20 early stopping, and global ℓ2 gradient clipping at 5.0.Early stopping monitors validation weighted Kendall’s τ.
  • A.4.3 Ranking Objectives: The ranker combines listwise, pairwise BPR, and pointwise MSE objectives, with the auxiliary regression loss stabilizing early optimization.Listwise batches use 8 datasets, while pairwise batches contain 1024 anchor–negative pairs.
  • A.4.4 Evaluation Protocol: Evaluation averages weighted Kendall’s τ across groups and additionally reports NDCG@k, Hit@k, and Recall@k for k ∈{1, 10, 30, 50}.The reported checkpoint is selected by the best validation weighted Kendall’s τ.
  • A.4.5 Compute Resources: A full training run typically converges in approximately 6–8 hours on 1 A6000 GPU, while complete test-grid evaluation requires less than 5 minutes.Recommendation inference uses leaderboard interactions and metadata, with complexity linear in the number of candidate models.
  • A.5 Baseline Details: Baselines include forward-pass feature-based transferability methods, metadata or interaction-based feature-free methods, and Model Size and Model Popularity heuristics.Feature-free methods avoid running models on the target dataset, whereas feature-based methods compute candidate scores from forward-pass statistics.
  • A.6 Recommended Model Pools for Routing: For PopQA, HotpotQA, MuSiQue, and Bamboogle, ModelLens generates replacement pools under comparable inference constraints using predicted dataset-semantic compatibility.The supplied descriptions state that candidates are selected without direct evaluation on the target benchmark.
  • A.7 Feature Ablation and Results: Feature ablation compares model-side and dataset-side inputs using leave-one-out performance drops and Shapley-style average marginal contributions across feasible feature subsets.These attribution views are illustrated in Figure 6.
  • A.8 Unseen-Family Generalization: Unseen-family generalization targets newly released Llama-, Qwen-, or Phi-class checkpoints, evaluated through a Modern-Cohort family hold-out split with three models.This tests a model population shift beyond random hold-out and held-out-dataset splits.

A.8.1 Data Split · A.8.2 Models Compared · A.8.3 Results

ModelLens evaluates held-out modern LLM families using a family-disjoint split, compares a holdout-trained checkpoint with an in-distribution ceiling, and reports ranking metrics under task-size constraints. The test set spans 364,517 rows, 4,943 models, 2,040 datasets, and 19,850 ranking tasks.

  • A.8.1 Data Split: 13 modern LLM families are held out for testing, while all other model families form the training pool.The held-out families are qwen, llama, mistral, gemma, phi, deepseek, yi, falcon, granite, aya, olmo, zephyr, and solar.
  • A.8.1 Data Split: A 5% model-disjoint validation slice is carved from the training pool to drive early stopping out of distribution.The supplied passage specifies that validation is model-disjoint, though its final wording is truncated.
  • A.8.1 Data Split: Held-out family identifiers remain in the global vocabulary, with embeddings present but receiving no training gradient.This preserves inference-time embedding slots while testing whether other architectural signals can support held-out-family prediction.
  • A.8.1 Data Split: 364,517 test rows cover 4,943 unique models, 2,040 datasets, and 19,850 unique (dataset, metric) ranking tasks.These counts characterize the resulting family-hold-out test split.
  • A.8.2 Models Compared: The evaluation compares three checkpoints on the held-out test set, including Holdout-Family and AllSeen.The supplied passages explicitly describe two of the three checkpoints; the third is not specified in the provided text.
  • A.8.2 Models Compared: Holdout-Family uses the family-hold-out training split with the family embedding pathway enabled.This checkpoint tests held-out-family performance while retaining the family embedding pathway.
  • A.8.2 Models Compared: AllSeen is trained on the full data including modern families and provides an in-distribution upper bound under the same architecture and vocabulary.It serves as the ceiling comparison for the family-hold-out experiment.
  • A.8.3 Results: NDCG@K, Hit@K, and Recall@K are averaged only over tasks with at least K candidate models, whereas weighted Kendall τ is averaged over all tasks.Tables 15–17 summarize overall results and break them down by held-out family and dataset overlap with training.

A.8.4 Discussion … A.12 Assets and Licenses

ModelLens retains strong Top-K recommendation quality under modern-family holdout, though fine-grained ordering and some family shifts remain difficult. The framework standardizes heterogeneous evaluations, probes learned size and family priors, and acknowledges leaderboard, modality, coverage, and licensing constraints.

  • A.8.4 Discussion: A.8.4 Discussion: 0.037 is the NDCG@10 gap between Holdout-Family and the AllSeen ceiling, a 4.9% relative drop.Hit@10 is 0.008 higher under holdout, at 0.7751 versus 0.7672.
  • A.8.4 Discussion: A.8.4 Discussion: −0.075 versus 0.157 on size-weighted Kendall τ shows that fine-grained ordering degrades more than identifying strong candidates.The holdout model still recovers the correct candidate pool, even when it cannot fully order near-equivalent modern checkpoints.
  • A.8.4 Discussion: A.8.4 Discussion: yi and granite remain within 0.013 NDCG@10 of the ceiling, while deepseek, olmo, gemma, and qwen show drops between −0.13 and −0.17.This heterogeneity indicates that some family-specific effects are not recoverable from size and descriptions alone.
  • A.8.4 Discussion: A.8.4 Discussion: NDCG@10 gaps are −0.034 for Seen-Dataset model-only OOD and −0.045 for Unseen-Dataset model-plus-dataset OOD.The similar gaps suggest that family-holdout difficulty arises mainly from missing family-level signals rather than unfamiliar datasets.
  • A.9 Computing Standardized Advantage and Learned Priors: A.9 Computing Standardized Advantage and Learned Priors: DATA summarizes empirical performance, while PROBE reflects learned bias from model size and family embeddings.The evaluation table represents model–dataset–metric scores as rows containing t, d, µ, m, and v.
  • A.9.1 Standardized Advantage (DATA): A.9.1 Standardized Advantage (DATA): scores are standardized within each (t, d, µ) group, clipped for small-sample noise, and averaged across groups per model.Group-level advantage is the mean model z-score for a size bucket or family, with insufficiently sampled groups discarded.
  • A.9.2 Learned Prior (PROBE): A.9.2 Learned Prior (PROBE): size and family effects are analyzed independently by marginalizing the other factor and standardizing PROBE values across bins.Reported analyses include Spearman correlations, linear slopes, and family η2 variance explained; scripts reproduce scores and plots from logged evaluations and checkpoints.
  • A.10 Limitations: A.10 Limitations, A.11 Broader Impacts, A.12 Assets and Licenses: leaderboard bias, sparse cross-modality coverage, and limited proprietary-model representation constrain evaluation, while the method may reduce selection costs.The work uses public metadata and benchmark results, and all datasets and models remain subject to their original licenses and terms of use.
Loading 2605.07075v1…