Source-linked AI summary
Discovering Hidden Gems in Model Repositories
Jonathan Kahana, Eliahu Horwitz, Yedid Hoshen
TL;DR
Public repositories contain millions of fine-tuned models, but users often concentrate on a few popular checkpoints, raising whether superior models are overlooked. The paper evaluates over 2,000 models, formulates discovery as fixed-budget bandit search, and accelerates Sequential Halving. It finds hidden gems that outperform popular counterparts and identifies elite models with only 50 queries per candidate, while noting remaining evaluation and task-coverage limits.
Problem
As repositories scale, it is unclear whether popular checkpoints are the best available and how superior models can be found without exhaustive evaluation.
Method
The paper evaluates models within shared model trees and formulates discovery as fixed-budget best-arm identification using an accelerated Sequential Halving search.
Results
More than 2,000 evaluations consistently find unpopular hidden gems that outperform popular counterparts, while the method identifies elite models with only 50 queries per candidate.
Takeaways & Limitations
Model selection can uncover strong alternatives to popular foundation checkpoints without increasing inference costs within shared model trees.
Takeaways & Limitations
The approach still evaluates every model on a small number of queries, and new tasks require reevaluating models for those tasks.
Abstract
from arXiv · showhide
Public repositories host millions of fine-tuned models, yet community usage remains disproportionately concentrated on a small number of foundation checkpoints. We investigate whether this concentration reflects efficient market selection or if superior models are systematically overlooked. Through an extensive evaluation of over 2,000 models, we show the prevalence of "hidden gems", unpopular fine-tunes that significantly outperform their popular counterparts. Notably, within the Llama-3.1-8B family, we find rarely downloaded checkpoints that improve math performance from 83.2% to 96.0% without increasing inference costs. However, discovering these models through exhaustive evaluation of every uploaded model is computationally infeasible. We therefore formulate model discovery as a Multi-Armed Bandit problem and accelerate the Sequential Halving search algorithm by using shared query sets and aggressive elimination schedules. Our method retrieves top models with as few as 50 queries per candidate, accelerating discovery by over 50x.
1 Introduction
Public repositories make many fine-tuned models available, but users often default to foundation or official instruction checkpoints because documentation is incomplete. The paper asks whether popular models are best and how to find better alternatives efficiently.
- Motivation: Over two million models are hosted on Hugging Face, while users commonly default to official Qwen or Llama checkpoints.Model cards are frequently incomplete or missing, making repository-wide selection difficult.
- Research questions: The paper investigates whether the most popular models are actually the best performers and how superior models can be identified efficiently.
- Findings: More than 2,000 evaluated models reveal unpopular fine-tunes that significantly outperform popular counterparts across several model families and diverse tasks.These lost high-performers are termed hidden gems.
- Approach: Model discovery is formulated as a Multi-Armed Bandit problem because exhaustive evaluation may require billions of inferences.The goal is to rank an entire model population from scratch without relying on previous rankings.
2 Do Hidden Gems Exist?
Usage is highly concentrated, yet evaluation finds unpopular models that strictly outperform popular choices. These hidden gems are difficult to identify because documentation and predictable repository location provide little guidance.
- Centralization of Usage: 0.0015% of models account for 95% of downloads, leaving most of the repository ecosystem inactive.This concentration motivates competing efficient-discovery and information-asymmetry hypotheses.
- Defining Hidden Gems: A Hidden Gem is unpopular, top-tier by performance, and strictly better than the best model in the popular-consensus group.The definition compares the top 1% by downloads with the top 1% by performance within a model tree.
- Evidence of Hidden Gems: 83.5% to 89.0%: a Qwen-3B math fine-tune improves GSM8K accuracy while using less than half the parameters of the best Qwen-7B base version.
- Evidence of Hidden Gems: Hidden gems occur across math, coding, question answering, and general performance, rather than only on niche tasks.Every evaluated tree includes unpopular models that are better generalists than popular base versions.
- Failure of Heuristics: Over 90% of identified gems lack relevant performance documentation, and their locations do not follow predictable trajectories in model trees.These findings undermine popularity and graph centrality as reliable search heuristics.
3 Efficient Model Discovery
The paper treats selecting the best model under a query budget as fixed-budget best-arm identification and accelerates Sequential Halving for this setting. Its modifications eliminate weak candidates earlier and allocate more queries to elite survivors.
- Problem Formulation: The task is to identify the best-performing model in a model tree under a global query budget.Candidate models are arms, individual evaluations are actions, and correctness provides binary rewards.
- Problem Formulation: Fixed-budget Best-Arm Identification frames discovery as pure exploration, minimizing the final gap between the selected model and the best model.
- Sequential Halving: Sequential Halving repeatedly evaluates candidates, eliminates the bottom half by empirical accuracy, and increases testing for survivors.Standard rounds begin with a small uniform allocation of 5-25 queries per model.
- Evaluation: Table 2 reports mean retrieval rank and accuracy for each method and query budget, averaged over 100 repetitions.
- Accelerated Search: The fail-fast schedule reduces the candidate pool to 100 models in the first round, reallocating compute toward distinguishing elite candidates.This addresses budget wasted on obviously poor models under standard 50%-per-round elimination.
4 Experiments
The experiments evaluate model discovery across four model pools and compare the proposed method with eight established bandit and elimination baselines. The method consistently finds hidden gems under both low- and mid-budget settings, while standard baselines often fail to surpass popular base models or find the strongest available models.
- Experimental Setting: Four model pools from Qwen-3B, Qwen-7B, Mistral-7B, and Llama3-8B trees are evaluated with 10- and 50-query-per-model budgets across 100 repetitions.Performance is reported using mean rank and top-1 accuracy against eight established algorithms.
- Results: At N = 10, standard baselines often fail to outperform popular base versions in the Qwen and Llama trees.The Mistral-7B tree is an exception, with many models surpassing its base model.
- Results: At N = 10, the proposed method consistently identifies Hidden Gems that significantly outperform the popular consensus across all trees.
- Results: At N = 50, the proposed method consistently converges to a top-3 model with much better accuracy than elimination-based baselines.The method uses correlated sampling and aggressive pruning; ablations confirm both design choices.
5 Conclusion
The paper identifies hidden gems as unpopular models that outperform foundation checkpoints and proposes accelerated Sequential Halving to make finding them practical. With only 50 queries per candidate, the method consistently identifies elite models.
- Hidden Gems are unpopular models that strictly outperform foundation checkpoints.
- The paper addresses the computational intractability of finding hidden gems with an accelerated Sequential Halving algorithm.
- 50 queries per candidate are sufficient for the method to consistently identify elite models.
A Limitations
The evaluation does not cover every possible task, and alternative weight-space learning approaches remain limited to small-scale networks and benchmarks. These boundaries constrain how broadly the findings and future acceleration strategies can be applied.
- Weight-space learning could further accelerate discovery but is currently limited to small-scale networks and benchmarks.
- The study evaluates hidden gems in math, coding, question answering, and general performance, but not all possible tasks.Finding gems for a new task would require reevaluating the models on that task.
- Weight-space learning aims to represent neural networks directly in weight space for search, comparison, and discovery, but faces extreme parameter dimensionality and parameter-space symmetries.
C Implementation Details
The implementation evaluates selected models from four large model trees using uniformly sampled benchmark queries and consistent decoding conditions. A custom scheduler aggressively reduces the candidate pool early to improve discovery efficiency and detected-model quality.
- Model Selection: For each tree, the study samples 400 fine-tunings and, when available, 400 adapters, prioritizing 300 popular models and adding 100 random models.Download or execution failures reduce the final evaluated pool.
- Query Selection: The evaluation uses 2,500 uniformly allocated queries across ARC-C, Winogrande, MMLU, MBPP, and GSM8K.Queries within each task are randomly sampled from the original benchmarks.
- Model Evaluation: Models are evaluated with and without system prompts, retaining the higher-accuracy version for each task.Greedy decoding is used, with maximum response lengths of 50 tokens for multiple-choice and short-answer tasks and 512 tokens for coding and math.
- Scheduler: The custom scheduler keeps 100 models after its first elimination round, approximately 20% of the model tree on average.The scheduler is designed around skewed accuracy distributions and improves the accuracy and rank of detected models.
D Ablation Studies
The ablations show that the aggressive scheduler improves retrieval under fixed budgets, while correlated sampling becomes beneficial at larger budgets.
- D Ablation Studies: At 10 queries per model, the custom schedule decreases average retrieval rank by over 30 across trees.The scheduler reallocates computation toward more informative comparisons.
- D Ablation Studies: At very small budgets, correlated and independent sampling perform comparably because evaluations are highly noisy.
- D Ablation Studies: The ablation evaluates mean rank and accuracy for each model tree across 100 repeated experiments.
- D Ablation Studies: At a budget of 50 queries per model, correlated sampling achieves better results in all trees and retrieves a top-3 available model.
E Additional Budgets Results
Across additional query budgets, increasing evaluation generally saturates performance, but the proposed approach remains competitive or superior while using fewer queries.
- E Additional Budgets Results: At 25 queries per model, the proposed method exceeds every baseline given 50 queries per model.A similar advantage appears when comparing 50-query results with baseline results at 100 queries.
- E Additional Budgets Results: Under equal budgets, the approach consistently identifies models with higher average accuracies and lower average ranks.
F Model Accuracy Distributions
Cumulative accuracy distributions across model trees show that most models are substantially worse than the best hidden gem, while documentation rarely makes gems searchable.
- F Model Accuracy Distributions: Across all tasks and trees, the vast majority of models are more than 10% worse than the best model.The distributions mark thresholds at 10% and 5% below the best gem model.
- F Model Accuracy Distributions: Only 2 of 24 hidden gems could have been found through relevant performance documentation.Nineteen had no documentation, while three documented irrelevant tasks.
- F Model Accuracy Distributions: Text search would likely miss a Qwen-3B coding gem because its documentation reported Fortran performance while the target task was Python coding.
H Additional Atlas Visualizations
Atlas visualizations map task-specific and overall performance across several model trees, showing that hidden gems are usually scattered and less downloaded than popular base versions.
- H Additional Atlas Visualizations: The atlas color-codes model trees by GSM8K math, MBPP coding, and RouterBench overall performance.
- H Additional Atlas Visualizations: Hidden gems often have limited download counts and are scattered across trees rather than concentrated in predictable branches.
- H Additional Atlas Visualizations: In the Qwen-7B tree, top coding models reside within the Qwen-7B Coder Instruct subtree.This is a notable exception to the broader scattering pattern.
- H Additional Atlas Visualizations: Curated official branches are not universal, and even large relevant subtrees can make exhaustive search computationally prohibitive.