Source-linked AI summary
Diversed Model Discovery via Structured Table Discovery
Zhengyuan Dong, Renée J. Miller
TL;DR
Semantic model search can return relevant but homogeneous candidates, limiting comparison across measurable model differences. The paper introduces Structured Semantic Search, which discovers and integrates model-card tables alongside semantic retrieval and evaluates query-relevant evidence with nuggets. Across the reported model-recommendation evaluation, the structure-aware pipeline shows improved nugget coverage over the semantic baseline.
Problem
Existing model search relies mainly on semantic similarity over model-card text, limiting diverse comparison-oriented candidate sets.
Method
Structured Semantic Search combines semantic anchor retrieval with table discovery, model-card remapping, orientation-aware table integration, and nugget-based evidence evaluation.
Results
The structure-aware pipeline shows improved nugget coverage over semantic baseline methods across 597 adapted model-recommendation queries.
Takeaways & Limitations
Table-centric retrieval complements semantic search by surfacing diverse, structured evidence and producing comparison-ready candidate sets.
Takeaways & Limitations
Table integration remains challenging for transposed or schematically heterogeneous tables, and many model cards are incomplete or only partially table-backed.
Abstract
from arXiv · showhide
Model cards describe model behavior through a mixture of textual descriptions and structured artifacts, including performance, configuration, and dataset tables. Existing model search systems rely predominantly on semantic similarity over text, which can produce homogeneous result sets and limit exploration of alternatives. We argue that model search is inherently comparative: users want models that are task-aligned yet differentiated in measurable ways. We hypothesize that this balance requires retrieval over condensed, high-quality evidence rather than verbose descriptions, and much of that evidence is concentrated in structured tables. We present StructuredSemanticSearch, a table-driven model search framework built on the ModelTables benchmark. Given a query, StructuredSemanticSearch combines a semantic baseline for task alignment with a structure-aware pipeline that discovers query-related model-card tables using table discovery operators such as unionability, joinability, and keyword search. Retrieved tables are mapped back to model cards under a controlled top-k budget, enabling fair comparison between text-based and table-based retrieval. Beyond retrieval, StructuredSemanticSearch adapts table integration to the model-table domain through orientation-aware integration, producing compact integrated views of tables from partially overlapping and sometimes transposed evidence tables. For evaluation, we introduce a nugget-based, auditable protocol that extracts compact evidence items from model cards, matches queries to condition- or intent-specific nuggets, and measures evidence coverage and diversity over retrieved model-card candidate sets. This protocol also provides a scalable path toward approximate, evidence-based labeling in dynamic model lakes. Experiments on 597 model-recommendation queries show improved nugget coverage for the structure-aware pipeline than semantic baseline
1 INTRODUCTION
Model search must find task-aligned models while exposing measurable, non-redundant alternatives for comparison. The paper therefore centers structured tables as condensed evidence and evaluates retrieval with auditable nuggets.
- Model lakes organize machine-learning models whose cards describe training data, evaluation results, and intended usage.
- Semantic and keyword-based systems find individually relevant models but provide limited support for comparison-oriented candidate sets.
- Task alignment and diversity are competing requirements because textual similarity can cluster results around dominant model families.
- Structured performance, benchmark, and configuration tables concentrate decision-critical evidence with less stylistic variation than narrative descriptions.
- Nuggets are fixed-attribute tuples covering model, base model, model variant, dataset, metric name, and metric value.
- The paper contributes table-driven discovery, nugget-based coverage and diversity evaluation, orientation-aware integration, and an end-to-end implementation.
2 RELATED WORK
Related work places model search within model-lake management and data discovery, while highlighting limits of document-level retrieval and the value of atomic evidence units.
- Model-lake research covers attribution, provenance, versioning, search, benchmarking, reporting, and documentation generation.
- Model cards record model details, intended use, training data, evaluation results, and limitations, but can be incomplete, inconsistent, or difficult to compare.
- Data discovery research studies finding useful datasets and tables, including keyword, joinable, and unionable search in heterogeneous data lakes.
- Traditional document-level retrieval metrics do not directly measure whether retrieved results cover the breadth of an information need.
- The paper borrows leaderboard literature’s tuple-oriented representation of methods, datasets, metrics, and scores without targeting leaderboard construction.
3 METHODOLOGY
Structured Semantic Search combines semantic anchor retrieval with table discovery, then maps discovered tables back to model cards under the retrieval budget.
- Unstructured Semantic Search retrieves model cards through keyword or natural-language search, while Structured Semantic Search adds table-aware candidate generation.
- The semantic baseline has dense, sparse, and hybrid implementations using Sentence-BERT with FAISS, Pyserini, and hybrid reranking.
- Structured Semantic Search follows a query-to-card-to-table-to-card workflow that expands candidates through shared structured evidence.
- The pipeline begins with a top-ranked anchor card constrained to have at least one associated table.
- Table discovery uses keyword, joinable, and unionable operators over headers, first-column values, shared entities, and alignable columns.
- Each discovered table is mapped to one model card by selecting the associated card with the highest semantic retrieval similarity.
4 MODEL RANKING EVALUATION STRATEGY
The evaluation compares semantic and table-based retrieval through query-aware nugget coverage and qualitative table integration under the same top-k budget.
- The evaluation targets comparative evidence quality rather than a static ranking benchmark, because model lakes continuously expand and fixed annotations become stale.
- Nugget-based Quantitative Evaluation: Nugget evaluation represents query-relevant evidence as compact, auditable units extracted from model cards and matched to query constraints.
- Nugget-based Quantitative Evaluation: Each nugget is a six-attribute tuple containing model, base model, model variant, dataset, metric name, and metric value.
- Nugget-based Quantitative Evaluation: Nuggets are extracted from heterogeneous card content into a fixed-schema table, with extraction needed only for newly added cards.
- Nugget-based Quantitative Evaluation: The candidate-set score counts unique query-relevant nuggets in the union of retrieved cards, preventing overlapping evidence from being counted repeatedly.
- Table-based Qualitative Evaluation: Orientation-aware integration organizes partially overlapping, noisy, or transposed retrieved tables into a coherent comparable view.
5 EXPERIMENTS
Experiments compare text-based and structure-aware model retrieval on 597 adapted model-search queries, evaluating nugget coverage across retrieval budgets and qualitative table-based comparisons. Structure-aware retrieval generally surfaces broader, more diverse, and more directly comparable evidence, while performance varies with operator and top-k budget.
- 5.1 Dataset: The evaluation uses 597 adapted LitSearch queries and compares three Unstructured Semantic Search variants with three Structured Semantic Search variants.Queries are rewritten from paper-recommendation requests into model-search requests, then evaluated using returned top-k model-card candidates.
- 5.1 Dataset: Evidence-based queries dominate the adapted benchmark, while other query-intent categories occur less frequently.The intent analysis is descriptive and precedes the quantitative evaluation.
- 5.2 ModelCard-Level Quantitative Evaluation: Across top-1, top-3, and top-5, unionable search is the strongest structure-aware operator, while joinable and keyword search remain competitive with most semantic baselines.Joinable search is more selective, keyword search is more brittle, and sparse retrieval remains strong when exact lexical overlap captures task alignment.
- 5.2 ModelCard-Level Quantitative Evaluation: At top-10, retrieval rankings shift and sparse and hybrid methods become more competitive, showing that retrieval depth affects accuracy.The relative advantage of the strongest structure-aware operator becomes less pronounced at the larger budget.
- 5.3 Table-Level Qualitative Evaluation: Structured Semantic Search produces broader, more diverse comparison views by integrating table-backed evidence across related model cards.The edge-device example organizes attributes including device, chipset, runtime, latency, memory usage, precision, and compute unit.
- 5.3 Table-Level Qualitative Evaluation: For OCR-heavy document understanding, unionable table search retrieves compatible benchmark tables that directly support comparison and table question answering.Dense model-level retrieval returned top cards without usable tables, whereas unionable search returned two tables with nearly identical schemas.
6 CONCLUSION
Structured Semantic Search treats model-card tables as searchable and integrable evidence, complementing semantic retrieval with more diverse, comparison-ready model information. Evaluation and case studies support its usefulness for evidence-grounded model decisions, while integration over heterogeneous tables remains an open challenge.
- Structured Semantic Search treats tables as searchable and integrable evidence units rather than relying only on model-card-level semantic retrieval.
- The structure-aware pipeline retrieves benchmark tables aligned with tasks, enabling direct model comparison and fine-grained version analysis within model families.
- Evaluation on 597 model-recommendation queries uses nugget-based quantitative evidence coverage, complemented by qualitative cases of coherent, comparison-ready integrated tables.
- Future work must address transposed or schematically heterogeneous tables, incomplete table-backed model cards, and query workloads with different intent distributions.