Source-linked AI summary
Modelpedia: A Catalog of Model Findings for the Meta-Science of AI
Franciszek Bernat, Dawid Płudowski, Michał Jan Włodarczyk, Luca Longo, Jianlong Zhou, Andreas Holzinger, Riccardo Guidotti, Wojciech Samek, Przemysław Biecek
TL;DR
Findings about AI models are scattered across papers and difficult to organize, retrieve, compare, and reuse. Modelpedia addresses this gap with an automated, LLM-assisted pipeline that extracts, links, validates, and aggregates findings into a searchable catalog. The prototype captured 1026 findings from two conferences and used the catalog to examine how the community investigates models.
Problem
Findings about model behavior are scattered across papers and reports, making them difficult to retrieve, compare, and reuse as knowledge about individual models.
Method
Modelpedia extracts findings from published papers, links them to models and related information, and aggregates them in a structured searchable catalog.
Results
1026 findings were captured from two conferences and validated with Claude Opus 5 with extra thinking.
Takeaways & Limitations
The catalog can support meta-analysis of which models and benchmarks receive attention and reveal under-studied areas, including reliance on correlational evidence.
Takeaways & Limitations
Modelpedia presents published findings as-is and does not evaluate whether the methodologies or findings are correct.
Abstract
from arXiv · showhide
Scientific knowledge about AI models is produced faster than the community can organize it. Every few months a new foundation model reshapes the field and hundreds of papers, blogs, and technical reports document how each behaves or fails. Yet, these findings remain scattered and effectively unretrievable. To address this gap we present Modelpedia, an automated, LLM-assisted framework that extracts findings about models from published papers, links it to the model, dataset, method, and concept it concerns, and aggregates the result into a searchable public catalog. Applying the prototype to accepted ICLR 2024 and 2025 papers, we extract over a thousand findings and, treating the catalog itself as an object of study, run a meta-analysis of how the community investigates models. Now, we invite the community to explore, contribute to, and build on the open catalog, and to help establish model findings as a shared foundation for the meta-science of AI.
1 Introduction
Modelpedia addresses the scattering and poor retrievability of findings about model behavior by organizing them into a unified catalog. It provides an automated, LLM-assisted prototype that links findings to models and related information and supports meta-analysis.
- The gap: Findings about model behavior are scattered across papers and reports, phrased inconsistently, and difficult to retrieve, compare, or reuse.Existing meta-analyses focus on broader research areas or model applications rather than accumulating behavioral findings about individual models.
- The proposed vision: Modelpedia represents model findings in a unified, structured form analogous to shared documentation templates for models and datasets.The catalog is intended to make findings comparable and reusable.
- The prototype: Modelpedia is an automated, LLM-assisted framework that extracts findings from published papers and aggregates them into a single knowledge catalog.The prototype links extracted findings to models from papers accepted to ICLR 2024 and 2025.
- Contributions: The prototype extracts, verifies, links, and aggregates findings into a searchable catalog and provides a web service for access.The paper also uses the catalog for a meta-analysis of which models and benchmarks concentrate community attention.
2 Modelpedia Architecture
Modelpedia combines a structured database with an LLM-assisted extraction pipeline for organizing findings about models. The prototype ranks ICLR articles for relevance, extracts and links findings, and validates 1026 findings from two conferences.
- Database structure: Each finding is linked to a model variant, dataset, method, concept, and source, with optional links to related works.The database uses one main findings table and five related tables; concept includes categories such as failure mode or shortcut.
- Findings extraction: The prototype uses OpenReview articles, Qwen3.8-27B, and an ICLR-only extraction process to identify potential findings about models.Extracted articles are converted to text files and supplied to an LLM prompt designed around entity–quote pairs.
- Article selection: Approximately the top 15% of articles are selected using abstract- and review-based relevance scores, trading computational cost against relevance.Approximately 45% of selected articles are subsequently found to lack any finding.
- Entity linking: LLM-based entity matching links extracted records to existing database entries or proposes new records when matching is not possible.This process is intended to avoid duplicates such as GPT-4o and ChatGPT-4o.
- Validation: 1026 findings were captured from two conferences and validated with Claude Opus 5 with extra thinking.The validator confirmed all findings, with a small discrepancy about the key metric field on approximately 20% of records.
3 The case for Modelpedia
Modelpedia’s meta-analysis shows that model research relies mainly on correlational evidence and concentrates on language and vision-language models, with vision evaluations centered on a few established datasets.
- Most articles report correlational evidence, while observational or interventional evidence is used only by a minority.This reliance occurs for both closed- and open-source models.
- Observational and interventional findings rely mostly on open-source models because interventions often require access to model weights.The authors present this pattern as highlighting the need for greater emphasis on analyzing open-source models.
- Vision-based evaluation mostly uses ImageNet and COCO, whereas language-based benchmarks are much more diverse.
- Language and vision-language models are by far the most analyzed modalities, especially Llama, GPT, and Mistral families.Many findings also concern closed-source GPT models, which restrict researchers mostly to correlational evidence.
4 Discussion
Modelpedia treats model findings as citable, attributable knowledge and uses an expanding catalog to support meta-analysis of what the field has studied and overlooked. Its current prototype does not verify whether the underlying published findings are methodologically correct.
- Modelpedia treats findings about models as first-class, creditable units that can be cited, versioned, and attributed.This is intended to preserve observations that might otherwise remain buried in appendices or scattered across papers.
- As the catalog grows across conferences, it can reveal both what the field has learned and what it has not studied.The authors identify reliance on correlational over interventional evidence as one visible blind spot.
- The authors position the current version as shared infrastructure that the community can expand, curate, and build on over time.
- Modelpedia presents findings as-is and does not evaluate whether the methodologies in source articles are correct.Verifying whether published findings hold remains an unsolved community problem, although the catalog makes it more tractable.
A Data structure
Modelpedia organizes each finding around shared information nodes, retrieval-oriented links, related work, comprehension fields, and an evidence-type classification. Its ontology was developed from manually read seed papers and experience in explainable AI.
- Each finding links a model, dataset, method, source paper, and optional generalized concept.Models are foundational because findings are inherently tied to a model.
- Models, datasets, methods, and sources receive anchors and artifacts to support retrieval, although links are not always available.An anchor links to the presenting paper, while an artifact links to a downloadable model or dataset site.
- Related work records whether a study builds on, is compared to, or depends on prior methods, models, or papers for context.
- Titles, short summaries, and caveats are designed to help users understand findings without consulting the original paper.
- Evidence type distinguishes observational patterns, interventional changes producing different outputs, and correlational quantities.
- The ontology was formulated through close reading of manually selected exemplary papers and experience in explainable AI.Those seed papers generated the database’s initial manual-extraction records.
B Example of Finding
Modelpedia presents a structured finding record that connects a model-behavior claim to its source, concepts, method, related work, and available metadata. The example shows how qualitative evidence and explicit caveats are preserved alongside the extracted finding.
- Caveat and scope: The caveat limits the evidence to a local explanation and rejects a general causal claim that printed text drives CLIP matching.Neither the frequency of the pattern nor an intervention removing the printed word is reported.
- Linked entities: The record links the finding to CLIP, its source paper, the Weighted Banzhaf interaction index, and related work on visual shortcomings in multimodal LLMs.Modelpedia pages expose these linked nodes and source metadata to support navigation and context.
- Caveat and scope: The example omits Dataset and Key metric fields because the finding focuses on one qualitative case rather than a measured dataset-level result.The source paper references a dataset, but it is represented in a separate finding from the same paper.
- Finding record: The example records a qualitative CLIP finding in which FIXLIP links the caption token “doll” to an image patch containing “dollar.”The authors describe this as a case where the model may be right for the wrong reasons.
- Finding record: The finding is classified as observational and linked to the Shortcut and Feature interaction concepts.Its method is the Weighted Banzhaf interaction index.
C Scoring algorithm
Modelpedia uses a lightweight keyword-scoring pipeline to rank accepted ICLR papers for likely model findings. It combines separately scored abstracts and reviews, then downloads the highest-scoring papers for extraction.
- Corpus selection: 972 accepted papers were extracted from ICLR 2024 and 2025: 521 from ICLR 2024 and 451 from ICLR 2025.The prototype obtained conference metadata through the OpenReview API.
- Scoring algorithm: The scoring system matches keywords from abstracts and reviews, using seven weighted groups with per-group ceilings.Groups contribute w · min(m, c), where m is the number of distinct matched keywords and c is the group ceiling.
- Scoring algorithm: Review keywords count only when at least half of the paper’s reviewers use them independently.This agreement gate is intended to reduce noise from broad reviewer vocabulary.
- Scoring algorithm: The method matches terms as prefixes or whole words, with 30–60 keywords per group and eight generic behavior terms removed from review vocabulary.The removed terms include limitation, robustness, and bias.
- Ranking: Abstract and review scores produce two independent rankings that are merged by summing the scores.The five highest-scoring ICLR 2024 papers are shown in Table 2.
D LLM prompts
Modelpedia’s extraction prompts use constrained LLM decisions embedded in deterministic verification and registry stages. The pipeline extracts findings, validates entities and citations, assigns concepts, and fills model attributes while making refusals explicit.
- D LLM prompts: The extractor uses Qwen/Qwen3.8-27B-FP8 with medium thinking across four prompt templates and three extraction steps.The model must select identifiers from closed lists or explicitly state that none applies.
- D LLM prompts: Between LLM stages, citations are checked against source text, candidate names are matched to the registry, and accepted entities are written into the registry.Anchors are derived from the verified citations.
- D.1 Step I: Extraction: Step I defines a finding as a third-party, post-hoc claim about a specific pre-existing model, excluding models trained by the paper’s authors.The prompt requires citations for every named entity and records results that do not qualify as findings.
- D.1 Step I: Extraction: The extraction output exposes considered models, covered results, and near-miss concepts so that refusals and exclusions remain visible.Empty lists are valid when no entity or concept applies.
- D.1 Step I: Extraction: In the example, GPT-4 is retained as a released model, while the authors’ fine-tuned imitation models are rejected as independently uncitable checkpoints.The resulting GPT-4 finding concerns the same style-over-factuality preference observed in human ratings.
- D.1 Step I: Extraction: The GPT-4 example is marked correlational, and its interpretation is limited because rating agreement does not demonstrate a causal style-based proxy.The extracted claim concerns matching trends with human crowdworker evaluations.
- D.2 Step II: Entities: Step II asks whether newly encountered entity names deserve permanent registry entries, preventing accidental splitting or merging of shared entities.The example adopts Stanford Alpaca as a named dataset and refuses SGD because optimisers are excluded.
- D.3 Step III: Concepts and model attributes: Step III matches findings to closed-list concepts using quoted definitions and assigns model modality, task, and domain from registry entries.The Sybil example assigns feature-interaction using a definition about joint effects, while empty answers remain valid when attributes cannot be defined.
E Extended analysis
The extended analysis examines conference-score selection, the breadth of datasets and models per finding, and evidence-type coverage across models and articles. It highlights broad reliance on correlational evidence and sparse interventional evidence for closed-weight models.
- Evidence and extraction: The analysis reports that most findings use correlational evidence, while observational or interventional evidence appears only for a minority of researchers.Interventional evidence is concentrated mainly on open-source models, where weight access makes interventions feasible.
- Conference scores: Figure 3 compares the distributions of conference finding scores for ICLR 2024 and ICLR 2025, distinguishing papers above the threshold from the remainder.Purple marks papers above the threshold and gold marks the remaining papers.
- Finding composition: Most ICLR 2025 findings rely on one dataset and one model, with long tails reaching 11 datasets and 19 models.Zero datasets indicate either a newly proposed dataset or unavailable online data.
- Evidence and extraction: Closed-weight models show near-zero interventional finding shares, unlike the broader spread observed for open-weight models.Figure 5 also reports that approximately 45% of ICLR 2025 articles yielded no clear findings, with at most 5 findings per article.