Source-linked AI summary
Automatic Model Card Generation Using an LLM
Tajkia Rahman Toma, Balreet Grewal, Cor-Paul Bezemer
TL;DR
Model-card documentation requires information about model development, limitations, and appropriate use, while many models lack documentation. The paper introduces LLM-based approaches for reorganizing and generating model cards, reporting high retention and accurate placement for reorganization. It also indicates that generation quality is constrained by source alignment and grounding.
Problem
Model cards are intended to provide essential information about model development, limitations, and appropriate use, but many models still lack documentation altogether.
Method
The paper introduces MCTidy to reorganize model cards and examines how input data influences generated model cards, including derivative-model information from YAML.
Results
MCTidy retains most content during reorganization, with 4.5% of checklist items missing and 1.9% incorrect placement; results also show high semantic consistency across runs.
Takeaways & Limitations
LLM-based methods show potential for reorganizing model-card content while preserving information and placing it correctly.
Takeaways & Limitations
Improving source alignment and grounding could further enhance factual accuracy.
Abstract
from arXiv · showhide
Model cards are structured documents that summarize key information about machine learning models to improve transparency, usability, and accountability. However, they often lack a consistent structure, and many models provide no model cards, making comparison and interpretation difficult. This paper presents two contributions. First, we propose MCTidy, an LLM-based approach that reorganizes existing model cards into a standardized template to improve clarity and comparability. Second, we introduce MCGenie, an LLM-based system that generates model cards directly from model repository data. We apply MCTidy to 48 Hugging Face model cards and evaluate information retention, section alignment, hallucination, and stability. Our findings show high information retention with minimal textual loss, accurate section assignment, rare hallucinations primarily in descriptive sections, and strong stability across runs. We assess MCGenie by generating model cards for the same 48 models and assessing semantic similarity, factual correctness, and sensitivity to input resources. The generated model cards achieved high semantic similarity (mean around 0.9); over half were fully correct, and most remaining errors were minor. Generation quality depended strongly on the availability of supporting resources, particularly associated papers. Overall, our findings demonstrate the potential of LLM-based methods to enable scalable, standardized model card documentation.
1 Introduction
The paper addresses inconsistent or missing model-card documentation with two LLM-based approaches: MCTidy reorganizes existing cards, while MCGenie generates cards from repository resources. Across 48 Hugging Face models, both approaches showed strong retention and quality, with generation depending particularly on supporting papers.
- MCTidy: MCTidy automatically reorganizes existing model-card content into a standard template using an LLM.Its goals are improved structure, readability, consistency, comparison, and evaluation.
- MCTidy: 48 existing Hugging Face model cards were reorganized and evaluated for retention, section alignment, hallucination, and run-to-run stability.
- MCTidy results: 4.5% of checklist information items were missing after MCTidy reorganization, while only 1.9% of sections contained misplaced content.
- MCTidy results: MCTidy’s hallucinations and misinterpretations were concentrated in specific sections, and focused human oversight was considered sufficient to catch occasional errors.
- MCTidy results: 0.97 median average semantic similarity across three runs supported high stability, with 87.5% of subsections scoring at least 0.90.
2 Study Setup
The study uses a standard model-card template and repository resources to evaluate MCTidy and MCGenie on a curated set of 48 Hugging Face model cards. It prepares information checklists, verifies reorganizations, and compares generated cards with validated references.
- Template and workflows: MCTidy uses an off-the-shelf LLM to reorganize model-card content according to a standard template with section and subsection descriptions.
- Template and workflows: MCGenie uses the same template to structure general model documentation and repository resources, including associated papers, tokenizer files, and configuration files.
- Model-card selection: 48 model cards remained after filtering Hugging Face repositories for English, sufficient descriptive length, base-model status, non-duplication, and manual curation.
- Information checklist preparation: Information checklists decompose original-card sentences into distinct information units for systematic preservation checks after reorganization.
- Information checklist preparation: 0.03 was the distance between GPT-4o mini-generated and manually corrected checklists, indicating that only minimal edits were required.
- MCGenie generation: 26 model cards were generated without exceeding Gemini’s maximum input-token limit, and 18 more could be generated after extracting key information from files.
3 Evaluation of MCTidy
Across 48 model cards, MCTidy retained most information, placed content accurately, introduced limited hallucinations, and produced stable reorganizations across runs. Manual review identified remaining omissions and corrections, especially in descriptive sections.
- Information retention: MCTidy retained a median of 93.8% of original model-card information, with 1.6% of checklist items partially missing and 4.5% completely missing.Six model cards retained all checklist items, while the lowest retained 47.9%; missing-item counts were not significantly correlated with checklist size (p = 0.1310).
- Section alignment: MCTidy placed 98.1% of content in the correct section, with 28 of 1,440 (sub)sections confirmed as containing misplaced content.The confirmed cases were broadly distributed, with the highest count being four in Quantitative Analyses/Unitary results.
- Hallucination: Hallucinations occurred most often in Primary intended users and Ethical Considerations, where content was more descriptive or interpretive.Observed errors included broadening user groups, inserting template ethical statements, inferring unsupported motivations, and mishandling referential terms.
4 Evaluation of MCGenie
MCGenie generated model cards with high document-level semantic similarity and generally preserved original informational content, but similarity varied across sections and depended on source availability. More than half were fully factually correct, while remaining errors were usually limited and often reflected source inconsistencies or unsupported interpretation.
- Document-level similarity: 0.9 mean and median cosine similarity indicate that generated model cards generally preserved the key informational content of the originals.Section-level similarity ranged from 0.62 to 0.99, with an average of average subsection scores of 0.84 across 48 model cards.
- Section-level similarity: 0.88–0.91 similarity was achieved by sections with straightforward content, whereas Evaluation Data, Training Data, and Caveats averaged 0.76–0.80.Sections requiring interpretation, synthesis, or contextual judgment were harder to replicate fully.
- Empty sections: 16 subsections became more often empty and 13 became less often empty than in reorganized model cards, with changes below 30 percentage points except Evaluation Data/Preprocessing.That subsection’s proportion of empty subsections decreased by 33.3 percentage points.
- Factual correctness: 54.17% of model cards had all sections correct, while the remaining 22 cards averaged 1.41 incorrect sections, or 4.14% of all sections.The median number of incorrect sections among the remaining cards was 1.0, with a range from 1 to 3.
- Error patterns: 60% of inaccuracies arose from mismatches between generated content and cited sources, while about 23% involved unsupported interpretive or inferential claims.Examples included incorrect dataset sizes, misreported hyperparameters, conflicting schedules, and plausible but unverifiable statements.
5 Threats to Validity
The evaluation faces threats from subjective manual judgments, LLM-based assessment, inconsistent repository evidence, a uniform template, and a limited sample. These constraints limit confidence in generalizing the findings across model types, documentation quality levels, and repository-resource configurations.
- External validity: Repository information may be outdated, scattered, ambiguous, or mutually inconsistent, potentially producing conflicting or incomplete generated model cards.Such source variation is an internal and external validity concern for evaluating MCGenie.
- Measurement: Partial-match scoring and manual interpretation remain subjective, potentially introducing variability into completeness assessments.Checklist items differ in granularity, so some contain multiple pieces of information.
- Internal validity: LLM juries provide scalable evaluation but may reduce reliability because hallucination, nondeterminism, superficial heuristics, bias, and opaque reasoning can diverge from expert judgment.Human validation was added to mitigate these risks, but the initial reliance on LLM evaluation remains a validity threat.
- Construct validity: MCTidy sometimes places uncertain content in Additional Information, which can concentrate material there and distort the intended organization.This fallback helps avoid information loss but threatens construct validity.
- Construct validity: A uniform template may underrepresent domain-specific documentation patterns and overlook nuances required in specialized contexts.The paper notes that documentation practices vary with domain and model type.
- External validity: The 48-model evaluation sample is not statistically representative, so findings may generalize poorly to less curated or lower-quality model cards.MCGenie’s performance may also vary when repositories contain different types of resources.
6 Related Work
Related work documents gaps in model-card content and proposes standards, quality dimensions, extraction tools, interactive formats, and authoring support. This paper differs by targeting a more widely adoptable approach aligned with popular documentation formats.
- Documentation gaps: Studies of existing model documentation identify gaps involving ethics, behavioral risks, intended use, risk mitigation, datasets, biases, licenses, and limitations.Analyses also report inconsistent transparency documentation and insufficient information across model-card sections.
- Documentation recommendations: Prior recommendations emphasize causality, decision provenance, computational tests, and detailed factual information to improve transparency and accountability.Other qualitative work finds developers selective about documenting ethical issues and hesitant to grant models full autonomy.
- Authoring support: Existing authoring tools include Model Card Toolkit, which organizes required information, and DocML, which generates documentation sections in computational notebooks.The Model Card Toolkit can also support user-facing interfaces and JSON-schema-based content generation.
- Alternative formats: FactSheets provide a seven-step, user-centered method for stakeholders to identify and progressively build documentation information.The approach represents an alternative documentation format rather than standard model cards.
- Automated extraction: AIMMX automatically extracts high-level contextual information from model repositories and supports scalable model discovery through a searchable catalog.Its motivation includes reducing manual effort and addressing missing or inconsistent standardized documentation.
- Interactive documentation: Interactive Model Cards use expert and non-expert input to create a more accessible format, evaluated through interviews and a prototype comparison with traditional cards.The evaluation involved 10 ML/AI experts and 20 non-expert analysts.
- Positioning: Because prior approaches often rely on metadata stores, computational notebooks, or alternative formats, this study aims for a more widely adoptable approach aligned with commonly used documentation formats.The stated positioning emphasizes compatibility with popular model-card practices.
7 Conclusion
The paper presents MCTidy for standardizing existing model cards and MCGenie for generating new cards from repository data. Across 48 Hugging Face models, both approaches showed promising scalability, while MCGenie remained sensitive to source-resource quality.
- MCTidy: MCTidy reorganizes model-card content into a standard template and was evaluated on 48 Hugging Face model cards.The approach aims to improve structure and clarity without altering the original content.
- MCTidy: 93.8% of checklist items were retained by MCTidy, while 1.9% of (sub)sections contained misplaced content.These results indicate high retention and mostly correct section placement.
- MCTidy: 10.1% of (sub)sections contained hallucinated content, which typically appeared in non-technical areas.The hallucinations were described as rare and generally easy to identify and correct.
- MCTidy: MCTidy achieved a 0.97 semantic similarity score for full documents and at least 0.90 for 87.5% of (sub)sections across runs.The reported stability supports automated standardization with minimal human oversight.
- MCGenie: MCGenie generated cards with mean document-level semantic similarity around 0.9, and 54.17% of cards were entirely correct.Most remaining cards contained only a small number of minor inaccuracies.
- Overall implications: Together, the findings indicate that LLM-based reorganization and generation can support scalable, low-effort model-card documentation with minimal human-in-the-loop requirements.The paper also released MCTidy, MCGenie resources, and reorganized model cards in a replication package.
A Model Card Sections’ Description for LLM
The model-card template is described for LLMs by listing its sections and identifying additions to the Mitchell et al. standard. Sections introduced by Toma et al. are visually highlighted.
- Template structure: Listing 1 describes the model-card sections proposed by Mitchell et al. and the additional sections introduced by Toma et al.The listing is intended to inform LLMs about the template structure.
Model Details
The Model Details section provides contextual and technical information needed to understand a model, its development, and its provenance. It covers model identity, architecture, training, creators, dates, versions, resources, licensing, and contact channels.
- Model identity: Model Details should explain the model’s context, architecture, category, core components, size, and supported context length when available.Examples include Transformer, Convolutional Neural Network, text generation, image classification, and reinforcement learning.
- Provenance: The section should identify the developing person or organization and provide background, affiliations, profiles, or official websites when available.This information helps stakeholders identify creators and assess credibility.
- Timeline and versions: Model dates should record development milestones and release timing, while model versions should explain improvements, fixes, features, or architectural changes.These details help readers track the model’s timeline and revisions.
- Training: Training details should cover algorithms, parameters, hyperparameters, fairness constraints, optimization techniques, and other methodologies with enough depth to explain the model’s development.Examples include supervised learning, reinforcement learning, learning rate, and number of layers.
- Resources and access: The section should link papers, repositories, technical blogs, documentation, licenses, and contact channels that support understanding, use, attribution, and feedback.License information should state usage restrictions, while citation details should provide formats such as BibTeX.
Intended Use
The Intended Use section explains the model’s applications, users, usage process, and boundaries. It specifies primary tasks and audiences while documenting out-of-scope or potentially misapplied uses.
- Primary uses: Intended Use should describe the model’s primary purposes, target tasks or domains, capabilities, and input-output structure.It may also suggest alternative models when they are more suitable for particular contexts.
- Users: The section should identify intended users, including researchers, developers, businesses, or educators, and describe their expertise and typical use cases.The audience description clarifies who the model is designed to serve.
- Boundaries: Out-of-scope uses should list applications and potential misuse cases for which the model was not designed.The template asks authors to identify contexts where the model might be misapplied or confused with related technologies.
- Usage: The section also explains how to use the model through input-output details, settings, code snippets, example outputs, documentation, and tutorials.These materials are included when available.
Factors
The Factors section identifies variables that may affect model performance and distinguishes relevant factors from those analyzed during evaluation.
- Relevant factors may include demographic variations, environmental conditions, and data collection methods.The section also asks how these factors were identified and why they matter.
- Evaluation factors should be identified and reported, with explanations when they differ from the broader relevant factors.For example, an evaluation may prioritize accuracy metrics over demographic fairness.
Metrics
The Metrics section explains how model effectiveness is measured, how decision thresholds are selected, and how variation or uncertainty is estimated.
- Model performance measures: Model effectiveness may be assessed using accuracy, F1 score, precision, or recall, with justification for metric selection.The section asks why chosen metrics are more suitable than alternatives.
- Decision thresholds: Decision thresholds should be described alongside their rationale and any empirical evidence supporting them.Spam-email classification is given as an example application.
- Variation approaches: Performance estimation should report uncertainty measures and statistical procedures such as cross-validation or bootstrapping.These methods are presented as ways to support robust estimates.
Evaluation Data
The section specifies documentation for evaluation and training data, preprocessing, disaggregated results, hardware requirements, ethical considerations, caveats, and recommendations.
- Evaluation Data: Evaluation-data documentation covers dataset details, motivation, and preprocessing, including size, diversity, source, relevance, and steps such as normalization or filtering.The template also asks how preprocessing aligns with the model’s design and intended use.
- Training Data: Training-data documentation covers dataset characteristics, suitability, and preprocessing steps such as tokenization, resizing, or outlier removal.The guidance asks authors to explain how preprocessing improved training.
- Disaggregated Evaluation Results: Disaggregated evaluation reports unitary results for individual factors and intersectional results across factor combinations.Examples include demographic groups, environmental conditions, and demographic groups within geographic locations.
- Memory or Hardware Requirements: Hardware documentation addresses memory and hardware requirements for loading, deploying, serving, and training the model.Examples include RAM or VRAM, disk space, and CPU, GPU, or TPU requirements.
- Ethical Considerations and Caveats: Ethical documentation covers sensitive data, application risks, affected groups, mitigation strategies, known fraught use cases, and unresolved areas for exploration.The caveats guidance also identifies evaluation gaps and suggests future testing or research.