What Carries Forward? Pretraining Disclosure Across Language Model Releases

• Pretraining information is needed to investigate model behavior and interpret evaluation results, but weights alone do not ensure access to training data or development documentation.Naming a source does not establish which material was selected or how it was transformed.Source: §1 · p. 1
• In the Llama family, Llama 3.1 recovers processing disclosure but not Llama 1’s named-source sampling breakdown, while acquisition scores continue to decline.More detailed corpus-preparation documentation need not provide a more specific account of corpus origins.Source: §1 · p. 1
• The document-based audit examines 41 model entries from 17 families using 24 pretraining indicators and tracks changes across 24 adjacent comparisons in ten families.It compares overall scores and indicator profiles using technical reports and associated official documentation.Source: §1 · pp. 1–2
• Scores decreased in 11 of 24 comparisons, increased in six, and remained unchanged in seven, with 38 indicator losses and 28 gains overall.Stable or improving totals can conceal losses in particular information.Source: §1 · p. 2

2.1 Pretraining Data and Documentation

• Corpus audits and experiments identify contamination, multilingual data-quality problems, exclusion through filtering, and effects of deduplication, data age, composition, and filtering.The paper distinguishes research on corpora or curation effects from its own analysis of disclosure about these aspects across releases.Source: §2.1 · p. 2

2.2 Openness and Independent Scrutiny

• Openness assessments distinguish available weights from access to training data, code, and documentation, while this audit assesses explicit requirements without inspecting corpora or testing reproducibility.Higher disclosure in the open-source/reproducible group is partly expected because development-resource availability informs both classification and scoring.Source: §2.2 · pp. 2–3

2.3 Transparency across Releases

• Prior transparency studies examine developer-level assessments, GPT documentation, and repository gaps; this audit tracks gains, losses, and retained pretraining disclosures across selected families under common criteria.Its approach distinguishes changes in total scores from changes in the information documented.Source: §2.3 · p. 3

3 Methodology

• The audit covers base-model pretraining, including continued pretraining when part of an included release, and applies 24 selected 2025 FMTI indicators.Instruction tuning, preference optimization, reinforcement learning, and downstream fine-tuning are excluded.Source: §3 · p. 3

3.1 Model Selection

• The selected corpus includes qualifying entries documented in developer-authored reports or research papers, with two official Marin retrospectives treated as exceptions.Grouped entries receive credit when evidence meets a criterion for at least one included variant, so their scores do not imply equal documentation across variants.Source: §3.1 · p. 3

3.2 Evidence Sources

• Evidence came from qualifying reports and associated official materials, and earlier documentation carried forward only when a later release explicitly established its continued applicability.The comparisons concern information documented for each entry; they do not establish that earlier documents were withdrawn.Source: §3.2 · pp. 3–4

3.3 Disclosure Indicators

• The fixed indicator subset covers acquisition, processing, data properties, compute, methods, model dependencies, and train–test overlap, with criteria restricted to pretraining.The subset retains the original indicator definitions and binary requirements while excluding topics outside its scope.Source: §3.3 · p. 4

3.4 Coding and Verification

• Indicators were scored as 1 when evidence met requirements and 0 otherwise, summed into equally weighted scores out of 24, and classified across adjacent entries as gains, losses, or retained disclosures.A zero means insufficient disclosure under a criterion, not that the underlying practice was absent; both authors verified the dataset, resolving discrepancies through source review and discussion.Source: §3.4 · p. 4

4.1 Distribution of Pretraining Disclosure

• The 41 entries averaged 8.90 of 24 points, with scores from 1 to 18, and none met all selected requirements.The median was 6.Source: §4.1 · p. 4

4.2 Disclosure Changes across Releases

• 11 of 24 adjacent comparisons had lower scores, six had higher scores, and seven were unchanged across ten families.The selected entries were assessed at the time of review; sequences include branches, continued-pretraining releases, and grouped variants rather than uniformly comparable generations.Source: §4.2 · p. 5

4.3 Aggregate Scores Conceal Changes in Disclosure

• 38 indicator losses and 28 gains occurred across the 24 comparisons, so an unchanged or improved total could still conceal lost disclosures.A loss means later evidence, including explicitly applicable earlier documentation, did not meet a previously satisfied requirement; it does not establish document removal or changed practice.Source: §4.3 · p. 5
• Data processing had nine gains and 13 losses, while compute had six gains and ten losses; these are transition counts, not rates.Families had different numbers of comparisons, and subdomains contained different numbers of indicators.Source: §4.3 · p. 5
• OLMo 2 and OLMo 3 both scored 17 despite three gains and three losses, while 14 criteria remained satisfied.Gains concerned crawling, data-domain composition, and compute hardware; losses concerned final-training compute usage, compute provider, and model dependencies.Source: §4.3 · p. 5, Table 2 · p. 5
• Llama 2 to Llama 3.1 rose from 6 to 8 through four gains and two losses, including losses for training-data usage and language composition.Across the Llama sequence, acquisition scores fell from 2 to 1 to 0, while processing changed from 3 to 0 to 3; Llama 3.1 did not restore Llama 1’s named-source breakdown.Source: §4.3 · p. 5

4.4 Persistent Gaps in Pretraining Disclosure

• All 41 entries met the model-stages criterion and 37 met each objectives and data-size criterion, but only two met language composition and three domain composition.None met the selected requirements for external data access or train–test overlap.Source: §4.4 · p. 5
• All 14 open-source/reproducible entries met three processing criteria, data replicability, and training-code access, yet none met language composition and only three met domain composition.Among 27 open-weight entries, 23 met data size, five acquisition methods, one public-dataset disclosure, and two language composition; none met domain composition, data replicability, or training-code access. Only two entries met final-training compute usage and two development duration; partial compute details did not always meet full requirements.Source: §4.4 · pp. 5–6

5 Discussion

• Pretraining disclosure is a property of individual releases that needs maintenance as models develop; later totals can improve without preserving earlier information.The comparisons extend lifecycle transparency work across selected families using common pretraining criteria.Source: §5 · p. 6, §5.1 · p. 6

5.1 Maintaining Disclosure across Releases

• A shared family name does not establish that earlier data sources, processing procedures, or training arrangements still apply to a later release.An explicit reference can identify unchanged earlier documentation without requiring developers to repeat it in every report.Source: §5.1 · p. 7

5.2 What Disclosure Scores Establish

• A total score cannot establish continuity: OLMo’s stable score offset gains with losses, while Llama’s improving score accompanied losses in data-related criteria.Hardware information and language composition answer different questions; adding information in one area does not resolve another.Source: §5.2 · pp. 7–8
• FMTI scores measure whether evidence meets selected requirements, not reproducibility, documentation usefulness, or responsible development.Open-source/reproducible releases’ higher scores partly follow from development-resource availability informing both group classification and scoring.Source: §5.2 · p. 8

5.3 Implications for Independent Scrutiny

• The audit identifies where documentation does not meet requirements; it does not establish comparable problems in the selected models’ corpora.Available data and code may permit further investigation, but deriving an answer from artifacts differs from receiving an explicit release-specific account.Source: §5.3 · p. 8
• Release documentation should identify applicable or changed pretraining descriptions and link them to supporting evidence, including the limits of reported overlap analyses.The paper motivates assessing both how much information is disclosed and whether its applicability remains clear as models evolve.Source: §5.3 · p. 8

6 Conclusion

• The audit examines 41 entries from 17 families using 24 selected FMTI indicators.It tracks gains, losses, and retained disclosures across model releases.Source: §6 · p. 8
• Across 24 within-family comparisons, disclosure scores decreased more often than they increased, and stable or improving totals sometimes concealed losses.The findings motivate release-specific documentation that identifies changes and the applicability of inherited information.Source: §6 · p. 8
• The findings motivate tracking both the extent and continuity of pretraining disclosure.The paper calls for release-specific documentation of changes and inherited information's applicability.Source: §6 · p. 8

Limitations

• The selected corpus is not representative of all language models and may favor releases with developer-authored technical reports.Findings characterize the selected entries, not ecosystem-wide disclosure prevalence.Source: Limitations · p. 9
• Family sequences are not controlled comparisons between equivalent successors, and pooled counts weight longer sequences more heavily.Branches, continued-pretraining releases, and grouped variants may differ in scope and reporting conventions.Source: Limitations · p. 9
• Grouped entries can combine criterion evidence from different variants, obscuring differences between model sizes.A group score does not imply that one variant meets the complete indicator profile.Source: Limitations · p. 9
• Assessment-time documentation and explicit inheritance rules mean unchanged practices may remain uncredited, while observed losses do not establish withdrawn information or changed practices.Earlier releases may benefit from later documentation updates or more time to accumulate supporting material.Source: Limitations · p. 9
• The 24 binary, equally weighted FMTI requirements do not capture every difference in disclosure specificity or usefulness.Scores do not directly establish reproducibility, data quality, or responsible development.Source: Limitations · p. 9
• Sequential verification against existing records does not provide an independent estimate of coding reliability, and errors may remain.Openness groups are operational classifications, and their score differences partly reflect criteria also used to classify them.Source: Limitations · p. 9

A Model Inventory

• Table 3 inventories 41 entries, their family groupings, documentation, and position in selected family sequences.Sequence order does not establish direct ancestry, and grouped entries receive credit when at least one variant meets a criterion.Source: A Model Inventory · p. 11

B Pretraining Disclosure Criteria

• The audit fixes a 24-indicator subset of the 2025 FMTI before coding, restricted to pretraining, including continued pretraining.The indicators cover origins and preparation of data, corpus properties, training resources, and base-model development.Source: B Pretraining Disclosure Criteria · p. 14

B.3 Data Properties

• Data-property criteria cover data size, language and domain composition, third-party access, and whether data descriptions support external replication.Size is measured for data directly used in building the model, while language composition requires proportions for languages making up at least 1% of the text data.Source: §B.3 · p. 15

B.4 Compute

• Compute criteria assess final-run compute use, provider, duration, and hardware.Compute guidance calls for appropriate units and a measurement methodology; duration guidance specifies continuous time and hardware hours.Source: §B.4 · p. 16

B.5 Methods

• Methods criteria ask whether development stages and their associated learning objectives or model updates are clearly described.The criteria recognize that developers may use different terminology or conceptualize stages differently.Source: §B.5 · pp. 16–17

B.6 Model Dependencies

• The model-dependencies criterion asks whether the models underlying or directly supporting a base model, and the derivation method, are disclosed.An explicit statement that a model has no such dependency can also meet the stated guidance.Source: §B.6 · p. 17

B.7 Train–Test Overlap

• The train–test overlap criterion asks for overlap and its calculation methodology for each reported capability evaluation.The guidance gives n-gram matching as an example of a methodology.Source: §B.7 · p. 17

C.1 AI-Assisted Coding

• ChatGPT assisted with evidence extraction and provisional coding, and both authors checked decisions against supporting sources.The second reviewer checked existing records sequentially, rather than independently assigning labels.Source: §C.1 · p. 17, §C.2 · p. 18

C.2 Evidence and Scoring Rules

• Evidence had to be attributable to the entry's pretraining, and earlier disclosures were carried forward only when later applicability was explicit.Binary scores indicate qualifying disclosure, not necessarily use; silence did not establish non-use.Source: §C.2 · p. 18

D Examples of Criterion Application

• The criterion-application examples explain the evidence considered and the basis for each final score.They illustrate the difference between relevant information and satisfying a complete disclosure requirement.Source: D Examples of Criterion Application · p. 18

D.1 Llama 1 / 1: Public datasets

• Llama 1's disclosed seven-source mixture and shares identify its largest public sources by volume.The listed mixture includes CommonCrawl at 67%, C4 at 15%, GitHub, Wikipedia, and Books at 4.5% each, arXiv at 2.5%, and Stack Exchange at 2%.Source: §D.1 · p. 18

D.2 Llama 2 / 0: Data acquisition methods

• Llama 2's description of a new mix of public sources does not distinguish existing datasets from direct crawling or provide an exhaustive acquisition-method inventory.The example therefore finds the disclosure insufficient under the FMTI acquisition taxonomy.Source: §D.2 · p. 18

D.3 Llama 3.1 / 16: Data language composition

• A fastText pipeline covering 176 languages is described, but whole-corpus proportions for languages at or above 1% are not reported.The FMTI criterion requires proportions plus the labeling method, so the criterion is not met.Source: §D.3 · p. 18

D.4 Llama 3.1 / 17: Data domain composition

• The 50/25/17/8 training-category mixture is not a complete corpus domain-composition analysis, so the Data domain composition criterion is not fully met.Source: §D.4 · p. 18

D.5 Llama 3.1 / 20: Compute usage for final training run

• The reported pretraining budget and GPU-hours do not establish the measurement or estimation methodology required for final-training compute credit.Source: §D.5 · p. 19

D.6 Llama 3.1 / 45: Train-test overlap

• An 8-gram contamination analysis covers key benchmarks, but omitted results and evaluations without overlap reporting leave the every-evaluation criterion unmet.Source: §D.6 · p. 19

E Supplementary Analyses

• Family-balanced prevalence and family coverage complement entry-level summaries, but do not establish prevalence in an unsampled model population; rankings and ties can differ.Source: E Supplementary Analyses · pp. 19, 21, Table 4 · p. 20
• Adjacent-release counts classify each indicator as gained, lost, retained, or continuously unmet; their totals include repeated transitions and are not independent statistical trials.Source: E Supplementary Analyses · p. 21, Table 5 · p. 22

F Complete Scores and Indicator Matrix

• Tables 6–8 provide complete entry and subdomain scores and the final 41-entry binary indicator matrix; a value of 1 means the pretraining-scoped requirement was met.Source: F Complete Scores and Indicator Matrix · p. 23, Table 6 · p. 24, Table 7 · p. 24

G.1 Information Accompanying Broader Disclosures

• 5 of 37 entries credited for data size also met language or domain composition criteria, and 2 of 24 credited for hardware or provider also met compute usage.These conditional counts describe overlap, not causation.Source: §G.1 · p. 27
• 24 of 28 entries credited for processing purpose also met methods, and 20 of those 24 met techniques; these are descriptive associations, not independent effects of openness.Source: §G.1 · p. 27

G.2 Criterion-Level Release Transitions

• The 24 comparisons comprise 20 open-weight and four open-source/reproducible comparisons; each comparison classifies all four transition counts, with changed criteria identified by original FMTI IDs.Source: §G.2 · p. 27, Table 11 · p. 28
• Llama 2 to Llama 3.1 gained three processing criteria and hardware while losing usage-data disclosure and language composition; OLMo 2 to OLMo 3 gained crawling, domain composition, and hardware but lost compute usage, provider, and dependencies.Source: §G.2 · p. 27
• Some unchanged totals reflect no changed criteria, while OLMo’s unchanged total masks replacement; losses mean unmet later-entry requirements, not document withdrawal or direct model ancestry.Source: §G.2 · p. 27

H Literature Search

• The exploratory literature search identified 51 studies to support methodological scoping and related work, rather than to provide a systematic review.Source: H Literature Search · p. 29