PaperlayerView exact revision on arXiv

Source-linked AI summary

Which Models Are Our Models Built On? Auditing Invisible Dependencies in Modern LLMs

Sanjay Adhikesaven, Haoxiang Sun, Sewon Min

arXiv:2606.12385v1cs.CL

TL;DR

Modern LLM development depends on recursively linked models and artifacts whose fragmented documentation makes dependency tracing difficult. ModSleuth formalizes and reconstructs these dependencies from public evidence, recovering over a thousand verified relationships across four releases and surfacing issues such as license-relevant multi-hop paths and documentation inconsistencies.

  • Problem

    Recursive LLM dependencies are fragmented across heterogeneous, inconsistently documented artifacts, making their full structure difficult to trace manually.

  • Method

    ModSleuth recursively reconstructs evidence-grounded dependency graphs using direct and indirect dependency semantics, operation-centered relationships, and artifact-identity resolution.

  • Results

    Over a thousand verified dependency relationships across four public-artifact-rich LLM releases form graphs that surface license-relevant paths and documentation inconsistencies.

  • Takeaways & Limitations

    The resulting graphs provide a foundation for auditing, understanding, and governing recursively dependent modern AI ecosystems.

  • Takeaways & Limitations

    Because ModSleuth uses only public artifacts, its graphs are evidence-grounded lower bounds and cannot recover undocumented, proprietary, or inaccessible dependencies.

Abstract

from arXiv · show

Modern LLM training pipelines increasingly rely on other models to generate data, filter corpora, judge outputs, and guide development decisions. These dependencies are recursive: a model may depend on an upstream artifact whose own dependencies are documented only in separate releases and artifacts. As a result, the full dependency structure is fragmented across heterogeneous public artifacts, with complexity and recursive depth far outpacing humans' ability to trace. We introduce ModSleuth, an agentic system that recursively reconstructs LLM dependency graphs from public artifacts with source-grounded evidence. We find that the primary challenge is no longer information extraction, but defining what constitutes a dependency and reconciling artifact references across inconsistent documentation. We address these challenges through a formalization that distinguishes direct and indirect dependencies, represents heterogeneous pipeline roles through operation-centered relationships, and resolves artifact identities across names, versions, and repositories. Applying ModSleuth to four public-artifact-rich LLM releases, we recover 1,060 source-verified dependencies and construct large-scale dependency graphs of modern LLM development. These graphs reveal multi-hop license obligations, train-evaluation coupling, discrepancies between released and training-time artifacts, and documentation inconsistencies that would otherwise be difficult to uncover. We release ModSleuth and the resulting dependency graphs to support transparent analysis of the increasingly complex ecosystems underlying modern LLMs.

1 Introduction

Modern LLM development depends recursively on other models across diverse stages, but existing documentation is too incomplete and flat to represent these structures. ModSleuth reconstructs source-grounded dependency graphs by formalizing dependency types and reconciling inconsistent artifact identities, revealing deep interconnectedness while identifying only declared dependencies.

  • Motivation: Modern LLMs rely on other models for data generation, rewriting, filtering, evaluation, preference learning, and other development stages, creating recursive dependencies across heterogeneous public artifacts.Upstream dependencies may be documented across technical reports, model cards, repositories, and datasets.
  • Motivation: Opaque dependencies can silently propagate license restrictions and contamination, while shared ancestry between judges and evaluated systems can create circular evaluations.These risks span multi-hop paths that standard decontamination may not trace.
  • Approach: ModSleuth recursively reconstructs LLM dependency graphs from public artifacts, focusing on dependency definition and artifact-identity resolution rather than information extraction alone.The system addresses inconsistent names, versions, model families, development stages, and repositories.
  • Approach: The framework distinguishes direct dependencies affecting model weights from indirect development influences, represents operation-centered relationships, and reconciles references with an identity lattice.Operations include generation, filtering, rewriting, OCR, and evaluation.
  • Findings: Dependency chains extend up to eight hops and include synthetic-data generation, curation, rewriting, annotation, OCR, evaluation, and auxiliary components’ training data.The resulting graphs portray the LLM ecosystem as highly interconnected and compositional beyond model initialization or dataset reuse.
  • Limitations: ModSleuth identifies declared rather than true dependencies, so detected graphs are likely lower bounds and cross-model comparisons may reflect disclosure differences.Limited public disclosure can make a model appear to have fewer or no dependencies.

2 Background & Related Work

Foundation-model development has evolved from relatively simple training pipelines into complex, recursive ecosystems where models support data processing, generation, filtering, and evaluation. ModSleuth addresses the resulting auditing gap by recursively tracing declared dependencies across public artifacts, while noting that undocumented dependencies remain outside its scope.

  • Background: Foundation Model Training: Foundation-model artifacts span closed, partially open, and fully open-source releases, reflecting a rapidly consolidated ecosystem.Examples include OpenAI, Gemini, and Anthropic; Llama, DeepSeek, and Qwen; and Olmo, Nemotron, and Marin.
  • Background: Foundation Model Training: Modern LLM pipelines increasingly use upstream models for OCR, synthetic data and reasoning-trace generation, rewriting, filtering, and other stages.This contrasts with earlier pipelines centered on web-corpus pretraining followed by post-training with curated human annotations.
  • Background: Foundation Model Training: ModSleuth formalizes recursive LLM dependency tracing and extracts model dependencies from publicly available sources, exposing deeply recursive structures such as hundreds of upstream artifacts for Olmo 3.The system also surfaces possible license and terms implications associated with these dependencies.
  • Related Work: Auditing ML Artifacts: Prior auditing methods emphasize weight initialization, finetuning, and dataset reuse, leaving operational dependencies such as filtering, OCR, rewriting, judging, and synthetic-data generation difficult to recover without disclosure.ModSleuth instead reconstructs declared dependencies from public artifacts and supports recursive tracing of heterogeneous relationships.
  • Related Work: Auditing ML Artifacts: The analysis uses only reported information from official sources, including technical reports, Hugging Face pages, and code releases, and therefore likely significantly underestimates undocumented model-model dependencies.Inferring unreported dependencies is identified as an important direction for future work.

3 Design of ModSleuth

ModSleuth reconstructs evidence-grounded dependency graphs from public release artifacts by formalizing dependency semantics, representing relationships as operations, and resolving ambiguous artifact identities. Its staged pipeline separates discovery from normalization and recursively validates claims against official source evidence.

  • Graph objective: ModSleuth models artifacts as nodes and evidence-backed relationships as edges describing how upstream models or datasets shape a target release.The graph is reconstructed using only public release evidence.
  • Dependency semantics: Direct dependencies affect training or weights, while indirect dependencies substantially influence development decisions without directly entering training.Direct dependencies include initialization models, synthetic-data generators, OCR systems, and filters; indirect dependencies include evaluators, ablations, and adopted recipes.
  • Relationship representation: Dependencies are represented as operations containing natural-language descriptions, coarse type labels, and supporting source excerpts, allowing one artifact to have distinct roles across pipeline stages.Operations group edges corresponding to a single pipeline event rather than relying only on a fixed dependency taxonomy.
  • Identity resolution: Artifact identity is organized as a lattice from underspecified families through partially resolved nodes to canonical releases anchored by URLs or identifiers.The lattice uses open-vocabulary facets such as family, size, and stage, so claims attach at the most specific justified level.
  • Recovery pipeline: The recovery pipeline gathers official public artifacts, preserves source-local mentions before cross-source normalization, and admits only source-grounded, validated dependency claims.Because undocumented dependencies are outside the evidence scope, the recovered graph is a lower bound on the true dependency structure.

4 Evaluation

Because exhaustive ground-truth graphs are impractical, ModSleuth is evaluated by source-verified dependency relationships recovered across four public-artifact-rich LLM releases. It substantially outperforms single-prompt baselines, recovering 1,060 verified relationships unbounded and 1,654 under BFS reachability.

  • Evaluation challenge: Exhaustive human-annotated dependency graphs proved impractical because experts spent many hours tracing even one model without producing a reasonably exhaustive graph.The evaluation therefore cannot rely on complete ground-truth graphs.
  • Evaluation protocol: Only dependency relationships independently judged verified from cited evidence count toward the evaluation metric; refuted and unclear relationships are excluded.Claude Sonnet 4.6 with web search assists post-hoc verification by corroborating cited URLs and returning JSON verdicts.
  • Evaluation setup: The evaluation covers Olmo 3, Nemotron 3 Super, DR Tulu, and SmolLM3, spanning fully open, industrial open-recipe, post-training, and compact fully open releases.The systems are compared against GPT-5.5 Pro, GPT-5.4 Pro, CC-single, and ChatGPT Deep Research.
  • Results: 171–314 verified dependencies were recovered by single-prompt baselines, versus 484 for ModSleuth at depth-1, 1,060 unbounded, and 1,654 with BFS reachability.Depth-1 counts target-subject relations; unbounded includes recursively discovered forward-reachable relations, while BFS additionally includes findings from separate investigations.

5 Findings

Across four releases, ModSleuth recovers 2,526 artifact nodes, 9,112 dependency edges, and 36,187 evidence anchors, revealing dependencies that often enter through data operations rather than checkpoint inheritance. Recursive tracing exposes hidden multi-hop models, training–evaluation coupling, concentrated model-mediated selection, and provenance and licensing issues.

  • Graph structure: 2,526 artifact nodes, 9,112 dependency edges, and 36,187 evidence anchors are recovered across the analyzed releases.The graph contains 1,443 dataset nodes and 1,083 model nodes.
  • Graph structure: 72.0% of edges are direct dependencies, while 28.0% are indirect dependencies shaping development through evaluation, ablation, or methodology borrowing.Among direct dependencies, data operations account for 350 verified edges (21.2%).
  • Multi-hop upstream models: Recursive tracing reveals upstream models hidden behind intermediate datasets, filters, classifiers, teachers, and tools.Examples include Claude Sonnet 3.7 behind DR Tulu’s ScholarQA trajectories and Qwen2.5-Coder-32B-Instruct behind Olmo 3 RL-Zero code-data transformation.
  • Training–evaluation coupling: Training and evaluation are structurally coupled when benchmark prompts, splits, auxiliary resources, or validation environments become training artifacts while the benchmark family remains an evaluation target.Olmo examples include IFEval-derived prompts and GSM8K train splits with synthetic expansion, while both benchmark families remain evaluation targets.
  • Model-mediated selection: 552 edges connect Qwen to 167 downstream artifacts, compared with 264 edges for Llama across 157 artifacts, 125 for GPT-4 across 65, and 162 for DeepSeek across 81.These are one-hop training-side dependencies, showing concentration among a small set of major model families.
  • Model-mediated selection: 70 uses for Qwen2.5-32B-Instruct and 43 for DeepSeek-R1 lead generator, filter, transformer, and judge edges, concentrating synthetic-data construction around a few upstream models.Other high-use models include Llama-3.3-70B-Instruct (36), Qwen3-32B (33), and GPT-4.1 (32).

6 Conclusion

The paper formalizes recursive LLM dependency tracing as constructing evidence-grounded graphs over model and dataset artifacts and presents ModSleuth to recover them from public artifacts. It shows that auditing these dependencies requires explicit semantics for dependency scope, pipeline roles, and artifact identity reconciliation.

  • System and formalization: ModSleuth recovers evidence-grounded dependency graphs over model and dataset artifacts from public artifacts.The system is presented as an agentic approach to recursive LLM dependency tracing.
  • Audit requirements: Auditing modern LLM dependencies requires explicit graph semantics defining what counts as a dependency.The paper identifies dependency scope as a central requirement for constructing the graphs.
  • Audit requirements: The graph representation must encode heterogeneous pipeline roles and reconcile artifact identities across inconsistent sources.These requirements address how different pipeline functions and artifact references are represented and matched.

Limitations

ModSleuth’s dependency graphs are evidence-grounded lower bounds because undocumented or inaccessible dependencies cannot be recovered. Evaluation is limited by absent ground truth, reliance on Claude Code for extraction and verification, and comparisons restricted to fully automated baselines.

  • Artifact coverage: Public-artifact reconstruction cannot recover undocumented, proprietary, or inaccessible dependencies, so the graphs are lower bounds on the true structure.The gap is likely largest for closed or partially disclosed releases.
  • Evaluation coverage: Because complete ground-truth dependency graphs do not exist, evaluation covers only four well-documented LLM releases with extensive supporting artifacts.Coverage across less thoroughly documented ecosystems remains for future work.
  • Verification dependence: ModSleuth and automated verification both rely on Claude Code, so shared modeling biases may affect extraction and verification despite evidence grounding and deterministic validation.Independent pipelines based on different model families would provide a stronger assessment of extraction quality.
  • Baseline scope: Comparisons use fully automated baselines and omit expert-guided workflows, which could reduce the performance gap and separate task decomposition from agent capabilities.Human-in-the-loop evaluation could clarify how much performance comes from ModSleuth’s decomposition versus the underlying agent.

A Evaluation Protocol Details · B Baseline Prompt

The evaluation pools candidate relationships, verifies them against public evidence, and attributes verified or refuted relationships to target models under multiple graph-based scopes. Baselines receive a shared prompt template to recover direct and indirect dependencies in a single JSON graph.

  • A Evaluation Protocol Details: Evaluation proceeds by pooling candidate relationships, verifying each against public evidence, and attributing verified or refuted relationships to target models.Candidates include subject and object artifacts, dependency descriptions, relation metadata, and supporting evidence URLs or excerpts.
  • A Evaluation Protocol Details: Each candidate is checked by Claude Sonnet 4.6 with web search, while BFS and full-graph aggregates receive separate Claude Opus 4.7 audits.Audits may verify submitted excerpts or locations and search for independent public corroboration.
  • A Evaluation Protocol Details: Depth-1 attributes a relationship to target T only when its subject canonicalizes to T’s canonical identifier, matching the strict per-target baseline scope.Canonicalization lowercases strings, collapses non-alphanumeric runs to hyphens, and preserves HuggingFace organization prefixes.
  • A Evaluation Protocol Details: Unbounded attribution requires both investigation provenance and forward reachability from T through subject →object edges in the merged graph.Provenance can come from the subject, target-specific seed anchors, or workers uniquely co-occurring with a target’s seed directory.
  • A Evaluation Protocol Details: BFS reachability attributes edges whose subjects are forward-reachable from each target and reports the union across four seeds, allowing multiple-target attribution.This scope captures each target’s full transitive dependency footprint.
  • A Evaluation Protocol Details: 1,060 of 9,112 merged-graph relations receive unbounded attribution to at least one target, while BFS reaches 1,654 unique edges.These figures use different attribution scopes: unbounded provenance plus reachability versus graph reachability alone.
  • B Baseline Prompt: All baselines use the same dependency-reconstruction prompt, with target-specific identity, provider, release, authoritative URLs, scope, and recursion-depth fields.The prompt instructs recovery of direct training-pipeline and indirect development dependencies and requests a single JSON graph containing subject and nodes.

C Additional Quantitative Results · C.1 Recovered Graph Scale by Target · D Additional Qualitative Findings

The appendix quantifies recovered dependency-graph scale and depth by target, showing that lineage depth reflects pipeline complexity. It also catalogs recurring audit patterns involving hidden model-mediated construction, benchmark exposure, artifact mismatches, and reproducibility.

  • C.1 Recovered Graph Scale by Target: Ancestor counts measure unique transitive upstream artifacts reachable from each target model.Table 6 reports these counts alongside maximum graph depths.
  • C.1 Recovered Graph Scale by Target: Maximum-depth variance reflects pipeline complexity rather than recovery quality.The comparison covers a pretraining-only release and instruction/reasoning families with substantially more training stages.
  • C.1 Recovered Graph Scale by Target: SmolLM3-Base has recoverable lineage dominated by web-scrape and filter steps, reaching depth 3.Its release is characterized as pretraining-only.
  • C.1 Recovered Graph Scale by Target: Olmo 3 Instruct/Think families reach depth 8 through midtraining, multiple SFT/DPO/RL stages, and judge-mediated synthetic-data construction.These stages create deeper recovered lineages than the pretraining-only example.
  • C.1 Recovered Graph Scale by Target: Single-source edges rely on anchors from one source class, whereas multi-source edges rely on anchors from multiple source classes.Table 7 presents the recovered-edge distribution using this distinction.
  • D Additional Qualitative Findings: Additional qualitative examples cover hidden model-mediated data construction.The appendix presents these as recurring audit patterns also represented in the main text’s curated findings.
  • D Additional Qualitative Findings: The qualitative findings also address benchmark exposure, release-artifact mismatches, and reproducibility issues.These examples extend the same recurring audit patterns discussed in the main text.

D.1 Hidden Model-Mediated Data Construction

Nemotron-CC-v2.1 incorporates Common Crawl data synthetically rephrased and translated with Qwen3-30B-A3B, while Nemotron-3-Super uses training content generated partly by Nemotron-Nano-9B-v2.

  • D.1 Hidden Model-Mediated Data Construction: Nemotron-CC-v2.1 extends Common Crawl-derived data through synthetic rephrasing and translation with Qwen3-30B-A3B.User-facing model cards describe the result as synthetic Common Crawl, whereas dataset documentation exposes the upstream rephrasing model and subset construction.
  • D.1 Hidden Model-Mediated Data Construction: Nemotron-3-Super uses training content generated in part by Nemotron-Nano-9B-v2.

D.2 Benchmark Exposure and Decontamination · D.3 License and Release Hygiene · D.4 Code and Configuration Reveal Details Hidden by Cards

The recovered artifacts expose benchmark-related development coupling, explicit decontamination rules, broad teacher-model licensing exposure, and release changes. Code, configuration, and data-preparation files also reveal mixture details and weights omitted from model cards.

  • D.2 Benchmark Exposure and Decontamination: LiveCodeBench serves both as an RL validation split and an evaluation benchmark, creating potential coupling to checkpoint selection or training decisions.A Nemotron configuration references livecodebench_v5_validation, while LiveCodeBench-style results are reported as evaluation metrics.
  • D.2 Benchmark Exposure and Decontamination: Released decontamination artifacts anchor AMC-related edges, with specified exclusions operating over question/answer fields using n-gram overlap.The artifacts include utils/decont_utils/data/{benchmark}.jsonl.
  • D.3 License and Release Hygiene: Tulu 3 DPO data draws on teachers from OpenAI, Anthropic, Mistral/Mixtral, 01-ai/Yi, MosaicML/MPT, and InternLM-family models.This cross-organization teacher panel is a point where multiple license and terms-of-service regimes may accumulate.
  • D.3 License and Release Hygiene: Olmo 3 releases complete and redacted Dolma 3 variants, while its reproduction data changed.The passage identifies these release differences without specifying the changed reproduction-data contents.
  • D.4 Code and Configuration Reveal Details Hidden by Cards: 1.36% of the rlvr1 mix comes from DAPO-Math-17k, while Skywork-OR1-RL-Data contributes 5.44%.These placeholder percentages are recovered by joining YAML mix-weight definitions with upstream dataset cards and fill_placeholders.py.
  • D.4 Code and Configuration Reveal Details Hidden by Cards: SmolLM3 YAMLs specify individual language shards and exact pretraining mix weights beyond the card’s broad web, code, math, and multilingual categories.The shards include FineWeb2-HQ slices and Stack-Edu buckets for Python, Java, and Rust.

D.5 Methodology and Ecosystem-Level Dependencies

SmolLM3 inherits methodology from upstream models in addition to data. Recovered inspired_by edges show that such method transfer can shape downstream development without involving weight inheritance or training-data reuse.

  • D.5 Methodology and Ecosystem-Level Dependencies: Two recovered inspired_by edges document methodology transfer into SmolLM3.One connects SmolLM3’s intra-document attention masking to Llama 3, while the other connects FineMath’s 13-gram decontamination to Qwen2.5-Math [74].
  • D.5 Methodology and Ecosystem-Level Dependencies: SmolLM3 uses intra-document attention masking similar to Llama 3.This is an example of method transfer rather than weight inheritance or training-data reuse.
  • D.5 Methodology and Ecosystem-Level Dependencies: FineMath follows Qwen2.5-Math [74]’s 13-gram decontamination procedure.The example illustrates how upstream methodology can shape downstream model development independently of data or weights.
Loading 2606.12385v1…