Source-linked AI summary
Tracing the Roots: A Multi-Agent Framework for Uncovering Data Lineage in Post-Training LLMs
Yu Li, Xiaoran Shang, Qizhi Pei, Yun Zhu, Xin Gao, Honglin Lin, Zhanping Zhong, Zhuoshi Pan, Zheng Liu, Xiaoyang Wang, Conghui He, Dahua Lin, Feng Zhao, Lijun Wu
TL;DR
Post-training datasets are often treated as isolated artifacts even though they form interdependent lineages, making provenance, redundancy, and contamination difficult to assess. The paper introduces a multi-agent framework that reconstructs dataset dependency graphs from documentation and uses them for lineage-aware curation. Large-scale analysis reveals domain-specific evolution and hidden structural risks, while root-anchored sampling produces a more diversity-oriented dataset.
Problem
Post-training datasets form recursively derived, interdependent networks, but systematic mechanisms for tracking their provenance and detecting inherited redundancy or benchmark contamination remain limited.
Method
The paper defines data lineage as a directed dataset-dependency graph and reconstructs it with a multi-agent framework that mines documentation and performs provenance tracing.
Results
17 of 83 examined datasets exceed 1% redundancy, while 19 datasets exhibit benchmark leakage propagation; the analysis also characterizes distinct domain-specific evolutionary structures.
Takeaways & Limitations
Lineage analysis supports root-anchored provenance-based sampling to counter derivative reuse and improve instruction diversity in post-training data construction.
Takeaways & Limitations
The framework depends on documentary transparency and requires human verification for low-confidence extractions because LLM-based tracing can hallucinate.
Abstract
from arXiv · showhide
Post-training data plays a pivotal role in shaping the capabilities of Large Language Models (LLMs), yet datasets are often treated as isolated artifacts, overlooking the systemic connections that underlie their evolution. To disentangle these complex relationships, we introduce the concept of \textbf{data lineage} to the LLM ecosystem and propose an automated multi-agent framework to reconstruct the evolutionary graph of dataset development. Through large-scale lineage analysis, we characterize domain-specific structural patterns, such as vertical refinement in math-oriented datasets and horizontal aggregation in general-domain corpora. Moreover, we uncover pervasive systemic issues, including \textit{structural redundancy} induced by implicit dataset intersections and the \textit{propagation of benchmark contamination} along lineage paths. To demonstrate the practical value of lineage analysis for data construction, we leverage the reconstructed lineage graph to create a \textit{lineage-aware diversity-oriented dataset}. By anchoring instruction sampling at upstream root sources, this approach mitigates downstream homogenization and hidden redundancy, yielding a more diverse post-training corpus. We further highlight lineage-centric analysis as an efficient and robust topological alternative to sample-level dataset comparison for large-scale data ecosystems. By grounding data construction in explicit lineage structures, our work advances post-training data curation toward a more systematic and controllable paradigm.
1 Introduction
Post-training datasets form an undocumented evolutionary network rather than isolated artifacts, creating hidden redundancy and contamination risks. The paper introduces lineage analysis to reconstruct these dependencies, diagnose structural problems, and guide more diverse curation.
- Motivation: Post-training datasets are recursively derived through semantic evolution, knowledge distillation, and structured fusion, forming a dense evolutionary network.These developmental connections are largely undocumented, limiting systematic provenance tracking.
- Lineage analysis: The framework reconstructs an evolutionary graph from 83 seed datasets, yielding 430 unique nodes and 971 inheritance edges across four domains.Its analysis finds horizontal expansion in general domains and deeper vertical refinement in math-oriented data.
- Structural risks: 17 of 83 examined datasets exceed 1% redundancy, with open-instruct-v1 reaching 46.48% because it includes its own superset.The analysis uses lineage structure to expose implicit dataset intersections.
- Structural risks: 19 datasets show benchmark leakage propagation, including Caco-1.3M inheriting 37.95% of Omni-Math samples from contaminated upstream sources.The downstream dataset need not explicitly include the benchmark for contamination to propagate along inheritance paths.
- Lineage-aware curation: Provenance-based sampling anchors selection at upstream roots to counter derivative reuse and produce a lineage-aware diversity-oriented dataset.The approach shifts curation toward lineage structure and topological reasoning rather than only sample-level comparison.
2 Related Work
Post-training data construction has shifted toward multidimensional synthesis and extensive reuse, while existing analysis tools mainly address documentation, quality, mixing, profiling, or compliance. Dataset-level evolutionary dependencies therefore remain insufficiently characterized by prior approaches.
- Data construction: Modern post-training data combines semantic evolution, knowledge distillation, structured fusion, and multimodal augmentation rather than being built entirely from scratch.This repurposing creates deeply nested dependencies among datasets.
- Existing analysis: Prior tools cover documentation, quality filtering, cross-domain mixing, corpus profiling, and licensing audits, but their focus is fragmented.These approaches address dataset properties or compliance rather than reconstructing the full evolutionary network.
3 Data Lineage
The paper formalizes data lineage as a directed dependency graph and uses a multi-agent pipeline to recover it from heterogeneous documentation. Agents validate datasets, retrieve sources, infer actual provenance, canonicalize identities, and recursively expand the graph with consistency checks.
- Formalization: Data lineage is represented as a directed graph G = (V, E), where nodes are datasets and edges encode upstream inheritance.Internal nodes have identifiable upstream sources, while leaf nodes terminate automated exploration.
- Challenges: Lineage tracing is difficult because provenance is scattered across papers, repository READMEs, and technical blogs in nonstandard formats.Deeply nested dependencies can also cause combinatorial growth during recursive expansion.
- Framework: The framework coordinates agents that fuse evidence from multiple sources and apply semantic reasoning to extract structured lineage.The pipeline is designed for noisy and incomplete documentation.
- Pipeline: A centralized queue drives target-to-source recursion through candidate validation, information retrieval, semantic source inference, and aggregation with recursive expansion.The stages verify dataset availability, gather external documentation, distinguish actual sources from incidental mentions, and consolidate extraction records.
- Verification: Canonical identifiers, API verification, temporal validity checks, and confidence filtering support graph construction and recursive traversal.Depth-First Search terminates at foundational datasets or sources released before 2020.
4 Landscape Analysis
The study maps the post-training dataset ecosystem as a lineage graph and finds distinct domain-specific evolutionary patterns, alongside structural redundancy and contamination risks. Lineage analysis also supports targeted curation recommendations for improving diversity and preventing leakage.
- Scope and Graph Construction: The study analyzes 83 high-impact textual datasets and reconstructs a lineage graph of 430 datasets linked by 971 inheritance edges.The analysis combines topological, cross-domain, and temporal perspectives.
- Evolutionary Patterns: General-domain datasets favor broad accumulation, with shallow structures averaging 1.05 depth and a 68.42% leaf ratio.FineWeb exemplifies large-scale aggregation, while FLAN has limited downstream influence with dout = 7.
- Evolutionary Patterns: Mathematics follows deep recursive refinement, averaging 2.92 depth and 1.54 downstream connections, with hendrycks math and gsm8k serving as central roots.NuminaMath developed from these foundational datasets and became a later cornerstone.
- Temporal Evolution: Temporal trends show general-domain leaf-node additions falling from 41 in 2024 to 5 in 2025, while mathematics intermediate nodes rose from 18 to 33.Science intermediate outputs also doubled from 6 to 12, indicating increased attention to specialized reasoning.
- Structural Risks: Lineage tracing exposes hidden source intersections, including 5.30% redundancy in Fast-Math-R1-SFT and 46.48% in open-instruct-v1.The analysis uses exact instruction-input-output triplet matches and identifies nested source patterns across collections.
- Structural Risks: Lineage analysis traces benchmark contamination across five benchmarks and 19 datasets, including 37.95% inherited Omni-Math contamination in Caco-1.3M.The study recommends screening upstream composition and excluding compromised evaluation sources before integration.
5 Analysis and Discussion
The lineage graph supports provenance-aware dataset construction and topological analysis, revealing how upstream roots, redundancy, contamination, and evolutionary relationships shape post-training data.
- Lineage-Centric Analysis: The analysis identifies structural redundancy and benchmark contamination as ecosystem-level issues that lineage can diagnose and help address through data curation.The paper’s broader motivation links implicit source overlap to reduced effective diversity and inherited benchmark leakage to compromised evaluation credibility.
- Lineage-Guided Data Construction: 570K unique instructions are produced by sampling root datasets prioritized by domain metadata and out-degree, followed by MinHash duplicate removal.The strategy treats root nodes as upstream knowledge anchors rather than sampling the entire corpus.
- Lineage-Guided Data Construction: A Vendi Score of 452.44 and Centroid Distance of 0.6385 exceed those of evaluated datasets ranging from 300K to 1.2M samples.The lineage-aware dataset also outperforms OpenHermes-2.5, MegaScience, and OpenThoughts3 despite its smaller size.
- Lineage-Guided Data Construction: Root-only sampling excludes internal evolutionary variants and isolates provenance disentanglement as the basis for the reported diversity results.The authors frame this design as preserving original provenance without exhaustive filtering over the derivative space.
- Lineage-Centric Analysis: The graph shifts dataset comparison from sample-level similarity to ancestry and inheritance paths, enabling faster overlap estimation without traversing millions of samples.This topological unit of comparison is intended to remain useful when semantic drift weakens surface similarity.
- Lineage-Centric Analysis: Lineage tracing groups related datasets into families and exposes recurring construction patterns such as textbook sourcing, synthetic Q&A generation, and chain-of-thought refinement.These recurring parent-to-child transformations provide guidance on source and refinement combinations in high-impact lineages.
6 Conclusion
The paper presents a multi-agent framework for reconstructing post-training dataset lineage, then uses the resulting graph to analyze contamination, redundancy, and diversity-oriented data construction.
- Conclusion: The framework reconstructs a large-scale lineage graph for post-training datasets and traces structural redundancy and benchmark contamination across the ecosystem.The conclusion also describes lineage-graph applications to diversity-focused data construction and future research directions.
Limitations
The framework’s reliability and coverage are constrained by model hallucinations and the transparency of dataset documentation.
- Limitations: LLM-based extraction carries hallucination risks, so low-confidence results require human verification to ensure graph reliability.This limitation concerns the reliability of automatically extracted provenance relationships.
- Limitations: Lineage reconstruction cannot recover dependencies that dataset creators fail to report or intentionally conceal in technical documentation.The method is therefore bounded by documentary transparency.
A Implementation Details of Automatic Provenance Framework
The provenance framework distributes sourcing, extraction, tracing, and aggregation tasks across specialized agents, using selected models and 83 seed datasets to initialize analysis.
- Pipeline Architecture: The pipeline assigns specialized agents to sourcing, extraction, tracing, and aggregation to balance accuracy, efficiency, and cost.Different models are selected according to task-specific performance characteristics.
- Sourcing Agent: The Sourcing Agent identifies data-information entry points in repository README files using GPT-5.1 to reduce hallucination risk.Its role is to retrieve reliable initial metadata and discover relevant external resources.
- Extracting Agent: The Extracting Agent uses Gemini-2.5-Flash to process large volumes of repository, blog, and related textual content efficiently.The agent summarizes identified sources into a specified format for downstream lineage analysis.
- Tracing Agent: The Tracing Agent uses GPT-5.1 to extract specific source-target relationships while emphasizing precision in lineage identification.Its task is to distinguish actual provenance relationships from incidental documentation mentions.
- Aggregation Agent: The Aggregation Agent uses Gemini-2.5-Pro to standardize dataset names, merge sources, and resolve ambiguous identifiers through web retrieval.It consolidates dispersed information into unified dataset nodes.
- Seed Dataset Selection: The analysis begins with 83 seed datasets selected by download volume, community engagement, and downstream-model performance.All selected datasets are described as verifiable and retrievable on Hugging Face.
A.3 Relation Types in Data Lineage
The lineage graph classifies dataset derivations into five relation types and stores these relations as edge attributes. This enables analysis of reuse, aggregation, and evolutionary depth across the dataset ecosystem.
- Relation types: Five relation types capture semantic evolution, CoT distillation, synthetic generation, structured fusion, and direct inclusion or subset relationships.These categories distinguish whether data is reformulated, teacher-generated, synthesized, combined across sources, or included unchanged.
- Relation inference: The tracing agent infers relation types from dataset documentation and records them as edge attributes in the lineage graph.When descriptions are vague, the agent uses contextual semantics or defaults to Direct Inclusion.
- Graph utility: Relation attributes let the graph answer how one dataset derives from another, supporting deduplication, contamination tracking, and quality assessment.The relation method provides fine-grained signals for downstream analysis.
- Structural analysis: Ecosystem analysis measures reuse rate by out-degree, information aggregation by in-degree, and evolutionary depth.These dimensions characterize dataset roles and evolutionary patterns; corresponding statistics appear in Tables 6–8.
C Source Intersection Details
The intersection analysis uses strict triplet hashing to detect exact duplicates and reveals hidden upstream overlap along lineage paths. These findings motivate verification before assembling new training pools.
- Intersection detection: Exact matching of complete instruction, input, and output triplets detects significant redundancy across multiple datasets.The protocol computes hash values for complete triplets rather than relying on looser semantic similarity.
- Intersection paths: Identified upstream intersection paths show that some datasets inadvertently incorporate large portions of earlier source datasets.Table 9 records the specific paths from targets to upstream sources.
- Curation implication: Curators should verify upstream intersections to prevent selecting identical data sources redundantly when constructing training pools.The recommendation follows directly from the observed lineage overlaps.
D Data Contamination Details
The paper traces benchmark contamination through downstream lineage using exact matching and documents risks from temporal benchmark updates. It also describes root-anchored curation and complementary metrics for constructing and evaluating a less redundant dataset.
- Contamination analysis: Exact matching of benchmark instruction-input pairs against downstream training data measures contamination across five benchmarks.The analysis expands beyond the main text to include LiveCodeBench and TruthfulQA.
- Temporal benchmarks: LiveCodeBench v53 test samples appeared in AM-Thinking-v1-Distilled and DeepCoder-Preview-Dataset, exposing risks from neglected version control and temporal cutoffs.The issue arises because LiveCodeBench updates chronologically.
- Contamination pathways: Contamination statistics report leaked-sample counts and percentages, while lineage tables trace the pathways through which benchmark data reaches downstream training datasets.The findings empirically confirm cascading contamination from upstream sources into derivatives.
- Root-anchored sampling: The sampling workflow uses leaf nodes with d_in = 0 as upstream knowledge anchors to reduce structural redundancy at its source.The graph contains 212 leaf datasets, which are ranked by downstream usage after excluding nodes with zero downstream usage.
- Curation pipeline: Domain filtering retains 31 post-training datasets, format unification produces approximately 8.7M samples, and two-stage deduplication yields 570K instruction samples.The pipeline removes niche or non-QA data, filters length and language, then applies exact Q matching followed by MinHash deduplication.
- Diversity evaluation: Vendi Score measures effective semantic cluster count, while Centroid Distance measures embedding-space dispersion around the global centroid.The Vendi calculation uses eigenvalues of a cosine-similarity kernel; higher values indicate more independent modes or broader semantic spread, respectively.