Source-linked AI summary
Data Science and Technology Towards AGI Part I: Tiered Data Management
Yudong Wang, Zixuan Fu, Hengyu Zhao, Chen Zhao, Chuyue Zhou, Xinle Lin, Hongya Lyu, Shuaikang Xue, Yi Yi, Yingjiao Wang, Zhi Zheng, Yuzhou Zhang, Jie Zhou, Chaojun Xiao, Xu Han, Zhiyuan Liu, Maosong Sun
TL;DR
Current LLM development relies heavily on scaling data despite bottlenecks in availability, cost, and training efficiency. The paper proposes an L0-L4, model-driven tiered data management framework for the full training lifecycle and reports consistent performance gains as data quality rises from L1 to L3.
Problem
Current data-scale-driven LLM research faces limited data availability, acquisition costs, training-efficiency constraints, and fragmented management practices.
Method
The paper introduces a model-driven L0-L4 framework that progressively refines data and assigns tiers to different LLM training stages.
Results
Across English, Chinese, mathematics, and code, performance improves steadily from L1 to L3, with the hierarchy L3 > L2 > L1 holding universally.
Takeaways & Limitations
Tier-aware data utilization provides a systematic way to allocate data according to quality, cost, and training-stage requirements.
Takeaways & Limitations
Future work is needed to develop rigorous data-value assessment, dynamic data-model feedback, broader modality coverage, and tighter integration with large-scale training systems.
Abstract
from arXiv · showhide
The development of artificial intelligence can be viewed as an evolution of data-driven learning paradigms, with successive shifts in data organization and utilization continuously driving advances in model capability. Current LLM research is dominated by a paradigm that relies heavily on unidirectional scaling of data size, increasingly encountering bottlenecks in data availability, acquisition cost, and training efficiency. In this work, we argue that the development of AGI is entering a new phase of data-model co-evolution, in which models actively guide data management while high-quality data, in turn, amplifies model capabilities. To implement this vision, we propose a tiered data management framework, designed to support the full LLM training lifecycle across heterogeneous learning objectives and cost constraints. Specifically, we introduce an L0-L4 tiered data management framework, ranging from raw uncurated resources to organized and verifiable knowledge. Importantly, LLMs are fully used in data management processes, such as quality scoring and content editing, to refine data across tiers. Each tier is characterized by distinct data properties, management strategies, and training roles, enabling data to be strategically allocated across LLM training stages, including pre-training, mid-training, and alignment. The framework balances data quality, acquisition cost, and marginal training benefit, providing a systematic approach to scalable and sustainable data management. We validate the effectiveness of the proposed framework through empirical studies, in which tiered datasets are constructed from raw corpora and used across multiple training phases. Experimental results demonstrate that tier-aware data utilization significantly improves training efficiency and model performance. To facilitate further research, we release our tiered datasets and processing tools to the community.
1 Introduction
AI development is framed as a progression in how data is organized and used, with current data-scale-driven learning approaching sustainability bottlenecks. The paper proposes data-model co-evolution and tiered management to allocate data across the LLM training lifecycle.
- AI development is presented as successive shifts in data-driven strategies and data utilization paradigms that improve model capability.
- Current mainstream research primarily expands data scale to enhance models, but this approach faces sustainability bottlenecks.
- Data-model co-evolution uses models to improve data management while high-quality data further refines model performance.
- Tiered data management supports the full LLM training lifecycle by matching heterogeneous data requirements with strategically deployed resources.
- The proposed L0-L4 framework evolves from raw resources toward structured data through filtering, selection, and editing processes.
- Empirical studies across English web, Chinese web, mathematics, and code data report improving performance as data quality rises from L1 to L3.
2 Tiered Data Management Framework
The paper motivates tiered data management by noting that mixed-quality training data can underuse valuable samples. It presents released datasets and tools as resources for implementing the framework.
- Indiscriminately mixing data of varying quality can hinder the use of high-value samples and produce suboptimal model performance.
- Concentrating high-quality data during annealing or mid-training has been reported to significantly enhance model performance.
- Table 1 lists the open-source datasets and tools released in the paper.
2.1 Existing Data Management Frameworks
Existing data management approaches are organized around training stages or processing methods, but remain fragmented and insufficiently hierarchical. The paper identifies missing unified standards, quality metrics, feedback, and lineage support as central shortcomings.
- Stage-oriented management: Stage-oriented frameworks organize management around pre-training, mid-training, and post-training objectives with different requirements for scale, diversity, and signal-to-noise ratio.
- Stage-oriented management: Pre-training management balances data scale and information density while preserving domain diversity for broad knowledge and linguistic representation.
- Stage-oriented management: Mid-training management prioritizes specialized knowledge and structured reasoning for vertical tasks, including mathematics and code.
- Stage-oriented management: Post-training management curates instruction-fine-tuning and reinforcement-learning data to improve instruction responsiveness and behavioral adherence.
- Method-oriented management: Method-oriented frameworks form a hierarchy from parsing and filtering through selection, editing, and synthesis as processing complexity increases.
- Identified limitations: Existing practices lack systematic hierarchical management, unified quality metrics, closed-loop feedback, and adequate data-lineage tracking.
2.2 Tiered Data Management Framework
The L0–L4 framework organizes data by progressively increasing purity, trustworthiness, and computational cost, assigning each tier a distinct management and training role. It spans raw archival resources, cleaned and selected corpora, refined data, and organized knowledge assets.
- L0–L4 defines five hierarchical levels that progressively increase data purity and trustworthiness while raising acquisition and computational costs.The framework is centered on fine-grained quality standards and uses specialized operators to move from raw data toward high-density knowledge assets.
- 2.2.1 L0: Raw Data: L0 preserves heterogeneous raw resources as an archival reserve, but its low information density and high noise make it unsuitable for direct model training.Sources include web content, scholarly literature, code, audio, video, and imagery collected through crawling, downloading, and format parsing.
- 2.2.2 L1: Filtered Data: L1 applies low-cost cleaning operations to produce scalable foundational corpora for large-scale pre-training.Its pipeline includes URL filtering, text extraction, language identification, heuristic filtering, and global deduplication; FineWeb processes data to approximately 36 trillion tokens.
- 2.2.3 L2: Selected Data: L2 uses model-driven selection, quality scoring, labeling, and domain-specific classifiers to increase information density and retain high-value samples.Ultra-FineWeb significantly outperforms the L1-level FineWeb dataset, supporting model-driven selection over rule-based filtering.
- 2.2.4 L3: Refined Data: L3 refines L1 and L2 data through editing, restoration, and synthesis, supporting mid-training, SFT, and RL while enhancing reasoning, mathematics, and instruction following.These operations address semantic flaws and logical incompleteness while adding information density, diversity, and creative value beyond raw data distributions.
- 2.2.5 L4: Organized Data: L4 transforms fragmented information into structured, verified, searchable knowledge assets that serve as authoritative sources for knowledge-intensive applications such as RAG.Its structural rigor and credibility enable efficient retrieval, supporting factual precision and defense against model hallucinations.
3 Experiments
The experiments evaluate tiered data management across four domains and multiple training stages, finding progressively stronger data quality and more effective late-stage learning with higher tiers.
- Experimental Setting: The framework is evaluated across English web, Chinese web, mathematics, and code using tiered datasets and multiple verification strategies.The experimental setting includes pre-training, efficient, and decay verification, with OpenCompass benchmarks.
- Data Construction: Tiered datasets are constructed by assigning L1, L2, and L3 resources across domains, including synthetic content at L3 for English and Chinese web data.The construction uses FineWeb, Ultra-FineWeb, and higher-tier synthetic datasets, with domain-specific selections.
- Data Quality Analysis: Across all four domains, performance follows the strict hierarchy L3 > L2 > L1, with gains from L1 to L3 of 1.70pp in English, 2.04pp in Chinese, 7.06pp in Math, and 1.79pp in Code.Average scores increase from 52.26pp to 53.96pp in English, 49.44pp to 51.48pp in Chinese, 23.78pp to 30.84pp in Math, and 34.49pp to 36.28pp in Code.
- Mathematics Case Study: Math-L3 improves average performance beyond mathematics, reaching 54.77pp on English, 53.48pp on Chinese, and 41.10pp on code benchmarks.The passage reports gains over Math-L1 and Math-L2 across non-mathematical domains and improvements on reasoning-intensive and code-generation tasks.
- Multi-Stage Training: Tiered training improves overall performance by 1.49pp over mix training, achieving 31.66pp versus 30.17pp across four evaluation domains.Improvements are reported in English, Chinese, mathematics, and code, with stronger gains on several reasoning- and knowledge-intensive tasks.
- Multi-Stage Training: During later training, tiered training rises by 3.31pp from 28.35pp to 31.66pp, while mix training rises by approximately 1.91pp from 28.26pp to 30.17pp.The later-stage advantage accompanies the introduction of L2 and L3 data after both strategies show similar early growth.
4 Conclusion
The paper frames sustainable LLM progress as data-model co-evolution rather than continued reliance on data-scale expansion. It introduces tiered data management as a scalable foundation and identifies scientific data valuation, dynamic feedback, and broader deployment as future priorities.
- The L0–L4 framework structures data from raw resources to organized and verifiable knowledge while aligning quality, cost, and training objectives.
- Tier-aware data utilization is reported to improve training efficiency and model performance in empirical studies on math and web data.
- Future work targets rigorous data-value assessment, dynamic data–model feedback, broader modalities and domains, and tighter integration with large-scale training systems.