Data-Juicer: A One-Stop Data Processing System for Large Language Models

Motivation

• The system targets heterogeneous data recipes, expensive recipe evaluation, limited configurability, and massive data volumes that constrain existing LLM data-processing workflows.These challenges arise from diverse sources and mixtures, costly model evaluation, users’ need to configure pipelines, and corpora containing billions or trillions of tokens.Source: §1 · pp. 1–2

System design

• Data-Juicer provides a fine-grained, composable abstraction with over 50 operators and tools for generating traceable, comparable, and refinable data recipes.Its end-to-end configuration supports both pre-training and fine-tuning workflows.Source: §1 · p. 2
• Visualization and auto-evaluation create a timely feedback loop for iteratively understanding data and improving recipes across LLM development stages.This feedback loop is intended to support exploration of recipe possibilities and evaluation of their effects on model performance.Source: §1 · pp. 2–3

Usability and extensibility

• Data-Juicer serves beginners through zero-code workflows, experienced users through configurable recipes, and advanced users through extensible operators and tools.Interactive tutorials further support users with different backgrounds and needs.Source: §1 · p. 2

Scalability and integration

• The system uses a unified data representation, optimization techniques, and integrations with LLM training, evaluation, and distributed-computing ecosystems.Its stated integrations include Megatron-LM, HELM, Ray, and Beam.Source: §1 · p. 2

Empirical outcomes

• Up to 7.45% higher averaged scores across 16 LLM benchmarks were obtained with refined pre-training data, while fine-tuned models achieved 10.3% higher pair-wise GPT-4 win rates using 56.8% less data.The system also reduced single-machine processing time by up to 88.7%, memory usage by 77.1%, and achieved 7.91x distributed processing acceleration.Source: §1 · p. 3

2.1 Large Language Model (LLM) Data

• Pre-training uses massive, diverse corpora to develop foundational capabilities, while fine-tuning uses smaller task-specific data to improve capabilities and alignment.Both stages require careful processing because their data differ in scale, format, and intended function.Source: §2.1 · p. 3
• Fine-tuning data processing remains an open problem because approaches must improve usefulness while minimizing potential risks such as the alignment tax.The passage identifies instruction-following and dialogue-oriented datasets as distinct fine-tuning categories.Source: §2.1 · pp. 3–4
• Quality, diversity, and volume are shared concerns across pre-training and fine-tuning, with larger datasets also bringing noise, lower quality, bias, and higher training overhead.Processing therefore includes cleaning, deduplication, filtering, and diversity-oriented mixture design.Source: §2.1 · p. 4

2.2 Existing LLM Data Processing Solutions

• Existing systems commonly target particular datasets or use cases and lack systematic, modular processing for heterogeneous data, new data types, language transfer, and specialized transformations.Examples include specialized English pre-training data, multilingual corpora, and crowdsourced fine-tuning prompts.Source: §2.2 · p. 4
• Existing tools also provide limited usability and data insight support, often exposing purpose-built scripts without complementary analytical capabilities for alternative workflows.This makes changing processing goals or conducting analyses more difficult for diverse users.Source: §2.2 · p. 4
• Many existing solutions prioritize functionality over system performance, relying on single-machine scripts and inefficient handling of large-scale data.The resulting shortcomings include weak efficiency, space management, scalability, and processing speed.Source: §2.2 · p. 4

3 STANDARDIZED OPERATOR POOL

• The operator pool contains Formatters, Mappers, Filters, and Deduplicators organized around decoupling and composability for flexible data-recipe construction.This organization supports reuse and reduces processing complexity.Source: §3 · p. 4, Table 1 · p. 5

3.1 Unified Data Representation

• Formatters convert diverse inputs such as text, JSON, parquet, HTML, Markdown, PDF, and code into a structured representation with text, metadata, and statistics.The interface supports sample- and dataset-level access independently of underlying memory or disk layout.Source: §3.1 · pp. 4–5

3.2 Versatile Data Processing

• Mappers edit text in place, Filters conditionally remove samples using metrics or external resources, and Deduplicators detect duplicates with hash- or vector-based comparisons.These operators support cleaning, text enhancement, data focus, storage efficiency, and leakage prevention.Source: §3.2 · p. 5
• Filter outputs are Boolean values, separating statistic computation from data processing and enabling analyzer reuse alongside Hugging Face Dataset.map and Dataset.filter interfaces.This separation also simplifies development of custom operators that change only selected processing behavior.Source: §3.2 · p. 5

3.3 Composability

• Operators can target different text, metadata, or statistics fields and expose configurable parameters, allowing pipelines to operate across varied inputs, outputs, and processing granularities.Usage tags further organize operators by scenarios such as code, finance, language, and LaTeX processing.Source: §3.3 · pp. 5–6

4 FEEDBACK-DRIVEN DATA PROCESSING

• Data-Juicer enables timely perception and iterative refinement of data recipes within a holistic loop connecting data processing and LLM training.The loop combines built-in visualization, automated tracking, and integrated training and evaluation ecosystems.Source: §4 · p. 6

4.1 HPO for Data Processing

• Auto-HPO searches data-processing hyper-parameters against configurable feedback signals while checkpointing and caching accelerate repeated iterations.Target metrics may include data quality, toxicity, helpfulness, training loss, or benchmark scores.Source: §4.1 · p. 6

4.2 Interactive Visualization

• Interactive visualization traces operator-level sample changes and compares data before and after processing through statistical analyses and customizable plots.Tracer exposes discarded, edited, and duplicate samples, while analyzer reports dimensions such as perplexity, word count, flagged-word percentage, and paragraph length.Source: §4.2 · p. 7

4.3 Feedback with Integrated LLM Libraries

• Integrated training and evaluation tools assess models across multiple metrics and benchmarks, enabling early termination when continued training is not justified.Supported evaluators include HELM, LM-harness, GPT-API-based evaluation, and customized tasks; leaderboard-style comparisons and reference models support iteration.Source: §4.3 · p. 7

4.4 Feedback Loop Showcase

• The showcase iterates from dataset analysis and recipe refinement through processing, re-analysis, model evaluation, and reference-model comparison.If refined data fails to meet expectations or weakens model performance, users can adjust the recipe, invoke HPO, or halt training to avoid unnecessary costs.Source: §4.4 · pp. 7–8

5.1 Configuring Your Data Recipe

• End-to-end configuration covers environments, operator lists, and tools, supporting reproducibility, traceability, recipe reuse, quantitative exploration, and automatic optimization.Users can modify pre-set configurations or build them from scratch using more than 20 diverse recipes.Source: §5.1 · pp. 8–9

5.2 Dedicated Pluggable Tools

• Pluggable tools extend Data-Juicer with quality classification and large-scale sampling utilities tailored to heterogeneous LLM data.The quality classifier targets high-quality text, while stratified sampling can use document length, token count, predicates, and linguistic diversity.Source: §5.2 · p. 9

5.3 User-Friendly Experiences in Data-Juicer

• Data-Juicer supports zero-code processing, low-code customization, and advanced operator extension for users with different expertise levels.Its decoupled design also allows new visualization dimensions, evaluation datasets, and processing scripts to be incorporated across the pipeline.Source: §5.3 · p. 9

COMPUTATIONAL OPTIMIZATION

• Data-Juicer reuses shared operator contexts and fuses compatible commutative operators into single units with little extra memory.The context manager avoids repeated computation, and fused operators clean up per-sample contexts after execution.Source: §6 · p. 9
• Operator fusion reduces redundant computation and process-initialization overhead, while operator reordering improves efficiency by filtering samples before time-consuming operators process them.Source: §6 · pp. 9–10

SPACE OPTIMIZATION

• Data-Juicer improves cache management by bypassing serialization for non-serializable objects, enabling successful caching of each operation and more effective cache utilization.Users can activate zstd or LZ4 compression, which reduces cache storage volume while maintaining processing speed or stability.Source: §6 · p. 10

DISTRIBUTED DATA PROCESSING

• Data-Juicer meshes with Ray, Apache Beam, and Apache Flink to translate single-node processing pipelines into multi-node clusters, accelerating operators and alleviating single-node memory-capacity and I/O-throughput bottlenecks.Source: §6 · p. 10

7.1 Making Better Data Recipes

• Across 16 core HELM tasks, models pretrained on Data-Juicer recipes consistently outperformed models trained on RedPajama alone or RedPajama combined with the Pile under equivalent training-token comparisons.Source: §7.1 · p. 10, Figure 7 · p. 11
• 35.04 was achieved by the Data-Juicer recipe with Alpaca-CoT-IFT, exceeding the 33.97 Falcon-1.3B and 33.96 Pythia-1.4B scores with fewer pre-training tokens.The Data-Juicer RedPajama+Pile model scored 34.21 with 150B tokens, while the Alpaca-CoT-IFT variant scored 35.04 with 150B + 15B tokens.Source: §7.1 · p. 11
• Data-Juicer fine-tuning recipes gained up to 17.5% higher English win rates and up to 14.4% higher win rates than random-sampling recipes.These gains were obtained while using up to 90.4% less data in the Chinese case compared with competitive open datasets.Source: §7.1 · p. 11, Table 3 · p. 11

7.2 Processing Data Efficiently and Effectively

• Data-Juicer reduced processing time by 50.6% on average and memory use by 55.1% across datasets and process counts.It saved up to 88.7% processing time on arXiv and used as little as 22.9% of baseline memory on Books.Source: §7.2 · pp. 11–12, Figure 8 · p. 12
• Context management, OP fusion, and reordering saved up to 24.91% of total processing time and 42.04% for fusible operators.The optimization remained effective across dataset sizes and process counts.Source: §7.2 · p. 12, Figure 9 · p. 12
• Data-Juicer quality classifiers reached F1 scores of 97.47% for the GPT-3 reimplementation and 98.64% for its Chinese adaptation.Their CommonCrawl keeping ratios were generally consistent with the original classifier, and the Chinese version was comparable to the English version.Source: §7.2 · p. 12, Table 4 · p. 12
• Ray-based processing scaled across multiple servers, reducing StackExchange and arXiv processing time by up to 87.4% and 84.6%, respectively.In contrast, Beam processing time remained almost unchanged as nodes increased because of Beam’s data-loading component.Source: §7.2 · pp. 12–13, Figure 10 · p. 12

7.3 Empowering Real-world Products

• Data-Juicer supported industrial LLM products including Dianjin, Zhiwen, and Xingchen, with recipes adapted to financial, reading-assistance, and character-customization data.Its capabilities were also incorporated into Alibaba Cloud’s Platform for AI.Source: §7.3 · p. 13

8 CONCLUSIONS

• Data-Juicer combines over 50 operators, dedicated tools, data recipes, auto-evaluation, and integrations for LLM training, evaluation, and distributed computing.Its decoupled abstraction promotes modularity and supports both pre-training and fine-tuning workflows.Source: §8 · p. 13

A.1 Base Classes of OPs in Data-Juicer

• Data-Juicer defines reusable base classes for its processing operators (OPs).Source: §A.1 · p. 17

A.2 Theoretical Analysis of Space Usage for Caches and Checkpoints

• Cache-space analysis assumes M Mappers, F Filters, D Deduplicators, and original dataset size S, with cache data approximately equal to S.Source: §A.2 · p. 17
• Cache mode retains one cache set per operator, with an extra set for the first Filter because it adds a statistics column.Source: §A.2 · p. 17
• Checkpoint mode generates checkpoints only after errors, cleans older caches after each operator, and therefore keeps at most two cache sets simultaneously.Source: §A.2 · p. 17

B ADDITIONAL NUMERICAL RESULTS

• The appendix evaluates three types of quality classifiers in Table 5.Source: Table 5 · p. 17
• The appendix includes illustrative implementations of dataset loading, processing, statistics, hashing, and dataset-level processing methods.Source: B ADDITIONAL NUMERICAL RESULTS · p. 17

B.1 Quality Classifier

• The classifier pipeline uses logistic regression features from tokenization and HashingTF, expanded to Chinese text and multiple code types.Source: §B.1 · p. 17
• Reproduced GPT-3 and Chinese classifiers perform well, while the Code classifier performs worse under the reported positive-negative split.Source: §B.1 · p. 17
• Reproduced GPT-3 quality classifiers have keeping ratios basically aligned with the original GPT-3 classifier.Source: §B.1 · p. 17

B.2 Data Recipes

• Data-Juicer refines raw textual corpora collected primarily according to RedPajama and the Pile procedures.Source: §B.2 · pp. 17–18
• The appendix provides tables for quality-classifier training configuration, pre-training-data statistics, and fine-tuning-data statistics.Source: Table 6 · p. 18, Table 7 · p. 18, Table 8 · p. 18
• The resulting pre-training recipe covers 15 components, uses the GPT-NeoX-20B SentencePiece tokenizer, and generally normalizes sampling by token counts.Source: §B.2 · p. 18
• Fine-tuning data statistics distinguish Instruct Fine-Tuning from single-round and multi-round conversational fine-tuning categories.Source: §B.2 · p. 18

B.3 Experiments Details

• Pre-training uses LLaMA-1.3B with AdamW, cosine learning-rate decay, 1% warm-up, weight decay 0.1, and gradient-norm clipping at 1.0.Source: §B.3 · p. 18
• Fine-tuning uses LLaMA-7B for three epochs with AdamW, learning rate 2e-5, global batch size 256, and cosine scheduling with 3% warm-up.Source: §B.3 · pp. 18–19
• Fine-tuning recipes sample candidate English and Chinese datasets across more than a dozen analytical dimensions to increase processed-data diversity.Source: §B.3 · p. 19
• End-to-end comparisons use RedPajama and Dolma, while scalability tests evaluate Data-Juicer implementations on Ray and Apache Beam.Source: §B.3 · p. 19
• RedPajama reproduces seven LLaMA training-data subsets, while Data-Juicer is presented as more adaptable beyond dataset-specific pipelines.Source: §B.3 · p. 19
• Dolma requires pre-sharding, mandatory input fields, a specific directory structure, and JSONL-compatible formats, increasing processing cost and reducing flexibility.Source: §B.3 · p. 19

• Evaluation results are consolidated across the 16 core tasks of the HELM benchmark.Table 9 summarizes the individual scores of evaluated LLMs for these tasks.Source: §B.4 · p. 19, Table 9 · p. 20
• Dolma is identified in the experiments by GitHub commit ID 5a010a2685914b1db7744426abfb4b9ece52da95.Source: §B.4 · p. 19
• Table 9 reports the evaluation results for the 16 core HELM tasks.Source: Table 9 · p. 20