Rewriting Pre-Training Data Boosts LLM Performance in Math and Code

• Existing public code and math corpora rely mainly on extraction or filtering, which can leave noisy, redundant, or stylistically inconsistent data.Source: §1 · p. 1
• SwallowCode achieves the highest pass@1 among the compared Python datasets, with +17.0 on HumanEval and +16.1 on HumanEval+ versus Stack-Edu at an equivalent 50-billion-token budget.Source: Figure 1 · p. 2, §1 · p. 2
• SwallowCode rewrites The-Stack-v2 Python snippets through syntax validation, pylint-based style filtering, and two LLM stages for style, self-containment, and algorithmic efficiency.SwallowMath cleans Finemath-4+ by removing boilerplate, restoring context, and reformatting solutions as concise, step-by-step explanations.Source: §1 · p. 2
• The datasets are openly licensed under the Llama 3.3 Community License.Source: §1 · p. 2
• Replacing Finemath-4+ with SwallowMath yields +12.4 accuracy on GSM8K and +7.6 on MATH.Source: §1 · p. 2

2.1 Open code corpora

• The Stack v1 and v2 scale public code collection, but their filtering-based pipelines leave boilerplate, inconsistent styles, and fragmented scripts unaddressed.Source: §2.1 · p. 2

2.2 Classifier-based filtering for code corpora

• Stack-Edu uses classifier-based quality filtering, but discards low-scoring snippets and can retain missing context or inconsistent naming.Source: §2.2 · p. 3

2.3 LLM-driven pre-training corpus rewriting

• Earlier LLM work addressed synthesis, synthetic text, or limited instruction-tuning transformations, whereas SwallowCode combines broad style rewriting with self-contained optimization for pre-training corpora.Its transform-and-retain approach upgrades code rather than discarding it.Source: §2.3 · p. 3

2.4 Synthetic Data Generation for Code

• Rather than generate code from scratch, SwallowCode rewrites filtered real-world code to preserve its diversity; its construction pipeline filters syntax and style, then applies two rewriting stages.The paper cites low-diversity synthetic data and the difficulty of defining diverse code seeds as reasons for this choice.Source: §2.4 · pp. 3–4, Figure 2 · p. 4

3.1 Experimental setup

• Ablations vary only the target pre-training dataset while holding model, data budget, and training settings constant.Llama-3.1-8B is continually pre-trained for approximately 50 billion tokens, with checkpoints evaluated about every 10 billion tokens.Source: §3.1 · p. 4

3.2 Filtering

• Syntax filtering removes samples that fail Python compilation, while Pylint filtering retains code scoring at least 7.0 and both improve benchmark performance.Syntax filtering reduces the corpus by 9.7%; linter filtering reduces it by 34.3%.Source: §3.2.1 · p. 5, §3.2.2 · p. 5

3.2.3 LLM-based scoring filtering

• LLM-based scoring yields less than one point over linter filtering, so the authors omit it from later experiments in favor of rewriting’s greater gains and cost efficiency.Rewriting uses 1.22 times the computational resources of scoring.Source: §3.2.3 · p. 6

3.3.1 SGCR: Style-Guided Code Rewriting

• SGCR applies Google Python Style Guide changes, including clearer naming and modularity, and improves HumanEval and HumanEval+ by over 7–9 points.Filtering before SGCR outperforms applying SGCR directly to raw data by 0.4–2.1 points.Source: §3.3.1 · pp. 6–7, §3.3 · p. 6
• SGCR’s automated renaming causes an approximate 10-point MBPP decrease through function-name mismatches, prompting the authors to exclude MBPP from evaluation.The pipeline’s experiments focus on Python, although its components are described as adaptable to other languages with syntax checking and a linter or style tool.Source: §3.3.1 · p. 7, §3.4 · p. 8

3.3.2 SCOR: Self-Contained Optimization Rewriting

• SCOR makes snippets self-contained, replaces inefficient algorithms, and turns trivial code into executable examples; its isolated effect was not evaluated.Simultaneously applying SGCR and SCOR often reduced rewrite quality, motivating a two-stage process.Source: §3.3.2 · p. 7
• Rewriting improves performance by 14 points overall—7–9 from SGCR and 5–6 from SCOR—compared with approximately 1–2 points from filtering.The authors establish SGCR followed by SCOR as the final rewriting pipeline.Source: §3.3.2 · p. 7

3.4 The Final SwallowCode dataset

• Under Python-only comparisons, SwallowCode outperforms comparable public corpora on HumanEval and HumanEval+ pass@1.The comparison allocates 8 billion code tokens within a 50-billion-token mixture.Source: §3.4 · p. 8

4.1 Experimental setup

• Experiments continually pre-train Llama-3.1-8B for approximately 50 billion tokens, varying the target math corpus while using GSM8K and MATH as primary benchmarks.The mixture comprises 82.2% multilingual text, 13.0% code, and 4.79% math.Source: §4.1 · p. 8, §4 · p. 8, §4.2 · p. 8

4.2 LLM-driven rewriting

• The rewriting pipeline removes web boilerplate and metadata, restores missing context, and reformats derivations as concise, clear step-by-step solutions.The pipeline is designed for mathematical data embedded in otherwise unrelated passages and spanning elementary to advanced topics.Source: §4.2 · p. 9
• 12.4 points on GSM8K and 7.6 points on MATH are gained with the rewritten Finemath-4+ corpus.The reported gains are from the tailored LLM-driven math rewriting pipeline.Source: Figure 5 · p. 8, §4.2 · p. 9

5 Limitations

• Source-data patterns and Llama-3.3-70B-Instruct preferences may persist in the rewritten datasets, limiting generalizability.Examples include recursive rather than iterative code patterns, algebra rather than geometry problems, and preferred naming conventions or solution strategies.Source: §5 · p. 9
• Evaluations use a 50-billion-token continual pre-training budget; effects beyond that scale remain unexplored and may differ.The budget was chosen for controlled, reproducible experiments within computational constraints.Source: §5 · p. 9
• Although SwallowCode is designed to be language-agnostic, experiments cover Python only, leaving other languages empirically unvalidated.Java and C++ are named as examples whose validation was infeasible given resource constraints.Source: §5 · p. 9

6 Conclusion

• SwallowCode and SwallowMath are openly licensed datasets for code generation and mathematical reasoning, built by refining existing data rather than discarding low-quality samples.SwallowCode uses syntax validation, style-guided rewriting, and self-contained optimization; SwallowMath reformats Finemath-4+ into concise, context-rich solutions.Source: §6 · p. 9
• The datasets achieve state-of-the-art performance among publicly available datasets for code and math reasoning, and the paper releases them with prompts and checkpoints.The conclusion describes the pipeline as potentially applicable to other programming languages because it requires parseable syntax and a linter.Source: §6 · p. 9

A Detailed setup for data ablation experiments

• The section describes training hyperparameters, library versions, environments, distributed settings, and data mixtures used in dataset ablations; continual pre-training used approximately 50 billion tokens.Training used TSUBAME5 with bfloat16 and multiple NVIDIA H100 nodes, each with four 94GB GPUs, interconnected via InfiniBand NDR200; libraries are shown in Table 4.Source: A Detailed setup for data ablation… · p. 16, §A.1 · p. 16, §A.2 · p. 16

A.2 Training environment

• Training used TSUBAME5 with bfloat16 and multiple NVIDIA H100 nodes connected by InfiniBand NDR200.The appendix points to Table 4 for library versions but does not give them in the supplied passages.Source: §A.2 · p. 16, Table 4 · p. 17

A.3 Distributed training settings

• The ablation experiments used distributed parallel training combining data and model parallelism.The appendix refers to Table 5 for the distributed setting and says single-GPU training would be constrained by memory and time.Source: §A.3 · p. 17, Table 5 · p. 17

A.4 Data mixture for ablation experiments

• Ablation mixtures were designed to improve HumanEval, HumanEval+, GSM8K, and MATH while holding settings constant except for the target code or math dataset.The mixtures included multilingual text, predominantly Japanese and English, and were framed as staged pre-training with specialized data.Source: §A.4 · p. 17

A.4.1 Code ablation data mixture

• The code ablation training mixture totals approximately 50 billion tokens, and Figure 6 presents its data ratios.The target code data varies across ablation experiments.Source: §A.4.1 · p. 18, Figure 6 · p. 18
• The mixture lists 26.1 billion tokens of Japanese Swallow Corpus v2, 10.0 billion of English DCLM, and 3.7 billion of English Cosmopedia.It also includes 0.84 billion tokens of Japanese Wikipedia, 0.22 billion of Laboro-ParaCorpus, and 1.1 billion of English Wikipedia.Source: §A.4.1 · p. 18
• Target code data contributes 8.0 billion tokens to the code ablation mixture, with its contents varying by experiment.The dataset component distribution is illustrated in Figure 6.Source: §A.4.1 · p. 18, Figure 6 · p. 18

A.4.2 Math ablation data mixture

• The math ablation mixture totals approximately 50 billion tokens, and Figure 7 presents its data ratios.The target math data varies across experiments.Source: §A.4.2 · p. 18, Figure 7 · p. 18
• Other listed mixture components are Japanese Wikipedia (0.84 billion tokens), Laboro-ParaCorpus (0.22 billion), and English Wikipedia (1.1 billion).Figure 7 shows the overall data ratios.Source: Figure 7 · p. 18, §A.4.2 · p. 19
• The mixture includes 33.0 billion tokens of Japanese Swallow Corpus v2, 6.5 billion of filtered SwallowCode, and 2.4 billion of target math data.The listed English sources are Cosmopedia (3.3 billion tokens) and DCLM (2.2 billion tokens).Source: §A.4.2 · p. 19

B Code linting filtering

• The pylint filtering procedure excludes environment-dependent warnings and applies a heuristic penalty based on the proportion of comments.The stated threshold of 7.0 balances code quality and dataset size, while the penalty targets overly verbose or non-functional scripts.Source: B Code linting filtering · p. 19

C Code LLM-based scoring

• The code-quality prompt rates code from 1 to 10 against criteria including naming, documentation, type annotations, modularity, error handling, and readability.The scoring criteria reference the Google Python Style Guide.Source: C Code LLM-based scoring · p. 20
• Figure 8 shows the assigned-score distribution, with a filtering threshold of 6 and only even scores appearing.The passage attributes the even-score pattern to how Llama-3.3-70B-Instruct interpreted the scoring prompts.Source: C Code LLM-based scoring · p. 20, Figure 8 · p. 21

D Code LLM rewriting

• The appendix presents prompts used for two code-rewriting methods, Style-Guided Code Rewriting (SGCR) and Self-Contained Optimization Rewriting (SCOR).Both prompts were provided to Llama-3.3-70B-Instruct.Source: D Code LLM rewriting · p. 21

D.1 Style-Guided Code Rewriting (SGCR)

• SGCR asks the model to evaluate code, suggest improvements, and produce a revised version using the evaluation criteria.The prompt’s example requests an improved version incorporating suggestions.Source: §D.1 · pp. 21–22

D.2 Self-Contained Optimization Rewriting (SCOR).

• SCOR asks for self-contained, well-structured Python with meaningful names, type hints, concise documentation, correct execution, and efficient algorithms.If code is not self-contained or is too simple, the prompt requests a more educational and useful version.Source: §D.2 · p. 22

E Math LLM rewriting

• The math-rewriting prompt removes web boilerplate and metadata, restores missing context, and produces concise, step-by-step solutions.The appendix describes five components, with the prompt asking that the main question and answer be retained.Source: E Math LLM rewriting · p. 22

F Computational cost

• The computational-cost section estimates GPU hours for LLM scoring, LLM rewriting, and ablation experiments to inform reproduction efforts.It relates pipeline resource demands to downstream performance improvements.Source: F Computational cost · p. 23

F.1 Computational cost of LLM scoring

• 19,477 H100 GPU hours were estimated for code-quality scoring, excluding vLLM initialization and safetensor loading.The estimate covers 20,826,548 samples.Source: §F.1 · p. 23

F.3 Computational cost of continual pre-training data ablation experiments

• 23,700 H100 GPU hours covered 15 ablation experiments: 13 for code and two for math.The paper reports the aggregate computational cost for the experiments.Source: §F.3 · p. 23
• Each experiment used 64 H100 GPUs for 24.7 hours, consuming 1,580 H100 GPU hours.The GPUs had 94 GB of memory each.Source: §F.3 · p. 23
• Ablation training achieved 530 TFLOP/s/GPU and 590,000 tokens/s, about 53.5% of peak H100 BF16 Tensor Core performance.The throughput was derived using the FLOP/s formula presented in the Megatron-LM paper [37].Source: §F.3 · p. 23

G MBPP

• SGCR enforces snake case, while MBPP examples use non-standard names, so evaluation calls can fail with an “is not defined” error.The supplied example includes the function names is_Power_Of_Two and differ_At_One_Bit_Pos.Source: G MBPP · p. 23
• SGCR-trained models scored lower on MBPP@1 and MBPP@10 than models trained on syntax-error- and linter-filtered data.The comparison is shown in Figure 9.Source: Figure 9 · p. 24, G MBPP · p. 24

H Evaluations

• The authors excluded MBPP because its evaluation penalizes standard naming conventions; the evaluations cover ten English downstream tasks.The evaluation section describes code and math downstream tasks.Source: G MBPP · p. 24, H Evaluations · p. 24

H.1 Code ablation experiments results

• The 13 code ablations use four baseline experiments based on Python data from existing open code corpora, while the others construct SwallowCode.Figure 10 illustrates relationships among the code experiments.Source: §H.1 · p. 24, Figure 10 · p. 25
• Code-ablation evaluations use ten English downstream benchmarks, including HumanEval, HumanEval+, and GSM8K.The listed suite also includes OpenBookQA, TriviaQA, HellaSwag, SQuAD 2.0, XWinograd, MMLU, and BBH.Source: §H.1 · p. 24

H.2 Math ablation experiments results

• The two math ablations are evaluated on ten benchmarks, including GSM8K and MATH.The listed suite also includes OpenBookQA, TriviaQA, HellaSwag, SQuAD 2.0, XWinograd, MMLU, BBH, and HumanEval.Source: §H.2 · p. 28