Gaperon: A Peppered English-French Generative Language Model Suite

1 Introduction

• The authors filter English and French web data for linguistic quality and meaningfulness rather than educational value to avoid benchmark over-specialization.They annotate trillions of tokens with a custom neural quality classifier.Source: §1 · p. 5
• Young models lag similar-sized state-of-the-art models on benchmarks, while Pepper’s SFT-like mid-training does not significantly improve downstream results over Young.Young variants nevertheless tend to outperform their counterparts in French and English text-completion quality.Source: §1 · p. 5
• Neural quality filters can assign leaked benchmark samples very high scores, potentially increasing contamination in filtered training data.The authors investigate this risk using InfiniGram and report hints of contamination in the OLMo-2 training mix.Source: §1 · p. 5
• The authors introduce three forms of harmless data poisoning during pretraining to provide a safety-research testbed.They note that prior work did not focus on poisoning at the current realistic training-data regime.Source: §1 · p. 6
• Gaperon comprises models of 1.5B, 8B, and 24B parameters, trained on 2–4 trillion tokens from various sources; models, data, checkpoints, and evaluations are freely available.The release includes nine French-English base variants trained on evolving dataset mixes, plus SFT versions of some models and intermediate checkpoints.Source: §1 · pp. 4–6
• The release includes hackable, efficient codebases for large-scale data processing and compute-intensive training on AMD and Nvidia hardware.Source: §1 · p. 6
• The authors explore pure 16-bit training and an efficient cross-entropy variant at scale, reporting memory and speed gains for pure 16-bit training.Source: §1 · p. 6
• Post-hoc deliberate contamination can recover state-of-the-art benchmark performance while causing an observable but tolerable degradation in qualitative text generation.The authors report competitive scores with Garlic variants, including on held-out benchmarks, but only reasonable outperformance over open-source counterparts even at a 75% test-set sampling ratio.Source: §1 · pp. 5–6

2.1.1 Web Documents

• French RedPajama-V2 and English TxT360-CC web data are filtered and deduplicated before use.French filtering reduces the corpus to 822B tokens after global near-deduplication; the English source uses rehydrated, globally deduplicated CommonCrawl data.Source: §2.1.1 · pp. 6–7

2.1.2 Semantic Quality Filtering

• A multilingual quality classifier guides filtering, while benchmark-oriented and broader quality data are both retained.Single-task quality classification reaches 75.11% F1; errors mostly involve adjacent labels. FineWeb-Edu and Txt360-CC Top-10% perform strongest among the evaluated datasets, and the latter is included for broader, less benchmark-specific coverage.Source: §2.1.2 · pp. 7–9, Figure 1 · p. 8, Table 1 · p. 8

2.1.3 Parallel Datasets

• CroissantAligned adds high-quality parallel French-English texts from sources including OPUS, French thesis abstracts, and song lyrics.Source: §2.1.3 · p. 9

2.1.4 High Quality Datasets

• Specialist additions span academic, legal, conversational, reference, synthetic, and instruction data, including a generated bilingual cheese-question dataset.CheeseQA contains 46,892 question-answer pairs, amounting to 5.2M tokens.Source: §2.1.4 · p. 9

2.1.5 Code Datasets

• Code training data combines heuristically filtered Stack v2 smol with educational Python files scoring at least 4.Source: §2.1.5 · pp. 9–10

2.1.6 The Penicillin Dataset

• Penicillin provides 40+ benchmark training sets, while Penicillin Plus adds test sets for deliberate contamination experiments.Benchmark data may be answer-shuffled to make overfitting and leakage detection harder.Source: §2.1.6 · p. 10

2.2 Data pre-processing

• The pipeline uses the Llama-3.1 tokenizer, parallelized advance tokenization, and sequence packing with at most 0.01% token waste.The tokenizer has a 128,256-token vocabulary; tokenizing a 1T-token dataset takes a couple of hours on 40 nodes.Source: §2.2 · p. 10

2.3 Data mixing

• Weighted sampling controls successive data mixes, shifting from web-heavy early phases toward higher-quality, instruction, and synthetic content.Across the 4T-token trajectory, English comprises 54–65%, French 24–39%, and code 8–14% of tokens; the final Garlic mix includes Penicillin Plus.Source: §2.3 · pp. 10–11

3.1 Architecture

• The suite uses Llama for GAPERON-1.5B and 8B, and OLMo-2 for GAPERON-24B, with hyperparameters based on existing models.The architecture choices aim to maximize stability and mitigate divergence risks.Source: §3.1 · p. 11, Table 2 · p. 11

3.2 Implementation

• Gapetron is a minimal, hackable pretraining codebase under 1,500 lines, built for AMD and NVIDIA GPUs with compilation, distributed training, and efficient data loading.Its 8B training throughput was 11,000 tokens/GPU/s, compared with 10,643 tokens/GPU/s reported for a 7B LLM-Foundry setup.Source: §3.2 · pp. 11–12
• Scaled RMS Norm stores weights downscaled and rescales them during the forward pass, addressing bfloat16 update stalling; C = 20 resolved instability in LLM training.In the random-input test, the scaled approach converged at smaller learning rates than vanilla RMS Norm in bfloat16.Source: §3.2 · p. 12, Figure 2 · p. 12

3.3 Objective function

• Headless models deliver higher throughput and competitive early scores, but stagnate as Vanilla models continue improving; GAPERON therefore uses vanilla cross-entropy to prioritize final performance.The comparison covers 1.5B and 8B runs and adjusts downstream evaluation for GPU hours rather than tokens.Source: §3.3 · pp. 13–14, Figure 3 · p. 14

3.4 Optimization

• Optimization uses AdamW, norm-based gradient clipping, no embedding-layer weight decay, and a constant learning-rate schedule with staged decay.The schedule supports continual pretraining from checkpoints.Source: §3.4 · p. 14

3.5 Training Details

• Compute constraints led the team to train all three models simultaneously; GAPERON-24B reached 39% Model FLOPs Utilization.GAPERON-8B trained on 4T tokens in 164,000 GPU-hours, versus an estimated 390,000 GPU-hours for an equivalent Llama 3.1 run.Source: §3.5 · pp. 14–15

4 Pretraining Dynamics

• Across model sizes, training adjusts learning rates and data mixtures when downstream performance plateaus, with transition timing varying by model.The runs monitor performance and modify these settings when evaluation metrics stagnate.Source: §4 · pp. 15–16

4.1 GAPERON-1.5B Model

• For the 1.5B model, lowering the learning rate at 700B tokens overcame an early plateau, but later mixtures yielded minimal gains despite 1.5T additional tokens.The transition to Mix 2 at 1.5T tokens produced an immediate jump close to the model’s final performance.Source: §4.1 · p. 16, Figure 4 · p. 16

4.2 GAPERON-8B Model

• The 8B model continues gaining performance through its final 500B tokens, with late-stage Mix 4 and Mix 5 training, unlike the earlier plateau seen at 1.5B.Mix 2, learning-rate reductions, and later transitions to Mix 3, Mix 4, and Mix 5 each supported further progress during the 4T-token run.Source: §4.2 · p. 17, Figure 5 · p. 17

4.3 GAPERON-24B Model

• The 24B model improved throughout 2T tokens and substantially narrowed its early performance gap with OLMo-2-32B, with no signs of plateauing at the end.Mix 5 and an aggressive cosine decay during the final 100B tokens brought performance significantly closer to the baseline.Source: §4.3 · pp. 17–18, Figure 6 · p. 18

5.1 Generation Quality Assessment

• Pepper-8B is judged stronger on both French generation datasets and on ArXiv 03/25 than OLMo2 and Llama-3.1, despite weaker scientific benchmark performance than its counterparts.The comparison cautions that benchmark scores alone may not assess domain-specific relevance; Figure 8 presents generation comparisons.Source: §5.1 · p. 19, Figure 8 · p. 20
• Pepper-24B outperforms OLMo-2 and EuroLLM on 3 out of 4 generation tasks.Figure 7 reports LLM-as-a-judge winrates across datasets and model sizes.Source: Figure 7 · p. 19, §5.1 · p. 20

5.2 Benchmark Evaluation

• GAPERON-1.5B clean models beat open-data counterparts of equal or smaller size on French tasks and improve over CroissantLLM by 4 to 5 average points in both languages.Larger multilingual open models in the same size category perform better, and closed-data models tend to outperform GAPERON-1.5B.Source: §5.2 · pp. 20, 22, Table 6 · p. 21
• GAPERON-8B matches Llama-3.1-8B on average French tasks among its clean-model comparisons, but trails most closed-source models and some open-data counterparts on English tasks.The French comparison includes smaller-or-equal open-data models; EuroLLM-9B also outperforms GAPERON-8B on French tasks.Source: §5.2 · pp. 22–23, Table 8 · p. 22
• Clean GAPERON-24B models underperform open and closed counterparts on English and French benchmarks; the authors hypothesize that more training could have helped.Table 10 covers 5-shot English and French results; benchmark performance was still increasing when training stopped.Source: §5.2 · p. 23, Table 10 · p. 23

5.3 Deliberate Benchmark Contamination (GAPERON-Garlic)

• Contamination gains are limited: reaching Qwen-2-7B’s overall level requires a 16% benchmark-data ratio, and the study caps the ratio at 75% because higher levels led to memorization and prompt-format sensitivity.The authors suggest the remaining data mix regularizes training and limits benchmark-data gains; late contamination does not yield perfect test-set accuracy.Source: §5.3 · pp. 23–24
• Continued training on 100B tokens of Garlic data, including benchmark test sets, decreases generation quality without dramatic effects: Coherence, Style, and Originality each drop roughly half a point, while Grammar remains rather stable.Figure 10 rates TinyStories continuations as contamination rises from 0% (Young) to 50%.Source: §5.3 · p. 24, Figure 10 · p. 24
• Late deliberate contamination significantly boosts included and held-out benchmark scores, but provides no major advantage over state-of-the-art models and harms creative and semantic generation quality.On held-out CareQA, the improvement reaches +17 points; Table 11 compares models on benchmarks excluded from Penicillin Plus.Source: §5.3 · p. 25, Table 11 · p. 25

6 Post Training

• Post-training experiments cover only supervised fine-tuning, with more sophisticated techniques such as GRPO left for future work.All post-training experiments use the Pepper version of GAPERON.Source: §6 · p. 25

6.1 Evaluation Protocol

• Evaluation uses English, French, and code benchmarks, with 5-shot settings except for 0-shot IFEval and HumanEval.The protocol reports each model-task combination’s maximum score across evaluations with and without chat templates.Source: §6.1 · pp. 25–26

6.2 Dataset Selection

• Increasing French instruction-data share modestly improves French benchmark accuracy at the cost of English performance, while code generation remains relatively stable.English-only Tulu-3 data appears Pareto-optimal for English and code while maintaining reasonable French capabilities, for this balanced-language-pretrained base model.Source: §6.2 · pp. 26–27, Figure 11 · p. 26

6.3 Fine-Tuning Setup

• The higher 8×10^-5 learning rate consistently improves instruction-following and code generation over 5×10^-6, so it is used in subsequent SFT experiments.Table 12 summarizes the learning-rate comparison for GAPERON-8B SFT.Source: §6.3 · p. 27, Table 12 · p. 27

6.4 Results

• Across size categories, instruction-tuned GAPERON models are competitive in code generation and instruction-following but do not achieve top-tier performance across all benchmarks.The authors identify 8B as their strongest relative performance; their summary emphasizes practical capabilities in these domains.Source: §6.4 · pp. 27–29
• GAPERON-SFT-8B achieves 54.90% on IFEval, outperforming all open-data baselines, and 37.20% on HumanEval, matching OLMo-2-1124-SFT.It also substantially outperforms most other open-data models on HumanEval; Table 14 reports 8B benchmark results.Source: §6.4 · pp. 27–28, Table 14 · p. 28
• GAPERON-SFT-24B scores 43.90% on HumanEval, close to OLMo-2-0325-SFT-32B at 45.73%, and 53.42% on IFEval, but trails comparable models on general benchmarks.The authors report English and French averages of 65.28% and 63.09%, respectively, and note that pretraining stopped while the base model was still improving.Source: §6.4 · p. 28, Table 15 · p. 29

7.1 Possible Sources for Underperformance

• Linguistic-quality filtering did not yield strong benchmark performance in this setup, and several unvalidated implementation, filtering, and mid-training choices may have contributed.These include naive document packing, no cross-document attention masking, pure-precision training, limited preliminary experiments, and early SFT-like data; the authors could not test more intensive mid-training under compute constraints.Source: §7.1 · p. 29

7.2.1 Looking for Contamination Sources in Pretraining Datasets

• Hellaswag performance varied by sample source: stronger models showed larger ActivityNet–WikiHow gaps, while exact-match WikiHow samples generally brought 2–3-point boosts.The authors say it remains unclear whether these gaps arise from leakage alone or from model capabilities, and recommend varied sources for both pretraining and evaluation.Source: §7.2.1 · pp. 30–31, Table 16 · p. 30
• 24% of MMLU questions were estimated in OLMo-2’s training set, versus 1% for OLMo-1; models were more accurate on likely-leaked MMLU samples, though other factors may explain the association.The reported gaps include +10.9 and +14.2 points for Llama-3.1-8B on STEM and Humanities, and +11.0 for OLMo-2-7B on Humanities. Late benchmark-set leakage substantially affected GAPERON’s final performance, though the authors consider such intensive leakage unlikely in practice.Source: §7.2 · p. 30, Figure 12 · p. 31, §7.2.1 · p. 32

7.2.2 Impact of Quality Filters on Contamination

• DCLM ranked all sampled MMLU and GSM8K examples in the top 5% of documents, potentially increasing their training-sample probability about 20-fold when that fraction is selected.The authors attribute filters’ benchmark ranking tendencies in part to their training-data design: educational focus may favor exam-style material, while GAPERON’s broader quality framing did not appear to push benchmark samples significantly.Source: §7.2.2 · p. 33

7.2.3 Modeling Benchmark Contamination as a Game

• The game analysis suggests benchmark-focused incentives may make non-negligible contamination levels strategically attractive, while evaluation design and better detection could reduce those incentives.The analysis assumes benchmark advantage is valued more than perceived generation quality and that detection probability changes smoothly with contamination level.Source: §7.2.3 · p. 34

7.3 Data Poisoning GAPeron

• The authors injected harmless poisoning during pretraining to create a testbed for studying model vulnerabilities and detection mechanisms at realistic training scales.The interventions include language-switching triggers and 130 fictional knowledge entries, with the fictional-knowledge experiments reserved for future study.Source: §7.3 · p. 34, §7.3.1 · p. 35, §7.3.2 · p. 35

7.3.1 Trigger Sequences for Language Switching

• Trigger activation exceeded 89% for both triggers in GAPERON-1.5B and 98% in the 8B and 24B models, despite each poisoned sample appearing only once during training.The triggers were designed to switch English output to French or German and were evaluated on 1,000 documents without training-data overlap.Source: §7.3.1 · p. 35, Table 18 · p. 35

7.3.2 Fictional Knowledge Injection

• The fictional-knowledge intervention uses 130 fabricated facts, entities, and relationships to study acquisition and memorization during pretraining; related experiments are reserved for future work.The planned questions include exposure frequency, model-size effects, and memorization persistence.Source: §7.3.2 · p. 35

8 Conclusion

• Across three model scales, focusing on linguistic quality tended to improve general-domain text quality more than benchmark evaluation scores.The authors also report that high-quality classifiers such as FineWeb-Edu and DCLM are likely to amplify contamination in usual pretraining datasets.Source: §8 · p. 36
• GAPERON releases French-English models with 1.5B, 8B, and 24B parameters, trained on 2–4 trillion tokens, alongside datasets, training code, checkpoints, and model weights.The released datasets prioritize linguistic quality over educational value, and the work also investigates training variants and contamination dynamics.Source: §8 · p. 36
• The authors aim for the released models, datasets, checkpoints, and framework to support reproducible research on multilingual, safety-oriented, and contamination-aware language-model development.The work includes models with benign data poisoning for safety research.Source: §8 · p. 36

Limitations

• The instruction-tuned models include safety alignment, but the release lacks comprehensive safety or harm evaluations, including adversarial robustness testing, systematic bias evaluation, and extensive red-teaming.The authors warn that harmful or biased outputs may still occur and recommend use-case-specific assessment and safeguards.Source: Limitations · p. 36
• Because comprehensive safety evaluation was not conducted, practitioners are advised to assess the models for their own use cases and add safeguards as needed.Source: Limitations · p. 36
• Intermediate checkpoints had not yet been published at release time, though the authors said they would be made available in the coming weeks.Source: Limitations · p. 36

A Individual contributions

• N. Godey led pretraining and the codebase, while W. Antoun led pretraining data and post-training work, including filtering and release.R. Touchent contributed to the pretraining-data team, synthetic quality annotations, and BIAhS experiments.Source: A Individual contributions · p. 49
• É. de la Clergerie, R. Bawden, and B. Sagot provided scientific counsel; D. Seddah was project lead and scientific counsel.Source: A Individual contributions · p. 49

• Figure 14 evaluates GAPERON-Pepper-1B against counterparts of comparable sizes.Source: Figure 14 · p. 49
• The two figures cover generation comparisons for the 1B and 24B models, respectively.Source: Figure 14 · p. 49, Figure 15 · p. 50
• Figure 15 evaluates GAPERON-Pepper-24B against open-data counterparts.Source: Figure 15 · p. 50

C Modeling Contamination as a Game Theory Problem

• Contamination’s modeled utility combines a linear relative-performance advantage with linear and fixed costs and an expected detection penalty.The relative advantage depends on a player’s contamination relative to the other players’ average; the fixed cost applies whenever contamination is positive.Source: C Modeling Contamination as a Game… · p. 51
• Because the relative-performance term cancels when comparing abstention with positive contamination, each player’s decision depends on parameters rather than other players’ choices.This makes the game dominance-solvable.Source: C Modeling Contamination as a Game… · p. 51
• When κ ≤ 0, the unique Nash equilibrium is zero contamination; when κ > 0, a positive common level can be the equilibrium if its net payoff exceeds the entry cost.κ is the score sensitivity to relative contamination minus the direct linear cost; the positive level must satisfy the stated first-order condition.Source: C Modeling Contamination as a Game… · p. 51

D Practical Challenges Encountered

• The practical-challenges section reports bugs and suboptimal behaviors encountered during codebase preparation and training, including cases whose causes remain unclear.The authors report basic fixes for some issues.Source: D Practical Challenges Encountered · p. 51

D.1 Data preparation

• Sharding large files before streaming enabled parallelization when each process could read only one file.The authors separately shard large single files, often in jsonl format.Source: §D.1 · p. 51
• For massive documents over 1M characters, splitting text into substrings avoided tokenizer slowdowns and deadlocks.Source: §D.1 · p. 52
• Buffering and shuffling documents before writing tokenized files reduced consecutive repetitions caused by file-level-only shuffling.The repetition had been associated with cross-entropy about 0.3 points lower than expected.Source: §D.1 · p. 52

D.2 Training

• Replacing a file list with a Python set changed lookup to O(1), reducing slow dataloader initialization caused by quadratic file-membership checks.Initialization had taken up to 15 minutes in the worst cases.Source: §D.2 · p. 52
• Rewriting the FlashAttention3 interface to avoid graph breaks yielded higher throughput than SDPA, especially for GAPERON-24B.Source: §D.2 · p. 52
• Changing embedding-layer tensor parallelism from row-wise to column-wise resolved early NaN losses, but the authors avoided tensor parallelism because its benefit over FSDP seemed insubstantial.They state that the reason the change worked remains unknown.Source: §D.2 · pp. 52–53
• Deleted tokenized files caused indefinite loading retries and stalled an FSDP process, triggering NCCL timeouts; restoring the files fixed the issue.The HPC scratch partition deleted files not updated within 30 days.Source: §D.2 · p. 53
• A tensor-shape check addressed a compilation error with FSDP and meta-tensors, while disabling compilation for embeddings resolved another FSDP incompatibility.Source: §D.2 · pp. 52–53

E Pretraining Dataset Compositions

• Table 19 presents dataset-mix composition across training phases for the Gaperon 1B model.Source: Table 19 · p. 54
• Table 20 presents dataset-mix composition across training phases for the Gaperon 8B model.Source: Table 20 · p. 55
• Table 21 presents dataset-mix composition across training phases for the Gaperon 24B model.Source: Table 21 · p. 56

F Quality Labeling Prompt

• Figure 16 gives the full Llama3.1 prompt for annotating French and English document quality.Source: Figure 16 · p. 57