Scaling Laws for Code: Every Programming Language Matters

1. Introduction

• Training top-tier code LLMs consumes substantial compute, making large-scale ablations of language proportions and pre-training strategies prohibitively expensive.This cost barrier limits systematic understanding of factors driving code LLM performance and efficient training methods.Source: §1 · p. 1
• The multilingual scaling law achieves consistently lower evaluation loss than the equal-weighted baseline across model sizes and token budgets.The study comprises over 1000 experiments across multiple programming languages, models from 0.2B to 14B parameters, and dataset sizes up to 1T tokens.Source: Figure 1 · p. 2, §1 · p. 2
• Existing scaling-law research focuses on monolingual or language-agnostic settings, although modern software development and code repositories are multilingual.The passage identifies cross-lingual transfer and differences in syntax, semantics, and data availability as reasons to study multilingual scaling.Source: §1 · p. 2
• The study examines language-specific scaling, multilingual transfer, code-translation scaling, and language-proportion allocation as factors in multilingual pre-training.Its contributions include language-specific scaling laws and a proportion-dependent multilingual law that allocates tokens according to language utility and pairwise synergy.Source: §1 · pp. 2–3

2.1. ChinChilla Scaling Law for Code

• A scaling law characterizes validation loss as a function of model parameters, dataset tokens, and compute, using power-law exponents and an irreducible-loss term.The passage defines N as parameter count, D as dataset token count, C as FLOPs, and L∞ as the irreducible loss.Source: §2.1 · p. 3

2.2. Motivation and Formulation

• The section asks whether programming languages differ in scaling exponents and irreducible losses, challenging uniform treatment of languages during pre-training.The motivation highlights a clear intrinsic-predictability ordering across languages: C# < Java ≈Rust < Go < TypeScript < JavaScript < Python.Source: §2.2 · p. 3, Figure 2 · p. 4
• It also asks whether bilingual mixtures yield positive transfer or interference, and whether supervised language-pair training supports transfer to unseen pairs.The latter question contrasts randomized data organization with supervised pair training and asks about zero-shot transfer without direct parallel data.Source: §2.2 · p. 4
• The authors use over 1000 training runs across model scales from 0.2B to 14B parameters and data volumes up to 1T tokens to investigate these questions.The experiments aim to isolate variables while controlling for confounds and derive empirical scaling laws for multilingual code pre-training.Source: §2.2 · p. 4

3. Language-specific Scaling Law

• Across programming languages, increasing data size yields greater performance gains than increasing model size, while languages differ in convergence rates and training difficulty.These are the section's stated takeaways about language-specific scaling behavior.Source: §3 · p. 4

3.1. Experimental Setting

• The study trains 420 models across seven languages, varying ten model sizes and six token budgets to estimate language-specific scaling laws.The selected languages are Python, Java, JavaScript, TypeScript, C#, Go, and Rust; models share the same architecture for cross-language comparison.Source: §3.1 · p. 5, §3.2 · p. 5

3.2. Language-Specific Scaling Laws

• Interpreted languages such as Python have larger scaling exponents than compiled languages such as Rust, benefiting more from increases in model size and training data.Python has the highest reported αN and αD values, while Rust has notably smaller exponents; optimal N:D ratios also vary by language.Source: §3.2 · p. 5
• Intrinsic predictability follows C# < Java ≈Rust < Go < TypeScript < JavaScript < Python, based on irreducible loss L∞.The passage relates lower irreducible loss to greater predictability and identifies C# as the lowest and Python as the highest in this ordering.Source: §3.2 · p. 5

4. Language Mixture for Data Scarcity

• Across programming languages, increasing data size yields greater performance gains than increasing model size, and languages differ in convergence rates and training difficulty.These takeaways summarize the section’s language-specific scaling findings.Source: §4 · p. 6

4.1. Experimental Setting

• The experiment compares 128B-token target-language self-repetition with a 64B-target plus 64B-auxiliary mixture, evaluating validation loss in the target language.Synergy gain is defined as the difference between the mixed-training and self-repetition validation losses; positive values indicate improvement from mixing.Source: §4.1 · p. 6

4.2. Bilingual Mixture Effects

• Six of seven languages show positive synergy across auxiliary-language combinations, with Java-C# yielding the largest gain (Δ = 0.186).Table 1 reports the synergy gain matrix.Source: §4.2 · p. 6, Table 1 · p. 7
• Java-C# reaches validation loss 0.718 versus 0.903 for Java self-repetition, a 20.5% improvement.JavaScript-TypeScript mixing also beats self-repetition: 0.517 vs 0.542; the reverse target-language pairing gives 0.517 vs 0.535.Source: §4.2 · p. 7
• Python as a target has small negative synergy with most auxiliary languages, while Python as an auxiliary language benefits targets such as Java (Δ = 0.054).The negative effects include JavaScript (Δ = −0.009) and Rust (Δ = −0.021); Java is the exception, with Python-target synergy Δ = 0.010.Source: §4.2 · p. 7
• With limited tokens, the authors recommend mixing syntactically related languages rather than simply upsampling one language; the optimal mixing ratio remains open.The experiments use 50:50 mixtures, while asymmetric ratios are left for investigation.Source: §4.2 · p. 7

5.1. Experimental Setting

• The study trains on 900B algorithmically equivalent code tokens across seven PLs plus 100B FineWeb-Edu tokens, then evaluates 2,100 instances over 42 translation directions.Python is the pivot language, with direct non-Python pairs absent from training.Source: §5.1 · p. 8

5.2. Pre-training on Unsupervised Multilingual Data

• Unsupervised multilingual pre-training yields emergent cross-lingual capability as model size increases, despite no direct alignment between language pairs.The reported zero-shot scaling parameters are A_z=0.1574, B_z=9.553, α_z=0.3470, β_z=0.8829, and L∞,z=0.1236.Source: §5.2 · p. 9

5.3. Pre-training on Supervised Multilingual Data

• For explicitly aligned Python–other pairs, concatenating code with its translation provides document-level alignment and supports efficient use of model capacity.The authors attribute this efficiency to the high scaling exponent α_a.Source: §5.3 · p. 9
• Document-level pairing also improves zero-shot translation between non-Python PLs, with the authors suggesting Python acts as an implicit bridge.Examples include Java↔Go and Rust↔JavaScript, although those pairs have no direct training alignments.Source: §5.3 · p. 9
• Parallel pairing yields superior fitted scaling on seen and unseen translation directions and better MultiPL-E generation than the baseline.Table 2 summarizes the scaling comparison; Table 3 evaluates multilingual code generation.Source: §5.3 · p. 10, Table 3 · p. 10, §5.4 · p. 10

5.4. Cross-Lingual Translation Strategies

• Parallel pairing improves translation on seen directions; on 30 unseen non-Python pairs, parallel-PL training also beats the baseline.The section fits power laws to seen and unseen validation loss; higher α indicates greater benefit from added model capacity, while lower L∞ indicates better asymptotic performance.Source: §5.4 · pp. 10–11, Figure 3 · p. 11
• Both prompt-based and direct concatenation outperform the baseline across model sizes, while translations from other PLs to Python score lower than other directions.Figure 4 aggregates translation scores across languages and directions into three categories.Source: Figure 4 · p. 11

6.1. Experimental Setup

• The comparison trains two 1.5B-parameter models on 400B tokens each, including 350B code tokens and 50B natural-language tokens.The baseline assigns 50B code tokens to each programming language; the optimized strategy reallocates the same code-token budget using fitted scaling laws and synergy analysis.Source: §6.1 · p. 12, Figure 5 · p. 12

6. Guideline for Code Pre-training

• The allocation guidance prioritizes high-utility languages, balances high-synergy pairs, and reduces tokens for fast-saturating languages.The recommendations also state that uniform allocation is suboptimal and that language synergy should guide corpus design.Source: §6 · p. 11, §6.1 · p. 12

6.2. Proportion-dependent Multilingual Scaling Law

• The proportion-dependent scaling law explicitly incorporates language proportions, with an effective data term capturing cross-lingual transfer.Its transfer coefficients are derived from the language-pair synergy analysis.Source: §6.2 · p. 12

6.3. Scaling Law under Optimal Allocation

• Under optimal allocation, the fitted exponents are α_D = 0.6859 and α_N = 0.2186, with L∞ = 0.2025.The coefficients come from weighted fitting of multilingual code-generation and translation loss.Source: §6.3 · p. 13

6.4. Evaluation on Multilingual Code Generation and Translation

• Guided allocation achieves higher average performance than uniform allocation under identical compute budgets, with no language suffering significant degradation.Evaluation covers MultiPL-E Pass@1 across seven programming languages and translation BLEU.Source: Figure 6 · p. 13, §6.4 · p. 13

Scaling Laws

• Prior code-scaling work [19] found code more data-hungry than natural language but analyzed only a single programming language.Source: §7 · p. 13
• Scaling-law research established predictable performance relationships with model size, data, and compute [16], while Chinchilla work [11] argued for compute-optimal scaling of parameters and training tokens.Source: §7 · p. 13

Code Pre-training

• Code pre-training progressed from CodeBERT [6] and GraphCodeBERT [7] to encoder-decoder models such as CodeT5 [26] and CodeGen [20], then multilingual foundation models.Source: §7 · p. 14

• Proportion-dependent scaling laws enable token allocation that improves average performance without degrading individual language capabilities.Source: §8 · p. 14
• Interpreted languages such as Python show larger scaling exponents than compiled languages such as Rust, with irreducible losses ordered from C# to Python.Source: §8 · p. 14
• Strategic language pairing yields synergistic benefits, particularly for syntactically similar languages such as Java-C#, and most languages benefit from multilingual exposure.Source: §8 · p. 14
• Parallel pairing of cross-lingual data significantly enhances translation capabilities and has favorable scaling properties.Source: §8 · p. 14

• Experiments cover seven programming languages, leaving low-resource and domain-specific languages such as SQL and assembly unexplored.Source: Limitations · p. 14
• The largest model is 14B parameters with 1T tokens, so scaling exponents at the greater-than-100B, multi-trillion-token frontier remain unvalidated.Source: Limitations · p. 14
• Evaluation covers code translation and generation benchmarks, which may not capture tasks such as program repair or multi-file completion.Source: Limitations · p. 14
• Synergy coefficients are fitted to one corpus, and the fixed-token-budget optimization does not explore dynamic curricula or adaptive sampling.Source: Limitations · p. 14