Competition-Level Code Generation with AlphaCode

• Competitive programming tests code generation beyond short snippets by requiring complex-language understanding, algorithmic reasoning, extensive implementation, and correctness on hidden edge cases.These newly created, battle-tested problems provide a meaningful benchmark that limits shortcuts and measures multiple aspects of intelligence.Source: §1 · p. 2
• AlphaCode generates many programs for each unseen problem, filters them using example-test execution results, and clusters the survivors into a small submission set.The system uses large transformers pretrained on selected GitHub code and fine-tuned on a curated competitive-programming dataset.Source: §1 · p. 3
• 34.2% of held-out CodeContests problems were solved using at most 10 submissions per problem, versus previously reported solve rates of around 1–5%.CodeContests reduced false-positive rates to 4%, compared with 30–60% in existing datasets.Source: §1 · p. 3
• AlphaCode ranked in the top 54.3% across 10 simulated Codeforces contests with over 5,000 participants each, achieving an estimated rating of 1238.The rating was within the top 28% of users who had participated in a contest during the previous six months.Source: §1 · p. 3, Figure 1 · p. 3
• AlphaCode’s analysis found that it creates original solutions rather than duplicating training code, relies heavily on natural-language descriptions, and cannot be assessed reliably by validation loss alone.The analysis also examined the problem types the model can and cannot solve.Source: §1 · p. 4

2.1. Competitive programming

• Competitive programming tasks require parsing natural-language specifications, deriving efficient algorithms, and implementing precise code that passes exhaustive hidden tests within time limits.Competitors typically solve several problems under time limits, with penalties for incorrect submissions and solution time.Source: §2.1 · pp. 5–6

2.2. Evaluation

• The n@k metric reports the percentage of problems solved when a system generates k samples and submits at most n≤k filtered samples against hidden tests.Filtering must use only information available to competitors, such as example tests, and a problem is solved if any submitted sample passes all tests.Source: §2.2 · p. 6
• Bootstrapped expectations over a larger sample pool reduce variance when estimating n@k, making comparisons more meaningful for small validation and test sets.The evaluation uses K≫k samples to estimate the metric while preserving the specified k-sample budget.Source: §2.2 · p. 6
• The competition setting uses 10@k, while pass@k equals k@k and serves as an upper-bound solve-rate metric when all k samples can be evaluated.Performance is reported across different k values because success at low sample budgets need not predict performance at high budgets.Source: §2.2 · p. 6

3. Datasets

• GitHub pre-training helps models represent and generate code fluently, while competitive-programming fine-tuning adapts them to the target domain and is critical for solving such problems.GitHub-pre-trained models solve simple programming problems but solve very few competitive-programming problems without dedicated fine-tuning.Source: §3 · p. 7, §3.2 · p. 7

3.1. Pre-training dataset

• The pre-training dataset contains 715.1 GB of filtered code from public GitHub repositories across 12 programming languages.Files larger than 1MB, files with lines longer than 1000 characters, and whitespace-insensitive duplicates were removed.Source: §3.1 · p. 7

3.2. CodeContests fine-tuning dataset

• CodeContests combines newly scraped Codeforces problems with Description2Code and CodeNet, using temporally separated training, validation, and test data to prevent leakage.Training data was released by 2021/07/14, validation problems appeared from 2021/07/15 to 2021/09/20, and test problems after 2021/09/21.Source: §3.2 · p. 7
• CodeContests includes problem descriptions, metadata, correct and incorrect submissions in C++, Python, and Java, plus accessible example and hidden test cases.Difficulty ratings and tags were collected but are unavailable at competition time and should not be used during testing.Source: §3.2 · p. 7

3.2.1. False positives and additional generated tests

• Additional tests were generated by mutating inputs and retaining mutations that 30 correct solutions produced identical outputs on.Generation ran for up to 10 CPU hours or 200 tests per problem, but complex formats prevented the full 200 tests for 6.3% of problems.Source: §3.2.1 · p. 8
• False positive rates fell from 62% to 4% after generated tests and filtering, although some slow but semantically correct solutions remained accepted.Problems were retained only with at least 5 hidden or generated tests producing at least 2 different outputs.Source: §3.2.1 · p. 9

4. Approach

• AlphaCode pre-trains on GitHub code, fine-tunes on competitive-programming data, generates many solutions, and filters them by tests and behavior into at most 10 submissions.This pipeline addresses sparse rewards, limited task data, and the restricted submission budget; large-scale sampling followed by filtering greatly improves problem solve rate.Source: §4 · p. 9

4.1. Model architecture

• AlphaCode treats problem solving as sequence-to-sequence translation with an encoder-decoder transformer that models p(Y|X) and autoregressively generates code from tokenized problem descriptions.The asymmetric design uses 1536 encoder tokens and 768 decoder tokens, while multi-query attention reduces sampling memory and cache-update costs.Source: §4.1 · p. 10

4.2. Pre-training

• Pre-training uses decoder next-token prediction and encoder masked language modeling on GitHub files, with masked language modeling essential for improving encoder representations.The largest 41B model was stopped early and was relatively undertrained compared with models at other scales.Source: §4.2 · p. 11

4.3. Fine-tuning

• Fine-tuning on CodeContests combines next-token and masked-language-modeling losses with tempering, value conditioning and prediction, metadata conditioning, and GOLD (Pang and He, 2020).Tempering with T = 0.2 reduces overfitting and smooths inference, while GOLD addresses the mismatch between many available solutions and the single-correct-solution submission metric.Source: §4.3 · pp. 11–12

4.4. Large scale sampling

• Millions of samples are generated per problem using parallel sampling, language and metadata variation, and relatively high temperature; regular temperature sampling outperformed neither top-k nor nucleus sampling.The system uses T′ = 0.25 with tempering and GOLD and T′ = 0.12 with tempering only, while optimal temperature generally rises with sample count.Source: §4.4 · p. 12

4.5. Filtering

• Example-test filtering removes approximately 99% of samples, but leaves tens of thousands of candidates for many problems and finds no passing program on approximately 10% of problems.The formulation still permits only 10 submissions per problem, preserving the contest penalty setting.Source: §4.5 · p. 13

4.6. Clustering

• Behavioral clustering uses generated test inputs to group semantically similar programs, and selecting one solution per cluster from largest to smallest performed best.The test-input model is initialized from the GitHub checkpoint, and imperfect or invalid generated tests can still help group sampled programs.Source: §4.6 · p. 13

5.1. Codeforces competitions evaluation

• 54.3%: AlphaCode ranked in the top 54.3% across 10 Codeforces competitions, improving to 48.8% with more than 10 submissions per problem.With 10 submissions per problem, this corresponded to an estimated rating of 1238, within the top 28% of recent contest participants.Source: Table 4 · p. 14, §5.1 · pp. 14–15

5.2. CodeContests evaluation

• 34.2% of validation problems were solved with one million samples, while 100,000 samples solved 31.8% of validation and 29.6% of test problems.The 41B model consistently outperformed the 9B model, and clustering consistently improved results.Source: §5.2 · p. 15, Table 5 · p. 15

5.3.1. Solve rates scale with respect to parameter count, compute, number of samples, and dataset size

• Solve rates scaled approximately log-linearly with samples and compute, while larger models achieved higher scaling slopes and reduced the samples needed for a target solve rate.Increasing samples continued improving 10@k performance despite only 10 submissions being allowed, but the required sample budget grew exponentially and became costly.Source: §5.3.1 · pp. 16–17, Figure 6 · p. 16, Figure 7 · p. 17

5.3.2. Architecture changes to improve sampling speed

• The encoder-decoder architecture with multi-query attention significantly increased sampling speed while maintaining sample quality relative to more expensive alternatives.These changes were motivated by the importance of drawing more samples within a fixed compute budget.Source: §5.3.2 · p. 17, Table 6 · p. 17

5.3.3. Choice of the pre-training dataset

• Pre-training on the full multilingual GitHub dataset significantly outperformed Python-only or MassiveText pre-training, while any pre-training beat training from scratch.The compared models were fine-tuned and sampled under matched procedures, except the Python-only model used Python-only fine-tuning and sampling.Source: §5.3.3 · p. 18, Table 7 · p. 18

5.3.4. Model enhancements

• Combining five AlphaCode enhancements increased the 10@100k solve rate from 15.2% to 24.1%, with each added setting improving performance.The contribution of the enhancements depended on the number of samples.Source: §5.3.4 · p. 18, Table 8 · p. 18

5.3.5. Filtering & clustering

• Example-test filtering removed more than 99% of samples, while filtering plus clustering enabled solve-rate scaling but remained below perfect sample selection.The largest 41B models generated samples passing example tests for over 90% of problems with 10^6 samples per problem.Source: §5.3.5 · p. 19, Figure 8 · p. 20

5.4. Results on APPS

• On APPS, a 1B AlphaCode model outperformed GPT-NEO at every difficulty level and Codex 12B on interview and competition problems, with performance improving as samples increased.Comparisons used fine-tuned models under reduced settings because required dataset information was missing.Source: §5.4 · p. 20

6. AlphaCode’s capabilities & limitations

• The analyses indicate that AlphaCode does not exploit obvious task weaknesses: it avoids copying training solutions and responds to changes in problem descriptions and conditioning metadata.Results are reported without clustering unless otherwise noted.Source: §6 · p. 20

6.1. Copying from training data

• AlphaCode does not appear to copy long blocks or core logic from training data; shared substrings occur at similar rates to human solutions and mostly contain input-parsing boilerplate.Model solutions have a slightly higher average longest common substring, while human solutions have a heavier tail.Source: §6.1 · p. 21, Figure 9 · p. 22

6.2. Model solution characteristics

• AlphaCode produces mostly syntactically correct Python, finds C++ syntax harder, and generates approximately the same amount of dead code as human solutions.Dead-code analysis removes unused imports, functions, and classes from correct Python solutions.Source: §6.2 · p. 22
• Larger models improve solve rates across problem types, performing relatively better on bitmasks, sorting, maths, and greedy algorithms but worse on dynamic programming and constructive algorithms.Models have significantly higher overall solve rates for easier problems and lower rates for harder problems.Source: §6.2 · pp. 22, 24, Table 11 · p. 22

6.3. Sensitivity to problem descriptions

• Meaningful description changes substantially affect solve rates, whereas synonyms and other apparently insignificant edits have limited effects; better models respond more to subtle description degradation.The model relies particularly on the specification, which describes how to read the input.Source: §6.3 · p. 24, Table 12

6.4. Sensitivity to provided metadata

• Random metadata conditioning improves exploration: tags alter generated algorithms, and random tags per sample outperform fixed random tags, while conditioning on CORRECT is preferred.For Gregor and Cryptography, the number-theory tag produced 29 solutions versus 9 with the brute-force tag, and loop-free solutions 12 versus 3.Source: §6.4 · pp. 25–26, Table 13 · p. 26

6.5. Loss is a poor proxy for solve rate

• Validation loss is a poor proxy for solve rate: it begins increasing after about 50k steps while solve rate continues improving, and its relationship to solve rate remains unresolved.Solve rate is the ultimate metric but has high variance and computational cost, complicating training-step decisions.Source: §6.5 · pp. 26–27, Figure 13 · p. 27

7.1. Program synthesis

• Program synthesis traditionally searches language-defined program spaces under constraints, while later approaches use input/output examples, sketches, and learned models to guide or generate programs.Deductive synthesis uses theorem proving, FlashFill uses input/output specifications, and sketch-based methods reduce search spaces with program skeletons; learned systems generate sketches, code, or policies.Source: §7.1 · p. 27
• Scaling beyond short completions remains difficult because larger programs increase both the depth and width of the search problem.Earlier synthesis and code-completion systems commonly restricted outputs to single lines, short programs, domain-specific languages, or brief snippets.Source: §7.1 · p. 28

7.2. Transformers for program synthesis

• Transformers advanced program synthesis, but prior systems mainly addressed simpler function completion, whereas AlphaCode targets full solutions to held-out competitive-programming problems.Codex achieved high success on hand-specified Python functions and supported interactive synthesis and autocomplete, but competitive-programming tasks have longer descriptions and solutions and omit suggested implementations.Source: §7.2 · p. 28

7.3. Scaling sampling

• Repeated sampling increases the chance of finding correct solutions, while majority voting and verifiers can select or rerank a small set of samples.These approaches were demonstrated by Chen et al. (2021), Austin et al. (2021), and Cobbe et al. (2021) on a much smaller scale than AlphaCode’s sampling and filtering.Source: §7.3 · p. 28

7.4. Evaluation metrics

• Functional execution-based evaluation better measures program correctness than token, syntax-tree, or reference-code matching, but limited test coverage can mark incorrect programs as correct.This limitation affects benchmarks including Chen et al. (2021) and Hendrycks et al. (2021), motivating stronger evaluation for programming problems.Source: §7.4 · p. 29

7.5. Competitive programming

• Competitive-programming datasets progressed from small collections to APPS and multilingual corpora, but prior evaluations reported near-zero solve rates and suffered from nonrepresentative formats or false positives.AlphaCode’s dataset builds on Caballero et al. (2016) and Puri et al. (2021), adding scraping from Codeforces; APPS contains 10,000 coding-competition problems.Source: §7.5 · p. 29

8.1. Applications

• Human-readable code generation could improve developer productivity, programming accessibility, and education, although most real-world applications require future work.Potential uses include extended completions, code optimization, higher-level programming, idiomatic alternatives, code documentation, and natural-language software creation.Source: §8.1 · pp. 29–30
• Code generation may help bad actors create evasive malware, increase malicious coding productivity, and gain unfair advantages in competitions or technical interviews.These risks arise alongside the broader possibility that improved developer tools can also improve threat-defender capabilities.Source: §8.1 · p. 30

8.2. Potential risks and benefits

• Human-readable outputs are easier to analyze for correctness and bias, while passing sufficient tests may improve generalisation to out-of-distribution inputs.These properties could make code generation safer, fairer, and more reliable in real-world environments.Source: §8.2 · p. 30
• Training-data deficiencies can reproduce bias, outdated APIs, bugs, and security weaknesses, while training and extensive sampling require hundreds of petaFLOPS days.Code generation also raises intellectual-property concerns about training data, attribution, and use of people’s code; AlphaCode partly addresses licensing through dataset filtering.Source: §8.2 · pp. 30–31
• Automation could reduce demand for programmers despite shifting work toward higher abstraction, and advanced coding systems could recursively improve themselves and accelerate advanced AI risks.Once synthesized, programs can generally run cheaply on ordinary computers, potentially enabling broad deployment despite high training and sampling costs.Source: §8.2 · p. 31

9. Conclusion

• AlphaCode performs roughly at the level of the median Codeforces competitor while generating novel solutions to unseen programming problems.The analysis found no evidence that AlphaCode copies important parts of prior solutions or exploits weaknesses in problem structure.Source: §9 · p. 31
• Large-scale sampling followed by filtering and clustering, sampling-efficient transformer architectures, a clean dataset, and robust evaluation were essential to AlphaCode’s performance.The authors also present model probings and discuss broader impacts and hazards of code-generation models.Source: §9 · p. 31

A.2. Program judging

• Program judging executes submissions on test cases and compares outputs with expected outputs, while accounting for problem-specific formats, multiple correct outputs, interactive inputs, and resource limits.The same judging process is used for final submissions and for filtering based on example tests.Source: §A.2 · pp. 38–40

A.1. Hidden tests

• Hidden test cases used to evaluate correctness are substantially longer and more demanding than the example tests visible to participants and used within AlphaCode.These held-out tests verify solutions against cases not exposed in the problem statement.Source: §A.1 · p. 38, §A.2 · p. 39

A.2. Program judging

• The judging implementation permissively treats numerical differences below 10^-5, case differences, and whitespace differences as equivalent, without finding false positives from these choices in dataset checks.This permissive behavior does not exactly match all problem-specified formats.Source: §A.2 · p. 39
• About 1/4 of validation problems are classified as multiple-output problems and judged against a majority human-solution output, which can underestimate model performance.Accepting any output produced by a human solution would reduce false negatives but substantially increase false positives.Source: §A.2 · p. 39
• Interactive problems are not explicitly handled, potentially causing false negatives and false positives, although no interactive false positives were found.Interactive problems are substantially rarer than multiple-output problems.Source: §A.2 · pp. 39–40

A.3. Evaluation metrics

• n@k measures the fraction of problems solved when generating k samples but submitting only n for evaluation.The metric evaluates model performance under a sampling-and-submission constraint.Source: §A.3 · p. 40
• For a single model, n@k is estimated by averaging solve rates across S subsamples of size k drawn without replacement from K≥k samples.The same K samples are reused to calculate n@k across different k values for scaling curves.Source: §A.3 · p. 40
• 95% confidence intervals for ablation results are obtained by bootstrap-resampling trained models and their samples, using the 2.5th and 97.5th percentiles.For Table 8, at least three models were fine-tuned per setting and their n@k results were averaged to address model-to-model variance.Source: §A.3 · p. 40
• Clustering results use five subsamples per k and model, with means averaged across models; 9B and 41B results use single models because multiple-model training is costly.Clustering results in Table 8 and Figure 8 use five models fine-tuned in the "+ GOLD" setting.Source: §A.3 · pp. 40–41

B.1. GitHub dataset composition

• 715GB of GitHub code spanning multiple programming languages forms the pre-training dataset, with its exact composition reported in Table A1.The corpus contains 86.31 million files and 715.1GB of data.Source: §B.1 · p. 41, §B.2 · p. 42

B.2. Dataset cleaning

• The dataset-cleaning pipeline removes duplicate problems and submissions, then normalizes C++ by compiling, formatting, standardizing includes, and expanding preprocessor directives and typedefs.When cleaning fails for part of the dataset, the original solution is retained.Source: §B.2 · pp. 41–42
• Execution-based filtering removes submissions that pass no tests, tests solved by fewer than 10% of remaining submissions, and submissions passing fewer than 10% of remaining tests.The procedure is applied to all Python and C++ solutions on all test cases.Source: §B.2 · p. 42

B.3. Data leakage and temporal split

• Training–evaluation leakage is especially problematic in code because duplicated solutions are common and competition participants often publish solutions online.Models can sometimes copy training data verbatim, but using information unavailable at competition time would constitute cheating.Source: §B.3 · p. 42
• A strict temporal split ensures training data contains only information available to a typical human competition participant.The split is applied to guard against leakage from duplicated code and solutions published online after competitions.Source: §B.3 · p. 42
• 0% solve rate with a temporal split shows that models must create novel solutions rather than simply remember the training set.A training-solution baseline reached 4.1% with a random split but 0% temporally; a 1B GitHub-pretrained model reached 0.8% randomly but 0% temporally.Source: §B.3 · p. 42
• Temporal-split validity was checked separately for both pre-training and fine-tuning using solve rates on CodeContests validation problems.The random-split comparisons were designed to test whether evaluation performance could arise from overlap with training data.Source: §B.3 · p. 42

C. Approach and Results

• The ensemble analysis distinguishes cases where the ensemble is better than all components from cases where strong components outperform the ensemble.These two cases summarize the dependence of ensemble performance on component quality.Source: §B.3 · p. 43
• Good ensembles can outperform individual components, whereas bad ensembles can perform worse than an individual component.The appendix figure presents both cases as distinct ensemble-performance outcomes.Source: §B.3 · p. 43

C.1. Ensembling

• Ensembling pools model samples before filtering and clustering, helping when component performances are similar but typically hurting when one run is much better.The better-run dominance in the latter case can waste the sample budget, whereas similar components cover the search space more effectively.Source: §C.1 · p. 43
• The 41B+9B ensemble improved validation performance slightly but was not better than the 41B model alone on the test set.With 1 million samples per problem, it achieved 32% solve rate with 10 submissions and 35.5% with clustering on validation; the test-set regression was slight.Source: §C.1 · pp. 43–44

C.2. Metadata conditioning

• Metadata conditioning improved solve rates by sampling ratings from 800 to 3500, tag combinations, and languages rather than using unavailable contest metadata.Ratings were sampled in increments of 100, tags from the 50 most popular combinations, and language uniformly between C++ and Python to diversify approaches.Source: §C.2 · p. 44
• Chosen metadata values were prepended to the natural-language prompt during sampling.This provides the model with the sampled rating, tags, and language conditions used to diversify generated programs.Source: §C.2 · p. 44

C.3. GOLD

• GOLD modifies maximum-likelihood training with an off-policy importance weight so the model emphasizes precision and the chance of generating at least one correct sample.The objective learns from tokens already assigned high likelihood while ignoring tokens outside the model’s distribution, shifting focus away from modeling every training solution.Source: §C.3 · p. 44
• Combining GOLD with tempering creates a distribution-selection problem: the non-tempered distribution becomes too smooth, whereas the tempered distribution is initially too sharp.These dynamics can respectively remove GOLD’s selection benefits or cause overly strong selection at the start of fine-tuning.Source: §C.3 · p. 45
• A short intermediate tempering phase without GOLD smooths the pretrained distribution before fine-tuning uses the tempered distribution for both loss and importance weighting.During fine-tuning, logits are divided by the temperature before computing the loss.Source: §C.3 · p. 45

C.4. Additional results for filtering and clustering

• Just over 1/3 of validation problems have p_pass example test significantly above zero, and the distribution varies substantially across problems and model sizes.Figure A3 sorts problems by the 41B model’s p_pass example test.Source: §C.4 · p. 45
• Clustering used 50 test inputs and 8192 model samples because invalid or nondiscriminating tests can create ambiguous clusters or split correct solutions across clusters.Increasing either clustering hyperparameter did not further improve performance.Source: §C.4 · p. 45

C.5. HumanEval comparison

• Our decoder-only baselines achieve HumanEval solve rates within about 1–3% of comparable Codex models and outperform GPT-Neo and GPT-J at all comparable settings.Encoder-decoder models were significantly worse on HumanEval and are therefore not reported in Table A3.Source: §C.5 · p. 46, §C.4 · p. 46
• The HumanEval gap likely reflects task alignment: encoder-decoder models suit contest descriptions with outputs and example tests, whereas decoder-only models learn more efficiently from GitHub code.Decoder-only models compute loss on all tokens, while encoder-decoder models compute loss on only one-third of tokens in this setting.Source: §C.5 · p. 46

C.6. APPS dataset settings

• APPS example tests were parsed with the Figure A4 procedure, which failed for less than 2% of test problems.CodeContests was excluded from APPS training because the datasets have a non-empty intersection; problems were assumed to provide only available examples.Source: §C.6 · pp. 46–47

C.7. Best settings for sampling

• A sampling temperature of T=0.25 performs well across a wide range of sample budgets, while top-k and nucleus sampling do not outperform simple temperature sampling.Sampling temperature affects solve rate, making sampling settings important when generating ≥1M samples per problem.Source: §C.7 · p. 47

C.8. Scaling with dataset size

• Increasing dataset size improves model performance, with adding more problems having a larger positive impact than adding more solutions per problem.This scaling result extends earlier analyses of model size, compute, and sample count.Source: §C.8 · p. 47

E. Additional analysis of AlphaCode’s capabilities and limitations

• Sampling analysis found lower temperatures better for small sample budgets, higher temperatures better for large budgets, and performance tolerant across a wide temperature range.Top-k sampling did not outperform temperature sampling, while nucleus sampling did not significantly improve the baseline and worked best near nucleus size 1.0.Source: D. Codeforces contest evaluation · p. 48

E.1. Model sample statistics

• Increasing either the number of finetuning problems or solutions improved solve rate for an early 1B-parameter model, with a smaller effect from increasing solutions.Table A7 separately reports the percentage of syntactically correct C++ and Python samples.Source: D. Codeforces contest evaluation · p. 49, §E.1 · p. 50

D. Codeforces contest evaluation

• AlphaCode’s evaluation procedure computes Codeforces placements using contest scoring, including solve timing and incorrect-submission penalties.Per-contest scores, official-scoreboard placements, and overall results are reported in Tables A5 and A6.Source: §D.1 · p. 49, §D.3 · p. 50

D.1. Simulation

• Simulated contests used 3,750 TPUv4 and 3,750 TPUv4i chips to continuously sample, evaluate example tests, cluster outputs, and submit up to 10 samples per problem at scheduled conditions.Submission timing depended on remaining contest time or the relative number of samples passing example tests, with 120 seconds added for clustering.Source: §D.1 · p. 49, D. Codeforces contest evaluation · p. 49
• The simulation omitted the hacking phase and replaced multiple-output example answers with consensus outputs from human solutions, affecting approximately five problems AlphaCode solved.The authors considered the hacking omission minor because AlphaCode did not hack other submissions, but the consensus-output change was a more consequential simulation artifact.Source: §D.1 · p. 50

D.2. Multiple evaluations

• Repeated evaluations measured variance from sample ordering and clustering while keeping the trained models and sample sets fixed.Samples were reshuffled, clustering used different generated inputs, and cluster selection used a different random seed; retraining and resampling were avoided because of compute limitations.Source: §D.2 · p. 50

D.3. Results

• The evaluation found that some problems were solved only when more than 10 submissions per problem were allowed in the second and third evaluations.The reported results include contest placements with penalties and an estimated Codeforces Elo computed separately for all three evaluations.Source: §D.3 · p. 50
• The estimated Codeforces Elo largely matched reproduced participant scores, differing by less than 15 points, but remains an estimate because AlphaCode did not compete live.The estimate reproduces the Codeforces Elo method and follows the contests in release order.Source: §D.3 · p. 50

E.2. Solve rate for different problem difficulty ratings

• The models perform significantly better on low-difficulty problems, with performance quickly degrading as difficulty rises; high solve rates in high-difficulty buckets reflect a 46% slow-positive rate and small buckets.The validation and test sets contain 117 and 165 problems, respectively.Source: §E.2 · pp. 50, 52

E.3. Sensitivity to the problem descriptions

• Overall, the problem description is critical to AlphaCode’s performance, and the models do not simply brute force possible solutions.Most analyses use the 10@1024 solve rate and sample already-trained models because retraining on modified descriptions was computationally infeasible.Source: §E.3 · p. 52

E.3.1. Simplification of problem descriptions

• Simplifying descriptions to make the required algorithm more explicit significantly improves performance, increasing the 1B model’s solve rate on one problem from 12% to 55%.The analysis covered five problems: three from the validation set and two written by the authors.Source: §E.3.1 · p. 53

E.3.2. Incorrect or irrelevant rewordings

• Rewording experiments show that AlphaCode strongly conditions on descriptions rather than brute forcing solutions, parsing algorithm descriptions from symbols or natural language and ignoring irrelevant explanations.The model performed better with more language-heavy descriptions, possibly because the training distribution is verbose.Source: §E.3.2 · p. 54

E.3.3. Capturing variables and their relations

• Variable-renaming tests examine consistent and inconsistent substitutions across up to six variables, with inconsistent renaming making problems ill-posed.Variables are replaced by other existing names while preserving character case.Source: §E.3.3 · p. 54
• As model size increases, performance drops more on ill-formed inputs while sensitivity to consistent renaming decreases, indicating improved capture of variable relationships.The 300M model is largely unaffected by both consistent and inconsistent renaming, whereas larger models show increasingly pronounced drops on ill-formed inputs.Source: §E.3.3 · pp. 54–55
• Non-trivial solve rates on ill-posed problems indicate that models can leverage other cues in the problem description even when variable relations are disrupted.This finding suggests that information beyond explicit variable relationships contributes to solving modified problems.Source: §E.3.3 · p. 55

Appendix Figure A10 | Solve rates under different word-level change regimes

• Solve rates deteriorate roughly linearly with introduced typos, while stronger word permutations or deletions reduce performance; low noise remains relatively robust at p = 0.05 and N = 2.Typos swap adjacent characters, permutations reorder nearby words, and deletions remove words probabilistically.Source: §E.3.4 · pp. 55–56

E.3.4. Sensitivity to word-level changes

• Solve rates show little degradation under synonym substitutions and no significant change when type-specific words are replaced with generic terms.The substitutions replace integer, array, and string with number, sequence, and sequence of characters.Source: §E.3.4 · p. 55

E.3.5. Description section ablations

• Removing any task-description section hurts performance, with IO examples least consequential, the description intermediate, and the specification most important; section reordering has only a small effect.Without the specification, parsing the problem input becomes difficult.Source: §E.3.5 · p. 56

E.4.1. Problem ratings

• Conditioning on an easier problem rating generally improves solve rate, whereas a uniformly random rating is competitive with any fixed rating.The study examines how supplied ratings alter generated samples and solve rates.Source: §E.4.1 · p. 56

E.4. Sensitivity to problem metadata

• Variable renaming is robust when invariant but degrades with inconsistent renaming, especially less for larger models.Inconsistent renaming makes descriptions ill-posed by replacing occurrences independently.Source: §E.4.1 · p. 57

E.4.1. Problem ratings

• For harder validation problems, conditioning on a harder rating is relatively better; for easier problems, harder ratings produce a larger negative impact.The comparison uses four validation-set difficulty quartiles and different conditioning ratings.Source: §E.4.1 · p. 57

E.4.2. Solution correctness

• Incorrect-solution conditioning hurts 10@k but not pass@k, while removing correctness conditioning hurts both metrics, though not significantly.The pass@k pattern suggests more samples may pass visible examples without passing hidden tests.Source: §E.4.2 · p. 57

F. Complete prompt and model examples

• Appendix Figures A14 and A15 present complete model C++ and Python solutions for Codeforces problems, with randomly sampled tags, ratings, and languages.The examples include both the full problem prompt and a correct model solution.Source: F. Complete prompt and model examples · pp. 60–61

F.1.1. Solution decompositions

• Figures A16–A19 decompose human and model solutions for ‘Digits Sum’ and ‘Pizzaforces’ into substrings from the finetuning dataset.Color repetition is not meaningful, and matching colors do not imply relationships between human and model substrings.Source: §F.1.1 · pp. 62–65

F.1.2. Very long common subsequences between human solutions and finetuning data

• Human validation solutions for ‘The Miracle and the Sleeper’ and ‘Integers Have Friends’ contain very long dataset common subsequences of lengths 914 and 666, respectively.The remaining solution portions are composed of much smaller substrings.Source: §F.1.2 · pp. 66–67

F.2.1. Simplified rewordings

• The simplified-rewording examples replace detailed competitive-programming statements with concise task formulations while preserving the underlying computational objective.Examples cover Nim winner determination, counting binary strings without consecutive zeros, Fibonacci computation, array products, and bitwise operations.Source: §F.2.1 · pp. 68–69
• The simplified examples retain concrete test-case structure and constraints, such as multiple cases, sequence lengths, element bounds, and required output formats.These details are explicitly shown across the Nim, binary-string, and Mocha and Math examples.Source: §F.2.1 · p. 68
• The rewordings also express interval and adjacent-element optimization tasks directly, including maximizing max-min products and neighboring-value products.The supplied formulations retain input constraints and requested outputs for these tasks.Source: §F.2.1 · p. 69
• For Mocha and Math, the original interval bitwise-AND operation is simplified to computing the AND of all sequence elements.The original asks to minimize the sequence maximum through repeated mirrored interval operations, whereas the simplified version directly requests the global bitwise AND.Source: §F.2.1 · p. 70
• The simplified prompt asks for the bitwise AND of all n integers in each test case.Each input value satisfies 0 <= a_i <= 10^9, and the required output is one integer per test case.Source: §F.2.1 · p. 70
• The original Balanced Substring task asks for any non-empty substring containing equal numbers of ’a’ and ’b’, or -1 -1 if none exists.The string length is at most 50, and each test case outputs the substring’s endpoints when one exists.Source: §F.2.1 · p. 71
• The balanced-substring examples state that one testcase has no valid substring, while others contain multiple valid choices, including "abbaba" and "baba".These examples illustrate that any qualifying interval may be returned.Source: §F.2.1 · p. 71
• The simplified version reduces Balanced Substring to finding adjacent unequal letters and printing their indexes, or -1 -1 when no such pair exists.The examples distinguish test cases with no unequal adjacent pair from those containing such pairs.Source: §F.2.1 · p. 71

F.2.2. Incorrect and verbose rewordings

• Some rewordings incorrectly change the adjacent-product objective from maximization to minimization or from adjacent pairs to arbitrary pairs.The corresponding variants explicitly request the minimum adjacent product or the maximum product over all l < r.Source: §F.2.2 · pp. 72–73
• The incorrect arbitrary-pair variant defines f(l, r) = (a_l . a_r) and illustrates a maximum of 12 for the first test case and 9 for the second.Its examples evaluate nonadjacent as well as adjacent pairs.Source: §F.2.2 · p. 73
• The intended array task is to maximize a_l times a_{l + 1} over adjacent positions, with n between 2 and 10^5 and total n at most 3 . 10^5.Each test case contains n integers in the range 1 <= a_i <= 10^6.Source: §F.2.2 · pp. 72, 74
• The task is to find the maximum product of two consecutive array members, with the shown test cases producing maxima of 12 (first) and 6 (second).Source: §F.2.2 · p. 74