PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating Large Language Models Trained on Code

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, Wojciech Zaremba

arXiv:2107.03374v2cs.LG

TL;DR

Can large language models generate functionally correct Python from natural-language docstrings? The paper fine-tunes GPT on GitHub code and evaluates Codex on HumanEval, finding strong performance that improves with multiple samples.

  • Problem

    The paper asks whether large language models can produce functionally correct Python functions from natural-language docstrings.

  • Method

    The authors fine-tune GPT on GitHub code and evaluate standalone Python functions on 164 hand-written HumanEval problems using unit tests.

  • Results

    28.8% of HumanEval problems are solved by 12B Codex with one sample, rising to 77.5% when multiple samples are generated.

  • Takeaways & Limitations

    Fine-tuning on code and producing multiple samples both improve functional code synthesis from docstrings.

  • Takeaways & Limitations

    Codex struggles with binding operations to variables when docstrings contain many operations and variables.

Abstract

from arXiv · show

We introduce Codex, a GPT language model fine-tuned on publicly available code from GitHub, and study its Python code-writing capabilities. A distinct production version of Codex powers GitHub Copilot. On HumanEval, a new evaluation set we release to measure functional correctness for synthesizing programs from docstrings, our model solves 28.8% of the problems, while GPT-3 solves 0% and GPT-J solves 11.4%. Furthermore, we find that repeated sampling from the model is a surprisingly effective strategy for producing working solutions to difficult prompts. Using this method, we solve 70.2% of our problems with 100 samples per problem. Careful investigation of our model reveals its limitations, including difficulty with docstrings describing long chains of operations and with binding operations to variables. Finally, we discuss the potential broader impacts of deploying powerful code generation technologies, covering safety, security, and economics.

1. Introduction

The paper introduces Codex, a GPT model specialized for code, and evaluates standalone Python function synthesis from docstrings using automatically checked unit tests. Codex substantially outperforms GPT baselines, with repeated sampling further improving the chance of finding correct solutions.

  • Motivation: Codex specializes GPT for coding tasks, motivated by GPT-3’s ability to generate simple Python programs despite not being trained explicitly for code generation.The authors hypothesize that abundant publicly available code could support a specialized model that excels at coding tasks.
  • Evaluation: The evaluation generates standalone Python functions from docstrings and checks correctness automatically with unit tests on 164 original programming problems.The problems assess language comprehension, algorithms, and simple mathematics.
  • Results: 28.8%: a single-sample 12B-parameter Codex solves these problems, compared with 11.4% for 6B-parameter GPT-J and near 0% for all GPT models.A 300M-parameter Codex solves 13.2% of the same problems with one sample.
  • Repeated sampling: 77.5%: Codex-S generates at least one correct function within 100 samples, demonstrating the effectiveness of repeated sampling for function synthesis.The approach approximates iterative programming through multiple attempts and bug fixes.
  • Repeated sampling: 44.5%: selecting the sample with the highest mean log-probability yields a unit-test-passing solution, offering a heuristic alternative to fully evaluating every sample.This selection strategy may be useful when evaluating all samples is impractical in deployment.

2. Evaluation Framework

The evaluation framework uses functional correctness via pass@k rather than match-based metrics, benchmarks 164 hand-written HumanEval problems, and safely executes generated code in a sandbox. It addresses metric limitations, provides an unbiased pass@k estimator, and protects hosts from untrusted programs.

  • Functional correctness: Functional correctness counts a generated sample as correct when it passes unit tests, addressing match-based metrics’ inability to capture functionally equivalent programs.The paper also reports that functionally inequivalent programs can have higher BLEU scores than functionally equivalent ones.
  • pass@k: The pass@k metric considers a problem solved when any of k generated samples passes its unit tests.The evaluation generates n = 200 samples with k ≤100 and uses an unbiased estimator based on the number of correct samples.
  • HumanEval: 164 hand-written programming problems comprise HumanEval, with each problem containing a function signature, docstring, body, and unit tests.The problems average 7.7 tests each and assess language comprehension, reasoning, algorithms, and simple mathematics; the dataset is released publicly.
  • Sandbox environment: The sandbox safely runs untrusted generated programs against unit tests while preventing host modification, persistence, sensitive-resource access, and data exfiltration.The framework uses gVisor to introduce a security boundary between containers and hosts, with eBPF firewall rules protecting network-adjacent services.

3. Code Fine-Tuning

Codex is produced by fine-tuning GPT models on a large filtered corpus of public GitHub Python code, with performance scaling predictably with model size. HumanEval results show that sampling temperature and sample-selection strategy substantially affect functional correctness, while BLEU is an unreliable proxy.

  • Training data: 159 GB of filtered Python code came from 54 million public GitHub repositories collected in May 2020.The initial corpus contained 179 GB of unique Python files under 1 MB before filtering.
  • Model and tokenization: Whitespace-run tokens reduced code representations by approximately 30% while retaining the GPT-3 text tokenizer as the lexer foundation.Whitespace encoding was identified as the largest tokenizer inefficiency for GitHub code.
  • Scaling: The code-fine-tuned model’s test loss follows a power law, (N 5.92×107)−0.13, where N is the number of non-embedding parameters.This scaling behavior is similar to the power-law relationship observed for GPT-3 language-model test loss.
  • HumanEval evaluation: For a 679M-parameter model, optimal sampling temperatures are T∗=0.2 for pass@1 and T∗=0.8 for pass@100.Higher temperatures are favored at larger k because they increase sample diversity, and pass@k rewards any correct solution among the samples.
  • HumanEval evaluation: Choosing the sample with the highest mean token log probability outperforms random selection, whereas sum log probability can perform slightly worse than random selection.This comparison applies when multiple samples can be generated but only one can be evaluated, without access to unit tests.
  • Limitations: BLEU-score distributions overlap substantially for correct and incorrect solutions, so higher BLEU may not indicate improved functional correctness.Incorrect solutions are functionally inequivalent to their reference solutions, making BLEU an unreliable correctness measure in this setting.

4. Supervised Fine-Tuning

Codex-S adapts training to standalone-function synthesis through automatically curated programming problems and supervised fine-tuning. It improves Codex across model sizes, especially when generating many samples, while tracing-based curation remains limited by executable, serializable functions and task-quality issues.

  • Training-data construction: Codex-S uses standalone-function problems from competitive programming websites and continuous-integration repositories for additional supervised fine-tuning.The goal is to reduce the distribution mismatch between general GitHub code and HumanEval-style function synthesis.
  • Training-data construction: 10,000 problems were curated from contest and interview-preparation websites using problem statements as docstrings and tests derived from examples or incorrect submissions.These problems provide broad algorithmic coverage and hidden-test-based functional-correctness settings.
  • Training-data construction: About 40,000 problems were collected from continuous-integration traces, limited by functions lacking input-output behavior and runtime objects that could not be pickled or restored.Tracing captured inputs and outputs for invoked functions, including builtins and library calls, producing tasks focused on following docstring instructions.
  • Data quality: Curated-problem quality is difficult to control because underspecified prompts can be wrongly penalized and stateful problems can produce different outcomes across executions.The authors use Codex-12B-generated samples and repeated verification to filter ambiguous, too-difficult, stateful, or nondeterministic problems.
  • Results: 6.5 percentage points was Codex-S’s average pass@1 improvement over Codex, rising to 15.1 percentage points for pass@100 across model sizes.Codex-S uses T ∗= 0 for pass@1 and T ∗= 1 for pass@100.
  • Results: 11.6 percentage points was Codex-S-12B’s average benefit over random sample ranking when ranking 1–100 samples by mean log probability.This benefit exceeded the corresponding benefit for Codex by over 2 percentage points.

5. Docstring Generation

The section develops Codex-D to generate docstrings conditioned on code, motivated by describing the intent of generated programs for safety. Because automatic evaluation is unavailable, samples are hand-graded, revealing comparable but lower pass rates than code generation and characteristic failure modes.

  • Training: Codex-D is trained on concatenated function signatures, reference solutions, and docstrings by minimizing negative log-likelihood of the docstring.This constructs code-conditional docstring-generation examples from the training problems.
  • Evaluation: Docstring samples are hand-graded because no comparable automatic evaluation exists, with correctness requiring a unique and accurate specification of the code body.The evaluation grades 10 samples per problem across 1640 problems from Codex-D-12B.
  • Failure modes: Codex-D often omits important details or invents unrelated problems by over-conditioning on function names.Copied code bodies are also not counted as correct, and generated unit tests are ignored during grading.
  • Results: Codex-D has lower but comparable pass rates to Codex-S at the same temperature.The paper gives no strong hypothesis for which direction should yield higher pass rates, citing differing syntax strictness and possible docstring quality.
  • Sample selection: Back-translation ranking underperforms mean log-probability ranking but outperforms random ranking, while appearing to overfit quickly.The method selects samples by maximizing P(ground truth docstring|generated sample), evaluated using Codex-D.

6. Limitations

Codex has important limitations despite solving many HumanEval problems: it is costly to train and struggles with long chains of operations and binding operations to variables. These limitations inform assessments of the hazards and broader societal impacts of deploying code generation systems.

  • Training efficiency: Codex is not sample efficient to train, requiring hundreds of millions of lines of publicly available Python code from GitHub.The dataset comprises a significant fraction of publicly available Python code, exceeding the amount even seasoned developers encounter over their careers.
  • Long operation chains: As chained building blocks increase in docstrings, Codex performance decreases exponentially, unlike the expected behavior of a human programmer.The synthetic problems use deterministic string transformations assembled from 13 basic building blocks.
  • Long operation chains: With each additional chained component, Codex-12B pass rates drop by roughly a factor of 2-3.This result comes from synthetically generated docstrings containing increasing numbers of chained components.
  • Variable binding: Codex makes mistakes binding operations to variables, especially when docstrings contain many operations and variables.In one example, Codex-12B fails to decrement w and return the product of all numbers.
  • Broader impacts: These limitations inform assessment of the potential hazards and broader societal impacts of using Codex generatively.The paper connects Codex’s limited system-level synthesis capabilities to its assessment of generative deployment risks.

7. Broader Impacts and Hazard Analysis

Codex could support programming, education, and exploration, but its deployment raises hazards involving over-reliance, misalignment, harmful outputs, malware, economic effects, environmental costs, and rare training-data matches.

  • Over-reliance: Codex may produce superficially correct code that fails user intent, creating over-reliance risks especially for novice programmers and potentially worsening automation bias as capabilities improve.Reliable safeguards require empirical study across user experience levels, interface designs, and tasks.
  • Alignment: Codex can generate incorrect code despite having the capability to be more helpful, and this misalignment may persist or worsen as model size and capabilities increase.A highly capable but misaligned model could produce obfuscated code that appears acceptable while doing something undesirable or harmful.
  • Bias and harmful outputs: Codex can be prompted to generate racist, denigratory, and otherwise harmful code comments, raising bias and representation concerns.The authors identify interventions through risk mitigation as necessary responses.
  • Economic and labor-market impacts: Codex may reduce software-production costs by increasing programmer productivity, but this effect may be limited because engineers also confer, write specifications, and upgrade software stacks.The paper frames these as possible economic and labor-market impacts rather than a complete forecast.
  • Security: Codex’s non-determinism could enable more advanced malware by generating diverse software that challenges fingerprinting- and signature-based detection systems.Software diversity may sometimes aid defenders but creates distinctive challenges for traditional malware detection and antivirus systems.
  • Environmental and data-reproduction considerations: < 0.1% of studied generations appeared to match training-data code snippets, and Codex also incurs training and inference energy costs.The reported matching cases consisted of common programming-language expressions or conventions; training GPT-3-12B and fine-tuning Codex-12B each consumed hundreds of petaflop/sdays of compute.

8. Related Work

Related work spans program induction and program synthesis, including AST-based, language-model, and large-Transformer approaches. The broader coding landscape also includes functional-correctness evaluation, code datasets, unit-test generation, autocomplete, and bug fixing.

  • Program induction: Program induction generates outputs from latent program representations, with later systems incorporating inductive biases from modern computing devices.Examples include Learning to Execute, Neural Turing Machines, memory networks, and Neural GPUs.
  • Program synthesis: Program synthesis explicitly generates programs from natural-language specifications, often using probabilistic grammars to construct abstract syntax trees.Later work learned state vectors for conditioning child-node expansion and applied the approach to text-to-code retrieval and text-conditional generation.
  • Neural code generation: Code has also been synthesized without ASTs using n-gram and character-level language models, while large Transformers have extended synthesis to docstring-function translation.CodeBERT trained on paired docstrings and functions, and PyMT5 used the T5 objective for code translation.
  • Evaluation: Functional correctness and sampling-based evaluation connect this work to SPoC and TransCoder, which likewise treated functional correctness as a meaningful capability measure.SPoC addressed functionally correct code generation from pseudocode under a fixed compilation budget, similar to pass@k.
  • Broader coding tasks: Related coding applications include broader GitHub-derived datasets, unit-test generation, autocomplete, and static or dynamic approaches to locating and fixing bugs.Tufano et al. generated unit tests, Aye et al. studied autocomplete trained on accepted completions, and earlier work used code analysis for debugging.

9. Conclusion · A. Estimating pass@k

The paper finds that fine-tuned GPT models can produce functionally correct code from natural-language docstrings, while noting similar reverse-task performance and substantial limitations. For pass@k, unbiased estimation is essential because the empirical pass@1 estimator systematically underestimates performance.

  • 9. Conclusion: Fine-tuned GPT models produced functionally correct code bodies from natural-language docstrings on human-written problems comparable to easy interview questions.The paper also reports that performance improved when training data matched the evaluation distribution more closely.
  • 9. Conclusion: The reverse task of generating docstrings from code bodies was simple to train, and its models exhibited similar performance profiles.
  • 9. Conclusion: The paper discusses broader impacts and identifies significant room for improvement because code-generating models retain important limitations.
  • A. Estimating pass@k: Only the Kulal et al. (2019) empirical estimator and estimator (1) are unbiased among the previously discussed pass@k estimators.The paper emphasizes unbiased evaluation with any number of samples n for fair comparison.
  • A. Estimating pass@k: The empirical pass@1 plug-in estimate consistently underestimates pass@k, and the gap remains even when n > 5k.Results can therefore appear better with more samples despite the estimator’s bias.
  • A. Estimating pass@k: Estimator (1) is unbiased because it estimates the fail probability (1−pass@1)^k as the probability of drawing k failed samples without replacement.
  • A. Estimating pass@k: In estimator (1), c, the number of correct samples passing unit tests, follows Binom(n, p), where p is pass@1, and the estimator equals 1 when n − c < k.
  • A. Estimating pass@k: Figure 13 shows that the apparently correct estimator has considerable downward bias, whereas the unbiased estimator initially has slightly higher variance but supports fair comparisons across sample counts.

B. Random Problems and Solutions from Codex-12B · C. Building Blocks for Synthetic Tasks

Codex-12B is illustrated on random HumanEval problems through multiple sampled completions, which include both correct and incorrect solutions. Synthetic tasks are constructed by composing 13 one-line string-manipulation building blocks into docstrings and code bodies.

  • B. Random Problems and Solutions from Codex-12B: The examples expose varied failure modes, including using inappropriate string splitting, omitting required edge cases, and confusing a task’s requested operation with a different computation.Wrong completions include s.split() for mixed comma-or-space input, placeholder pass, and computing products or sums unrelated to the specified digit or palindrome conditions.
  • C. Building Blocks for Synthetic Tasks: 13 building blocks define synthetic tasks for evaluating performance as a function of docstring complexity.Each block is specified by one line of text and one line of code, covering operations such as removing letters, changing case, deleting characters, and rearranging words.
  • C. Building Blocks for Synthetic Tasks: The building blocks include string transformations such as removing vowels, dropping halves, replacing spaces, reversing words, inserting “apples,” changing alternating character case, and deleting punctuation.The listed operations span character-level, word-level, positional, spacing, insertion, and punctuation transformations.
  • C. Building Blocks for Synthetic Tasks: The blocks are composed by concatenating their one-line descriptions into a docstring and their implementations into a code body.The example combines making every other character uppercase with replacing spaces by triple spaces in a single function.

D. Details of Specification-based Evaluation Framework … E.2. How can alignment be defined and evaluated in models like Codex?

The framework evaluates code-generation difficulty through specification complexity and expressivity, while alignment analysis asks whether models produce what users want despite latent capabilities. Codex may instead reproduce confused, insecure, or biased patterns from its training distribution, motivating measurable sufficient conditions for intent misalignment.

  • D. Details of Specification-based Evaluation Framework: The framework proposes measuring natural-language specifications by abstraction level and attributes such as variable dependencies, inter-procedural reasoning, and computational interleavings.These attributes adapt expressivity and complexity measures from formal specifications to natural-language prompts.
  • D. Details of Specification-based Evaluation Framework: Higher-level specifications are more ambiguous, while lower-level specifications define architectural and programming constructs more precisely.Codex has nevertheless shown preliminary ability to solve high-level specifications, whereas existing synthesis methods mainly address tightly constrained tasks.
  • D. Details of Specification-based Evaluation Framework: The evaluation framework includes variable interdependencies, temporal reasoning, concurrency and parallelism, hyperproperties, and nondeterminism.Examples include tracking nested variable states, safety and liveness, computational interleavings, noninterference, and differing outputs across executions.
  • D. Details of Specification-based Evaluation Framework: Specification-independent coding practices include code and parameterized reuse, automatic program-architecture determination, and a wide range of programming constructs.These practices support increasingly complex and higher-level specifications without requiring every implementation construct to be stated explicitly.
  • E.1. Why evaluate alignment?: Alignment evaluation targets problems that may persist or worsen as model capability improves, even if they currently cause little harm.The authors present alignment as one category of such long-term problems.
  • E.1. Why evaluate alignment?: Codex is better characterized as continuing prompts according to its training distribution than as trying to help users.This creates ambiguity because it is unclear whether Transformer models have intent in the relevant sense.
  • E.1. Why evaluate alignment?: Codex tends to continue confused, insecure, or biased code with similarly flawed code, and may introduce such flaws even after receiving fairly good inputs.This behavior can occur despite the model’s capability to produce secure, unbiased, and high-quality code.
  • E.2. How can alignment be defined and evaluated in models like Codex?: A model is considered intent misaligned when it outputs B instead of the user-preferred A despite being capable of A and distinguishing when the user wants A or B.Capability can be established through prompt engineering, limited fine-tuning, model surgery, or success on a task Y requiring X.

E.3. Results of alignment evaluations · E.4. Areas for Further Work · E.5. Experiment Details

Alignment evaluations indicate that Codex produces more bugs after exposure to subtly buggy code, which the authors interpret as misalignment while noting possible robustness confounds. The paper proposes dataset curation, quality conditioning, fine-tuning, RLHF, verification-assisted feedback, and improved alignment metrics, and details evaluations using HumanEval prompts with correct or buggy examples.

  • E.3. Results of alignment evaluations: Codex outputs code with a higher frequency of bugs when prompted with buggy code, leading the authors to identify misalignment.The evaluation contrasts prompts containing high-quality code with prompts containing buggy code.
  • E.3. Results of alignment evaluations: The authors caution that subtly buggy prompts might instead expose a robustness failure if they are sufficiently out-of-distribution.They consider this explanation unlikely to dominate because GitHub contains plenty of poor-quality code.
  • E.4. Areas for Further Work: Alignment evaluations should become standard practice, and the evaluation datasets are publicly available.The datasets are provided through the cited code-align-evals-data repository.
  • E.4. Areas for Further Work: Potential alignment improvements include curating or labeling pre-training data, fine-tuning on high-quality bug-free code, and applying RLHF.Formal filtering may be needed because humans find it difficult to write bug-free code, while RLHF can use human judgments of correctness and helpfulness.
  • E.4. Areas for Further Work: Fully aligning capable code models remains challenging because hard tasks may exceed human labelers’ expertise and alignment is difficult to determine from behavior alone.The authors call for better alignment metrics and transparency tools.
  • E.4. Areas for Further Work: A fully aligned code-generating model would write its best code, avoid deliberately introducing bugs, and follow user instructions, making it a more helpful coding assistant.The authors present successful alignment as likely to be very useful despite its difficulty.
  • E.5. Experiment Details: The experiments use 158 HumanEval problems and prepend three independently sampled correct or subtly buggy solutions from a 30-problem subset to evaluation prompts.Models are compared with correct-solution context, no prepended solutions, and subtly buggy-solution context; the current task is excluded, with T = 0.2.

F. Supplemental Bias Analysis … G.1. Threat actors

Codex can encode and reproduce harmful social biases in generated code and text, while its unfiltered research outputs should be treated as untrusted until reviewed. The analysis also highlights limitations of its probes and a threat landscape resembling language models, though misuse may differ.

  • F. Supplemental Bias Analysis: Codex generated biased code, creating potential allocative or representational harms in the contexts where code is used and reused.The paper connects these harms to code’s role in foundations for world-changing applications.
  • F. Supplemental Bias Analysis: Unfiltered Codex outputs should be treated as untrusted until users review and verify their accuracy and fitness for purpose.The models can inherit outdated or troublesome ideas rather than functioning as objective tools.
  • F.1. Probes for classification prompts and completions that encode bias: Codex often assumed binary gender in completions for def gender(x): and produced commonly generated completions encoding harmful bias for def race(x):.The probes examined single- and multi-line autocompletions for classification prompts involving protected classes.
  • F.1. Probes for classification prompts and completions that encode bias: After age-classification prompts, Codex sometimes suggested more sensitive classifications, including emotion.The paper characterizes these cases as potentially exacerbating harm when engineers do not realize they are veering into harmful territory.
  • F.2. Analyzing bias in text generated by Codex: Codex comments reproduced biases similar to GPT-3 across gender, race, and religion, with less output diversity; for Islam, both produced “terrorist” and “violent” more often than for other groups.GPT-3 produced more variants on these themes, while co-occurrence testing did not measure contextual subtleties and used constrained prompts.
  • F.2. Analyzing bias in text generated by Codex: Codex’s typical use is less open-ended than GPT-3’s, because users tend to prompt it more precisely and neutrally, though not always.This difference limits direct inference from GPT-3’s likely impact to Codex’s textual harms.
  • G.1. Threat actors: Codex faces a threat landscape similar to language models, spanning low-skilled actors through well-resourced APT groups with objectives including money, chaos, information, or organizational goals.The paper notes that the ways Codex may be misused will likely differ from those of language models.

G.2. Potential misuse applications · G.3. Insecure code generation

Codex presents potential misuse and security risks, but generally does not outperform conventional tools for offensive cybersecurity, vulnerability discovery, or phishing. Nevertheless, it can generate harmful components and frequently produce insecure code, with no clear improvement from scaling in the tested cryptographic tasks.

  • G.2. Potential misuse applications: Codex may assist threat actors with malware production, phishing, and other unauthorized offensive purposes, although it does not differentially enable offensive cybersecurity capabilities.The paper characterizes Codex as excelling at boilerplate and assesses it as no more efficient or effective than conventional tools for offensive cybersecurity.
  • G.2. Potential misuse applications: Codex is not proficient at standalone malicious code generation but can produce components incorporated into more complex systems.It struggled with SQL and shell injection payloads but generated code that recursively encrypts files in a directory.
  • G.2. Potential misuse applications: Codex performed worse than even rudimentary Static Application Security Testing tools for vulnerability discovery.SAST tools generally find simple rule-based vulnerabilities, while Codex did not perform well in comparison.
  • G.2. Potential misuse applications: Codex generally cannot suggest specific vulnerable, malicious, or typosquatted package versions because version information lies outside its prompt context.Package versions are specified in manifest files or installed package files rather than the prompt context available to Codex.
  • G.2. Potential misuse applications: Code-trained models offered no advantage over conventional language models for generating phishing pretexts because the domains differ fundamentally.The paper also identifies a trust boundary in training on public data and warns that adversarial inputs could induce vulnerable, malicious, or misaligned suggestions.
  • G.2. Potential misuse applications: Codex may suggest compromised dependencies, insecure function calls, or secrets from training data, creating a potential supply-chain risk if widely deployed.The paper also judges standalone safety-critical system synthesis unlikely because Codex lacks system-level generation capabilities.
  • G.3. Insecure code generation: Codex models of varying sizes frequently used clearly insecure configurations when prompted to generate RSA keys or AES contexts.The evaluation tested calls to cryptographic libraries for generating cryptographic contexts and assessed whether outputs were clearly insecure.
  • G.3. Insecure code generation: Over more than 1 order of magnitude of parameters, insecure code production showed no robust model-size trend, leaving improvement with scale unclear.The authors interpret this result as an alignment issue and suggest that a larger study of common insecure vulnerabilities is needed.

H. Supplemental economic analysis … H.5. Future directions

The economic effects of Codex remain preliminary and uncertain, spanning programmer productivity, labor-market changes, accessibility, non-engineering automation, package-selection biases, and potential security risks. The paper therefore emphasizes scenario-specific analysis and further research rather than strong conclusions.

  • H. Supplemental economic analysis: Codex can produce clearly insecure cryptographic configurations, including RSA keys shorter than 2048 bits and AES contexts using ECB mode.The appendix describes these as configurations that consensus among cryptography experts generally advises against.
  • H.1. Impacts on programmers and engineers: Codex may reduce software-production costs, but programming work also includes collaboration, design specifications, system upgrades, prompt framing, and code review.These additional tasks and model-related overhead mean code generation would not reduce software labor costs to zero, even with perfect accuracy.
  • H.1. Impacts on programmers and engineers: Codex may create complementary engineering work, alter coding-job screening, and encourage new roles for workers skilled at using code-generation tools.The paper links these possibilities to prompt engineering, Codex’s coding-challenge performance, and employers’ potential reconsideration of interview processes.
  • H.2. Differential impacts among engineers: Codex’s effects may differ across engineers: Python-dominant roles may be more affected, while tool adoption could either substitute for labor or enhance productivity and bargaining power.Because Python is growing and prominent in education, Codex might also make engineering more accessible to people from more diverse demographic backgrounds.
  • H.3. Impacts on non-engineers: Code-generation tools may broaden entry into programming, change the skills new programmers need, and simplify repetitive-task automation in non-engineering roles.Codex may lower barriers by helping people work with unfamiliar codebases or languages, while generated tools can automate repetitive work outside engineering.
  • H.4. Effects of differential package import rates: Codex imports substitutable packages at different rates, potentially causing subtle errors, improving robustness, or increasing dominance within the software supply chain.Users may increasingly rely on Codex as a package decision-making tool, which could entrench existing packages, miss newer ones, or suggest deprecated methods.
  • H.5. Future directions: The authors recommend studying economic value, documentation and testing changes, real-world effects on productivity and wages, and barriers to entering programming.They also call for deeper analysis of deployment scenarios, capabilities, and disparate outcomes across groups because precise prediction without user or market signals is difficult.
  • H.5. Future directions: As code-generation capabilities improve, their effects on high-skill workers could be substantial, motivating additional research and updated policy views about AI substitution.The paper presents its economic analysis as highly preliminary and intended primarily to motivate further related work, not strong conclusions.
Loading 2107.03374v2…