PaperlayerView exact revision on arXiv

Source-linked AI summary

Execution-based Code Generation using Deep Reinforcement Learning

Parshin Shojaee, Aneesh Jain, Sindhu Tipirneni, Chandan K. Reddy

arXiv:2301.13816v4cs.LGcs.AIcs.CLcs.PL

TL;DR

Pre-trained code models can generate programs that fail to meet code-specific requirements such as compilability and functional correctness. PPOCoder applies PPO using execution and structure feedback, and the reported experiments show improvements over SOTA baselines across code-generation tasks and programming languages.

  • Problem

    Pre-trained models rely on text-generation objectives that may not ensure code-specific properties such as compilability, syntactic correctness, and functional correctness.

    Source: §1 · p. 1, §2.2 · p. 3

  • Method

    PPOCoder fine-tunes pre-trained code models with PPO using compiler feedback, AST and DFG matching scores, and a KL-divergence penalty.

    Source: §1 · p. 2, §3.2.2 · p. 6, §3.2.3 · p. 7

  • Results

    Experiments across three code-generation tasks report that PPOCoder outperforms SOTA baselines in compilation rate and functional correctness across different programming languages.

    Source: §1 · p. 3, §5 · p. 13

  • Takeaways & Limitations

    PPOCoder provides a framework for applying code-specific execution and structure feedback to code-generation model optimization.

    Source: §1 · p. 2, §5 · p. 13

Abstract

from arXiv · show

The utilization of programming language (PL) models, pre-trained on large-scale code corpora, as a means of automating software engineering processes has demonstrated considerable potential in streamlining various code generation tasks such as code completion, code translation, and program synthesis. However, current approaches mainly rely on supervised fine-tuning objectives borrowed from text generation, neglecting unique sequence-level characteristics of code, including but not limited to compilability as well as syntactic and functional correctness. To address this limitation, we propose PPOCoder, a new framework for code generation that synergistically combines pre-trained PL models with Proximal Policy Optimization (PPO) which is a widely used deep reinforcement learning technique. By utilizing non-differentiable feedback from code execution and structure alignment, PPOCoder seamlessly integrates external code-specific knowledge into the model optimization process. It's important to note that PPOCoder is a task-agnostic and model-agnostic framework that can be used across different code generation tasks and PLs. Extensive experiments on three code generation tasks demonstrate the effectiveness of our proposed approach compared to SOTA methods, achieving significant improvements in compilation success rates and functional correctness across different PLs.

1 Introduction

Pre-trained code models support varied generation tasks but may fail to produce compilable, functionally correct programs. PPOCoder combines PPO with execution and code-structure feedback to optimize code generation across tasks and programming languages.

1 Introduction

  • Pre-trained code models often struggle with execution correctness, while existing approaches are frequently tailored to particular tasks or programming languages.Generated code must maintain syntactic and functional correctness, including passing compilation and unit tests.Source: §1 · pp. 1–2
  • PPOCoder initializes actor and critic networks from a pre-trained PL model, samples a program, evaluates reward using execution feedback and target code, then updates both networks.The critic estimates values used with returns for the updates.Source: Figure 1 · p. 2
  • PPOCoder combines compiler feedback, AST/DFG matching, and a KL-divergence penalty in a PPO-based framework for code generation.The KL penalty controls exploration and helps prevent deviations from pre-trained priors.Source: §1 · pp. 2–3
  • PPOCoder uses compiler or unit-test feedback alongside syntactic and semantic matching scores to provide richer guidance than sparse terminal compiler feedback alone.The matching scores compare AST subtrees and DFG edges between generated code and executable targets.Source: §1 · pp. 2–3
  • Experiments across code completion, code translation, and program synthesis in C++, Java, Python, C#, PHP, and C report improvements over SOTA baselines in compilation rate and functional correctness.The paper also investigates reward components and PPO optimization through ablation studies.Source: §1 · p. 3

2 Related Work

Prior work uses pre-trained models, structural representations, and reinforcement learning for code generation, but existing methods still face accuracy and generalization challenges. PPOCoder combines execution and structure feedback within a task- and model-agnostic RL framework.

2.1 Pre-trained Models for Code Generation

  • Pre-trained code models use self-supervised objectives, yet can struggle to preserve syntactic and operational accuracy in generated code.Examples include encoder-only, decoder-only, and encoder-decoder models trained on code corpora.Source: §2.1 · p. 3

2.2 Leveraging Structure in Code Generation

  • Structure-aware models incorporate ASTs, sketches, or DFGs, but may remain misaligned with non-differentiable objectives such as compilability and passing tests.GraphCode-BERT incorporates DFG semantics in encoded representations, while StructCoder makes encoder and decoder components structure-aware.Source: §2.2 · p. 3

2.3 RL for Sequence Generation

  • RL can optimize non-differentiable sequence metrics, but code-generation methods may be task- or language-specific and rely on sparse compiler signals or token-matching losses.PPOCoder combines compiler feedback with AST and DFG matching and uses a KL penalty; its reward also includes these elements alongside a penalty relative to the pre-trained model.Source: §2.3 · pp. 3–4, Figure 2 · p. 4

3 PPOCoder

PPOCoder fine-tunes pretrained code models with PPO, combining compiler feedback, syntax and semantic structure alignment, and a KL penalty. Its reward and policy-optimization design targets code correctness while constraining divergence from the pretrained model.

3 PPOCoder

  • PPOCoder trains an actor and critic with PPO, using compiler feedback, syntactic and semantic matching, and a KL-divergence penalty as reward components.The components guide generation while addressing sparse compiler feedback and the risk of policy divergence from pretrained models.Source: §3 · pp. 4–5, §3.2 · p. 6

3.1 Problem Formulation

  • Generation is framed as a finite-horizon token-level MDP initialized from the pretrained model; conservative policy iteration limits variation and large per-iteration policy changes.The policy aims to maximize expected reward for generated code.Source: §3.1 · p. 5

3.2 Reward Function

  • The reward combines compiler, syntactic-match, and semantic-match signals received at episode end with a KL penalty applied at each time step.The penalty balances against the other reward terms and controls the active policy’s actions relative to pretrained priors.Source: §3.2 · p. 6

3.2.1 Compiler Signal

  • Compiler feedback rewards functional correctness when unit tests are available; otherwise, it uses the compiler’s signal for syntactic correctness.With tests, the reward depends on passing all unit tests; without them, the compiler returns syntactic correctness.Source: §3.2.1 · p. 6

3.2.2 Syntactic Matching Score

  • The syntactic matching score is the percentage of reference AST sub-trees matched in the generated hypothesis, capturing structural correctness beyond token overlap.AST comparisons can flag structural differences such as a missing operator or erroneous type casting.Source: §3.2.2 · pp. 6–7

3.2.3 Semantic Matching Score

  • The semantic matching score measures the percentage of reference data-flow edges matched, representing variable dependencies in generated code.DFG nodes represent variables, and edges represent the sources of their values.Source: §3.2.3 · p. 7

3.2.4 KL-Divergence Constraint

  • A KL-divergence penalty keeps the active policy near the pretrained model and controls exploration versus exploitation; a larger β means less exploration and more exploitation.The penalty uses log-probabilities from the active policy and pretrained model at each time step.Source: §3.2.4 · p. 7

3.3 Loss Function

  • The PPO loss optimizes the actor’s advantage objective and the critic’s token-level value estimates, using a clipped policy ratio and a weighted combination of loss terms.For each source-target pair, the procedure samples multiple hypotheses, estimates advantages, computes PPO loss, and updates policy and value-head parameters.Source: §3.3 · p. 8

4 Experiments

PPOCoder is evaluated on code completion, translation, and program synthesis, with experiments assessing compilation, code quality, functional correctness, and model components. Results show improvements in compilation and synthesis performance, while ablations examine reward design, optimization objectives, exploration, and sample count.

4.1 Code Completion

  • 97.68% Comp Rate for PPOCoder + CodeT5 versus 52.14% for fine-tuned CodeT5 on code completion.PPOCoder performs similarly to other baselines on Edit Sim and xMatch, so the higher compilation rate is not attributed to generating arbitrary easy-to-compile code.Source: §4.1 · p. 9, Table 1 · p. 9

4.2 Code Translation

  • PPOCoder + CodeT5 raises Comp Rate over CodeT5 by 9.92%, 22.22%, 21.62%, 13.20%, 7.46%, and 6.11% for C++, Java, Python, C#, PHP, and C target PLs, respectively.It also obtains comparable and mostly better CodeBLEU scores than other baselines.Source: §4.2 · p. 9, Table 2 · p. 10

4.3 Program Synthesis

  • On APPS, PPOCoder + CodeT5 improves CodeT5’s program-synthesis performance and achieves pass@k results comparable to or better than CodeRL + CodeT5.The smaller CodeT5 encoder-decoder also outperforms larger models, and PPOCoder + CodeT5 further improves on it.Source: §4.3 · pp. 10–11, Table 3 · p. 11
  • On zero-shot MBPP, PPOCoder + CodeT5 records pass@80 of 63%, compared with 61.4% for GPT-137B; CodeRL + CodeT5 records 68.2%.The models were fine-tuned on APPS and evaluated on MBPP.Source: §4.3 · p. 11, Table 4 · p. 12

4.4 Ablation Study

  • The best reward configuration uses all four reward elements; KL-divergence penalty improves compilation rate, while compiler feedback alone yields a minor improvement.Adding syntactic or semantic matching scores further boosts performance, with AST matching improving compilation rate more than DFG matching.Source: §4.4 · pp. 11–12, Figure 3 · p. 12
  • Proximal conservative policy iteration with value optimization achieves the best performance among tested objectives, exceeding policy-gradient-only and policy-gradient-plus-value-function variants.The tested progression corresponds to REINFORCE, A2C, and PPO-style optimization.Source: §4.4 · p. 12, Figure 3 · p. 12
  • Action-space size 1 may limit exploration, while a vocabulary-sized action space can hinder learning; the experiments use k = 5 as a balance.Increasing synthetic samples from 1 to 10 improves performance, but further increases cause a slight decline and add computational complexity.Source: Figure 3 · p. 12, §4.4 · p. 13

5 Conclusion

The paper presents PPOCoder, a PPO-based framework intended to improve code generated by programming-language models. Experiments report improved syntactic and functional correctness, while the approach adds optimization time and may not improve metrics outside those directly targeted.

5 Conclusion

  • PPOCoder combines compiler feedback, unit tests, and syntactic and semantic feedback in an RL optimization framework for programming-language models.The framework is designed for programming languages rather than natural language.Source: §5 · p. 13
  • Experiments show improvements over baselines in the syntactic and functional correctness of generated code.Ablations also show that different model components affect final performance.Source: §5 · p. 13
  • RL-based optimization adds computational time and may not yield significant gains on evaluation metrics that are not directly targeted.The paper motivates PPOCoder as a more cost-effective way to improve smaller models than training substantially larger ones.Source: §5 · p. 13

A Implementation Details

Implementation uses shared optimizer and graph-matching procedures, with task-specific settings for code completion and translation. Experiments use PyTorch and four Quadro RTX 8000 GPUs.

A Implementation Details

  • Across experiments, training uses batch size 32, AdamW with weight decay 0.05, and a learning rate warming from 1e−7 to 2e−5 before inverse-square-root decay.Tree-sitter identifies AST subtrees; DFG edges are built from AST-leaf variables and their relations, with matching counts compared against the reference structures.Source: A Implementation Details · pp. 17–18

A.1 Code Completion

  • For Python code completion on CSN, PPOCoder +CodeT5 is initialized from fine-tuned CodeT5; RL uses top-k sampling with k = 5, three synthetic outputs per sample, maximum sequence length 400, and six epochs.PPOCoder observes 40K×3=120K input-output sample pairs with synthetic outputs during RL optimization.Source: §A.1 · p. 18

A.2 Code Translation

  • XLCoST translation fine-tunes the policy from CodeT5 for each language pair and uses top-k k = 5, with synthetic-sample counts of 100 for C, 20 for PHP, and 10 otherwise.RL optimization runs for up to 15 epochs, with maximum source and target lengths of 400; compilation uses language-specific compilers or syntax checks.Source: §A.2 · p. 18

A.3 Program Synthesis

The APPS program-synthesis implementation uses difficulty-stratified problem data and evaluates generated programs with pass@k. PPOCoder is initialized from APPS-fine-tuned CodeT5 and also evaluated zero-shot on MBPP.

A.3 Program Synthesis

  • APPS supports program-synthesis experiments using natural-language problem descriptions paired with Python programs across Introductory, Interview, and Competition difficulty levels; pass@k evaluates functional correctness by whether code passes all problem-specific unit tests.Source: §A.3 · pp. 18–19
  • GPT-3 and Codex are evaluated few-shot without fine-tuning, while other baselines and PPOCoder + CodeT5 start from APPS fine-tuning.PPOCoder + CodeT5 uses top-k k = 5 and five synthetic samples per problem, with up to eight RL epochs and maximum source/target lengths of 600/512.Source: §A.3 · p. 19
  • PPOCoder + CodeT5 is also evaluated on MBPP in a zero-shot setting, alongside GPT and CodeT5 models fine-tuned on MBPP for 60 epochs.The MBPP comparison uses maximum source and target lengths of 400.Source: §A.3 · p. 19

B Additional XLCoST Dataset Details

The XLCoST translation experiments use program-level parallel data across six languages, filtered to retain compilable source-target pairs. The appendix reports dataset statistics for both the original and filtered data.

B Additional XLCoST Dataset Details

  • Experiments use only program-level XLCoST data, retaining parallel examples whose source and target programs compile.The authors hypothesize that error-free ground truth provides a stronger signal for correcting syntax than data that may contain flawed programs.Source: B Additional XLCoST Dataset Details · p. 19
  • XLCoST covers C++, Java, Python, PHP, C, and C#; the reported statistics distinguish parallel snippets from parallel programs.Table 5 places snippet counts in the upper triangle and program counts in the lower triangle.Source: B Additional XLCoST Dataset Details · p. 19, Table 5 · p. 19
  • The filtered dataset statistics cover compilable programs, with JavaScript excluded.The experiments use an automated compilation pipeline to identify compilable programs.Source: B Additional XLCoST Dataset Details · p. 19, Table 6 · p. 20

C Additional Evaluation of Code Translation on TransCoder Benchmark

The TransCoder benchmark evaluates models trained on XLCoST using filtered functions with unit tests. PPOCoder+CodeT5 improves computational accuracy across the evaluated language pairs and reduces compilation errors.

C Additional Evaluation of Code Translation on TransCoder Benchmark

  • TransCoder is used only for evaluation because its unit-tested sets are too small for RL fine-tuning; models are trained on XLCoST instead.The benchmark has 466, 481, and 463 test functions with unit tests for C++, Java, and Python, respectively.Source: C Additional Evaluation of Code… · p. 20
  • About 30% of TransCoder test functions overlap with XLCoST, so evaluation uses only filtered functions absent from XLCoST training data.Table 7 reports overlap statistics for TransCoder functions with XLCoST.Source: C Additional Evaluation of Code… · p. 20, Table 7 · p. 20
  • PPOCoder+CodeT5 improves CodeT5 computational accuracy across all six evaluated translation directions on TransCoder.The reported directions are C++→Java, C++→Python, Java→C++, Java→Python, Python→C++, and Python→Java.Source: C Additional Evaluation of Code… · p. 20, Table 8 · p. 21
  • PPOCoder+CodeT5 reduces compilation errors across the six translation directions, while runtime errors occur predominantly when Python is the target language.The authors attribute the latter pattern to Python’s more permissive compilation process.Source: C Additional Evaluation of Code… · pp. 20–21
  • The evaluation reports computational accuracy, compilation errors, runtime errors, failed unit tests, and timeouts.Timeouts are generally attributed to infinite loops, which appear more prevalent in Java↔Python translations.Source: C Additional Evaluation of Code… · pp. 20–21, Table 8 · p. 21

D Additional Ablation Experimetns

The additional ablations vary reward components and RL objectives across translation and synthesis settings. They examine compiler feedback, structure-based reward signals, and the relationship between compilation and functional accuracy.

D Additional Ablation Experimetns

  • Python-C# translation ablations vary reward and loss configurations, divergence penalty, action-space size, and the number of synthetic samples.Figure 4 summarizes these configuration comparisons.Source: D Additional Ablation Experimetns · p. 21, Figure 4 · p. 21
  • With compiler signal, Java-Python variants achieved higher compilation rates than their counterparts without it: +cs+dfg 0.732 vs +dfg 0.674, +cs+ast 0.874 vs +ast 0.801, and +cs+dfg+ast 0.921 vs +dfg+ast 0.879.The variants include the KL-divergence penalty by default; Figure 5 presents the compiler-signal comparison.Source: D Additional Ablation Experimetns · p. 21, Figure 5 · p. 22
  • The MBPP ablations compare reward components and RL objective terms, while Figure 8 frames pass@1 against model size for PPOCoder+CodeT5 and larger models.The Figure 8 caption names CodeGen and PaLM as comparison models.Source: D Additional Ablation Experimetns · p. 21, Figure 8 · p. 23
  • On MBPP, adding only compiler signal raised pass@80 by approximately 16%, and +CPI+VF gave the best results among the tested RL algorithms.The reward-component variants include the KL-divergence penalty by default; all RL algorithms improved on base CodeT5.Source: D Additional Ablation Experimetns · p. 22, Figure 6 · p. 22
  • On MBPP, +cs+dfg+ast achieved the best reported combination: 96.5% compilation rate and 26.1 pass@1.Figure 7 compares pass@1 with compilation rate across reward configurations.Source: Figure 7 · p. 22, D Additional Ablation Experimetns · p. 23
  • On MBPP, +cs+ast reached 90.2% compilation but 18.3 pass@1, below +cs+dfg+ast at 96.5% compilation and 26.1 pass@1.The reported results show a general positive correlation between compilation rate and pass@1, but the AST-only addition does not follow a strictly increasing pattern.Source: D Additional Ablation Experimetns · p. 23

E Discussion and Comparison with SOTA Large Models

On MBPP, PPOCoder+CodeT5 is competitive with several larger models while using a smaller model. It exceeds all listed CodeGen-Multi models but trails the largest PaLM and larger CodeGen-Mono models.

E Discussion and Comparison with SOTA Large Models

  • PPOCoder+CodeT5 scores 26.1 pass@1, exceeding all CodeGen-Multi models (7.4, 18.0, 18.3, 20.9), PaLM-8B (5.0), PaLM-62B (21.4), and CodeGen-Mono-350M (14.5).It trails PaLM-540B (36.8) and CodeGen-Mono models at 2.7B, 6.1B, and 16.1B parameters (27.3, 32.4, and 35.2).Source: E Discussion and Comparison with SOTA… · p. 23
  • PPOCoder+CodeT5 occupies the top-left of the performance-size comparison, representing higher pass@1 with the smallest model size among the compared models.The authors describe this placement as a favorable performance-resource trade-off.Source: E Discussion and Comparison with SOTA… · pp. 23–24

F Case Studies

The case studies illustrate PPOCoder +CodeT5 improving PHP-to-Python and Java-to-C++ translation compilability, while the APPS example highlights functional errors that compilation alone does not catch.

F Case Studies

  • In PHP-to-Python translation, PPOCoder +CodeT5 produces compilable code where CodeT5 has syntax, variable-use, and semantic errors.The example’s errors include an invalid loop layout, a mismatched loop index, and an incorrect translation of PHP_INT_MAX.Source: Figure 9 · p. 24, F Case Studies · p. 24
  • In Java-to-C++ translation, PPOCoder +CodeT5 improves compilation by using a standard vector type and avoiding errors in CodeT5’s translation.CodeT5 also uses an undefined loop variable and omits a parenthesis in a function call.Source: F Case Studies · pp. 24–25, Figure 10 · p. 25
  • In the APPS synthesis example, both models generate compilable programs, but CodeT5’s incorrect array initialization and loop condition fail hidden unit tests.These logical errors are not compilation errors.Source: F Case Studies · p. 25, Figure 11 · p. 26
Loading 2301.13816v4…