PaperlayerView exact revision on arXiv

Source-linked AI summary

NatGen: Generative pre-training by "Naturalizing" source code

Saikat Chakraborty, Toufique Ahmed, Yangruibo Ding, Premkumar Devanbu, Baishakhi Ray

arXiv:2206.07585v2cs.PLcs.AIcs.LGcs.SE

TL;DR

Many software-engineering tasks require high-quality labeled data, but such data is scarce and pre-training performance depends on the objective. NatGen naturalizes artificially transformed code into developer-like equivalents using semantics-preserving transformations, achieving state-of-the-art results across generative tasks and strong zero-shot and few-shot transfer.

  • Problem

    High-quality labeled data is scarce for many software-engineering tasks, while pre-trained model performance depends substantially on the pre-training objective.

  • Method

    NatGen pre-trains a sequence-to-sequence model to reconstruct natural developer code from artificially de-naturalized, semantically equivalent code.

  • Results

    NatGen achieves state-of-the-art performance across code generation, translation, and bug-fixing tasks, with especially effective transfer in zero-shot and few-shot settings.

  • Takeaways & Limitations

    Code naturalization provides a self-supervised pre-training objective that supports learning to ingest and generate code when labeled data is limited.

  • Takeaways & Limitations

    NatGen currently reverts only one de-naturalization transformation when inputs contain more than one, limiting the scope of its transformation coverage.

Abstract

from arXiv · show

Pre-trained Generative Language models (e.g. PLBART, CodeT5, SPT-Code) for source code yielded strong results on several tasks in the past few years, including code generation and translation. These models have adopted varying pre-training objectives to learn statistics of code construction from very large-scale corpora in a self-supervised fashion; the success of pre-trained models largely hinges on these pre-training objectives. This paper proposes a new pre-training objective, "Naturalizing" of source code, exploiting code's bimodal, dual-channel (formal & natural channels) nature. Unlike natural language, code's bimodal, dual-channel nature allows us to generate semantically equivalent code at scale. We introduce six classes of semantic preserving transformations to introduce un-natural forms of code, and then force our model to produce more natural original programs written by developers. Learning to generate equivalent, but more natural code, at scale, over large corpora of open-source code, without explicit manual supervision, helps the model learn to both ingest & generate code. We fine-tune our model in three generative Software Engineering tasks: code generation, code translation, and code refinement with limited human-curated labeled data and achieve state-of-the-art performance rivaling CodeT5. We show that our pre-trained model is especially competitive at zero-shot and few-shot learning, and better at learning code properties (e.g., syntax, data flow).

1 Introduction

NatGen addresses scarce high-quality labels by pre-training models to transform artificially unnatural code into semantically equivalent code resembling developers’ original code. The approach uses code’s formal and natural channels to generate training pairs at scale and achieves state-of-the-art results across generative software-engineering tasks.

  • Motivation: High-quality labels are limited for many software-engineering tasks, while simply collecting GitHub code can produce low-quality and duplicated data.This motivates self-supervised pre-training objectives that transfer across code generation, translation, repair, and related tasks.
  • Approach: NatGen naturalizes weird synthetic code into semantic-equivalent code that a human developer would have written.The task is designed to make the model understand input code and generate well-formed output code.
  • Approach: Code’s formal channel permits meaning-preserving transformations, enabling large-scale pairs of de-naturalized and original natural code.The natural channel captures developer-oriented syntax and expression, while the formal channel preserves semantics during transformation.
  • Approach: NatGen is a transformer sequence-to-sequence model pre-trained to reconstruct developers’ original code from artificially de-naturalized inputs.The model learns to generate complete code, supporting downstream tasks that require code generation.
  • Results: NatGen achieves state-of-the-art performance on several downstream settings, including code generation, code translation, and bug fixing.The paper also reports especially strong performance when labeled data is scarce.
  • Contributions: The authors introduce Code Naturalization, release a large pre-training dataset and sequence-to-sequence model, and share implementation resources.The released resources include source code, a data-download script, and a pre-trained model.

2 Background & Problem Formulation

The paper formulates naturalizing as a code pre-training task that preserves formal semantics while making syntax more developer-like, then evaluates its transfer to generative tasks under limited supervision. It reports state-of-the-art downstream performance, strong low-data behavior, and uneven difficulty across transformation types.

  • 2.1 The Dual Channels of Code: Source code combines a precise formal channel with a noisier natural channel involving names and comments.The formal channel supports semantic-preserving transformation, while the natural channel supports human comprehension and communication.
  • 2.1 The Dual Channels of Code: Naturalizing keeps code meaning constant, changes its syntax into an unnatural form, and trains the model to generate code a developer would write.This objective simultaneously targets code comprehension and code generation.
  • 2.2 “Naturalizing” vs. De-noising: Naturalizing extends denoising pre-training by starting from syntactically correct but artificially unnatural code and producing semantically equivalent natural code.The paper characterizes this as a more subtle and demanding alternative to syntactic noise such as token masking or deletion.
  • 2.3 Research Questions: The study asks whether naturalization improves code generation and how its components and transformation challenges affect performance.The transformation study includes dead-code insertion and variable renaming.
  • 2.3 Research Questions: Variable Renaming is the hardest transformation to undo at 40% accuracy, whereas Dead Code is easiest at 99% accuracy.These figures come from reconstruction on a held-out validation dataset matching the training transform distribution.
  • 2.3 Research Questions: After fine-tuning, NatGen reaches state-of-the-art performance on Java generation from text, Java↔C# translation, and bug fixing.Generated code is reported as syntactically and semantically closer to the expected code.
  • 2.3 Research Questions: The low-data evaluation studies zero-shot and few-shot transfer by removing task-specific fine-tuning or subsampling benchmark training data.The paper reports that NatGen is efficient in low-data training.

3 Methodology

NatGen creates pre-training pairs by transforming natural source code into semantically equivalent unnatural forms, then trains a sequence-to-sequence model to reconstruct the original code. Its transformations operate over program structure and include six rule classes, while the workflow uses AST-based location selection and context adaptation to preserve functionality.

  • Overall Pipeline: NatGen de-naturalizes source code, pre-trains on reconstructing original developer-written code, and then fine-tunes the model for task-specific code generation.The pipeline consists of de-naturalizing source code, naturalization pre-training, and task-specific fine-tuning.
  • Transformation Rules: The six transformation classes include loop transformation, dead code injection, block swap, operand swap, confusing code insertion, and variable renaming.These rules rewrite natural code into artificial forms while preserving semantics.
  • Transformation Rules: Loop transformation converts for loops to equivalent while loops and vice versa, adapting initializers, updates, and loop-breaking statements.The transformation preserves loop behavior by moving initialization and update expressions to appropriate positions.
  • Transformation Rules: Dead code injection inserts statements inside branches or loops guarded by unsatisfiable conditions, making the inserted code unreachable.The injected statements can be transplanted from the same input program.
  • Transformation Rules: Block and operand swaps preserve equivalence by negating branch conditions or adjusting logical operators when operands are exchanged.For asymmetric inequalities, operand swapping reverses the comparison direction; symmetric operators remain unchanged.
  • Transformation Workflow: The workflow parses code into an AST, selects nodes where applicable transformations exist, adapts transformed nodes to their context, and translates the result back into code.The transformation functions use AST analysis and approximate data-flow analysis and are designed to preserve the original functionality.

4 Experimental Setup

NatGen is evaluated through pre-training and fine-tuning configurations spanning multiple programming languages and three source-code generation tasks. The experiments use standard code-quality metrics and compare NatGen with established pre-trained models and task-specific baselines.

  • Tasks and Data: The experimental design evaluates NatGen on text-to-code generation, Java↔C# code translation, and bug fixing.Bug-fix datasets include small programs with fewer than 50 tokens and medium programs with up to 100 tokens.
  • Tasks and Data: Pre-training primarily uses CodeSearchNet across Java, Python, Go, JavaScript, Ruby, and PHP, plus 1M functions each for C and C#.The additional C and C# functions were collected from 5,000 active GitHub projects.
  • Model Configuration: NatGen uses a 12-layer, 12-head encoder-decoder configuration based on CodeT5 and initializes from the released CodeT5-base parameters.The model is pre-trained for 25K steps on two Nvidia GeForce RTX 3090 GPUs with an effective batch size of 1080.
  • Evaluation: Evaluation reports exact match, Syntax Match, Dataflow Match, and CodeBLEU for generated code.Syntax Match compares AST subtrees, while Dataflow Match compares anonymized def-use edges with the target code.
  • Baselines: Comparisons include GPT-2, CodeGPT, PLBART, SPT-Code, CodeT5, and task-specific baselines where applicable.Many fine-tuning results are reported against available CodeXGLUE leaderboard results.

5 Empirical Results

NatGen is evaluated during pre-training and on code generation, translation, bug fixing, zero-shot, and few-shot tasks. Across these settings, it generally produces syntactically and semantically closer code than comparison models, while transformation type affects pre-training difficulty.

  • Pre-training effectiveness: NatGen generates syntactically and semantically valid code more reliably than comparison models, with only 0.01% direct copying versus PLBART’s 7.05%.NatGen’s median edit distance is 10, compared with 3 for PLBART.
  • Pre-training effectiveness: Variable renaming is NatGen’s hardest transformation at about 40% accuracy, whereas dead-code elimination reaches about 99% accuracy.The paper attributes the dead-code result partly to the simplicity and formulaic nature of the insertion rule.
  • Pre-training effectiveness: After 1,000 pre-training steps, exact match reaches 61%, syntax match 97%, dataflow match 94%, and CodeBLEU 95%.At step 0, the corresponding values are 0%, 13.93%, 19.86%, and 9.74%.
  • Fine-tuning tasks: On text-to-code generation, NatGen’s final model surpasses CodeT5 by 2.8% in SM, 5.28% in DM, and 4.74% in CB.

6 Limitations & Threats

The paper identifies limitations in NatGen’s transformation coverage, evaluation comparisons, and scalability. It presents NatGen as a proof of concept while noting retained CodeT5 capabilities and future data needs.

  • Transformation bias: NatGen reverses only one de-naturalization transformation when inputs contain multiple transformations, limiting the current transformation coverage.The authors leave broader natural-code patterns and their effect on code generation for future work.
  • Knowledge retention: NatGen’s Code summarization BLEU is 19.547 versus CodeT5’s 19.551, suggesting it mostly retains CodeT5’s natural-language generation capacity.NatGen starts further pre-training from a CodeT5-base checkpoint, which raises the possibility of catastrophic forgetting.
  • Comparison constraints: The comparison with CodeT5 is constrained because NatGen is initialized from CodeT5 and the authors do not further pre-train CodeT5 under its original objective.The authors also note that CodeT5’s pre-training code and additional checkpoints were unavailable.
  • Computational cost: CodeT5’s original pre-training required 16 NVIDIA A100 GPUs with 40G memory each for 12 days, making additional large-scale comparison expensive.The authors use the publicly available CodeT5 checkpoint to avoid repeating this costly pre-training.
  • Refactoring scope: NatGen’s code-refactoring potential would require more training data and a larger variety of transformations for more accurate and powerful tools.The paper frames this capability as a future direction rather than a completed refactoring system.

7 Related Works

Related work progresses from encoder-only code understanding models to generative encoder-decoder models. NatGen retains a similar architecture to CodeT5 but replaces denoising objectives with semantic-preserving code naturalization.

  • Pre-trained code models: BERT-style models such as CodeBERT and GraphCodeBERT use pre-training to capture code representations, while GPT- and BART-style models target code generation.These model families reflect different pre-training strategies for understanding and generating source code.
  • Generative limitations: CodeBERT and GraphCodeBERT perform well on code understanding but are less suited to generation because they require an untrained decoder during generative-task fine-tuning.PLBART addresses this limitation with a generative denoising autoencoder.
  • CodeT5: CodeT5 extends T5 with identifier tagging and masked identifier prediction, supporting both code understanding and generative tasks.Its encoder-decoder architecture makes it a close architectural comparison for NatGen.
  • NatGen: NatGen uses CodeT5-like architecture but de-naturalizes code with meaning-preserving transformations and trains the model to recreate natural code.The approach exploits code’s dual-channel nature, where semantically equivalent rewrites can be generated.

8 Conclusion

The paper introduces Code-Naturalization pre-training, which transforms developer code into unnatural but equivalent forms and trains NatGen to restore natural code. Fine-tuned NatGen reaches state-of-the-art results across three generative Software Engineering tasks and is especially effective in zero- and few-shot settings.

  • Conclusion: Code-Naturalization trains NatGen to generate natural source code from unnatural counterparts created through six semantic-preserving transformations.The objective targets code that is syntactically and semantically closer to developer-written code.
  • Conclusion: NatGen achieves state-of-the-art performance across three downstream tasks requiring code generation after fine-tuning.The conclusion identifies these tasks as variations of code-generation-related Software Engineering problems.
  • Conclusion: NatGen’s naturalizing pre-training is especially effective in resource-constrained zero-shot and few-shot transfer-learning settings.This conclusion highlights performance under limited task-specific supervision.

A Further Examples of Code Transformations

Figures 10 and 11 provide examples of input generated code from different pre-trained models. Together, they serve as qualitative examples for comparing model-generated code.

  • Figure 10: Figure 10 presents examples of input generated code produced by different pre-trained models.The figure is intended as a qualitative comparison of generated-code examples.
  • Figure 11: Figure 11 continues the examples of input generated code from different pre-trained models.It extends the qualitative examples presented in Figure 10.

B Pre-training progression for all metrics

Figure 12 tracks different metrics for different languages in the validation dataset across pre-training steps.

  • Figure 12 shows validation-dataset metrics for different languages over the number of pre-training steps.
Loading 2206.07585v2…