Source-linked AI summary
Unified Pre-training for Program Understanding and Generation
Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, Kai-Wei Chang
TL;DR
PLUG tasks need models that connect programming-language syntax and semantics with natural-language meaning, despite limited task-specific supervision. PLBART pretrains a sequence-to-sequence model on unlabeled Java, Python, and natural-language data using denoising autoencoding, then applies it across generation and understanding tasks. It outperforms or rivals state-of-the-art methods across these tasks, while vulnerability detection remains an area where graph-based models perform best.
Problem
PLUG applications require shared understanding of programming-language and natural-language syntax, semantics, and dependencies, while most available data are unlabeled.
Method
PLBART is a bidirectional autoregressive transformer pretrained on unlabeled programming-language and natural-language data with denoising sequence-to-sequence learning.
Results
PLBART outperforms or rivals state-of-the-art methods across program generation, translation, summarization, repair, clone detection, and vulnerability detection tasks.
Takeaways & Limitations
Pretraining enables PLBART to learn syntax, identifier naming conventions, and data flow that support program understanding and generation, including with limited annotations.
Takeaways & Limitations
PLBART is not state-of-the-art in vulnerability detection, where graph-based models perform best.
Abstract
from arXiv · showhide
Code summarization and generation empower conversion between programming language (PL) and natural language (NL), while code translation avails the migration of legacy code from one PL to another. This paper introduces PLBART, a sequence-to-sequence model capable of performing a broad spectrum of program and language understanding and generation tasks. PLBART is pre-trained on an extensive collection of Java and Python functions and associated NL text via denoising autoencoding. Experiments on code summarization in the English language, code generation, and code translation in seven programming languages show that PLBART outperforms or rivals state-of-the-art models. Moreover, experiments on discriminative tasks, e.g., program repair, clone detection, and vulnerable code detection, demonstrate PLBART's effectiveness in program understanding. Furthermore, analysis reveals that PLBART learns program syntax, style (e.g., identifier naming convention), logical flow (e.g., if block inside an else block is equivalent to else if block) that are crucial to program semantics and thus excels even with limited annotations.
1 Introduction
PLUG applications require models to connect programming-language syntax and semantics with natural-language meaning across summarization, generation, and translation. PLBART addresses this need by pretraining shared representations from unlabeled program and natural-language data.
- Motivation: PLUG applications automate code summarization, generation, and translation, requiring understanding of both programming and natural languages.Natural language appears in documentation, commit messages, bug reports, and developer forums.
- Motivation: Equivalent program operations can differ across languages, so translation and summarization require cross-language semantic alignment.Python sorted corresponds to Java Arrays.sort, while tuple indexing appears as x[0] or Tuple.get(0).
- Motivation: PLUG tasks share a prerequisite: learning programming- and natural-language syntax, semantics, and their mutual dependencies.Unlabeled data can support transferable representations across tasks, reducing reliance on large task-specific annotations.
- Motivation: Existing sequence-to-sequence generation methods lack a pretrained decoder and therefore require substantial parallel data for decoder training.Denoising sequence-to-sequence pretraining was proposed to reconstruct corrupted text and address this limitation.
- Contribution: PLBART is a bidirectional autoregressive transformer pretrained on unlabeled programming-language and natural-language data for broad PLUG applications.The paper evaluates it on summarization, generation, translation, repair, clone detection, and vulnerability detection.
2 PLBART
PLBART is a sequence-to-sequence Transformer pretrained by reconstructing corrupted Java, Python, and English data, then fine-tuned for generation and classification tasks. Its design combines denoising objectives with shared handling of code and natural-language inputs and outputs.
- Denoising Pre-training: PLBART uses denoising sequence-to-sequence pretraining to reconstruct original programming or natural-language text from corrupted inputs.The model learns to predict the original instance after applying a noise function.
- Data & pre-processing: Pretraining uses Java and Python functions from GitHub and English questions and answers from StackOverflow.The data sources provide programming-language and natural-language instances for shared pretraining.
- Data & pre-processing: PLBART balances data sources through multinomial sampling with smoothing parameter α = 0.3 because programming-language data outnumber natural-language data.The sampling probabilities depend on the number of instances in each language.
- Architecture: The model uses a BARTbase-style sequence-to-sequence Transformer with six encoder layers, six decoder layers, 768-dimensional representations, and 12 attention heads.An additional layer-normalization layer is added to both encoder and decoder.
- Noise function: The corruption strategies are token masking, token deletion, and token infilling, with infilled span lengths sampled from a Poisson distribution with λ = 3.5.Thirty-five percent of tokens are masked in each instance.
- Fine-tuning: For sequence generation, the encoder receives source code or text and the autoregressive decoder generates summaries, code, or translations.For sequence classification, a final decoder representation is passed to a linear classifier.
3 Experiment Setup
PLBART is evaluated across generation, translation, repair, and classification tasks using public downstream benchmarks and comparisons with scratch-trained and pre-trained baselines. The setup measures sequence quality, code correctness, and program understanding across multiple task types.
- Evaluation Tasks: The evaluation covers code summarization, code generation, code translation, program repair, clone detection, and vulnerability detection.These tasks assess PLBART’s ability to capture semantics in source code and associated natural-language text.
- Evaluation Tasks: Code summarization generates an English summary from code in Ruby, JavaScript, Go, Python, Java, or PHP.
- Evaluation Tasks: Code generation maps a natural-language description and Java class environment to a target Java function.
- Evaluation Tasks: Code translation generates semantically equivalent code across programming languages, while program repair modifies buggy code into a fixed version.The translation and repair tasks probe syntax, semantics, and semantic changes in code.
- Evaluation Tasks: Classification evaluates clone detection and vulnerable-code detection from single or paired source-code inputs.
- Evaluation Metrics: BLEU measures n-gram overlap, CodeBLEU additionally evaluates grammatical and logical correctness, and Exact Match checks exact sequence equality.Smoothed BLEU-4 is used for code summarization, while corpus-level BLEU is used for other generation tasks.
- Baselines: PLBART is compared with models trained from scratch and models pre-trained on unlabeled corpora before fine-tuning.The pre-trained comparisons include encoder-only models with task-specific decoders and decoder-only autoregressive models.
- Baselines: The Transformer baseline matches PLBART’s parameter count, isolating the contribution of PLBART’s pre-training.
4 Results & Analysis
PLBART performs strongly across generation, translation, repair, and classification tasks, while qualitative and ablation analyses indicate that pre-training captures syntax, semantics, and logical flow.
- Code Summarization: PLBART improves code summarization over baseline methods in five of six programming languages, averaging 0.49 BLEU-4 over CodeBERT.The largest improvement is approximately 16% in Ruby, despite Ruby being absent from PLBART’s pre-training languages.
- Code Generation: PLBART outperforms all baselines on code generation in BLEU and CodeBLEU, while CodeGPT-adapted achieves the best Exact Match score.PLBART’s large CodeBLEU margin over CodeGPT-adapted indicates more syntactically and logically correct generated code.
- Analysis: With only 10K fine-tuning examples, PLBART outperforms all baselines in CodeBLEU, supporting learned syntax and data flow during pre-training.The authors attribute valid generation to denoising sequence-to-sequence pre-training that helps model program syntax and logical flow.
- Code Translation: PLBART outperforms translation baselines on EM, BLEU, and CodeBLEU, improving over CodeBERT by 9.5% for Java-to-C# and 10.5% for C#-to-Java.Generated code can differ textually from references while remaining semantically equivalent through alternative control-flow structures and identifier usage.
- Program Repair: In program repair, PLBART generates 17.13% and 74.03% more correct bug fixes than CodeBERT on Javasmall and Javamedium, respectively.PLBART performs comparably to GraphCodeBERT, which uses structure-aware pre-training.
- Classification: PLBART outperforms CodeBERT on clone detection and vulnerable code detection, including improved performance on unseen C/C++ code.The authors note that neither PLBART nor CodeBERT is state-of-the-art for vulnerability detection because graph-based models perform best.
5 Related Work
Prior work established Transformer-based pretraining and deep-learning approaches for software-engineering tasks, but most methods learned task-specific representations. PLBART builds on transferable, unsupervised representations for program and language tasks.
- Transformer pretraining learned representations from large-scale unlabeled data across natural language, computer vision, and software engineering.
- Deep learning has been applied to code summarization, translation, clone detection, and code completion using large code repositories and developer forums.
- Most prior software-engineering approaches used task-specific representation learning, while some learned transferable representations without supervision.
- CodeBERT is a closely related model pretrained on bimodal data, providing context for PLBART’s unified pretraining approach.
6 Conclusion
The paper presents PLBART as a pretrained sequence-to-sequence model for broad program and language understanding and generation. It reports strong downstream performance and learned program characteristics useful for understanding.
- PLBART is a sizeable pretrained sequence-to-sequence model for program and language understanding and generation tasks.
- PLBART achieves state-of-the-art performance across code summarization, code generation, and code translation tasks.
- Discriminative-task experiments establish PLBART’s effectiveness for program understanding in addition to generation.
- Pretraining leads PLBART to learn syntax, identifier naming conventions, and data flow.
Broader Impact
Automating program-and-language applications can reduce routine developer workload and support productivity. The paper identifies documentation, synthesis, and cross-language translation as applications that can be automated.
- Automation in software engineering is presented as a way to increase programmers’ productivity by reducing tedious daily work.
- Reducing routine workload could give developers more time to address significant problems affecting society’s wellbeing.
- Software-development applications suitable for automation include code documentation, summarization, synthesis, and translation across programming languages.