Source-linked AI summary
SPT-Code: Sequence-to-Sequence Pre-Training for Learning Source Code Representations
Changan Niu, Chuanyi Li, Vincent Ng, Jidong Ge, Liguo Huang, Bin Luo
TL;DR
Existing source-code pre-training often excludes the decoder, omits code structure, or depends on bilingual code-description data. SPT-Code uses a seq2seq encoder-decoder model with AST-based inputs and three code-specific tasks, achieving state-of-the-art performance across five downstream tasks after fine-tuning.
Problem
Existing source-code pre-training often omits decoder training and code structure, while learning natural-language descriptions can require a limiting bilingual corpus.
Method
SPT-Code jointly pre-trains an encoder-decoder model on code, AST, and method-description information using adapted MASS, Code-AST Prediction, and Method Name Generation.
Results
SPT-Code achieves state-of-the-art performance on five code-related downstream tasks after fine-tuning.
Takeaways & Limitations
Code-specific seq2seq pre-training can combine source-code, syntactic-structure, and concise description information without relying on a bilingual corpus.
Takeaways & Limitations
The CodeSearchNet pre-training data is imbalanced across six programming languages, and validity beyond those languages is not guaranteed.
Abstract
from arXiv · showhide
Recent years have seen the successful application of large pre-trained models to code representation learning, resulting in substantial improvements on many code-related downstream tasks. But there are issues surrounding their application to SE tasks. First, the majority of the pre-trained models focus on pre-training only the encoder of the Transformer. For generation tasks that are addressed using models with the encoder-decoder architecture, however, there is no reason why the decoder should be left out during pre-training. Second, many existing pre-trained models, including state-of-the-art models such as T5-learning, simply reuse the pre-training tasks designed for natural languages. Moreover, to learn the natural language description of source code needed eventually for code-related tasks such as code summarization, existing pre-training tasks require a bilingual corpus composed of source code and the associated natural language description, which severely limits the amount of data for pre-training. To this end, we propose SPT-Code, a sequence-to-sequence pre-trained model for source code. In order to pre-train SPT-Code in a sequence-to-sequence manner and address the aforementioned weaknesses associated with existing pre-training tasks, we introduce three pre-training tasks that are specifically designed to enable SPT-Code to learn knowledge of source code, the corresponding code structure, as well as a natural language description of the code without relying on any bilingual corpus, and eventually exploit these three sources of information when it is applied to downstream tasks. Experimental results demonstrate that SPT-Code achieves state-of-the-art performance on five code-related downstream tasks after fine-tuning.
1 INTRODUCTION
Existing code pre-training models leave gaps in jointly learning encoder-decoder representations, code structure, and natural-language descriptions without bilingual data. SPT-Code addresses these gaps with a seq2seq model, three code-specific pre-training tasks, and evaluation across five downstream tasks.
- Research gap: Most existing models pre-train only the Transformer encoder, omit code structure, or rely on natural-language pre-training tasks and bilingual code-description data.The paper identifies these as three unresolved issues in applying pre-trained models to software-engineering tasks.
- Approach: SPT-Code jointly pre-trains the Transformer encoder and decoder using code sequence, AST, and method-description information derived from each method.Method names and invoked-method names provide a succinct description without requiring a bilingual corpus.
- Pre-training tasks: The model introduces adapted MASS, Code-AST Prediction, and Method Name Generation to learn code knowledge, syntactic structure, and concise natural-language descriptions.Each task targets one of the three information types in an input instance.
- Model design: SPT-Code uses a simplified, linearized AST representation alongside natural language as input and can support both classification and generation tasks.The representation is presented as an extension of pre-trained-model inputs for source code.
- Evaluation: State-of-the-art results were achieved on five downstream code-related tasks after pre-training on unlabeled monolingual code across six programming languages.The evaluated tasks include code summarization, completion, bug fixing, translation, and search.
2 RELATED WORK
Prior work increasingly uses seq2seq architectures and structural inputs, but most software-engineering pre-training still reuses natural-language objectives. SPT-Code selects ASTs as its structural representation while positioning code-specific objectives as an under-investigated design area.
- Pre-training models in NLP and SE: Pre-trained models are compared by their modules, objectives, and input information, including natural language, code, and structural code information.The overview distinguishes whether the encoder, decoder, or both are pre-trained.
- Pre-training models in NLP and SE: Newer models such as T5, BART, and T5-learning jointly pre-train encoder and decoder modules, unlike most earlier work.The paper characterizes these models as seq2seq pre-training approaches.
- Pre-training models in NLP and SE: Except for GraphCodeBERT and TreeBERT, most software-engineering models reuse natural-language objectives, making objective selection under-investigated.Masked language modeling is identified as the most popular reused task.
- Structural information of source code: ASTs contain syntactic structure that code sequences cannot express, while data-flow and control-flow representations are less structurally rich but require less hardware and training time.The paper motivates structural representations through this expressiveness–cost contrast.
- Structural information of source code: SPT-Code chooses ASTs for structural information and linearizes them before encoding.The paper notes that ASTs must be flattened with a traversal method before being fed to an encoder.
3 SPT-CODE
SPT-Code is a Transformer encoder-decoder model that represents each method with code tokens, a linearized AST, and natural-language information. It uses X-SBT and three code-specific pre-training tasks to learn code, syntax, and method intent.
- Architecture: SPT-Code is a 12-layer Transformer encoder-decoder with 768-dimensional representations, 3072-dimensional feed-forward layers, 12 attention heads, and 0.1 dropout.
- Model Input: The model input combines code tokens, a linearized AST, and natural-language tokens separated by [SEP].The natural-language sequence is derived from method names and API calls after splitting CamelCase and snake_case tokens.
- Linearized AST: X-SBT shortens original SBT traversals by more than half and retains AST nodes at or above the expression level to reduce lexical redundancy.Code tokens already provide lexical information, so expression-level X-SBT is used to emphasize syntactic structure.
- Pre-Training Tasks: Code-AST Prediction labels the AST as IsAST or NotAST, using the actual code-corresponding AST half the time and a random dataset AST otherwise.This binary task is intended to teach the model structural information represented by X-SBT.
- Pre-Training Tasks: Modified MASS masks a code fragment for sequence-to-sequence reconstruction, training the model to understand, infer, and generate code sequences.
- Pre-Training Tasks: Method Name Generation removes method-name-derived natural-language tokens from the input so the model learns to generate the method name from the remaining information.
4 EXPERIMENT
The experiment section evaluates SPT-Code through pre-training, downstream fine-tuning, four research questions, and manual quantitative and qualitative analysis.
- Evaluation: The evaluation covers pre-training settings, downstream-task fine-tuning, four research questions, and manual quantitative and qualitative analysis.
4.1 Pre-Training
SPT-Code is pre-trained on CodeSearchNet, with experiments comparing access to the full corpus against subsets used by prior models. Its three tasks are trained sequentially with specified optimization settings.
- Dataset: CodeSearchNet contains more than 6.4M code instances across Java, Python, JavaScript, PHP, Go, and Ruby, paired with processed documentation.
- Dataset: SPT-Code can use all 6.4M CodeSearchNet instances because its inputs can be extracted from completely unlabeled data.CodeBERT and GraphCodeBERT use documentation labels, while T5-learning uses only the Java subset in the reported implementation.
- Training: The three pre-training tasks are run sequentially for 10 CAP epochs, 30 MASS epochs, and 30 MNG epochs.All tasks use cross-entropy; optimization uses AdamW with a 5e-5 initial learning rate and 2000 warmup steps.
- Training: The tokenizer uses BPE for code and natural language and a regular word tokenizer for X-SBT sequences.The tokenizer is built on the whole pre-training dataset and reused for downstream tasks without modification.
4.2 Fine-Tuning on Downstream Tasks
SPT-Code is fine-tuned across five software-engineering tasks using task-specific datasets, baselines, and evaluation metrics. The section covers code summarization, completion, bug fixing, translation, and search.
- Code summarization: Code summarization generates concise natural-language descriptions of source-code methods.The section considers classical Java and Python datasets alongside CodeSearchNet.
- Code completion: Any-code completion generates unrestricted code from surrounding program context.Examples target expressions larger than a single AST node, using the remainder of the method as context.
- Bug fixing: Bug fixing generates corrected code from buggy method-level code.The BFPsmall and BFPmedium datasets contain buggy/fixed method pairs extracted from GitHub bug-fixing commits.
- Code translation: Code translation migrates legacy code between programming languages using Java–C# project pairs.Evaluation uses accuracy and BLEU against baselines including Transformer and code-pretrained models.
- Code search: Code search ranks code snippets against natural-language queries using cosine similarity between code and query representations.Negative queries are sampled with a BLEU constraint, and nearest candidate vectors are evaluated using ranking metrics.
4.3 Evaluation
The evaluation addresses four research questions concerning SPT-Code’s downstream effectiveness, component contributions, data scalability, and fine-tuning-data sensitivity.
- Evaluation design: SPT-Code is evaluated through four research questions.The questions organize the subsequent experiments on downstream performance, ablations, pre-training data, and fine-tuning data.
RQ1: How effective is SPT-Code compared with the state-of-the-art baselines and other code pre-trained models on five downstream tasks?
SPT-Code outperforms baselines in most reported downstream cases and remains competitive on code search. Ablations show that pre-training tasks, ASTs, and natural language contribute differently across tasks, while more pre-training data improves performance.
- Overall effectiveness: SPT-Code clearly outperforms dedicated and pre-trained baselines in most cases across summarization, completion, bug fixing on BFPmedium, and translation.The reported comparison covers five downstream tasks and identifies broad gains rather than a single benchmark result.
- Overall effectiveness: SPT-Code is slightly less accurate than S2S+COPYSPAN on BFPsmall bug fixing.The authors attribute this to short inputs and COPYSPAN’s suitability for localized modifications, then report an improvement after removing MNG.
- Overall effectiveness: SPT-Code performs comparably to GraphCodeBERT on code search without sacrificing classification performance.This result contrasts the model’s generation-oriented design with its classification-mode evaluation.
- Ablation analysis: Removing all pre-training tasks or removing AST and natural-language inputs causes considerable performance drops.Removing individual tasks generally reduces results, with CAP most useful for search, MASS for translation, and MNG most influential for summarization.
- Ablation analysis: Natural language and AST inputs both improve performance, but their relative importance depends on the downstream task.Natural language helps summarization, bug fixing, and search more, whereas ASTs help completion and translation more.
- Data scaling: Shrinking pre-training data from 6.4M to 2.3M or 1.5M significantly decreases performance.SPT-Code remains advantageous or comparable with equal data and benefits from additional unlabeled pre-training data.
- Fine-tuning data: Reducing fine-tuning data lowers performance across evaluation metrics.With one-tenth of the training data, summarization becomes comparable to CodeBERT; with two-tenths, it is similar to or higher than GraphCodeBERT.
4.4 Quantitative Analysis
The quantitative analysis evaluates outputs from SPT-Code and baselines across downstream tasks using judgments from five software-engineering graduate students.
- Human evaluation: Five non-author software-engineering graduate students assess outputs across a wide range of downstream-task test samples.Participants are randomly assigned equal numbers of samples, with aligned samples across multiple datasets within a task.
4.5 Qualitative Analysis
The qualitative analysis finds that SPT-Code captures code semantics more accurately and globally than other pre-trained models, producing more complete method summaries. Its advantage is especially visible in recognizing both operations and their relationships to code objects.
- SPT-Code captures identifier semantics more accurately and distributes code-segment semantics more globally than other pre-trained models.The analysis reports that its semantic representations are more widely and evenly distributed across code segments.
- In code summarization, SPT-Code produces more complete and accurate descriptions of a method’s overall functionality.The qualitative comparison uses an example Java method from CodeSearchNet.
- SPT-Code and GraphCodeBERT recognize the queue operation, whereas CodeBERT and T5-learning fail to capture it.The compared models all recognize the queue object, but only GraphCodeBERT and SPT-Code identify the operation conducted on it.
- SPT-Code describes the operation more appropriately than GraphCodeBERT, which captures the object-operation relationship but chooses less suitable wording.The analysis attributes this difference to data flow capturing code structure only partially rather than fully.
5 THREATS TO VALIDITY
The validity discussion identifies threats from CodeSearchNet overlap and duplication, empirically chosen hyperparameters, and language imbalance. Duplicate removal causes little performance decrease, but its effect is confounded with reducing the pre-training dataset.
- Construct Validity: Using CodeSearchNet for both pre-training and evaluating code summarization and code search may expose test samples during pre-training.This creates an unfair comparison with methods not pre-trained on CodeSearchNet, and duplicate removal was not performed in the original setup.
- Construct Validity: After removing test-set overlaps and duplicates, SPT-Code’s results decrease very little.The cleaned experiment re-pre-trained and fine-tuned SPT-Code on JCSD, code completion, and BFPmedium bug fixing.
- Construct Validity: The effect of duplicate removal remains uncertain because the cleaned experiment also shrank the pre-training dataset by about one-tenth.
- Internal Validity: SPT-Code’s hyperparameters were set empirically rather than tuned experimentally, so other settings may yield better results.
- External Validity: Because CodeSearchNet is imbalanced across six programming languages, SPT-Code may perform differently across languages and is not guaranteed to generalize beyond them.
6 CONCLUSION
SPT-Code is an encoder-decoder source-code model trained with three code-specific tasks and a representation combining natural language and AST information. Fine-tuning achieves state-of-the-art results on five downstream tasks, while ablations show that task and input contributions vary by task.
- SPT-Code is a large source-code model based on an encoder-decoder architecture.
- Three code-specific pre-training tasks enable SPT-Code to train on completely unlabeled data.
- SPT-Code combines natural-language and AST representations, including the improved X-SBT AST traversal method.
- Fine-tuned SPT-Code achieves state-of-the-art performance on five code-related downstream tasks.
- Ablations show that the three pre-training tasks have different impacts across downstream tasks, while AST and natural-language inputs improve performance.