Source-linked AI summary
A Transformer-based Approach for Source Code Summarization
Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, Kai-Wei Chang
TL;DR
Source code summarization aims to produce readable descriptions that facilitate program comprehension, while existing RNN-based representations struggle with source code structure and long-range dependencies. The paper explores a Transformer with self-attention, relative position representations, and copy attention, finding that it outperforms state-of-the-art approaches by a large margin. The authors also identify design trade-offs involving relative positions, AST-based inputs, and future software-engineering applications.
Problem
Source code summarization must represent non-sequential code structure and long-range token dependencies, which RNN-based models may fail to capture.
Method
The paper uses a Transformer with self-attention, relative position representations, and copy attention to generate natural-language summaries from source code.
Results
The Transformer with relative position representations and copy attention outperforms state-of-the-art source-code summarization approaches by a large margin.
Takeaways & Limitations
Relative position representations help model pairwise relationships between source-code tokens, while directional information remains important.
Takeaways & Limitations
Future work is needed to incorporate code structure effectively into the Transformer and apply these techniques to other software-engineering sequence-generation tasks.
Abstract
from arXiv · showhide
Generating a readable summary that describes the functionality of a program is known as source code summarization. In this task, learning code representation by modeling the pairwise relationship between code tokens to capture their long-range dependencies is crucial. To learn code representation for summarization, we explore the Transformer model that uses a self-attention mechanism and has shown to be effective in capturing long-range dependencies. In this work, we show that despite the approach is simple, it outperforms the state-of-the-art techniques by a significant margin. We perform extensive analysis and ablation studies that reveal several important findings, e.g., the absolute encoding of source code tokens' position hinders, while relative encoding significantly improves the summarization performance. We have made our code publicly available to facilitate future research.
1 Introduction
Source code summarization supports program comprehension by producing readable descriptions of program functionality. This work addresses limitations of RNN-based approaches by using Transformer self-attention and relative position representations to model long-range token relationships.
- Source code summarization creates readable descriptions of program functionality, helping reduce developers’ effort in program comprehension.
- RNN-based models process code sequentially, limiting their ability to represent non-sequential structure and long-range dependencies in long source-code sequences.
- The proposed approach uses Transformer self-attention to capture long-range dependencies between source-code tokens.
- Relative position representations model pairwise relationships between code tokens and significantly improve summarization over absolute position representations.
- Experiments on two GitHub datasets and ablation studies show that the simple approach outperforms state-of-the-art techniques and quantify Transformer design choices.
2 Proposed Approach
The proposed approach uses a Transformer with self-attention to summarize source code, modeling token order and pairwise relationships. It combines relative position representations with copy attention, while finding that absolute source-code positions lead to poor summarization performance.
- Architecture: The model uses a Transformer composed of stacked multi-head attention and parameterized linear transformation layers in both encoder and decoder.Each multi-head attention layer performs self-attention with multiple attention heads.
- Copy Attention: Copy attention lets the decoder generate vocabulary words or copy rare tokens such as function and variable names from the source code.The additional attention layer learns a copy distribution on top of the decoder stack.
- Position Representations: Absolute source-code position embeddings encode token positions in dmodel dimensions, but capturing this order is reported to produce poor summarization performance.Absolute position embeddings are used for source-code tokens, while alternative summary-token position representations are not studied.
- Position Representations: Relative position representations encode pairwise relationships between input elements within the self-attention mechanism.The representation uses learned key and value terms for relative positions, with the maximum relative distance clipped to k.
- Experimental Comparison: The experiments compare the vanilla Transformer using absolute positions with a Full Model using relative positions and copy attention.The comparison is reported in Table 2 alongside baseline methods.
- Position Representations: The studied directional variant ignores whether token j is to the left or right of token i and learns 2k + 1 relative position representations.The direction-agnostic representation clips relative distance using the maximum value k.
3 Experiment
Experiments on Java and Python datasets evaluate Transformer design choices using BLEU, METEOR, and ROUGE-L. Results favor relative positional representations, copy attention, token splitting, and deeper models, while AST information does not improve summarization.
- Setup: Experiments use Java and Python datasets, with BLEU, METEOR, and ROUGE-L as evaluation metrics.The study compares the Transformer approach with five baseline methods reported in prior work.
- Overall results: The Base model outperforms the baselines except for ROUGE-L on Java, while the Full model improves performance further.The Full model includes additional design choices evaluated in the experiments.
- Tokenization: 0.60, 0.72 BLEU and 1.66, 2.09 ROUGE-L points are lost on Java and Python, respectively, when CamelCase and snake_case tokens are not split.Splitting code tokens reduces the Java vocabulary from 292,626 to 66,650 unique tokens.
- Position representation: Absolute positional encoding slightly hurts performance, whereas relative position representations improve it across the experiments.Directional information matters, while relative distances of 16, 32, and 2^i produce similar performance.
- Model structure: Deeper models perform better than wider models, while incorporating AST information does not improve summarization and increases input length from 120 to 172 tokens.Transformer complexity is O(n^2 × d), so AST-based inputs add computational cost.
- Qualitative analysis: Copy-enabled models generate shorter summaries with more accurate keywords than Vanilla Transformer models, and relative positions increase copy probability for frequent code tokens.These observations come from qualitative examples on the Java and Python datasets.
4 Related Work
Related work primarily uses recurrent sequence-to-sequence models, while other approaches incorporate convolution, AST structure, API usage, reinforcement learning, dual learning, or retrieval. This work studies Transformer design choices without altering its core architecture.
- Neural summarization: Most neural code summarization approaches use recurrent encoder-decoder networks with attention for sequence generation.A convolutional attention model is another reported approach for producing short, name-like summaries.
- Structural information: Recent methods encode program structure with Tree-LSTM, Tree-Transformer, or graph neural networks, while SBT flattens ASTs into sequences.Prior work reports improvements from SBT-based representations and from separating code structure from code tokens.
- Additional enhancements: Other enhancements leverage API usage information, reinforcement learning, dual learning, and retrieval-based techniques.The paper notes that these techniques could be combined with a Transformer but are outside its focused design study.
5 Conclusion
The paper reports that Transformer with relative position representations and copy attention substantially outperforms state-of-the-art source-code summarization approaches. Future work will incorporate code structure and extend these techniques to other software-engineering sequence-generation tasks.
- Transformer with relative position representations and copy attention outperforms state-of-the-art approaches by a large margin.
- Future work will study effective incorporation of code structure into the Transformer.
- The authors also plan to apply these techniques to other software-engineering sequence-generation tasks, such as commit message generation.
A Hyper-Parameters
The appendix summarizes the hyper-parameters used in the experiments, including Transformer depth, attention-head count, and relative-position clipping distance.
- Table 7 summarizes the hyper-parameters used in the experiments.
- l denotes the number of Transformer layers.
- h denotes the number of Transformer attention heads, while k denotes the clipping distance in relative position representations.
B Recurrent Encoder-Decoder vs. Transformer on Python Dataset
On the Python dataset, the authors compare recurrent Seq2seq models with their Transformer approach after revisiting prior recurrent-model results and hyper-parameter tuning. The Transformer outperforms the recurrent Seq2seq models by a large margin.
- The comparison evaluates recurrent Seq2seq models and Transformer on the Python dataset.
- The authors suspect prior lower performance resulted from incorrectly tuned hyper-parameters, motivating a fairer model comparison.
- The authors report that their recurrent Seq2seq model performs much better than results reported in prior work.
- Copy attention does not improve the recurrent Seq2seq model and is associated with lower training and validation performance than the base recurrent model.
- The proposed Transformer-based approach outperforms recurrent Seq2seq models by a large margin.
C Qualitative Examples
The qualitative-example appendix presents model outputs for Python and Java datasets, enabling qualitative comparison across different models.
- Table 9 presents a qualitative example of different models’ performance on the Java dataset.
- Together, the tables provide qualitative examples for comparing model performance across Java and Python datasets.
- Table 10 presents a qualitative example of different models’ performance on the Python dataset.