Source-linked AI summary
Retrieve and Refine: Exemplar-based Neural Comment Generation
Bolin Wei
TL;DR
Neural comment generators commonly rely only on source code, while software reuse offers comments from similar code that could guide generation. This paper retrieves similar code and uses its comment as an exemplar in a neural generation model. The approach significantly outperforms the compared state-of-the-art methods on a Java dataset.
Problem
Existing neural comment-generation systems use source-code information alone and do not take advantage of software reuse, while sometimes producing short or degenerate comments.
Source: §I · p. 1Method
The model retrieves similar code with an open-source search engine and uses its comment as an exemplar to guide a seq2seq neural network.
Source: §I · p. 1, A. Retrieve Module · p. 2, B. Refine Module · p. 2Results
Our approach significantly outperformed vanilla seq2seq, DeepCom, and HAD on BLEU and METEOR; its BLEU score improved about 10% over HAD.
Source: §IV · p. 2Takeaways & Limitations
Exemplar guidance is presented as a way to reduce template-construction workload while controlling comment generation.
Source: §I · p. 1The token-based search is too simple to retrieve semantically similar code, and evaluation relies on machine-translation metrics rather than human assessment.
Source: §V · p. 2
Abstract
from arXiv · showhide
Code comment generation is a crucial task in the field of automatic software development. Most previous neural comment generation systems used an encoder-decoder neural network and encoded only information from source code as input. Software reuse is common in software development. However, this feature has not been introduced to existing systems. Inspired by the traditional IR-based approaches, we propose to use the existing comments of similar source code as exemplars to guide the comment generation process. Based on an open source search engine, we first retrieve a similar code and treat its comment as an exemplar. Then we applied a seq2seq neural network to conduct an exemplar-based comment generation. We evaluate our approach on a large-scale Java corpus, and experimental results demonstrate that our model significantly outperforms the state-of-the-art methods.
I. INTRODUCTION
Comments help programmers understand and maintain code, but generating them is time-consuming. The paper argues for guiding generation with comments from similar code because source-only neural models can produce short or degenerate outputs.
I. INTRODUCTION
- Comments aid program understanding and software maintenance, while generating proper comments is time-consuming.The paper presents automatic comment generation as important for software development.Source: §I · p. 1
- Source-only neural models can produce short comments and degenerate outputs, motivating the use of information beyond the input code.On the Java dataset, about 2% of DeepCom predictions were under four words, and over seventy HAD predictions repeated a token twenty times.Source: §I · p. 1
- Comments retrieved for similar code may contain information inconsistent with the target code, so the paper proposes using them as exemplars rather than direct replacements.Software reuse makes searching large software libraries for similar code possible,, and traditional IR-based methods use retrieved comments directly –.Source: §I · p. 1
- Using comments from similar code as exemplars is presented as a template-like approach that reduces template-construction work and controls generation.The paper characterizes exemplar-based generation as similar to template-based generation, without requiring templates to be defined manually.Source: §I · p. 1
- The proposed Retrieve and Refine model searches for similar code, encodes the input code, similar code, and its comment, then generates a new comment using their correlation.The Refine module uses LSTMs.Source: §I · p. 1
II. BACKGROUND AND RELATED WORK
Prior comment-generation work includes template-based, IR-based, and deep-learning approaches. The paper identifies failure to exploit software reuse and limited control in existing neural methods as shortcomings.
II. BACKGROUND AND RELATED WORK
- Prior approaches use manually defined templates, retrieve terms or similar code through IR, or learn from large-scale code-comment pairs.Deep-learning systems generally encode a source-code sequence or AST and decode a comment.Source: §II · p. 1
- Fig. 2 presents the approach as exemplar-based and marks the similar-code/comment pair with a dashed box.Source: Figure 2 · p. 2
- Existing deep-learning models do not exploit software reuse and lack generation control, making short and meaningless comments easier to produce.Source: §II · p. 2
III. APPROACH OVERVIEW
The approach has two parts: Retrieve searches for similar code, and Refine generates a comment. The overview marks the similar-code/comment pair as an exemplar.
III. APPROACH OVERVIEW
- The approach overview divides exemplar-based comment generation into a Retrieve module and a Refine module.Source: §III · p. 2
- Retrieve searches for similar code, while Refine performs comment generation.Source: §III · p. 2
A. Retrieve Module
Retrieve finds a similar-code comment to use as an exemplar. It uses Lucene to search a code library and selects the first-ranked match.
A. Retrieve Module
- Retrieve assumes that comments of similar code have similar patterns, turning exemplar selection into a search for similar code.Source: A. Retrieve Module · p. 2
- Given input code, Retrieve searches a code library built with Lucene and treats a similar code’s comment as the exemplar.The study keeps Lucene’s default settings.Source: A. Retrieve Module · p. 2
- For each input code, the system selects the first-ranked similar code and uses its comment as the exemplar.Source: A. Retrieve Module · p. 2
B. Refine Module
Refine uses a seq2seq network to improve a retrieved exemplar by learning differences between the input code and similar code. Its decoder combines input-code and exemplar-comment information, with attention guided by their similarity.
B. Refine Module
- The network learns differences between the input and similar code because the exemplar may contain information, such as different variables, that does not fit the input.Source: B. Refine Module · p. 2
- Three LSTMs encode the input code, similar code, and its comment; a sigmoid function estimates the similarity between the two code snippets.The decoder initializes its hidden state from the final states of the input code and exemplar comment.Source: B. Refine Module · p. 2
- When input and similar code have lower similarity, the decoder should pay more attention to the input code.For each output token, attention computes context vectors from input-code tokens and exemplar-comment tokens, which are combined using the similarity score.Source: B. Refine Module · p. 2
- Training minimizes cross-entropy between predictions and ground truth, and inference generates comments with beam search.Source: B. Refine Module · p. 2
IV. EXPERIMENTS
On Java methods, the exemplar-based approach significantly outperforms the three neural baselines on BLEU and METEOR. The evaluation uses about 370,000 method-comment pairs from GitHub.
IV. EXPERIMENTS
- The Java corpus contains about 370k method-comment pairs, split into about 330k training, 20k validation, and 20k test pairs.Comments use the first sentence of each method's Javadoc.Source: §IV · p. 2
- Performance is evaluated with BLEU and METEOR, the metrics used by the compared comment-generation studies.Source: §IV · p. 2
- About 10% higher BLEU than HAD, with significant gains over vanilla seq2seq, DeepCom, and HAD on both BLEU and METEOR.Source: §IV · p. 2
V. CONCLUSION AND FUTURE WORK
The paper concludes that Retrieve and Refine significantly outperforms previous state-of-the-art methods on the Java dataset. It also identifies limitations in token-based retrieval and machine-translation-based evaluation.
V. CONCLUSION AND FUTURE WORK
- The method significantly outperforms previous state-of-the-art methods on the Java dataset.Source: §V · p. 2
- Retrieve finds a comment for similar code as an exemplar, while Refine uses the correlation between input and similar code to generate a comment under its guidance.Source: §V · p. 2
- Token-based code search is too simple to retrieve semantically similar code, motivating future use of advanced code-clone detection.Source: §V · p. 2
- Evaluation uses machine-translation metrics; the authors propose human evaluation to assess applicability in software development.Source: §V · p. 2