PaperlayerView exact revision on arXiv

Source-linked AI summary

A Neural Model for Generating Natural Language Summaries of Program Subroutines

Alexander LeClair, Siyuan Jiang, Collin McMillan

arXiv:1902.01954v1cs.SE

TL;DR

Source-code summarization tools still depend heavily on meaningful internal documentation, leaving undocumented or poorly documented code difficult to summarize. This paper introduces a neural model that separately processes code words and AST structure, achieving competitive standard-experiment performance and 9.5 BLEU when no code words are available.

  • Problem

    Existing documentation tools leave programmers to write text and examples, while neural code-summarization approaches largely rely on meaningful internal documentation such as identifier names.

  • Method

    The model separately accepts a word representation of code and an abstract syntax tree, allowing the two information sources to be processed differently.

  • Results

    The ensemble reaches 20.9 BLEU in the standard experiment, while the model achieves 9.5 BLEU versus 0 for every baseline in the AST-only challenge experiment.

  • Takeaways & Limitations

    The approach supports automatic documentation generation and helps summarize code when useful internal documentation is absent.

  • Takeaways & Limitations

    Results may differ on another dataset, and the random train/validation/test split was not evaluated with cross-validation.

Abstract

from arXiv · show

Source code summarization -- creating natural language descriptions of source code behavior -- is a rapidly-growing research topic with applications to automatic documentation generation, program comprehension, and software maintenance. Traditional techniques relied on heuristics and templates built manually by human experts. Recently, data-driven approaches based on neural machine translation have largely overtaken template-based systems. But nearly all of these techniques rely almost entirely on programs having good internal documentation; without clear identifier names, the models fail to create good summaries. In this paper, we present a neural model that combines words from code with code structure from an AST. Unlike previous approaches, our model processes each data source as a separate input, which allows the model to learn code structure independent of the text in code. This process helps our approach provide coherent summaries in many cases even when zero internal documentation is provided. We evaluate our technique with a dataset we created from 2.1m Java methods. We find improvement over two baseline techniques from SE literature and one from NLP literature.

I. INTRODUCTION

Source-code summarization aims to generate natural-language descriptions of program subroutines, but existing methods often depend on meaningful identifier names or comments. This paper separates code words from AST structure in a neural model and evaluates it in standard and no-code-word settings.

  • Source-code summaries describe subroutine behavior and support programmers’ understanding of a program’s role and behavior.
  • Existing documentation tools automate presentation but still leave programmers responsible for writing much of the text and examples.
  • Traditional template-based methods have largely given way to data-driven approaches inspired by neural machine translation.
  • Nearly all source-code summarization techniques rely on meaningful identifier names or comments, although code behavior is dictated by program structure and control or data flow.
  • The proposed model processes code words and an AST as separate inputs, using separate GRU encoders and attention mechanisms before predicting summaries word by word.
  • 2.1m Java methods form the evaluation dataset, which supports standard comparisons and a challenge experiment using only ASTs.

II. PROBLEM AND OVERVIEW

The paper targets automatic summaries of Java methods, motivated by sparse high-quality documentation and broader questions about representing source code for AI. It situates the approach among template-based and data-driven work using ASTs and sequence-to-sequence models.

  • The target is automatic generation of natural-language descriptions of Java methods like those used in JavaDocs.
  • Only about 10% of 51m Java methods in Sourcerer have any method summary, while about 4% meet the paper’s basic quality filters.
  • Automatic summaries could help programmers document code more quickly and understand code that lacks documentation.
  • The paper also seeks insight into how source code should be represented for AI within an ongoing debate about neural techniques in software engineering.
  • Related work includes heuristic and template-driven summarization as well as AI and data-driven approaches.
  • Hu et al.’s SBT method flattens ASTs and associates code words with structural context before feeding the representation to a sequence-to-sequence model.
  • SBT’s reported 38 BLEU is not directly comparable because its dataset was split by function, unlike the experiments described here.
  • Other data-driven work summarizes API-call sequences or uses custom code-word representations with off-the-shelf sequence-to-sequence models.

2) Heuristic/Template-based:

Attentional encoder-decoder models translate an input sequence into an output sequence one word at a time. Attention addresses the limitation of compressing the entire input into one vector by selecting input information relevant to each output.

  • B. Neural Machine Translation: An attentional encoder-decoder is the main architecture underlying most neural machine translation systems.
  • B. Neural Machine Translation: The encoder converts an arbitrary-length input sequence into a fixed-length vector, while the decoder converts that representation into another arbitrary-length sequence.
  • B. Neural Machine Translation: The sequence-to-sequence setup represents translation as converting one language sequence, such as English, into another, such as French.
  • B. Neural Machine Translation: During training, encoder-decoder models learn to predict output sentences one word at a time rather than all at once.
  • B. Neural Machine Translation: The training input includes the whole input sequence, previously generated output words, and the correct next word.
  • B. Neural Machine Translation: At inference time, each predicted word is fed back with the input to produce the next word until an end-of-sentence token is predicted.
  • B. Neural Machine Translation: The encoder’s single-vector strategy is limited because different input words have different importance for particular output words.
  • B. Neural Machine Translation: Attention addresses this limitation by replacing the single input representation with an attention mechanism between the encoder and decoder.

IV. OUR PROPOSED MODEL

The proposed model uses separate attentional encoders for code/text and AST representations, combining their context with decoder information to generate summaries one word at a time.

  • The model uses two encoders—one for code/text and one for AST data—and concatenates their attention outputs.Each encoder uses embedding and recurrent layers of equal size.
  • Three input layers represent the code/text sequence, comment sequence, and flattened AST sequence.Code/text and AST sequences use length 100, while comments use length 13; shorter sequences are padded and longer ones truncated.
  • Separate processing lets attention focus independently on code/text and AST inputs rather than treating them as one sequence.The design also preserves separate embedding spaces and avoids truncating one input because of the other.
  • The model applies separate attention mechanisms to code/text and AST encoder outputs, producing context vectors for each decoder position.Unlike vanilla sequence-to-sequence models, each output location receives its own context vector.
  • The decoder predicts one word at a time using its recurrent state together with the code/text and AST context vectors.The combined vectors are passed through a dense layer for each decoder-sequence element before next-word prediction.

C. Hardware Details

The model was implemented, trained, and tested using a CPU, 64 GB of RAM, and two GPUs with 16 GB of VRAM each.

  • Training required GPUs with 16 GB of VRAM because of the model’s large size.The hardware included one Xeon E5-1650v4 CPU, 64 GB RAM, and two Quadro P5000 GPUs.

V. CORPUS PREPARATION

The authors curated Java methods with JavaDoc summaries, removed likely auto-generated code, split data by project, and constructed standard and AST-only datasets.

  • The corpus began with over 51 million Java methods from more than 50,000 projects in the Sourcerer repository.The repository was selected because newer downloads substantially overlapped with it.
  • Around 4 million methods remained after filtering for JavaDoc comments, extracting first sentences, and retaining English comments.The first JavaDoc sentence was treated as the method-behavior summary.
  • Removing files marked as generated reduced the dataset to around 2 million methods and eliminated auto-generated cases found during manual inspection.The aggressive filter addressed duplicated code-comment patterns that could inflate BLEU scores.
  • Projects were randomly divided into training, validation, and test groups using 90%, 5%, and 5% of projects, respectively.Because projects differ in size, the resulting method split was 91%, 4.8%, and 4.2%.
  • SBT-AO preserves AST structure while replacing code words, except official Java API class names, with <OTHER>.This representation supports the challenge setting in which only AST information is available.
  • The standard dataset contains preprocessed source code, comments, and SBT-AO representations, while the challenge dataset contains comments and SBT-AO representations.A default SBT dataset was also retained for baseline comparisons.

VI. EVALUATION

The evaluation compares the proposed approach with baselines on standard and challenge datasets.

  • The evaluation compares the approach to baselines over standard and challenge datasets.The standard experiment uses training, validation, and testing partitions and follows common SE and NLP practice.

A. Research Questions

The paper asks how its approach compares with competitive methods when internal documentation is available and how it performs using only an AST when meaningful code words are unavailable.

  • RQ1 compares the approach with competitive methods in the standard setting, assuming internal documentation is available.
  • RQ2 evaluates performance in the challenge setting, where only an AST is available and internal documentation may be absent.
  • The challenge setting reflects bytecode-only, poorly documented, or deliberately obfuscated code, while an AST may still be extractable.

B. Baselines

The evaluation uses closely matched encoder-decoder baselines and a common training and measurement procedure to isolate input-representation differences.

  • The attendgru baseline is a generic attentional encoder-decoder without the proposed AST encoder and concatenation.
  • The SBT baseline uses Hu et al.’s code representation with the same GRU-based encoder-decoder, enabling an apples-to-apples input-format comparison.
  • The codenn baseline is adapted from a publicly available implementation originally designed for C# and SQL to function for Java.
  • All approaches use greedy inference, minimizing experimental variables and computation cost.
  • For RQ1, models train for 10 epochs, validation performance is measured after each epoch, and model selection uses validation results.Validation performance began degrading after five or six epochs.
  • Performance is reported using composite BLEU and BLEU1 through BLEU4, which measure similarity between predicted and reference summaries.

D. Threats to Validity

The evaluation reports performance patterns across standard and challenge settings while identifying dataset and splitting choices as threats to validity.

  • The evaluation may produce different results on another dataset, and it does not use cross-validation.Random train/validation/test splits mitigate but do not eliminate this risk.
  • Results: 19.6 BLEU versus 19.4 BLEU shows roughly equal standard-experiment performance for ast-attendgru and attendgru.
  • Threats to validity: The prior SBT result may differ because of architectural differences or project-overlapping splits that expose similar methods across training and test sets.
  • Results: Ast-attendgru and attendgru make orthogonal predictions, with different methods favoring each approach and roughly 33k methods favoring either one in the 91k test set.
  • Results: 9.5 BLEU is achieved by ast-attendgru in the challenge experiment, versus 0 for every baseline.

C. Explanation and Example

The examples show how AST information helps ast-attendgru summarize methods when code words are ambiguous or unavailable, while attention visualizations reveal different model behaviors.

  • Example 1: The tokenUrl method is summarized as setting the token URL by ast-attendgru, while attendgru and SBT produce less precise alternatives.The method assigns tokenUrl and returns this; ast-attendgru identifies the setter behavior despite the ambiguous method name.
  • Example 1: Attendgru attends strongly to the code word “return” when predicting “sets,” reflecting patterns learned from short getter methods.Its code/text heatmap shows high activation at position 14, corresponding to return.
  • Example 1: Ast-attendgru receives both code/text and AST inputs, allowing AST attention to supply structural evidence when the code/text signal is uncertain.Its code/text attention includes the start-of-sentence token as a likely uncertainty signal, while a separate AST attention mechanism focuses on structural regions.
  • Example 1: All AST-input models correctly chose “sets,” but SBT could not determine what was being set.The authors attribute this difference to SBT blending textual and structural information into one input representation.
  • Example 1: The challenge experiment reduced ast-attendgru’s first-word accuracy to 33%, compared with 52% in the standard experiment.Even under this difficulty, the model characterized one method as setting a property and used an unknown token when it could not identify which property.
  • Example 2: In the disconnect example, ast-attendgru performs best in the no-code-words setting by using AST patterns associated with close and cleanup behavior.Similar try/catch structures, method calls, and assignments to null or false appeared in training examples with summaries describing cleanup actions.

VIII. ENSEMBLE DECODING AND FUTURE WORK

The paper combines attendgru and ast-attendgru through ensemble decoding by averaging their output vectors during prediction. This simple combination reaches 20.9 BLEU, exceeding either model alone and motivating more advanced ensemble and input-specific techniques.

  • Ensemble decoding: Ensemble decoding averages the two models’ output vectors element-wise and selects the maximum value of the combined vector at each prediction step.The models therefore cooperate word by word rather than selecting one model for the entire sentence.
  • Motivation: The ensemble is intended to combine attendgru’s strengths when code/text words are clear with ast-attendgru’s strengths when those words are unclear.This complementary behavior is the motivation for combining the models during prediction.
  • Results: 20.9 BLEU is achieved by the ensemble, compared with 19.6 for ast-attendgru and 19.4 for attendgru.The authors describe this as more than a full BLEU-point improvement from a relatively simple procedure.
  • Future work: The result motivates future work on advanced ensemble decoding, network optimization, and different processing techniques for each input type.Suggested directions include predicting when to use one model or another, dropout, parameter tuning, and input-specific data processing.
Loading 1902.01954v1…