PaperlayerView exact revision on arXiv

Source-linked AI summary

Improved Code Summarization via a Graph Neural Network

Alexander LeClair, Sakib Haque, Lingfei Wu, Collin McMillan

arXiv:2004.02843v2cs.SEcs.CL

TL;DR

Source code summarization benefits from structural information, but prior work did not combine a graph-based AST encoder with a separate source-code sequence encoder. This paper combines ConvGNN-based AST encoding with sequence modeling and reports a 4.6% BLEU improvement over recent baselines.

  • Problem

    Prior source-code summarization models used AST structure without providing the source-code sequence as a separate model input, motivating evaluation of their combination.

  • Method

    The model combines a GNN-based encoder for each subroutine’s AST with an RNN-based encoder for the subroutine’s source-code sequence.

  • Results

    4.6% BLEU improvement was demonstrated on a large published dataset compared with recent baselines.

  • Takeaways & Limitations

    Combining source-code tokens with AST structure and ConvGNNs helps the model learn when to copy source tokens and improves AST token representations.

  • Takeaways & Limitations

    Automated BLEU and ROUGE scores may not represent human judgments of summary quality.

Abstract

from arXiv · show

Automatic source code summarization is the task of generating natural language descriptions for source code. Automatic code summarization is a rapidly expanding research area, especially as the community has taken greater advantage of advances in neural network and AI technologies. In general, source code summarization techniques use the source code as input and outputs a natural language description. Yet a strong consensus is developing that using structural information as input leads to improved performance. The first approaches to use structural information flattened the AST into a sequence. Recently, more complex approaches based on random AST paths or graph neural networks have improved on the models using flattened ASTs. However, the literature still does not describe the using a graph neural network together with source code sequence as separate inputs to a model. Therefore, in this paper, we present an approach that uses a graph-based neural architecture that better matches the default structure of the AST to generate these summaries. We evaluate our technique using a data set of 2.1 million Java method-comment pairs and show improvement over four baseline techniques, two from the software engineering literature, and two from machine learning literature.

1 INTRODUCTION

Automatic code summarization has shifted toward neural methods, but treating code as a sequence overlooks its structural relationships. This paper combines graph-based AST encoding with source-code sequence encoding and reports improved summarization performance.

  • Motivation: Source code is a complex web of interacting components, making graph or tree representations better suited than purely sequential representations.Code contains relationships among classes, routines, statements, and identifiers that sequences do not directly represent.
  • Related approaches: Neural code summarization approaches increasingly incorporate AST information through annotations, separate inputs, AST paths, or graph neural networks.Prior work explored several ways to augment or replace sequential representations with structural information.
  • Research gap: The literature had not thoroughly examined how GNN-based AST representations improve code summarization or why they help.Existing evidence indicated performance gains, but their degree and underlying reasons remained insufficiently understood.
  • Contribution: The proposed model combines a GNN-based AST encoder with an RNN-based source-code sequence encoder for summarizing program subroutines.The approach adapts graph2seq with software-engineering-specific modifications.
  • Contribution: 4.6% BLEU improvement was reported on a large published dataset compared with recent baselines.The paper also uses explainable-AI techniques to investigate when and why the approach performs better.

2 PROBLEM, SIGNIFICANCE, SCOPE

The paper targets automatic generation of short natural-language summaries for program subroutines. Such summaries can support documentation, code comprehension, and search, while graph representations address challenges in more complex methods.

  • Scope: The task is to generate a short natural-language description from the source code of a program subroutine.The scope is automatic summarization of program subroutines.
  • Problem: Current summarization systems have succeeded mainly on shorter methods with simpler code and comments, but struggle with more complex structures or language.The paper identifies graph representations as a promising direction for this limitation.
  • Significance: Automatic documentation could reduce programmers’ reading and writing effort while supporting code comprehension and search.The cited benefits include decreased time to understand code, improved comprehension, and greater searchability.

3 BACKGROUND AND RELATED WORK

Related work progresses from heuristic and sequence-based summarization toward models that incorporate ASTs, paths, and graphs. The paper’s background motivates ConvGNNs as a way to propagate structural information across AST nodes.

  • Source-code summarization: Source-code summarization research spans heuristic or template-driven methods and newer AI- or data-driven methods.The paper frames recent work as increasingly dominated by neural approaches.
  • Related approaches: Multi-input sequence models separate structural AST information from source-code tokens to help learn code structure.LeClair et al. used an SBT sequence with identifiers removed alongside source-code tokens.
  • Related approaches: Recent models represent code with source sequences, ASTs, flattened ASTs, GNNs, and AST paths, sometimes combining multiple representations.Table 1 categorizes data, architectures, and approaches by these attributes.
  • Neural background: The encoder-decoder architecture maps encoded inputs to generated output sequences using recurrent layers and attention mechanisms.Attention learns which input tokens are useful for predicting each output token.
  • Graph neural networks: ConvGNNs use message passing to encode spatial information and allow nodes to aggregate information across multiple hops.Each hop extends the structural neighborhood contributing to a node’s representation.

4 APPROACH

The approach encodes source tokens sequentially and AST nodes relationally, attends to both representations, and decodes a natural-language summary token by token. Its central design replaces a flattened AST encoder with ConvGNN layers while retaining separate source-code input.

  • Pipeline: The model embeds source-code tokens and AST node tokens, encodes them with an RNN and ConvGNN respectively, then predicts the summary sequence.The pipeline consists of embedding, dual encoding, attention, decoding, and next-token prediction.
  • Model architecture: The architecture has four inputs: source-code tokens, AST nodes, AST edges, and the predicted sequence so far.Source-code and AST nodes share an embedding because their vocabularies substantially overlap.
  • Attention and decoding: Separate attention mechanisms connect the decoder to the source code and AST, and their outputs are concatenated into a context vector for next-token prediction.The combined context is passed through a dense layer to predict each next token.
  • AST encoder: A ConvGNN replaces the flattened AST encoder and learns AST node representations from neighboring nodes and edges.The best-performing implementation uses one ConvGNN layer per hop, with two hops in the best model.
  • Data: The experiments use a published dataset containing 2.1 million Java method-comment pairs.The dataset was provided in filtered raw and tokenized versions.

5 EXPERIMENT DESIGN

The experiment evaluates a graph-based AST encoder combined with source-code sequence input against established baselines, while also examining hop size, interpretability, and validity threats.

  • Research Questions: RQ1 evaluates performance against baselines, RQ2 tests graph hop sizes, and RQ3 investigates whether performance differences relate to ConvGNN use.The research questions separately address comparative performance, hop-size effects, and evidence concerning the ConvGNN.
  • Methodology: The study compares the proposed approach with baselines using established source-code summarization and neural machine translation metrics.BLEU and ROUGE are used for quantitative evaluation, with visualizations of model weights supporting interpretability analysis.
  • Methodology: Five ConvGNN models vary hop size across 1, 2, 3, 5, and 10 while other hyperparameters remain frozen.The models use source code and AST inputs and are evaluated with the same best-of-ten training strategy.
  • Baselines: The four baselines represent flattened-AST, AST-path, and graph-neural-network encoding schemes across different architectures.All baselines use AST information, but they differ in how that structure is supplied to the model.
  • Baselines: The study excludes one reinforcement-learning baseline because its extra variables complicate attribution, its encoding was superseded, and its results could not be reproduced.Heuristic approaches are also excluded because prior work had already reported that data-driven methods outperform them on automated metrics.
  • Threats to Validity: Automated metrics may not represent human judgment, while limited hardware prevented extensive hyperparameter optimization.The authors mitigate these concerns through example-based analysis and consistent hyperparameter settings and baseline reproduction procedures.

6 EXPERIMENT RESULTS

The experiments evaluate model configurations, hop sizes, and attention behavior. The graph-based model combining source-code and AST inputs generally outperforms flattened-AST and graph-based baselines, while attention examples illustrate how structural information contributes.

  • RQ1: Quantitative Evaluation: 19.93 BLEU-A and 56.08 ROUGE-LCS made code+gnn+BiLSTM the highest-performing configuration.It outperformed the nearest graph-based baseline by 4.6% BLEU-A and 0.06% ROUGE-LCS, and the flattened-AST baseline by 5.7% BLEU-A and 12.72% ROUGE-LCS.
  • RQ3: Graph AST Contribution: Source-code and AST attention can align on a token, supporting direct copying of source tokens into generated summaries.When the attentions differ, the model often relies more on AST attention for the next-token prediction.
  • RQ1: Quantitative Evaluation: The code+gnn+BiLSTM model exceeded code+gnn+GRU by 0.05 BLEU-A and 0.3 ROUGE-LCS.Because the BLEU-A difference was only 0.05 BLEU-A, subsequent tests used code+gnn+GRU; BiLSTM also incurred significantly higher computational cost.
  • RQ2: Hop Size Analysis: Two ConvGNN hops produced the best overall AST performance for the evaluated model configurations.Two hops performed 1.5% better than three hops and 2.8% better than ten hops.
  • RQ3: Graph AST Contribution: ConvGNN attention focuses on more specific AST tokens, whereas flattened-AST attention generally covers larger structural regions.The examples associate this finer-grained attention with learned node representations and improved decisions about copying words from source code.
  • RQ3: Graph AST Contribution: In one example, code+gnn+GRU predicted the correct token ‘first’ while ast-atendgru predicted ‘specified’.The models attended to the same important token in different contexts, while the graph-based model focused on more specific structural information.

7 DISCUSSION AND FUTURE WORK

The discussion attributes the gains to graph-based AST processing and separate source-code encoding. It also identifies ensembling and method-specific feature combinations as directions for future work.

  • Discussion: ConvGNN layers outperform a flattened AST representation in the evaluated summarization setting.The paper attributes this to improved AST node embeddings that combine structural elements.
  • Discussion: A separate source-code encoder lets the model combine source-code and AST attention as a copy mechanism.Aligned attention can directly copy source tokens, while disagreements shift reliance toward AST information.
  • Discussion: ConvGNN-based node embeddings represent AST tokens as combinations of structural elements.The examples show more specific AST attention than the flattened-AST model’s broader structural attention.
  • Future Work: Ensembling these models is identified as a future direction because prior work found performance improvements from ensembles.The paper also suggests combining source-code features may help different types of methods.

8 CONCLUSION

The paper presents a neural architecture combining source-code tokens with ConvGNN-encoded Java-method ASTs to generate summaries, reporting improved aggregate BLEU-A over graph-based and flattened-AST approaches.

  • The model combines source-code tokens with ConvGNNs that encode a Java method’s AST to generate natural-language summaries.The architecture uses sequence and graph representations as separate inputs.
  • The combined inputs help the model learn when to copy source-code tokens and produce better AST-token representations.
  • Over 4.6% higher BLEU-A than other graph-based approaches and 5.7% higher BLEU-A than flattened-AST approaches were reported.
  • The paper analyzes how ConvGNN layers contribute to the performance increase and discusses implications for future work.

9 REPODUCIBILITY

The paper points to an online repository for its models, source code, and data, and illustrates attention visualizations and generated summaries for a Java indexOf method.

  • Reproducibility: Models, source code, and data used in the work are available in an online repository.The passage provides a repository URL.
  • Example summaries: For Method ID 25584536, the reference and code+gnn+GRU summaries identify the first occurrence index, while ast-attendgru identifies an object’s index in the list.
  • Attention visualization: Example 3 visualizes source-code and AST attention for code+gnn+GRU and ast-attendgru.
Loading 2004.02843v2…