PaperlayerView exact revision on arXiv

Source-linked AI summary

Retrieval-Augmented Generation for Code Summarization via Hybrid GNN

Shangqing Liu, Yu Chen, Xiaofei Xie, Jingkai Siow, Yang Liu

arXiv:2006.05405v5cs.LGcs.AI

TL;DR

Automatic code summarization is challenged by the gap between code and natural language, program complexity, and trade-offs between retrieval and generation methods. The paper proposes retrieval-augmented generation with hybrid static–dynamic graph reasoning and evaluates it on a new large C benchmark. The approach achieves state-of-the-art performance, improving existing approaches by 1.42, 2.44 and 1.29 on BLEU-4, ROUGE-L and METEOR.

  • Problem

    Code summarization must bridge heterogeneous code and natural-language representations, while retrieval methods generalize poorly and generation methods cannot exploit similar retrieved examples.

  • Method

    HGNN combines retrieval-based augmentation with hybrid message passing over a static code property graph and an attention-based dynamic graph.

  • Results

    The approach achieves state-of-the-art performance, improving existing approaches by 1.42, 2.44 and 1.29 on BLEU-4, ROUGE-L and METEOR, respectively.

  • Takeaways & Limitations

    The framework combines complementary retrieval-based and generation-based capabilities while modeling both local and global code structure.

  • Takeaways & Limitations

    Most GNN encoders capture only local neighborhood information because small-hop message passing is used to avoid over-smoothing.

Abstract

from arXiv · show

Source code summarization aims to generate natural language summaries from structured code snippets for better understanding code functionalities. However, automatic code summarization is challenging due to the complexity of the source code and the language gap between the source code and natural language summaries. Most previous approaches either rely on retrieval-based (which can take advantage of similar examples seen from the retrieval database, but have low generalization performance) or generation-based methods (which have better generalization performance, but cannot take advantage of similar examples). This paper proposes a novel retrieval-augmented mechanism to combine the benefits of both worlds. Furthermore, to mitigate the limitation of Graph Neural Networks (GNNs) on capturing global graph structure information of source code, we propose a novel attention-based dynamic graph to complement the static graph representation of the source code, and design a hybrid message passing GNN for capturing both the local and global structural information. To evaluate the proposed approach, we release a new challenging benchmark, crawled from diversified large-scale open-source C projects (total 95k+ unique functions in the dataset). Our method achieves the state-of-the-art performance, improving existing methods by 1.42, 2.44 and 1.29 in terms of BLEU-4, ROUGE-L and METEOR.

1 INTRODUCTION

Automatic code summarization must bridge heterogeneous natural-language and code representations while handling complex program structure. HGNN addresses these challenges by combining retrieval augmentation with hybrid static–dynamic graph reasoning, supported by a new C benchmark and state-of-the-art results.

  • Code summarization is difficult because code and natural-language summaries differ in tokens, synonyms, and structure, while program logic is complex.
  • Retrieval-based methods exploit similar code but generalize poorly to dissimilar programs, whereas generation-based methods generalize better but cannot use retrieved examples.
  • Existing GNN encoders mainly pass messages within small k-hop neighborhoods, limiting their ability to capture global interactions and incurring high cost when deepened.
  • HGNN combines retrieval-based augmentation with a hybrid GNN that fuses a static code property graph and an attention-based dynamic graph.
  • The authors release a challenging C benchmark containing 95k+ functions from popular and diversified projects.
  • 1.42, 2.44 and 1.29 improvements over existing approaches are reported for BLEU-4, ROUGE-L and METEOR, respectively.

2 HYBRID GNN FRAMEWORK

HGNN combines retrieval augmentation with static and dynamic graph processing to generate code summaries. Its framework constructs CPG-based representations, augments them with similar code and summaries, and uses global attention alongside local graph structure.

  • Framework overview: HGNN comprises retrieval-augmented static graph construction, attention-based dynamic graph construction, hybrid message passing, and an attention-based LSTM decoder.The decoder generates the final natural-language summary from representations produced by the graph components.
  • Static graph construction: A function is represented as a multi-edged Code Property Graph whose nodes encode AST-derived sequences and types, while edges capture structures such as Flow To, Control, Define/Use, and Reach.The initial node matrix contains one embedding per AST node.
  • Retrieval-based augmentation: The retrieval mechanism selects the most similar code-summary pair, injects retrieved code semantics into the current graph through attention, and weights retrieval features by similarity.The similarity score is computed from normalized text edit distance, and the augmented representation merges retrieved and original node features.
  • Retrieval-based augmentation: Retrieved summaries are encoded with a BiLSTM and concatenated with the graph encoding, after multiplying their hidden states by the retrieval similarity score.This provides the decoder with both graph representations and retrieved-summary information.
  • Dynamic graph construction: The dynamic graph uses structure-aware global attention over every node pair, including pairs without static edges, and row-normalizes its dense adjacency matrix for message passing.Edge embeddings and augmented node representations contribute to the attention-based adjacency construction.
  • Hybrid message passing: Hybrid Message Passing propagates information on both the retrieval-augmented static graph and the attention-based dynamic graph.The design combines local neighborhood information with global node dependencies.

3 EXPERIMENTS

Experiments evaluate HGNN against retrieval-, sequence-, and graph-based baselines on C and Python code-summarization datasets. HGNN outperforms existing methods overall, while ablations support retrieval augmentation and hybrid static–dynamic graph modeling.

  • 3.1 SETUP: 95k+ unique function-summary pairs form the deduplicated CCSD benchmark, with separate in-domain and out-of-domain evaluations.The dataset contains 84,316 training, 4,432 in-domain validation, and 4,203 in-domain test functions; the supplied passage truncates the remaining split details.
  • 3.2 COMPARISON WITH THE BASELINES: HGNN outperforms existing methods on both in-domain and out-of-domain datasets, indicating strong generalization performance.Retrieval-based methods are competitive on in-domain BLEU-4 but perform worse on ROUGE-L and METEOR and do not perform well out of domain.
  • 3.2 COMPARISON WITH THE BASELINES: 14.01 BLEU-4, 30.89 ROUGE-L and 14.50 METEOR exceed Rencos by 1.42, 2.44 and 1.29, respectively, on the overall CCSD dataset.These are the reported overall-dataset results in Table 1.
  • 3.3 ABLATION STUDY: Removing retrieval augmentation, the static graph or the dynamic graph decreases performance, supporting the contributions of all three components.Without static graph performs worse than without dynamic graph on ROUGE-L and METEOR, while BLEU-4 is higher without dynamic graph.
  • 3.3 ABLATION STUDY: Combined code- and summary-based augmentation reaches 14.01, 30.89 and 14.50 on BLEU-4, ROUGE-L and METEOR, outperforming either augmentation alone.Summary-based augmentation reaches 13.76, 30.59 and 14.11, whereas code-based augmentation reaches 10.22, 27.54 and 12.49.
  • 3.4 HUMAN EVALUATION: Human evaluation finds HGNN summaries more relevant to source code and more similar to ground-truth summaries than the compared methods.The evaluation compares HGNN with NNGen, Transformer, Rencos and SeqGNN on relevance and similarity.
  • 3.5 CASE STUDY: Qualitative examples show HGNN captures a self-defined struct variable and the functionality of releasing a cedar reference more precisely than baselines.The first example produces a token object for variable p, while other models produce only a string.
  • 3.6 EXTENSION ON THE PYTHON DATASET: On PCSD, HGNN improves over the best competing results by 0.40 BLEU-4, 3.70 ROUGE-L and 1.41 METEOR.The PCSD ablation results also support contributions from both static and dynamic graphs.

4 RELATED WORK

Prior code-summarization methods span retrieval, sequence-to-sequence, AST-based, and graph-based approaches. Graph representations incorporate richer code structure, while existing sequential approaches can miss code semantics.

  • Source Code Summarization: Early code summarization used information retrieval, while later methods generated summaries with attentional Seq2Seq models or AST representations.These approaches differ in whether they retrieve existing summaries or generate summaries from sequential code representations.
  • Source Code Summarization: Sequential models can ignore rich code semantics when source programs contain structured relationships beyond token order.The related-work passage contrasts sequential models with graph-based representations.
  • Graph Neural Networks: GNN-based approaches represent programs as graphs containing structures such as control flow and program dependencies.Graph representations were introduced to better learn program semantics and achieved promising performance over sequence-based approaches.

5 CONCLUSION AND FUTURE WORK

The paper concludes that HGNN combines retrieval augmentation with hybrid static–dynamic graph message passing for automatic code summarization. It reports substantial improvements and identifies extensions involving API knowledge, code augmentation and robustness evaluation.

  • Conclusion: HGNN combines retrieval-based and generation-based approaches through a retrieval-augmented mechanism.The framework is presented as general-purpose for automatic code summarization.
  • Conclusion: Hybrid message passing over static and dynamic graphs is designed to capture global semantics among code nodes.The static and dynamic graph representations are combined within the GNN.
  • Conclusion: Evaluation shows that HGNN substantially improves state-of-the-art techniques.The conclusion does not provide additional metric values beyond this overall comparison.
  • Future Work: Future work will add API knowledge, explore more code-based augmentation techniques and evaluate robustness of the trained model.These directions are explicitly listed by the authors.

A DETAILS ON CODE PROPERTY GRAPH

The code property graph combines syntactic, control-flow and dependency information to represent program semantics. Its components encode syntax, possible execution order and data or control dependencies.

  • Code Property Graph: A Code Property Graph combines different edge types on an AST to represent program semantics.The supplied passage names Flow to and Control edges as examples.
  • Abstract Syntax Tree: An AST preserves syntactic information while omitting details irrelevant to program semantics.AST nodes contain code sequences and type attributes, with child-parent relations represented by black arrows.
  • Control Flow Graph: A CFG represents statement execution order, including possible orders and the conditions required for execution.Statements are modeled as nodes alongside designated entry and exit nodes; Flow to edges connect control-flow relations.
  • Program Dependency Graph: A PDG represents data and control dependencies between program statements.A Reach edge connects a variable definition to a later use, exemplified by variable b.

B DETAILS ON BASELINE METHODS

The study compares the proposed approach with retrieval-, sequence-, and graph-based baselines, reproducing published methods when source code is available and reimplementing others.

  • Baselines are grouped into retrieval-based, sequence-based, and graph-based approaches.

B.1 RETRIEVAL-BASED APPROACHES

Prior code summarization approaches include retrieval-based, generation-based, reinforcement-learning, dual-training, and graph-neural-network methods. Rencos most closely combines retrieval with generation, while this work differs by using retrieval augmentation during training.

  • TF-IDF retrieves summaries from the most similar programs using cosine similarity over term-frequency and inverse-document-frequency vectors.
  • NNGen retrieves top-k similar code snippets with bag-of-words representations and prioritizes their summaries by BLEU-4 scores.
  • Generation-based approaches include attention-based Seq2Seq, Transformer self-attention, reinforcement learning with ASTs and sequences, and dual training of summarization and generation.
  • Rencos uses a pretrained Seq2Seq model at test time, conditioning summary generation jointly on source code and retrieved similar code.
  • Graph-based baselines apply GCNs, GATs, or GGNNs to code property graphs alongside sequence encoders for source-code summarization.
  • The implementation uses 40,000-word embeddings, 512-dimensional representations, one GNN hop, Adam optimization, batch size 64, and beam width 5.

D DETAILS ON DATA PREPARATION

The paper introduces a C-language code summarization dataset built from popular open-source repositories and evaluates generated summaries with retrieved examples. The dataset includes randomly split data and an out-of-domain test set.

  • The dataset addresses the lack of high-quality C code summarization resources by extracting separate function-summary pairs from popular GitHub repositories, including Linux and QEMU.
  • Table 5 presents additional generated summaries on the CCSD test set together with retrieved code and summaries.
  • The random train/validation/test split contains 84,316/4,432/4,203 examples, while the out-of-domain test set contains 2,330 examples.
  • Joern was used to construct the code property graph for the dataset examples.
  • The authors report that HGNN generates higher-quality summaries in the Table 5 examples using their retrieval-augmented approach.
Loading 2006.05405v5…