PaperlayerView exact revision on arXiv

Source-linked AI summary

PassSum: Leveraging Paths of Abstract Syntax Trees and Self-Supervision for Code Summarization

Changan Niu, Chuanyi Li, Vincent Ng, Jidong Ge, Liguo Huang, Bin Luo

arXiv:2610.15435v1cs.SE

TL;DR

Neural code summarizers face inefficient AST encoding and dependence on costly labeled data. PassSum combines path-based AST encoding with self-supervised pre-training on unlabeled methods. The authors report that it outperforms state-of-the-art summarizers across training-data amounts, while its AST integration improves PassSum's performance.

  • Problem

    Neural code summarizers rely on large labeled datasets that are costly to build and use inefficient AST encodings, limiting their suitability when labeled data is scarce.

    Source: §1 · p. 2, §2.2 · p. 3

  • Method

    PassSum combines root-to-terminal AST paths and code-sequence input in a Transformer summarizer, and uses three self-supervised objectives to pre-train on unlabeled methods.

    Source: §1 · p. 2

  • Results

    PassSum outperforms state-of-the-art code summarizers regardless of training-data amount; its Path Embedding outperforms code2seq-like encoding and SBT.

    Source: §1 · p. 2, §8 · p. 17

  • Takeaways & Limitations

    As training data decreases, pre-training makes PassSum's performance deteriorate less quickly, and incorporating ASTs allows them to improve PassSum's performance.

    Source: §8 · p. 17

    PassSum's hyperparameters were set empirically rather than tuned experimentally, so other settings may yield better results.

    Source: §6 · p. 16

Abstract

from arXiv · show

Code summarization is to provide a high-level comment for a code snippet that typically describes the function and intent of the given code. Recent years have seen the successful application of data-driven code summarization. To improve the performance of the model, numerous approaches use Abstract Syntax Trees (ASTs) to represent the structural information of the code, which is considered by most researchers to be the main factor that distinguishes code from natural language. Then, such data-driven methods are trained on large-scale labeled datasets to obtain a model with strong generalization capabilities that can be applied to new examples. Nevertheless, we argue that state-of-the-art approaches suffer from two key weaknesses: (1) inefficient encoding of ASTs; (2) reliance on a large labeled corpus for model training. As a result, such drawbacks lead to (1) oversized model, slow training, information loss and instability; (2) inability to be applied to programming languages with only a small amount of labeled data. In light of these weaknesses, we propose PassSum, a code summarization approach that addresses the aforementioned weaknesses via (1) a novel input representation which contains an efficient AST encoding method; (2) introducing three pre-training objectives and pre-training our model with a large amount of (easy-to-obtain) unlabeled data under the guidance of self-supervised learning. Experimental results on code summarization for Java, Python and Ruby methods demonstrate the superiority of PassSum to state-of-the-art methods. Further experiments demonstrate that the input representation we use has both temporal and spatial advantages in addition to performance leadership. In addition, pre-training is also shown to make the model more generalizable with less labeled data, and also to speed up the convergence of the model during training.

1 INTRODUCTION

Code summarization generates natural-language descriptions of source code, but current neural approaches face inefficient AST encoding and costly reliance on labeled data. PassSum pairs a path-based AST representation with self-supervised pre-training, and the authors report stronger performance and input-representation efficiency.

1 INTRODUCTION

  • Code summarization generates concise natural-language comments describing a method's overall actions, and comments can support code understanding, reuse, and maintenance.Source: §1 · p. 1
  • Existing neural summarizers depend on large, high-quality labeled datasets that are costly to build, while their AST encoding is inefficient.Code comments also support comprehension and maintenance, but code and comments may not co-evolve and documentation may become outdated.Source: §1 · pp. 1–2
  • PassSum uses root-to-terminal AST paths with self-attention instead of code2vec-style path-contexts, and adds the code sequence to capture lexical as well as syntactical information.The path input is reduced from P(n, 2) to n.Source: §1 · p. 2
  • PassSum is pre-trained with three self-supervised objectives on unlabeled code to improve performance robustness when labeled training data is limited.The paper describes unlabeled methods as source-code methods without comments.Source: §1 · p. 2
  • PassSum outperforms state-of-the-art code summarizers regardless of the amount of training data, and its input representation demonstrates time and space efficiency.Source: §1 · p. 2

RIZATION TECHNIQUES

The section identifies inefficient AST encoding and dependence on labeled data as shortcomings of current code summarizers, then proposes path encoding and self-supervised pre-training to address them.

2.1 Inefficient Encoding of ASTs

  • Tree and graph neural encoders can be inefficient or costly, and GNNs also face long-dependency, over-smoothing, over-squashing, and variable-shape challenges.Tree-LSTMs repeat shared-node inputs without communication between branches, while RvNNs are computationally expensive due to recursion.Source: §2.1 · p. 3
  • Subtree encoding can obscure detailed correlations between terminal nodes in different parts of an AST, including distinct starts that share an endpoint.When two critical paths start at different leaves in one subtree and end at the same leaf, encoding that subtree as one vector makes them difficult to distinguish.Source: §2.1 · p. 3
  • SBT linearization can produce sequences more than three times the source-code length, potentially increasing training time and reducing spatial efficiency.The paper reports evaluating these effects in Section 5.2.2.Source: §2.1 · p. 3
  • Path-context encoding creates n(n −1) ordered terminal-node pairs, so code2vec commonly samples k contexts, which can make predictions less stable.The cited study reports that snippets reduced to two or fewer statements can retain the key features models use to predict labels.Source: §2.1 · p. 3

2.2 Reliance on a Large Amount of Labeled Data

  • Large labeled training sets are often unavailable: most code lacks comments, and existing comments may be outdated, limiting application to languages with little labeled data.The section identifies broad performance across programming languages as a research challenge.Source: §2.2 · p. 3
  • The proposed response to both shortcomings is path encoding together with self-supervised pre-training.The section introduces these as the approaches for addressing inefficient AST encoding and reliance on labeled data.Source: §2.2 · p. 3

3 PRELIMINARIES

The preliminaries review approaches to representing code structure and self-supervised pre-training. They introduce root-to-terminal AST paths and code-focused pre-training tasks as the paper’s choices.

3.1 Encoding Structural Information of Source Code

  • Prior structural encoders feed ASTs to tree or graph networks, split them into subtrees, or linearize them into sequences and path-contexts.The reviewed examples include Tree-LSTMs, RvNNs, GNNs, subtree-based methods, traversal-based sequences, and terminal-node path-contexts.Source: §3.1 · p. 4
  • The paper instead serializes paths from the AST root to each terminal node, which it says shortens the representation, reduces information loss, and improves training speed and space use.The authors refer to experiments in Section 5 as demonstrating these advantages.Source: §3.1 · p. 4

3.2 Self-supervised Pre-training

  • Self-supervised pre-training derives training labels automatically from the input, with masked language modeling and next-sentence prediction given as NLP examples.The section presents BERT as an example of the pre-training-and-fine-tuning paradigm.Source: §3.2 · p. 4
  • The paper designs pre-training tasks for code- and input-related features rather than directly adopting existing NLP pre-training tasks.The preliminaries situate this choice alongside code pre-training work using source code, text, or code structure.Source: §3.2 · p. 4

4 APPROACH

PassSum combines code tokens with AST Path Embedding in a Transformer, representing paths from the AST root to terminal nodes. Self-attention learns relationships among these paths, while their order in the source code is encoded positionally.

4 APPROACH

  • PassSum is a Transformer-based code summarizer that uses code tokens and path embeddings as its two input components.The architecture description specifies a 78M-parameter model.Source: §4 · p. 4, §4.2 · p. 4, §4.1 · p. 4

4.3.2 Step 2: Embed Paths

  • Each path contains its AST node sequence and terminal-node identifier, which separate GRUs encode before concatenation and projection into a path vector.Identifiers are split into subtokens before encoding.Source: §4.3.1 · p. 5, §4.3.2 · pp. 5–6

4.3.1 Step 1: Extract Paths

  • Path extraction yields n root-to-terminal paths instead of code2vec’s n(n −1) paths, reducing input size without random sampling.The paths are sorted by the source-code positions of their terminal nodes, which the authors say increases model stability.Source: §4.3.1 · p. 5

4.3.2 Step 2: Embed Paths

  • Path Embedding returns one vector per path and adds positional information for terminal-node order, which many current methods treat as unordered.The authors describe this position information as important for program understanding.Source: §4.3.2 · p. 6

4.2 Input Representation

  • PassSum’s Transformer uses self-attention to learn relationships between path representations, avoiding information loss that may result from random path sampling.The model input includes code-token vectors, a [SEP] embedding, and path vectors.Source: §4.2 · pp. 4–5, §4.3.2 · p. 7

4.3.2 Step 2: Embed Paths

  • Path Embedding does not explicitly represent variable dependencies such as define-use links; the authors leave supplying them explicitly to future work.They believe self-attention may find valuable dependencies from path representations.Source: §4.3.2 · p. 7

4.4 Pre-training Objectives

PassSum is pre-trained on uncommented source code using three self-supervised objectives: predicting masked code tokens, shuffled path order, and masked method names. The resulting model is then fine-tuned for code summarization.

4.4 Pre-training Objectives

  • The paper frames self-supervised pre-training as a way to use large amounts of unlabeled data, with targets generated from the input.Its unlabeled pre-training data consists solely of uncommented source code.Source: §4.4 · p. 7

4.4.1 Objective 1: Masked Language Model

  • Masked Language Model (MLM) masks 15% of code tokens and trains the encoder to predict those tokens.MLM is applied only to the code-token portion of the input.Source: §4.4.1 · p. 7

4.4.2 Objective 2: Path Order Prediction

  • Path Order Prediction (POP) shuffles paths and trains the encoder to predict their original order in the code.The order reflects the source-code positions of the paths’ terminal nodes.Source: §4.4.2 · p. 7

4.4.3 Objective 3: Method Name Generation

  • Method Name Generation (MNG) trains the decoder to generate a method’s masked name, using subtokens as a short sequence.The authors use method names as labels that reflect a method’s function and intent.Source: §4.4.3 · pp. 7–8

4.4 Pre-training Objectives

  • Pre-training combines three objectives with equal probability before fine-tuning on labeled code-summary data for up to 200 epochs.The fine-tuning objective is cross-entropy loss.Source: §4.4 · p. 7, §4.4.3 · p. 8, §4.5 · p. 8

5 EVALUATION

The evaluation is organized around PassSum’s benchmark performance, input representation, Path Embedding efficiency, pre-training effects, and human evaluation.

5 EVALUATION

  • The evaluation asks whether PassSum is effective against state-of-the-art baselines on code summarization.This is RQ1.Source: §5 · p. 8
  • RQ2 examines whether PassSum’s input representation settings provide a performance advantage.RQ3 asks whether Path Embedding has time and space efficiency advantages over other structural encodings.Source: §5 · p. 8
  • RQ4 evaluates pre-training’s effects on performance, robustness, and convergence speed.RQ5 compares PassSum with the best baseline through human evaluation.Source: §5 · p. 8

5.1 Experimental Setup

The experimental setup uses public labeled code-summary datasets, unlabeled code, automatic metrics, and human evaluation to compare PassSum with non-pretrained and pretrained baselines.

5.1.1 Dataset

  • The labeled datasets are JSCD for Java and PCSD for Python, and the experiments also use CodeSearchNet, which spans six programming languages.The unlabeled data comes from Java-small and a monolingual Python dataset.Source: §5.1.1 · p. 8

5.1.2 Pre-processing

  • Preprocessing splits code and comment tokens into subtokens and removes selected constructors, methods, and duplicate instances.The paper uses tree-sitter to obtain code ASTs and cleans comment markup.Source: §5.1.2 · pp. 8–9

5.1.3 Metrics

  • Automatic evaluation uses BLEU, METEOR, and ROUGE-L, alongside a human evaluation.The passage directs readers to a later section for human-evaluation details.Source: §5.1.3 · p. 9

5.1.4 Baselines

  • Baselines include models trained without pre-training and models pre-trained on unlabeled data before fine-tuning.The latter group includes CodeBERT, GraphCodeBERT, and T5-learning.Source: §5.1.4 · p. 9

5.1.5 Settings

  • Experiments use a maximum of 100 paths per sample, 512-dimensional embeddings, and a batch size of 32.The paper reports training on four NVIDIA V100 32G GPUs with mixed precision.Source: §5.1.5 · pp. 9–10
  • The Table 3 caption reports that PassSum significantly outperforms baselines (p < 0.05), but notes that ASTNN and code+gnn+GRU were not evaluated on Python.The caption identifies code summarization results as the table’s scope.Source: Table 3 · p. 10

5.2 Results

The results evaluate PassSum’s performance, input representation, and pre-training across reported experiments, alongside human judgments of its summaries.

5.2.1 RQ1: How effective is PassSum compared with the state-of-the-art baselines on the code summarization task?

  • PassSum's reported improvement is larger on JCSD (2.24) than Python (1.71), attributed to larger Java ASTs benefiting more from Path Embedding and more Java pre-training data.The average AST terminal-node counts are 119.50 for JCSD and 54.83 for PCSD.Source: §5.2.1 · p. 10
  • PassSum performs better on unseen JavaScript, PHP, Go, and Ruby, which the authors attribute to shared AST structure and pre-training's general language understanding.The authors describe this as evidence of cross-language generalization.Source: §5.2.1 · p. 10

5.2.2 RQ2: Is there a performance advantage owing to the settings of the input representation used by PassSum?

  • Removing Path Embedding caused considerable performance fall; increasing SBT maximum length from 100 to 600 yielded only minor improvement, while PassSum code2seq (200) showed minor gain over PassSum code2seq.The authors attribute the SBT result to difficulty extracting functional features from excessive length; code2seq’s 200-token setting was described as a good sweet spot for information capture and practical GPU memory use.Source: §5.2.2 · pp. 10–11

5.2.1 RQ1: How effective is PassSum compared with the state-of-the-art baselines on the code summarization task?

  • PassSum outperforms state-of-the-art competitors; on CodeSearchNet, it significantly outperforms baselines (p < 0.05).Fine-tuned GraphCodeBERT is also reported to outperform the baselines.Source: §5.2.1 · p. 10, Table 4 · p. 11

5.2.2 RQ2: Is there a performance advantage owing to the settings of the input representation used by PassSum?

  • PassSum's concatenation method outperformed all four separate-encoder strategies, with the authors attributing the difference to earlier, more comprehensive interaction between lexical and syntactic features.The “flat” strategy performed somewhat worse than the other three separate-encoder strategies, which performed around the same.Source: §5.2.2 · p. 11

5.2.3 RQ3: Compared with other encoding methods for structural information, does Path Embedding have advantages in time and space efficiency?

  • PassSum, except for the SBT variant, had the highest time and space efficiency; SBT was most efficient but incurred a considerable performance loss, while ASTNN and code+gnn+GRU were less efficient.The authors attribute code2seq's lower efficiency than PassSum to numerous, often longer path-contexts that slow LSTM propagation; overall, PassSum was more efficient on Python than Java.Source: §5.2.3 · p. 11, Table 6 · p. 11

5.2.4 RQ4: What are the impacts of pre-training on PassSum from Performance, Robustness, and Convergence Speed perspectives?

  • Pre-training improved all PassSum evaluation metrics and partly preserved performance as labeled data shrank; on Ruby, PassSum outperformed other pre-training baselines and pre-training improved robustness.The Ruby CodeSearchNet dataset contained 24,927 training, 1,400 validation, and 1,261 test samples; its training-size trend was comparable to Java and Python but fluctuated more.Source: §5.2.4 · p. 12
  • Pre-trained PassSum converged faster, with lower training loss and higher validation BLEU, and stopped earlier; on Ruby, the un-pretrained model had lower training loss but lower validation BLEU.The authors hypothesize that the un-pretrained model overfits the smaller Ruby dataset, and conclude that pre-training enables faster convergence and better performance in less time.Source: §5.2.4 · p. 12, Figure 11 · p. 13

5.2.5 RQ5: How effective is PassSum compared with the best baseline according to human evaluation?

  • In survey C, each group was evaluated by two participants, who each assessed four groups; the resulting score distribution is reported in Table 9.Scores ranged from 0, when ground truth was better, to 5, when PassSum summaries were better.Source: §5.2.5 · p. 14
  • In almost half of cases, PassSum summaries were essentially the same as ground truth; 6.25% contained no code functionality, while 15.75% better reflected code functionality.The authors conclude that PassSum generally produces summaries similar to ground truth, but a small number are not useful.Source: §5.2.5 · p. 14

5.3 Qualitative Analysis

PassSum performs best on longer code with many, especially nested, control components; the authors attribute this pattern to its structural path input and pre-training. Its examples also show limits when methods are short or important API semantics are not apparent from names.

5.3 Qualitative Analysis

  • The human evaluation randomly sampled generated summaries for 200 test examples and revisited them to identify where PassSum succeeded or failed.The examples include Java and Python cases.Source: §5.3 · p. 15
  • PassSum outperforms baselines on longer code with many nested control components, but is usually comparable when control is sparse; the authors credit Path Embedding and pre-training.They suggest structural information is harder to extract from code sequences alone in the more control-heavy examples.Source: §5.3 · p. 15
  • PassSum can miss details in short methods, while opaque API semantics in one Python example led all compared approaches to fail.The authors also note that PassSum still recognizes an if-condition in the Java example.Source: §5.3 · p. 15
  • In Table 10's Java and Python examples, PassSum captures key information, including control information in a Python access-check example, more accurately than GraphCodeBERT.The authors report that PassSum's Python summary specifies checking access to each path, while GraphCodeBERT's wording is ambiguous.Source: §5.3 · p. 15, Table 10 · p. 16

6 THREATS TO VALIDITY

The authors identify limits in model comparability, hyperparameter selection, and the representativeness of the datasets used for evaluation.

Construct Validity

  • Different model architectures prevented matching total parameter counts, although the experiments kept other settings as consistent as possible and matched hidden-layer size.The parameter-count differences were especially present in Table 3; multi-encoder models in Tables 5 and 6 also had more parameters due to two encoders.Source: §6 · p. 16

Internal Validity

  • PassSum's hyperparameters were set empirically rather than tuned experimentally, so other settings may yield better results.Source: §6 · p. 16

External Validity

  • Experiments used two widely used public datasets, which may not represent other programming languages.Source: §6 · p. 16

7 RELATED WORK

The related work covers code summarization methods and pre-training objectives designed for source code, with PassSum's POP and MNG identified as code-specific objectives.

7 RELATED WORK

  • The related-work section frames its coverage around code summarization and source-code pre-training, noting that POP and MNG were designed specifically for source code.Source: §7 · p. 16

7.1 Code Summarization

  • The review spans a shift from IR methods to neural code summarization, including AST-based, path-based, retrieval, graph, and Transformer approaches.Source: §7.1 · pp. 16–17

7.2 Pre-training Tasks for Source Code

  • Reviewed source-code pre-training objectives include predicting masked data-flow edges, recovering obfuscated identifiers, predicting segment order, and aligning representations of functionally similar programs.Source: §7.2 · p. 17

8 CONCLUSION

The conclusion presents PassSum as addressing inefficient AST encoding and weak performance with small training sets. It reports advantages for Path Embedding and pre-training, and that incorporating ASTs improved PassSum's performance.

8 CONCLUSION

  • Path Embedding outperformed code2seq-like encoding and SBT, while pre-training slowed PassSum's performance decline as training data decreased.Source: §8 · p. 17
  • The authors modified Transformer to accept two input sequences and report that incorporating ASTs improved PassSum's performance.They contrast this with previous work in which Transformer did not benefit from ASTs.Source: §8 · p. 17
  • The conclusion identifies inefficient AST encoding and limited robustness with small training sets as the two weaknesses PassSum addresses.Source: §8 · p. 17

Related papers

Loading 2610.15435v1…