PaperlayerView exact revision on arXiv

Source-linked AI summary

code2seq: Generating Sequences from Structured Representations of Code

Uri Alon, Shaked Brody, Omer Levy, Eran Yahav

arXiv:1808.01400v6cs.LGcs.PLstat.ML

TL;DR

Generating natural-language sequences from source code supports applications including summarization, documentation, retrieval, and captioning. CODE2SEQ represents code through sampled AST paths and attends over those paths during decoding, outperforming previous programming-language-oriented works and state-of-the-art NMT models across the reported tasks and languages.

  • Problem

    Generating natural-language sequences from source code is needed for applications including summarization, documentation, retrieval, and captioning, but direct approaches encode code as flat token sequences.

  • Method

    CODE2SEQ samples paths between AST terminals, encodes the paths with LSTMs, and attends over their representations while generating the target sequence.

  • Results

    The model performs significantly better than previous programming-language-oriented works and state-of-the-art NMT models across method-name prediction, code captioning, and documentation generation in two programming languages.

  • Takeaways & Limitations

    AST-path encoding provides a basis for code-to-sequence generation across multiple tasks, datasets, and programming languages.

  • Takeaways & Limitations

    Reported NMT baselines used many more parameters than CODE2SEQ because size-matched versions performed worse.

Abstract

from arXiv · show

The ability to generate natural language sequences from source code snippets has a variety of applications such as code summarization, documentation, and retrieval. Sequence-to-sequence (seq2seq) models, adopted from neural machine translation (NMT), have achieved state-of-the-art performance on these tasks by treating source code as a sequence of tokens. We present ${\rm {\scriptsize CODE2SEQ}}$: an alternative approach that leverages the syntactic structure of programming languages to better encode source code. Our model represents a code snippet as the set of compositional paths in its abstract syntax tree (AST) and uses attention to select the relevant paths while decoding. We demonstrate the effectiveness of our approach for two tasks, two programming languages, and four datasets of up to $16$M examples. Our model significantly outperforms previous models that were specifically designed for programming languages, as well as state-of-the-art NMT models. An interactive online demo of our model is available at http://code2seq.org. Our code, data and trained models are available at http://github.com/tech-srl/code2seq.

1 INTRODUCTION

Generating natural-language sequences from source code supports tasks including summarization, documentation, retrieval, and captioning. CODE2SEQ replaces flat token encoding with AST-path representations and outperforms prior code-specific and NMT models on two tasks.

  • Source-code-to-language generation supports code summarization, documentation, retrieval, and other natural-language generation tasks.
  • Token-based seq2seq approaches treat code as a source-token sequence and have achieved state-of-the-art results despite extremely long inputs.
  • CODE2SEQ represents code as compositional paths over its AST, encodes each path with an LSTM, and attends over path vectors during decoding.
  • The model evaluates Java method-name summarization and C# code captioning using predictions generated from code snippets.
  • CODE2SEQ outperforms code-specific models, TreeLSTMs, and state-of-the-art NMT models on both evaluated tasks.

2 REPRESENTING CODE AS AST PATHS

CODE2SEQ represents code structurally through sampled paths between AST terminals rather than only as a token sequence. This exposes recurring syntactic patterns while keeping the variable-sized path set computationally manageable.

  • An AST uniquely represents a source-code snippet according to a programming language’s grammar, with terminals as leaves and nonterminals as syntactic structures.
  • Pairwise paths between AST terminals combine terminal and nonterminal nodes with terminal values to represent a code snippet.
  • Structurally similar paths can reveal common patterns between functionally equivalent methods with different token sequences.
  • The model samples k paths from each code snippet and resamples them at every training iteration to avoid sampling bias.

3 MODEL ARCHITECTURE

CODE2SEQ replaces flat token encoding with separate representations of AST paths, then generates output sequences by attending over those path representations. Its architecture combines path structure and terminal-token values, aggregates paths to initialize decoding, and dynamically selects paths during generation.

  • AST encoder: CODE2SEQ encodes each AST path separately rather than reading source code as a flat token sequence.Each path is represented with a bidirectional LSTM over AST nodes and child indices.
  • Decoder: The decoder produces target tokens sequentially, conditioning each next-token probability on previously generated tokens and the attended context.The context vector and decoder state are combined to predict the current target token with softmax.
  • AST encoder: AST paths combine structural representations with subtoken-based embeddings of their terminal values.Terminal tokens are split into subtokens, embedded, summed, and concatenated with the path representation before a fully connected layer produces the combined vector.
  • Decoder: The decoder is initialized from the mean of the combined representations of all k paths.This aggregation makes the initial decoder state independent of the order of the sampled input paths.
  • Decoder: During generation, attention dynamically selects a weighted distribution over the encoded AST paths for each output token.The decoder attends over path vectors rather than source-token representations.

4 EXPERIMENTS

The experiments evaluate CODE2SEQ on Java method-name summarization and C# code captioning, using datasets ranging from about 700K to 16M examples and comparisons with code-specific, tree-based, and NMT baselines. Across these evaluations, CODE2SEQ improves over competing approaches, including on code documentation and varying input lengths.

  • Tasks and datasets: CODE2SEQ is evaluated on Java method-name summarization and C# code captioning, with an additional comparison on Javadoc generation.The summarization task predicts a Java method’s name from its body, while captioning generates natural-language descriptions of C# snippets.
  • Tasks and datasets: The summarization experiments use Java-small, Java-med, and Java-large datasets containing about 700K, 4M, and 16M examples, respectively.The splits are project-disjoint, with training, validation, and test projects separated across datasets.
  • Code summarization: CODE2SEQ improves over the best baselines by 4 to 8 F1 points across all three summarization benchmarks.The comparisons include PL-oriented models, syntactic-path models, TreeLSTMs, and NMT baselines strengthened with subtoken splitting and other adjustments.
  • Code summarization: 7.3% relative improvement over the best baseline is achieved on Java-large, increasing to 13% on Java-med and 22% on Java-small.Relative improvements over the Transformer are 23% on Java-large and 37% on Java-small, indicating stronger gains on smaller datasets.
  • Sensitivity to input length: CODE2SEQ remains superior to all examined baselines across input-code lengths, although every model performs best on snippets shorter than 3 lines.Performance declines as code length increases and becomes stable for lengths of 9 lines and above; inputs over 30 lines are grouped at 30 lines in Figure 4.
  • Code captioning: 23.04 BLEU improves code captioning by 2.51 points, or 12.2% relative, over CodeNN and the other baselines.The evaluation concerns natural-language descriptions of C# code snippets.
  • Code documentation: 14.53 BLEU is obtained for Javadoc generation, a 62% relative gain over Hu et al. (2018) under the same dataset and matched LSTM sizes.The paper reports this additional documentation comparison as outside the main focus of the work.

5 ABLATION STUDY

The ablation study evaluates how decoder design, token representations, AST structure, attention, and path sampling affect code summarization. Sequential decoding, token splitting, syntactic paths, attention, and fresh path sampling each contribute to performance.

  • The study evaluates alternative model designs on Java-med validation data for code summarization.It measures changes in performance for several ablations, including removing AST nodes, decoding, token splitting, tokens, attention, or path re-sampling.
  • Removing AST nodes decreases precision by 5.16 and recall by 4.30, while still outperforming the baselines.The authors attribute this partly to AST leaves focusing attention on named tokens and ignoring functional tokens such as brackets and semicolons.
  • Removing the decoder reduces recall by more than one-third, showing that method-name prediction benefits from sequential decoding.The target method names are short, but the ablation treats the whole sequence as a single prediction.
  • Removing token splitting or terminal tokens drastically reduces the score, indicating that both subtoken and syntactic-path information matter.Without tokens, the model still reaches around half the full-model score despite lacking identifier names, types, APIs, and constant values.
  • Removing attention shows a contribution close in relative value to attention in seq2seq models, while fixed path samples lose the regularization benefit of re-sampling.Freshly sampling k paths on every training iteration improves performance compared with using the same paths throughout training.

6 RELATED WORK

Prior work often represents code as sequences, characters, API calls, linearized syntax, or task-specific graphs. CODE2SEQ instead uses compositional AST paths and is evaluated on broader code-to-language generation settings.

  • Earlier models represented code as token, character, or API-call sequences, treating code as a sequence rather than a tree.This requires the models to relearn predefined programming-language syntax implicitly.
  • Syntactic representation models had mainly been evaluated on easier blank-filling or code-classification tasks.The cited prior models generally lacked compositional syntactic relations, whose number was fixed before or after processing.
  • Oda et al. generated pseudo-code from Python using line-by-line statistical machine translation, whereas CODE2SEQ maps whole snippets to shorter outputs without assuming input-output alignment.The paper distinguishes these settings because its tasks do not align individual input elements with output elements.
  • Hu et al. linearized ASTs before applying a standard seq2seq model, while CODE2SEQ uses an encoder that directly assumes tree-structured input.On Hu et al.’s dataset, CODE2SEQ improves over their BLEU score by 62%.
  • Allamanis et al. used expert-designed syntactic and semantic graph relations, whereas CODE2SEQ makes minimal assumptions about the input language and does not require expert semantic knowledge.The contrast concerns the design of graph edges versus the model’s more general AST-based representation.

7 CONCLUSION

CODE2SEQ encodes source code through sampled AST paths and generates natural-language sequences by attending to those path representations. Across multiple tasks, datasets, and languages, it outperforms prior programming-language models and state-of-the-art NMT models.

  • CODE2SEQ samples AST paths, encodes them with an LSTM, and attends to them while generating the target sequence.This combines syntactic code structure with sequential modeling of natural-language output.
  • The model predicts method names, captions, and method documentation across datasets of varying sizes and two programming languages.The demonstrated tasks include method-name prediction, natural-language captioning, and Javadoc generation.
  • CODE2SEQ performs significantly better than previous programming-language-oriented models and state-of-the-art NMT models in the reported settings.The conclusion summarizes the cross-task evaluation rather than a single benchmark result.
  • The authors make their code, datasets, and trained models publicly available and present the approach as a basis for broader source-code and natural-language tasks.They also state that the principles can extend to other kinds of generated outputs.

A ADDITIONAL EVALUATION

The additional evaluation compares CODE2SEQ with code2vec and documents the paper’s dataset and qualitative examples. The comparison favors stricter project-level splitting to avoid method-name leakage across files.

  • CODE2SEQ achieves a higher F1 score than code2vec on code2vec’s proposed dataset.The paper reports that code2vec nevertheless achieves a high F1 score on that dataset.
  • The authors attribute code2vec’s poorer performance on their dataset partly to different train, validation, and test splitting procedures.Their datasets split by project, while code2vec’s dataset splits by file, allowing related files from one project to cross partitions.
  • File-level splitting can make evaluation easier because method names may leak across files and duplicates may occur within the same project.The paper cites prior work finding that file-based splitting is easier than project-based splitting.
  • The authors use project-level splitting for their comparisons despite achieving better results on code2vec’s dataset.They choose the stricter approach to keep comparisons on split-by-project datasets.
  • The paper includes dataset statistics and qualitative test-set examples for code captioning and code summarization.Figure 5 additionally illustrates timestep-level decoding with the top-attended AST path at each step.

D CODE CAPTIONING RESULTS

Figure 9 visualizes BLEU scores for the model and baselines in the code captioning task, using the values from Table 2.

  • The figure compares BLEU scores for the model and baselines on code captioning.The task predicts natural-language descriptions for C# code snippets.

E CODE SUMMARIZATION RESULTS

Figure 10 visualizes F1 scores for the model and baselines in the code summarization task, using the F1 columns from Table 1.

  • The figure compares F1 scores for the model and baselines across code summarization datasets.The task predicts method names in Java.

F ABLATION STUDY RESULTS

The ablation analysis reports relative precision and recall decreases, while accompanying figures visualize attention paths, examples, and performance comparisons.

  • Figure 11 shows the relative decrease in precision and recall for each ablation.The ablations are described in Section 5 and presented in Table 3.
  • Figure 5 illustrates decoding-step predictions and the top-attended paths used by the model for a C# code snippet.Path widths represent attention weights, while only the top-attended path at each step is shown.
  • Figures 9 and 10 visualize BLEU and F1 comparisons between the model and baselines for code captioning and code summarization.The figures use values from Tables 2 and 1, respectively, and report significantly higher results for the model.
Loading 1808.01400v6…