PaperlayerView exact revision on arXiv

Source-linked AI summary

Language-Agnostic Representation Learning of Source Code from Structure and Context

Daniel Zügner, Tobias Kirschstein, Michele Catasta, Jure Leskovec, Stephan Günnemann

arXiv:2103.11318v1cs.LGcs.SE

TL;DR

Code models have typically emphasized source-code sequence or AST structure, although both represent useful aspects of a program. CODE TRANSFORMER combines their distances in self-attention using language-agnostic features. It achieves state-of-the-art monolingual results across five languages, and multilingual training improves performance across languages, especially those with fewer resources.

  • Problem

    Code-learning models have traditionally relied predominantly on either source-code Context or AST Structure, despite their complementary information.

    Source: §1 · p. 1

  • Method

    CODE TRANSFORMER combines Context and AST distances in self-attention and uses language-agnostic features computed from source code and ASTs.

    Source: §1 · p. 2

  • Results

    The model achieves state of the art in monolingual code summarization on five languages, while multilingual training improves all languages and Context-only multilingual training does not yield the same improvements.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The largest multilingual improvement occurs for the language with the fewest resources, indicating that multilingual training can improve learning for less widely used languages.

    Source: §6 · p. 9

Abstract

from arXiv · show

Source code (Context) and its parsed abstract syntax tree (AST; Structure) are two complementary representations of the same computer program. Traditionally, designers of machine learning models have relied predominantly either on Structure or Context. We propose a new model, which jointly learns on Context and Structure of source code. In contrast to previous approaches, our model uses only language-agnostic features, i.e., source code and features that can be computed directly from the AST. Besides obtaining state-of-the-art on monolingual code summarization on all five programming languages considered in this work, we propose the first multilingual code summarization model. We show that jointly training on non-parallel data from multiple programming languages improves results on all individual languages, where the strongest gains are on low-resource languages. Remarkably, multilingual training only from Context does not lead to the same improvements, highlighting the benefits of combining Structure and Context for representation learning on code.

1 INTRODUCTION

Source code and its AST offer complementary representations, while prior models have generally emphasized one or the other. CODE TRANSFORMER combines their distances in self-attention using language-agnostic features, and the paper reports strong monolingual and multilingual summarization results.

1 INTRODUCTION

  • Source code and its AST are complementary representations, but machine-learning models have traditionally relied predominantly on one or the other.The paper frames their joint use as improving source-code representation learning.Source: §1 · p. 1
  • AST-based graph neural networks make code structure explicit, but message passing can leave their learned representations local and weak at long-range interactions.The paper contrasts this with models based on source-code sequences.Source: §1 · p. 1
  • GREAT combines graph and sequence representations but uses language-specific graph features that require specialized pipelines and static analysis.Its features include data-flow, control-flow, and syntactic edges.Source: §1 · p. 1
  • The Figure 1 example contrasts token pairs that are close in AST distance but far apart in sequence, and pairs showing the reverse pattern.The caption uses these cases to motivate unifying Structure and Context.Source: Figure 1 · p. 2
  • CODE TRANSFORMER combines distances from source-code Context and AST Structure in self-attention using language-agnostic features.Pairwise AST distances, such as shortest-path lengths, make full structure accessible at each layer.Source: §1 · p. 2
  • The model achieves state of the art on monolingual code summarization across five languages, while multilingual training improves all languages unlike Context-only multilingual training.The strongest multilingual gains are on low-resource languages.Source: §1 · p. 2

2 RELATED WORK

Prior code-representation work spans language models, structured models, and graph neural networks. This paper situates its approach among research that supplies or learns structural information, while distinguishing prior feature-engineering and AST-linearization strategies.

2 RELATED WORK

  • Early code language models used source regularities to learn representations with semantic properties, which were applied to identifier and API prediction and autocomplete.The passage also describes applications including Android deobfuscation.Source: §2 · p. 2
  • Prior probabilistic source-code models often use parse trees, while other approaches use hand-crafted node features or explicit semantic edges.The passage distinguishes this work from those feature-augmentation approaches.Source: §2 · p. 2
  • The paper describes its language-agnostic features as avoiding language-specific features such as program-analysis edges, facilitating multilingual training through unified AST representations.The passage says such representations can be generated for different programming languages.Source: §2 · p. 2
  • Other AST-based approaches linearize the graph by traversing it before modeling.The cited examples are Alon et al. (2018) and Li et al. (2017).Source: §2 · p. 3
  • Prior research argues that explicit structure can aid generalization, long-range dependencies, and compositionality; this work pursues that inductive-bias approach for source code.The passage also notes tree-structured methods in language and source-code modeling.Source: §2 · p. 3
  • Graph neural networks recursively aggregate neighboring nodes and have achieved state-of-the-art results across node, link, clustering, and graph-classification tasks.The passage presents GNNs as learning node and graph representations end to end.Source: §2 · p. 3

3 INTEGRATING STRUCTURE AND CONTEXT IN THE CODE TRANSFORMER

The CODE TRANSFORMER incorporates relative position information into self-attention so it can use both source-code sequence context and AST structure. It combines several AST relations and groups distance values to make relational attention computation more efficient.

3 INTEGRATING STRUCTURE AND CONTEXT IN THE CODE TRANSFORMER

  • Relative position encodings replace absolute positions with token-to-token distances, allowing self-attention to operate on structured inputs such as graphs.The paper notes that absolute positions can be ambiguous when processing two sentences at a time.Source: §3 · p. 4

3.1 INTEGRATING SOURCE CODE AND AST REPRESENTATIONS OF PROGRAMS.

  • Separate key projections let the model sum contributions from sequence and AST relations in the raw attention score.This makes multiple relations between tokens available to the attention computation.Source: §3.1 · p. 4
  • AST shortest-path distances use an undirected graph and provide global access to the graph structure at each layer, unlike edge-only attention.The paper contrasts this with edge-based attention, which exchanges information only between immediate neighbors.Source: §3.1 · p. 4
  • Ancestor and sibling hop distances retain direction and child order, while personalized PageRank adds local graph proximity that can differ from hop counts.For ancestor distances, the paper gives opposite signed values for the two directions; sibling distances likewise encode ordered hops.Source: §3.1 · pp. 4–5
  • Each source token receives concatenated subtoken, assigned AST-node-type, and tokenizer token-type embeddings; unselected internal AST nodes still affect distances.The assigned AST node is the one with the shortest source range containing the token.Source: §3.1 · p. 5

3.2 EFFICIENT RELATIVE ATTENTION COMPUTATION.

  • To avoid materializing an N × N × d encoding tensor, the model groups distances into k ≪ N^2 bins, using wider bins for larger distances.The experiments use k = 32, and the authors report relative insensitivity to the number of bins.Source: §3.2 · p. 5

4 EXPERIMENTAL SETUP

The experiments evaluate code summarization by predicting a function's name from its body across four CodeSearchNet languages and Java. The setup includes data preprocessing, a pointer-enabled decoder, and comparisons with code summarization baselines.

4 EXPERIMENTAL SETUP

  • Code summarization predicts a function's name from its body, with evaluation on Python, JavaScript, Go, Ruby, and Java-small.The CodeSearchNet datasets were deduplicated; Java-small contains roughly 700K samples from 11 open-source Java projects.Source: §4 · pp. 5–6
  • Java-small uses nine projects for training, one for validation, and one for testing; Java-medium and Java-large are also used for pretraining before fine-tuning.The Java-small test and validation projects are removed from the pretraining data to avoid leakage.Source: §4 · p. 6
  • Preprocessing splits code names into subtokens, limits method names to six subtokens, and retains snippets with at most 512 tokens.An open-source AST parser produces the ASTs, and the vocabulary includes subtokens occurring at least 100 times in training.Source: §4 · p. 6
  • Transformer-based decoders use a pointer network, and comparisons include code2seq, GREAT, and a BiLSTM+GNN→LSTM+Pointer model.The Fernandes et al. model is compared only on Java-small because its preprocessing is language-specific; GREAT receives AST-derived edges for comparison.Source: §4 · p. 6

5 RESULTS

The CODE TRANSFORMER improves code summarization across monolingual and multilingual settings, with multilingual training producing substantial gains across languages and the strongest improvement on Ruby. Ablations and qualitative analyses further examine the roles of AST distances, Structure and Context, and shared multilingual representations.

5.1 MONOLINGUAL CODE SUMMARIZATION

  • The CODE TRANSFORMER substantially outperforms other models on all but one language in monolingual evaluation, matching its Context-only variant on Ruby.The authors attribute Ruby’s exception to its smaller dataset and dynamic typing; Context-only also outperforms GREAT on every language.Source: §5.1 · pp. 6–7, Table 2 · p. 7
  • On Java-small, the model without pretraining achieves state of the art among pointer-network models, while Java-medium and Java-large pretraining improves precision, recall, and F1 after fine-tuning.Pretraining excludes the Java-small validation and test projects to avoid leakage.Source: §5.1 · p. 7, Table 3 · p. 7
  • Removing either Structure or Context substantially reduces performance with a pointer network, while without it the Structure-only variant can outperform the full model.The Structure-only variant without a pointer network performs better on longer method names, which strongly influence micro-F1.Source: §5.1 · p. 7, Table 3 · p. 7
  • Each AST distance alone performs substantially worse than their combination, although personalized PageRank performs best among the individual distances.The comparison uses the Structure-only setting without a pointer network.Source: §5.1 · pp. 7–8, Table 4 · p. 8

5.2 MULTILINGUAL CODE SUMMARIZATION

  • Joint multilingual training substantially outperforms monolingual models on all four languages, with the strongest improvement on low-resource Ruby.Fine-tuning on individual languages after joint training has only a marginal effect.Source: Table 2 · p. 7, §5.2 · p. 8
  • On ogbg-code24, the monolingual Python model exceeds the leaderboard state of the art by 112% in sample-F1, and the multilingual model with LM pretraining exceeds it by 122%.These comparisons are reported as of submission.Source: §5.2 · p. 8
  • The multilingual snippet embeddings show language intersections: methods beginning with parse tend to cluster across Python and JavaScript, and an example pair is semantically similar despite different method names.The visualization uses masked method-name representations from validation snippets in four languages.Source: §5.2 · pp. 8–9, Figure 4 · p. 9, Figure 5 · p. 9
  • Learned language embeddings for languages with similar roots in language design are close to one another.The pairwise similarity matrix is shown in the appendix.Source: §5.2 · p. 9

6 CONCLUSION

The CODE TRANSFORMER jointly learns from program Structure and Context using language-agnostic features. It achieves state-of-the-art code summarization results on five languages, while multilingual training improves every language and benefits the lowest-resource language most.

6 CONCLUSION

  • The CODE TRANSFORMER combines Structure and Context using only language-agnostic features.Its features are limited to source code and features computed from the AST.Source: §6 · p. 9
  • The model achieves state-of-the-art code summarization performance on five programming languages.Source: §6 · p. 9
  • Multilingual training substantially outperforms monolingual training on every language, with the largest improvement on the language with fewest resources.Source: §6 · p. 9
  • Multilingual training from Context alone does not yield the same improvements, highlighting the benefits of combining Structure and Context.Source: §6 · p. 9

A.1 DISTANCE ENCODING FUNCTION

Scalar relation values are encoded as vectors with the model's embedding dimension. The distance encoding uses the sinusoidal function from Vaswani et al. (2017), with no trainable parameters.

A.1 DISTANCE ENCODING FUNCTION

  • Scalar relation values are mapped to vectors in R^d, where d is the model's embedding dimension.Source: §A.1 · p. 14
  • The encoding function is the sinusoidal function presented in Vaswani et al. (2017).Source: §A.1 · p. 14
  • The encoding has no trainable parameters; its constant M is set to 10,000, and k indexes positions with 1 ≤ k < d/2.Source: §A.1 · p. 14

A.2 MULTILINGUAL REPRESENTATION ANALYSIS

The learned language embeddings show similar cosine similarities for Python–Ruby and JavaScript–Go. The section also presents selected snippets and their best embedding matches across languages.

A.2 MULTILINGUAL REPRESENTATION ANALYSIS

  • The learned language embeddings give similar pairwise cosine similarities for Python–Ruby and JavaScript–Go.Source: §A.2 · p. 14
  • The Python–Ruby and JavaScript–Go embedding similarities align with language-design roots and common use cases.Source: §A.2 · p. 14
  • Selected snippets beginning with “is,” “main,” or “load” are paired with their best embedding matches from other languages.Source: §A.2 · p. 14

A.3.1 TEXTUAL CODE SNIPPET PREPROCESSING

The preprocessing pipeline tokenizes code, removes comments and empty lines, masks literals, and standardizes indentation and sub-token inputs. It also removes residual whitespace and discards snippets the tokenizer cannot parse.

A.3.1 TEXTUAL CODE SNIPPET PREPROCESSING

  • Code snippets are tokenized with Pygments, then multiline, single-line, and documentation comments are removed.Source: §A.3.1 · p. 14
  • Empty lines are removed, and hard-coded strings and numbers are replaced with [MASK STRING] and [MASK NUMBER] tokens.Source: §A.3.1 · p. 16
  • Indentation changes are represented by [INDENT] or [DEDENT] tokens after detecting the snippet's indentation style.Source: §A.3.1 · p. 16
  • Tokens are split into sub-tokens, with five input sub-tokens used and shorter sequences padded with [PAD].Source: §A.3.1 · p. 16
  • Remaining whitespace-only tokens are removed except line breaks, and snippets with tokens Pygments cannot parse are discarded.Source: §A.3.1 · p. 16

A.3.2 STAGE 1 PREPROCESSING (GENERATION OF ASTS)

The appendix details AST-based preprocessing, model inputs, and experimental comparisons, then presents code-summarization examples and pointer-network and CSN evaluations.

A.3.2 STAGE 1 PREPROCESSING (GENERATION OF ASTS)

  • ASTs provide node types and source ranges; tokens map to the smallest encompassing node, with a shorter-range heuristic for overlapping child ranges.Snippets with AST parse errors are discarded.Source: §A.3.2 · p. 16

A.3.3 STAGE 2 PREPROCESSING (CALCULATION OF DISTANCE MATRICES)

  • AST relations include PPR, shortest paths, ancestor and sibling paths, then binning and filtering constrain inputs.The appendix also removes punctuation and discards training snippets over 512 tokens, while test inputs may reach 1,000 tokens.Source: §A.3.3 · p. 17

A.4 INPUT EMBEDDINGS TO THE MODEL

  • Each input token combines five subtoken embeddings with token-type and AST-node-type embeddings, then projects them to the model’s embedding dimension.Source: §A.4 · p. 17

A.5 INPUT TO THE GREAT BASELINE

  • For the GREAT baseline, the authors supplied their AST distances as thresholded edges because GREAT’s preprocessing pipeline was unavailable.Source: §A.5 · p. 17

A.6 EXPERIMENTAL SETUP

  • Ablations compare structure-only and context-only inputs, distance masking, and distance-bin counts.A five-hop mask substantially deteriorated results, while 16 or 64 bins had no noticeable performance effect.Source: §A.6 · p. 18

A.7 CODE SUMMARIZATION EXAMPLES

A.8 ESTIMATION OF POINTER NETWORK POTENTIAL

  • The pointer network improves test F1 for every language except Go, where test performance drops despite about a 3-point validation gain.Figure 7 compares label-token overlap between method bodies and labels; the authors attribute Go’s train/validation–test mismatch to an apparent violation of the i.i.d. assumption.Source: §A.8 · p. 22, Figure 7 · p. 22

A.9 CODE SUMMARIZATION RESULTS ON THE CSN DATASET (SAMPLE-F1)

Related papers

Loading 2103.11318v1…