PaperlayerView exact revision on arXiv

Source-linked AI summary

code2vec: Learning Distributed Representations of Code

Uri Alon, Meital Zilberstein, Omer Levy, Eran Yahav

arXiv:1803.09473v5cs.LGcs.AIcs.PLstat.ML

TL;DR

The paper addresses how to represent code snippets so their semantic information can be learned across programs and used to predict properties such as method names. It introduces attention-based code embeddings built from AST paths, demonstrates them on more than 14,000,000 methods, and reports generalization across projects alongside semantic similarities and analogies in learned name vectors.

  • Problem

    The paper asks how to represent code snippets so they capture semantic information, support learning across programs, and enable prediction of properties such as method labels.

  • Method

    The model extracts syntactic paths between AST leaves and uses soft attention to aggregate their representations into a single fixed-length code vector.

  • Results

    The model was trained on more than 14,000,000 methods and generalizes to predicting names in files across different projects, while learned method-name vectors capture semantic similarities and analogies.

  • Takeaways & Limitations

    The approach provides reusable code representations whose predictions are interpretable and whose learned name vectors support meaningful semantic combinations and analogies.

  • Takeaways & Limitations

    The closed label vocabulary prevents the model from composing method names that were not observed during training, especially highly specific names.

Abstract

from arXiv · show

We present a neural model for representing snippets of code as continuous distributed vectors ("code embeddings"). The main idea is to represent a code snippet as a single fixed-length $\textit{code vector}$, which can be used to predict semantic properties of the snippet. This is performed by decomposing code to a collection of paths in its abstract syntax tree, and learning the atomic representation of each path $\textit{simultaneously}$ with learning how to aggregate a set of them. We demonstrate the effectiveness of our approach by using it to predict a method's name from the vector representation of its body. We evaluate our approach by training a model on a dataset of 14M methods. We show that code vectors trained on this dataset can predict method names from files that were completely unobserved during training. Furthermore, we show that our model learns useful method name vectors that capture semantic similarities, combinations, and analogies. Comparing previous techniques over the same data set, our approach obtains a relative improvement of over 75%, being the first to successfully predict method names based on a large, cross-project, corpus. Our trained model, visualizations and vector similarities are available as an interactive online demo at http://code2vec.org. The code, data, and trained models are available at https://github.com/tech-srl/code2vec.

1 INTRODUCTION

The paper introduces code embeddings that aggregate syntactic paths into fixed-length vectors for predicting semantic labels, especially method names. It addresses cross-project generalization and demonstrates semantic structure in learned method-name vectors.

  • Motivation: 77.34% is the model’s most likely prediction for the reverseArray label on a method containing only low-level array assignments.The example illustrates semantic labeling from method bodies rather than explicit high-level operations.
  • Challenges: The central challenge is aggregating possibly hundreds of expressions and statements into one descriptive semantic label.The representation must capture a method’s entire content while producing a compact descriptor.
  • Approach: The model learns continuous code embeddings and predicts label distributions conditioned on code by aggregating multiple syntactic paths into one vector.Its input is a code snippet paired with a label, and the model estimates P(L|C).
  • Results and applications: Learned method-name vectors capture semantic similarities and analogies, including the relation “receive is to send as download is to upload.”Applications include method-name review, retrieval, API discovery, and similarity-based code pipelines.
  • Evaluation: The approach targets cross-project method-name prediction, a difficult benchmark where prior results were low, and reports dramatic improvement over previous techniques.The paper presents the benchmark as evidence for broader code-property prediction tasks.
  • Approach: Soft attention distributes weights over syntactic paths, enabling precise aggregation of path-context information into a vector for the entire snippet.The paper identifies soft attention over syntactic paths as the key source of improved results over previous work.

2 OVERVIEW

code2vec represents code snippets as learned vectors built from syntactic paths, using attention to aggregate contexts and predict semantic labels. Examples show it distinguishes structurally similar methods and assigns descriptive names by weighting informative paths.

  • 2 OVERVIEW: The model extracts AST paths between code leaves, embeds their paths and values, and aggregates them into a fixed-length code vector for prediction.Each path-context combines two terminal values with the syntactic path connecting them.
  • 2.1 Motivating Example: The model predicts contains, get, and indexOf for three methods sharing similar structure but differing in behavior.All three methods iterate over elements and test target, yet their return behavior differs.
  • 2.1 Motivating Example: Attention scores identify the most influential path-contexts, with path width in the visualizations proportional to attention.The model learns weights relative to the other contexts in each snippet.
  • 2.1 Motivating Example: The model assigns descriptive and semantically related name suggestions, including contains and containsExact for the motivating method.The examples show that several high-ranked suggestions can share characteristics with the method.
  • 2.1 Motivating Example: Syntactic paths distinguish contexts that token bigrams would treat identically, such as a parameter versus a local variable with the same name and type.The AST path records an intermediate Parameter node, allowing the distinction to affect prediction.

3 BACKGROUND - REPRESENTING CODE USING AST PATHS

The paper represents code using path-contexts derived from an abstract syntax tree. Each path-context links two terminal values through the AST path between them, with path length and width optionally bounded to control data size and sparsity.

  • 3 BACKGROUND - REPRESENTING CODE USING AST PATHS: The representation of a snippet is formed from path-contexts derived from pairs of AST terminal nodes, excluding self-pairs.The formal representation applies the path construction to the whole code snippet.
  • 3 BACKGROUND - REPRESENTING CODE USING AST PATHS: An AST is defined as nodes, terminals, values, a root, child mappings, and a function assigning values to terminal nodes.This formal structure supplies the tree from which paths and path-contexts are extracted.
  • 3 BACKGROUND - REPRESENTING CODE USING AST PATHS: An AST path connects two terminal nodes through intermediate nonterminals, recording upward and downward movement directions.The path starts and ends at terminals and passes through a common-ancestor structure.
  • 3 BACKGROUND - REPRESENTING CODE USING AST PATHS: A path-context is a triplet containing the values of the start and end terminals and the AST path connecting them.It therefore describes two actual tokens together with their syntactic relation.
  • 3 BACKGROUND - REPRESENTING CODE USING AST PATHS: Maximum path length and width are limited empirically to reduce training-data size and sparsity.Width bounds the difference in child indices between child nodes of an intermediate node.

4 MODEL

The model embeds path-contexts and learns a weighted aggregation into one code vector, which is then used to predict labels. Its design favors attention-based composition and distributed parameters for efficient training and prediction.

  • 4 MODEL: A bag of context vectors is aggregated into one vector by learning how much attention each context should receive.The weighted average uses each context's interaction with a global attention vector.
  • 4.2 Path-Attention Model: The path-attention network combines multiple contexts into a code vector and predicts the probability of each target label from that vector.The architecture is designed for code snippets supplied with labels, names, or tags.
  • 4.2 Path-Attention Model: The network maps each path-context's two terminal values and connecting path to embeddings, concatenates them, and transforms the result through a learned layer.Path and value embeddings are learned jointly with the remaining network parameters.
  • 4.3 Training: Cross-entropy training minimizes the negative log-likelihood assigned to the true label.The loss decreases as the predicted probability of the true label approaches one.
  • 4.5 Design Decisions: More than 95% of test-set paths appeared during training, supporting a faster, more memory-consuming design choice.The selected approach reaches 95% of its final results after six hours of training.

5 DISTRIBUTED VS. SYMBOLIC REPRESENTATIONS

Distributed representations avoid the combination-specific parameterization of symbolic CRFs by learning compact vectors for atomic values, paths, and labels. In the method-name task, this reduces resource requirements while improving the reported F1 score.

  • 5 DISTRIBUTED VS. SYMBOLIC REPRESENTATIONS: Symbolic CRFs model only combinations observed in training, whereas distributed representations compute likelihoods for combinations of observed atomic values.The distributed model stores a small fixed number of parameters per atomic value and composes their combinations algebraically.
  • 5 DISTRIBUTED VS. SYMBOLIC REPRESENTATIONS: Symbolic modeling of terminal values, paths, and labels requires parameters for combinations whose count grows polynomially.The relevant sets are denoted X, P, and Y.
  • 5 DISTRIBUTED VS. SYMBOLIC REPRESENTATIONS: The distributed model uses O(d · (|X| + |P| + |Y|)) parameters instead of combination-level factors.Its vectors have size d and are assigned to atomic terminal values, paths, and labels.
  • 5 DISTRIBUTED VS. SYMBOLIC REPRESENTATIONS: 49.9 F1 is achieved by the reproduced CRF baseline, while the distributed model relatively improves it by 17%.The CRF requires 104% more parameters and about 10 times more memory.

6 EVALUATION

The evaluation compares code2vec with prior methods on cross-project method-name prediction and tests its speed, data efficiency, attention designs, representation components, and qualitative semantic behavior.

  • Evaluation goals: The model aggregates arbitrary-sized code snippets into fixed-size semantic vectors, using method-name prediction as its benchmark.This benchmark tests whether the resulting code vector captures a method body's functionality and semantic role.
  • Quantitative evaluation: Our model achieves significantly better precision and recall than the compared prior models.The evaluation compares precision, recall, and F1 across models.
  • Speed: In 3 hours, the model reaches 88% of its final results, and in 6 hours it reaches 95%, while exceeding baseline results.The model achieves its best results after 30 hours.
  • Data efficiency: With 20% of the training data, the model’s F1 score drops by only 50%.The paper also reports that reducing the data further produced worse results, without specifying the additional percentage here.
  • Alternative designs: The alternative attention model reaches F1 59.3 but is harder to interpret and trains more slowly than standard soft attention.It assigns different attention scores within a combined context vector rather than one score for the whole context.
  • Data ablation study: The full path-context representation outperforms reduced representations, showing that paths, identifiers, and keywords jointly support prediction.Dropping identifiers and keywords hurts more than dropping paths, while combining all components performs significantly better.
  • Qualitative analysis: The model predicts descriptive names for methods whose bodies contain only lower-level operations, including reverseArray, isPrime, sort, and bubbleSort.The highlighted examples are longer than the dataset’s average method length of 7 lines.
  • Semantic analysis: Learned method-name vectors capture semantic similarities and analogies, such as combining equals with toLowerCase to retrieve equalsIgnoreCase.Other examples include the analogy download − receive + send yielding upload.

7 LIMITATIONS OF OUR MODEL

The model is limited by a closed label vocabulary, sparse representations that require substantial data and memory, and dependence on informative variable names.

  • Closed labels vocabulary: Only labels observed exactly during training can be predicted, so highly specific names are often reduced to their main idea.For example, findUserInfoByUserIdAndKey may become findUserInfo.
  • Sparsity and Data-hunger: Whole-symbol terminals, monolithic AST paths, and whole-symbol targets create sparsity across the model’s representations.These design choices prevent unseen paths, terminal values, and target values from being represented.
  • Sparsity and Data-hunger: The resulting parameter count increases GPU memory consumption, enlarges the stored model to about 1.4 GB, and demands substantial training data.The authors trained on 14M examples, while performance may be weaker on smaller datasets.
  • Sparsity and Data-hunger: Path abstractions are proposed as an alternative for reducing sparsity by omitting or merging selected AST-node information.This is presented as a potential future direction rather than part of the described model.
  • Dependency on variable names: Predictions are usually less accurate when variable names are uninformative, obfuscated, or adversarial because training data contained generally good names.Suggested remedies include mixed datasets and preceding variable de-obfuscation.

8 RELATED WORK

Related work spans bimodal code-and-language modeling, code summarization, structural code representations, traditional programming-language models, attention, and distributed representations. The paper distinguishes its approach through whole-snippet embeddings and soft attention over syntactic paths.

  • Bimodal modelling of code and natural language: Bimodal modeling treats source code as both executable for machines and readable for humans, enabling research on conditioning natural language and code on each other.Prior work includes token-based LSTM and attention models for translation and summarization.
  • Bimodal modelling of code and natural language: Unlike sliding-window token attention in prior summarization work, this approach leverages code syntax and is designed to scale more easily to large corpora.The cited comparison reports that the prior approach had worse results when trained across several projects, while this model achieved significantly better results on a multi-project corpus.
  • Representation of code in machine learning models: The representation resembles prior syntactic-relation approaches but extends them to whole snippets, using soft attention to predict properties of an entire code fragment.The paper presents this whole-snippet embedding and attention mechanism as its main novelty in code representation.
  • Traditional machine learning: Earlier programming-language models include decision trees, CRFs, probabilistic context-free grammars, and n-grams.These methods represent the broader tradition of machine learning for programming languages.
  • Attention in machine learning: Prior syntax-based models identify context nodes or select one most informative context, whereas this model uses the full path and soft attention over multiple contexts with different weights.The paper contrasts this with hard-attention-like single-context selection and notes that non-neural methods cannot express soft attention.
  • Distributed representations: Distributed representations map code elements, paths, and method names into vectors that support generalization, use fewer parameters than symbolic representations, and place semantically similar names near one another.This work applies distributed representations beyond words, sentences, and documents to code snippets and method names.

9 CONCLUSION

The paper represents arbitrary-sized code snippets with fixed-length vectors by applying soft attention to AST-derived syntactic paths. It demonstrates the approach through cross-project method-name prediction trained on more than 14,000,000 methods.

  • 9 CONCLUSION: The model applies soft attention over syntactic paths extracted from an AST and aggregates their representations into one learned fixed-length vector.This architecture is intended to represent arbitrary-sized code snippets.
  • 9 CONCLUSION: More than 14,000,000 methods were used to demonstrate method-name prediction with the proposed representation.The conclusion presents method-name prediction as the paper’s example application.
  • 9 CONCLUSION: The model generalizes to predict names in files across different projects, and the attention mechanism makes its prediction results interpretable.The authors conjecture that generalization stems from the model’s relative simplicity and distributed nature.
  • 9 CONCLUSION: The authors propose the attention-based structural model as a basis for a wide range of programming-language processing tasks.The code and trained model are publicly available.
Loading 1803.09473v5…