Source-linked AI summary
A Literature Study of Embeddings on Source Code
Zimin Chen, Martin Monperrus
TL;DR
Source code has language-like structure and multiple granularities, prompting interest in applying word embeddings to programs. The paper surveys this work through a literature search and a five-category organization, and concludes that embeddings have been applied across code granularities and downstream tasks.
Problem
Programming languages have language-like repetitive and predictable patterns, and their multiple granularities motivate examining how word embeddings apply to source code.
Source: §1 · p. 1Method
The authors identify relevant papers through author inquiries and an extensive Google Scholar search, then organize them into five embedding categories and provide visualizations and data links.
Source: §1 · p. 1Results
The literature study finds diverse embedding techniques applied to different source-code granularities and downstream tasks.
Source: §5 · p. 6Takeaways & Limitations
Contextual word embeddings are a promising direction because code elements can have context-dependent meanings.
Source: §4.2 · p. 6
Abstract
from arXiv · showhide
Natural language processing has improved tremendously after the success of word embedding techniques such as word2vec. Recently, the same idea has been applied on source code with encouraging results. In this survey, we aim to collect and discuss the usage of word embedding techniques on programs and source code. The articles in this survey have been collected by asking authors of related work and with an extensive search on Google Scholar. Each article is categorized into five categories: 1. embedding of tokens 2. embedding of functions or methods 3. embedding of sequences or sets of method calls 4. embedding of binary code 5. other embeddings. We also provide links to experimental data and show some remarkable visualization of code embeddings. In summary, word embedding has been successfully applied on different granularities of source code. With access to countless open-source repositories, we see a great potential of applying other data-driven natural language processing techniques on source code in the future.
1 INTRODUCTION
The survey examines how word-embedding techniques are being applied to source code, motivated by language-like patterns in programming and encouraging experimental results. It organizes this work into five categories and contributes visual representations and a curated list of publicly available embedding data.
- Programming-language tokens and statements are natural targets for word embeddings because programming languages share repetitiveness and predictability with natural languages.Source: §1 · p. 1
- Source-code embeddings can represent different granularities, from variables and expressions to statements and methods.Source: §1 · p. 1
- The authors identify relevant articles by asking related-work authors and conducting an extensive Google Scholar search.Source: §1 · p. 1
- The survey categorizes source-code embeddings as tokens, expressions, APIs, methods, and miscellaneous embeddings.Source: §1 · p. 1
- The survey provides visual representations of code embeddings and a curated list of publicly available embedding data.Source: §1 · p. 1
2 BACKGROUND
The background defines embeddings and describes visualization and similarity tools used to inspect and compare them.
2.1 EMBEDDINGS
- An embedding maps objects to real-valued vectors; word embeddings map words or word sequences to vectors that can represent semantic information.Source: §2.1 · p. 2
2.2 VISUALIZATION
- PCA and t-SNE reduce high-dimensional embeddings to two or three dimensions for visualization.Source: §2.2 · p. 2
2.3 SIMILARITY METRICS IN THE EMBEDDING SPACE
- Cosine similarity compares vector angles independently of magnitude, while high-dimensional distance behavior favors smaller Lk norms such as L1 over L2.Source: §2.3 · p. 2
3 EMBEDDING FOR SOURCE CODE
The survey organizes code embeddings by granularity and application: token, function or method, sequence or API-call, binary-code, and other representations. It describes varied construction methods and downstream uses across these categories.
3.1 EMBEDDING OF TOKENS
- Token embeddings support vulnerability prediction, program repair, and student-submission profiling, using embeddings as classifier or encoder inputs or to compare code pieces.Source: §3.1 · p. 2
3.2 EMBEDDING OF FUNCTIONS OR METHODS
- Method embeddings represent AST paths or contextual method-name features for name prediction and generation, while function embeddings from control-flow paths support clone detection.Source: §3.2 · pp. 2–3
- Other function or method representations use hyperbolic embeddings of function-call graphs, encoder states for sketches, AST features for variable-misuse repair, or LSTM and Siamese networks for clone detection.Source: §3.2 · p. 3, Figure 1 · p. 3
3.3 EMBEDDING OF SEQUENCES OR SETS OF METHOD CALLS
- Embeddings of traces, APIs, queries, and code snippets support tasks including cross-language API-usage matching, natural-language-to-API translation, and buggy-code identification.Source: §3.3 · p. 4
3.4 EMBEDDING OF BINARY CODE
- Binary-code methods embed statements from LLVM contextual-flow-graph paths, combine binary instructions with LSTMs for block similarity, preserve cross-architecture instruction relations, or embed control-flow graphs for function similarity.Source: §3.4 · pp. 4–5, Figure 3 · p. 4
3.5 OTHER EMBEDDINGS
- Other embeddings encode AST expressions with TreeNN, represent source-code edits as sequences or graphs, and use intent and intent-filter embeddings to classify possible Android component links.Source: §3.5 · p. 5, Figure 4 · p. 5
- Program and behavior embeddings aggregate variable traces or event-pattern embeddings, while AST graph embeddings add data-flow and type-hierarchy edges.Source: §3.5 · p. 5
- The survey also lists publicly available code embeddings and experimental data in Table 1.Source: §3 · p. 2, Table 1 · p. 6
4 FUTURE RESEARCH DIRECTIONS
The section proposes future directions for code embeddings, focusing on evaluating embeddings through downstream tasks and representing context-dependent meanings.
4.1 EMBEDDING VERSUS DOWNSTREAM TASKS
- Generic evaluation of code-embedding quality is difficult, so the authors point to downstream tasks rather than generic measures as the preferred evaluation approach.Analogies and lists of nearest elements are also possible evaluation approaches.Source: §4.1 · p. 6
- Existing code embeddings can serve as initialization for downstream tasks.Source: §4.1 · p. 6
4.2 CONTEXTUAL WORD EMBEDDING
- Contextual word embeddings account for surrounding context, relevant because a code element such as Java’s static can mean different things in fields and methods.Source: §4.2 · p. 6
5 CONCLUSION
The study collects and discusses source-code embedding work, finding diverse techniques used across downstream tasks and describing the field as still emerging.
5 CONCLUSION
- The study collects and discusses articles that define or use embeddings on source code.Source: §5 · p. 6
- The surveyed embedding techniques are diverse and are applied to different downstream tasks.Source: §5 · p. 6
- The authors describe code embedding as an early-stage field and expect many future papers to use the concept.Source: §5 · p. 6