PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of Machine Learning for Big Code and Naturalness

Miltiadis Allamanis, Earl T. Barr, Premkumar Devanbu, Charles Sutton

arXiv:1709.06182v2cs.SEcs.LGcs.PL

TL;DR

Software engineering and programming-languages research lacks an extensive synthesis of probabilistic machine-learning models for source code and the design choices arising from code’s relationship to natural language. This survey contrasts code and text, presents a taxonomy of source-code models, reviews applications, and discusses challenges; it concludes that these models have broad potential, while data, deployment, and unsupervised-learning constraints remain open.

  • Problem

    Software development needs more reliable and maintainable tools, while prior reviews did not provide an extensive synthesis of probabilistic machine-learning models for source code.

  • Method

    The survey contrasts programming and natural languages, presents a taxonomy of probabilistic source-code models, and reviews their software-engineering and programming-languages applications.

  • Results

    The reviewed evidence shows that source code contains exploitable statistical patterns, including local pattern density associated with performance improvements in code completion.

  • Takeaways & Limitations

    Probabilistic source-code models offer opportunities for learning from existing code, reasoning about new artifacts, and transferring knowledge between developers and projects.

  • Takeaways & Limitations

    Data sparsity remains unsolved because semantic representations, compiled projects, and substantial runtime data are difficult to obtain at scale.

Abstract

from arXiv · show

Research at the intersection of machine learning, programming languages, and software engineering has recently taken important steps in proposing learnable probabilistic models of source code that exploit code's abundance of patterns. In this article, we survey this work. We contrast programming languages against natural languages and discuss how these similarities and differences drive the design of probabilistic models. We present a taxonomy based on the underlying design principles of each model and use it to navigate the literature. Then, we review how researchers have adapted these models to application areas and discuss cross-cutting and application-specific challenges and opportunities.

1 INTRODUCTION

The survey frames probabilistic modeling of source code as a data-driven complement to formal software-engineering approaches, enabled by the scale of open-source repositories and motivated by the need for more reliable, maintainable software tools.

  • 1 INTRODUCTION: Open-source repositories expose billions of code tokens and millions of metadata instances, creating a large empirical resource for software-engineering research.Examples include Linux, MySQL, Django, Ant, and OpenEJB, with metadata about authorship, bug fixes, and reviews.
  • 1 INTRODUCTION: The survey argues that machine learning should complement formal program structure with statistical properties learned from code.It highlights machine learning’s ability to generalize from examples and handle noise, while emphasizing the assumptions and consequences behind these methods.
  • 1 INTRODUCTION: The article reviews probabilistic source-code models, including n-gram and deep-learning methods, and organizes them around their design principles.Its coverage extends beyond bag-of-words representations to sequences, trees, and continuous representations.
  • 1 INTRODUCTION: Applications span recommender systems, debugging, program analysis, and program synthesis, alongside a taxonomy and discussion of challenges and future directions.The survey also covers related research areas and aims to explain core methods and techniques.
  • 1 INTRODUCTION: Existing reviews had summarized the area’s progress, but the authors position this work as an extensive literature review.The survey therefore consolidates work from software engineering and programming-languages perspectives.

2 THE NATURALNESS HYPOTHESIS

The naturalness hypothesis treats software as human communication whose large corpora contain exploitable statistical regularities, motivating probabilistic models that learn how developers write and use code.

  • 2 THE NATURALNESS HYPOTHESIS: The naturalness hypothesis states that software is human communication and that software corpora have statistical properties resembling natural-language corpora.These properties can be exploited to build better software-engineering tools.
  • 2 THE NATURALNESS HYPOTHESIS: Large code corpora are expected to contain rich patterns because coding is an act of communication, enabling probabilistic machine-learning models for software tools.The hypothesis was supported by early evidence that natural-language models could work surprisingly well for source code.
  • 2 THE NATURALNESS HYPOTHESIS: The hypothesis motivates models that learn how developers naturally write and use code.Such models can augment existing tools with statistical information and support new machine-learning-based tools.
  • 2 THE NATURALNESS HYPOTHESIS: Probabilistic methods can express hypotheses with confidence values and learn uncertain correspondences between code and documents such as requirements, comments, and blog posts.The passages emphasize probability as a natural representation of uncertainty in these correspondences.
  • 2 THE NATURALNESS HYPOTHESIS: Code naturalness is connected to developers’ preference for conventional, idiomatic, and familiar code because it supports understanding and maintenance.The supplied passage introduces this connection but does not complete its broader explanation.

3 TEXT, CODE AND MACHINE LEARNING

Code resembles text in having exploitable patterns but differs through bimodality, executability, formality, and structural complexity; these differences shape how machine-learning methods must model source code.

  • 3 TEXT, CODE AND MACHINE LEARNING: Source code is bimodal: it communicates with humans through explanatory content and with computers through executable algorithms.This dual audience drives both similarities and differences between code and natural language.
  • 3 TEXT, CODE AND MACHINE LEARNING: Code is semantically brittle because small changes can drastically alter meaning, motivating combinations of probabilistic and formal methods.Natural language is generally more tolerant of mistakes than executable code.
  • 3 TEXT, CODE AND MACHINE LEARNING: Programming languages are designed formally and top-down, whereas natural languages emerge through social dynamics and change gradually.Programming languages can instead undergo punctuated changes, including releases that break backward compatibility.
  • 3 TEXT, CODE AND MACHINE LEARNING: Code can be less surprising than text, and its local pattern density has improved performance in applications such as code completion.The passage attributes this pattern density partly to code reuse and library-based development.
  • 3 TEXT, CODE AND MACHINE LEARNING: Unlike natural language, programming languages require syntactic and approximate semantic unambiguity because they are translated into machine code.This allows code models to exploit rich structure while still confronting issues such as polymorphism and aliasing.
  • 3 TEXT, CODE AND MACHINE LEARNING: Mapping code units to textual units remains open because statements, blocks, and functions do not universally correspond to sentences or other text units.This complicates tasks such as code-to-comment generation and code search.

4 PROBABILISTIC MODELS OF CODE

The survey organizes probabilistic code models by their inputs, outputs, and probability-distribution design choices, distinguishing code-generating, representational, and pattern-mining families. It then relates these families to model assumptions, structural representations, and software-engineering applications.

  • Model taxonomy: Probabilistic code models estimate distributions over code artifacts, making simplifying assumptions that improve tractability while introducing model-specific strengths and weaknesses.The survey groups models by shared design choices and notes that suitability depends on the application.
  • Model taxonomy: Code-generating models stochastically construct code from smaller elements such as tokens or AST nodes, whereas representational models predict code properties from abstractions such as token contexts or data flow.Pattern-mining models instead infer latent structure without supervision.
  • Pattern-mining models: Pattern-mining models infer latent structure without supervision and can identify reusable, human-interpretable patterns, but evaluating their outputs is difficult because quality is rarely quantifiable.Applications include API-pattern documentation, summarization, and anomaly detection.
  • Model taxonomy: The three families parallel established NLP approaches: generation resembles language modeling and translation, representation resembles classification and entity recognition, and pattern mining resembles topic modeling.These analogies help position source-code models within broader machine-learning methods for structured data.
  • Code-generating models: Existing code-generating models do not always produce code that parses, compiles, and executes because structural complexity requires simplifying assumptions.Some models impose constraints to address this limitation, but the survey does not claim universal validity.
  • Code-generating models: Code-generating models factor code generation into sequential predictions because directly modeling complete structured code is infeasible, using token-, syntax-, or semantics-aware representations.N-gram models capture local dependencies but struggle with long-range scope; parse annotations and cache mechanisms add context or exploit identifier repetition.
  • Representational models: Representational models learn conditional distributions over code properties, such as variable types, using transformed code representations and methods that often target specific applications.They complement generative models by predicting facts useful to engineers and downstream tasks such as static analysis.

5 APPLICATIONS

Probabilistic source-code models support applications across software engineering and programming-language research by handling uncertainty and making some analyses more tractable. In recommender systems, they use code context to rank completions and infer developer intent.

  • Probabilistic source-code models support applications in software engineering and programming-language research by enabling probabilistic reasoning under uncertainty.They can also simplify or accelerate analyses that would otherwise be computationally costly.
  • Recommender Systems: Recommender systems apply machine learning to software artifacts for tasks such as code completion and recommending likely code reviewers.
  • Recommender Systems: Probabilistic recommender systems infer developer intent from the context of partial code while quantifying uncertainty about the latent intent.
  • Recommender Systems: Statistical code completion learns probabilities from existing code and returns suggestions ranked by predicted relevance rather than alphabetically.Systems may complete specific constructs such as method calls and parameters or attempt to complete all code tokens.
  • Recommender Systems: Code-completion research progressed from context features for method invocations and constructors to Bayesian graphical models and cache-based n-gram language models.A context-based model captures object usages and models the probability distribution for the next call, while a local cache acts as a domain-adapted n-gram.

5.2 Inferring Coding Conventions

Machine-learning models infer coding conventions from source-code surface structure and can quantify uncertainty over convention decisions. Related defect-detection work uses code probabilities and abstractions, but sparsity, rare defects, and diverse correct code constrain precision and coverage.

  • Coding conventions impose syntactic constraints beyond grammar, governing formatting and naming choices intended to prevent some bugs and improve comprehension, navigation, and maintenance.
  • Surface-structure models infer emergent coding conventions from source code while quantifying uncertainty over their decisions.A central challenge is sparse code constructs caused by diverse, non-repeatable code within projects and domains.
  • Code Defects: Defect detection is difficult because defects are rare and correct source code is extremely diverse.
  • Code Defects: Language-model probability can indicate defects: buggy code tends to be less natural and language models have matched popular tools such as FindBugs in preliminary evidence.
  • Code Defects: Defect models use different abstraction levels, and each model captures only the defect family determined by its chosen representation.Examples include coarse-grained n-grams and paths over API calls that remove control-flow dependencies.
  • Code Defects: Probabilistic anomaly detection has seen limited industrial uptake, with code diversity creating sparsity that contributes to imprecision.

5.4 Code Translation, Copying, and Clones

Machine-learning translation models adapt statistical machine translation to source code, while semantic constraints reduce invalid translations. Current work largely addresses similar programming paradigms, leaving cross-paradigm and memory-management differences as important limitations.

  • Code Translation: Statistical machine translation can replace tedious rule maintenance for translating between evolving programming languages, but it tends to produce invalid code.
  • Code Translation: Adding semantic constraints to the translation process reduces translation errors.
  • Code Translation: Existing models mainly translate between languages with similar paradigms and structures, such as managed-memory object-oriented C# and Java.
  • Code Translation: Translating across paradigms or memory-management models requires preserving conceptual and semantic differences, including deallocation in C and loop-to-map-reduce transformations.
  • Code Translation: Translation models are evaluated using exact matches, syntactic or semantic correctness, and BLEU.
  • Copying and Clones: Structured prediction and distributed representations can adapt variables in copied snippets to target contexts, automating naming cleanup and reducing collisions.

5.5 Code to Text and Text to Code

Probabilistic models connect natural-language text and source code for applications including synthesis, traceability, search, documentation, and readability. Code-to-text systems generate natural-language descriptions, while text-to-code systems aim to help users write programs from language.

  • Connecting text and code is difficult because both are diverse, text is ambiguous, code is compositional, and software uses layered abstractions.Probabilistic machine-learning models provide a principled way to model and resolve ambiguities in both.
  • Code to Text: Code-to-text generation supports documentation and readability by translating code into pseudocode, summaries, or comments.Examples use machine translation, neural attention, n-gram, and topic models.
  • Text to Code: Text-to-code generation aims to help developers and end users write programs more easily.The area is closely related to semantic parsing, which converts natural-language utterances into representations of meaning.
  • Text to Code: Neural architectures have been proposed for general-purpose code generation.

5.6 Documentation, Traceability and Information Retrieval

Probabilistic models connect natural-language documentation and source code for search, traceability, information extraction, and synthesis. The surveyed work spans language models, neural sequence models, graphical models, classifiers, and learned search heuristics.

  • Documentation and Information Retrieval: Documentation and code search are central software-engineering problems where probabilistic models can integrate natural-language text with code.Applications include code search, requirements and specifications, traceability, and documentation maintenance.
  • Documentation and Information Retrieval: Sequence-to-sequence neural networks have been trained to map natural-language queries into API sequences for code search.
  • Documentation and Information Retrieval: Language models can identify code fragments in free text and software-relevant messages in social-media text.
  • Program Synthesis: Program synthesis generates full or partial programs from formal specifications, input/output pairs, or natural-language descriptions.Natural-language specifications correspond to semantic parsing.
  • Program Synthesis: Programming-by-example research uses graphical models, parameterized probabilistic context-free grammars, classifiers, and learned heuristics to guide synthesis.These methods learn from task commonalities, input/output features, or synthesized-program features.
  • Program Synthesis: A character-level LSTM can generate large numbers of valid-looking OpenCL benchmarks for compiler fuzzing rather than satisfy a specification.

5.8 Program Analysis

Probabilistic source-code models support program analysis by estimating uncertain properties, reducing false positives, and generating plausible verification hypotheses. The surveyed approaches trade some formal guarantees for statistical predictions or use learning to guide sound analyses.

  • Program Analysis: Probabilistic models address uncertainty in program analysis, where sound analyses can become imprecise and produce many false positives.
  • Program Analysis: One family relaxes soundness and produces probabilistic results, such as learned distributions over JavaScript variable types.
  • Program Analysis: Classifiers and recurrent neural networks learn to parameterize analyses or distinguish false-positive from true-positive static-analysis warnings.
  • Program Analysis: A second paradigm uses machine learning to generate plausible formal-verification hypotheses that can subsequently be proved.

6 CHALLENGES AND FUTURE DIRECTIONS

The survey identifies uncertainty, sparse and compositional data, representation gaps, deployment constraints, imperfect evaluation, and limited application data as major challenges for machine learning on source code. Future progress depends on models that better incorporate programming-language structure and support practical deployment.

  • Motivation and Research Waves: Machine learning is introduced to handle uncertainty and ambiguity in software development and to avoid hard-coded heuristics.The surveyed direction targets questions about code purpose, test behavior, specifications, input domains, and crashes.
  • Motivation and Research Waves: The second wave learns directly from source code without manual feature extraction, while a third wave is envisioned around programming-language semantics.
  • Motivation and Research Waves: Source-code machine learning is not uniformly improved by deep learning: deterministic problems may not require it, and simple models can outperform generic deep methods.
  • Representations and Communities: Bridging symbolic programming-language representations and continuous machine-learning representations remains a central challenge for reasoning about code structure and semantics.
  • Model Design: Models must balance explicit source-code structure and constraints against general-purpose methods, with domain inductive biases often improving performance with less data.Modular neural architectures are proposed as one way to compose structure-aware components.
  • Data Sparsity and Generalization: Data sparsity is unresolved because reusable software yields few examples of distinct industrial programs, unlike the abundance of repeated natural-language topics.
  • Data Sparsity and Generalization: Highly compositional code remains difficult to model because relations between objects across abstraction levels are hard to capture.
  • Deployment: Large models can be difficult to deploy locally, while cloud use raises privacy concerns and rapid code evolution demands costly retraining.

7 RELATED RESEARCH AREAS

The survey relates statistical code modeling to software-repository mining, probabilistic programming, formal software models, and non-learning probabilistic code models. These areas overlap in subject matter but differ in direction, modeling goals, and guarantees.

  • Software Repository Mining and Big Code: Mining software repositories and Big Code analyze source code and metadata from repositories to improve software tools and processes.
  • Probabilistic Programming: Probabilistic programming applies programming-language concepts to make machine-learning algorithm development easier, whereas statistical code models apply machine learning to program development.
  • Formal and Statistical Models: Formal semantic models and statistical source-code models are distinct, but combining them is identified as an interesting direction for future research.

8 CONCLUSIONS

The survey finds that probabilistic source-code models have broad potential for program analysis and software engineering, while enabling learning, reasoning, and knowledge transfer across code artifacts, developers, and projects.

  • Probabilistic source-code models could support new tools across almost every area of program analysis and software engineering.
  • The survey organizes existing research through a taxonomy of probabilistic machine-learning models and their applications.
  • These models learn from existing code, reason probabilistically about new source-code artifacts, and transfer knowledge between developers and projects.
  • Online datasets used in probabilistic source-code research are listed alphabetically in Table 4.
Loading 1709.06182v2…