Source-linked AI summary
Learning Natural Coding Conventions
Miltiadis Allamanis, Earl T. Barr, Christian Bird, Charles Sutton
TL;DR
Implicit coding conventions are difficult for programmers and rule-based tools to infer, despite their importance to readable and maintainable software. NATURALIZE learns a codebase’s local style with statistical language-modeling techniques and suggests naming and formatting revisions through four tools. It achieved 94% accuracy in top identifier-name suggestions, and 14 of 18 patches submitted to five open-source projects were accepted.
Problem
Implicit coding conventions emerge from consensus and are difficult to codify, while roughly one third of code reviews contain convention feedback.
Method
NATURALIZE learns local coding conventions from a training corpus and ranks candidate naming or formatting revisions by statistical naturalness.
Results
NATURALIZE achieved 94% accuracy in its top identifier-name suggestions, and 14 of 18 submitted patches for five open-source projects were accepted.
Takeaways & Limitations
NATURALIZE provides convention-aware tools for improving stylistic consistency during development, release management, and code review.
Takeaways & Limitations
Naming evaluators were not developers of the test projects, so those developers may have held different opinions about the suggestions.
Abstract
from arXiv · showhide
Every programmer has a characteristic style, ranging from preferences about identifier naming to preferences about object relationships and design patterns. Coding conventions define a consistent syntactic style, fostering readability and hence maintainability. When collaborating, programmers strive to obey a project's coding conventions. However, one third of reviews of changes contain feedback about coding conventions, indicating that programmers do not always follow them and that project members care deeply about adherence. Unfortunately, programmers are often unaware of coding conventions because inferring them requires a global view, one that aggregates the many local decisions programmers make and identifies emergent consensus on style. We present NATURALIZE, a framework that learns the style of a codebase, and suggests revisions to improve stylistic consistency. NATURALIZE builds on recent work in applying statistical natural language processing to source code. We apply NATURALIZE to suggest natural identifier names and formatting conventions. We present four tools focused on ensuring natural code during development and release management, including code review. NATURALIZE achieves 94% accuracy in its top suggestions for identifier names and can even transfer knowledge about conventions across projects, leveraging a corpus of 10,968 open source projects. We used NATURALIZE to generate 18 patches for 5 open source projects: 14 were accepted.
1. INTRODUCTION
Coding conventions emerge from programmers’ local stylistic choices, but implicit conventions are difficult to codify and follow. NATURALIZE learns these conventions from code and suggests natural naming and formatting revisions, supported by tools for development and release management.
- Coding conventions shape source readability, portability, accessibility to newcomers, reliability, and maintainability.
- Roughly one third of code reviews contain feedback about following coding conventions, showing that teams enforce them rigorously.
- Implicit conventions emerge from consensus and are difficult for fixed rule-based formatters to codify or enforce.
- NATURALIZE learns local conventions descriptively and recommends changes only when the codebase provides sufficient evidence of consensus.
- NATURALIZE identifies surprising names or formatting through statistical language modeling and proposes less surprising alternatives.
- NATURALIZE provides four tools for convention-aware development and release management, including pre-commit checks, formatter rules, an Eclipse plugin, and code-review profiling.
- NATURALIZE applies language-independent techniques using identifiers, keywords, and operators, supporting domain-specific, esoteric, and multi-language projects.
- 94% accuracy in top identifier-name suggestions and a mean formatting-suggestion accuracy never below 96% were reported.
2. MOTIVATING EXAMPLE
Coding conventions affect readability and maintainability, yet developers may violate them during collaborative review. NATURALIZE addresses this by learning project conventions and suggesting more stylistically consistent names and formatting.
- Coding-convention violations can arise even when submitted code is functionally correct, prompting reviewer feedback about names, line length, and formatting.The motivating example shows a Microsoft developer’s class modified after review requests.
- NATURALIZE can evaluate selected code, identify names inconsistent with a codebase’s parameter conventions, and suggest alternatives.The motivating scenario applies NATURALIZE to parameter names such as res and str.
- Release management brings individual coding styles into contact, making convention adherence important for code readability and maintainability.The paper frames release management as committing, reviewing, and promoting changes.
- Because statistically inferred changes may introduce bugs, NATURALIZE’s use cases require that the benefit of a suggested modification justify its cost.The paper assumes human approval of suggestions while retaining concern about modification risk.
- NATURALIZE provides tools for individual or selected-code suggestions, codebase convention profiles, formatter-rule generation, and high-precision pre-commit checks.The tools include devstyle, styleprofile, genrule, and stylish?.
- NATURALIZE learns conventions from an existing codebase or from a pre-packaged model trained on popular open-source projects.The current codebase supplies domain-specific conventions, while the packaged model supports reuse across projects.
3. THE NATURALIZE FRAMEWORK
NATURALIZE infers coding conventions from code and ranks candidate revisions by naturalness, supporting suggestions for names, formatting, and formatter rules. Its architecture combines proposers that generate alternatives with scoring and confidence mechanisms that limit low-quality recommendations.
- Core architecture: NATURALIZE takes a code snippet, generates replacement candidates, scores them for naturalness, and presents a short ranked list or no suggestion.The framework can also return a binary naturalness decision for applications such as pre-commit checking.
- Core architecture: Proposers modify input snippets into candidate revisions, such as replacing a variable name with names used in similar project contexts.Filtering logic removes implausible alternatives before scoring.
- Scoring and suggestions: Naturalness is measured against a training corpus for a particular codebase, using a scoring function such as a statistical language model.The framework builds on statistical NLP because prior work found language models effective at capturing code naturalness.
- Scoring and suggestions: The suggest function sorts candidates by score, keeps at most k alternatives, filters candidates below a confidence gap t, and avoids suggesting the unchanged input when it is already best.These controls reduce excessive or low-quality recommendations.
- Binary decisions: The binary decision function rejects a snippet when the best available improvement exceeds threshold T, trading fewer false rejections against more unnatural snippets passing through.Increasing T makes rejection less frequent and reduces the chance of rejecting well-written snippets.
- Binary decisions: Confidence thresholds can be selected by estimating false-positive rates on random training-corpus snippets and searching for a threshold under an acceptable bound α.The procedure assumes the training corpus contains natural code.
- Scoring and suggestions: NATURALIZE can transfer convention information across projects by averaging global and application-specific language models in a cross-project model.The global model is trained on a collection of stylistically good projects, while the local model uses the current application.
- Formatter integration: The inferred conventions can be converted into constraints and settings for a code formatter.Formatter rules represent settings such as brace placement or spacing together with constraints over those settings.
4. EVALUATION
NATURALIZE was evaluated through empirical studies, automatic perturbation tests, robustness analyses, qualitative inspection, and patches submitted to open-source projects. The evaluations found that convention violations matter to developers, suggestions can achieve high accuracy, and project developers sometimes accepted or acted on the recommendations.
- 4.1 The Importance of Coding Conventions: Two empirical studies found that programmers violate coding conventions, while project members correct such violations.The evaluation examined commit messages and code reviews to assess both nonadherence and concern about adherence.
- 4.1 The Importance of Coding Conventions: Between 7,560 and 18,900 Microsoft reviews received feedback about formatting, while 21,420 to 39,060 resulted in name changes during one month.These ranges are based on confidence intervals for just over 126,000 reviews completed during February 2014.
- 4.2 Suggestion: 94% suggestion accuracy across identifier types was achievable even when NATURALIZE made suggestions at half of possible opportunities.Higher confidence thresholds reduce suggestion frequency while increasing accuracy; performance varies by project and identifier type.
- 4.2 Suggestion: Cross-project language models improved suggestion accuracy at k = 5, supported by 14.8% of variable names being shared across projects.The cross-project model was evaluated against a base model to assess whether naming information could transfer across projects.
- 4.2 Suggestion: At a false-positive rate of at most 0.05, stylish? correctly rejected 40% of perturbed snippets.The system performed somewhat worse on perturbed variable names than on formatting, but the authors describe its filtering performance as suitable for a pre-commit script.
- 4.3 Robustness of Suggestions: NATURALIZE preserved unusual identifier names: for reasonable thresholds, it proposed alternatives for only about 20% of identifiers unknown to its language model.The authors interpret this as evidence that the naming suggestions are qualitatively reasonable, while noting that evaluators were not developers of the test projects.
5. RELATED WORK
NATURALIZE extends prior work on coding conventions, identifiers, formatters, code completion, and language models by inferring local style across naming and formatting.
- NATURALIZE is presented as the first system to infer coding conventions from a codebase across naming and formatting.
- Coding conventions support consistent measurement, reduce systematic error, and eliminate trivial convention-enforcing commits.
- Existing rule-based formatters constrain simple naming patterns or whitespace but cannot capture contextual conventions such as using i as a loop variable.
- Unlike API recommenders and code-completion systems, NATURALIZE focuses on revising existing code rather than helping developers find entities during editing.
- Language models provide a principled basis for code suggestions, but their use for programming languages and revision of existing text remains relatively limited.
6. CONCLUSION
NATURALIZE learns local style from a codebase and suggests changes that improve stylistic consistency. It recommends changes only when the codebase provides sufficient evidence of emerging consensus, while achieving 94% accuracy for identifier-name suggestions and revising mature open-source projects.
- NATURALIZE learns local style from a codebase and provides suggestions to improve stylistic consistency.
- NATURALIZE suggests changes only when there is sufficient evidence of emerging consensus in the codebase.
- 94% accuracy was achieved in NATURALIZE’s top suggestions for identifier names.
- NATURALIZE generated useful revisions for mature, high-quality open-source projects.