Source-linked AI summary
Big Code != Big Vocabulary: Open-Vocabulary Models for Source Code
Rafael-Michael Karampatsis, Hlib Babii, Romain Robbes, Charles Sutton, Andrea Janes
TL;DR
Source-code NLMs struggle with large, sparse vocabularies, OOV identifiers, and rare words, limiting their scalability. The paper studies vocabulary choices, develops an open-vocabulary BPE NLM, and evaluates it across code corpora. The resulting model scales to 13,362 projects and outperforms prior models across completion and bug-detection tasks in Java, C, and Python.
Problem
Large and sparse source-code vocabularies, unseen identifiers, and rare words hinder closed-vocabulary NLM scalability and prediction.
Method
The paper studies vocabulary-design choices and develops an open-vocabulary source-code NLM using subword units, BPE, beam search, dynamic adaptation, and caching.
Results
The open-vocabulary NLM scales to 13,362 projects and outperforms recent state-of-the-art models for code completion and bug detection across Java, C, and Python.
Takeaways & Limitations
Subword-based open-vocabulary modeling handles unseen identifiers while shrinking vocabulary by three orders of magnitude for large-scale code modeling.
Takeaways & Limitations
The study evaluates only three languages and omits some model architectures and combinations because of resource constraints.
Abstract
from arXiv · showhide
Statistical language modeling techniques have successfully been applied to large source code corpora, yielding a variety of new software development tools, such as tools for code suggestion, improving readability, and API migration. A major issue with these techniques is that code introduces new vocabulary at a far higher rate than natural language, as new identifier names proliferate. Both large vocabularies and out-of-vocabulary issues severely affect Neural Language Models (NLMs) of source code, degrading their performance and rendering them unable to scale. In this paper, we address this issue by: 1) studying how various modelling choices impact the resulting vocabulary on a large-scale corpus of 13,362 projects; 2) presenting an open vocabulary source code NLM that can scale to such a corpus, 100 times larger than in previous work; and 3) showing that such models outperform the state of the art on three distinct code corpora (Java, C, Python). To our knowledge, these are the largest NLMs for code that have been reported. All datasets, code, and trained models used in this work are publicly available.
1 INTRODUCTION
Source-code language models support software-engineering tools, but identifier growth creates vocabulary and OOV challenges that limit neural models’ scalability. This paper studies vocabulary choices, develops an open-vocabulary NLM, and evaluates it across languages and tasks.
- Source-code language models have been applied to code completion, readability improvement, program repair, bug identification, and API migration.
- Identifier creation makes software vocabulary extremely large and sparse, while unseen identifiers create an OOV problem for neural language models.
- Vocabulary design choices change vocabulary size by up to three orders of magnitude, and BPE adaptations are needed for manageable open vocabularies.
- The proposed open-vocabulary NLM combines BPE, beam search, and caching and scales to 13,362 software projects.
- Across Java, C, and Python, the open-vocabulary NLM outperforms n-gram and closed-vocabulary models for code completion and improves buggy-code highlighting.
- The improved model may support future software-development tools beyond autocompletion, including readability and program repair tools.
2 BACKGROUND AND RELATED WORK
Language models face scalability, OOV, and rare-word problems when vocabularies are large. Open-vocabulary subword methods address these issues, motivating their application to source code.
- NLMs model longer context than n-grams, but large vocabularies still create memory and computation costs.
- Large vocabularies make NLM embedding layers and full next-token probability predictions expensive to store and compute.
- Closed-vocabulary models replace unseen test words with an unknown token, losing information and preventing direct prediction of those words.
- Rare words receive unreliable embeddings because they are observed too infrequently for meaningful updates.
- Open-vocabulary character models eliminate OOV tokens but must model longer dependencies, while subword models combine character-level coverage with token-level units.
- Cache and copy mechanisms reuse previously seen words but cannot help when an OOV word appears for the first time.
- In source code, arbitrary and compound identifiers cause vocabulary explosion, limiting closed-vocabulary NLM scalability.
- Convention splitting handles some OOV tokens but cannot represent arbitrary subtoken combinations and leaves many subtokens infrequent.
3 DATASETS
The study evaluates language models on Java, C, and Python corpora selected for differences in programming style and language characteristics. The datasets use held-out validation, testing, and subword-encoding subsets.
- The corpora cover Java, C, and Python, whose differing programming paradigms and typing characteristics may affect language-model performance.
- Lexical analysis used a Java lexer for Java and Pygments for C and Python, while preprocessing followed the prior corpus setup.
- For Python and C, 1% of each corpus was reserved for validation and 1% for testing.
- Python and C also used a separate 10% sample to learn BPE subword encodings, with the remaining data used for training.
- The Java split preserved the validation and test sets from prior work and used 1,000 random projects for subword-encoding data.
4 MODELING VOCABULARY
The study evaluates vocabulary-design choices for source code by measuring scalability, information loss through OOV rates, and word frequency. It finds that splitting and BPE substantially reduce vocabulary and OOV problems, while some alternatives impose costly trade-offs.
- Baseline and evaluation criteria: More than 11 million unique Java words produce over 40% test-set OOV, while a 75K vocabulary leaves close to 80% of test vocabulary unseen.The baseline also has 83% of vocabulary words occurring 10 times or less, including 25% occurring once.
- Word splitting: Convention-based word splitting reduces vocabulary by nearly an order of magnitude and decreases OOV by a factor of 5 in the unfiltered case.The reduction comes with a 57% increase in corpus size and only 3% more words occurring over 10 times.
- Subword splitting: 37% vocabulary reduction from digit-and-character modeling requires only a 2% corpus-size increase, with a 2% OOV improvement for a 75K vocabulary.It also increases the share of words occurring at least 10 times to 28%.
- Subword splitting: Spiral splitting yields a further 26% vocabulary reduction after number splitting and reports 9% unseen words for the evaluated vocabulary configuration.The passage describes Spiral as the state-of-the-art token splitter used in the study.
- Alternative approaches: Stemming reduces vocabulary by 5% but loses information, whereas character models eliminate OOV issues while drastically increasing sequence lengths.The paper therefore does not consider character modeling desirable and views stemming as an unfavorable trade-off.
- BPE: BPE constructs a vocabulary from characters and learned merges, then segments new data by applying the ordered merge operations.The number of merge operations controls the final vocabulary size.
- BPE: With 10K BPE merges, OOV issues vanish and more than 97% of remaining words occur over 1,000 times.The corpus grows, but by no more than the earlier vocabulary choices explored.
- BPE: BPE shrinks source-code vocabulary effectively and makes most vocabulary entries frequent, improving embeddings.
5 NEURAL LANGUAGE MODEL FOR CODE
The paper’s open-vocabulary neural language model predicts BPE-derived subwords rather than complete tokens. Beam search reconstructs ranked token predictions, while caching and project adaptation address code locality and new-project behavior.
- Model architecture: The model uses an RNN language model over BPE subword units, with a small single-layer GRU as its base architecture.The base model uses 512-feature representations and a GRU state of the same size.
- Subword modeling: Training segments train, validation, and test tokens with BPE encodings learned from held-out projects using 2,000, 5,000, or 10,000 maximum merges.
- Beam search: A custom beam-search procedure approximates top-k complete-token predictions from subword-unit sequences.A token is complete when its final subword ends with </t> and earlier subwords do not.
- Caching: The cache stores identifiers following each five-token history and retrieves them when the same history recurs.The cache weight is 0.3 and this mechanism operates only at test time.
- Project adaptation: Dynamic adaptation updates the global NLM with one gradient step on each encountered sequence after testing it on a new project.The procedure is equivalent to one training epoch on that project while avoiding training on the test set.
6 EVALUATION
The evaluation measures intrinsic language-model quality and extrinsic code-completion performance under multiple deployment scenarios. It also isolates identifier prediction and distinguishes static, dynamic, and maintenance settings.
- Intrinsic evaluation: Cross entropy measures average bits required per prediction, with lower values indicating better language-model performance.It rewards both correct predictions and high confidence.
- Intrinsic evaluation: For subword models, token probability is computed as the product of the probabilities of its constituent subwords.This product permits cross-entropy computation at the token level.
- Code completion: Code completion predicts each test-file token from preceding tokens and is evaluated with mean reciprocal rank over 10 ranked predictions.A rank-1 correct suggestion contributes 1, rank 2 contributes 0.5, and rank 10 contributes 0.1.
- Completion scenarios: Static tests evaluate globally trained models on separate projects without adaptation, while dynamic tests permit parameter updates after predictions on test-project files.
- Completion scenarios: Maintenance tests train on all other files in the test project and evaluate one held-out file at a time, requiring retraining for each test file.
- Identifier evaluation: Identifier-only evaluation reports entropy, MRR, recall at rank 1, and recall at rank 10 for identifier usages excluding primitive types.
7 RESEARCH QUESTIONS
The research questions test comparative performance, scalability, cross-language robustness, dynamic adaptation, and usefulness beyond code completion. The study compares subword NLMs with established language-model baselines and evaluates buggy-code detection.
- RQ1: Comparative performance: RQ1 compares subword NLMs with n-gram, cache, nested-cache, token-level, and heuristic-splitting language models for code.The comparison covers code-completion performance.
- RQ2: Scalability: RQ2 asks whether subword NLMs scale to large code corpora and whether additional training data improves performance despite diminishing returns and rising resource costs.The study considers runtime, memory, and storage as scalability outcomes.
- RQ3: Programming languages: RQ3 evaluates whether language-agnostic NLM methods remain effective across Java, C, and Python.The paper specifically considers C’s terseness and Python’s lack of type information as possible challenges.
- RQ4: Dynamic adaptation: RQ4 tests whether dynamic updating adapts subword NLMs to new projects containing identifiers absent from a large cross-project corpus.The technique is compared with cache and nested-cache n-gram models.
- RQ5: Beyond completion: RQ5 tests usefulness beyond completion by evaluating whether NLMs improve on n-gram models for detecting buggy code.The broader transfer-learning scenarios discussed are outside the paper’s scope.
8 RESULTS
Across evaluation settings, open-vocabulary NLMs outperform closed-vocabulary and n-gram baselines, scale better with larger corpora, and remain effective across Java, C, and Python. Dynamic adaptation, caching, and larger capacity further improve performance while preserving practical resource usage.
- RQ1. Performance of Models: The BPE NLM outperforms nested-cache n-gram models on Java, predicting 74% of identifiers within its top 10 dynamic predictions.Nearly 56% of identifiers are predicted in first position.
- RQ1. Performance of Models: Open-vocabulary NLMs significantly outperform both closed-vocabulary variants and remain trainable in the maintenance setting.The closed models use full tokens or convention-based token splitting, while the open model uses the same architecture otherwise.
- RQ1. Performance of Models: The cache improves MRR by 3 to 4%, while increased capacity adds nearly 3% on the small corpus; together they raise identifier MRR by close to 6%.The two improvements are complementary, with the cache especially helping low ranks relevant to completion.
- RQ2. Large Corpora: Larger corpora improve open-vocabulary NLMs more than n-gram models, increasing static MRR by about 6% and dynamic MRR by 2 to 3%.For dynamic adaptation, entropy improves by 0.5 bits, whereas nested-cache n-gram improvements remain below 0.4% MRR.
- RQ3. Programming Languages: The NLM results hold across Java, C, and Python, with slightly worse overall performance for C and somewhat better performance for Python.N-gram models perform considerably worse in Python, which the authors hypothesize is related to smaller project sizes and reduced caching opportunity.
9 CONCLUSIONS
The paper finds that open-vocabulary modeling is necessary for scalable source-code NLMs and reports strong performance across languages and tasks. It also identifies evaluation limits imposed by computational resources and scope.
- Open-vocabulary NLMs are the only viable option because alternative vocabulary choices produce large vocabularies, high OOV rates, and rare words.
- Subword units let the model handle unseen identifiers while shrinking vocabulary by three orders of magnitude and scaling to datasets over 100 times larger than prior code NLMs.
- The NLM outperforms recent state-of-the-art nested-cache n-gram models for code completion and bug detection across varied scenarios and three programming languages.
- Resource constraints prevented evaluation of all model and training combinations, additional NLM architectures, more than three languages, and code duplication effects.
- Improved code language models could support tools for readability, program repair, synthesis, and translation between programming languages.
10 ARTIFACTS
The study releases the data, preprocessing code, open-vocabulary NLM implementation, evaluation scripts, and trained models to support replication and reuse.
- The study provides DOI references for the specific artifact versions used in the research.
- The Java, C, and Python datasets are reused or preprocessed for vocabulary analysis and NLM training with defined training and test sets.
- The codeprep library supports varied source-code preprocessing and was used to gather vocabulary statistics.
- The OpenVocabCodeNLM library contains the open-vocabulary model, training procedure, and evaluation scripts for extension by researchers.
- Released trained models support inference for code completion and can be fine-tuned for tasks such as classification.