Source-linked AI summary
Code Completion with Neural Attention and Pointer Networks
Jian Li, Yue Wang, Michael R. Lyu, Irwin King
TL;DR
Code completion for dynamically typed languages is limited by scarce type information, long-range dependencies, and incorrect prediction of OoV words. The paper introduces AST-aware attention and a pointer mixture network that switches between global generation and local copying. Experiments on JavaScript and Python benchmark datasets demonstrate the effectiveness of these approaches.
Problem
Code completion for dynamically typed languages faces limited type information, long-range dependencies, and an OoV problem that standard neural language models cannot correctly predict.
Method
The paper combines AST parent-child attention with a pointer mixture network that switches between a global attentional RNN and a local pointer component.
Results
Experiments on JavaScript and Python benchmark datasets demonstrate the effectiveness of the proposed attention mechanism and pointer mixture network.
Takeaways & Limitations
The model learns when to generate a vocabulary word and when to copy an OoV word from local context for code completion.
Takeaways & Limitations
Pointer networks alone cannot predict words beyond the current input sequence, while the parent-location index is set to 1 when it exceeds the context length.
Abstract
from arXiv · showhide
Intelligent code completion has become an essential research task to accelerate modern software development. To facilitate effective code completion for dynamically-typed programming languages, we apply neural language models by learning from large codebases, and develop a tailored attention mechanism for code completion. However, standard neural language models even with attention mechanism cannot correctly predict the out-of-vocabulary (OoV) words that restrict the code completion performance. In this paper, inspired by the prevalence of locally repeated terms in program source code, and the recently proposed pointer copy mechanism, we propose a pointer mixture network for better predicting OoV words in code completion. Based on the context, the pointer mixture network learns to either generate a within-vocabulary word through an RNN component, or regenerate an OoV word from local context through a pointer component. Experiments on two benchmarked datasets demonstrate the effectiveness of our attention mechanism and pointer mixture network on the code completion task.
1 Introduction
Code completion is especially challenging for dynamically typed languages because type information is limited, neural models struggle with long-range dependencies and OoV words, and pointer networks lack global vocabulary coverage. The paper addresses these issues with AST-aware attention and a pointer mixture network that switches between generating vocabulary words and copying locally repeated words.
- Code completion is less supported for dynamically typed languages because they lack type annotations that traditionally guide next-token prediction.
- Standard RNN language models struggle with long-range dependencies because they compress sequence information into a fixed-size hidden state.Program source code commonly reuses identifiers declared many lines earlier.
- Attention retrieves relevant previous hidden states, and the proposed mechanism additionally exploits parent-child structure in program ASTs.
- A limited vocabulary replaces rare OoV words with UNK, which cannot provide useful code-completion recommendations.The issue is particularly important when program corpora contain many unique words.
- The pointer mixture network switches between a global RNN that generates vocabulary words and a local pointer that copies words from context.This design targets locally repeated terms while retaining access to the global vocabulary.
- Experiments on JavaScript and Python benchmark datasets report improvements over state-of-the-art methods.
2 Approach
The approach represents programs as AST-node sequences and predicts node types and values with separate neural language models. It combines AST-aware attention with a pointer mixture network to balance global vocabulary generation and local copying for code completion.
- AST-Based Code Completion: Programs are parsed into ASTs, flattened by in-order depth-first traversal, and represented as sequences of nodes containing type and value attributes.AST-based completion preserves structural information and can complete larger code blocks than token-only prediction.
- AST-Based Code Completion: The task predicts the next AST node type and value separately, using one model for each prediction task.Given preceding words w_1, ..., w_t−1, the models predict the next word w_t and its type or value.
- Neural Language Model: Each LSTM cell receives Type and Value embeddings, while the prediction conditions on the current hidden state, context vector, and parent vector.The resulting representation is projected into vocabulary space and normalized with a softmax.
- Attention Mechanism: The attentional LSTM retrieves previous hidden states through context attention and retrieves a parent-node hidden state using recorded AST parent locations.When a parent lies beyond the memory window, its location is set to 1.
- Pointer Mixture Network: The pointer mixture network combines an attentional LSTM global component with a local pointer component and a switcher balancing their outputs.The pointer distribution reuses attention scores; the selected output is generated from the global vocabulary or copied from local context.
3 Evaluation
The evaluation uses JavaScript and Python AST datasets to compare LSTM variants, attention, and pointer mechanisms. Results show consistent benefits from attention and pointer copying, especially for out-of-vocabulary prediction and state-of-the-art comparisons.
- Dataset: Experiments use 150,000 JavaScript and Python program files each, with 100,000 for training and 50,000 for testing.The ASTs are serialized using in-order depth-first traversal to generate training and evaluation queries.
- Experimental Setup: The models compare vanilla LSTM, attentional LSTM, and the proposed Pointer Mixture Network.The attentional LSTM uses context and parent attention, while the Pointer Mixture Network combines attentional generation with pointer copying.
- OoV Prediction: Across vocabulary sizes, vanilla LSTM has the lowest accuracy, attentional LSTM improves on it, and the Pointer Mixture Network achieves the highest accuracy.The mixture model’s gain over attentional LSTM is largest with a 1k vocabulary, where the OoV rate is highest.
- State-of-the-Art Comparison: Three of four benchmark tasks reach state-of-the-art performance using the attentional LSTM and Pointer Mixture Network.The attentional LSTM leads next-type prediction on both datasets; the pointer mixture matches Raychev et al. on JavaScript values, exceeds Liu et al. on JavaScript, and exceeds the prior best on Python values.
- Pointer Analysis: A random pointer distribution performs worse than the learned pointer mixture on 1k-vocabulary value prediction in both datasets.This comparison supports that the pointer component learns when and where to copy OoV values.
- Case Study: In the case study, the Pointer Mixture Network copies the OoV value employee id from an earlier occurrence, unlike the other LSTM variants.Vanilla LSTM predicts EMPTY, while attentional LSTM predicts UNK rather than the actual value.
4 Related Work
Related work applies statistical models, probabilistic grammars, neural networks, and pointer mechanisms to code completion. Pointer approaches differ by task setting and whether they reproduce rare in-vocabulary identifiers or address OoV words.
- Prior code-completion research includes n-gram models, probabilistic grammars, and neural networks for modeling source code.
- Bhoopchand et al. proposed a sparse pointer mechanism in an RNN to better predict identifiers in Python source code.
- Gulcehre et al. combined RNN generation with local pointing for sequence-to-sequence tasks such as neural machine translation, unlike this language-modeling scenario.
- Merity et al. used pointing to reproduce rare words that remain in the global vocabulary, rather than out-of-vocabulary words.
5 Conclusion
The paper applies neural language models and AST-informed attention to code completion, while using a pointer mixture network to address OoV values. Experiments demonstrate the effectiveness of these approaches.
- The paper develops AST attention using parent-children information and a pointer mixture network for OoV values in code completion.
- The pointer mixture network learns either to generate a new value with an RNN or copy an OoV value from local context with a pointer component.
- Experimental results demonstrate the effectiveness of the proposed attention mechanism and pointer mixture network.