PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Transfer Learning for Source Code Modeling

Yasir Hussain, Zhiqiu Huang, Yu Zhou, Senzhang Wang

arXiv:1910.05493v2cs.LGcs.NEcs.SEstat.ML

TL;DR

Problem-specific, data-hungry code models require retraining for related tasks. This work transfers knowledge from pretrained RNN and GRU models through an attention learner to source-code suggestion, reporting significant improvement over state-of-the-art language models on real-world data.

  • Problem

    Deep-learning code models are problem-specific, data-hungry, and may require days of training, while related tasks require training from scratch.

    Source: §1 · p. 2

  • Method

    The approach uses pretrained RNN and GRU models as knowledge sources and combines their learned representations in an attention learner fine-tuned for source-code suggestion.

    Source: §1 · p. 2

  • Results

    Extensive evaluation on a real-world dataset reports significant improvement over state-of-the-art language models.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper presents transfer learning as a potentially beneficial direction for source-code modeling and a starting point for further research.

    Source: §6.4 · p. 17

    The approach could be improved through hyperparameter optimization, more complex architectures, or training on a larger dataset.

    Source: §6.4 · p. 17

Abstract

from arXiv · show

In recent years, deep learning models have shown great potential in source code modeling and analysis. Generally, deep learning-based approaches are problem-specific and data-hungry. A challenging issue of these approaches is that they require training from starch for a different related problem. In this work, we propose a transfer learning-based approach that significantly improves the performance of deep learning-based source code models. In contrast to traditional learning paradigms, transfer learning can transfer the knowledge learned in solving one problem into another related problem. First, we present two recurrent neural network-based models RNN and GRU for the purpose of transfer learning in the domain of source code modeling. Next, via transfer learning, these pre-trained (RNN and GRU) models are used as feature extractors. Then, these extracted features are combined into attention learner for different downstream tasks. The attention learner leverages from the learned knowledge of pre-trained models and fine-tunes them for a specific downstream task. We evaluate the performance of the proposed approach with extensive experiments with the source code suggestion task. The results indicate that the proposed approach outperforms the state-of-the-art models in terms of accuracy, precision, recall, and F-measure without training the models from scratch.

1. Introduction

Deep-learning code models are task-specific, data-hungry, and costly to train, motivating transfer learning. The paper combines pretrained RNN and GRU models in an attention learner and reports improved performance on source-code suggestion.

1. Introduction

  • The introduction situates the work among deep-learning applications in code suggestion, syntax-error detection and correction, and code summarization.Prior code-suggestion models use a fixed-size context to predict the next token.Source: §1 · p. 2
  • Problem-specific deep-learning approaches require retraining for related tasks, need large datasets, and can take days to train.Transfer learning reuses knowledge from a pretrained model for a related downstream task.Source: §1 · p. 2
  • The proposed approach transfers learned knowledge from pretrained source-code language models to related tasks and fine-tunes it without training from scratch.The downstream task evaluated here is source-code suggestion.Source: §1 · p. 2
  • The approach combines two pretrained recurrent models, RNN and GRU, in an attention learner for downstream-task fine-tuning.The models are presented for transfer learning in source-code modeling.Source: §1 · p. 2
  • Extensive evaluation on a real-world dataset reports significant improvement over state-of-the-art language models.The evaluated downstream task is source-code suggestion.Source: §1 · p. 2

2. Related Work

The related-work section surveys source-code modeling and transfer learning. It contrasts task-specific code-suggestion systems with an approach that transfers knowledge from two pretrained recurrent models and fine-tunes it with attention.

2.1. Source Code Modeling

  • Prior source-code modeling work spans n-gram and cache models, PCFGs, structured generation, decision trees, and graph-based API recommendation.The subsection also surveys API-use mining, completion tools, and deep-learning applications across code tasks.Source: §2.1 · p. 3
  • Earlier source-code suggestion approaches are described as problem-specific, whereas this work transfers pretrained knowledge and fine-tunes it for a related task without training from scratch.The cited comparison focuses on RNN-based code completion and source-code completion approaches.Source: §2.1 · pp. 3–4

2.2. Transfer Learning

  • The paper applies transfer learning to source-code modeling by combining knowledge from two pretrained recurrent models and fine-tuning it with an attention learner.The subsection places transfer learning in the context of work across vision, finance, image classification, and text classification.Source: §2.2 · p. 4

3. Preliminary

The preliminary section introduces technical background for the proposed approach. It discusses recurrent networks and attention as foundations for the later method.

3. Preliminary

  • The preliminary section provides background and a technical overview of the work.The following subsections cover recurrent neural networks and the attention learner.Source: §3 · p. 4

3.1. Recurrent Neural Network

  • A vanilla RNN’s vanishing-gradient drawback can be overcome with a GRU, which exposes the full hidden content at each timestep.Source: §3.1 · p. 4

3.2. Attention Learner

  • The attention model calculates a context vector as the weighted mean of the state sequence.Source: §3.2 · p. 5

4. Proposed approach

The approach combines pre-trained source-code models, downstream-task preparation, and an attention learner for source-code suggestion. It uses RNN and GRU variants for transfer learning, then applies their learned knowledge to the downstream task.

4. Proposed approach

  • The workflow prepares pre-trained models and source code for suggestion, then uses an attention learner to leverage and fine-tune the learned knowledge.The proposed approach is evaluated on the downstream task of source-code suggestion.Source: §4 · p. 5, §4.2 · p. 7

4.1. Transfer Learning

  • The authors train RNN and GRU variants for source-code transfer learning, using source code because its grammar makes image- and text-classification models unsuitable for this purpose.The collected projects comprise 13 million code tokens and a vocabulary of 177,342.Source: §4.1 · pp. 5–6

4.1.1. Pre-Training Models for Transfer Learning

  • The pre-training section includes an architecture summary for the deep learning models used to pre-train source-code models for transfer learning.Source: Table 2 · p. 7

4.2. Learning to Transfer Knowledge

  • For source-code suggestion, the pre-trained models' learned knowledge is kept unchanged while an attention learner fine-tunes task-specific features.The task recommends the next possible source-code token given a context.Source: §4.2 · p. 7

4.2.1. PreProcessing

  • The preprocessing overview identifies normalization, tokenization, and feature extraction as steps for preparing source code for suggestion.Table 3 is presented as an example of these steps.Source: §4.2.1 · p. 8, Table 3 · p. 8

Normalization

Normalization removes or generalizes source-code content considered unnecessary for the task.

Normalization

  • Normalization removes blank lines and comments, replaces numeric constants with generic type tokens, and replaces constant strings with StringVal.Source: Normalization · p. 8

Tokenization

Tokenization turns normalized source files into code-token sequences and partitions them into fixed-size contexts.

Tokenization

  • Each file is parsed into space-separated code tokens, then divided into subsequences with context size 20.Source: Tokenization · pp. 8–9

Vectorization

The vectorization steps map source-code tokens into vocabulary-index sequences suitable for deep-learning training.

Vectorization

  • Tokens appearing only once are replaced with the special token unk before a global vocabulary is built.Source: Vectorization · p. 9
  • Each unique source-code token receives a vocabulary entry, and each token is mapped to the positive integer for its vocabulary index.Source: Vectorization · p. 9
  • The proposed model architecture is described as a transfer-learning-based attention model.Source: Figure 3 · p. 9

5. Evaluation

The evaluation compares the proposed approach with state-of-the-art and baseline models for source code suggestion, examines normalization, and uses separate Java projects for testing. It describes the training and prediction setup and the metrics used, but the supplied passages do not report comparative results.

5. Evaluation

  • The evaluation asks whether the approach outperforms state-of-the-art and baseline models for source code suggestion and whether normalization improves performance.The three research questions cover state-of-the-art comparison, baseline comparison, and normalization.Source: §5 · p. 10

5.0.1. Data set

  • The evaluation uses five top-starred GitHub Java projects not used to train the pre-trained models, repeating experiments separately for each project.Each project is split into ten folds: one for testing, one for validation, and the rest for training; Table 5 lists the projects.Source: §5.0.1 · p. 10, Table 5 · p. 10

5.1. Training and Prediction

  • The approach is evaluated alongside RNN and GRU baselines, using 300-dimensional vocabulary vectors, 300 hidden units, context size 20, Adam, dropout, and early stopping.Training uses Adam with learning rate 0.001 and stops after three consecutive validation-loss hits.Source: §5.1 · p. 11

5.2. Metrics

  • For each prediction, the model uses preceding source-code context to suggest the next token; evaluation includes top-k accuracy, MRR, precision, recall, F-measure, and ANOVA testing.A prediction counts as successful when it matches the original next token.Source: §5.1 · p. 11, §5.2 · pp. 11–12

6. Results

The proposed approach outperforms the RNN and GRU baselines on source-code suggestion accuracy, F-measure, and MRR. Normalization improves accuracy, while the additional findings describe its parameter and latency profile, a qualitative suggestion example, and possible directions for improvement.

6.1. RQ1: Comparison against the baseline approaches

  • 68.36 average F-measure for the proposed approach exceeds RNN's 39.73 and GRU's 46.20.Source: §6.1 · p. 12, Table 6 · p. 13
  • 66.15%@k=1, 90.68%@k=5, and 93.97%@k=10 average accuracy for the proposed approach exceeds RNN (45.01%, 65.56%, 68.55%) and GRU (50.06%, 64.38%, 73.27%).The baseline values are reported in the same k=1, k=5, and k=10 order.Source: §6.1 · p. 12, Figure 4 · p. 14

6.2. RQ2: Comparative analysis for Source Code Suggestion Task

6.3. RQ3: Impact of Normalization

  • 66.15@k=1 accuracy with normalization compares with 56.27@k=1 without normalization, which removed comments from the source code.Source: Table 9 · p. 16, §6.3 · p. 16

6.4. Additional Findings

  • For a matrix-printing code context, the model ranked println first among its next-token suggestions.Source: §6.4 · p. 17
  • The proposed approach averages 58M parameters versus 9M for baselines, with minimal-to-no time overhead and next-token suggestions in under 20 milliseconds.Training is offline; the authors also report that LSTM performed worse than RNN and GRU and was not selected for transfer learning.Source: §6.4 · pp. 16–17, Figure 6 · p. 17
  • The authors attribute performance to transferred pre-trained features and an attention learner that focuses on task-specific features, supporting generalization without training from scratch.Source: §6.4 · p. 17
  • The authors identify hyperparameter optimization, more complex architectures, and larger training data as possible improvement directions.Source: §6.4 · p. 17

7. Threats to Validity

The study addresses construct-validity risk through several evaluation metrics and ANOVA testing, while noting possible baseline-implementation and hyperparameter risks. Its results may not generalize beyond the GitHub projects used, which may not fully represent other languages or Java source code.

  • To address metric-selection risk, the evaluation uses Top-k accuracy, precision, recall, F-measure, MRR, and ANOVA testing for statistical significance.Source: §7 · p. 18
  • Re-implemented baselines follow the original manuscripts and are checked twice, but unobserved inaccuracies may remain; training, validation, or testing set and hyperparameter changes may affect performance.Source: §7 · p. 18
  • The GitHub projects used may not represent other languages or Java source code entirely, limiting the generality of results.Source: §7 · p. 18

8. Conclusion

Extensive evaluation on source code suggestion suggests that transfer learning significantly improves performance over state-of-the-art models. The approach uses pre-trained RNN and GRU models, trained on over 13 million code tokens, directly for transfer without retraining.

  • Extensive evaluation suggests transfer learning significantly improves performance over state-of-the-art models on source code suggestion.Source: §8 · p. 18
  • The pre-trained RNN and GRU models are trained on over 13 million code tokens and can be used for transfer without retraining.Source: §8 · p. 18
  • The proposed source code language model applies transfer learning to the downstream task of source code suggestion.Source: §8 · p. 18
Loading 1910.05493v2…