PaperlayerView exact revision on arXiv

Source-linked AI summary

Comparing the Pre-trained Models of Source Code by Re-pre-training under a Unified Setup

Changan Niu, Chuanyi Li, Vincent Ng, Bin Luo

arXiv:2610.15440v1cs.SE

TL;DR

The paper asks how to design more powerful CodePTMs with limited data and computation. It re-pre-trains models under a unified setup and compares their pre-training tasks; for bug fixing, fine-grained tasks are reported to contribute more than others. The comparison uses a single training budget, although models may surpass one another at different budgets.

  • Problem

    The paper asks how to design more powerful CodePTMs with limited data and computation.

    Source: §I · p. 2

  • Method

    The authors re-pre-train CodePTMs using their declared architectures and pre-training tasks under a unified experimental setup, then evaluate them on software engineering tasks.

    Source: §I · p. 2

  • Results

    For bug fixing, fine-grained generative and categorical pre-training tasks are described as having greater contribution than other tasks.

    Source: 2) Bug Fixing (BF) · p. 9

  • Takeaways & Limitations

    The authors hope their comparison will give subsequent researchers comparable insight into current CodePTM techniques to facilitate designing more powerful models.

    Source: §I · p. 2

    All models were pre-trained for a single budget of 500K steps, although models can surpass one another at different budgets.

    Source: Internal Validity · p. 9

Abstract

from arXiv · show

Recent years have seen the successful application of large pre-trained models of source code (CodePTMs) to code representation learning, which have taken the field of software engineering (SE) from task-specific solutions to task-agnostic generic models. By the remarkable results, CodePTMs are seen as a promising direction in both academia and industry. While a number of CodePTMs have been proposed, they are often not directly comparable because they differ in experimental setups such as pre-training dataset, model size, evaluation tasks and datasets, etc. In this paper, we first review the experimental setup used in previous work and propose a standardized setup to facilitate fair comparisons among CodePTMs to explore the impacts of their pre-training tasks. Then, under the standardized setup, we re-pre-train CodePTMs using the same model architecture, input modalities, and pretraining tasks, etc., as they declared and fine-tune each model on each evaluation SE task for evaluating. Finally, we present the experimental results and make a comprehensive discussion on the relative strength and weakness of different pre-training tasks w.r.t. each SE task. We hope our view can inspire and advance the future study of more powerful CodePTMs.

I. INTRODUCTION

CodePTMs differ in pre-training tasks and experimental setups, making their relative strengths difficult to compare. The paper re-pre-trains models under unified conditions to examine pre-training-task impacts across software engineering tasks.

I. INTRODUCTION

  • Existing CodePTMs vary in datasets, model sizes, and evaluation tasks, so their relative strengths and weaknesses remain unclear.Although CodeXGLUE spans 14 datasets across 10 tasks, no CodePTM had been evaluated on all of them.Source: §I · pp. 1–2
  • Differences in vocabulary size, runtime settings, and input and output length limits are additional sources of uncertainty in comparisons.Runtime settings include batch size, iteration count, and early-stopping patience.Source: §I · p. 2
  • The paper asks how to design more powerful CodePTMs with limited data and computation.It also aims to provide a comprehensive comparison of recent CodePTMs for AI researchers.Source: §I · p. 2
  • The paper compares current code pre-training techniques by re-pre-training models under a unified experimental setup.The setup covers the pre-training dataset, model size, downstream tasks and datasets, and evaluation metrics; models retain their declared architectures and pre-training tasks.Source: §I · p. 2

II. EVALUATION SE TASKS, DATASETS AND METRICS

The paper selects representative, commonly used software engineering tasks and datasets, then classifies tasks by purpose and input-output types and identifies their evaluation metrics.

II. EVALUATION SE TASKS, DATASETS AND METRICS

  • The evaluation covers 10 of 18 previously identified software engineering tasks, chosen to span task types and use representative tasks and widely adopted datasets.The selection avoids tasks of the same type and tasks used to evaluate only one CodePTM.Source: §II · p. 2
  • Tasks are classified as understanding or generation and by input-output types, with C, NL, and V denoting code, natural language, and extracted or predicted values.Table I also reports task definitions, datasets, data splits, and primary metrics.Source: §II · p. 2
  • Evaluation uses standard retrieval and classification metrics and generation metrics including BLEU, ROUGE-L, CodeBLEU [33], and exact match.Retrieval and classification metrics include accuracy, F1, precision, recall, MRR, and MAP.Source: §II · p. 3

III. SELECTED CODEPTMS FOR RE-PRE-TRAINING

The selected CodePTMs are organized by architecture, input modality, and pre-training task. The section also cautions that existing performance results cannot isolate the effects of pre-training tasks because models differ in other settings.

A. Architecture

  • The comparison selects Transformer-based models and excludes LSTM-based models because their sizes cannot be controlled to match the Transformer models.Transformer encoder and decoder models are matched in size, while Transformer encoder-decoder models are twice the size; comparisons are therefore restricted to like-sized architecture groups.Source: A. Architecture · p. 3

B. Modality

  • The selected models cover code, natural-language, and structure inputs, including unimodal, bimodal, and multimodal combinations.OSCAR is excluded because it uses an intermediate representation of source code as its code modality.Source: B. Modality · p. 4

C. Pre-Training Tasks

  • Pre-training tasks are grouped as natural-language-aware or code-aware, code-aware-only, structure-aware-only, or cross-modal-aware tasks.Cross-modal tasks are divided into code-natural-language and code-structure categories; tasks are also classified as generative or categorical.Source: C. Pre-Training Tasks · p. 4

D. Performance Discussion

  • Existing CodePTMs differ in architecture, input modality, pre-training tasks, and settings such as datasets and model sizes, preventing attribution of performance differences to pre-training tasks alone.The section therefore discusses findings from tasks to which more than one CodePTM has been applied.Source: D. Performance Discussion · p. 4

1) Defect Detection:

Observed associations between pre-training choices and performance vary by software engineering task: structure-aware tasks do not always help, while natural-language inputs and fine-grained tasks are useful in particular settings.

1) Defect Detection:

  • Cross-modal-aware pre-training does not appear to improve defect-detection performance, and the effect of code structure remains uncertain.Source: 1) Defect Detection · p. 4

2) Clone Detection:

  • For clone detection, NL/code-aware tasks can match the state of the art, while structure-aware tasks can bring TE models close to it.The passage relates this pattern to prior work arguing that token sequences alone are insufficient for modeling program structure.Source: 2) Clone Detection · p. 4

3) Code Translation:

  • For code translation, adding code structure and structure-aware pre-training still does not achieve state-of-the-art performance.Source: 3) Code Translation · p. 4

4) Bug Fixing:

  • For bug fixing, natural-language input helps, and combining it with code-aware-only and code-natural-language-aware pre-training can achieve state-of-the-art performance.The passage says structural information and structure-related pre-training did not improve performance.Source: 4) Bug Fixing · p. 4

5) Code Summarization:

  • For code summarization, the state of the art uses natural-language input and code-natural-language-aware pre-training, while method-name-based tasks appear less useful than document-based tasks.The passage notes that the performance gap to the state of the art is greater for models using natural-language information than for those without it, and says differing setups may warrant exploration.Source: 5) Code Summarization · p. 4

6) Code Generation:

The paper compares code-generation performance across CodePTMs but notes that the state-of-the-art result cannot be attributed specifically to natural-language input or cross-modal pre-training. It prepares models from available implementations and uses CodeSearchNet's training split as the unified pre-training dataset.

6) Code Generation:

  • The code-generation state-of-the-art model combines natural-language input and Code-NL-aware, IT, and SIMLM pre-training, so their individual contributions remain unresolved.The paper says further experiments are needed to determine whether the improvement comes from natural-language input or cross-modal pre-training.Source: 6) Code Generation · p. 4

1) Model Preparation:

  • Models were prepared through direct use of public code, re-implementation of pre-training, or re-implementation from papers when fully correct public code was unavailable.These categories cover models with complete code, models with only checkpoints and fine-tuning code, and models lacking fully correct public code.Source: 1) Model Preparation · p. 5

2) Unified Pre-training Dataset:

  • To prevent validation- and test-set leakage for downstream tasks that also use CodeSearchNet, the authors use only its training set as pre-training data.CodeSearchNet includes more than 6.4M codes from six programming languages, including more than 2.1M with paired documentation; because “Post” is unavailable, PLBART uses “Docs” as natural-language information in pre-training data.Source: 2) Unified Pre-training Dataset · p. 5

3) Unified Model Size:

The study standardizes model scale and fine-tuning procedures while retaining task-appropriate input and output handling for different model architectures. Re-implemented models were not significantly different from the original reported results under the tested settings.

3) Unified Model Size:

  • The unified model uses a base Transformer with 512-dimensional model states, 2,048-dimensional feed-forward layers, six layers, eight attention heads, and a 512-token length limit.The vocabulary target is 50,000±1%, allowing for tokenizer merge variation.Source: 3) Unified Model Size · p. 5
  • Re-implemented models' results were not significantly different from the original reports under the original pre-training and fine-tuning settings.This check was used to verify the re-implementations.Source: 3) Unified Model Size · p. 5

B. Fine-tuning Settings

  • For understanding, TE-based models use designated classification-position states, TD-based models use the last hidden state, and TF-based models use architecture-specific representations or text-to-text labels.For generation, TE models receive a randomly initialized decoder, TD models remain autoregressive, and TF models are used without modification.Source: B. Fine-tuning Settings · p. 5
  • Each model-task pair is fine-tuned with three random seeds, validated each epoch, and early-stopped after 20 consecutive validations without a new best score.The epoch limit is 504; hyperparameters are selected by grid search over learning rate, warm-up steps, and batch size.Source: B. Fine-tuning Settings · pp. 5–6

C. Other Settings

The paper reports that TE- and TD-based models pre-train fastest, while TF-based models are slowest, and relates speed differences to architecture, input length, objectives, and input processing. Pre-training follows the original or public-code settings where available, with defaults otherwise.

C. Other Settings

  • Pre-training uses 500K steps and follows original or public-code hyperparameters where available, defaulting to AdamW, a 5e-5 learning rate, 2,000 warm-up steps, and batch size 256 otherwise.Experiments run on four NVIDIA Tesla V100 GPUs.Source: C. Other Settings · p. 6

V. RE-PRE-TRAINING SPEED OF CODEPTMS

  • TE- and TD-based models have the highest pre-training speed, while TF-based models have the lowest.A six-layer TF model makes a 12-layer pass and is theoretically more than twice as slow as encoder-only or decoder-only models.Source: §V · p. 6, Table V · p. 6
  • Input length affects speed even with unified length limits: code-and-documentation inputs make CoTexT slower than code-only DeepDebug, and three-modality SPT-Code is slowest.The models use padding where inputs are shorter, but modalities result in different input sequence lengths.Source: §V · p. 6
  • Objectives that generate longer output sequences take more time; PLBART's denoising objective regenerates the full input, unlike SMLM and SNP's subsequence generation.The cited comparison concerns PLBART, T5-learning, and ProphetNet-Code, all of which are TF-based.Source: §V · p. 6
  • TreeBERT is slower than PLBART despite shorter AST Paths because forming those paths requires node embeddings, concatenations, and other processing.Both models are TF-based and generate the entire code sequence.Source: §V · p. 6

VI. PERFORMANCE OF RE-PRE-TRAINED CODEPTMS

Under unified evaluation settings, the paper finds that the usefulness of pre-training tasks varies by downstream task. Its task-level synthesis identifies patterns across structure-aware, MLM-style, code-NL-aware, and FLM objectives.

VI. PERFORMANCE OF RE-PRE-TRAINED CODEPTMS

  • On Bug Fixing, T5-learning’s exact matches on small/medium datasets were 10/3 in the original paper and 16.58/7.30 in our results, suggesting extra pre-training data in a different programming language can improve downstream performance.Source: §VI · p. 7
  • The task comparisons use unified pre-training and evaluation datasets and model size, with results checked as significantly different across models for each SE task.The authors use Approximate Randomization tests with P-values below 0.05.Source: §VI · p. 7
  • The paper summarizes the most frequent pre-training-task combinations among each downstream task's top five models.The summary groups task combinations as code-aware only, structure-aware only, code-NL-aware, and code-structure-aware.Source: §VI · p. 7, Table VIII · p. 7
  • Structure-aware tasks benefit most downstream tasks except translation, completion, and generation; MLM variants suit most tasks except completion, code-NL-aware tasks suit generation except completion, and FLM suits completion.These are the paper's task-level conclusions from the best-performing models.Source: §VI · p. 7, Table VIII · p. 7

A. SE Task Analysis

Task difficulty varies across the evaluated SE tasks, while pre-training-task strengths differ between defect detection and clone detection. The analysis also cautions that some datasets may not distinguish models well.

A. SE Task Analysis

  • All models score highly on BigCloneBench, while Code Search is harder on filtered AdvTest than on CodeSearchNet, where reported scores exceed 70.Defect detection and Code Question Answering remain moderately difficult; the authors suggest a more difficult clone-detection dataset is needed to distinguish models.Source: A. SE Task Analysis · p. 7
  • Bug Fixing has low exact match (EM) on both subsets, but medium scores exceed small on BLEU and CodeBLEU because longer code allows more n-gram matches.EM is lower on medium because that subset contains longer code than small.Source: A. SE Task Analysis · p. 7
  • Code Summarization and Generation are challenging, and summarization models score differently across languages, which the authors attribute to dataset difficulty.Source: A. SE Task Analysis · pp. 7–8

1) Defect Detection:

  • FLM performs strongly on Defect Detection: CugLM outperforms TE models using MLM alone or MLM combined with other tasks, and CodeGPT-adapted scores above all TE models.The analysis also reports good results from code-only input for TE- and TD-based models.Source: 1) Defect Detection · p. 8
  • TreeBERT is the best-performing model on Defect Detection overall, and its TMLM and NOP tasks learn code-related structural features.The authors interpret this as evidence that code-specific characteristics matter and that TF can reach a level similar to TE and TD on understanding tasks.Source: 1) Defect Detection · p. 8

2) Clone Detection:

  • MLM-based tasks are associated with strong Clone Detection results: CugLM and C-BERT lead among TE models, while MLM+DOBF, TreeBERT, and PLBART lead among TF models.Source: 2) Clone Detection · p. 8

3) Code-to-Code Retrieval:

The analysis finds that code-specific structural features are useful for Code-to-Code Retrieval, while Code Search and Question Answering favor models trained on code and natural language. For Code Translation, generative-task effectiveness depends on task complexity and architecture.

3) Code-to-Code Retrieval:

  • Code-to-Code Retrieval favors code-specific features: GraphCodeBERT leads using DFG structure, and TreeBERT uses AST structure for the best TF-based result.The task requires distinguishing snippets that differ in form but are functionally identical; the authors say NLP-borrowed tasks alone do not suffice for strong results.Source: 3) Code-to-Code Retrieval · p. 8

4) Code Search and Question Answering:

  • Models trained on both code and natural language generally score relatively higher on Code Search and Question Answering, which require code-documentation alignment.Source: 4) Code Search and Question Answering · p. 8
  • GraphCodeBERT significantly improves on CodeBERT in Code Search and Question Answering, a comparison the authors attribute to code-specific pre-training; TreeBERT and SPT-Code also support this observation.Source: 4) Code Search and Question Answering · p. 8

1) Code Translation (CT):

  • Both TE and TF models using generative tasks can reach SOTA on certain Code Translation metrics, but TF requires more complex generative tasks for such results.With other settings held the same, T5-learning's SMLM does not surpass Prophet-net or PLBART to reach SOTA on any metric; SMLM masks independent tokens, unlike DAE or FNM, which generate consecutive token sequences.Source: 1) Code Translation (CT) · p. 8
  • For TE models, adding categorical NSP to C-BERT produces significant drops on four of six Code Translation results, while adding structural EP and NA improves three metrics, two to SOTA.Source: 1) Code Translation (CT) · p. 8
  • For Code Translation, generative pre-training is more effective with architectures suited to generation, such as TD and TF, than with TE.Source: 1) Code Translation (CT) · p. 8

2) Bug Fixing (BF):

Bug Fixing favors fine-grained generative and categorical pre-training tasks, whereas Code Completion results favor FLM and FNP. Code Summarization and Generation show task- and setting-specific differences in the value of natural language and structural information.

2) Bug Fixing (BF):

  • Bug Fixing benefits more from fine-grained generative tasks than coarse-grained ones, and token- or node-level categorical tasks contribute positively, with GraphCodeBERT achieving SOTA on three metrics.The authors link this pattern to bug fixing's localization and repair stages and to bugs generally occurring at token level; TE structure-aware tasks also appear more powerful here than in Code Translation.Source: 2) Bug Fixing (BF) · pp. 8–9

3) Code Completion (CC):

  • Code Completion SOTA results come from FLM and FNP, but the authors attribute ProphetNet-code's lag behind CodeGPT-adapted to natural-language input and leave code-only TF with FNP as an open question.Source: 3) Code Completion (CC) · p. 9

4) Code Summarization (SM):

  • For Code Summarization, code-only pre-training can outperform code-plus-natural-language settings, and CodeGPT-adapted exceeds TE SOTA models on five of six datasets.MASS based on method names can outperform document-based generative tasks in the compared settings; structural-information tasks show no significant contribution.Source: 4) Code Summarization (SM) · p. 9

5) Code Generation (CG):

  • CugLM reaches TE SOTA on Code Generation without natural-language pre-training or input, while CodeT5 leads under TF and CoTexT is second-best.The authors say NL is useful for Code Generation; whether TF with FNP trained on code and NL in the same instance would outperform CoTexT remains for further study.Source: 5) Code Generation (CG) · p. 9

VII. THREATS TO VALIDITY

The threats section bounds the comparison through implementation, hyperparameter, training-budget, and benchmark-scope qualifications. The conclusion presents the work as a unified re-pre-training comparison intended to inform subsequent CodePTM research.

VII. THREATS TO VALIDITY

  • Different code implementations may vary in efficiency and effectiveness, although re-implemented and original models showed no significant difference under the reported t-test (p < 005).Source: Construct Validity · p. 9
  • Original or default hyperparameter settings may not be optimal or affect models equally, and base-scale standardization may yield results below those of the original models.Source: Internal Validity · p. 9
  • All models were pre-trained for 500K steps, although models can surpass one another at different training budgets.Source: Internal Validity · p. 9
  • Conclusions are guaranteed only for the selected tasks and datasets, whose capabilities and data distributions may differ from others.Source: External Validity · p. 9

VIII. CONCLUSION

  • The conclusion presents a comprehensive, fair comparison of pre-training tasks under a unified setting and offers analysis of their relative strengths and weaknesses.Source: §VIII · pp. 9–10

Related papers

Loading 2610.15440v1…