PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Transferability of Pre-trained Language Models for Low-Resource Programming Languages

Fuxiang Chen, Fatemeh Fard, David Lo, Timofey Bryksin

arXiv:2204.09653v1cs.PLcs.CLcs.SE

TL;DR

The paper investigates how monolingual and multilingual PLMs transfer to Ruby for Code Summarization and Code Search, including how to select languages for fine-tuning. It reports that monolingual PLMs have better performance-to-time ratios and that the proposed selection strategies improve task performance.

  • Problem

    Prior work had not analyzed fine-tuning monolingual PLMs, and evidence on PLM transfer to low-resource Ruby remained limited despite differences between programming languages.

    Source: §1 · pp. 1–2

  • Method

    The authors evaluate over a hundred PLMs on Code Summarization and Code Search and propose selecting fine-tuning languages for multilingual PLMs based on similarity to Ruby.

    Source: §1 · p. 2

  • Results

    Monolingual PLMs have better Performance-to-Time Ratios, and the proposed language-selection strategies improve Code Summarization and Code Search performance over the combined multilingual dataset.

    Source: §1 · p. 2, §9 · p. 10

  • Takeaways & Limitations

    The authors suggest considering monolingual PLMs fine-tuned on combined multilingual data for greater efficiency with similar or better task performance.

    Source: §1 · p. 2

    The study covers six programming languages and two tasks, so its results might not generalize to other languages and tasks.

    Source: §8 · p. 10

Abstract

from arXiv · show

A recent study by Ahmed and Devanbu reported that using a corpus of code written in multilingual datasets to fine-tune multilingual Pre-trained Language Models (PLMs) achieves higher performance as opposed to using a corpus of code written in just one programming language. However, no analysis was made with respect to fine-tuning monolingual PLMs. Furthermore, some programming languages are inherently different and code written in one language usually cannot be interchanged with the others, i.e., Ruby and Java code possess very different structure. To better understand how monolingual and multilingual PLMs affect different programming languages, we investigate 1) the performance of PLMs on Ruby for two popular Software Engineering tasks: Code Summarization and Code Search, 2) the strategy (to select programming languages) that works well on fine-tuning multilingual PLMs for Ruby, and 3) the performance of the fine-tuned PLMs on Ruby given different code lengths. In this work, we analyze over a hundred of pre-trained and fine-tuned models. Our results show that 1) multilingual PLMs have a lower Performance-to-Time Ratio (the BLEU, METEOR, or MRR scores over the fine-tuning duration) as compared to monolingual PLMs, 2) our proposed strategy to select target programming languages to fine-tune multilingual PLMs is effective: it reduces the time to fine-tune yet achieves higher performance in Code Summarization and Code Search tasks, and 3) our proposed strategy consistently shows good performance on different code lengths.

1 INTRODUCTION

The study examines how monolingual and multilingual PLMs transfer to Ruby, a low-resource programming language, across Code Summarization and Code Search. It evaluates more than a hundred models and reports task-dependent results, including an effective strategy for selecting fine-tuning languages.

1 INTRODUCTION

  • The study addresses uncertainty about whether single-language or multilingual PLMs transfer better, especially because programming languages can differ structurally and datasets often underrepresent languages such as Ruby.Ruby is selected as a low-resource study subject; datasets for SE tasks are described as dominated by high-resource languages such as Java and Python.Source: §1 · pp. 1–2
  • The study investigates monolingual versus multilingual PLMs, zero-shot transfer, performance-to-time trade-offs, code-length effects, and selection of fine-tuning languages.The language-selection question concerns finding high-resource programming languages suitable for fine-tuning multilingual PLMs.Source: §1 · p. 2
  • The evaluation covers more than a hundred pre-trained and fine-tuned models on Code Summarization and Code Search using CodeSearchNet’s six programming languages.The study uses RoBERTa for pre-training and treats understanding transferability, rather than beating the state of the art, as its purpose.Source: §1 · p. 2
  • The proposed language-selection strategy improves performance over fine-tuning on the combined multilingual dataset, while results for that dataset differ between Code Summarization and Code Search.The combined multilingual dataset is not best for Code Summarization but is best for Code Search; the proposed strategy is reported to improve performance over it.Source: §1 · p. 2
  • The authors contribute a quantitative and qualitative evaluation, a language-selection strategy, and a comparison of PLMs trained on different programming languages for two SE tasks.They suggest that developers consider monolingual PLMs fine-tuned on combined multilingual datasets for greater efficiency with similar or better performance, depending on the task.Source: §1 · p. 2

2 RELATED WORK

Prior work introduced PLMs for code representation and evaluated them on varied code tasks, often using either one or several programming languages. The authors identify limited evidence comparing single-language and multilingual pre-training and probing PLM transfer across settings.

2 RELATED WORK

  • Existing code PLMs use varied architectures and training languages, including Python-focused CuBERT and JavaScript-focused SCELMO, as well as models trained on multiple languages.The cited work applies PLMs to tasks such as program repair, code generation, summarization, retrieval, and completion.Source: §2 · pp. 2–3
  • The review identifies limited evidence on whether single-language pre-training outperforms multilingual pre-training and on PLM zero-shot transfer across programming languages and settings.The authors position their study as an initial effort to address this gap.Source: §2 · p. 3
  • Empirical studies examine PLMs on code completion, summarization, comment generation, and program synthesis, while one study recommends time-segmented data for evaluating comment generation and method naming.Prior zero-shot Code Summarization work on C# was brief and was not its main focus.Source: §2 · p. 3

3 RESEARCH METHOD

The study frames five questions about PLM transfer, efficiency, zero-shot use, code length, and language selection. Its design compares monolingual and multilingual models across two tasks, with dedicated analyses for these questions.

3 RESEARCH METHOD

  • The method investigates PLM applicability and transferability in Software Engineering through a set of research questions and a corresponding study design.The questions cover performance, efficiency, zero-shot settings, code lengths, and selection of programming languages for fine-tuning.Source: §3 · p. 3

3.1 Research Questions

  • RQ1 compares individual-language training and fine-tuning with multilingual PLMs fine-tuned on multilingual datasets to test whether additional programming languages help.The multilingual PLMs named as baselines are CodeBERT and GraphCodeBERT.Source: §3.1 · p. 3

RQ2: Which PLM has the best Performance-to-Time Ratio

  • The remaining questions examine performance-to-time trade-offs, zero-shot transfer, code-length effects, and whether a selected language can work well for a low-resource target.The authors note that fine-tuning-language selection may need to account for task and programming-language differences.Source: RQ2: Which PLM has the best… · pp. 3–4

3.2 Study Design

  • The study uses CodeSearchNet’s six languages to compare monolingual PLMs and multilingual baselines on Code Summarization and Code Search, including human evaluation for summarization.CodeBERT and GraphCodeBERT serve as baselines fine-tuned on multilingual datasets.Source: §3.2 · p. 4
  • Performance-to-Time Ratio divides normalized task performance by normalized fine-tuning time; the study also compares zero-shot models, quartile-based code-length groups, and selected-language fine-tuning.Summarization performance uses BLEU and METEOR, while Code Search uses MRR; the longest training time and highest metric scores provide normalization denominators.Source: §3.2 · p. 4

4 SELECTING A PROGRAMMING LANGUAGE FOR FINE-TUNING A PLM

For Code Summarization, the proposed selection approach ranks candidate programming languages using semantic and textual similarity to the target language. For Code Search, the authors instead recommend fine-tuning on the combined multilingual dataset.

4.1 Code Summarization

  • For Code Summarization, the approach selects a fine-tuning language using both semantic similarity and textual similarity to the target programming language.Semantic similarity is based on normalized cosine similarity of code embeddings; textual similarity uses normalized code-clone counts detected by CCFinder, with a minimum of 30 tokens.Source: §4.1 · pp. 4–5

4.2 Code Search

  • PLMs fine-tuned on the combined multilingual dataset perform best in Code Search, so the authors propose using that dataset for the task.The passage reports this result with reference to Table 5.Source: §4.2 · p. 5

5 EXPERIMENTAL SETUP

The setup uses RoBERTa-based PLMs and CodeSearchNet for Ruby-focused code summarization and code search experiments, with quantitative metrics and a human-rated summary analysis.

5.1 The PLM and the Pretraining Dataset

  • RoBERTa is the common pre-training base, and CodeSearchNet supplies bimodal code–natural-language pairs and unimodal data across six programming languages.The corpus contains 2.1M bimodal data points and 6.4M unimodal codes.Source: §5.1 · p. 5

5.2 Downstream Tasks and Datasets

  • Code Summarization generates natural-language descriptions from code, while Code Search retrieves relevant code for a natural-language query.Summarization is fine-tuned on code-comment pairs; Code Search uses test sets pairing each correct example with 999 incorrect pairs.Source: §5.2 · p. 5

5.3 Evaluation Metrics

  • BLEU and METEOR evaluate generated summaries, while Code Search is evaluated using Mean Reciprocal Rank (MRR).BLEU measures n-gram precision, METEOR aligns words and phrases, and MRR averages reciprocal ranks of the first correct answers.Source: §5.3 · pp. 5–6

5.4 Qualitative Analysis for Code Summarization

  • MTurk workers rated randomly selected generated summaries for informativeness, relevance, fluency, and comprehension.Three workers rated each summary from one to five; only ratings from workers who understood the code were accepted.Source: §5.4 · p. 6

6 RESULTS

The results show task-dependent performance: combined multilingual fine-tuning performs best within PLMs, while monolingual PLMs outperform multilingual baselines in Code Search. Monolingual PLMs also have higher Performance-to-Time Ratios in both tasks.

6.1.1 Code Summarization.

  • Multilingual PLMs outperform monolingual PLMs in Code Summarization, while monolingual PLMs perform better in Code Search.Across PLMs, fine-tuning on the combined dataset gives the best Code Summarization performance.Source: §6.1.2 · p. 6, §6.1.1 · p. 6, Table 3 · p. 7

6.2.1 Code Summarization.

  • Monolingual PLMs have higher BLEU-4 and METEOR Performance-to-Time Ratios in Code Summarization than multilingual PLMs.The reported figure compares the two metrics’ PTRs.Source: §6.2.1 · p. 6, Figure 1 · p. 7

6.2.2 Code Search.

  • Monolingual PLMs have higher MRR Performance-to-Time Ratios in Code Search; the Ruby-trained monolingual PLM has the highest score among monolingual models.The PTR uses normalized MRR and fine-tuning time.Source: §6.2.2 · p. 6, Figure 2 · p. 7

6.1.2 Code Search.

  • 5%–35.1% in MRR over CodeBERT and 2.3%–32.6% over GraphCodeBERT were achieved by monolingual PLMs; combined multilingual fine-tuning performed best within each PLM.The reported comparisons are for Code Search.Source: §6.1.2 · p. 6, Table 5 · p. 7

6.3 RQ3: What are the best settings for zero-shot downstream tasks?

For zero-shot Ruby tasks, monolingual PLMs fine-tuned on Python perform best in both Code Summarization and Code Search. The Code Search results also show a statistically significant difference between experiments using datasets with and without Ruby.

6.3.1 Code Summarization.

  • Python fine-tuning gives the best zero-shot performance across the monolingual PLMs in Code Summarization.Table 6 reports BLEU and METEOR scores and shows only each PLM’s best-performing fine-tuned model.Source: §6.3.1 · p. 8, Table 6 · p. 8

6.3.2 Code Search.

  • Python fine-tuning also gives the best zero-shot MRR performance for Code Search.Table 7 reports the best-performing fine-tuned model for each monolingual PLM.Source: §6.3.2 · p. 8, Table 7 · p. 8
  • The comparison between datasets with and without Ruby yields p = 0.00604, which the authors report as statistically significant.Source: §6.3.2 · p. 8

6.4 RQ4: What effect does the PLMs have on different code length?

Across code lengths, the better fine-tuning dataset differs by task. Combined multilingual fine-tuning usually leads in Code Summarization and leads for every tested length in Code Search.

6.4.1 Code Summarization.

  • For Code Summarization, combined multilingual fine-tuning performs best for most PLMs in the shortest and longest code-length groups; Ruby or Python fine-tuning performs better for some PLMs at other lengths.Source: §6.4.1 · p. 8

6.4.2 Code Search.

  • For Code Search, combined multilingual fine-tuning performs best across all code lengths, with MRR varying by no more than ±0.05 across lengths.Most PLMs’ MRR scores vary by no more than ±0.03.Source: §6.4.2 · p. 8
  • Code Search performance is reported as similar when PLMs are tested across all code lengths.Source: §6.4.2 · p. 8

6.5 RQ5: How effective is our strategy to decide in advance a language that can work well for a target low-resource language?

The language-selection strategy improves Code Summarization and Code Search over the combined multilingual fine-tuning baseline. Its selected source languages and the reported gains differ by task.

6.5.1 Code Summarization.

  • Fine-tuning monolingual and multilingual PLMs on Python, PHP, Go, and Ruby improved BLEU-1/2/3/4 by 0.6%–4.3%, 1.1%–13.5%, 11.9%–30%, 3.8%–41.7%, and METEOR by 1.5%–5.6%.Compared with Ruby, suitability scores were Python 0.55, PHP 0.92, Java 0.14, Javascript 0.52, and Go 0.46.Source: §6.5.1 · p. 8

6.5.2 Code Search.

  • For Code Search, the strategy improves MRR by 2.3%–35.1%; monolingual PLMs also outperform CodeBERT and GraphCodeBERT by 5%–35.1% and 2.3%–32.6%, respectively.Source: §6.5.2 · pp. 8–9

7 DISCUSSIONS

The discussion recommends comparing monolingual and multilingual PLMs and choosing fine-tuning languages by task. It also notes that CodeSearchNet may contain non-exact code duplicates.

7 DISCUSSIONS

  • The efficiency and performance comparison depends on the task: monolingual PLMs are more efficient by PTR, while Code Search favors monolingual PLMs over multilingual ones.For Code Summarization, multilingual PLMs perform well on automatic metrics, but MTurkers find no major difference in generated summaries.Source: §7 · p. 9
  • The authors recommend testing language-selection strategies by downstream task, since a strategy that works for Code Summarization may not work as well for Code Search.Source: §7 · p. 9
  • CodeSearchNet may contain Type 2–4 clones, so this evaluation includes data with possible non-exact code duplicates.The authors state that excluding these duplicates may not necessarily yield more robust models.Source: §7 · p. 10

8 THREATS TO VALIDITY

The study reports threats involving limited task and language coverage, possible under-training from a fixed pre-training schedule, and evaluation measures.

External Validity

  • Results cover six programming languages and two tasks, so they might not generalize to other languages or tasks.The languages are Go, Java, JavaScript, PHP, Python, and Ruby; the tasks are Code Summarization and Code Search.Source: §8 · p. 10

Internal Validity

  • Pre-training every PLM for 50 epochs may leave some models under-trained, while optimal stopping criteria remain an open problem.The authors report no hard rule for the number of pre-training steps and use 50 epochs for consistency.Source: §8 · p. 10

Construct Validity

  • The authors identify evaluation measures as a construct-validity threat and use multiple commonly used downstream-task metrics to mitigate possible bias.The cited measures include BLEU, METEOR, and MRR.Source: §8 · p. 10

9 CONCLUSION AND FUTURE WORKS

The authors conclude that monolingual PLMs have a higher Performance-to-Time Ratio than multilingual PLMs and that their language-selection strategies are efficient. They recommend these approaches while identifying broader task coverage and strategy generalization as future work.

Conclusion

  • Monolingual PLMs have a better Performance-to-Time Ratio than multilingual PLMs, and the proposed language-selection strategies are efficient and can improve current state-of-the-art performance.The authors suggest considering a monolingual PLM fine-tuned on a combined multilingual dataset because of its higher Performance-to-Time Ratio.Source: §9 · p. 10
  • The authors recommend considering a monolingual PLM fine-tuned on a combined multilingual dataset and using their strategies for Code Summarization and Code Search.The recommendation is based on the reported Performance-to-Time Ratio and strategy results.Source: §9 · p. 10

Future Work

  • Future work will examine more downstream tasks and seek a strategy that generalizes across multiple tasks.The authors frame both broader task applicability and cross-task strategy generalization as future work.Source: §9 · p. 10

Related papers

Loading 2204.09653v1…