Source-linked AI summary
Exploring Distributional Shifts in Large Language Models for Code Analysis
Shushan Arakelyan, Rocktim Jyoti Das, Yi Mao, Xiang Ren
TL;DR
Code-capable language models face an underexplored distribution-shift problem across software organizations, projects, and components. The paper evaluates adaptation methods, including retrieving labeled examples from other domains, and finds that these examples help address out-of-domain performance drops, while simultaneous multi-domain adaptation has different tradeoffs across tasks.
Problem
How CodeT5, Codex, and ChatGPT generalize across software-domain shifts remains underexplored, including shifts between organizations, projects, and components.
Source: §1 · p. 1Method
The paper evaluates three code-capable models on generation and summarization across organization, project, and component domains, testing adaptation with retrieved out-of-domain examples.
Source: §1 · pp. 1–2, §5 · p. 8Results
Takeaways & Limitations
Retrieving similar labeled training examples is the most effective approach for adapting to new domains when in-domain data is absent.
Source: §5 · p. 8Filtering skews the dataset toward larger projects and removes samples that could come from smaller projects.
Source: §6 · p. 9
Abstract
from arXiv · showhide
We systematically study how three large language models with code capabilities - CodeT5, Codex, and ChatGPT - generalize to out-of-domain data. We consider two fundamental applications - code summarization, and code generation. We split data into domains following its natural boundaries - by an organization, by a project, and by a module within the software project. We establish that samples from each new domain present all the models with a significant challenge of distribution shift. We study how established methods adapt models to better generalize to new domains. Our experiments show that while multitask learning alone is a reasonable baseline, combining it with few-shot finetuning on examples retrieved from training data can achieve very strong performance. Moreover, this solution can outperform direct finetuning for very low-data scenarios. Finally, we consider variations of this approach to create a more broadly applicable method to adapt to multiple domains at once. We find that for code generation, a model adapted to multiple domains simultaneously performs on par with those adapted to a single domain
1 Introduction
The paper examines how code-capable language models generalize across software domains, a distribution-shift challenge not systematically studied across the software hierarchy. It finds out-of-domain performance drops and evaluates adaptation strategies for new domains and multiple domains at once.
1 Introduction
- The study probes Codex, CodeT5, and ChatGPT on code generation and summarization across organization, project, and component-level domains.These domain boundaries reflect the hierarchical organization of software systems.Source: §1 · p. 1, Figure 1 · p. 1
- Distribution shifts across software’s hierarchical structure remain understudied for code-capable large language models.Prior work addressed other shifts, including older-to-newer code, projects, authors, and code representations.Source: §1 · p. 1
- All three models lose out-of-domain performance; for code summarization, few in-domain examples improve performance by over 10 BLEU score points on average.The study compares out-of-domain performance with adaptation using limited data from random or test domains.Source: §1 · pp. 1–2
- Retrieved labeled out-of-domain examples perform on par with or better than adaptation using 8 or 16 in-domain examples on CodeSearchNet.The authors identify retrieved-example adaptation as useful for low-data domains, where labeled in-domain examples may be scarce.Source: §1 · p. 2
- For multiple-domain adaptation, CodeT5 has a minor summarization drop and negligible generation drop, making simultaneous adaptation feasible with minimal tradeoff.This avoids storing separate model copies for each domain.Source: §1 · p. 2
2 Background
Prior work on code analysis has examined cross-project shifts, while shifts across software’s hierarchical domains remain underexplored for large language models. The background introduces transfer and adaptation approaches used in the study.
2 Background
- Earlier code-analysis studies primarily examined cross-project shifts using traditional machine-learning methods, while recent work considers author, timeline, and token-distribution shifts.The paper identifies large language models’ capabilities under distribution shift as still underexplored.Source: §2 · p. 2
- The study tests MAML for gradient-based meta-learning and jointly fine-tunes a shared language model across tasks for multi-task learning.Both approaches aim to improve initialization for few-shot downstream performance.Source: §2 · p. 2
- Low-Rank Adaptation updates a low-rank parameterization as a parameter-efficient alternative to fine-tuning all model parameters.The background notes that parameter-efficient methods can obtain comparable performance while tuning a small fraction of parameters.Source: §2 · p. 2
- The study tests in-context learning on Codex and similarity-based retrieval to select informative examples for domain adaptation.In-context learning supplies task descriptions and examples without updating model weights.Source: §2 · p. 3
3 Problem setting
The paper defines out-of-domain code analysis around unseen organizations, projects, and project components, then builds evaluation domains from CodeSearchNet’s repository paths. It evaluates code summarization and generation with three language models.
3 Problem setting
- The study treats code from an unseen organization, project, or project component as a new-domain setting for model evaluation.Functions are grouped by organization, repository, or lowest-level folder, with each same-domain set denoted τ_i.Source: §3 · p. 3, Figure 2 · p. 3
3.1 Data
- CodeSearchNet’s JavaScript data is split into training and evaluation sets, with evaluation domains overlapping training domains removed to preserve out-of-domain status.Domains with fewer than 96 total samples are filtered; development data provides a proxy for performance if a domain had been seen during training.Source: §3.1 · p. 3
3.2 Applications and Metrics
- Code summarization maps a function to its docstring and is evaluated with BLEU-4, while code generation produces a function from a natural-language description and is evaluated with CodeBLEU.The authors note that Codex’s undisclosed training data means they cannot establish whether it included CodeSearchNet.Source: §3.2 · pp. 3–4
3.3 Models
- CodeT5, Codex, and ChatGPT are evaluated, ranging from a 700-million-parameter encoder-decoder to a GPT-3-based decoder with over 100 billion parameters.ChatGPT is described as a chat-optimized version of InstructGPT fine-tuned with reinforcement learning from human feedback.Source: §3.3 · p. 4
4 Analysis
The analysis tests how code models generalize to new domains and compares adaptation strategies, including retrieved supervision and multi-domain training. Models struggle out of domain; retrieved examples can help in low-data settings, while combining domains has different effects for summarization and generation.
4 Analysis
- The analysis compares few-shot in-domain adaptation with out-of-domain-only models; CodeT5 updates weights, whereas Codex receives in-domain examples as prompt demonstrations.CodeT5 experiments use 8, 16, or 32 supervision examples and full fine-tuning or LoRA.Source: §4 · pp. 4–5
Finding: Models struggle on new domains
- CodeT5 summarization gains about 200% relative improvement from few-shot in-domain data, while Codex gains up to 50% on generation; CodeT5’s generation shift is nearly negligible.The authors hypothesize that distribution shift is less noticeable on the more challenging generation task.Source: Finding: Models struggle on new domains · p. 5
- ChatGPT outperforms Codex at zero-shot, but its cautious refusals on vague or noisy inputs produce zero scores and dampen apparent gains from demonstrations.The authors do not compare ChatGPT further in the paper.Source: Finding: Models struggle on new domains · p. 5
- CodeT5 methods include multitask and meta-learning training, plus retrieving similar training examples for supervision; cosine similarity performs better overall than IsoScore.The retrieved set contains the k most similar examples for each test example, with k set to 4, 8, or 32.Source: Finding: Models struggle on new domains · p. 6
Codex and ChatGPT
- For Codex and ChatGPT, adaptation uses in-context learning; Codex retrieval demonstrations are the query’s 4 or 8 most similar training samples.Demonstration order is shuffled five times and results averaged; the prompt limit permits up to eight demonstrations.Source: Codex and ChatGPT · p. 5, Codex · p. 6
Finding: Strategic adaptation is advantageous in very low data scenarios
- Performance gains saturate as stratified supervision or demonstration examples increase: CodeT5 with 32 examples per test instance is almost always worse than with 4 or 8; Codex performs approximately the same with 4 or 8.Source: Finding: Strategic adaptation is… · p. 7
- For code summarization, retrieving 4 or 8 out-of-domain examples can match or beat adaptation with 8 in-domain examples; CodeT5 generation can exceed adaptation with 32 in-domain examples.Retrieval gains plateau, while in-domain supervision does not show the same trend.Source: Finding: Strategic adaptation is… · p. 7
Finding: Multi-domain code generation models do not require a large performance sacrifice.
- A single model trained on combined retrieved samples has a moderate drop for code summarization and a negligible drop for code generation; generation shows little negative transfer across domains.The paper describes this as a viable solution for code generation.Source: Table 5 · p. 8, Finding: Multi-domain code generation… · p. 8
5 Conclusion
The paper evaluates CodeT5, Codex, and ChatGPT on code generation and summarization across organization-, project-, and module-level shifts. It finds that retrieved training examples are the strongest adaptation approach without in-domain data, while multi-domain adaptation works well for generation but trades off summarization performance.
5 Conclusion
- CodeT5, Codex, and ChatGPT are evaluated on code generation and summarization across organization-, project-, and module-level domain shifts.All evaluated models are susceptible to reduced performance under domain shifts.Source: §5 · p. 8
- Retrieved out-of-domain training examples are the most effective adaptation approach without in-domain data, while multi-domain adaptation works very well for generation but trades off summarization generality.The paper also reports that multi-domain-adapted models perform very well for code generation.Source: §5 · p. 8
6 Limitations and Threats to Validity
Filtering makes the dataset more reliable in the authors’ view, but skews it toward larger projects and excludes many potential samples from smaller ones.
6 Limitations and Threats to Validity
- Filtering skews the dataset toward larger projects and removes many potential samples from smaller projects.The authors consider filtering necessary because very small test sets can have high variance, and ask readers to interpret findings with this selection effect in mind.Source: §6 · p. 9
8 Appendix
The appendix documents methods and supplementary materials, including few-shot learning approaches and a visualization of domain similarity. The visualization indicates that most domains are internally uniform yet dissimilar to other domains.
8.1 Javascript Keywords
- The appendix notes that JavaScript keywords used in the CodeBLEU implementation are listed in a table, and a supplementary table compares in-domain and random out-of-domain code-generation ChrF.The table caption identifies ChrF as higher-is-better; no comparative values are supplied here.Source: §8.1 · p. 12, Table 6 · p. 13, Table 7 · p. 13
8.2.1 Meta-learning and Multi-task-learning
- MAML learns an initialization intended for fast fine-tuning with few examples, while multitask learning jointly learns related tasks and shares model parameters.Both methods aim to capture a prior from training tasks for future target tasks; the authors experiment with both to compare few-shot performance in this setting.Source: §8.2.1 · p. 12
8.2.2 Few-shot Methods
- The appendix describes parameter-efficient methods such as adapters, LoRA, prompt tuning, and prefix-tuning, alongside retrieval-based example selection methods from prior work.The cited retrieval approaches select examples using encoded similarity or task-specific supervision.Source: §8.2.2 · p. 12
8.3 Domain split visualization
- Figure 6 plots each domain’s average within-domain example similarity against its average similarity to all other domains.Its x-axis reflects within-domain uniformity, and its y-axis reflects similarity to examples in other domains.Source: §8.3 · p. 12, Figure 6 · p. 13
- Most domains cluster as internally uniform but dissimilar to other domains; the less common upper-left cases include more folders than repositories, and more repositories than organizations.The authors hypothesize that functional similarities across domains may explain these cases, such as utility folders whose individual functions differ but resemble functions in other utility folders.Source: §8.3 · p. 13
CodeT5:
The appendix describes model settings, supplementary experiments, retrieval variants, and sample outputs. It notes that retrieved examples can be near duplicates of test examples, while in-domain fine-tuning can use relevant member names.
CodeT5:
- CodeT5-large is used without architecture changes, Codex is evaluated on both tasks without training, and ChatGPT is accessed through the GPT-3.5 Turbo API.The appendix also identifies their model backgrounds and notes ChatGPT’s 4096-token API limit.Source: CodeT5 · pp. 13–14
8.5 Hyperparameters and training details
- Full CodeT5 fine-tuning uses 500 steps, LoRA uses rank 4 and 1000 steps, and multitask learning uses 150K steps; MAML uses 10K steps.Batch sizes and checkpoint-selection procedures vary across methods.Source: §8.5 · p. 14
8.6 The Vault
- The Vault data are processed to align them with CodeSearchNet, addressing differences in function comments and docstring parameter documentation.CodeSearchNet functions may contain inlined comments and shorter docstrings, whereas Vault functions are stripped of comments and often include parameter documentation.Source: §8.6 · pp. 14–15
8.7 Additional experimental results
- Supplementary code-generation results report chrF, RougeL, and CodeBERTScore, and Figure 7 compares CodeT5 LoRA with regular fine-tuning.The supplied passages identify the metrics and comparison but do not provide their result values.Source: Table 8 · p. 14, Table 9 · p. 14, §8.7 · p. 15
8.8 IsoScore
- IsoScore retrieval selects training examples that maximize the IsoScore of the combined test-example and training-example embedding set.The method tests 4×32, 8×32, or 32×32 retrieved examples, matching the counts used with cosine similarity.Source: §8.8 · p. 15, Table 10 · p. 15
8.9 Fast vote-k
- Fast vote-k selects 128, 256, or 1024 supervision examples for further fine-tuning of a CodeT5 multitask model.The appendix reports these results in Table 11.Source: §8.9 · p. 15, Table 11 · p. 15
8.10 Instructions for Codex and ChatGPT
- Table 12 contains the instructions and demonstration templates used with Codex and ChatGPT in instruction-only and in-context-learning experiments.The table caption identifies the materials as task instructions and demonstration templates.Source: §8.10 · p. 15, Table 12 · p. 16
8.11 Sample outputs
- Retrieved out-of-domain examples can be near duplicates of test examples, producing model outputs extremely close to gold; in-domain fine-tuning can also use relevant member names.These observations are illustrated through sample outputs from different models.Source: §8.11 · p. 15, Table 13 · p. 17