CodeRetriever: Unimodal and Bimodal Contrastive Learning for Code Search

1 Introduction

• Token-based pre-training is described as producing poor code semantics because frequent-token embeddings dominate representation space and mixed-language code is difficult to unify.The paper identifies anisotropy and cross-language representation as the two issues.Source: §1 · pp. 1–2
• CodeRetriever learns function-level representations with text and code encoders, measuring code-text or code-code relevance through dense-vector similarity.Its training minimizes unimodal and bimodal contrastive losses: one draws functionally similar code closer, while the other models code-text relevance.Source: §1 · p. 2
• The method builds positive code-code pairs from function names and documentation, and uses documentation and in-line comments as text-code pairs.In-line comments can convey code-internal logic, such as swapping descending adjacent elements to sort an array in ascending order.Source: §1 · p. 2
• CodeRetriever achieves new state-of-the-art performance on eleven code-search datasets spanning six languages and function-, snippet-, and statement-level code.The paper also says the approach can alleviate anisotropy and cross-language representation issues.Source: §1 · p. 2

2 Preliminary: Code Search

• CodeSearchNet contains 2.1M function-document pairs and 6.4M unpaired functions from open-source, non-fork GitHub repositories.The paper describes it as the largest publicly available code dataset.Source: §2 · p. 2
• Earlier code-search work uses paired code-document data, while rich unlabeled code is often abandoned or used only for token-level pre-training.The paper argues token-level objectives do not explicitly learn function-level representations and are sub-optimal for code search.Source: §2 · p. 2
• CodeRetriever uses code-document and code-comment pairs for bimodal learning and code-code pairs for unimodal learning of function-level semantics.The paper describes the model as serving downstream domain- or language-specific datasets after pre-training.Source: §2 · p. 2, Figure 2 · p. 3

3 Approach

• Siamese code and text encoders represent inputs as dense vectors, with semantic similarity calculated for code–code and text–code pairs.The paper defines cosine similarity as the similarity operation.Source: §3 · p. 3

3.1 Unimodal Contrastive Learning

• Unimodal contrastive learning compares a paired code sample with in-batch code negatives, including batches that can mix programming languages.The set of candidates contains the paired code and N−1 unpaired samples; the temperature is set to 1.Source: §3.1 · p. 3

3.2 Bimodal Contrastive Learning

• Bimodal contrastive learning trains on paired text and code, using documentation or comments to convey code semantics and align programming-language representations with natural language.The text–code batch may contain codes from hybrid programming languages.Source: §3.2 · p. 3

3.3 Overall Pre-training Objective

• The overall pre-training objective combines unimodal code–code loss with two bimodal contrastive losses.The two bimodal losses correspond to code–document and code–comment training.Source: §3.3 · p. 3

4.1 Code-Document

• Code and their corresponding documentation are positive pairs, with natural-language descriptions serving as a pivot across programming languages.The authors use documentation because it describes code functionality and provides semantic information.Source: §4.1 · p. 3

4.2 Code-Comment

• In-line comments provide fine-grained information about code logic; the pipeline extracts comments and filters noisy or low-information examples.Cleaning merges continuous comment lines, removes short, TODO, automated-checking, and non-text comments, and excludes functions with little semantic information; about 1.9 million pairs remain.Source: §4.2 · pp. 3–4

4.3 Code-Code

• Because manually judging code-pair semantics is costly and difficult, the authors use similar function names or documentation to collect candidate pairs.The collection uses NameMatcher and DocMatcher, retaining pairs whose retrieval scores exceed 0.75.Source: §4.3 · p. 4
• CrossModel is trained on the less noisy documentation-matched pairs and random negative pairs, then filters low-scoring pairs from both candidate sets.The remaining denoised sets form the final code–code corpus.Source: §4.3 · pp. 4–5
• The resulting code–code corpus contains 23.4 million pairs, with natural language used as an anchor for cross-language pairs.The authors state that such pairs can include different programming languages and mitigate the cross-language representation issue.Source: §4.3 · p. 5

5.2.2 Results

• CodeRetriever improves over GraphCodeBERT by 4.0 average absolute points on CodeSearch and outperforms UniXcoder on all tasks with reported results.It also outperforms baselines on Adv, CoSQA, CoNaLa, SO-DS, and StaQC.Source: §5.2.2 · p. 6
• AR2 improves over In-Batch Negative by 3.0 average absolute points and over Hard Negative by 1.1 average absolute points on average.In-Batch Negative outperforms the other approaches on StaQC; the authors suggest its training set contains more false query-code pairs.Source: §5.2.2 · p. 6

5.3.1 Low-Resource Code Search

• With training examples varied from 500 to FULL (19K) on CoSQA, CodeRetriever reaches more reasonable low-resource performance than GraphCodeBERT.The setting uses only a few hundred paired query-code examples for fine-tuning.Source: §5.3.1 · pp. 6–7

5.3.2 Cross-Language Code Search

• On cross-language search, unimodal contrastive learning significantly helps, and adding bimodal contrastive learning yields better performance.The experiment fine-tunes on query-Python data and evaluates on a query-Java test set.Source: Performance · p. 7

Performance

• Without unimodal contrastive learning, Java and Python representations form separate clusters; with it, they overlap, indicating a more unified code representation space.The comparison uses t-SNE visualizations of representations with and without unimodal contrastive learning.Source: Performance · p. 7

5.3.3 Code-to-Code Search Results

• CodeRetriever outperforms other pre-trained models on code-to-code retrieval evaluated with MAP on POJ-104.The authors describe this as evidence of potential for other code-understanding tasks.Source: §5.3.3 · p. 7

5.3.4 Uniformity and Alignment

• During training, uniformity loss declines gradually and alignment loss also declines, which the authors associate with alleviated anisotropy and improved alignment of code and natural-language representations.The authors say these metrics indicate a reduced gap between pre-training and fine-tuning compared with previous code pre-trained models.Source: §5.3.4 · p. 7

5.3.5 Ablation Study

• Undenoised code-code pairs slightly degrade performance, while denoising brings significant improvement; document-code and comment-code pairs bring further gains.The authors conclude that unimodal learning depends on positive-pair quality and leave more powerful pair construction methods for future work.Source: §5.3.5 · p. 8

Token-Level Code Pre-training

• Earlier token-level pre-training includes language-model and BERT-like approaches, while CodeGPT, PLBART, CoTexT, and CodeT5 target code generation.The passage notes token-level objectives can cause anisotropy and differ from code search’s function-level representation focus.Source: §6 · p. 8

Contrastive Learning for Code

• Prior contrastive code methods construct pairs through semantics-preserving transformations, bug injection, or program structures such as ASTs and CFGs.The passage says rule-based approaches can limit naturalness and diversity.Source: §6 · p. 8
• CodeRetriever builds positive pairs from code-code, code-documentation, and code-comment data, with code-code pairs drawn from real-world code.Source: §6 · p. 8

• CodeRetriever combines unimodal and bimodal contrastive learning as pre-training tasks for code search.Source: §7 · p. 8
• The authors report significant improvement and new state-of-the-art results on all evaluated publicly available benchmarks.Source: §7 · p. 9
• Further analyses report effectiveness in low-resource and cross-language code search and support the effectiveness of both contrastive-learning schemes.Source: §7 · p. 9

Limitations

• Only GraphCodeBERT initializes the experiments because of limited computing infrastructure; other pre-trained models are left for future work.The authors specifically name UniXcoder as an alternative for future experiments.Source: Limitations · p. 9
• Code-code and code-comment pairs still contain noise, and stronger denoising methods are planned for future work.Source: Limitations · p. 9
• Pre-training uses CodeSearchNet; the authors leave exploring larger corpora, such as full GitHub repositories, for future work.Source: Limitations · p. 9

A Statistics of Bimodal Pairs

• Table 7 is identified as reporting statistics for CodeRetriever’s code-text pairs.Source: Table 7 · p. 12

B Code-Code Pairs Construction

• Algorithm 1 is identified as the code-code pair construction procedure, with fragments naming documentation-, name-, and general-pair collections.Source: B Code-Code Pairs Construction · p. 12

C The Hyper-parameters for building code-code pairs.

• Table 8 is identified as listing hyperparameters for the matchers and CrossModel used to build code-code pairs.Source: Table 8 · p. 12

D Statistics of Unimodal Pairs

• Table 9 is identified as reporting statistics for CodeRetriever’s code-code pairs.Source: Table 9 · p. 12

E Implementation Details

• CodeRetriever shares code- and text-encoder weights in a siamese model initialized from a 12-layer GraphCodeBERT checkpoint.The checkpoint has hidden size 768 and 12 attention heads.Source: E Implementation Details · p. 12
• The training corpus contains 2.1 million code-document pairs, 23.4 million code-code pairs, and 1.9 million code-comment pairs.Source: E Implementation Details · p. 12
• Training uses eight V100 GPUs for 1.8 days, with batch size 256, learning rate 4e-5, and 100K steps.Maximum sequence lengths are 128 for text and 320 for code.Source: E Implementation Details · p. 13

F Statistics of Fine-tuning Data

• Table 10 is identified as reporting statistics for the downstream datasets.Source: Table 10 · p. 13

G Hyper-parameters of AR2

• Table 11 is identified as listing AR2 hyperparameters.Source: Table 11 · p. 13