Self-Supervised Contrastive Learning for Code Retrieval and Summarization via Semantic-Preserving Transformations

1 Introduction

• Code retrieval needs large collections of semantically equivalent or nonequivalent snippets, but labeling them requires substantial human effort or noisy heuristics.Test-based labeling incurs code-execution costs, while shared comments, method names, or documents do not guarantee semantic equivalence.Source: §1 · p. 1
• Corder trains neural networks to distinguish semantically equivalent snippets generated by program transformations from dissimilar code.Transformations create syntactically different versions of a snippet while preserving its semantics.Source: §1 · pp. 1–2
• Corder’s pretrained encoder produces code vectors for retrieval and can be fine-tuned with a small amount of labeled data for downstream tasks.The paper evaluates code-to-code retrieval, text-to-code retrieval, and code summarization.Source: §1 · p. 2
• Corder-pretrained models outperform other pretext-task models for unsupervised code-to-code retrieval and perform better than scratch-trained models and other pretext tasks for downstream tasks.The introduction names text-to-code retrieval and code summarization as downstream tasks.Source: §1 · p. 2

Deep Learning Models of Code

• Earlier code models use graph neural networks over dependency-enriched ASTs or recurrent networks over paths and flattened subtrees to learn code representations.The related work names Allamanis et al. [3], Code2vec [6], Code2seq [4], and ASTNN.Source: §2 · p. 2
• Surveys on code embeddings [17] report a need to reduce labeled-data requirements and encourage learning source-code models from unlabeled data.Source: §2 · p. 2
• The cited self-supervised program-repair approach [73] is task-specific, while language-model pretraining [23] [34] is described as missing code syntax and semantic features.Source: §2 · p. 2

3.1 Approach Overview

• Corder transforms unlabeled snippets into positive pairs, encodes them as vectors, and uses contrastive prediction to distinguish same-origin variants from other snippets.The framework combines a program transformation module, a shared encoder, and a contrastive loss.Source: §3.1 · pp. 2–3, Figure 2 · p. 3

3.2 Approach Details

• For each minibatch, two randomly selected transformations generate variants of each snippet; the shared encoder maps them to vectors, and NCE trains on the resulting pairs.Other transformed snippets in the minibatch serve as negatives, and pair losses are computed in both directions.Source: §3.2 · p. 3

3.2.1 Program Transformation Operators.

• Corder uses variable renaming, dead-code insertion, independent statement permutation, loop exchange, and switch-to-if conversion to create semantically preserved variants with different code structures.The operators alter textual information or AST structure in different ways; statement permutation is restricted to statements without dependencies.Source: §3.2.1 · pp. 3–4

4 Use Cases

• The pretrained Corder models are intended for three downstream tasks: code-to-code retrieval, text-to-code retrieval, and code summarization.The encoders are used either to produce code vectors or to initialize supervised models.Source: §4 · p. 4

4.1 Using the Pre-trained Encoders to Produce Code Vectors for Downstream Task

• For code-to-code search, Corder maps codebase snippets and a code query to vectors, then retrieves the query’s top-k nearest neighbors by cosine similarity.The retrieved candidates are intended to be semantically equivalent to the query.Source: §4.1.1 · p. 4, §4.1 · p. 4

4.2 Fine-Tuning the Encoders for Supervised Learning Downstream Tasks

• For supervised downstream tasks, the Corder-pretrained encoder weights initialize the model, rather than starting the code encoder from random weights.The paper describes this transfer as a use of pretraining followed by fine-tuning.Source: §4.2 · p. 4

4.2.1 Text-to-Code Retrieval

• For text-to-code retrieval, the pretrained code encoder is paired with a randomly initialized text encoder; for code summarization, the pretrained encoder initializes the source-code encoder.Text and code embeddings are mapped into a shared space for retrieval.Source: §4.2.1 · pp. 4–5, §4.2.2 · p. 5

5.1.1 Data Preparation

• JavaSmall contains about 700k examples from 11 GitHub Java projects, and JavaMed contains about 4M examples from 1,000 top-starred GitHub Java projects.These datasets train the Corder pretext task; they are not the evaluation datasets.Source: §5.1.1 · p. 5

5.1.2 Encoders

• The evaluation compares AST-based Code2vec and TBCNN encoders with token-based two-layer Bi-LSTM and Transformer encoders.All techniques use node-type and token information to initialize AST nodes; the described embedding dimensions are 128.Source: §5.1.2 · p. 5

5.1.3 Research Questions

• The research questions ask whether pretrained code vectors help unsupervised code-to-code search and whether pretraining improves fine-tuned text-to-code search and code summarization.The latter comparison is against training downstream models from scratch.Source: §5.1.3 · p. 5

5.2.1 Datasets, Metrics, and Baselines

• Code-to-code retrieval takes a code snippet as query and seeks semantically related snippets from a candidate collection; OJ and BigCloneBench are used for evaluation.The paper motivates both datasets by their use in detecting semantically duplicated or similar code.Source: §5.2.1 · pp. 5–6
• OJ queries are sampled as 50 programs from each of 104 classes, while BigCloneBench evaluation samples clone and non-clone pairs and uses 5,000 clone-pair queries.OJ has 5,200 total queries; results are limited to the top 10 for both datasets.Source: §5.2.1 · p. 6
• The comparison includes Word2vec, Doc2vec, ElasticSearch, FaCoy, and encoders pretrained with masked language modeling or method-name prediction.The Transformer and Code2vec pretraining baselines use the JavaMed dataset.Source: §5.2.1 · p. 6, §5.2.2 · p. 6

5.2.2 Results

• Corder-pretrained Code2vec and BiLSTM on JavaMed outperform FaCoy, and Corder pretraining beats MNP and Masked-LM for code retrieval.Table 1 reports code-to-code retrieval results; the authors describe Corder as better than the alternative pretext tasks.Source: Table 1 · p. 6, §5.2.2 · p. 6

5.3 Fine-tuning Pre-trained Encoders for Text-to-code Retrieval

• The evaluation uses approximately 16 million preprocessed Java methods paired with docstrings from DeepCS.The retrieval task takes natural language as input and searches candidate code snippets.Source: §5.3.1 · p. 7

5.3.1 Datasets, Metrics, and Baselines

• Retrieval is evaluated with Precision@k for k=1, 5, and 10, and Mean Reciprocal Rank.The chosen Precision@k values reflect typical result-list sizes that users inspect.Source: §5.3.1 · p. 7
• The study compares models trained from scratch with fine-tuned Corder-pretrained encoders, including neural bag-of-words, BiLSTM, Transformer, TBCNN, and Code2vec.A Code2vec model pretrained with method-name prediction is also included as a fine-tuning baseline.Source: §5.3.1 · p. 7

5.3.2 Results

• Corder pretraining significantly improves text-to-code retrieval across encoders, with larger pretraining datasets producing higher-margin gains and Corder-pretrained Code2vec outperforming MNP-pretrained Code2vec.Table 2 reports text-to-code retrieval performance.Source: §5.3.2 · p. 7

5.4 Fine-tuning Pre-trained Encoders for Code Summarization

• The task predicts a full natural-language sentence from a short code snippet, using DeepCS data of approximately 16 million Java methods and corresponding docstrings.The target sequence averages about 12.3 in length.Source: §5.4.1 · p. 7

5.4.1 Dataset, Metric, and Baselines

• Performance is measured with BLEU and compared against 2-layer bidirectional LSTMs, Transformer, Code2seq, MOSES, and an information-retrieval baseline.The comparison also uses token splitting into subtokens and UNK replacement during inference.Source: §5.4.1 · p. 7
• Corder pretraining outperforms MNP and Masked-LM pretraining in BLEU for every encoder on code summarization.The authors describe the margin as significant.Source: §5.4.2 · p. 7

5.5 Compared Against Supervised Methods

• Corder-trained encoders significantly outperform the other supervised representation-learning methods on code classification.Table 4 reports the comparison against supervised methods.Source: Table 4 · p. 8, §5.5 · p. 8
• The evaluation follows a representation-quality protocol: embeddings are extracted for training samples and a Logistic Regression classifier is trained on top.Accuracy is the metric, and the OJ task classifies each code snippet by functionality.Source: §5.5 · p. 8
• The supervised representation baselines include AdaSent and InferSent, whose encoders are trained on supervised tasks, plus Code2vec pretrained with method-name prediction.All are evaluated through the classifier trained on their embeddings.Source: §5.5 · p. 8

• Corder outperforms most baselines on code-to-code retrieval, text-to-code retrieval, and code summarization.Source: §5.6 · p. 8
• Corder embeddings also perform better than embeddings from other supervised methods on code classification.Source: §5.6 · p. 8
• Pretraining comparisons are limited because some tasks are difficult to adapt across encoders, and unbiased settings are not guaranteed.The study therefore adapts MNP for Code2vec and Masked-LM for Transformer.Source: §5.6 · p. 8

6 Analysis and Ablation Study

• The analysis includes an ablation study of individual operators and a visualization of code embeddings.Source: §6 · p. 8, §6.1 · p. 8

6.1 Impact of Transformation Operators

• With downsampled data, SF, PS, and US outperform VR on text-to-code retrieval and code summarization, and SF leads text-to-code retrieval.With the original data, PS and US perform best most of the time, while SF is comparable to VR despite applying to fewer snippets.Source: §6.1 · p. 9
• The operator results support the conclusion that changing code structure is crucial; VR changes textual information, whereas PS, US, and SF modify structure extensively.Source: §6.1 · p. 9

6.2 Visualization for Code Embeddings

• The embedding visualization compares Corder-Code2vec with MNP-Code2vec on vectors from six OJ Dataset classes.The authors conclude that instance discrimination is a better pretext task than method-name prediction for the same Code2vec encoder.Source: Figure 5 · p. 9, §6.2 · p. 9

• Corder applies program transformation operators to create syntactically diverse, semantically equivalent code snippets.Source: §7 · p. 9
• Its contrastive objective minimizes distances between representations of similar snippets and maximizes distances between dissimilar snippets.Source: §7 · p. 9
• Corder pretraining significantly outperforms models without contrastive learning on code-to-code retrieval, text-to-code retrieval, and code summarization.Source: §7 · p. 9