PaperlayerView exact revision on arXiv

Source-linked AI summary

Synthetic Semantic Supervision for Contrastive Code Representation Learning in Small Transformers: An Empirical Study

Kenneth Paulsen, Florian Tambon, Mike Papadakis, Shin Yoo

arXiv:2609.03702v1cs.AI

TL;DR

Existing code-embedding methods rely on costly large models or labor-intensive and imperfect supervision, while compact models need effective semantic training signals. This paper trains small encoders with synthetic code descriptions and contrastive learning, evaluating them across eight tasks. The approach significantly improves over same-size pretrained baselines on five tasks, matches them on two more, and remains competitive with much larger models and execution-aware supervision.

  • Problem

    Large embedding models incur high computational and financial costs, while human-written supervision is labor-intensive and can be inconsistent or poorly generalizing.

  • Method

    SyncDesc pairs code with synthetically generated semantic descriptions and trains a compact dual encoder using a contrastive objective.

  • Results

    SyncDesc significantly improves over same-size pretrained baselines on five of eight tasks, reaches parity on two more, and is comparable to existing baselines on vulnerability detection.

  • Takeaways & Limitations

    Synthetic semantic supervision combined with contrastive learning offers a scalable alternative for compact code representation learning.

  • Takeaways & Limitations

    The evaluation covers isolated method-level code in C, C++, and Java, leaving industrial, multi-file, project-level distributions and other languages untested.

Abstract

from arXiv · show

General-purpose code embeddings power tools for code search, classification, and retrieval. Compact transformer encoders for code typically rely on either human-written docstrings (labor-intensive and inconsistent) or mined structural signals such as execution traces (setting-specific and costly to collect). We empirically study an alternative: contrastive pretraining of small encoders with synthetically generated natural-language descriptions emphasizing code functionality and intent, paired with code in a dual-encoder framework at training and discarded at inference. We benchmark this approach against pretraining-based baselines, generalist LLMs, and embedding-specific models on eight retrieval, classification, and generation tasks across C, C++, and Java. Synthetic semantic supervision yields statistically significant gains over pretraining baselines of the same inference-time size on five of eight tasks, with parity on two more; once fine-tuned, it matches or exceeds zero-shot models two orders of magnitude larger on classification, and it stays on par with execution-aware supervision at matched pretraining data, suggesting a scalable, effective alternative to existing code-representation paradigms.

1 Introduction

Code representations support automated code tools, but existing embedding approaches face cost, supervision-quality, and generalization limitations. The paper studies SyncDesc, which uses synthetic semantic descriptions with contrastive learning to train compact code embeddings and evaluates it against multiple paradigms.

  • Code representations translate source code into formats usable for code search, documentation, analysis, and coding assistance.
  • Large language models and specialized embedding models can generate code embeddings, but their computational and financial costs scale poorly for large embedding workloads.
  • Human-written specifications provide semantic supervision but are labor-intensive, inconsistent, and limited in generalization across tasks, languages, and evaluation settings.
  • Structural supervision automates signal collection through traces, instrumentation, or program analysis, but the paper contrasts it with synthetic semantic supervision.
  • The study asks whether synthetic semantic descriptions paired with contrastive learning can produce compact code embeddings that rival or exceed existing methods.
  • SyncDesc uses synthetically generated semantic descriptions paired with code in a dual-encoder contrastive objective, then discards descriptions at inference.

2 Experimental Setup

The experiments compare compact transformer representations trained with different supervision paradigms against larger generalist and embedding models. Evaluation covers eight retrieval, classification, and generation tasks using established datasets and task-specific protocols.

  • The compact-model baselines use a 125M-parameter UnixCoder backbone with standard pretraining, execution-aware TRACED, or synthetic-code ContraCode objectives.
  • SyncDesc generates synthetic natural-language descriptions for code snippets and aligns them with code through a dual-encoder contrastive learning pipeline.
  • ContraCode uses approximately 1.8M JavaScript methods, while TRACED uses a CodeNet corpus and SyncDesc uses Java method-level code–description pairs.
  • The study evaluates eight downstream tasks spanning retrieval, classification, and generation, fine-tuning small transformers while using large language models without further fine-tuning.
  • Evaluation uses ranking and similarity metrics for retrieval, accuracy and F1 for classification, and embedding-based semantic similarity for generation.

3 Results

SyncDesc improves small-code-encoder performance across retrieval and classification while remaining competitive with much larger models and execution-aware supervision. Ablations indicate that code-conditioned synthetic semantics and contrastive learning jointly drive the gains, with improved embedding geometry.

  • 3.1 Comparison to transformer baselines: SyncDesc significantly outperforms standard transformer and execution-aware baselines on five of eight tasks, with parity on two more.The comparison covers retrieval, classification, and generation tasks across the evaluated languages.
  • 3.1 Comparison to transformer baselines: At matched data, SyncDesc is on par with TRACED while requiring no execution infrastructure.The comparison controls for corpus, data volume, language, backbone, and fine-tuning protocol.
  • 3.2 Scale comparison: A fine-tuned 125M SyncDesc encoder reaches F1 0.287 versus 0.270 for prompted large language models despite being over two orders of magnitude smaller.The authors note that this comparison contrasts fine-tuning with zero-shot prompting.
  • 3.2 Scale comparison: SyncDesc is competitive with or outperforms large embedding models on title and tag retrieval while trailing on docstring matching.It achieves this with a model approximately 56× smaller than Nomic Embed Code.
  • 3.3 Ablation and latent-space analysis: Ablation analyses exclude Docstring Matching and focus on tasks whose evaluation targets are independent of the supervision source.The selected analyses include Clone Detection, Java Tag Classification, and CodeChef Tag Matching.
  • 3.3.1 Ablation Study: Every ablation degrades performance, with Clone Accuracy 0.724 and Tag F1 0.276 for SyncDesc; semantic content and contrastive learning are primary drivers.On CodeChef Tag Matching, the full model leads every ablated arm by at least 2.5 points.
  • 3.3.1 Ablation Study: Code-conditioned synthetic descriptions outperform GPT-4o paraphrases of human docstrings, while Qwen2.5-Coder-7B generation matches within seed noise.Clone accuracy is 0.752 versus 0.728, and tag F1 is 0.260 versus 0.230.
  • 3.3.1 Ablation Study: Synthetic descriptions improve cluster separation under both semantic category and difficulty labels, whereas human docstrings show weaker separation.Synthetic descriptions also have higher lexical diversity and lower identifier overlap than human docstrings.

4 Related Work

Prior code representation methods use denoising, execution traces, generative objectives, augmentations, or retrieval-specific architectures. This work instead makes contrastive code–text alignment central and varies the supervision source by pairing code with synthetic semantic descriptions.

  • Pretraining-based code encoders: Early code encoders use denoising objectives, auxiliary token or data-flow signals, and encoder–decoder architectures.Examples include CodeBERT, GraphCodeBERT, and UnixCoder.
  • Structural supervision: Execution-aware methods incorporate traces, coverage, and runtime values but require executable code, runtime infrastructure, and dataset-specific tooling.These requirements limit scalability.
  • Large code and embedding models: Large autoregressive code models emphasize next-token prediction, while embedding-oriented models train dual encoders contrastively on large code–docstring corpora.These approaches benefit from scale and specialized training data.
  • Contrastive Code–Text Alignment: Contrastive learning aligns code and text, but its effectiveness remains limited by the supervision source when used as a secondary objective.Making contrastive alignment primary directly optimizes embedding similarity and discrimination.
  • Contrastive Code–Text Alignment: Prior contrastive-code work varies augmentations, dropout, hard negatives, retrieval architectures, or other design axes, whereas this contribution varies supervision source.The paper identifies combining synthetic supervision with those advances as future work.
  • Contribution: The proposed approach treats contrastive alignment as central and pairs code with synthetic semantic descriptions for consistent instance-level grounding without relying on missing or low-quality docstrings.This positions synthetic semantic supervision as an orthogonal design dimension to existing contrastive methods.

5 Conclusion

SyncDesc uses synthetically generated semantic descriptions to train compact code representations and performs competitively across diverse evaluations. Its benefits are strongest for semantic tasks, while future work targets broader code settings and deeper structural reasoning.

  • Conclusion: SyncDesc transfers a large teacher’s semantic extraction into a compact encoder for embedding workloads.The encoder is trained with synthetic semantic descriptions and then used without the descriptions at inference.
  • Conclusion: Across eight tasks in C, C++, and Java, SyncDesc performs better on five tasks and remains comparable on structurally oriented tasks.The evaluation spans retrieval, classification, and generation against transformer baselines.
  • Conclusion: SyncDesc matches more complex baselines with matched pretraining data and surpasses them when synthetic descriptions are scaled on semantic tasks.The conclusion frames semantic supervision as complementary to, rather than a replacement for, dynamic supervision.
  • Conclusion: Future work could extend supervision to more languages, multi-file or project-level code, description-property analysis, and structured signals such as type information.These directions target representations for tasks requiring deeper semantic or structural reasoning.

Limitations

The study’s conclusions are bounded by synthetic-description quality, method-level evaluation, and unequal adaptation effort across model families. These constraints limit claims about reproducibility, broader deployment settings, and fair cross-scale comparisons.

  • Synthetic supervision: GPT-4o-generated supervision may contain omitted semantics or stylistic biases rather than purely generalizable semantic structure.A fixed prompting template is used, but description quality is not independently verifiable and exact regeneration depends on a changing model.
  • Evaluation scope: Evaluation covers isolated methods in C, C++, and Java, leaving industrial, multi-file, project-level, repository-level, and other-language settings untested.The excluded settings include functions whose semantics depend on surrounding modules, frameworks, or build context.
  • Comparison design: 125M baselines are fine-tuned per task, whereas LLM-scale embedding and prompted models are evaluated as-is, so adaptation effort is not matched.A frozen-encoder evaluation partially addresses this asymmetry within the small-model family.

A Full results with standard deviations

The full numerical results confirm significant improvements over the strongest 125M baseline on five of eight tasks, with two nonsignificant improvements and one vulnerability-detection shortfall.

  • Significance results: Five of eight headline comparisons are statistically significant under paired instance-level bootstrap tests.The tests use 10k resamples on saved per-item predictions.
  • Significance results: Two tasks improve on average without statistical significance: description generation and CodeForces tag classification.The appendix reports these as better on average but not significantly better.
  • Significance results: Vulnerability detection is at parity-to-below: SyncDesc-S trails UnixCoder with F1 0.557 vs. 0.570, while exceeding TRACED at matched pretraining data with 0.557 vs. 0.545.The UnixCoder comparison is nonsignificant.

B Detailed downstream-task comparison

Detailed comparisons show that synthetic semantic supervision is strongest on semantic retrieval and matching, while gains depend on task, data composition, and whether models are fine-tuned or frozen.

  • Standard baselines: Vulnerability detection remains narrow: F1 is 0.557 vs. 0.570 against UnixCoder, while SyncDesc exceeds TRACED at matched data with 0.557 vs. 0.545.The UnixCoder difference is not significant.
  • Single-source versus dual-source pretraining: Adding Java data to C/C++ single-source pretraining yields +1.5 MAP on clone retrieval, from 0.855 to 0.870.The dual-source effect depends on the relative richness of the single-source corpus.
  • Zero-shot frozen-encoder evaluation: Frozen SyncDesc-1S-C reaches 0.752 zero-shot clone MAP, +24.6 points over frozen UnixCoder and 88% of its fine-tuned 0.855.The frozen comparison is restricted to language-matched settings.
  • Table conventions: Table 1 averages metrics over three seeds with standard deviation and marks statistically significant SyncDesc improvements using paired bootstrap tests.The table distinguishes single-source and dual-source variants and identifies best and second-best approaches per task.
  • Significance analysis: Table 2 tests each SyncDesc headline variant against the strongest 125M baseline using 10k resamples and 95% confidence intervals on metric differences.The analysis uses saved per-item predictions.
  • Zero-shot frozen-encoder evaluation: Cross-language frozen clone retrieval collapses, with 1S-Java at 0.258, while fine-tuning erases these differences.The CodeForces frozen columns remain flat and the large embedding LLMs retain their lead there.

C Detailed ablation study

The ablations show that synthetic semantic supervision and contrastive learning are central to SyncDesc’s gains, while controlled comparisons extend this result from classification to retrieval. Evaluation also highlights adaptation and protocol boundaries when comparing small encoders with larger models.

  • Alternative supervision sources: Human docstrings reduce performance by approximately 3 percentage points on Clone detection and 5 percentage points on Tag classification.The comparison keeps the rest of the training process unchanged and indicates that native docstrings generally underperform synthetic descriptions.
  • Alternative supervision sources: Random text causes average drops of approximately 7 percentage points on Clone detection and 3 percentage points on Tag classification.This tests whether gains arise from the training procedure alone rather than meaningful supervision content.
  • Training objectives: Replacing contrastive learning with masked language modelling drops Clone detection by 4.2 percentage points and Tag classification by 3.2 percentage points.The result supports a specific contribution from the contrastive objective.
  • Retrieval alignment: On CodeChef Tag Matching, the full model exceeds every ablated arm by at least 2.5 points, extending the classification ordering to retrieval.The task uses Java-aligned supervision, avoiding the cross-language confound noted for a C++ target.
  • Controlled supervision sources: In the controlled 50k supervision-source suite, code-conditioned synthetic descriptions outperform rewritten and raw human docstrings, while Qwen2.5-Coder-7B matches GPT-4o within seed noise.Clone accuracy is 0.752 for code-conditioned synthetic descriptions versus 0.728 for rewritten docstrings; tag F1 is 0.260 versus 0.230.

D.1 Example synthetic descriptions vs. human docstrings

The examples show synthetic descriptions replacing identifier-heavy human docstrings with longer, domain-neutral summaries of functionality and control flow. Across examples, they abstract concrete names while preserving operational intent.

  • Recommendation example: 56 tokens versus 12 tokens: the synthetic recommendation description explains score calculation, conditional accumulation, entity counting, and return behavior.
  • Description characteristics: Synthetic descriptions summarize functionality and control flow in domain-neutral terms, unlike terse human docstrings that reuse code identifiers.The installation example describes validation, downloading, finalization, logging, and exception handling rather than naming concrete identifiers.
  • Installation example: 56 tokens versus 17 tokens: the synthetic installation description summarizes download, validation, and finalization with negligible identifier overlap.
  • Circuit example: 50 tokens versus 15 tokens: the synthetic circuit description abstracts Gate and Wire into nodes and links while summarizing topology-preserving reconstruction.

E Latent-space (UMAP) analysis

The UMAP analysis projects code embeddings into two dimensions to compare synthetic-description and human-docstring training under semantic and non-semantic labels.

  • Visualization setup: UMAP projections compare synthetic-description and human-docstring embeddings across algorithmic semantic labels and non-semantic difficulty labels.The first and third panels use synthetic descriptions, while the second and fourth use human docstrings.

F Decontamination audit

The audit examines overlap between pretraining corpora and downstream test sets, while documenting generation, cost, and deployment boundaries. Reported overlap was small, and synthetic supervision avoided sending test items to the generation API.

  • Audit method: Exact duplicates use MD5 hashes of normalized code, while near-duplicates use MinHash-LSH with 128 permutations, 5-gram shingles, and Jaccard ≥0.7.
  • Audit findings: 911 BigCloneBench functions, or 0.11%, had verified exact-hash overlap; expected metric shift on affected metrics was under 0.5%, and C-side test sets had zero overlap.
  • Generator-side leakage: No test items were sent to the GPT-4o API for description generation, so synthetic supervision did not encode test-set content.
  • Cost and deployment: USD 91 of one-time description generation was followed by 8 A100-hours of contrastive pretraining, with the pretrained encoder reused across tasks.
  • Comparison with execution traces: Execution-trace collection required about 4,020 sequential CPU-hours on the same CodeNet C corpus, whereas description generation applies uniformly across languages.
  • Inference boundary: Inference reached roughly 204 snippets/s unbatched on an RTX 4090, without per-query API cost, latency, rate limits, or code egress.
Loading 2609.03702v1…