PaperlayerView exact revision on arXiv

Source-linked AI summary

SynCoBERT: Syntax-Guided Multi-Modal Contrastive Pre-Training for Code Representation

Xin Wang, Yasheng Wang, Fei Mi, Pingyi Zhou, Yao Wan, Xiao Liu, Li Li, Hao Wu, Jin Liu, Xin Jiang

arXiv:2108.04556v3cs.CLcs.AIcs.PL

TL;DR

Existing code pre-training approaches treat code largely as token sequences or incorporate structure, while leaving identifier properties, AST-edge syntax, and relationships among code modalities underexplored. SynCoBERT combines identifier and AST-edge prediction with contrastive learning across code, comments, and ASTs. It achieves state-of-the-art results on four code-intelligence tasks using the same pre-training corpus and model size.

  • Problem

    Code pre-training has not sufficiently explored identifiers, syntax encoded by AST edges, or mutual information among code, comments, and AST modalities.

    Source: §1 · pp. 1–2

  • Method

    SynCoBERT combines identifier and AST-edge prediction objectives with multi-modal contrastive learning over code, comments, and ASTs.

    Source: §1 · p. 2

  • Results

    SynCoBERT achieves state-of-the-art results on four code-intelligence tasks with the same pre-training corpus and model size.

    Source: §6 · p. 7

  • Takeaways & Limitations

    The experiments support using identifier and AST-edge objectives together with cross-modal learning for code representation across the four evaluated tasks.

    Source: §6 · p. 7

Abstract

from arXiv · show

Code representation learning, which aims to encode the semantics of source code into distributed vectors, plays an important role in recent deep-learning-based models for code intelligence. Recently, many pre-trained language models for source code (e.g., CuBERT and CodeBERT) have been proposed to model the context of code and serve as a basis for downstream code intelligence tasks such as code search, code clone detection, and program translation. Current approaches typically consider the source code as a plain sequence of tokens, or inject the structure information (e.g., AST and data-flow) into the sequential model pre-training. To further explore the properties of programming languages, this paper proposes SynCoBERT, a syntax-guided multi-modal contrastive pre-training approach for better code representations. Specially, we design two novel pre-training objectives originating from the symbolic and syntactic properties of source code, i.e., Identifier Prediction (IP) and AST Edge Prediction (TEP), which are designed to predict identifiers, and edges between two nodes of AST, respectively. Meanwhile, to exploit the complementary information in semantically equivalent modalities (i.e., code, comment, AST) of the code, we propose a multi-modal contrastive learning strategy to maximize the mutual information among different modalities. Extensive experiments on four downstream tasks related to code intelligence show that SynCoBERT advances the state-of-the-art with the same pre-training corpus and model size.

1 Introduction

Code representation learning supports code-intelligence tasks, but existing approaches leave important properties of programming languages and relationships among code modalities underexplored. SynCoBERT addresses these gaps with objectives for identifiers and AST edges and contrastive learning across code, comments, and ASTs.

1 Introduction

  • Code representation learning provides representations for downstream code-intelligence tasks, including code search, clone detection, defect detection, and program translation.Source: §1 · p. 1
  • Programming-language properties remain underexplored, including the symbolic and syntactic information carried by identifiers, which should not be treated as ordinary code tokens.Source: §1 · p. 1
  • Edges linking AST non-leaf nodes to leaf nodes carry syntactic structure information that existing approaches should consider.For example, the AST represents x + y with a binary-operator node pointing to x, y, and +.Source: §1 · p. 2
  • Code snippets, comments, and parsed structures such as ASTs are semantically equivalent modalities that may provide complementary information, but prior work leaves their potential mutual information unexplored.Source: §1 · p. 2
  • SynCoBERT introduces Identifier Prediction and AST Edge Prediction, alongside multi-modal contrastive learning over code, comments, and ASTs.The objectives predict whether code tokens are identifiers and predict edges between AST nodes; contrastive learning targets mutual information across modalities.Source: §1 · p. 2

2 SYNCOBERT

SynCoBERT pre-trains on code, comments, and ASTs with four objectives. The framework combines masked-token prediction, identifier and AST-edge prediction, and multimodal contrastive learning.

2.1 Preliminary

  • SynCoBERT concatenates comment, code, and depth-first AST-token sequences, using [CLS]/[SEP] delimiters; its encoder has 12 layers, 768 hidden units, and 12 attention heads.For code without comments, the input contains code and AST sequences.Source: §2.1 · p. 2

2.2 Multi-Modal Masked Language Modeling (MMLM)

  • MMLM masks 15% of tokens across comment, code, and AST, replacing selected tokens with [MASK] 80% of the time, random tokens 10%, and leaving them unchanged 10%.The objective uses cross-entropy to predict the masked tokens.Source: §2.2 · p. 2

2.3 Identifier Prediction (IP)

  • IP applies binary classification to every code token, predicting whether it is an identifier or a non-identifier rather than masking only a subset.Identifiers are treated as symbols that can be renamed without changing code logic.Source: §2.3 · p. 3

2.4 AST Edge Prediction (TEP)

  • TEP masks AST edges and predicts whether node pairs are connected, encoding tree structure directly without adding model modules.The prediction uses the representations of the two nodes in each pair.Source: §2.4 · p. 3

2.5 Multi-Modal Contrastive Learning (MCL)

  • MCL uses paired comment–code/AST matches and order-swapped inputs as positives; unpaired data swaps code and AST, with 2N - 2 negatives per input.The objective seeks to maximize mutual information across code, comment, and AST modalities.Source: §2.5 · p. 4

2.6 Training Objective

  • The total training objective sums MMLM, IP, TEP, and MCL losses with L2 regularization to help prevent overfitting.The regularization coefficient is λ.Source: §2.6 · p. 4

3 Experiment Setup

The experiments reuse CodeSearchNet for pre-training and evaluate code understanding and translation across multiple datasets, comparing SYNCOBERT with models trained from scratch and pre-trained models.

3.1 Pre-Training Dataset and Settings

  • CodeSearchNet supplies 2.1M code-comment pairs and 6.4M standalone code samples across six languages; SYNCOBERT uses a 12-layer Transformer initialized from CodeBERT and trains for 110K steps.Training used batch size 128 and took about 80 hours on eight NVIDIA Tesla V100 GPUs.Source: §3.1 · p. 4

3.2 Evaluation Tasks, Datasets, and Metrics

  • Evaluation covers code search, clone detection, defect detection, and C#–Java translation, using task-specific datasets and metrics.The datasets include AdvTest and CodeSearch for code search; BigCloneBench and POJ-104 for clone detection; Defects4J for defect detection; and CodeTrans for translation.Source: §3.2 · p. 5

3.3 Baseline Methods

  • Baselines fall into two groups: models trained on each evaluation task from scratch and models pre-trained on unlabeled corpora before fine-tuning.The evaluation uses the default training, validation, and test splits for the datasets, except CodeSearch.Source: §3.3 · p. 5, §3.2 · p. 5

Training from Scratch

  • Scratch-trained baselines include NBow for code search, naive copy and PBSMT for translation, and TextCNN, BiLSTM, and Transformer models.The Transformer baseline matches pre-trained models in layer count and hidden size.Source: Training from Scratch · p. 5

Pre-Trained Models

  • Pre-trained baselines include RoBERTa variants, code2vec, CodeBERT, and GraphCodeBERT, which incorporate source code, AST paths, or data flow in different ways.CodeBERT is pre-trained on PL–NL pairs; GraphCodeBERT builds on CodeBERT and integrates code data flow.Source: Pre-Trained Models · p. 5

4 Results and Analysis

SYNCOBERT outperforms comparison models across code search, clone detection, defect detection, and translation; an ablation also finds a larger performance drop when MCL is removed.

4.1 Natural Language Code Search

  • SYNCOBERT outperforms all methods on both search datasets, exceeding CodeBERT by 10.9 points on AdvTest and 4.7 points on CodeSearch, and GraphCodeBERT by 2.9 and 2.7 points, respectively.The CodeSearch advantage over GraphCodeBERT is an average; both models use the same pre-training corpus as SYNCOBERT.Source: §4.1 · p. 6

4.2 Code Clone Detection

  • On POJ-104, SYNCOBERT exceeds CodeBERT and GraphCodeBERT by 5.57 and 3.08 points, and leads on BigCloneBench Recall and F1-score but not Precision.GraphCodeBERT is slightly better on BigCloneBench Precision.Source: §4.2 · p. 6

4.3 Code Defect Detection

  • On Defects4J, SYNCOBERT outperforms representative models, by 2.42 over CodeBERT and 1.29 over GraphCodeBERT.The authors also report that SYNCOBERT was not pre-trained on C, unlike code2vec, yet outperformed it.Source: Table 3 · p. 6, §4.3 · p. 6

4.4 Program Translation

  • On C#→Java translation, SYNCOBERT exceeds CodeBERT by 4.38 BLEU points, 2.5 exact-match points, and 2.81 CodeBLEU points.The authors note that SYNCOBERT was not pre-trained on C# and attribute cross-language generalization to syntactic and semantic similarities between C# and Java.Source: §4.4 · p. 6, Table 4 · p. 7

4.5 Ablation Study

  • Removing TEP and IP slightly hurts CodeSearch performance, while additionally removing MCL degrades it substantially.The ablation uses progressive component removal; the authors suggest MCL may help maximize mutual information across modalities for sentence-level representations.Source: §4.5 · p. 6, Table 5 · p. 7

5 Related Work

Prior code pre-training work varies in its use of syntax and symbolic information, while earlier code contrastive-learning approaches focus on a single modality. Multi-modal contrastive methods have also been explored in other domains.

Pre-Trained Models on Programming Language

  • Code pre-training includes language-model, encoder-decoder, and tree-based approaches, but the cited work notes gaps in exploiting AST edges, programming-language symbolic properties, and multi-modal potential.TreeBERT uses tree masked language modeling and node order prediction, but does not consider AST edges.Source: §5 · p. 7

Contrastive Learning

  • Contrastive learning has been applied to language and code, but the cited code approaches compare similar and dissimilar snippets within a single modality.The passage identifies this as leaving programming languages’ multi-modal characteristics unaddressed.Source: §5 · p. 7
  • Prior multi-modal contrastive methods in vision and graph learning combine information across modalities or structural views to learn representations.The examples include visual representation learning, multi-modal clustering, and contrasting graph views.Source: §5 · p. 7

6 Conclusion

SynCoBERT combines syntax-guided prediction objectives with multi-modal contrastive learning for code representation. Experiments on four code intelligence tasks show state-of-the-art performance with the same pre-training corpus and model size.

  • Experiments on four code intelligence tasks show state-of-the-art performance with the same pre-training corpus and model size.Source: §6 · p. 7
  • Identifier Prediction and AST Edge Prediction target identifiers and edges between AST nodes, respectively.Source: §6 · p. 7
  • Multi-modal contrastive learning maximizes mutual information among code, comment, and AST modalities to exploit their complementary information.Source: §6 · p. 7
Loading 2108.04556v3…