Source-linked AI summary
What do pre-trained code models know about code?
Anjan Karmakar, Romain Robbes
TL;DR
The paper asks whether code-model embeddings capture source-code characteristics broadly enough for downstream use, beyond success on individual software engineering tasks. It probes four models with tasks targeting surface, syntactic, structural, and semantic information. GraphCodeBERT is most consistent overall, while BERT performs surprisingly well on some tasks, leaving the small performance gap between code-aware models and BERT for further investigation.
Problem
It remains unclear whether pre-trained code embeddings comprehensively encode source-code characteristics well enough to support a broad range of downstream software engineering tasks.
Source: §I · p. 1Method
The paper applies four probes for surface, syntactic, structural, and semantic information to BERT, CodeBERT, CodeBERTa, and GraphCodeBERT.
Source: §I · p. 1Results
GraphCodeBERT is the most consistent model, with up to 10-12% improvements in accuracies from the BERT baseline, while BERT performs competitively and better on surface information.
Source: D. Discussion · p. 4Takeaways & Limitations
The probes help identify model strengths and weaknesses, compare hidden-layer roles, test linear extractability, and examine sample efficiency.
Source: §V · p. 4
Abstract
from arXiv · showhide
Pre-trained models of code built on the transformer architecture have performed well on software engineering (SE) tasks such as predictive code generation, code summarization, among others. However, whether the vector representations from these pre-trained models comprehensively encode characteristics of source code well enough to be applicable to a broad spectrum of downstream tasks remains an open question. One way to investigate this is with diagnostic tasks called probes. In this paper, we construct four probing tasks (probing for surface-level, syntactic, structural, and semantic information) for pre-trained code models. We show how probes can be used to identify whether models are deficient in (understanding) certain code properties, characterize different model layers, and get insight into the model sample-efficiency. We probe four models that vary in their expected knowledge of code properties: BERT (pre-trained on English), CodeBERT and CodeBERTa (pre-trained on source code, and natural language documentation), and GraphCodeBERT (pre-trained on source code with dataflow). While GraphCodeBERT performs more consistently overall, we find that BERT performs surprisingly well on some code tasks, which calls for further investigation.
I. INTRODUCTION
The paper asks what code properties pre-trained embeddings capture, since strong performance on individual software engineering tasks does not establish broad code understanding. It adapts probing to code models with four tasks and reports that code-trained models tend to do better on code-specific probes, but their advantage over BERT is smaller than expected.
I. INTRODUCTION
- The paper investigates whether pre-trained embeddings capture code structure, syntax, and semantics broadly enough to support accurate, robust, and generalizable code models beyond single-task systems.The authors note that successful downstream performance leaves open what the models learn about code.Source: §I · p. 1
- The study compares BERT with CodeBERT, CodeBERTa, and GraphCodeBERT to examine embeddings from models with different code and structural pre-training.BERT is the English-pretrained baseline; the code models add source code, documentation, or data-flow information.Source: §I · p. 1
- Four probes assess syntactic, structural, surface-level, and semantic information through AST Node Tagging, Cyclomatic Complexity Prediction, Code Length Prediction, and Invalid Type Detection.The authors say these tasks cover commonly identifiable abstractions, but more tasks are needed to probe each abstraction thoroughly.Source: §I · p. 1
- The paper introduces four probing tasks and releases their corresponding datasets publicly.It presents probing as a preliminary empirical study and potential pseudo-benchmark for pre-trained models.Source: §I · p. 1
- Probing suggests that models encode syntactic and semantic properties to varying degrees, while code-aware models tend to perform better on code-specific probes by less than expected over BERT.The authors call for further study and more effective pre-training procedures that capture diverse source-code characteristics.Source: §I · p. 1
II. BACKGROUND
Probes use simple diagnostic classifiers on frozen model representations to test whether specific properties are encoded. While NLP probing has studied varied linguistic properties and BERT layers, probing of source-code models remains rare.
II. BACKGROUND
- A probe pairs an auxiliary diagnostic task with a classifier to test whether a specific property is encoded in pre-trained model weights without fine-tuning on downstream task data.Tasks are often kept simple to reduce interpretability problems.Source: §II · p. 2
- The classifier uses vectors from frozen hidden layers and is usually linear, so successful prediction suggests the property is encoded in those representations.Studies generally use probes to compare property encoding across models or layers, rather than focus on raw accuracy.Source: §II · p. 2
- NLP probing has examined linguistic properties and BERT layer hierarchies, whereas source-code probing was described as nearly absent, with one known coarse programming-language identification example.The cited BERT work places surface, syntactic, and semantic features at different levels of its representation hierarchy.Source: §II · p. 2
III. PROBING SOURCE CODE
The paper constructs four probing tasks to test whether code-model embeddings reflect surface, syntactic, structural, and semantic characteristics. It probes four transformer models with linear classifiers using balanced task datasets derived from Java projects.
AST Node Tagging (AST)
- AST Node Tagging tests syntactic information by predicting AST node tags, which the paper treats as a potential prerequisite for code tasks such as completion.ASTs underpin many structured source-code representations, and the probe examines syntax encoded in model layers.Source: §III · p. 2
Cyclomatic Complexity (CPX)
- Cyclomatic Complexity Prediction probes structural information by asking models to predict snippet complexity, which depends on the number of linearly independent paths.The paper notes that predicting complexity from a token sequence may be challenging.Source: §III · p. 2
Code Length Prediction (LEN)
- Code Length Prediction probes whether transformer embeddings encode the surface-level length of a code snippet.The authors conjecture that length may be easy to predict when code is input as a token sequence.Source: §III · p. 2
Invalid Type Detection (TYP)
- Invalid Type Detection tests semantic awareness by distinguishing snippets with intentionally misspelled primitive types from syntactically legitimate code.The negative examples are created by swapping consecutive characters at random indexes in type names.Source: §III · p. 2
Probing data & labels
- Each task has 10k class-balanced samples from method-level Java code; labels include 20 AST node classes, complexity values from 0 to 9, five length bins, and intentionally misspelled types.The source is a subset of the 50K-C dataset of compilable Java projects.Source: §III · p. 2
Pre-trained Models and Probing Classifier
- The four models are English-pretrained BERT and CodeBERT, CodeBERTa, and GraphCodeBERT, trained on code and documentation or code with data-flow information.A simple linear classifier uses hidden-layer vectors to test which code properties are linearly correlated with model embeddings.Source: §III · p. 2
IV. EARLY RESULTS AND DISCUSSION
The authors frame their probe analysis around differences between models rather than general probe accuracy. Table I is identified as reporting probing-task accuracies.
IV. EARLY RESULTS AND DISCUSSION
- The analysis prioritizes differences between models over the probes' general accuracy.Source: §IV · p. 3
- Table I is labeled as presenting probing-task accuracies.Source: Table I · p. 3
A. Model Analysis
GraphCodeBERT performs most consistently overall, but BERT is unexpectedly competitive; results vary across surface, syntactic, structural, and semantic probes.
A. Model Analysis
- BERT leads code length prediction with 76.05% accuracy, which the authors relate to its extensive training data and the token-counting nature of the task.Source: A. Model Analysis · p. 3
- With enough training samples, all models exceed 85% accuracy on AST node tagging, reaching 92.55% for CodeBERTa [34].The authors interpret this as evidence that syntax-related information needed for node tagging is encoded in hidden layers.Source: A. Model Analysis · p. 3
- GraphCodeBERT reaches 46.70% accuracy and BERT 42.65% on cyclomatic complexity prediction, the most challenging structural probe.The results raise the question of whether BERT encodes structural information or code models fail to encode it effectively.Source: A. Model Analysis · p. 3
- GraphCodeBERT achieves the highest invalid-type prediction accuracy at 97.20%, while every code model exceeds 90% and BERT is not far behind.The authors caution that this binary task has a higher naive baseline accuracy.Source: A. Model Analysis · p. 3
- GraphCodeBERT is most consistent overall, while English-pretrained BERT remains surprisingly competitive against code models.The authors call for further investigation of this pattern.Source: A. Model Analysis · p. 3
B. Layer analysis
Code models show heterogeneous layer-wise probe performance, whereas BERT’s performance is fairly homogeneous across layers.
B. Layer analysis
- The layer analysis uses embeddings from hidden layers 1–12, except CodeBERTa’s six layers; layer 0 represents naive baseline accuracy.Source: B. Layer analysis · p. 3
- No single layer is optimal across tasks for the code models, suggesting that learned code-property abstractions are spread across multiple layers.Source: B. Layer analysis · p. 3, Figure 1 · p. 3
- Unlike the code models, BERT exhibits fairly homogeneous performance across layers rather than localized performance for code properties.The paper contrasts this with prior observations about BERT for English.Source: B. Layer analysis · p. 3
C. Sample Size Analysis
Accuracy generally rises with more probe-training samples, and layer-to-layer irregularity flattens; GraphCodeBERT shows a marked sample-efficiency advantage on invalid-type prediction.
C. Sample Size Analysis
- The study evaluates probes with 10% and 1% of the dataset and caps training at 10,000 samples.The authors expected underlying syntactic, semantic, and structural rules to be learnable sample-efficiently without memorization.Source: C. Sample Size Analysis · p. 3
- Probe accuracies increase as training data grows from 100 to 1,000 to 10,000 samples.Source: C. Sample Size Analysis · p. 3
- Layer-to-layer accuracy irregularity flattens as sample size increases, a trend reported for all tasks.Source: C. Sample Size Analysis · p. 4
- GraphCodeBERT’s accuracy with 100 samples exceeds BERT’s with 10,000 on invalid-type prediction, a sample-efficiency pattern not seen for AST or CPX.The authors attribute the result to GraphCodeBERT extracting more signal from fewer samples.Source: C. Sample Size Analysis · p. 4
D. Discussion
The discussion finds that linear probes access some code properties more readily than others, with CPX particularly difficult. BERT’s competitiveness and code models’ uneven advantages motivate further investigation and improved pretraining.
D. Discussion
- Linear classifiers extract some code characteristics from pretrained embeddings, but cyclomatic complexity less effectively; the authors propose testing nonlinear classifiers such as an MLP.They also argue that more effective pretraining procedures should be explored.Source: D. Discussion · p. 4
- BERT is competitive across tasks, while CodeBERTa and CodeBERT show promising syntax and semantic results; GraphCodeBERT is most consistent, with up to 10–12% accuracy improvements over BERT.GraphCodeBERT struggles to exceed 85% on AST tagging; the authors hypothesize it may lack data-flow context when predicting from a single token.Source: D. Discussion · p. 4
- BERT’s strong baseline results are surprising, but source-code models show an advantage on invalid-type prediction with fewer samples, particularly GraphCodeBERT.Source: D. Discussion · p. 4
- The explanation that code models under-encode the probed properties rather than BERT possessing comparable code knowledge remains a hypothesis needing confirmation on more tasks.If confirmed, the authors say more effective pretraining procedures should be explored.Source: D. Discussion · p. 4
V. CONCLUSION & FUTURE WORK
The paper uses probes to assess code-model capabilities and hidden-layer roles, finding a slim performance gap between models expected to know code and those that are not. It outlines broader probes and model comparisons as future work.
V. CONCLUSION & FUTURE WORK
- The study surveys four publicly available pre-trained source-code models using representative probes of diverse code characteristics.The motivation is the rapid introduction of such models in IDE extensions, plugins, and web engines.Source: §V · p. 4
- Probes can gauge model strengths and weaknesses, characterize hidden-layer roles, test linear extractability, and provide insight into task-specific sample efficiency.The authors frame these uses as a way to inspect large pre-trained models.Source: §V · p. 4
- Initial probes found a slim performance margin between models expected to know code and those that should have no code knowledge, a result needing further investigation.If confirmed, the authors say this would suggest room for more advanced source-code pre-training techniques.Source: §V · p. 4
- Future work will add probing tasks for more source-code characteristics and compare additional models, including CuBERT, C-BERT [10], PLBART, and Codex [11].The authors also plan more probes based on structure, syntax, and semantics.Source: §V · p. 4
- The authors plan context-based probes for code search and summarization and extension to multiple languages, with a longer-term goal of a pseudo-benchmark for model development.The proposed suite would assess whether new models encode important source-code characteristics.Source: §V · p. 4
- The authors made code, datasets, and experimental results available online for replication.The passage identifies GitHub as the repository location.Source: §V · p. 4