What Do They Capture? -- A Structural Analysis of Pre-Trained Language Models for Source Code

1 INTRODUCTION

• Prior empirical work compared where code embeddings perform well but did not explain their inner mechanisms or what pre-trained models capture.Code representations support tasks including completion, search, and summarization.Source: §1 · pp. 1–2
• The paper analyzes CodeBERT and GraphCodeBERT using attention alignment, structural probing of contextual embeddings, and syntax-tree induction without training.The analyses target whether models represent code syntax in attention weights, vectors, and induced trees.Source: §1 · p. 2
• Attention captures high-level AST motif structure, and structural probing finds syntax structure implicitly embedded in transformed contextual word vectors.The paper reports evidence for these findings from its attention analysis and probing approach.Source: §1 · p. 2
• Pre-trained code models can induce syntax trees to a certain extent without training.The authors describe this as the third analysis contribution.Source: §1 · p. 2

2.1 Self-Attention-Based Transformer

• A Transformer maps code tokens to layerwise contextual representations using blocks with multiple self-attention heads that aggregate information across tokens.Positional encodings are injected into input embeddings to represent token order.Source: §2.1 · pp. 2–3

2.2 Pre-Training Language Model

• BERT pre-training uses masked language modeling and next-sentence prediction, with selected input tokens masked, left unchanged, or replaced.The described masking procedure selects 15% of tokens, then applies the 80% masked, 10% unchanged, and 10% random-replacement split.Source: §2.2 · p. 3
• CodeBERT masks concatenated source code and natural-language descriptions, while GraphCodeBERT adds data-flow information among variables during pre-training.These are the two representative code models selected for the paper’s analysis.Source: §2.2 · p. 3

3 MOTIVATION

• In the Layer 5 averaged attention heatmap, groups of code tokens correspond to tokens in the same AST branch, as illustrated by an if statement.The example is the token sequence “if exit_code is not None.”Source: §3 · p. 4
• In Layer 5, Head 12, tokens in an AST motif are highlighted as closely connected by self-attention.The figure marks connected nodes that appear in the AST motif structure.Source: §3 · p. 4
• These attention patterns motivate quantitative, systematic analysis of attention distributions and generated hidden vectors to investigate captured code features.The analysis examines both outputs under the Transformer framework.Source: §3 · p. 4

4.1 Attention Analysis

• Attention analysis measures whether high-weight token pairs share an AST parent and examines how attention varies across inputs.The analysis excludes tokens adjacent in textual order; high variability suggests content-dependent attention, while low variability suggests content-independent attention.Source: §4.1 · pp. 4–5

4.2 Structural Probing on Word Embedding

• Structural probing tests whether a learned linear mapping makes embedding distances correspond to syntax-tree path lengths, so distance-one pairs identify neighboring nodes.The mapping is trained on tree distances between token pairs in source-code sequences.Source: §4.2 · p. 5

4.3 Syntax Tree Induction

• Syntax-tree induction derives adjacent-token distances from a pre-trained model and uses them to induce a binary tree without additional training.The approach assumes that similar token representations or attention distributions indicate tokens that are close in the syntax tree.Source: §4.3 · pp. 5–6

Distance measurement

• Syntactic distances between adjacent tokens can be measured from intermediate representations or attention distributions using different distance functions.The listed options include L1 and L2 for representations and Jensen-Shannon divergence and Hellinger distance for attention distributions.Source: §13 · p. 6, Table 1 · p. 6

Tree reconstruction

• A greedy top-down recursive partitioning algorithm converts the distance vector into a binary syntax tree.The text notes that bottom-up reconstruction is an alternative left for future exploration.Source: §13 · p. 6

Right-skewness bias

• A linear bias term adjusts syntactic distances to make induced trees moderately right-skewed, reflecting the observed right-skew of ASTs.The authors treat unchanged unsupervised-analysis performance after bias injection as evidence that the model captures this bias to some extent.Source: §13 · p. 6

Tree similarity

• Tree similarity is measured by converting gold and induced trees into sets of intermediate-node leaf pairs and calculating their F1 score.Figure 5 provides a toy example of the gold-standard AST and induced-tree comparison.Source: §13 · p. 6, Figure 5 · p. 6

5.1 Experimental Design

• The experiments examine 12-layer, 12-head CodeBERT and GraphCodeBERT models, each with 768-dimensional layer representations, using CodeSearchNet data for Python, Java, and PHP.Source: §5.1 · p. 7

5.2 RQ1: Attention Analysis

• Attention heads vary between position- and content-focused patterns, with the most AST-aligned example focusing on content.Other illustrated heads focus on the next-token position or disperse attention roughly evenly.Source: Figure 7 · p. 8, §5.2 · p. 8
• 67.25% for CodeBERT and 59% for GraphCodeBERT are the highest reported concentrations of high-confidence attention on AST motifs, both in deep layers.The maxima occur at Layer 11, Head 1 for CodeBERT and Layer 12, Head 9 for GraphCodeBERT.Source: §5.2 · p. 8

5.3 RQ2: Probing on Word Embedding

• 84% (CodeBERT) and 86% (GraphCodeBERT) are the fifth-layer Python peaks; GraphCodeBERT scores higher at every layer across Python, Java, and PHP.The layerwise metric is Spearman correlation between predicted and gold-standard AST distances.Source: Figure 9 · p. 9, §5.3 · p. 9

5.4 RQ3: Syntax Tree Induction

• Right-branching is the strongest tree-induction baseline, while bias injection reduces Assignment performance, possibly because some subtrees are not right-skewed.The reported F1 scores cover six common intermediate-node types in Python ASTs.Source: §5.4 · pp. 9–10

5.3 RQ2: Probing on Word Embedding

• Transformer-layer representations outperform CodeBERT-0 in Python probing, and syntax structure is reported as preserved across different hidden layers.CodeBERT-0 uses word embeddings before the Transformer layers; the heatmaps compare gold-standard and predicted distances.Source: §5.3 · p. 9, Figure 10 · p. 10

5.4 RQ3: Syntax Tree Induction

• Pre-trained code models significantly outperform the tree-induction baselines without bias injection, and right-skew bias raises induction performance by up to 5%.The comparison covers Python syntax-tree induction.Source: §5.4 · p. 9, Table 3 · p. 10
• Parsing from attention distributions performs better than parsing from contextual word representations in the reported tree-induction experiments.The paper reports the best results among tested distance functions for each Transformer layer.Source: §5.4 · p. 10

6.1 Observed Findings

• Attention aligns with AST structure, more strongly in deeper layers, while some position-based heads do not use textual context.The authors suggest removing those heads might not affect final results and could reduce model parameters.Source: §6.1 · p. 10
• Hidden layers encode global syntactic distances between word pairs, and CodeBERT can induce syntax trees to a certain extent.Figure 11 presents a CodeBERT syntax-tree-induction case study for a Python snippet.Source: §6.1 · p. 10, Figure 11 · p. 11

6.2 Limitations and Future Work

• The analysis covers AST structure, which may be only one aspect of source code relevant to pre-trained models’ performance.The authors identify code tokens, control-flow graphs, and data-flow graphs as other aspects for future investigation, and analyze only CodeBERT and GraphCodeBERT among self-supervised models.Source: §6.2 · p. 11
• The structural probing is simple, and the tree-induction algorithm is a top-down recursive binary method with a right-skewness bias designed for natural-language constituency trees.The authors note that ASTs are more complex than the induced tree and that the grammar-induction algorithm has room for improvement.Source: §6.2 · p. 11

• Embedding studies examine contextual information in pretrained representations [10] [26] and probe sentence embeddings for linguistic properties [6] [23].Source: §7 · p. 11
• BERT syntax and semantic knowledge has been studied through edge probing [38], attention-based dependency extraction, and structural probing of word representations.Source: §7 · p. 11
• Attention work analyzes individual BERT heads [21], while exBERT supports flexible, interactive investigation of model-internal reasoning.Source: §7 · p. 11
• This paper adapts interpretation techniques from NLP to examine feature correlations captured by pre-trained code models.Source: §7 · p. 11

• The analysis aligns self-attention with syntax, probes whether Transformer contextual representations capture code syntax, and tests syntax-tree induction without training.Source: §8 · p. 11
• The analysis reports findings intended to inspire future studies on code representation learning.Source: §8 · p. 11
• The experimental data and source code are to be integrated into the open-source NaturalCC toolkit.Source: §8 · p. 11