Pre-training Code Representation with Semantic Flow Graph for Effective Bug Localization

1 INTRODUCTION

• The paper addresses inadequate deep code semantics, limited contrastive negative samples, and ignored lexical similarity in BERT-based bug localization.These limitations affect code representation and the matching of bug reports with changesets.Source: §1 · p. 2
• Semantic Flow Graph is a directed, multiple-label representation capturing data flow, control flow, element types, and computational roles.It is designed to compactly and adequately capture code semantics.Source: §1 · p. 2
• SemanticCodeBERT uses SFG and novel pre-training tasks to learn code representations for downstream code-related tasks.The model supplements masked language modeling with structure-aware objectives.Source: §1 · p. 2
• HMCBL combines memory-bank momentum contrastive learning with hierarchical contrastive loss to model large-scale negatives and semantic and lexical similarity.The memory bank stores changesets from different batches, while hierarchical loss uses similarities at different levels.Source: §1 · p. 2
• The evaluation uses 450,000 Java functions for pre-training and a six-project Java dataset for changeset-based bug localization.Ablation studies report improvements from SFG and advantages over existing bug localization architectures.Source: §1 · p. 2
• The study reports state-of-the-art bug localization performance and a large-scale evaluation of the proposed method.The contributions include the SFG representation, SemanticCodeBERT, HMCBL, and empirical evaluation.Source: §1 · p. 2

2 RELATED WORKS

• Changeset-based bug localization is increasingly used because simultaneously changed code parts are related and can facilitate bug fixing.Information-retrieval methods match bug reports with code elements or changesets.Source: §2 · p. 2
• VSM-based methods rely on exact term matching, which is weakened by lexical gaps between bug reports and developer naming practices.Neural methods instead seek semantic relatedness between reports and software artifacts.Source: §2 · pp. 2–3
• Deep neural bug localization methods include transfer-learning and unsupervised approaches for cross-project or changeset-based retrieval.Examples include TRANP-CNN, COOBA, and Bug2Commit [56].Source: §2 · p. 3
• BERT-based changeset retrieval has achieved impressive results, but FBL-BERT does not adequately capture deep program semantics and retains two stated limitations.Its design choices include changeset granularity and special tokens for semantic representation.Source: §2 · p. 3
• Pre-trained code models such as CodeBERT, GraphCodeBERT, CuBERT, C-BERT, TreeBERT, and UniXcoder use token, graph, tree, or cross-modal information.The paper argues that existing models either treat code as token sequences or capture only shallow structure.Source: §2 · p. 3
• SFG and SemanticCodeBERT are proposed to capture code semantics more completely than token-sequence or data-flow-only representations.The related-work discussion identifies this limitation as the motivation for the paper's representation and pre-training design.Source: §2 · p. 3

3 SEMANTIC FLOW GRAPH

• SFG addresses the limitation of existing graph representations by encoding which program elements and operations are related, distinguishing otherwise identical data-flow patterns.For example, it differentiates a function-call computation from a Boolean conjunction by incorporating variable types and operations.Source: §3 · pp. 3–4
• SFG is defined as a tuple of nodes, directed edges, type mappings, and computation-role mappings.Nodes represent variables and control instructions; edges encode data, control, or sequential computation flow.Source: §3 · p. 4
• Its node and edge design captures data dependencies, control dependencies, and sequential computation flow at fine granularity.Variable nodes map one-to-one to variables, while control instructions may map to condition, branch, and convergence nodes.Source: §3 · p. 4
• Type and role labels encode what program elements are related and through which operations they participate in computation.For example, If-Then-Else control nodes receive distinct types, while variables can receive roles such as Assigned, Assignment, InvocationTarget, and InvocationArgument.Source: §3 · pp. 4–5
• SFG remains compact by excluding additional program-element nodes while retaining adequate semantic information for learning across programs.The representation is implemented for Java methods with 55 node types and 43 variable roles, supporting Java versions up to Java 16.Source: §3 · p. 5

4 SEMANTICCODEBERT

• SemanticCodeBERT adapts GraphCodeBERT’s architecture to SFG and adds pre-training tasks tailored to the graph representation.BERT provides the backbone, while the model processes code together with its semantic graph information.Source: §4 · p. 5, §4.1 · p. 5

4.1 Model Architecture

• The model concatenates comments, source code, SFG nodes, node types, and node roles into one input sequence.The architecture defines 55 possible types and 43 possible computation roles for the graph nodes.Source: §4.1 · p. 5

4.2 Masked Attention

• Graph-guided masked attention restricts Transformer information flow to source-node alignment, node dependencies, type links, and role links.Unrelated token pairs receive an infinitely negative attention score and are blocked from transmitting signals.Source: §4.2 · p. 6

4.3 Pre-training Tasks

• Besides masked language modeling and node alignment, SemanticCodeBERT pre-trains on edge prediction, type prediction, and role prediction.These tasks respectively encourage learning graph structure, node attributes, and computation-role correlations.Source: §4.3 · p. 6
• The auxiliary objectives mask graph relations to predict node origins, graph edges, and node types, and incorporate node roles; edge and type prediction each randomly mask 20% of relevant edges.Source: §4.3 · pp. 6–7

5.1 Problem Definition

• Bug localization learns a similarity function that ranks bug-inducing changesets above non-bug-inducing changesets for each bug report.Source: §5.1 · p. 7

5.2 Representation learning

• BERT encodes bug reports while SemanticCodeBERT encodes relevant and irrelevant changesets into feature vectors for representation learning.Source: §5.2 · p. 7
• A projector maps report and changeset features into a shared embedding space, using batch normalization instead of dropout for regularization.Source: §5.2 · p. 7
• A momentum-updated key model and memory bank provide large-scale, slowly evolving negative changeset representations for contrastive learning.The query model is updated by back-propagation, while the key model is momentum-updated; the experiment uses momentum 0.999 and enqueues new negatives while removing old ones.Source: §5.2 · p. 7

5.3 Similarity Estimation

• Hierarchical contrastive learning combines feature-level, model-level, and bank-level cosine similarities to capture lexical, semantic, and broad negative-sample relationships.InfoNCE losses are defined for the three levels and combined with balancing hyperparameters.Source: §5.3 · pp. 7–8

5.4 Offline Indexing And Retrieval

• Offline retrieval stores encoded changesets in an IVFPQ index, searches the nearest partition, and reranks only the top-N′ candidates.Source: §5.4 · p. 8

6.1 Dataset

• The dataset covers six Java projects and evaluates commit-, file-, and hunk-level bug-inducing changesets.File- and hunk-level instances are derived by dividing bug-inducing changesets into smaller code changes.Source: §6.1 · p. 8

6.2 Evaluation Metrics

• Precision@K, MAP, and MRR evaluate top-K relevance, recovery of all relevant changesets, and the rank of the first relevant changeset, respectively.These metrics assess complementary retrieval behaviors for bug reports and changesets.Source: §6.2 · pp. 8–9

6.3 Experimental Setup

• The experiments use Java CodeSearchNet pre-training, fixed training and test splits by bug-opening date, three changeset encodings, and filtered code sequences.SemanticCodeBERT is initialized from GraphCodeBERT and trained for 600K batches, while changesets use D-, ARC-, or ARCL-encoding.Source: §6.3 · p. 9

6.4 Retrieval Performance

• 140.78% to 188.79% MRR improvement over FBL-BERT was reported across six projects at commit-level granularity, with gains significant at p < 0.01.The method also consistently outperformed GraphCodeBERT and UniXcoder in almost all configurations.Source: §6.4 · p. 9, Table 2 · p. 10

6.5 Ablation Study

• Adding Type and Role Prediction universally improves commits-level performance, indicating that node attributes are vital for learning code representations.The ablation compares Node Alignment, Edge Prediction, Type Prediction, and Role Prediction pre-training tasks.Source: §6.5 · p. 10, Table 3 · p. 10
• HMCBL improves MRR for more than 80% of projects by 15.48% to 66.67% and generalizes across advanced pre-training encoders, including BERT, GraphCodeBERT, and SemanticCodeBERT.Its hierarchical momentum contrastive learning provides large-scale negative-sample interactions for representation learning.Source: §6.5 · pp. 10–11

6.6 Threats to Validity

• SZZ noise and tangled changes may make bug-report–changeset mappings imprecise, although manual validation and widespread dataset use reduce these concerns.The authors note that completely removing tangled changes is extraordinarily difficult.Source: §6.6 · p. 11
• External validity is limited because evaluation covers a limited number of bugs from several open-source projects.The projects vary in purpose and development style, and the dataset is widely used for changeset-based bug localization.Source: §6.6 · p. 11

• The paper identifies inadequate code-semantic modeling, limited contrastive negative samples, and ignored lexical similarity as key weaknesses in existing BERT-based bug localization.Source: §7 · p. 11
• Semantic Flow Graph (SFG) is a directed, multiple-label code graph designed to compactly and adequately capture code semantics.Source: §7 · p. 11
• The proposed system combines SFG-based SemanticCodeBERT with Hierarchical Momentum Contrastive Bug Localization (HMCBL).Source: §7 · p. 11
• Evaluation results confirm that the proposed method achieves state-of-the-art performance in bug localization.Source: §7 · p. 11