PaperlayerView exact revision on arXiv

Source-linked AI summary

Type-IV Code Clone Detection via Layer-Wise Non-Contrastive Representation Learning

Luciano Marchezan, Kevin Delcourt, Eugene Syriani, Houari Sahraoui

arXiv:2609.17338v1cs.SEcs.LG

TL;DR

Type-IV clone detection must capture behavioral equivalence despite substantial syntactic variation, while contrastive methods require difficult negative sampling. The paper proposes LWVIC4Code, which applies VICReg across transformer layers with cross-layer consistency and depth-dependent weighting. It reports strong multilingual performance, competitive or superior results without negative samples, and practical efficiency relative to zero-shot LLMs.

  • Problem

    Type-IV clones are semantically equivalent but syntactically diverse, and contrastive detection methods require difficult negative sampling that can introduce bias and complexity.

  • Method

    LWVIC4Code adapts VICReg to code and jointly applies layer-wise objectives with cross-layer consistency regularization and depth-dependent layer weighting.

  • Results

    LWVIC4Code achieves strong performance across Python, Java, and C# datasets, outperforming a contrastive baseline while requiring no negative samples.

  • Takeaways & Limitations

    Non-contrastive, layer-wise representation learning provides a promising approach for robust semantic code clone detection across programming languages.

  • Takeaways & Limitations

    Non-contrastive methods require substantial high-quality data, and transfer is less effective to syntactically divergent languages such as C.

Abstract

from arXiv · show

Software clones are fragments of code that are similar or functionally equivalent to each other. They pose significant challenges for maintenance, refactoring, and bug detection. Detecting Type-IV clones, which are semantically equivalent but may differ syntactically, is particularly difficult for traditional token- or syntax-based methods. Recent machine learning approaches rely on contrastive learning, which requires careful negative sampling and can introduce bias. In this paper, we propose LWVIC4Code, a non-contrastive representation learning approach specifically designed for Type-IV clone detection. Building on the Variance-Invariance-Covariance Regularization (VICReg) framework and prior layer-wise VICReg training, LWVIC4Code introduces cross-layer consistency regularization and depth-dependent layer weighting to progressively refine semantic information across transformer layers, producing robust and discriminative code representations. We conduct an empirical study comparing LWVIC4Code against a contrastive learning baseline and zero-shot large language models on Python (Kamino) and multi-language (GPTCloneBench) datasets. Results show that LWVIC4Code achieves competitive or superior performance without negative samples, benefits from layer-wise supervision, and generalizes effectively from Python to other languages, particularly Java and C#. These results demonstrate that non-contrastive, layer-wise representation learning is a promising direction for robust semantic code clone detection.

I. INTRODUCTION

Type-IV clone detection requires semantic representations because functionally equivalent code can differ substantially in syntax. LWVIC4Code addresses this with non-contrastive, layer-wise VICReg training and performs strongly across languages and datasets.

  • Motivation: Type-IV clones are functionally equivalent fragments that may differ entirely in syntax or structure, limiting traditional token- and syntax-based detection.Differences may involve control-flow strategies, library usage, or algorithmic approaches.
  • Approach: LWVIC4Code applies VICReg objectives across transformer layers and adds cross-layer consistency regularization with depth-dependent layer weighting.These components progressively refine semantic information and produce robust, discriminative code embeddings.
  • Empirical evaluation: The evaluation compares LWVIC4Code with CodeBERTCL, standard VICReg, and zero-shot LLMs on Python and multilingual clone datasets.The study also examines layer-wise supervision, component ablations, and transfer from Python to other languages.
  • Empirical evaluation: LWVIC4Code reaches F1 scores of 0.977 on C#, 0.968 on Java, and 0.920 on Python, outperforming CodeBERTCL while avoiding negative sampling.On GPTCloneBench, corresponding MCC values are 0.957, 0.938, and 0.840; CodeBERTCL obtains F1 0.899 and MCC 0.794 on C#.
  • Approach: The approach adapts VICReg to code, learning semantic embeddings without negative samples through invariance, variance, and covariance regularization.VICReg encourages equivalent views to align while preserving feature variability and diversity.

II. BACKGROUND

Type-IV clones are behaviorally equivalent despite potentially unrelated syntax, making them difficult for surface-based methods to detect. The background motivates non-contrastive representation learning as an efficient alternative to biased negative sampling and resource-intensive LLM inference.

  • A. Challenges and Motivation: Type-IV clones may use nearly unrelated implementations, such as nested loops versus map for the same matrix row-sum function.This behavioral equivalence can be difficult for traditional text- or token-based techniques to recognize.
  • A. Challenges and Motivation: Contrastive clone detection depends on carefully selected negatives, because overly dissimilar or syntactically similar negatives can encourage trivial cues, bias, or overfitting.Negative construction also increases training complexity.
  • A. Challenges and Motivation: Non-contrastive learning avoids negative pairs by constraining the embedding space, while VICReg jointly promotes invariance, variance, and covariance.These properties are suited to semantic equivalence across syntactically different code.
  • A. Challenges and Motivation: Existing datasets often lack balanced negatives or explicit semantic-equivalence labels, limiting their suitability for Type-IV clone detection.GPTCloneBench targets Type-IV clones across Python, Java, C#, and C but remains relatively small at approximately 25k pairs.
  • A. Challenges and Motivation: Real-time development use favors lightweight embedding models because LLM-based clone detection is resource-intensive and harder to integrate into IDEs.This practical constraint motivates efficient representation-learning approaches.

B. Code Embeddings for Clone Detection

Code embeddings represent structural and semantic properties of code and can support clone decisions through similarity thresholds. Finetuning improves task performance but commonly relies on difficult-to-construct negative examples for Type-IV clones.

  • Code embeddings: Pretrained models such as CodeBERT and CodeT5 generate embeddings that capture structural and semantic code patterns for downstream clone detection.Given code fragments, these representations can be compared to measure similarity.
  • Clone decisions: A threshold θ is applied to embedding similarity to determine whether a code pair constitutes a clone.The threshold is typically selected empirically using metrics such as F1 or MCC across candidate values.
  • Finetuning: Finetuning pretrained embeddings on labeled clone datasets can use single-language or multilingual data, but usually depends on labeled negatives.For Type-IV clones, constructing reliable negatives may introduce bias or degrade performance.

III. NON-CONTRASTIVE REPRESENTATION LEARNING FOR TYPE-IV CLONE DETECTION

VIC4Code adapts VICReg for Type-IV clone detection by learning semantic code embeddings without contrastive negative pairs. LWVIC4Code extends this design with layer-wise supervision, cross-layer consistency, and depth-dependent weighting.

  • VIC4Code: VIC4Code uses non-contrastive representation learning to capture semantic similarity between functionally equivalent code fragments.It builds on VICReg rather than contrastive objectives.
  • VIC4Code: The VICReg objective combines invariance, variance, and covariance constraints on projected embeddings.The encoder produces latent embeddings, and a learnable projector maps them into the optimization space.
  • VIC4Code: VIC4Code trains on paired code fragments, encodes and projects both fragments, then backpropagates the VICReg loss to update the model.Algorithm 1 initializes the encoder, projector, and optimizer before processing batches across epochs.
  • LWVIC4Code: LWVIC4Code applies VICReg objectives across transformer depths instead of only at the final layer.This is motivated by hierarchical transformer representations in which deeper layers encode higher-level semantics.
  • LWVIC4Code: LWVIC4Code adds cross-layer consistency and layer-wise weighting while training end-to-end with joint encoder updates.Its architecture also uses masked mean pooling at every transformer layer and layer-specific projectors.

1) Optimizing representations across layers:

LWVIC4Code obtains representations at every transformer layer, regularizes their semantic continuity, and jointly optimizes the resulting objectives end-to-end.

  • 1) Optimizing representations across layers:: LWVIC4Code uses masked mean pooling at each transformer layer, excluding padding while retaining information from valid code tokens.Each pooled representation is passed through a layer-specific projector.
  • 1) Optimizing representations across layers:: The layer-wise VICReg objective enforces invariance, variance, and covariance constraints separately at each depth.These constraints align semantically similar fragments, prevent collapse, and preserve complementary features.
  • 2) Enforcing cross-layer consistency:: Cross-layer consistency encourages deeper representations to refine earlier semantic information without making adjacent layers identical.The term promotes semantic continuity across the transformer hierarchy.
  • 2) Enforcing cross-layer consistency:: The cross-layer term is computed for both views of a clone pair and averaged using an L2 consistency objective.L2 constrains both direction and magnitude of representation changes between adjacent layers.
  • 2) Enforcing cross-layer consistency:: Training accumulates layer-specific and cross-layer losses, then backpropagates the total objective through the encoder and layer-specific projectors.The stop-gradient operation restricts direct updates through the previous-layer representation in the cross-layer term.

3) Applying layer-wise weighting:

LWVIC4Code combines layer-wise VICReg and cross-layer consistency losses using depth-dependent weights, while VICReg regularization preserves representation diversity.

  • 3) Applying layer-wise weighting:: The total layer-wise loss combines each layer’s VICReg loss with its cross-layer consistency term under a depth-dependent weight.The weighting is defined as w_l = (l/L)^2, emphasizing deeper layers, while α controls consistency strength.
  • 3) Applying layer-wise weighting:: Variance and covariance regularization counterbalance cross-layer attraction by maintaining activation variability and complementary features.This combination encourages semantic continuity without representation collapse.
  • 3) Applying layer-wise weighting:: The VICReg-based approach is less sensitive to batch composition than contrastive methods because variance and covariance regularization discourage collapse.The passage attributes this robustness to the structure of the regularization terms.

IV. EMPIRICAL EVALUATION

The empirical evaluation tests LWVIC4Code against contrastive learning and zero-shot LLMs, examines its layer-wise components, and measures cross-language transfer from Python training.

  • IV. EMPIRICAL EVALUATION: The evaluation compares LWVIC4Code with CodeBERTCL and zero-shot LLMs across Type-IV clone detection tasks.It examines performance without negative samples, layer-wise supervision, component contributions, and multilingual generalization.
  • A. Evaluation Protocol: The contrastive baseline requires positive and negative code pairs, whereas LWVIC4Code is evaluated as a non-contrastive alternative.CodeBERTCL learns by reducing clone-pair distance while enforcing a margin between non-clones.
  • A. Evaluation Protocol: Performance is assessed with precision, recall, F1, MCC, AUC, threshold analysis, and statistical significance procedures.Similarity thresholds are selected over explored ranges, while GEE logistic regression and clustered bootstrap account for repeated or dependent measurements.
  • RQ2–RQ3: The layer-wise study compares LWVIC4Code with VIC4Code under identical training conditions to test whether supervision across depths improves semantic representations.Ablations remove final-layer exclusivity, cross-layer consistency, or layer-wise weighting while retaining other settings.
  • RQ4: The LLM comparison uses fixed prompting without fine-tuning for deepseek-r1:14b, gpt-oss:20b, and qwen3.5:9.65b.The protocol focuses on publicly accessible models representative of current practice.
  • RQ5: The multilingual transfer study trains LWVIC4Code only on Python Kamino clones and evaluates it on Java, C#, and C GPTCloneBench clones.It evaluates MCC and F1 across cosine-similarity thresholds to assess sensitivity to threshold choice.

B. Datasets

The study evaluates Type-IV clone detection on Kamino, a large Python dataset, and GPTCloneBench, a multi-language dataset spanning Python, Java, C#, and C.

  • Kamino contains 78,771 Type-IV clone pairs of semantically equivalent Python fragments generated through validated, syntactically diverse transformations.The dataset uses unit-test execution, CodeBLEU filtering, clustering, and an 80/20 train-test split.
  • GPTCloneBench provides Type-IV clone pairs across Python, Java, C#, and C for evaluating cross-language generalization.The study uses 3,948 Python, 9,935 Java, 6,872 C#, and 4,823 C pairs.

C. Results

LWVIC4Code consistently outperforms CodeBERTCL and generally surpasses zero-shot LLMs, while layer-wise weighting provides the most consistent ablation benefit and some variants show dataset-dependent behavior.

  • C. Results: LWVIC4Code outperforms CodeBERTCL across datasets and languages, improving GPTCloneBench AUC from 0.8881 to 0.9864 and Kamino AUC from 0.9326 to 0.9751.It also uses only positive clone pairs during training; the corresponding gains reach 0.297 in F1 and 0.333 in MCC.
  • C. Results: LWVIC4Code improves over standard VIC4Code on GPTCloneBench, with higher F1 and MCC across languages and AUC of 0.9864 versus 0.9815.On Kamino, VIC4Code is marginally better in AUC, 0.9775 versus 0.9751, while both remain effective.
  • C. Results: Layer-wise weighting provides the most consistent ablation benefit, with its removal reducing AUC from 0.9864 to 0.9701 on GPTCloneBench and from 0.9751 to 0.9481 on Kamino.Removing cross-layer consistency also significantly affects Kamino, but its AUC difference is smaller, 0.9751 versus 0.9684.
  • C. Results: The last-layer-only variant achieves higher GPTCloneBench AUC, 0.9916 versus 0.9864, but lower Kamino AUC, 0.9751 versus 0.9840, indicating dataset-dependent behavior.It also yields higher clone-pair median similarities and lower variance, favoring higher similarity thresholds, whereas the full model is more robust across datasets and thresholds.
  • C. Results: LWVIC4Code generally outperforms zero-shot LLMs across languages, with up to 0.316 MCC improvement on GPTCloneBench, while DeepSeek-r1 surpasses it on Kamino.The LLM performance gap is smaller for DeepSeek-r1, whereas GPT-OSS and Qwen remain below LWVIC4Code; differences are statistically significant after correction.

5) RQ5: Transferability of Representations in Python:

Python-trained LWVIC4Code transfers effectively to Java and C#, while performance declines on the more syntactically divergent C language. The model’s layer-wise design and positive-only training support strong results across thresholds, but data diversity and threshold selection remain practical constraints.

  • Threshold sensitivity: F1 peaks at 0.925 for C# within θ ∈[0.6, 0.7], while C reaches only slightly above 0.7 F1 and approximately 0.6 MCC.For Java and C#, performance remains relatively stable up to θ = 0.75 before declining.
  • Transferability: Python-trained LWVIC4Code generalizes strongly to Java and C#, but its effectiveness decreases on C as the target language diverges further from Python.The paper attributes the gap to greater syntactic and paradigmatic differences between C and Python.
  • Transferability: LWVIC4Code achieves high GPTCloneBench scores for Java and C# despite relatively small language-specific training sets, including F1/MCC values of 0.968/0.938 and 0.977/0.957.The cited training sets contain 4,257 Java pairs and 2,944 C# pairs.
  • Model analysis: Layer-wise weighting and cross-layer consistency materially support performance, whereas last-layer-only representations are competitive mainly at stricter similarity thresholds.Removing layer-wise weighting consistently reduces performance, and removing cross-layer consistency causes significant degradation on Kamino.
  • Limitations: Non-contrastive training can require large, diverse positive-pair sets, and C performance illustrates lower scores and a bias toward high similarity thresholds.On C, F1 and MCC are 0.866 and 0.790, respectively, with θ = 0.9 needed to maintain precision.
  • Practical implications: LWVIC4Code is positioned for efficient deployment because clone detection over approximately 44k GPTCloneBench pairs takes less than 10 minutes, versus dozens of hours for zero-shot LLMs.The approach can also be incrementally fine-tuned on new positive examples without generating negative samples.

B. Threats to Validity

The threats analysis identifies limitations involving implementation and preprocessing choices, uncertain external generalization, imperfect validity measures, and small language-specific samples. The authors mitigate these concerns through consistent protocols, complementary datasets, threshold-wide evaluation, cross-language testing, and statistical controls.

  • Internal validity: Performance differences across contrastive, non-contrastive, and LLM approaches may reflect implementation details, hyperparameters, or preprocessing, although the study uses consistent settings and a tuned CodeBERT baseline.The mitigation strategy standardizes training settings, architectures, and evaluation protocols.
  • External validity: Results may not fully generalize to other codebases or programming languages despite evaluation on Python-only and multilingual datasets.The datasets include Kamino and GPTCloneBench, providing complementary language and data settings.
  • Construct validity: MCC, F1, and similarity thresholds may not capture every aspect of semantic equivalence, so the study evaluates all possible thresholds rather than one random θ.The authors also use nonoverlapping train-test splits and cross-dataset transfer evaluation to reduce similarity-related bias.
  • Conclusion validity: Small sample sizes in some language subsets, such as C, may reduce the reliability of the reported results.The study supplements MCC and F1 with GEE, bootstrap confidence intervals for AUC differences, McNemar’s test, and Benjamini–Hochberg correction.

A. AST and PDG-Based Approaches

Semantic clone detection research spans AST/PDG representations, graph and probabilistic transformations, contrastive embeddings, visual transfer, ensembles, and LLMs. The paper positions LWVIC4Code as a distinct non-contrastive approach that learns semantic code representations across transformer layers.

  • A. AST and PDG-Based Approaches: AST- and PDG-based methods incorporate structural and dependency information to detect complex Type-III and Type-IV clones, outperforming traditional metric-based approaches in reported studies.Other structural methods transform ASTs into Markov-chain or graph-based representations for semantic embedding learning.
  • Contrastive Representation Learning: Contrastive approaches can improve generalization and learn language-agnostic semantic representations, but they depend on carefully constructed positive and negative pairs.This limitation motivates examining non-contrastive training for code clone detection.
  • Alternative Approaches: Prior work also includes visual transfer and ensemble verification, but these methods either transform code into images or rely on engineered features rather than directly learning semantic code representations.The ensemble approach verifies candidate pairs instead of learning general-purpose semantic embeddings.
  • LLM-Based Approaches: LLMs capture contextual code information but have shown difficulty reliably identifying functional equivalence across languages, supporting comparison with dedicated embedding models.The cited literature reports mixed results, including strong generalization in some settings and struggles with cross-language equivalence.
  • LWVIC4Code: LWVIC4Code adapts VICReg to Type-IV clone detection by enforcing layer-wise consistency across transformer layers and learning embeddings without negative samples.The paper presents this as a new model class and systematically compares it with other machine-learning strategies.
  • Future Directions: Future work includes hybrid contrastive/non-contrastive training, broader languages and datasets, adaptive thresholds, and continual learning for scalable deployment.These directions target data scarcity, representation collapse, generalizability, and production adaptation.
Loading 2609.17338v1…