PaperlayerView exact revision on arXiv

Source-linked AI summary

Vulnerable Code Search: Transferable Attack for Code Language Models

Kaicheng Wang, Liyan Huang, Jesse Thomason, Weihang Wang

arXiv:2608.26031v1cs.SEcs.CR

TL;DR

Code-search CLMs can be manipulated through small, functionality-preserving identifier changes, raising concerns about reliable retrieval. The paper introduces a transferable surrogate-model attack that aligns irrelevant code with target queries and reports severe degradation across embedding and LLM-based retrieval systems. The findings indicate that current models rely heavily on lexical cues, while white-box access and computation constrain the direct attack framework.

  • Problem

    Code-search CLMs are susceptible to adversarial modifications of non-functional textual elements, threatening reliable code retrieval.

  • Method

    The attack replaces identifiers without altering functionality, optimizing similarity on a smaller white-box surrogate before transferring the adversarial snippets to other models.

  • Results

    Up to 77% absolute MRR degradation occurred across all tested models on CosQA, while attacks transferred across embedding models and LLM-based repository retrieval.

  • Takeaways & Limitations

    High standard benchmark scores can mask reliance on lexical cues rather than robust semantic understanding in code search models.

  • Takeaways & Limitations

    The proposed attack requires white-box model access and sufficient computation, prerequisites that may not hold for large-scale or proprietary black-box systems.

Abstract

from arXiv · show

Reliable code retrieval is crucial for developer productivity and effective code reuse. However, current neural code language models (CLMs) powering search tools are susceptible to adversarial attacks targeting non-functional textual elements. In this paper, we introduce a programming language-agnostic, transferable, adversarial attack that exploits this CLM vulnerability. Our approach perturbs identifiers within a code snippet without altering the snippet's functionality to artificially align the code with a target query. We demonstrate that our attack, even when computed using smaller code embedding models, such as CodeT5+, is highly effective and transferable to larger, closed-source embedding models, like Voyage-code-3, or LLMs like Gemini-3.1-Pro. Our attack can increase the similarity between the query and arbitrary, irrelevant code snippets, consequently degrading key retrieval metrics such as the Mean Reciprocal Rank (MRR) of state-of-the-art models by up to 77%. The experimental results highlight the fragility of current code search methods and underscore the need for more robust, semantic-aware approaches.

1 Introduction

Embedding-based code retrieval supports scalable search but remains vulnerable to functionality-preserving identifier changes that make irrelevant snippets resemble target queries. The proposed transferable attack exploits this weakness across models, producing severe retrieval degradation and exposing reliance on lexical rather than semantic cues.

  • Embedding-based retrieval encodes queries and code into a shared latent space to rank candidates by vector similarity.
  • Functionality-preserving adversarial examples can drastically change code embeddings, enabling query-specific attacks that crowd out legitimate or relevant results.
  • The attack replaces identifiers to maximize similarity with a target query and transfers from CodeT5+ to larger or closed-source models across five programming languages.
  • 77% absolute drop in Mean Reciprocal Rank (MRR) was observed across all tested models on CosQA.
  • High benchmark scores do not necessarily reflect true semantic understanding because current models can rely on brittle lexical features.

2 Related Work

Related work develops increasingly capable code-search representations, incorporates program structure and contrastive learning, and studies adversarial transformations under different attacker knowledge assumptions.

  • Code-search research progressed from CodeSearchNet and joint natural-language–code representations toward increasingly capable code language models.
  • Structural approaches use data-flow graphs or Abstract Syntax Trees, while contrastive learning became a dominant training technique.
  • Other code-search strategies adapt general embeddings, finetune LLMs, train on augmented data, or use black-box embedding services.
  • Programming languages enable adversarial attacks through semantic-preserving transformations, whereas discrete natural-language tokens make small valid perturbations difficult.
  • Attack strategies differ by attacker knowledge, with white-box methods requiring access to model gradients.

3 Adversarial Attack for Code Search

The attack optimizes identifier replacements against a surrogate model while preserving code structure and naming style, then transfers the resulting adversarial snippet to inaccessible victim models. Its formulation combines gradient influence with query-token similarity under practical indexing and access constraints.

  • Threat model: The attack aims to modify a target snippet so it ranks within the top-k results for a natural-language query while preserving execution logic.
  • Threat model: The attacker uses white-box access to a surrogate embedding model for similarity scores and gradients, while the victim model may differ.
  • Threat model: The attack targets corpus indexing, where injected or modified code can influence future retrieval after its adversarial variant is indexed.
  • Functionality preservation: AST preservation, consistent renaming, and distinct replacements constrain modifications so execution logic remains unchanged.
  • Gradient-guided optimization: Gradient-based optimization selects identifier-compatible tokens whose aggregated influence over token occurrences maximizes the surrogate similarity change.
  • Gradient-guided optimization: Disjoint token-position sets make first-order replacement effects separable, allowing greedy optimization instead of combinatorial search.
  • Identifier style: Style constraints filter replacements to preserve naming conventions such as camelCase and snake_case.
  • Query-token similarity: Query-token similarity augments gradient influence so replacements use vocabulary aligned with the target query; the method sets α = 0.1.

4 Experiments and Results

Experiments show that identifier-based attacks transfer broadly across embedding models, queries, datasets, and LLM retrieval, sharply degrading search while exposing trade-offs among attack components and defenses.

  • 4.1 Effectiveness and Transferability: Over 97% of surrogate-model examples gain query-code similarity, with stronger gains for CodeT5+ than OASIS and larger effects on CosQA than CLARC.Dataset or language characteristics affect attack magnitude.
  • 4.1 Effectiveness and Transferability: Over 95% of surrogate/evaluation pairs transfer similarity gains across robustness-enhanced, finetuned, and closed-source models.High Pearson and Spearman correlations indicate surrogate gains generally predict transfer gains; failed transfers have smaller negative shifts.
  • 4.1 Effectiveness and Transferability: The attack also transfers across semantically equivalent queries, suggesting it targets shared intent rather than only the original query string.CodeT5+-based attacks tend to produce larger similarity changes, whereas OASIS-based attacks yield more consistent correlations.
  • 4.2 Comparison with Baselines: A 5× higher similarity change than CodeAttack accompanies a reduction from ∼10.8M API calls to 1k in the black-box setting.In the white-box setting, Attack Transfer also achieves larger similarity changes than DAMP while using less GPU time.
  • 4.3 Application on Code Search Benchmarks: Replacing 10% of irrelevant CosQA candidates drives MRR to single digits and R@5 below 15% for every evaluation model.Before attack, OASIS, Nomic-embed-code, and Voyage-code-3 report R@5 above 90% and MRR between 74% and 87%.
  • 4.4 Shared Corpus Attack: Injecting 20 adversarial snippets, or 4% of the corpus, sharply reduces MRR and NDCG while largely preserving Recall@5.With 1–2% attacked, the multi-query objective still degrades MRR and NDCG, but sharing snippets across more queries weakens the effect.

5 Conclusion & Future Work

The paper presents a transferable identifier-based adversarial attack that misleads code search models while preserving code functionality. Its cross-model effectiveness exposes reliance on lexical cues and motivates semantics-grounded defenses that preserve retrieval utility.

  • The attack modifies code identifiers to mislead code search models without changing snippet functionality.
  • The attack remains effective across embedding models, retrieval benchmarks, and LLM-based repository retrieval.
  • The results reveal that current CLMs rely heavily on lexical cues rather than robust semantic understanding.
  • Future work should develop defenses that preserve retrieval utility while investigating why attacks transfer across models.

A Limitations

The causes of attack transferability across code embedding models remain unclear, and limited public information about model pretraining prevents a definitive explanation.

  • The underlying causes of transferability across evaluated code embedding models remain unclear.
  • Overlapping pretraining data is one possible explanation, but limited public pretraining details prevent a definitive conclusion.

B Reproducibility

The reproducibility materials report the experimental hardware and attack batch sizes, while identifying the models and datasets used. The attack codebase is not publicly released because of misuse concerns.

  • Experiments used an AMD EPYC Milan 7643 CPU, 1TB RAM, and an NVIDIA L40 Ada 48GB GPU.
  • Attack batch sizes were 10 query-code pairs for CodeT5+ and 4 pairs for OASIS.
  • The attack codebase is not publicly released to mitigate potential misuse, though researchers may contact the first author about implementation details.
  • The experiments used OASIS, Nomic-embed-code, Voyage-code-3, CodeT5+, and the summarized benchmark datasets.

C.1 Model & Dataset License

The paper lists licenses and availability information for the models and datasets used in its experiments, including several open licenses and an unclear license for Voyage-code-3.

  • CodeT5+ is listed under the BSD 3-Clause License.
  • Nomic-embed-code is listed under the Apache-2.0 license.
  • The OASIS-code-embedding-1.5B model is referenced through its Hugging Face repository.
  • Voyage-code-3 has an unclear license, while CosQA is Apache-2.0 and CLARC is CC-BY-SA 4.0.
  • HumanEval-X is listed under the Apache-2.0 license.

D Implementation Details

The attack replaces identifiers while preserving code structure and functionality, then transfers across models, queries, and programming languages. Its effectiveness varies with search-space constraints, embedding density, dataset, and model sensitivity.

  • Attack construction: The method replaces function, variable, macro, and module identifiers to increase similarity between code and a target query.Replacements are selected using gradient information, identifier-style preservation, and query-token similarity.
  • Attack construction: Identifier consistency and no-duplicate constraints preserve coherent replacements for repeated symbols and distinct identifiers.The constraints maintain tokenizer-attached context and ensure the code’s AST structure remains identical.
  • Search space: The candidate vocabulary is typically reduced to 30–40% of the full tokenizer vocabulary by identifier-compatible style constraints.In the worst case, the candidate set can be as large as the full vocabulary.
  • Transferability: Most perturbations increase similarity, with non-positive changes described as almost negligible across model architectures.The passage attributes this pattern to high transferability and consistency across models.
  • Results: Mean ∆Sim. ranges from approximately 0.18 to 0.39 on CosQA and 0.13 to 0.27 on CLARC, indicating larger shifts for Python than C++ snippets.The passage reports consistently larger similarity changes on CosQA than CLARC.
  • Transferability: The attack remains effective for paraphrased queries and transfers across five languages, targeting semantic intent beyond exact query wording.Transfer is particularly strong against Java, JavaScript, and Go, while Python shows notably higher correlation coefficients.

E.7 Different Attack Ratio on Benchmark

The attack remains potent when only a small fraction of a retrieval corpus is manipulated, while degradation approaches saturation as the attack ratio increases.

  • Low attack ratios: 1% poisoning manipulates only 5 of 500 snippets yet causes substantial degradation in white-box and black-box retrieval settings.Recall@1 drops approximately 60% on CodeT5+ and 62.8% on Voyage-code-3.
  • Low attack ratios: 60% Recall@1 drop occurs on CodeT5+ under the 1% attack ratio.The passage describes this as an approximate drop on the surrogate model.
  • Low attack ratios: 62.8% Recall@1 drop occurs on Voyage-code-3 under the same 1% attack ratio.This demonstrates transfer of the vulnerability to a commercial embedding model.
  • Trend and saturation: Degradation at 5% attack rate is nearly equivalent to degradation under the 10% scenario, indicating a saturation effect.The passage states that 5% or less is often sufficient to dominate retrieval.

E.8 Application of the Adversarial Attack on Other Benchmarks

Across additional benchmarks, the attack remains transferable but varies in magnitude: CLARC is more resistant than CosQA, while CodeSearchNet confirms stable transfer behavior. Robust finetuning introduces a trade-off between attack resistance and retrieval quality.

  • CLARC: OASIS suffers a drop of over 68% in Recall@1 on CosQA but roughly 25% on CLARC.The passage attributes the disparity partly to dataset-specific semantic separation and transferability.
  • CLARC: CLARC presents a wider query-ground-truth versus query-irrelevant similarity gap, creating a larger barrier for adversarial snippets.For OASIS, the gap is 31.19 on CLARC versus 21.40 on CosQA.
  • CLARC: Surrogate-generated adversarial examples are less transferable on CLARC and often fail to reach similarity scores that displace the ground truth.This is identified as a transferability limitation of the attack on CLARC.
  • CodeSearchNet: The method remains consistently effective and highly transferable on CodeSearchNet, with correlations aligning closely with those reported elsewhere.The passage characterizes transferability as stable and predictable across datasets.
  • Robust finetuning: Robust-Only FT resists static attacks but reduces all three retrieval metrics to less than half the pretrained baseline.Mixed FT better preserves retrieval performance while mitigating static attacks.
  • Robust finetuning: Standard adversarial finetuning is insufficient because checkpoints either remain vulnerable to white-box attacks or sacrifice retrieval performance.The experiments report a trade-off between robustness and retrieval quality.

E.10 Ablation Studies

The ablations show that selecting the highest-similarity adversarial code is more effective than selecting a fixed iteration, while extending attacks beyond five iterations provides diminishing returns. The study also selects α=0.1 because it consistently yields the highest ΔSimilarity across models, and examples illustrate similarity increases on transfer targets.

  • Attack Code Selection: Selecting the highest-similarity code across iterations produces more effective attacks than selecting the code at a fixed iteration.Fixed-iteration selection plateaus and fluctuates after three iterations, whereas maximum-similarity selection performs better.
  • Iteration Limit: Extending the attack from 5 to 10 iterations increases ΔSimilarity by less than 0.02 across surrogate and transfer models.The additional iterations double computational cost, motivating termination at five iterations.
  • Choice of α: α=0.1 consistently yields the highest ΔSimilarity across CodeT5+, OASIS, and Nomic-embed-code.CodeT5+ scores are comparable at α=0.05 and α=0.1, while transfer targets score higher with larger α values.
  • Qualitative Examples: For a sorting-query example, Nomic-embed-code similarity rises from 0.0689 to 0.5925 after identifier perturbation.The reported increase is ΔSimilarity 0.5236.
  • Attack Code Selection: Figure 4 evaluates average similarity changes on CosQA for CodeT5+ and the transfer targets OASIS and Nomic-embed-code.Selecting the maximum similarity up to iteration k performs better, while extending from 5 to 10 iterations yields only marginal gains.
  • Qualitative Examples: For a string-copy query, Nomic-embed-code similarity reaches 0.5072 after adversarial identifier changes.The reported increase is ΔSimilarity 0.4036.
Loading 2608.26031v1…