PaperlayerView exact revision on arXiv

Source-linked AI summary

Adversarial Examples for Models of Code

Noam Yefet, Uri Alon, Eran Yahav

arXiv:1910.07517v5cs.LGcs.PL

TL;DR

Neural models of code can be forced into incorrect predictions through semantically preserving program changes, while exhaustive search over equivalent programs is prohibitively expensive. The paper presents DAMP, a gradient-guided white-box attack, and demonstrates its effectiveness across multiple code-model architectures alongside evaluated defenses and their trade-offs.

  • Problem

    The paper examines whether neural models of code can be forced to make adversarially chosen incorrect predictions using semantically equivalent program transformations.

  • Method

    DAMP derives desired predictions with respect to model inputs while holding weights constant, then uses gradient-guided variable renaming and dead-code insertion to perturb code.

  • Results

    DAMP succeeds in targeted and non-targeted attacks across code2vec, GGNN, and GNN-FiLM in Java and C#, outperforming the reported baselines in the evaluated attacks.

  • Takeaways & Limitations

    The evaluated defenses involve trade-offs across performance, robustness, and retraining requirements, while adversarial training is presented as contributing to more robust models.

  • Takeaways & Limitations

    The multi-step gradient-descent attack is not guaranteed to converge, and some adversarial targets may have no perturbation that forces the desired prediction.

Abstract

from arXiv · show

Neural models of code have shown impressive results when performing tasks such as predicting method names and identifying certain kinds of bugs. We show that these models are vulnerable to adversarial examples, and introduce a novel approach for attacking trained models of code using adversarial examples. The main idea of our approach is to force a given trained model to make an incorrect prediction, as specified by the adversary, by introducing small perturbations that do not change the program's semantics, thereby creating an adversarial example. To find such perturbations, we present a new technique for Discrete Adversarial Manipulation of Programs (DAMP). DAMP works by deriving the desired prediction with respect to the model's inputs, while holding the model weights constant, and following the gradients to slightly modify the input code. We show that our DAMP attack is effective across three neural architectures: code2vec, GGNN, and GNN-FiLM, in both Java and C#. Our evaluations demonstrate that DAMP has up to 89% success rate in changing a prediction to the adversary's choice (a targeted attack) and a success rate of up to 94% in changing a given prediction to any incorrect prediction (a non-targeted attack). To defend a model against such attacks, we empirically examine a variety of possible defenses and discuss their trade-offs. We show that some of these defenses can dramatically drop the success rate of the attacker, with a minor penalty of 2% relative degradation in accuracy when they are not performing under attack. Our code, data, and trained models are available at https://github.com/tech-srl/adversarial-examples .

1 INTRODUCTION

Neural models of code perform strongly across many tasks but, like deep models in other domains, can be manipulated into incorrect predictions. The paper introduces DAMP, which searches semantically preserving program modifications using input gradients, and evaluates attacks and defenses across architectures and languages.

  • Neural models of code achieve state-of-the-art performance on tasks including name and type prediction, summarization, generation, search, and bug finding.
  • Adversarial examples modify correctly labeled inputs so trained neural models produce an attacker-specified incorrect label.
  • The attack goal is to find a semantically equivalent program P′ that changes a model’s correct prediction y to an adversarial label y_bad.
  • The program-equivalence search space is exponentially large, making random brute-force exploration prohibitively expensive.
  • DAMP follows gradients of the desired prediction with respect to discrete program inputs while keeping model weights fixed, then selects semantic-preserving modifications such as renaming variables or adding dead code.
  • DAMP attacks three neural architectures in Java and C#, while the paper evaluates defenses that trade original accuracy against attack robustness.

2 OVERVIEW

The overview illustrates how small, semantics-preserving code changes can redirect predictions in code classification and bug-finding models. DAMP guides this discrete search with gradients over input distributions, producing targeted or non-targeted adversarial examples.

  • Bypass Semantic Labeling: In code2vec, renaming array to ttypes changes a sorting snippet’s prediction to the targeted label contains without changing functionality.
  • Bypass Semantic Labeling: Adding unused variable upperhexdigits makes code2vec predict the targeted label escape with 100% probability while preserving the snippet’s functionality.
  • Bypass Bug Detection: In a C# VarMisuse example, renaming destination in another method to scsqbhj makes a GGNN choose the incorrect but type-correct field SourceType.
  • Gradient-Based Exploration of the Program Space: DAMP explores equivalent programs toward a chosen target label for targeted attacks or away from the original label for non-targeted attacks.
  • Gradient-Based Exploration of the Program Space: The approach adapts gradient-based input exploration from continuous settings, where perturbations can directly follow gradients toward or away from labels.
  • Deriving with Respect to a Discrete Input: For discrete inputs, DAMP differentiates with respect to a one-hot distribution over values and uses argmax to select an adversarial discrete value.

3 BACKGROUND

Neural networks use gradient-based optimization to fit labeled data, but adversarial examples exploit vulnerabilities in these models. Discrete domains such as programs make semantic-preserving perturbations and gradient-based attacks especially difficult.

  • Neural network training: Neural networks model inputs as labels using trainable weights fitted to a labeled training set.Training adjusts the weights by optimizing a loss function.
  • Neural network training: Gradient descent repeatedly updates model weights toward lower loss, using a learning rate to control step size.The process may reach a local rather than global minimum.
  • Adversarial examples: Adversarial examples intentionally alter inputs to make trained models produce incorrect predictions.This vulnerability has been demonstrated across neural-network domains including computer vision.
  • Adversarial examples: Discrete inputs such as programs cannot receive imperceptible noise, and differentiating loss with respect to discrete symbols yields zero gradients.These properties distinguish program attacks from continuous-input attacks.
  • Adversarial examples: Natural-language adversarial methods do not directly transfer to code because code has variable reuse, semantic structure, compilation constraints, and readability requirements.These properties make semantic-preserving program perturbations particularly challenging.

4 ADVERSARIAL EXAMPLES FOR MODELS OF CODE

DAMP generates semantic-preserving adversarial examples for code by optimizing model inputs rather than fixed model weights. It supports targeted attacks through loss descent and non-targeted attacks through loss ascent, using discrete variable-name changes or dead-code insertion.

  • Perturbations: DAMP can rename local variables or insert dead code while preserving program semantics.Variable renaming can affect multiple occurrences while retaining compilability.
  • Definitions: Targeted attacks rename a local variable so the model produces a specific incorrect adversarial label.The attacker must select both a variable and an alternative name.
  • DAMP approach: DAMP holds model weights constant and follows gradients with respect to code inputs to select semantic-preserving perturbations.The approach targets neural models of code rather than retraining them.
  • Targeted Attack: For targeted attacks, the method decreases loss for the adversarial label by stepping opposite the gradient with respect to a variable’s one-hot representation.Argmax converts the resulting distribution into a concrete alternative variable name.
  • Non-targeted Attack: For non-targeted attacks, DAMP increases loss for the original label by moving in the gradient direction.The two attack types differ in target label and direction of progress.
  • Convergence: Success is not guaranteed because neural-network losses are generally non-convex and some adversarial targets may have no feasible perturbation.The limitation applies even when using multiple adversarial optimization steps.

5 DEFENSE AGAINST ADVERSARIAL EXAMPLES

The paper evaluates defenses that either preprocess inputs before an existing model or retrain the model for robustness. These approaches trade clean accuracy, training cost, and attack resistance in different ways.

  • Without retraining: Defenses without retraining place an independent gatekeeper before the code model to detect or mask suspicious input changes.Their modular form is f_θ∘g, where g processes inputs before f_θ.
  • Without retraining: No Vars replaces every variable with UNK at test time and is 100% robust by construction, but discards variable-name information.It can be applied to an already-trained model.
  • Without retraining: Outlier Detection replaces a variable with UNK when its L2 distance from other symbols exceeds a validation-tuned threshold.The threshold controls the trade-off between defense effectiveness and clean accuracy.
  • With retraining: Retraining aims to make models robust from the beginning rather than patching a vulnerable model with a separate defense.This approach requires additional model training.
  • With retraining: Train Without Vars removes variable-name reliance during both training and testing, but requires training a model from scratch.It is expected to use other predictive signals instead.
  • With retraining: Adversarial Training jointly minimizes original and adversarial losses using perturbed training examples, increasing training time to about three times slower.The method does not change model complexity.

6 EVALUATION

The evaluation measures DAMP’s ability to change downstream predictions under targeted and non-targeted attacks, then compares defenses that mitigate those attacks. It is not intended as a general robustness evaluation of the attacked models.

  • Evaluation scope: The experiments evaluate DAMP on targeted and non-targeted attacks and assess defensive techniques for mitigating them.The evaluation has separate attack-effectiveness and defense-robustness components.
  • Evaluation scope: The study evaluates defense techniques by their ability to provide robustness to a given model.The authors distinguish this objective from evaluating the attacked models’ intrinsic robustness.
  • Reproducibility: The paper releases its code, data, and trained models for the reported experiments.The materials are available through the project repository.

6.1 Setup

The evaluation attacks three neural code-model architectures across Java and C#, using two semantic-preserving perturbation strategies and gradient-guided search. The setup compares variable renaming and dead-code insertion while controlling search and dataset conditions.

  • Models and datasets: DAMP is evaluated on code2vec, GGNN, and GNN-FiLM, using Java for code2vec and C# for the two GNN models.The downstream models were obtained from their original authors as trained models.
  • Adversarial strategies: The attack focuses on variable renaming and dead-code insertion as semantic-preserving program transformations.Variable renaming changes an existing variable; dead-code insertion adds an unused variable declaration.
  • Adversarial strategies: Other transformations, including statement and operator swapping, are not differentiable and therefore do not enable targeted attacks.The paper consequently restricts its targeted attack strategies to transformations that can be safely applied without semantic analysis.
  • Models and datasets: The code2vec evaluation uses Java-large, containing more than 16M Java methods from 9500 top-starred GitHub projects.The dataset separates 9000 projects for training, 200 for validation, and 300 for testing.
  • Evaluation protocol: The code2vec test subset contains only examples predicted correctly by the original model, giving the filtered set 100% original accuracy by construction.This filtering establishes the starting point for measuring attack-induced changes.
  • Models and datasets: The C# evaluation uses about 220,000 graphs from 29 top-starred GitHub projects, with up to five type-correct variable candidates per example.DAMP modifies an unrelated variable rather than the correct variable or adversarial target.

6.2 Attack

DAMP uses gradient-guided search to target specific labels or any incorrect label, and it generally outperforms the supplied baselines across code2vec, GGNN, and GNN-FiLM. Attack effectiveness varies by architecture, target, and perturbation type.

  • Attack design: DAMP uses Equation (4) for targeted attacks and Equation (5) for non-targeted attacks, with desired labels sampled from training labels occurring at least 10K times.Robustness is the percentage of examples whose correct prediction remains unchanged under the relevant attack definition.
  • Baselines: TFIDF attacks code2vec by selecting the variable name with the highest label-conditioned training frequency for the desired adversarial label.CopyTarget instead renames a variable directly to the desired adversarial label.
  • code2vec results: DAMP outperforms the baselines overall in both targeted and non-targeted code2vec attacks.The table caption reports DAMP as more effective for 6 of 10 VarName targets and 8 of 10 DeadCode targets.
  • code2vec results: 6% robustness is achieved by DAMP for non-targeted code2vec VarName attacks, versus 34.10% for RandomVar and 53.53% for TFIDF.Lower robustness indicates a more effective attack.
  • code2vec results: 10.39% robustness occurs for DAMP targeting mergeFrom, compared with 72.79% for CopyTarget on the same target.DAMP is more effective than CopyTarget for each randomly sampled adversarial label in the reported targeted comparison.
  • Attack behavior: Non-targeted attacks generally produce lower robustness than targeted attacks because any incorrect label counts as success, rather than only the desired label.VarName is generally more effective than DeadCode because renaming affects multiple code occurrences, whereas inserted dead code affects a smaller part.
  • GGNN and GNN-FiLM results: DAMP is more effective than CharBruteForce on the GNN models, with GGNN at 69.00% robustness and GNN-FiLM at 87.62% robustness for targeted attacks.The corresponding baseline robustness values are 98.84% for GGNN and 96.19% for GNN-FiLM.
  • GGNN and GNN-FiLM results: GNN-FiLM is more robust than GGNN despite using 10 gradient steps versus 3, with robustness of 87.62% versus 69.00% for targeted DAMP attacks.The paper hypothesizes that GNN-FiLM’s source-and-target message computation makes attacks on a single node less effective.

6.3 Defense

The defense evaluation compares approaches that trade off robustness against adversarial attacks with performance on undefended inputs. Outlier Detection is the strongest modular defense, while Adversarial Training is strongest among retrained methods.

  • Outlier Detection achieves the best performance and highest robustness among defenses that do not require retraining.It achieves an F1 of 97.02 and above 75.35% robustness for targeted and non-targeted attacks.
  • 2.18% degradation in F1 is the penalty for Outlier Detection compared with No Defense.This makes it a performance–robustness compromise rather than a maximally defensive configuration.
  • 100% robustness is achieved by Train Without Vars, but its F1 is degraded by 9.6% compared with No Defense.No Vars also reaches 100% robustness, while being applicable without retraining and about 10 F1 points worse than Train Without Vars.
  • Reducing vocabulary size to 100K or 50K barely decreases undefended performance but leaves robustness roughly as poor as No Defense.A 10K vocabulary hurts performance without providing much robustness.
  • Outlier Detection’s similarity threshold controls a robustness–performance trade-off.Smaller thresholds improve robustness but reduce F1; larger thresholds improve performance while reducing robustness.

6.4 Additional Examples

Additional examples show that DAMP can steer predictions through variable renaming or dead-code insertion, including transferable perturbations and attacks that introduce real bugs.

  • The paper’s examples and appendix cases can be tested on the original models and linked implementations.The listed resources include code2vec.org and Microsoft’s tf-gnn-samples repository.
  • 99.99% probability changes sort to get, while 86.99% changes sort to indexOf, through targeted variable renaming.The adversarial labels were selected before finding the replacement names.
  • 99.99% probability changes multiple snippets’ predictions to sort after inserting the same unused variable declaration.The shared dead-code patch demonstrates transferability across examples.
  • A GGNN VarMisuse attack renames a local variable in another method and makes GetGetter predict _setterBuilder instead of _getterBuilder.The resulting prediction introduces a real bug in GetGetter.
  • Figure 13 presents a similar targeted GNN adversarial example.The examples extend the demonstrated VarMisuse attack pattern beyond the Figure 12 case.

7 RELATED WORK

Prior work established adversarial examples in images, language, malware, and code robustness, but the paper positions its contribution as targeted attacks on high-level code models.

  • Image attacks exploit small perturbations, while discrete NLP attacks modify characters or replace words with similar words.The cited NLP techniques described here support non-targeted attacks rather than selected adversarial labels.
  • Malware studies perturb binary files to make malicious programs appear benign, but did not perform targeted attacks.The passage also contrasts those approaches with gradient-based targeted attacks.
  • The paper claims to be the first to investigate adversarial attacks for models of high-level code.Earlier work identified robustness problems but did not provide a concrete attack or defense method.
  • Concurrent code-robustness work used semantic-preserving transformations for training but did not use gradients or targeted attacks.The cited comparison distinguishes that training-based direction from this paper’s attack formulation.
  • The Outlier Detection defense is similar in spirit to upstream misspelling-repair defenses in NLP.Both place a preprocessing or recognition component before the downstream model.

8 CONCLUSION

DAMP is presented as a general white-box attack technique demonstrated across major code-model architectures and languages. The paper also evaluates defenses and their trade-offs across performance, robustness, and retraining requirements.

  • DAMP generates targeted and non-targeted adversarial attacks by renaming variables and adding dead code.
  • DAMP works on code2vec, GGNN, and GNN-FiLM models in Java and C#.
  • The technique is general for white-box attacks on code models where gradients can be computed.
  • The authors compare defense techniques across performance, robustness, and whether retraining is required.
  • The paper argues its principles can support further adversarial attacks and defenses, and releases code, data, and trained models publicly.

A APPENDIX - ADDITIONAL EXAMPLES

The appendix gives concrete examples of DAMP manipulating variable names to redirect code2vec predictions and GGNN VarMisuse predictions. These transformations can target arbitrary labels or introduce real variable-selection bugs.

  • code2vec targeted attacks: Renaming txt changes code2vec’s correct escape prediction to the arbitrary targets contains or done.DAMP finds the replacement names expres and claimed.
  • code2vec targeted attacks: Renaming elem changes code2vec’s correct contains prediction to the arbitrary targets escape or load.The replacement names are upperhexdigits and musicservice.
  • code2vec targeted attacks: Renaming array or target changes code2vec’s correct count prediction to sort or contains.DAMP finds orderedlist and thisentry as the required replacement names.
  • GGNN VarMisuse attacks: A GGNN VarMisuse example changes the predicted variable in GetGetter by renaming setteril in another method, introducing a real bug.
  • GGNN VarMisuse attacks: Related GGNN VarMisuse examples redirect predictions in EmitProxy and Map, or introduce a real bug without a specified correct label.The manipulations rename builder, context, or source in another method.
Loading 1910.07517v5…