PaperlayerView exact revision on arXiv

Source-linked AI summary

Detecting Contaminated Code-Generation Prompt Batches via Influence Functions

Francesco Quinzan, Noor Munir, Yishun Lu, Stephen Roberts

arXiv:2608.14303v1cs.LG

TL;DR

Code-generation assistants can produce insecure implementations, while existing detectors depend on known vulnerability patterns and do not reliably cover novel attack classes. CodeSIFT uses influence functions to detect contaminated prompt batches without assuming a threat model, reaching up to 0.98 AUROC at moderate-to-high contamination.

  • Problem

    No reliable method detects prompts that elicit security-relevant weaknesses in generated code when the vulnerability class is unknown or novel.

  • Method

    CodeSIFT uses an influence-function-based statistical test comparing candidate and benign prompt batches’ parameter-space influence.

  • Results

    Up to 0.98 AUROC, CodeSIFT detected contaminated batches across both datasets and three models, while static baselines were inconsistent across models.

  • Takeaways & Limitations

    The results indicate that parameter-space influence provides a threat-model-agnostic detection signal that transfers across model families better than surface-level pattern matching.

  • Takeaways & Limitations

    The evaluation covers batch-level detection on models up to 7B parameters, Python, and the studied vulnerability families, leaving localization and broader coverage untested.

Abstract

from arXiv · show

Large language models (LLMs) are increasingly used for code generation, yet they remain vulnerable to prompts that elicit insecure implementations. Existing defenses typically rely on predefined threat models or known vulnerability patterns, limiting their effectiveness against novel attacks. We propose CodeSIFT, a threat-model-agnostic detection method that leverages influence functions to identify batches of prompts that induce anomalous model behavior. Rather than detecting specific vulnerabilities, CodeSIFT measures the parameter-space influence of generated code and uses a statistical test to determine whether a candidate prompt set deviates from a benign reference distribution. To evaluate our approach, we introduce two benchmark datasets covering a variety of vulnerabilities. We evaluate CodeSIFT on three open-weight code LLMs ranging from 3B to 7B parameters, achieving AUROC scores of up to 0.98 at moderate-to-high injection rates, while maintaining well-calibrated false positive rates and substantially outperforming static analysis baselines. These results suggest that influence-function-based detection is a promising direction for identifying malicious code-generation prompts without requiring prior knowledge of the underlying attack class.

1 Introduction

CodeSIFT addresses the lack of threat-model-agnostic methods for detecting prompts that induce security-relevant misalignment in code generation. It uses influence functions to test whether prompt batches deviate from benign model behavior, and is evaluated on two new datasets against static-analysis baselines.

  • Motivation: Code-generation assistants can produce vulnerable code in developer workflows where failures carry security and financial consequences.They may write authentication logic, database queries, and infrastructure code with little human review.
  • Limitations: Existing static analyzers, taint trackers, signatures, and rules are limited to known vulnerability patterns or threat models.They may miss variants outside their catalogs and do not directly target prompts eliciting flawed implementation choices.
  • Problem: No reliable method previously detected prompts eliciting code-generation misalignment when the vulnerability class is unknown or novel.Misalignment is defined as introducing a CWE/OWASP security-relevant weakness that a benign completion of the same task would not.
  • Approach: CodeSIFT proposes an influence-function statistical test that aggregates deviations from intended model behavior without assuming a threat model.The method characterizes benign internal behavior and flags incoming prompt sets that depart from that norm.
  • Evaluation: 200 prompts each comprise the AuthSec and InfraCloud datasets, paired with a benign reference set for evaluating malicious-prompt detectors.AuthSec covers authentication, authorization, and session security; InfraCloud covers infrastructure, cloud, and protocol exploitation.
  • Results: AUROC up to 0.98 is achieved against Bandit, Semgrep, and an AST-based taint tracker across coding LLMs of up to seven billion parameters.The baselines are inconsistent across models and are anti-correlated with the true label in several cases.

2 Related work

Prior work documents frequent vulnerabilities in LLM-generated code but does not target distinguishing adversarial prompt batches from benign ones. Influence functions provide a retrospective diagnostic framework whose scalability has been extended to models with up to 52 billion parameters.

  • Security of LLM-generated code: Roughly 40% of GitHub Copilot completions contain exploitable vulnerabilities across a broad set of CWE-relevant scenarios.This motivates detecting insecure code-generation behavior.
  • Security of LLM-generated code: Existing studies are not built to test whether detectors distinguish batches of adversarially crafted prompts from benign ones.This contrasts with the detection objective addressed by the paper.
  • Influence Functions: Influence functions originated in robust statistics and were reintroduced to machine learning for explaining predictions and identifying mislabeled or adversarial training examples.This use is retrospective, diagnosing data the model had already trained on.
  • Influence Functions: 52 billion parameters is the largest model scale reported for influence-score computation using EK-FAC approximations, TF-IDF prefiltering, and query batching.These techniques control gradient-computation costs over large candidate sets.

3 Preliminaries

The paper formalizes stochastic LLM outputs, cross-entropy objectives, and influence responses to upweighting token sequences. It then approximates these responses for large, open-weight models using in-distribution curvature data and computationally efficient K-FAC simplifications.

  • Notation: The model maps token sequences z to stochastic outputs sampled from P(ˆz | z, ˆθ), with deterministic decoding represented as a point mass.The notation uses ˆz = f(z, ˆθ), where ˆθ denotes converged parameters.
  • Notation: Cross-entropy L(ˆθ, z) measures sequence loss, while J(ˆθ) is its empirical average over the training dataset.The token-level expression uses log pˆθ(ˆzt | z<t).
  • Influence responses: Upweighting a sequence z by ε defines a response function whose first-order expansion approximates its local infinitesimal effect on optimized parameters.The linearized response is valid for sufficiently small ε under twice-continuous differentiability of J(θ).
  • Influence responses: The linearized response approximates the Proximal Bregman Response Function, which captures local parameter-and-behavior changes from upweighting z while remaining near the current model.The PBRF separates prediction deviation, the effect of adding z, and parameter deviation from ˆθ.
  • Unknown training datasets and computational issues: Because pretrained training datasets are undisclosed, the method estimates inverse curvature using an in-distribution curvature dataset Dc, implemented with permissively licensed code from The Stack v2.Dc is intended to approximate the population from which the unknown training data were sampled.

4 The Proposed Algorithm

CodeSIFT compares prompt-induced parameter-space influence against a benign reference without assuming a specific threat model. It uses Welch’s one-sided test to identify candidate batches whose average influence is significantly larger than the reference.

  • Setup and influence score: CodeSIFT characterizes benign behavior with reference dataset D∗ and tests whether candidate dataset D departs from those statistics.The candidate batch’s benign or malicious status is unknown.
  • Setup and influence score: The prompt influence score IFp,ε(z) averages the parameter shift required to absorb sampled responses while preserving the model’s training-distribution behavior.Averaging over responses generated for z yields the expected shift induced by that prompt.
  • Influence scores as a detection criterion: A candidate batch is classified as malicious when the test p-value is below the chosen significance threshold α; otherwise, it is classified as benign.The decision corresponds to rejecting H0 in favor of the larger-influence alternative H1.
  • Influence scores as a detection criterion: Misalignment-eliciting prompts are expected to produce gradients poorly explained by H−1 and therefore larger average influence scores than benign prompts.Averaging scores over D summarizes its aggregate influence pressure, while comparison with D∗ isolates excess pressure.
  • Influence scores as a detection criterion: Welch’s one-sided test evaluates whether E[z ∼D] IFp,ε(z) exceeds E[z ∼D∗] IFp,ε(z).Welch’s test is used because the two datasets need not have equal influence-score variance.

5 Dataset Construction

CodeSIFT’s evaluation uses paired malicious and benign prompt pools because contamination detection compares batch influence against a known-clean reference. The dataset comprises manually authored Python completion records organized around taxonomy-based weaknesses and mirrored benign implementation tasks.

  • Dataset rationale: The benchmark requires paired malicious and benign prompt pools rather than individual labeled examples, because detection compares completion influence against a known-clean reference.Existing benchmarks such as CyberSecEval score individual completions for insecure patterns and are not designed for this paired-batch evaluation.
  • Record format: Each record pairs a natural-language completion instruction with a Python skeleton containing one of three completion markers: # TODO:, # TASK:, or # INSTRUCTION:.The marker identifies the region to fill without naming the vulnerability; maliciousness is implied by the surrounding task framing.
  • Provenance: 800 records were authored manually, with original record content and weakness coverage drawn from the CWE catalogue and OWASP Top 10.Malicious records target taxonomy-defined weaknesses, while benign records were written alongside them against the same domain families.
  • Malicious pool: Malicious tasks are grouped into AuthSec and InfraCloud application targets, covering authentication, authorization, sessions, infrastructure, cloud, protocols, and related attacks.The listed coverage includes credential and password handling, JWT/OAuth misuse, CSRF, access-control bypasses, SSRF, cloud-metadata/IAM abuse, HTTP/WebSocket attacks, cache poisoning, request smuggling, and CI/CD or dependency attacks.
  • Benign reference pool: Benign prompt families mirror the malicious domains with ordinary input handling, authentication utilities, and infrastructure, I/O, and networking tasks without a vulnerability objective.Examples include parsing and transformation utilities, password hashing, token verification, cookie configuration, file I/O, HTTP clients, serialization, and small Flask endpoints.

6 Experiments

Experiments evaluate CodeSIFT on two malicious-prompt domains and three open-weight code models against threat-model-agnostic static-analysis baselines. CodeSIFT’s detection improves with contamination while maintaining calibrated false-positive control, including on the harder InfraCloud dataset.

  • Experimental setup: Experiments use 200 AuthSec and 200 InfraCloud malicious prompts, a 400-prompt benign reference pool, and 100 The Stack v2 prompts for curvature estimation.The benign pool is split across the two domains.
  • Experimental setup: The evaluation covers Granite-3B-Code-Instruct-2K, DeepSeek-Coder-6.7B-Instruct, and Qwen2.5-Coder-7B-Instruct.The models have 3B, 6.7B, and 7B parameters, respectively.
  • Baselines: The comparison uses Bandit, Semgrep, and AST Taint Tracker, applying the same t-test to averaged detector flags so only the underlying signal varies.Baselines are general-purpose or custom static-analysis detectors and are not tuned to the benchmark vulnerability classes.
  • Detection accuracy: 0.6 to 0.85 AUROC is achieved by CodeSIFT at the lowest AuthSec injection rate, rising to essentially 1.0 at roughly 40 to 50 percent contamination.CodeSIFT improves monotonically with injection rate across every tested model and dataset combination.
  • Detection accuracy: 0.88 to 0.98 AUROC is reached on InfraCloud, where detection rises more gradually because the dataset is harder and more heterogeneous.The InfraCloud result varies by model.
  • False positive control: 0.01 to 0.09 is the tested nominal significance-level range over which CodeSIFT’s empirical FPR tracks α closely across both datasets and all three models.This indicates calibration without systematic over- or under-rejection.

7 Discussion · Appendix

CodeSIFT detects contaminated prompt batches without assuming a threat model, with AUROC increasing as malicious-prompt contamination rises and reaching up to 0.98 at moderate-to-high contamination while remaining calibrated. Its parameter-space influence signal transfers across model families better than surface-level baselines, but prompt-level localization and evaluation beyond 7B models remain open directions.

  • 7 Discussion: CodeSIFT detects contaminated prompt batches across both datasets and all three tested models without assuming a threat model.
  • 7 Discussion: AUROC rises monotonically with the fraction of malicious prompts and reaches up to 0.98 at moderate-to-high contamination.
  • 7 Discussion: CodeSIFT remains well calibrated throughout the evaluated contamination settings.
  • 7 Discussion: Static baselines are inconsistent across models and are frequently uninformative or worse.
  • 7 Discussion: Parameter-space influence provides a detection signal that transfers across model families beyond surface-level pattern matching.
  • 7 Discussion: CodeSIFT does not require the vulnerability class to be known in advance.
  • Appendix: CodeSIFT currently operates as a batch-level test, leaving localization of individual malicious prompts within a batch as a next step.
  • Appendix: The evaluation covers models up to 7B parameters, leaving broader model scales outside the reported results.

A Missing Proofs

The proof of Proposition A.1 analyzes a convex loss through a quadratic expansion and linearized network output. It then applies stationarity and convexity to obtain a unique minimizer, with λI ensuring invertibility.

  • Proposition A.1: Proposition A.1 assumes that the loss is convex as a function of the network outputs.The proposition’s stated assumption concerns convexity in network outputs, not necessarily network parameters.
  • Proof setup: The proof expands the loss quadratically around the reference activations ˆy(i), which remain fixed for each input even when the output model is stochastic.It separately denotes the activations y(i) produced by θ for the same input.
  • Step 1: The Bregman term reduces to a quadratic form: The Bregman term is reduced to a quadratic form by computing the divergence of the quadratic loss approximation and substituting the linearized output.The proof defines the Bregman divergence using the loss values, gradient, and network output difference.
  • Step 2: Stationarity condition: Convexity implies Gˆθ ⪰0, while adding λI with λ > 0 guarantees invertibility and yields a unique minimizer.The conclusion follows by solving the stationarity equation under the stated positive regularization condition.

B Datasets vulnerability coverage

Table 1 organizes the malicious samples by vulnerability class and associates each class with representative CWE identifiers.

  • Table 1 groups the malicious samples by vulnerability class.
  • Each vulnerability class is represented using CWE IDs.
  • The listed CWE IDs are representative rather than exhaustive.

C Hyperparameters for the Main Experiments · D Additional experimental results

The main experiments use fixed estimation, damping, and significance hyperparameters across models, datasets, and injection rates. Appendix D instead varies candidate-set size, sample count, and injection rate as figure axes.

  • C Hyperparameters for the Main Experiments: Fixed hyperparameters apply across models, datasets, and injection rates unless otherwise noted.Appendix D is the stated exception, reporting sweeps directly as figure axes.
  • D Additional experimental results: Appendix D reports candidate-set size, sample count, and injection rate directly as figure axes.These quantities are swept rather than treated solely as fixed hyperparameters in that appendix.
  • C Hyperparameters for the Main Experiments: ε = 10−3 is used when estimating IFp,ε(z).This value is fixed for the main experiments.
  • C Hyperparameters for the Main Experiments: N = 25 sampled responses per prompt are averaged when estimating IFp,ε(z).The response count is fixed across the main experimental settings.
  • C Hyperparameters for the Main Experiments: K-FAC factors use a base damping coefficient of 0.1 and a per-projection relative damping term of 0.1.A damping floor γ is also applied.
  • C Hyperparameters for the Main Experiments: γ = 10−3 is used for Granite-3B-Code-Instruct-2K and DeepSeek-Coder-6.7B-Instruct, while γ = 10−2 is used for Qwen2.5-Coder-7B-Instruct.The paper attributes this difference to curvature-scale differences across architectures.
  • C Hyperparameters for the Main Experiments: The headline detection results use a fixed significance threshold of α = 0.01.Section 6.5 additionally sweeps the nominal significance level to assess calibration.

D.1 Overview across dataset size and sample count · D.2 Dataset size versus injection rate

CodeSIFT maintains strong detection as candidate-set size and sampled completions vary, reaching near-ceiling AUROC on AuthSec with small sets and few samples. Across both datasets, larger candidate sets and higher injection rates improve CodeSIFT’s AUROC, while L1 and L2 remain nearly identical and Bandit stays near chance.

  • D.1 Overview across dataset size and sample count: CodeSIFT reaches near-ceiling AUROC on AuthSec with only 20–30 prompts in the candidate set.The result is shown across the three evaluated code models.
  • D.1 Overview across dataset size and sample count: Five sampled completions per prompt already suffice for CodeSIFT on AuthSec, whose accuracy is essentially unaffected by additional samples.The figures compare CodeSIFT’s L1 and L2 variants with three static baselines.
  • D.1 Overview across dataset size and sample count: On InfraCloud, CodeSIFT shows the same dataset-size and sample-count pattern as AuthSec, but at a lower AUROC ceiling.The InfraCloud comparisons are reported for all three code models and the same detector set.
  • D.2 Dataset size versus injection rate: CodeSIFT’s AUROC increases with both candidate-set size and injection rate on AuthSec and InfraCloud.Larger candidate sets and higher contamination rates each make detection easier.
  • D.2 Dataset size versus injection rate: The L1 and L2 CodeSIFT variants behave near-identically throughout the dataset-size and injection-rate analyses.This pattern holds across both datasets and all three evaluated models.
  • D.2 Dataset size versus injection rate: Bandit remains largely flat around 0.5 to 0.6 regardless of contamination level or candidate-set size.This indicates no real dependence on either quantity in the reported analyses.
  • D.2 Dataset size versus injection rate: The figures show CodeSIFT increasing monotonically along both axes, whereas static baselines remain largely flat or decline as either quantity grows.This comparison is reported for AuthSec and InfraCloud across the evaluated models.

D.3 Dataset size versus number of sampled completions · D.4 Injection rate versus number of sampled completions

Across both sweeps, CodeSIFT’s AUROC is governed primarily by the tested contamination axis—dataset size or injection rate—while remaining largely insensitive to sampled completions per prompt. This pattern holds across AuthSec and InfraCloud and indicates sample-efficient detection, whereas static baselines show no clear dependence on the swept variables.

  • D.3 Dataset size versus number of sampled completions: D.3 Dataset size versus number of sampled completions: CodeSIFT’s AUROC increases steadily with dataset size while staying essentially flat as sampled completions per prompt increase.Figures 14–19 evaluate AuthSec and InfraCloud across Granite-3B-Code-Instruct-2K, DeepSeek-Coder-6.7B-Instruct, and Qwen2.5-Coder-7B-Instruct.
  • D.3 Dataset size versus number of sampled completions: D.3 Dataset size versus number of sampled completions: Static baselines show no clear dependence on dataset size or the number of sampled completions per prompt.This lack of structure is reported alongside CodeSIFT’s trends in the AuthSec and InfraCloud sweeps.
  • D.3 Dataset size versus number of sampled completions: D.3 Dataset size versus number of sampled completions: The dataset-size dependence and completion insensitivity indicate that CodeSIFT is sample efficient.The result is reported for both benchmark datasets and all three evaluated models.
  • D.4 Injection rate versus number of sampled completions: D.4 Injection rate versus number of sampled completions: CodeSIFT’s AUROC increases steadily as injection rate rises while remaining essentially flat across sampled completions per prompt.Figures 20–25 report the joint sweeps on AuthSec and InfraCloud at fixed dataset size for three models.
  • D.4 Injection rate versus number of sampled completions: D.4 Injection rate versus number of sampled completions: Static baselines again show no clear structure along injection rate or sampled completions per prompt.This comparison is made at a fixed dataset size in Figures 20–25.
Loading 2608.14303v1…