A LLM Assisted Exploitation of AI-Guardian

• The study uses AI-Guardian, a recent IEEE S&P defense, as a case study of GPT-4’s value as a human-directed research assistant.Source: §1 · p. 1
• GPT-4 implemented the attack from human guidance, often following imprecise instructions; brief exchanges or manual bug fixes corrected issues.All code for the paper was written by GPT-4 following human guidance.Source: §1 · pp. 1–2
• AI-Guardian’s claimed 98% robustness fell to 8% under the original paper’s threat model, and its authors acknowledged the break succeeded.Source: §1 · p. 1
• The paper discusses warning signs that AI-Guardian might be vulnerable and the researchers’ experience using GPT-4 for research.It identifies the defense’s claim of robustness to nearly unbounded adversarial examples as one warning sign.Source: §1 · p. 1

2.1 Adversarial Examples

• Adversarial examples imperceptibly alter inputs to cause misclassification [23]; gradient-based attacks refine perturbations using model gradients [10] [17], including targeted PGD within an ϵ-bounded ℓ∞ neighborhood [17].Targeted PGD seeks an example x′ in that neighborhood that minimizes the loss between the model’s prediction and target label y′.Source: §2.1 · pp. 2–3

2.2 Defending against adversarial examples

• Prior defenses are often bypassed [9] [1] [26], while certified defenses [15] [13] and adversarial training [17] often reduce clean accuracy; AI-Guardian claims robustness without that tradeoff.AI-Guardian relies on a secret transformation t: the defense assumes adversarial examples crafted without knowledge of t will fail, but the paper argues that this secret can be recovered.Source: §2.2 · p. 3

2.3 Large Language Models

• LLMs show strong language and functional-code generation capabilities [3] [12] [16], motivating study of GPT-4 as a research assistant for adversarial-robustness research.The paper focuses on GPT-4 because it appeared especially capable of implementing the attacks.Source: §2.3 · p. 3

3 Threat Model

• On MNIST, the L∞ distortion bound is 0.25, a standard choice [17] and the bound considered by the original paper.An attack need only break one of AI-Guardian’s claimed robustness guarantees to show the scheme is insecure.Source: §3 · p. 4
• The attacker has gradient access to the base model f and oracle access to defended-model logits, but does not know the secret transformation t.The defended model’s logits are those of the model incorporating t.Source: §3 · p. 4
• The threat model is reasonable because the defender may publish f while keeping t secret, and protection applies only when inputs are preprocessed as f(t(x)).Users can still use f(x) normally, but without the protection mechanism.Source: §3 · p. 4

The AI-Guardian Defense

• AI-Guardian constructs a binary mask M ∈ {0, 1}^W×H×C and pattern Z ∈ [0, 1]^W×H, then permutes the class labels with P.W, H, and C denote width, height, and channel dimensions.Source: §4 · pp. 4–5
• The model is trained to predict y on original examples and P[y] on transformed inputs t(x) = M·x + (1−M)·Z.The transformation applies M to x and fills the unmasked regions with Z.Source: §4 · p. 5
• At inference, AI-Guardian predicts with g(x) = P^-1[f(t(x))], applying the inverse permutation to restore the original class order.The transformed input is passed through the defended model before its output is reordered.Source: §4 · p. 5
• AI-Guardian adds an adversarial-training phase using adversarial examples generated on a previous version of the model.The stated purpose is to further enhance robustness against adversarial attacks.Source: §4 · p. 5

5.1 Mask recovery

• Using WH + 1 queries, the mask attack compares logits for the zero image and each basis vector; unchanged outputs identify masked pixels.The evaluation recovered masks perfectly on all 100 models with different random masks.Source: §5.1 · p. 5

5 Breaking AI-Guardian

• All three components were recovered perfectly across 100 randomly configured models each: masks, permutations, and patterns rounded to the nearest integer.The pattern recovery evaluation reports exact recovery after rounding to the nearest integer.Source: §5.1 · p. 5, §5.2 · p. 6, §5.3 · p. 7

5.2 Permutation recovery

• The permutation attack uses 10-step PGD across source-target class pairs to build a difficulty matrix, then assigns each source its easiest target.It exploits the model’s tendency to produce the permuted label for adversarially perturbed inputs; it requires access only to the original model, not the defended model.Source: §5.2 · p. 6

5.3 Pattern recovery

• Given the recovered mask and permutation, gradient descent on Z′ recovers the pattern from a few examples, reusing prior queries so no new queries are needed.The examples need only make the system overdetermined; on MNIST, 78 queries suffice, and earlier queries can be reused.Source: §5.3 · pp. 6–7

• Above 90% targeted attack success at ℓ∞ distortion ε = 0.25 reduced clean accuracy to 0% across 100 models with different random masks, patterns, and model choices.Source: §6 · p. 7
• 92% targeted attack success at ℓ∞ distortion ε = 0.25 reduced the official AI-Guardian models’ accuracy to 0%.Source: §6 · p. 7

7.1 Why this defense?

• The paper’s claims that replacing the central 90% yields under 15% success and ℓ∞ distortion 2.0 stays below 30% were exceptionally unlikely or impossible, the authors argue.At distortion 2.0, an adversary could replace an image with any other image.Source: §7.1 · p. 7
• On MNIST, AutoPGD success fell from 20% at distortion 0.75 to under 10% at 1.0; Figure 4 also shows other cases where increasing budgets lowered success [8].Source: §7.1 · p. 7
• The authors did not carry out the attacks, judging them so insufficient that they would not succeed even under almost infinite distortion; this is their stated reason for not doing so.Source: §7.1 · p. 7
• The evaluation omitted attacks that ensemble over random keys, attempts to extract the secret through queries, and a defined transformation keyspace.Without a mechanical procedure for sampling keys, the authors say formal robustness evaluation is difficult.Source: §7.1 · p. 7
• The authors call for better community processes to prevent papers with major evaluation oversights from being accepted at top conferences.Source: §7.1 · p. 8

7.2 On “AI”-Assisted Research

• GPT-4 implemented attack code from prompts in under a minute, drawing on familiar algorithms and handling boilerplate, though the author says manual coding would have been faster.The authors characterize its strengths as calculator-like rather than those of a human coauthor; its errors occurred in complicated algorithmic code.Source: §7.2 · p. 8
• GPT-4 required operator domain knowledge, had fixed knowledge and limited generalization, failed to ask clarifying questions, and repeatedly made the same errors after correction.Source: §7.2 · pp. 8–9
• Current models can implement code for novel research when tasks are precisely defined and require little human insight; future models may handle poorly defined tasks and devise unfamiliar algorithms.The authors note that fully automated coding, debugging, experimentation, and reporting are not prerequisites for present utility.Source: §7.2 · p. 9

Interaction overview

• The authors say the transcript’s false starts arose from their own bad attack ideas, and that documenting intermediate tasks can illuminate GPT-4 use and attack-design methodology.They report rewriting under-specified questions three times and never rolling back the conversation to hide failed attempts.Source: Appendix A · p. 12

Attack progression

• GPT-4’s mask-extraction code worked initially, but bad requested improvements were unhelpful; backdoor recovery failed until permutation recovery was recognized as necessary.The authors say the failed recovery code ran correctly, and later describe successful permutation recovery followed by a successful, expensive gradient-free backdoor-recovery attempt and a less-effective zero-query attack omitted for clarity.Source: Appendix A · p. 12
• After a gradient-free backdoor-recovery attack required hundreds of thousands of queries, the authors implemented the paper’s more query-efficient attack.The appendix says the gradient-free attack was query-expensive; it provides no quantitative query count for the later method in this part.Source: Appendix A · p. 12

PGD evaluation

• For an initial PGD evaluation, the authors corrected GPT-4’s attack direction and added the specified step size and CUDA changes.The request tested targeted attacks against both base and defended models, while noting that white-box access to the defended model’s mask or permutation was unfair.Source: Appendix A · p. 14

Mask extraction

• The authors’ query-based mask attack compares defended outputs after setting each pixel to zero versus one; GPT-4 proposed quadtree subdivision to reduce queries for sparse masks.The per-pixel method infers a mask value of one when outputs are nearly identical; the quadtree approach recursively splits regions when outputs differ.Source: Appendix A · p. 14
• The mask is estimated recursively by comparing model outputs and splitting regions into four segments; recursive 28×28 results must be sliced into the corresponding mask segments.For small masks, the comparison uses model.forward_base and model.forward_defended outputs, treating near-equal outputs (within 1e-4) as evidence for a mask value of 1.Source: Appendix A · p. 15

Backdoor-pattern recovery

• For each pixel where the mask is 0, the recovery routine tests all 256 values from 0 to 1 and selects the value minimizing the absolute difference between output L2 norms.The mask has shape (1, 28, 28), so the pixel-location indexing must use two indices.Source: Appendix A · pp. 15–16

Iterative refinement

• The code iteratively refines the pattern from an initial guess of 0.5, updating masked-pixel values to minimize the difference between base and defended outputs.A separate gradient-estimation approach perturbs a batch along random directions and keeps adjustments that reduce the L2 output difference.Source: Appendix A · p. 16

Permutation recovery

• Permutation recovery is proposed using at least 128 same-label training examples and gradient descent on a pattern Z to maximize incorrect predictions.The proposed loss is negative cross-entropy between predicted labels and the examples’ true label.Source: Appendix A · p. 17

Appendix A: Complete LLM Code Interaction

• The revised recovery function searches for a pattern that maps examples from an original label to a target label by maximizing negative cross-entropy.The earlier simplified recovery approach may fail because the optimization can be non-convex with multiple local optima.Source: Appendix A · p. 17
• The procedure tests all 10 × 10 source-target pairs using recover_permutation, chooses the permutation with lowest loss, and produces a length-10 mapping that allows fixed points.Source: Appendix A · pp. 17–18
• The transfer attack uses PGD with random noise on border pixels to disturb the added pattern, then evaluates adversarial examples on the defended model.The code also introduces EoT-style PGD, applying a different border-noise transformation at each iteration to seek examples robust to that noise.Source: Appendix A · pp. 18–19
• The targeted attack selects a different random target class for each input and should count success only when the defended model outputs that target.The interaction also modifies the targeted attack’s loss and returns an attack success rate.Source: Appendix A · p. 19
• The conversion code requires matching TensorFlow and PyTorch layer structures and permuting convolution weights; it does not cover complex architectures such as residual connections.TensorFlow convolution weights use height-width-input-output ordering, while PyTorch expects output-input-height-width ordering.Source: Appendix A · pp. 20–21
• The weight-loading corrections handle Conv2d and Linear modules separately, transposing Linear weights because TensorFlow and PyTorch use opposite feature ordering.The interaction notes that other layer types may require specific handling and that conversion changes can depend on the architecture and TensorFlow version.Source: Appendix A · p. 21
• The proposed pattern-recovery attack computes a backdoored image from guessed mask and permutation, runs the model, and compares its output with defended logits using a loss.The interaction frames this as a more efficient way to solve for the guessed pattern.Source: Appendix A · p. 21
• The routine uses SGD to minimize mean squared error between predictions on backdoored images and target logits y_defended by adjusting a randomly initialized guess_pattern.The number of epochs and learning rate are configurable.Source: Appendix A · p. 21
• Running the function requires a CUDA-enabled GPU and PyTorch, with the model and data moved to the GPU before calling it.Source: Appendix A · p. 21
• The author clarified that the permutation is over the model’s output labels and that the pattern should be clipped between -.5 and .5 at each step.Source: Appendix A · p. 21
• The interaction also requests that the final loss be printed.Source: Appendix A · p. 21