Source-linked AI summary
Generating Natural Adversarial Examples
Zhengli Zhao, Dheeru Dua, Sameer Singh
TL;DR
Existing adversarial perturbations can reveal classifier failures but are often unnatural, semantically weak, and difficult to use in language. The paper searches around latent representations learned by generative models to generate natural adversaries for black-box classifiers, which are reported as legible, grammatical, semantically close, and useful for interpretation and robustness evaluation.
Problem
Existing adversarial perturbations are often unnatural, not semantically meaningful, and difficult to apply to language, limiting their usefulness for analyzing black-box classifier behavior.
Method
The framework searches near an input’s latent representation and maps perturbed dense vectors through generative models to produce adversarial instances on the learned data manifold.
Results
Across image and text domains, the approach generates adversaries that are legible, grammatical, semantically close to inputs, and useful for interpreting and evaluating black-box classifiers.
Takeaways & Limitations
Natural adversaries support robustness evaluation and interpretation of black-box classifiers, including without labeled training data.
Takeaways & Limitations
The framework’s example quality depends directly on GAN capabilities, while adversary search is computationally expensive and requires an accurate inverter mapping.
Abstract
from arXiv · showhide
Due to their complex nature, it is hard to characterize the ways in which machine learning models can misbehave or be exploited when deployed. Recent work on adversarial examples, i.e. inputs with minor perturbations that result in substantially different model predictions, is helpful in evaluating the robustness of these models by exposing the adversarial scenarios where they fail. However, these malicious perturbations are often unnatural, not semantically meaningful, and not applicable to complicated domains such as language. In this paper, we propose a framework to generate natural and legible adversarial examples that lie on the data manifold, by searching in semantic space of dense and continuous data representation, utilizing the recent advances in generative adversarial networks. We present generated adversaries to demonstrate the potential of the proposed approach for black-box classifiers for a wide range of applications such as image classification, textual entailment, and machine translation. We include experiments to show that the generated adversaries are natural, legible to humans, and useful in evaluating and analyzing black-box classifiers.
1 INTRODUCTION
Existing adversarial examples expose model vulnerabilities but are often unnatural and poorly suited to understanding deployed behavior or complex domains such as language. The paper proposes natural adversaries generated through latent representations, producing interpretable changes across image and text applications.
- Motivation: Adversarial examples can expose vulnerabilities in machine-learning models, but worst-case perturbations are often unnatural and unlike inputs encountered during deployment.This limits insight into black-box decision behavior and robustness to natural data variations.
- Motivation: Unnatural input-space perturbations are difficult to apply to language because preserving grammar and semantic similarity is challenging.The paper also notes that meaningful changes such as lighting variations may evade conventional adversarial approaches.
- Approach: The framework generates natural adversarial examples that are meaningfully similar, valid or legible, and helpful for interpretation.It targets adversaries that remain on the data manifold rather than relying on random-looking noise.
- Approach: The method searches near an instance’s latent representation in a dense continuous space learned with generative adversarial networks.Perturbations are sampled around the corresponding representation, with the search range recursively tightened.
- Illustration: Compared with FGSM noise, generated digit adversaries look handwritten and reveal meaningful stroke changes that can fool the classifier.The example shows that thickening one stroke and thinning another changes the prediction while retaining visual similarity.
- Evaluation: Experiments and human evaluation apply the approach to image classification, textual entailment, and machine translation for assessing black-box robustness without labeled training data.The generated adversaries are described as natural, grammatical, semantically close to the input, and useful for interpreting local model behavior.
2 FRAMEWORK FOR GENERATING NATURAL ADVERSARIES
The framework generates adversarial examples by searching near an input’s latent representation and decoding candidate perturbations through a generative model. It combines GAN-based manifold modeling, inversion, and black-box search to find natural samples that change classifier predictions.
- Problem setup: The goal is to find the nearest instance on the data manifold that changes a black-box classifier’s prediction.The target adversary x* satisfies f(x*) ≠ f(x), while remaining close to x in the manifold defined by the data distribution.
- Latent-space modeling: A GAN learns to map dense latent vectors to domain samples, while an inverter maps data instances back to latent representations.The inverter is trained using reconstruction error and divergence between sampled latent vectors and reconstructed latent codes.
- Adversary generation: The method perturbs the inverted representation z′ near x and decodes candidate vectors through the generator to query the classifier.Searching in a dense latent space encourages generated adversaries to remain valid, legible, or grammatical.
- Image examples: The framework produces MNIST adversaries against RF and LeNet, with Table 1 comparing them against FGSM adversaries against LeNet.The table shows original test images, corresponding generated adversaries, and classifier predictions.
- Search algorithms: Hybrid shrinking search uses a coarse-to-fine range restriction with denser sampling, achieving a 4× speedup with similar results to iterative stochastic search.Both methods are sample-based, require no classifier gradients, and are applicable to black-box classifiers.
3 ILLUSTRATIVE EXAMPLES
The framework generates natural adversaries across image classification, textual entailment, and machine translation by searching continuous representations rather than directly modifying inputs. Examples show legible image changes, coherent sentence perturbations, and translation probes that expose model behavior.
- The approach applies to image classification, textual entailment, and machine translation, generating adversaries for black-box models.The framework is demonstrated across visual and textual domains.
- Image classification: Image adversaries are generated for MNIST and LSUN, with LSUN examples covering Church-versus-Tower classification.The LSUN classifier is an MLP trained on 64×64 images from the two categories.
- Image classification: For LSUN, Church-to-Tower changes sharpen roofs, narrow buildings, or transform trees into towers, while reverse changes make tower images more church-like.These transformations are presented as natural domain-specific changes.
- Textual entailment: The method searches continuous text representations trained to produce semantically and syntactically coherent sentences, avoiding direct discrete input edits.An ARAE encodes text into continuous codes, while an inverter maps them into Gaussian latent space.
- Textual entailment: Textual entailment adversaries perturb hypotheses while keeping premises unchanged, targeting classifiers with different architectures.The task distinguishes entailment, contradiction, and neutrality between premise–hypothesis pairs.
- Machine translation: Machine-translation adversaries use probing functions to introduce a target German word or detect a dropped verb in English-to-German translation.The probes define adversarial success through properties of the translation rather than class predictions.
- Machine translation: The translation probes suggest that an English gerund is often dropped by Google’s English-to-German system.This is presented as a vulnerability identified through generated adversaries.
4 EXPERIMENTS
Experiments evaluate whether natural adversaries can compare, interpret, and assess black-box models without labeled training data. Statistical analyses and pilot human studies find that adversary distance tracks classifier accuracy and that generated changes are natural or grammatical.
- The framework evaluates and interprets black-box classifiers on image and text data using statistical analyses and pilot user studies.The evaluation is designed to work without labeled training data.
- Robustness of Black-box Classifiers: More accurate classifiers require larger adversarial changes, measured by latent-space distance and by how often their adversaries have the highest ∆z.This pattern is reported for MNIST and textual entailment.
- Robustness of Black-box Classifiers: Varying neural-network size, dropout, regularization, layer sizes, and training data produces a strong correlation between test accuracy and average adversarial ∆z across 80 classifiers.On MNIST, more complex classifiers yield adversaries increasingly different from the original input.
- Human Evaluation: Human subjects judged generated MNIST adversaries natural and found RF adversaries closer to originals than LeNet adversaries.The study used 20 images and 13 responses per question.
- Human Evaluation: 78% of the time, subjects judged the proposed MNIST adversaries more natural than FGSM adversaries against LeNet.FGSM was not applicable to RF in this comparison.
- Human Evaluation: Pilot evaluation of textual-entailment adversaries found the generated sentences grammatical and consistent with the earlier evaluation results.The study presented 20 premise–hypothesis pairs and collected four responses per question.
5 RELATED WORK
Prior adversarial attacks often rely on gradients and produce uninterpretable perturbations, while this framework targets natural, interpretable adversaries for black-box models across image and text domains.
- Gradient-based attacks: Gradient-based attacks such as FGSM and JSMA modify inputs using gradients, while adversarial examples may also transfer across models.FGSM shifts inputs along a cost-minimizing direction; JSMA greedily changes important features until classification changes.
- Black-box settings: Existing attacks are often tied to differentiable classifiers with parameter access, whereas this method targets black-box models including Random Forests.The method does not require gradients and is designed for classifiers whose parameters are unavailable.
- Interpretability: The proposed natural adversaries provide interpretable insights into classifier decision behavior, unlike the uninterpretable noise produced by prior methods.The paper illustrates how localized stroke changes can reveal what alters a prediction.
- Text domains: Text adversarial-example research has received less attention because discrete inputs complicate perturbation, while rule-based substitutions and erasure can harm text integrity.The cited text methods use predefined substitutions or representation erasure rather than the paper’s natural-adversary framework.
6 DISCUSSION AND FUTURE WORK
The framework’s generated-example quality depends on GAN behavior, and its search procedure remains computationally expensive despite an efficiency improvement.
- Limitations: Generated-adversary quality depends directly on the capabilities of the GANs used as generative models.The authors note that GAN training can be brittle and that generator, critic, and inverter capacities must be balanced to avoid model collapse.
- Efficiency: The iterative stochastic search is computationally expensive because it uses naive sampling and local search.A hybrid coarse-to-fine shrinking search yields around 4× speedup with similar results.
7 CONCLUSIONS
The paper presents a GAN-based framework for generating natural adversaries against black-box classifiers in visual and textual domains. These adversaries are legible, grammatical, meaningfully similar, and useful for interpreting and evaluating models without labeled training data.
- Contribution: The framework generates natural adversaries against black-box classifiers in both visual and textual domains.Applications include image classification, textual entailment, and machine translation.
- Generated examples: The generated adversaries are legible, grammatical, and meaningfully similar to their inputs.These properties are reported across the paper’s visual and textual applications.
- Uses: Natural adversaries help interpret decision behavior and evaluate black-box classifier accuracy without labeled training data.The paper frames these uses as applications of the proposed adversarial examples.
A ILLUSTRATION WITH SYNTHETIC DATA
In the synthetic-data illustration, the framework maps inputs into latent space and searches there for a closer, manifold-consistent adversary rather than relying on input-space proximity.
- Latent representation: The inverter maps an input x to a latent dense vector z′, and the generator reconstructs x from that representation.The mappings are expressed as Iγ(x) and Gθ(Iγ(x)).
- Synthetic comparison: For the toy classifier, the latent-space adversary is the closest semantic point along the data-manifold curve, unlike the gradient-based input-space alternative.The illustration contrasts the left-side manifold point with a point directly above the input.
- Adversarial search: The search iteratively expands the latent-space range until generated samples change the black-box classifier’s prediction.A coarse-to-fine strategy first locates broad upper bounds and then tightens the search range.
C ARCHITECTURE FOR CONTINUOUS IMAGES
For continuous images, the framework learns a generator and inverter, then searches around an input in latent space for the closest adversarial sample generated on the data manifold.
- Model and training: The framework trains a WGAN generator and an inverter on relevant data before searching for adversaries.For MNIST, the WGAN uses a 64-dimensional latent space with transposed convolutional generator layers and a critic.
- Manifold-based generation: Because candidates are produced by the generator, the method seeks natural adversaries on the learned manifold rather than arbitrary off-manifold perturbations.Figure 5 contrasts the proposed manifold-based adversary with an existing adversarial sample outside the manifold.
- Adversarial search: The iterative stochastic search expands its radius until it finds a label-changing sample, then returns the candidate closest to z′.After finding an adversary, the search radius is reset to that candidate’s latent distance and the process continues refining the result.
- Adversarial search: Given an input x, it obtains its classifier label y and latent code z′, then perturbs z′ with random noise vectors.Each perturbation is decoded into a sample and evaluated by the black-box classifier.
- Adversarial search: The hybrid shrinking search recursively narrows the latent search interval before applying iterative refinement to locate a nearby adversary.When no adversary is found, the lower bound is moved inward; when one is found, the upper bound is updated to its latent distance.
D ARCHITECTURE FOR DISCRETE TEXT
For discrete text, the framework combines ARAE-based continuous representations with generator and inverter networks, then evaluates automatically generated perturbations on short SNLI sentences.
- Text representation: The text model uses ARAE to encode discrete sentences into continuous codes and decode continuous codes into discrete text.An LSTM encoder produces continuous representations, while a decoder reconstructs sentences.
- Text representation: An inverter maps continuous codes into a 100-dimensional Gaussian latent space, while MLPs learn mappings between noise and continuous codes.The text encoder uses four CNN layers and produces 300-dimensional continuous codes; the decoder is a single-layer LSTM.
- Evaluation: On SNLI sentences up to length 10, the approach uses Δr = 0.01 and N = 100 and generates perturbations described as grammatical and semantically close.Table 9 provides automatically generated examples of these text perturbations.
D.1 TEXTUAL ENTAILMENT EXAMPLES
Additional textual-entailment examples show generated adversarial hypotheses for premise–hypothesis pairs evaluated with three classifiers.
- Examples: The supplementary examples use premise–hypothesis pairs from the SNLI corpus.
- Examples: The examples generate adversarial hypotheses by perturbing the hypothesis while keeping the premise fixed.
- Examples: The examples correspond to the textual-entailment cases presented in the main text.
D.2 MACHINE TRANSLATION EXAMPLES
The machine-translation examples probe black-box translation behavior by perturbing English sentences and examining changed German outputs, including inserted or dropped words.
- Supplementary examples: The supplementary tables provide additional examples corresponding to the main-text probing functions and textual-entailment examples.
- Translation probes: The examples perturb English sentences to create translation adversaries for a Google translation system.
- Translation probes: One probing function introduces the word “stehen” into the translation system’s behavior.
- Translation probes: Another probing function searches for dropped verbs in English-to-German translation.
- Translation probes: The tables compare each original sentence with its adversary and show the corresponding translations with English glosses.