Adversarial Robustness Limits via Scaling-Law and Human-Alignment Studies

Motivation

• Prior adversarial training reaches only 71% robustness to ℓ∞ attacks on CIFAR10, despite neural networks matching human performance on several vision-and-text tasks.This gap motivates studying why a simple benchmark remains unsolved.Source: §1 · p. 1
• Reaching human performance with unlimited CIFAR10 data would require roughly 10^30 FLOPs, making simple scaling impractical.The estimate corresponds to about 3,000 years of TF32 matrix math on 25,000 MI300 or H100 GPUs.Source: §1 · p. 1

Implications

• The paper concludes that meaningful progress requires more efficient training algorithms and improved architectures rather than simply increasing scale.It separately argues that attack formulations should account for image validity.Source: §1 · p. 1

Motivation

• Around 90% human performance on AutoAttack data is the estimated ceiling because roughly 10% of adversarial images no longer fit their original labels.The paper attributes this error to invalid adversarial data, so benchmarking can make human-level robustness appear unattainable.Source: §1 · p. 1

Contributions

• The paper develops scaling laws for adversarial robustness that incorporate generative-model data quality and predict unseen configurations and resource allocations.The scaling laws also identify opportunities to reduce model size and increase inference speed.Source: §1 · p. 1

Scaling-law analysis

• Figure 1 predicts performance asymptotes near 90% even with arbitrarily large, high-quality synthetic data and infinite compute.The estimate uses a scaling-law approach that models synthetic-data quality through FID.Source: Figure 1 · p. 2

Results

• 73.71% AutoAttack accuracy improves the prior state of the art by 3% using compute-efficient scaling-law-guided configurations.The analysis also reports the same robustness with 20% fewer FLOPs and a 1% gain using a 3× smaller model at the same compute budget.Source: §2 · p. 2

Limitations

• The ℓ∞ attack formulation generates invalid adversarial images that humans also misclassify, preventing benchmarking from reaching a fraction of human-level robustness.The paper therefore argues that attack formulations should account for image validity.Source: §2 · p. 2

Adversarial Training Trends

• Adversarial training commonly uses 10-step attacks, making each iteration 9× more expensive than regular training, while larger models and datasets improve robustness.Synthetic CIFAR10 datasets with up to 50M samples have been used, and this paper’s SOTA model trains on 300M unique synthetic points.Source: §2 · p. 2
• Robustifying foundation models may be difficult or futile because their training costs are already high before adding adversarial defenses.This concern motivates studying the costs and limits of adversarial robustness on CIFAR10.Source: §2 · p. 2

Adversarial Examples That Fool Humans

• Figure 2 presents adversarial images that fool the SOTA neural network for readers to classify and compare with clean counterparts.The figure is used to illustrate potentially invalid data and brittleness in human visual perception.Source: Figure 2 · p. 3

Adversarial Training Trends

• The paper extends prior scaling laws by modeling synthetic-data quality through FID, so predictions change as training-data quality changes.Earlier concurrent studies did not model this effect.Source: §2 · p. 3

Adversarial Examples That Fool Humans

• Robust neural networks can generate adversarial data that fools humans, and such images may remain difficult to classify even with extra time.The related work includes higher-resolution and alternative-norm attacks that also transfer to humans.Source: §2 · p. 3
• Human susceptibility to these attacks can change the correct label, causing robustness benchmarks to frame human-level robustness as unattainable.The paper links this effect to invalid data generated by norm-constrained attacks.Source: §2 · p. 3

3.1. Background and Methodology

• The methodology varies model size, dataset size, and synthetic-data quality, then fits empirical loss estimators using adversarially evaluated models.Source: §3 · p. 3, §3.1 · p. 4

3.2. Scaling-law approaches

• Approach 1 fits compute-optimal model and dataset sizes for one dataset, whereas Approach 2 incorporates synthetic-data quality and Approach 3 models earlier overfitting on lower-quality data.Source: Figure 3 · p. 5, §3.2.3 · p. 6
• Improving data quality lowers minimum loss and moves scaling exponents toward balanced model and data allocation, while all approaches broadly agree that prior SOTA models were too large.Source: §3.2.1 · p. 5, §3.3.1 · p. 7

• 73.71% AutoAttack accuracy was achieved with 93.68% clean accuracy by training WideResNet-94-16 on 500M samples.Source: §3.3.1 · p. 7
• A model trained at the prior SOTA’s FLOPs budget reached 71.59% AutoAttack and 93.11% clean accuracy while using over 3× fewer FLOPs and parameters than the prior SOTA.Source: §3.3.1 · p. 7

4.1. Evaluating Human Performance

• Human accuracy drops by at least 21% for every user from clean to attacked images, supporting a robustness limit below 100%.Average performance on attacked images is 25% lower than on corresponding clean images.Source: §4.1 · p. 9
• 90.46% estimated average human accuracy on all adversarial images closely matches the scaling law’s predicted asymptotic accuracy.This estimate assumes humans correctly classify the 7371 adversarial images classified correctly by the SOTA model; GPT-4 would reach 85.31%.Source: §4.1 · p. 9

4.2. Addressing Invalid Data

• 727 of 2629 adversarial images are categorized as invalid because human performance indicates they lack a perceptible depiction of the labeled class.Valid images are defined as those correctly labeled by two of four users or by one high-confidence human user.Source: §4.2 · p. 9
• Invalid adversarial images include deceptive examples showing an incorrect class and ambiguous examples lacking a clear object.Deceptive images represent 39% of invalid images and can induce the same incorrect prediction in humans and the SOTA model.Source: §4.2 · pp. 9–10
• The revised benchmark removes images that AutoAttack makes invalid, separating model errors from errors caused by invalid test data.Table 4 compares SOTA performance with and without invalid data and estimates human-performance upper bounds on both benchmarks.Source: §4.2 · p. 10, Table 4 · p. 10

5. Discussion

• Existing ℓp-norm-bounded attacks generate invalid data on SOTA-robust models, preventing benchmarks from measuring a fraction of human-level robustness.The paper therefore argues that attack formulations should account for image validity.Source: §5 · p. 10

5.1. Limitations and Future Directions

• The study’s observations require broader testing across datasets, models, and robustness benchmarks, including the prevalence of invalid data beyond CIFAR10.The authors also report preliminary evidence that training attacks can create invalid data and identify scaling behavior that should be tested more generally.Source: §5.1 · p. 10

Impact Statement

• The authors identify no potential societal consequences that they feel require specific highlighting.Source: Impact Statement · p. 10
• The organization’s IRB determined that the work did not require IRB review.Source: Impact Statement · p. 11

Scaling laws

• Scaling laws model robustness across unseen model sizes, dataset sizes, and synthetic data qualities, enabling compute-optimal resource allocation.They also expose opportunities to reduce model size and improve inference speed.Source: A. Outline of Our Contributions and… · p. 14

Human study

• A small-scale human study examines the performance bottleneck of current adversarial training methodologies.The study focuses on adversarial data that fools the top-performing model.Source: A. Outline of Our Contributions and… · p. 14

Novelty

• The work reports a single-epoch, synthetic-data-only adversarial training run that outperforms prior state of the art.The prior method relied on non-synthetic data.Source: A. Outline of Our Contributions and… · p. 14

Results

• 3% AutoAttack accuracy: the best compute-efficient model beats prior SOTA, while matching prior robustness requires 20% fewer training FLOPs.A 3× smaller model also gains 1% AutoAttack accuracy at the same compute budget.Source: A. Outline of Our Contributions and… · p. 14

Implications

• Scaling compute alone is predicted to require thousands of years of state-of-the-art supercomputer time to reach human robustness levels.The paper therefore points toward more efficient algorithms and improved architectures rather than simple scaling.Source: A. Outline of Our Contributions and… · p. 14
• Human performance is also limited because current ℓ∞ attack formulations can generate invalid images that do not fit their original labels.The paper argues that attacks should produce valid images containing the information needed for the original label.Source: A. Outline of Our Contributions and… · p. 14

FLOP accounting

• The adversarial training loop requires 27× the FLOPs of one forward pass, compared with roughly 3× for regular training.The main source of extra cost is the 10-step PGD attack.Source: B. Counting the FLOPs of Adversarial… · p. 15, §B.1 · p. 15
• 27 forward-pass equivalents are required per adversarial training iteration, versus about 3 for regular training.The resulting adversarial iteration is approximately 9× as expensive.Source: §B.1 · p. 15

PGD cost

• 10 PGD steps account for roughly 20 forward-pass equivalents because each step computes an image gradient without updating weights.A clean-data forward pass raises this subtotal to 21.Source: §B.1 · p. 15

TRADES cost

• Two additional forward passes evaluate the attacked and clean outputs for TRADES, followed by a larger backward pass for the loss gradient.Reusing the clean output could save one forward pass.Source: §B.1 · p. 15

Human vulnerability

• Images produced by AutoAttack on the SOTA network trick both humans and neural networks.The paper relates this finding to prior human-fooling attacks across resolutions and perturbation norms.Source: C. Additional Discussion of… · p. 15

Human comparison

• Human performance is not a perfect gold standard because images that fool current networks can also cause human misclassification.The paper argues that robustness progress may therefore be less pessimistic than standard human comparisons suggest.Source: C. Additional Discussion of… · p. 15

Hyperparameter study

• The hyperparameter study varies dataset size, batch size, and learning rate, selecting settings with higher CW test accuracy for scaling-law runs.The study is preparatory rather than a robustness result itself.Source: Figure 10 · p. 16

Sampling robustness

• Infrequent resampling caused by interrupted jobs has no notable performance effect compared with sampling without replacement.This supports treating the sampling irregularity as negligible for the scaling-law models.Source: Figure 11 · p. 17

Training Dataset Sampling

• Infrequent sampling with replacement from interrupted jobs produces no notable performance difference from sampling without replacement.The effect arises when resumed jobs receive a new random sampler state.Source: E. Training Dataset Sampling · p. 17, Figure 12 · p. 18

Optimization

• Approach 1 uses log-space regressions to estimate compute-optimal model and dataset sizes, while Approach 3 finds them numerically with SciPy fsolve.The query FLOPs are restricted to points reflecting loss improvements as compute increases.Source: §F.1 · p. 18, §F.3.2 · p. 20

Fitting

• The fitting procedures use L-BFGS with Huber loss to learn scaling-law parameters, including synthetic-data quality effects.Quality modifies the fitted terms through learned exponents.Source: §F.2.1 · p. 18, §F.3.1 · p. 19
• Approach 3 fits all models, datasets, and data qualities with one shared parameterization.The fit includes a model trained on 300M synthetic images from EDM-20, PFGM++, and DG.Source: §F.3.1 · pp. 19–20, Figure 14 · p. 20
• The fitting excludes small models trained on large, high-quality datasets because those models underfit as dataset size increases.The restriction applies when model size is below 100M, data size exceeds 10M, and FID is below 10.Source: §F.2.1 · p. 19, §F.3.1 · p. 20

Scaling-law comparisons

• Approach 1 scales model size more aggressively than Approach 2, with exponents a = 0.56 versus a = 0.48.Approach 3 balances model and dataset scaling at low compute, then shifts toward model scaling as overfitting is anticipated.Source: §F.4 · p. 21

Scaling-law implications

• All three approaches indicate that prior SOTA models use too little data for their size, favoring smaller models trained on more data.This is consistent with robustness having large sample complexity.Source: §F.4 · p. 21

Compute-efficient region

• The compute-efficient region assumes DG-20 data with FID ≈1.65, although mixed-in CIFAR10 data may make the effective quality slightly higher.Higher data quality could lower the predicted N* because it tends to raise b.Source: §F.4 · p. 21

Human-evaluation setup

• The human evaluation classified 2,629 adversarial images that fooled the SOTA network and their clean counterparts, totaling 5,258 images per user.Three users classified images in batches of 100 through a web application.Source: §G.1 · p. 21, Figure 16 · p. 22
• Participants were research-group members familiar with CIFAR10, and the study provided limited additional training because the dataset has only 10 relatively distinct classes.This setting differs from studies that provide extensive participant training.Source: §G.1 · p. 21

GPT-4 evaluation

• GPT-4 correctly classified 1,160 of 2,629 adversarial images, but this does not show greater robustness because the attacks targeted the SOTA model.The evaluation therefore measures classification of examples generated against another model, not GPT-4’s own benchmark robustness.Source: §G.2 · p. 23

Invalid adversarial images

• Clean-image comparisons show that “Deceptive” adversarial examples can appear to depict birds, deer, or dogs despite those attributes being absent from the originals.Their perceived classes differed substantially from the original image classes and from the classifications of humans and the SOTA network.Source: §G.3 · p. 23, Figure 19 · p. 25

Training-time attack artifacts

• Benchmark attacks making 10% of CIFAR10 adversarial test images invalid suggest that weaker training-time attacks may also generate invalid data.A 10-step PGD training example appeared to humans as a dog, horse, or interpolation between them.Source: §G.4 · p. 23, Figure 20 · p. 25

Human-fooling examples

• Figure 21 presents adversarial images that can potentially fool humans alongside their clean counterparts.The comparison is intended to show how attack-generated manipulations relate to the original images.Source: §G.5 · p. 23, Figure 21 · p. 26

GPT-4 evaluation

• The evaluation used a fixed-temperature GPT-4 payload, with temperature set to 0 and images represented as Base64 data.The paper also reports that prompt variation could improve GPT-4V performance and clarify which images have lost label information.Source: §G.2 · p. 23, Figure 18 · p. 24