PaperlayerView exact revision on arXiv

Source-linked AI summary

RL-FAT: Reinforcement Learning for Fair Adversarial Training

Tejaswini Medi, Levan Mikeladze, Margret Keuper

arXiv:2608.29247v1cs.LG

TL;DR

Average robust accuracy can conceal substantial class-wise vulnerabilities, motivating robust-fairness evaluation beyond the average case. RL-FAT uses policy-gradient feedback from adversarial predictions together with fairness-emphasis loss, and experiments report improved worst-class robustness and robust fairness with competitive overall accuracy.

  • Problem

    Average robust accuracy can hide large class-wise robustness disparities, making worst-class robustness and class-wise fairness important evaluation criteria.

  • Method

    RL-FAT combines correctness-based adversarial prediction rewards, class-wise value estimates and advantages, and fairness-emphasis adversarial loss to adapt training emphasis across classes.

  • Results

    Across CIFAR-10, CIFAR-100, and ImageNette, RL-FAT consistently improves worst-class robust accuracy and robust fairness while maintaining competitive clean and robust accuracy.

  • Takeaways & Limitations

    RL-FAT promotes more balanced robustness across classes by focusing training on unreliable adversarial predictions and high-loss classes.

  • Takeaways & Limitations

    The robust-fairness metric is baseline-dependent, captures fairness only at the class level, and may not reflect subgroup robust fairness.

Abstract

from arXiv · show

Deep neural networks remain highly vulnerable to adversarial perturbations, and adversarial training (AT) has become a widely used approach for improving robustness. However, improvements in average robust accuracy often mask substantial class-wise disparities: while some classes become more robust, others may remain disproportionately vulnerable under attack. This imbalance raises an important adversarial fairness concern, particularly in vision tasks where reliable robustness is expected across all categories. To address this challenge, we propose \textbf{RL-FAT}, a reinforcement-learning-inspired fair adversarial training framework that uses policy-gradient based feedback from adversarial predictions. RL-FAT interprets the prediction distribution as a policy and combines correctness-based prediction rewards with class-wise value estimates to compute class-specific advantages for policy-gradient optimization. This enables the model to adaptively focus on class-wise misclassification. Furthermore, we introduce a fairness-emphasis adversarial loss that assigns stronger training pressure to classes with high adversarial loss, thereby mitigating class-wise robustness disparity. By combining reinforcement-driven adaptation with fairness-emphasis regularization, RL-FAT improves adversarial robustness while promoting a more balanced robustness distribution across classes. Extensive experiments demonstrate that our method achieves competitive robust accuracy and substantially reduces class-wise robustness imbalance compared with standard adversarial training baselines.

1 Introduction

Adversarial training improves average robustness, but average metrics can conceal severe class-wise vulnerabilities. RL-FAT addresses this robust fairness problem by learning class-aware training emphasis from adversarial prediction feedback and fairness-weighted loss.

  • Average robust accuracy can hide substantial disparities, leaving some classes disproportionately vulnerable under adversarial attacks.
  • ResNet-18 and XCiT-S12 exhibit larger gaps between average and worst-class accuracy under adversarial evaluation than under clean evaluation.
  • Existing robust-fairness methods use designed class-wise rules, predefined metrics, configuration choices, or additional optimization mechanisms.
  • RL-FAT interprets adversarial predictions as actions and uses correctness rewards, class-wise value estimates, and advantages to guide policy-gradient optimization.
  • RL-FAT adds fairness-emphasis adversarial loss, applying stronger training pressure to classes with above-average adversarial loss.
  • RL-FAT consistently improves worst-class robust accuracy and robust fairness under strong attacks while maintaining competitive overall clean and robust accuracy.

2 Related Work

Related work shows that average robust accuracy does not fully capture class-wise reliability. Existing approaches address robust unfairness through reweighting, risk-variance regularization, class-specific configurations, and other designed mechanisms, whereas RL-FAT learns training emphasis from adversarial prediction feedback.

  • Robust-fairness studies identify large class-wise disparities despite high average robust accuracy and emphasize worst-class robustness as an evaluation criterion.
  • FRL adjusts loss weights and adversarial margins, while FAT regularizes the variance of class-wise adversarial risk.
  • CFA calibrates perturbation margins, regularization strengths, and weight averaging separately for each class, while DAFA links fairness to inter-class similarity.
  • RL-FAT learns class-wise training emphasis directly from adversarial prediction feedback instead of relying mainly on predefined reweighting or class-specific configuration rules.

3 Methodology

RL-FAT uses a warm-up TRADES-based stage before jointly optimizing reinforcement-based class feedback and fairness-emphasis losses. Its feedback treats adversarial predictions as sampled actions, while its fairness term emphasizes classes with above-average adversarial loss.

  • Training stages: RL-FAT first performs TRADES-based warm-up adversarial training to obtain a stable robust initialization before fairness-aware optimization.The warm-up uses clean cross-entropy and KL-based consistency between clean and adversarial predictions.
  • Reinforcement-based class feedback: For each adversarial input, RL-FAT interprets the prediction distribution as a policy, samples a class action, and assigns reward according to prediction correctness.Correct predictions receive positive feedback, whereas incorrect predictions receive negative feedback.
  • Adversarial example generation: PGD generates adversarial examples within an ℓ∞ perturbation budget by maximizing KL divergence between clean and adversarial predictive distributions.The perturbations are projected back onto the threat-model ball and clipped to the valid image range after each step.
  • Post-warm-up objective: Post-warm-up training combines the reinforcement and fairness objectives as Lpost = LRL + λLfair.λ controls the contribution of the fairness term.
  • Reinforcement-based class feedback: Class-wise running action-value estimates provide baselines, and the resulting advantages reinforce or suppress sampled actions through the reinforcement loss.Positive advantages increase an action's log-probability; negative advantages suppress it, using accumulated class-wise reward history.
  • Fairness-emphasis adversarial loss: The fairness-emphasis loss assigns stronger pressure to classes whose adversarial losses exceed the mini-batch class average.Positive hard-class gaps are normalized into weights, distributing pressure across multiple vulnerable classes rather than only the single worst class.

4 Experiments

RL-FAT is evaluated on standard vision benchmarks against adversarial-training and robust-fairness baselines using CNN and transformer architectures. The experiments use adversarial evaluation and a training procedure that combines TRADES warm-up with RL-FAT objectives.

  • RL-FAT is evaluated on CIFAR-10, CIFAR-100, and ImageNette using TRADES and representative robust-fairness methods as baselines.Comparisons include FRL, WAT, CFA, and DAFA, with PGD training and AutoAttack evaluation.
  • The evaluation uses ResNet-18 as the main backbone and includes XCiT-S12 to examine generalization across CNN and transformer architectures.
  • Training first uses the TRADES objective for 110 epochs, then continues with the RL-FAT objective until epoch 150.The TRADES stage uses β = 6 to obtain a stable robust initialization.
  • During RL-FAT training, prediction actions receive correctness-based rewards, while class-wise value estimates and clipped advantages guide the reinforcement loss.The objective also applies higher fairness-loss weight to classes with above-average adversarial loss.
  • Experiments use five random seeds, training-time checkpoint selection, and final AutoAttack evaluation without using the test set for checkpoint selection.Evaluation uses the same perturbation budget ε = 8/255.

4.3 Evaluations on CIFAR-10

On CIFAR-10 with ResNet-18, RL-FAT improves worst-class robust accuracy over TRADES and robust-fairness baselines while targeting a better trade-off than average robust accuracy alone.

  • RL-FAT improves worst-class robust accuracy over FRL, WAT, CFA, and DAFA by 4.78%, 2.94%, 5.17%, and 1.36%, respectively.
  • Table 1 reports average and worst-class accuracy under clean and adversarial evaluation, with ρrob measuring fairness improvement relative to TRADES.Higher ρrob indicates stronger robust-fairness improvement after adjusting for changes in average robust accuracy.
  • The reported results characterize RL-FAT as improving worst-class robustness rather than only maximizing average robust accuracy.

4.4 Evaluations on CIFAR-100

Across CIFAR-100 and ImageNette evaluations, RL-FAT obtains the best reported worst-class robust accuracy while maintaining competitive overall robustness under adversarial evaluation.

  • 4.4 Evaluations on CIFAR-100: 2.20% worst-class robust accuracy is achieved by RL-FAT on CIFAR-100, outperforming TRADES, FRL, WAT, CFA, and DAFA.RL-FAT improves over TRADES by 0.90% while maintaining competitive average robust accuracy.
  • ImageNette evaluation: Table 3 reports mean and standard deviation for clean and adversarial accuracy and computes ρrob relative to the TRADES baseline.
  • ImageNette evaluation: 28.79% worst-class robust accuracy is achieved by RL-FAT on ImageNette, improving over TRADES by 5.89%.The evaluation uses 64×64 ImageNette inputs and AutoAttack.
  • ImageNette evaluation: On ImageNette, RL-FAT achieves 47.91% average robust accuracy and 75.92% average clean accuracy while CFA reaches 50.68% average robust accuracy.RL-FAT nevertheless has higher worst-class robust accuracy than CFA in the reported comparison.

4.6 Class-specific Robustness Steering

RL-FAT can steer training toward a selected class by replacing the fairness-emphasis loss with that class’s adversarial loss. Steering improves the target class but can redistribute performance across other classes.

  • Steering mechanism: Steering replaces the fairness-emphasis loss with Lsteer(t) = Lt, the adversarial cross-entropy loss for samples from target class t.Changing t redirects training pressure toward different class-specific robustness goals.
  • Table interpretation: Table 4 reports target-class, average, and worst-class accuracy under clean and AutoAttack evaluation for each CIFAR-10 steering target.Worst-class columns also identify the most frequent worst-performing class across five seeds.
  • Target-class results: Cat steering reaches 99.72% target clean accuracy and 94.42% target robust accuracy, while Frog, Dog, Deer, and Bird each exceed 90% target robust accuracy.
  • Overall distribution: Horse steering achieves 76.82% average clean accuracy, 47.32% average robust accuracy, and 16.74% worst-class robust accuracy.Truck produces the strongest worst-class clean accuracy at 36.12%.

4.7 Evaluation on Out-of-Distribution Detection

RL-FAT is evaluated for OOD detection using CIFAR-10 as in-distribution data and SVHN as OOD data, emphasizing both overall and worst-class MaxLogit performance. The broader evaluation also examines how the complete objective improves worst-class robustness and fairness under AutoAttack.

  • OOD detection: OOD detection uses CIFAR-10 as the in-distribution dataset and SVHN as the semantically different OOD dataset, with MaxLogit as the score.Overall and worst-class OOD performance are both considered to assess balanced separation across classes.
  • OOD detection: Figure 3 reports overall MaxLogit metrics on the left and worst-class MaxLogit metrics on the right, with higher values indicating better performance.Green circles identify RL-FAT results.
  • RL-FAT ablation: 32.38% is the best worst-class robust accuracy achieved by the complete RL-FAT objective in the CIFAR-10 AutoAttack ablation.The evaluation uses AutoAttack at ε = 8/255 with ResNet-18.
  • RL-FAT ablation: 38.77 is the highest ρrob score achieved by the complete RL-FAT objective in the same ablation.The result indicates the strongest robust fairness improvement among the evaluated objective variants.

4.9 Ablation on PGD-Based Warm-up

RL-FAT is tested beyond TRADES warm-up by using PGD adversarial training on CNN and transformer architectures. It improves worst-class robust accuracy and fairness while retaining competitive average robustness.

  • ResNet-18: 31.98% worst-class robust accuracy is achieved by RL-FAT with PGD warm-up on ResNet-18, outperforming PGD by 12.74%.RL-FAT also outperforms DAFA by 5.62% and achieves a ρrob score of 67.49.
  • ResNet-18: RL-FAT with PGD warm-up achieves the highest ρrob score of 67.49 on ResNet-18.PGD reaches 48.80% average robust accuracy but only 19.24% worst-class robust accuracy.
  • XCiT-S12: 37.80% worst-class robust accuracy is achieved by RL-FAT on the XCiT-S12 transformer, alongside the highest ρrob score of 27.76.Its average robust accuracy is 54.42%.
  • XCiT-S12: The XCiT-S12 experiment extends the fairness evaluation beyond CNN-based ResNet-18 models to transformer-based models.The setup uses PGD-based baselines and evaluates the best checkpoint over three seeds.

5 Conclusion

The conclusion frames RL-FAT as a reinforcement-learning-inspired approach to reducing class-wise robustness disparities that average robust accuracy can conceal. Across multiple datasets and training settings, it improves worst-class robustness while retaining competitive average performance.

  • Method: RL-FAT combines policy-gradient feedback from adversarial predictions with a fairness-emphasis adversarial loss.Class-wise reward feedback and stronger pressure on high-loss classes target vulnerable classes during adversarial training.
  • Results: Across CIFAR-10, CIFAR-100, and ImageNette, RL-FAT consistently improves worst-class robust accuracy while maintaining competitive average clean and robust accuracy.The conclusion presents this as the main cross-dataset outcome.
  • Results: The ablation results identify the fairness-emphasis component as important for improving the most vulnerable classes.PGD-based experiments also indicate generalization beyond a TRADES warm-up setting.
  • Takeaway: RL-FAT is presented as a simple and flexible direction for making adversarially robust models more balanced across classes.The stated scope is models that remain strong on average while improving class-wise balance.

Supplementary Material of RL-FAT

The supplementary material adds training details, OOD and corruption-robustness evaluations, and computational-cost comparisons. It documents RL-FAT’s two-stage procedure and reports balanced OOD performance alongside selected corruption gains and cost context.

  • Training procedure: The two-stage training procedure first uses TRADES warm-up training, then optimizes reinforcement-based class feedback together with fairness-emphasis loss.The algorithm initializes class-wise value estimates and later computes rewards, advantages, hard-class weights, and the combined post-warm-up loss.
  • OOD detection: RL-FAT’s OOD evaluation reports both overall and worst-class MaxLogit performance for SVHN detection with CIFAR-10 as in-distribution data.Higher AUROC, AUPR-IN, and AUPR-OUT indicate better OOD detection.
  • OOD detection: RL-FAT achieves worst-class AUROC 77.79, AUPR-IN 24.86, and AUPR-OUT 98.40, the best worst-class results in the reported OOD comparison.Its overall values are 83.78, 78.57, and 88.65, respectively, while FRL has the strongest overall performance.
  • Corruption robustness: The corruption analysis reports only settings where RL-FAT improves worst-class accuracy over TRADES across all five severity levels.Table 9 expresses gains as RL-FAT minus TRADES in percentage points for average and worst-class accuracy.
  • Computational cost: RL-FAT adds lightweight class-wise value updates after adversarial example generation rather than accelerating adversarial training.Training time and memory comparisons are reported on a single NVIDIA H100 with batch size 128.
Loading 2608.29247v1…