ROIC-DM: Robust Text Inference and Classification via Diffusion Model

1 Introduction

• Language models have advanced text inference and classification but remain vulnerable to adversarial inputs crafted to compromise their performance.Source: §1 · p. 1
• Existing defenses may require strong assumptions or sacrifice performance; detection and training need prior adversarial samples, while purification can modify clean-data information.The paper identifies preserving original meaning while removing adversarial perturbations as a challenge for purification.Source: §1 · p. 1
• Prior diffusion-model work in NLP focused on generation, motivating this paper's first investigation of diffusion models for text inference and classification.The paper argues denoising can help estimate the data space, while noting that classification use and comparable performance pose challenges.Source: §1 · pp. 1–2
• Experiments on three real-world datasets report that ROIC-DM is more robust than language models, including those with advanced defenses, and achieves comparable or better performance using language models as advisors.The introduction names AG NEWS, SST, and MRPC among the datasets.Source: §1 · p. 2
• ROIC-DM modifies diffusion models for text inference and classification, then uses traditional language models as advisors during denoising.The paper distinguishes its task-solving use of diffusion models from text purification, which uses them only for preprocessing.Source: §1 · p. 2

2.1 Textual Adversarial Attacks

• Textual attacks commonly replace words, such as with synonyms, using strategies that assume black-box access to output logits and search for substitutes.Source: §2.1 · p. 2

2.2 Textual Adversarial Defenses

• Training incorporates perturbations and detection filters adversarial samples, but both require prior attack knowledge; purification uses generative models yet can modify normal inputs.The paper therefore characterizes improving language-model robustness with these defenses as challenging.Source: §2.2 · p. 2

3 Preliminaries: Diffusion Models

• A diffusion model has a forward process that gradually transforms raw data into a Gaussian distribution and a reverse process that reconstructs it from Gaussian noise.Source: §3 · p. 2
• The forward process corrupts x0 into xt through a T-step Markov chain with fixed parameters, with a noise schedule controlling injected noise and a reparameterization adding Gaussian noise.Source: §3 · p. 2
• The reverse process is a learnable Markov chain that denoises xT toward x0, with each subsequent representation computed from the current one.Source: §3 · pp. 2–3
• Because x0 is unknown during reversal, a noise estimator predicts noise; the reverse phase is optimized using a variational lower bound and a diffusion loss.Figure 1 depicts the diffusion and reverse phases alongside a noise-estimator architecture.Source: §3 · p. 3, Figure 1 · p. 3

4 Methodology

• The methodology presents ROIC-DM as a diffusion-based approach to text classification and inference, with its overview shown in Figure 1.It first describes the diffusion classifier, then the pre-trained advisor, and finally the noise-estimator architecture.Source: §4 · p. 3

4.1 Diffusion Model for Text Classification and Inference

• ROIC-DM modifies diffusion models into classifiers, adding noise to labels and using text-conditioned reverse denoising to recover them.Training predicts noise from text context, noisy labels, and timestep; inference starts from Gaussian noise and repeatedly recovers the label before selecting the largest output position.Source: §4.1 · pp. 3–4

4.2 Pre-trained Advisor Improved ROIC-DM

• A fine-tuned pre-trained language model advises ROIC-DM during reverse denoising through its soft-label prediction for the input.The soft label is added to the noisy label representation.Source: §4.2 · p. 4

4.3 Model Architecture of fθ(·)

• The noise estimator encodes text with a BERT encoder, combines timestep and noisy-label features, smooths the fused vector, then projects it with text features to predict noise.The text feature is the [CLS] hidden state; the smoother uses softmax and layer normalization.Source: §4.3 · p. 4, Encoder · p. 4, Time Embedding and Linear Layer · p. 5

5.1 Datasets

• Experiments cover AG NEWS, SST-2, and MRPC, using full test sets for accuracy and random samples of 500 for robustness evaluation.The authors note that adversarial attack evaluation is seriously slow.Source: §5.1 · p. 5

5.2 Baselines

• ROIC-DM is compared with adversarial-training, regularization, and text-purification defenses.The baselines include PGD, FreeLB, InfoBERT, and Text Purification.Source: §5.2 · p. 5
• PGD minimizes empirical loss on adversarial examples, while FreeLB mixes embedding-perturbed virtual samples with normal training data.InfoBERT uses mutual-information-based regularizers to suppress noisy mutual information; Text Purification uses pre-trained models' mask-infill ability to recover noisy texts before prediction.Source: §5.2 · p. 5, InfoBERT · p. 5, Text Purification · p. 5

5.3 Attack Methods and Evaluation Metrics

• TextFooler and BERT-Attack assess robustness; the former iteratively replaces crucial words with synonyms, while the latter uses BERT to generate semantic-preserving substitutes.The paper selects these attacks because they can deceive robust models with limited perturbations and are widely used in robustness research.Source: §5.3 · p. 5
• Clean% measures accuracy on the clean test set, while Aua% measures prediction accuracy on adversarial data, with higher Aua% indicating greater robustness.The attacks used to generate adversarial data are evaluated individually.Source: Clean% · p. 5, §5.3 · p. 5
• Suc% is the fraction of involved texts successfully perturbed by an attack; a lower Suc% indicates greater robustness.Source: Clean% · p. 5

• ROIC-DM uses T = 1000 diffusion steps with a linear noise schedule from β1 = 1e−4 to βT = 0.02.Source: §5.4 · p. 5
• Training uses AdamW with a linear-decay learning rate starting at 1e-4 and a batch size of 64.Source: §5.4 · pp. 5–6
• Table 2 reports adversarial robustness, including MRPC attacks on both hypothesis and premise; † marks undefended fine-tuning baselines.Source: Table 2 · p. 6
• Table 3 compares accuracy for ROIC-DM with different pre-trained advisors on AG NEWS.Source: Table 3 · p. 6
• Baseline results use released PGD, FreeLB, and InforBERT settings; Text Purification is reimplemented from its paper description because no public code was available.Source: §5.4 · p. 6
• Adversarial attacks use their official OpenAttack implementations, and the experimental code was slated for release after double-blind review.Source: §5.4 · p. 6

• On clean data, ROIC-DM beats BERT on all three datasets by up to 2.9 accuracy scores and also exceeds defended BERT.Source: §5.5 · p. 6
• Additional AG NEWS analyses compare advisor ablation under both attacks, show a case study, and plot training losses.Source: Table 4 · p. 6, Table 5 · p. 7, Figure 2 · p. 7
• 27.7 Aua% on AG NEWS under TextFooler is ROIC-DM’s advantage over the best baseline, Text Purification; under stronger BERT-Attack, ROIC-DM still beats all baselines.Source: §5.5 · pp. 6–7

• Without an advisor, ROIC-DM performs slightly worse than traditional classifiers; the authors suggest their pretrained scale and mature architectures may explain the gap.Source: §5.6 · p. 7
• The advisor comparison reports AG NEWS results only, while stating that similar conclusions hold for the other two datasets.Source: §5.6 · p. 7
• ROIC-DM scores 1.1, 0.4, and 1.0 higher than BERT, DistillBERT, and ALBERT, respectively, when each serves as its counterpart advisor.Source: §5.6 · p. 7

5.7 The Impact of Advisor

• 60.3 Aua% on TextFooler and 46.4 Aua% on BERT-Attack for ROIC-DM without an advisor exceed BERT with the best baseline defender (51.0 and 44.5 Aua%, respectively).The comparison is on AG News and combines results from Tables 4 and 2.Source: §5.7 · p. 7
• ROIC-DM’s training loss is smoother and decreases faster than ROIC-DM without an advisor, suggesting the advisor can benefit training.Source: §5.7 · p. 7

5.8 Case Study

• On the meaning-preserving BERT-Attack example, only ROIC-DM retains the correct prediction; vanilla BERT and Text Purification based BERT do not.Source: §5.8 · p. 7

• Across three datasets and two competitive textual attacks, ROIC-DM is reported to be more robust and achieve better performance than conventional language models.Source: §6 · p. 7
• ROIC-DM is built on the diffusion-model framework for text inference and classification.Source: §6 · p. 7
• Conventional large language models are integrated as advisors within ROIC-DM’s reverse process to enhance performance.Source: §6 · p. 7