PaperlayerView exact revision on arXiv

Source-linked AI summary

Enhancing LLM Safety Through a Theoretical Minimax Game Lens

Yihe Deng, Yu Yang, Junkai Zhang, Wei Wang, Bo Li

arXiv:2502.05163v2cs.CLcs.LG

TL;DR

Multilingual guardrail training is constrained by scarce real-world safety data outside English. The paper proposes a two-player minimax RL framework that co-evolves synthetic-data generation and classifier training. Its reported model outperforms smaller and larger baselines while remaining compact and faster at inference.

  • Problem

    Open-source guardrail datasets and approaches focus predominantly on English, and real-world safety data in other languages are scarce.

    Source: §1 · p. 1

  • Method

    An iterative two-player RL framework uses a generator and guardrail classifier to synthesize multilingual training data and improve classification.

    Source: §1 · p. 2

  • Results

    The 0.5B model outperforms similarly sized baselines by over 20% and larger models by 10%, with a reported 3× inference speedup.

    Source: §7 · p. 8

  • Takeaways & Limitations

    The authors conclude that their minimax framework offers a compact approach to multilingual safety-data generation and guardrail training.

    Source: §7 · p. 8

    The current implementation focuses on English, French, German, and Spanish, although the approach retains Qwen-2.5's multilingual capacity for expansion.

    Source: §7 · pp. 8–9

Abstract

from arXiv · show

The rapid advancement of large language models (LLMs) necessitates effective mechanisms to ensure their responsible deployment by accurately distinguishing unsafe content from benign content. While substantial safety datasets are available in English, multilingual safety modeling remains underexplored due to limited open-source safety datasets in other languages. Even within English datasets, safe yet sensitive corner-case content is scarce, leading to shortcut learning by models and non-trivial false-positive rates. To mitigate these issues, we introduce a novel minimax reinforcement learning (RL) framework wherein a data generator and a classifier model co-evolve, facilitating the production of high-quality synthetic multilingual safety data. We theoretically formalize this interaction as a minimax game and rigorously demonstrate convergence to a Nash equilibrium. Empirical evaluations confirm that our synthetic data generation method significantly enhances the classifier model performance, enabling a substantially smaller model to surpass the state-of-the-art by nearly 10% on English benchmarks while achieving 4.5x faster inference speed. These results establish a scalable and efficient methodology for synthetic data generation, advancing the development of safer and more robust multilingual LLM deployments.

1 INTRODUCTION

The introduction identifies multilingual safety-data scarcity and asks whether a guardrail classifier can guide synthetic-data generation to improve its own training. It presents an iterative two-player RL framework, theoretical convergence guarantees, and strong multilingual benchmark results.

1 INTRODUCTION

  • Existing guardrail approaches and open-source training datasets focus predominantly on English, while non-English real-world safety data are scarce.The cited work notes performance declines when English safety-aligned models are applied to other languages.Source: §1 · p. 1
  • The paper proposes an iterative two-player RL framework in which a data generator and guardrail classifier co-evolve to improve synthetic data generation and classifier training.The introduction frames the research question as whether the guardrail can actively guide data generation for its own training.Source: §1 · p. 2
  • The authors formulate the interaction as a minimax game with a Nash equilibrium and report linear convergence of their algorithm.The introduction also identifies data filtering, self-judgment, and seed-data curation as practical components.Source: §1 · p. 2
  • DuoGuard, a 0.5B classifier, outperforms similarly sized guardrails across datasets and surpasses larger models, according to the stated contribution summary.The introduction reports average gains above 20% over similarly sized baselines and approximately 10% over larger guardrail baselines.Source: §1 · p. 2

2 RELATED WORK

Related work covers LLM guardrails, two-player reinforcement learning, and multilingual synthetic-data generation. The paper distinguishes its data-synthesis and classifier-training focus from prior collaborative and self-play approaches.

Fine-tuning LLMs via Two-player RL

  • Prior two-player RL work includes collaborative task-solving between LLM agents, while self-play work iteratively optimizes models on generation tasks such as math reasoning and instruction following.The paper distinguishes its focus on data synthesis and model training from real-time agent collaboration during inference.Source: §2 · p. 2

Guardrail Models for LLM Safety

  • Guardrail research includes LlamaGuard, WildGuard, Aegis, MD-Judge, and ShieldGemma; small-scale examples include LlamaGuard3 (1B) and ShieldGemma (2B).The section identifies F1 score, model scale, and inference efficiency as practical guardrail considerations.Source: §2 · pp. 2–3

Multilingual Synthetic Data Generation

3 PROBLEM SETTING AND PRELIMINARIES

The preliminaries describe guardrails as binary harmful-content classifiers, introduce preference optimization, and motivate DPO as an alternative to reward-model training. They also define the paper's LLM representation and classification setup.

LLM Representation

  • An LLM is represented by a parameterized probability distribution that generates a response conditioned on a prompt.The response is treated as a sample from the conditional distribution pθ(·|x).Source: §3 · p. 3

Preference Optimization

  • RLHF commonly models human preferences with the Bradley-Terry framework, but reward-model training can be computationally expensive and operationally challenging.DPO is presented as a simplified alternative using an implicit reward function defined by the LLM itself.Source: §3 · p. 3

Guardrail Models

  • Guardrails classify user inputs or LLM outputs as harmful or not, and this work prioritizes inference efficiency with a binary classification model.The paper describes converting a pretrained LLM's final layer to produce discrete safe or unsafe outputs.Source: §3 · p. 3

4 METHOD

The method couples a generator that proposes labeled synthetic examples with a classifier trained on them. Its theoretical analysis establishes equilibrium convergence, while practical procedures focus training on misclassified examples.

4 METHOD

  • The method starts from labeled real seed data and iteratively updates a generator that proposes target-language examples and a classifier trained to distinguish harmful from benign content.The generator preserves the seed example's toxicity label, while the classifier outputs a probability of toxicity.Source: §4 · p. 3, §4.1 · pp. 3–4

4.1 The Two-Player Game: Theoretical Convergence

  • The generator is trained to produce samples that elicit incorrect classifier predictions, while the classifier minimizes negative log-likelihood on generated examples.DPO is selected for generator training for stability and computational efficiency; higher negative log-likelihood of the correct label indicates greater classifier vulnerability.Source: §4.1 · p. 4, Figure 3 · p. 4
  • The analysis casts the generator-classifier interaction as a minimax game whose DPO objective shares a minimizer with a KL-regularized RL objective.The objective equivalence connects the generator's preference optimization to the classifier's training objective.Source: §4.1 · p. 4
  • With an appropriately chosen regularization parameter β, the minimax game admits a Nash equilibrium and iterative updates converge to it linearly.The paper attributes convergence to a contractive update map on the distribution space.Source: §4.1 · p. 5

4.2 The Two-Player Game: Practical Algorithm

  • The practical algorithm augments training data exclusively with generated samples the classifier misclassifies, preserving seed labels through separate safe and unsafe prompts.It also uses 12 harmful classes with a multi-label binary cross-entropy objective and retrains the classifier from scratch each iteration.Source: §4.2 · p. 5
  • Generator preference pairs treat misclassified samples as preferred and correctly classified samples as dispreferred in DPO training.The practical algorithm constructs as many preference pairs as available from the evolving dataset.Source: §4.2 · p. 5

5 EXPERIMENTS

DuoGuard is evaluated across four languages and six safety datasets, with results showing strong multilingual F1 and low inference cost. Data generated by the two-player scheme also generalizes across base models and scales, including from a 0.5B classifier to a 1.5B classifier.

5 EXPERIMENTS

  • Two-player training iterates over the seed training dataset to sample queries and update the classifier and generator; experiments use Qwen2.5-0.5B and Qwen2.5-1.5B as classifier base models.Both classifier and generator receive full fine-tuning.Source: §5 · pp. 5–6
  • The evaluation spans four languages and six safety datasets, while the seed corpus is linguistically imbalanced: English is 81.4%, compared with French at 8.9%, Spanish at 5.2%, and German at 4.5%.The seed data also includes benign sensitive-domain examples and downsampled harmful examples dominated by specific terms.Source: §5 · p. 6

5.1 Main Results

  • Across four languages, DuoGuard’s average F1 exceeds LlamaGuard3 (8B) by over 10% and similar-scale LlamaGuard3 (1B) and ShieldGemma (2B) by more than 30%.Its inference cost is 16.47 ms/input; Figure 4 reports the lowest performance decline across non-English languages relative to DuoGuard’s English performance.Source: §5.1 · p. 6, Figure 4 · p. 7

5.2 Weak-to-Strong Generalization

  • Data generated by the framework consistently outperforms same-base-model training baselines by more than 20% across architectures, and data from the 0.5B classifier improves the 1.5B classifier.Table 2 reports average F1 across languages for models trained with the generated dataset.Source: §5.2 · p. 7, Table 2 · p. 7

6 ABLATION STUDY

The seed-data analyses show that language composition affects multilingual performance, while iterative synthetic data improves non-English results and shifts data toward underrepresented languages.

6.1 Seed Data

  • Adding French to English seed data raises French-test F1 from 51.3 to 65.2 and improves Spanish- and German-test F1 by 7.4 and 12.9 points.Adding Spanish and German further improves their respective test sets, but reduces English and French data proportions and slightly lowers performance overall.Source: §6.1 · p. 7, Figure 5 · p. 8
  • Seed-data languages with more training data, such as English, achieve higher average F1 than lower-data Spanish and German across six benchmarks.F1 consistently declines as dataset size decreases, though the gap varies across test sets.Source: §6.1 · pp. 7–8, Figure 6 · p. 8

6.2 Synthetic Data

  • By iteration 2, Spanish and German reach scores comparable to French, while all non-English languages narrow their gap with English.After iteration 1, substantial gains appear for all non-English languages, especially Spanish and German.Source: §6.2 · p. 8, Figure 7 · p. 8
  • At iteration 1, English falls from 81% of seed data to 13% of synthetic data, while French, German, and Spanish rise to 27%, 35%, and 24%.At iteration 0, French, German, and Spanish collectively account for less than 20% of seed data.Source: §6.2 · p. 8, Figure 7 · p. 8

7 CONCLUSION

The paper presents DuoGuard as a two-player minimax RL framework for multilingual safety data scarcity. Its four-language implementation is bounded by the underlying LLM’s quality and current language coverage.

7 CONCLUSION

  • Across six languages, the 0.5B model outperforms similarly sized baselines by over 20% and larger models by 10%, with a 3× inference speedup.The framework integrates synthetic data generation with guardrail-model training and is formalized as a minimax game with theoretical convergence guarantees.Source: §7 · p. 8
  • Synthetic-data outcomes depend on the underlying LLM’s quality, with stronger models expected to yield superior results.The paper notes Qwen-2.5 supports over 29 languages.Source: §7 · p. 8
  • The implementation illustrates the framework using English, French, German, and Spanish, while retaining Qwen-2.5’s broader multilingual capacity for expansion.The paper describes this broader language support as potential for future expansion.Source: §7 · pp. 8–9

Checklist

The checklist records reproducibility materials and theoretical-result documentation, while the appendix introduces analysis of the two-player minimax framework and its optimization objectives.

Checklist

  • The checklist marks theoretical assumptions, complete proofs, and explanations of assumptions as included..Source: Checklist · p. 13
  • The checklist marks main-result reproduction materials and training details as provided, while reporting that experiment statistics and error bars are not applicable.It also marks computing-infrastructure details as provided.Source: Checklist · p. 13

A Theoretical Analysis

A.1.1 Generator

The appendix derives generator and classifier optima, then uses concavity and convexity of the game objective to establish a Nash equilibrium under stated regularity conditions.

A.1.1 Generator

  • The DPO objective and its corresponding KL-regularized RL objective share a minimizer, with the generator update given by a preference-weighted distribution.The reward is negative classifier log-likelihood; the Bradley–Terry reward minimizer is equivalent up to an additive term constant in the compared responses.Source: §A.1.1 · pp. 14–15

A.1.2 Classifier

  • The classifier’s maximizing conditional distribution is q*(y|ex) = p(y|ex).In the two-player view, each player’s update is a best response to the other player’s previous distribution.Source: §A.1.2 · pp. 15–16

A.2.1 Existence of Nash Equilibrium

  • The equilibrium argument assumes finite discrete token sets, classifier probabilities bounded below by γ > 0, and a strictly positive generator normalization term.Source: §A.2.1 · p. 16
  • The minimax objective is concave in the generator distribution and convex in the classifier distribution, supporting the minimax equality and Nash-equilibrium result.The derivation invokes Von Neumann’s Minimax Theorem.Source: §A.2.1 · pp. 16–17

A.2.2 Convergence to Nash Equilibrium

The analysis establishes convergence by showing that the update maps are Lipschitz and their composition is a contraction. Banach’s theorem then yields a unique fixed point and linear convergence, while related work distinguishes this data-synthesis game from inference-time collaboration.

A.2.2 Convergence to Nash Equilibrium

  • The proof first establishes that both Tθ and Tϕ are Lipschitz before proving that the algorithm converges to a fixed point.The regularity conditions support the Lipschitz bounds.Source: §A.2.2 · pp. 17–18
  • With suitable β, the update maps’ Lipschitz constants have product below 1, making T² a contraction and yielding linear convergence within O(log(1/ϵ)) iterations.This provides an ϵ-equilibrium policy.Source: §A.2.2 · pp. 18–19
  • Banach’s fixed-point theorem establishes a unique fixed point for T², and the proof uses convergence of the even and odd subsequences to establish convergence of the full sequence.The theorem applies to a contraction on a complete metric space.Source: §A.2.2 · p. 19

B Additional Related Work

  • Unlike two-player RL for collaborative task-solving, this work uses the framework for data synthesis and model training rather than real-time inference collaboration.The related-work discussion also covers adversarial two-player approaches.Source: B Additional Related Work · p. 19

C Data Details

The seed-data collection combines multilingual and English sources, then curates labels and examples for classifier and generator training. Synthetic proposals are filtered for alignment and classifier errors, and ranked into preference pairs for generator training.

C Data Details

  • Seed-data curation includes deduplication and benchmark decontamination, with source-specific rules for interpreting or filtering labels and samples.Examples include retaining only unanimous safe samples from one dataset and discarding non-suicidal examples from another.Source: C Data Details · p. 20
  • The seed collection draws on multilingual safe and unsafe datasets, including sources in English, French, German, and Spanish.The appendix lists sources and processing steps and depicts the collected seed data’s language proportions.Source: C Data Details · pp. 19–21, Figure 8 · p. 19
  • Unsafe examples receive one or more fine-grained labels, assigned from original labels with manual inspection and, when needed, Llama-3.1 self-consistency over three queries.The listed scheme contains 12 subcategories.Source: C Data Details · p. 21

C.1 Data Curation

  • Generated proposals are retained when their harmfulness ratings match seed labels, their length differs by no more than 200 characters, and they contain no refusal phrases.The current classifier then selects retained proposals that it misclassifies for classifier training.Source: §C.1 · pp. 21–22
  • For DPO, proposals are ranked by classifier misclassification and rating alignment, with Level 1 preferred over Level 2 and weaker Level 3 comparisons used when Level 2 is absent.Instances without Level 1 examples are excluded from preference pairing.Source: §C.1 · p. 22

D Experiment Details

The experiment details specify the training resources and prompts used for data generation and safety evaluation. The impact statement reports theoretical convergence guarantees and empirical gains across six multilingual safety benchmarks, while noting unresolved risks involving adversarial attacks and cultural nuance.

D Experiment Details

  • The appendix provides fine-tuning hyperparameter tables and reports approximate per-iteration times for classifier, synthetic-data generation, and generator training.The reported times are around 10, 48, and 16 hours, respectively.Source: D Experiment Details · p. 22, Table 3 · p. 22, Table 4 · p. 23
  • The experiment details include prompts for creating new toxic or safe prompts in another language, inspired by an English input and tailored to that language’s speakers.The safety-evaluation prompt asks for assessment in the target language and cultural context.Source: D Experiment Details · pp. 23–24

E Impact Statement

  • The impact statement reports convergence guarantees and empirical gains across six multilingual safety benchmarks.It describes these results as demonstrating the approach’s effectiveness and robustness.Source: E Impact Statement · p. 24
  • The approach does not address all risks from adversarial attacks or nuanced cultural contexts in safety assessments.The statement points to human oversight and refined synthetic-data generation as future research directions.Source: E Impact Statement · p. 24

Related papers

Loading 2502.05163v2…