PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-playing Adversarial Language Game Enhances LLM Reasoning

Pengyu Cheng, Tianhao Hu, Han Xu, Zhisong Zhang, Zheng Yuan, Yong Dai, Lei Han, Nan Du, Xiaolong Li

arXiv:2404.10642v3cs.CLcs.LG

TL;DR

The paper asks whether LLM reasoning can be improved without relying solely on externally collected reasoning data. It introduces SPAG, which trains LLMs as both players in Adversarial Taboo and reinforces winning self-play episodes. Across the evaluated models and benchmarks, reasoning performance improves broadly and continues to improve through repeated self-play, while the larger-model setting and possible harmful behaviors remain insufficiently studied.

  • Problem

    LLM reasoning remains challenged, while existing self-improvement methods can require broad high-quality queries and rely on potentially biased model judgments.

    Source: §1 · p. 2

  • Method

    SPAG first uses imitation learning to teach an LLM attacker and defender behavior, then repeatedly reinforces winning episodes from the LLM playing against a copy of itself.

    Source: §5 · p. 9

  • Results

    Reasoning performance uniformly improves across various benchmarks, and three repeated self-play epochs produce continuous improvement in the reported reasoning results.

    Source: §4.2 · p. 8, §5 · p. 9

  • Takeaways & Limitations

    Adversarial multi-agent self-play provides a path for improving fundamental LLM capabilities through reinforcement learning without requiring the game’s target-word interactions to be manually judged.

    Source: §1 · p. 3, §5 · p. 9

    Experiments cover only LLaMA-2-7B and Baichuan-2-13B, and the harmful effects of adversarial self-play, including learned cheating or bluffing, remain insufficiently studied.

    Source: §6 · p. 10

Abstract

from arXiv · show

We explore the potential of self-play training for large language models (LLMs) in a two-player adversarial language game called Adversarial Taboo. In this game, an attacker and a defender communicate around a target word only visible to the attacker. The attacker aims to induce the defender to speak the target word unconsciously, while the defender tries to infer the target word from the attacker's utterances. To win the game, both players must have sufficient knowledge about the target word and high-level reasoning ability to infer and express in this information-reserved conversation. Hence, we are curious about whether LLMs' reasoning ability can be further enhanced by Self-Playing this Adversarial language Game (SPAG). With this goal, we select several open-source LLMs and let each act as the attacker and play with a copy of itself as the defender on an extensive range of target words. Through reinforcement learning on the game outcomes, we observe that the LLMs' performances uniformly improve on a broad range of reasoning benchmarks. Furthermore, iteratively adopting this self-play process can continuously promote LLMs' reasoning abilities. The code is available at https://github.com/Linear95/SPAG.

1 Introduction

LLMs have strong language abilities but their reasoning remains challenged, motivating more fundamental and objective improvement methods. The paper proposes Adversarial Taboo self-play because its information-limited, automatically judged interactions exercise broad language and reasoning capabilities.

  • LLM reasoning remains challenged despite advances in understanding, generation, translation, summarization, and programming, although it is essential for complex problem-solving [Pan et al., 2023].Source: §1 · p. 2
  • Prompt engineering and auxiliary tools can improve reasoning but require additional designs and are sensitive to prompt patterns and model checkpoints [Turpin et al., 2024, Chu et al., 2023].Source: §1 · p. 2
  • Self-improvement methods use model-generated synthetic data, but often require broad high-quality query collections and may amplify incorrect or biased self-judgments [Singh et al., 2023, Huang et al., 2023, Burns et al., 2023, Chen et al., 2024; Raina et al., 2024].Source: §1 · p. 2
  • Adversarial Taboo hides a target word from the defender while the attacker tries to elicit it and the defender tries to infer it without uttering it.Source: §1 · pp. 2–3
  • The game supports broad topical coverage through a vast target-word vocabulary and explicit automatic judging from utterances and inference patterns.Source: §1 · p. 3

2 Preliminary

This section situates SPAG within reinforcement-learning methods for language models. It describes PPO, offline A-LoL, and ReST as approaches for optimizing or selecting model-generated samples while addressing stability and sampling costs.

  • RLHF learns a reward model from human preference pairs and optimizes an LLM policy to maximize expected reward [Ouyang et al., 2022].Source: §2 · p. 3
  • PPO is a mainstream RLHF optimizer that regularizes policy updates with a KL divergence from a pre-update reference policy [Schulman et al., 2017].Source: §2 · p. 3
  • PPO for LLMs is challenged by inefficient online natural-text sampling and unstable training, motivating offline improvements such as Advantage-Leftover-Lunch (A-LoL).Source: §2 · p. 3
  • ReST simplifies RLHF by selecting samples whose reward exceeds a threshold and reinforcing that offline subset [Gulcehre et al., 2023].Source: §2 · p. 3

3 Self-play of Adversarial Language Games

SPAG models Adversarial Taboo as a two-player zero-sum language game, first teaching an LLM to follow the roles and then updating it from winning self-play episodes. The method uses offline, regularized policy optimization to improve stability and preserve general language ability.

3.1 Adversarial Language Game Modeling

  • Adversarial Taboo assigns a hidden target word to the attacker, who must induce its utterance, while the defender infers it without saying it; incorrect guesses lose and unfinished games tie.Source: §3 · p. 3
  • The game is modeled as a two-player zero-sum Markov game whose states append alternating attacker and defender utterances, with rewards defined from the attacker’s perspective.Source: §3.1 · p. 4

3.2 Imitation Learning

  • Before self-play, imitation learning on GPT-4 game episodes trains the LLM to follow attacker and defender rules, while KL regularization preserves general language abilities.Source: §3.2 · pp. 4–5

3.3 Reinforcement Learning from Self-play

  • During self-play, one LLM copy alternates between attacker and defender policies to collect multi-turn episodes across target words, avoiding inefficient on-policy training through an offline scheme.Source: §3.3 · pp. 5–6
  • SPAG selects attacker-winning episodes with R(τ) > 0 and defender-winning episodes with R(τ) < 0, then applies separate policy losses with importance sampling and KL-regularized updates.Source: §3.3 · pp. 5–6
  • The objective retains supervised fine-tuning data, because the SFT term is intended to prevent loss of general language abilities during self-play training.Source: §3.3 · p. 6

4 Experiments

Experiments evaluate SPAG on LLaMA-2-7B and Baichuan-2-13B using reasoning benchmarks, game win rates, and ablations against CoT, SFT, and non-adversarial-game baselines. Imitation learning improves reasoning broadly, while iterative adversarial self-play further raises overall reasoning and game performance, with gains attributed mainly to SPAG rather than SFT data.

Imitation Learning

  • Both LLaMA-2-7B and Baichuan-2-13B improve uniformly across reasoning benchmarks after GPT-4 imitation learning, while general language ability is largely preserved.The IM LLaMA-2 model improves on MMLU; Baichuan-2-13B shows only an insignificant MMLU decrease despite reasoning gains.Source: §4.2 · p. 7
  • Adversarial Taboo imitation outperforms imitation on 20-Question and Guess-My-City, and exceeds CoT in overall reasoning geometric mean.Although LLaMA-2 CoT performs strongly on BBH and WinoGrande, the IM model achieves higher overall reasoning performance.Source: §4.2 · p. 8

Self-play Training

  • Reasoning performance generally increases across SPAG epochs for both models, with SPAG-2 surpassing IM on LLaMA-2-7B after an initial WinoGrande decline.MMLU does not improve continuously, especially for Baichuan-2-13B, but overall geometric-mean performance continues to rise significantly.Source: §4.2 · p. 8

Ablation Study

  • SPAG-3 defeats continuous Alpaca SFT on all LLaMA-2-7B benchmarks and maintains noticeable advantages on Baichuan-2-13B except Mutual.The authors therefore attribute the major reasoning gains primarily to the SPAG scheme rather than the included SFT data.Source: §4.2 · p. 8
  • Imitation gains saturate beyond 5K GPT-4 episodes, while β1 around 0.1, β2 around 0.2, and α at or below 0.5 are associated with stronger reasoning improvements.Additional self-play episodes still raise reasoning scores, although their marginal effect is smaller than during imitation learning.Source: §4.2 · p. 8, Figure 3 · p. 9

Game-Play Performance

  • SPAG models show continuous win-rate improvement against GPT-4, while defender performance also improves as self-play epochs increase.Supplementary statistics indicate that both players use fewer interactions and shorter utterances to achieve victory.Source: §4.2 · p. 9, Figure 4 · p. 9

5 Conclusion

SPAG trains LLMs through self-play in Adversarial Taboo, where each model alternates between attacker and defender roles against a copy of itself. Repeating self-play and reinforcement for three epochs continuously and uniformly improves reasoning performance across various benchmarks.

  • SPAG has an LLM imitate attacker and defender roles, collect self-play episodes against a copy, and reinforce selected winning episodes.The process repeats self-play and reinforcement for three epochs.Source: §5 · p. 9
  • Reasoning performance continuously and uniformly improves across various benchmarks after iterative self-play and reinforcement.The paper presents SPAG as a multi-agent self-play path for improving LLM fundamental capabilities.Source: §5 · p. 9

6 Limitations

The evaluation is limited to two open-source LLMs, and important training alternatives remain untested. The authors also have not sufficiently studied whether adversarial self-play induces harmful behaviors.

  • Effectiveness was verified only on LLaMA-2-7B and Baichuan-2-13B, leaving larger LLM sizes unevaluated.The authors also identify MCTS value estimation and actor-critic policy gradients as unexplored alternatives.Source: §6 · p. 10
  • The harmful impact of adversarial self-play remains insufficiently studied, including whether models learn cheating, bluffing, or other disgraceful tricks.Source: §6 · p. 10

7 Broader Impacts

The experiments suggest that self-play can continuously enhance LLM capacity on a particular task, while also raising safety concerns about increasingly capable systems and adversarial language tricks.

  • Self-play can continuously enhance LLM capacity on a particular task, suggesting a possible analogue of AlphaGo Zero’s self-improvement through play.The passage frames this as an indication rather than a demonstrated general result across all intelligence tasks.Source: §7 · p. 10
  • Adversarial language games may encourage harmful tricks such as cheating and bluffing, motivating exhaustive security checks on self-played LLMs.Source: §7 · p. 10

A.1 GPT-4 Data Collection

GPT-4 data collection and SPAG training use Adversarial Taboo conversations with hidden target words, role-specific prompts, broad word sampling, and outcome-based rewards. The reward design is zero-sum, emphasizes later actions, normalizes nonzero episode returns, and may encourage shorter interactions.

A.1 GPT-4 Data Collection

  • GPT-4 is prompted separately as attacker and defender, with game history and role-specific instructions supplied for each next-turn response.Source: §A.1 · p. 15
  • Adversarial Taboo assigns a hidden target word to the attacker, who induces the defender to utter it while avoiding saying it directly; the defender gets one guess.Games run for a randomly selected number of turns between 3 and 8, with ties if nobody wins.Source: §A.1 · p. 15, §A.3 · p. 16
  • The word pool uses the top 50k CoCA words after removing short words and stop words, with GPT-4 collection using the top 30k and open-source self-play using all 50k.Source: §A.2 · p. 15
  • Prompt templates are rewritten into eight expressions and mixed across instruction-tuning and chat formats to reduce overfitting to one prompt template.Source: §A.3 · p. 16

B Reward Design Details

  • The reward is zero-sum, assigns total returns of 1, −1, or 0 to attacker wins, defender wins, or ties, and weights later dialogue actions more heavily through γ.Nonzero total rewards are normalized to unit magnitude for training stability.Source: B Reward Design Details · pp. 16–17
  • The heuristic reward design naturally encourages the players to interact less in order to win the game.Source: B Reward Design Details · p. 17

C.1 Reasoning Evaluation Benchmark

The evaluation uses established reasoning benchmarks with specified few-shot protocols and compares SPAG-related models with non-adversarial game baselines. Additional analyses examine game interactions, utterance length, and advantage estimation.

Benchmark setup

  • Reasoning evaluation uses MMLU, BBH, Mutual, ARC, LogiQA2, WinoGrande, and PIQA through the Language Model Evaluation Harness.MMLU is evaluated 5-shot, BBH 3-shot, and the remaining benchmarks 0-shot.Source: §C.1 · p. 17
  • The benchmark suite spans multi-task knowledge, hard reasoning, dialogue reasoning, logical reading comprehension, commonsense reasoning, and physical commonsense reasoning.The supplied benchmark descriptions define these task scopes across MMLU, BBH, Mutual, LogiQA2, WinoGrande, and PIQA.Source: §C.1 · p. 17

Outcome judgement

  • Game outcomes use rule-based checks over target-word derivatives, including plural and tense forms, to determine whether players violate the game rules.The attacker is checked for speaking any listed target-word form; the defender is evaluated through the corresponding target-word prediction procedure.Source: §C.2 · p. 17

Optimization

  • Advantage estimates use temporal-difference residuals, then approximate attacker and defender advantages by the attacker reward and its negative in the zero-sum game.The approximation assumes equal-level self-play value functions are zero, while the original reward is defined from the attacker’s perspective.Source: D Advantange Value Estimation · p. 18

Baselines

  • The study compares SPAG with 20-Question and Guess-My-City, using GPT-4-derived imitation data and matched 20k dialogue histories for each game.Imitation learning and first-epoch self-play are conducted in these non-adversarial target-word games.Source: E Non-adversarial Game Baselines · pp. 18–19

F Additional Experimental Results

Additional experiments evaluate SPAG on a 150-word test set and provide self-play examples across imitation, SPAG, and GPT-4 models. Training produces shorter games and shorter utterances.

Testing setup

  • The 150-word test set contains manually selected target words from diverse topics for evaluating SPAG models.The old-version game performances and interaction statistics are reported for this testing setup.Source: F Additional Experimental Results · p. 19

Game behavior

  • As SPAG training progresses, dialogue rounds continuously decrease and players’ utterances become shorter on the testing set.The paper presents these trends through the additional game-performance and interaction analyses.Source: F Additional Experimental Results · p. 19

Qualitative examples

Related papers

Loading 2404.10642v3…