Language Alignment via Nash-learning and Adaptive feedback

1. Introduction

• The paper argues that iterative self-reward training can be robust to noisy mechanisms that might otherwise produce biased, sub-optimal demonstrations.The authors identify noisy self-evaluation as a risk and state that direct learning from such data does not guarantee a better optimal model.Source: §1 · p. 1
• The proposed alternative updates LLM policies through adaptive feedback from improved opponents without a learned preference model or human-annotated preference data.Earlier works are described as assuming a preference model learned in advance.Source: §1 · p. 1
• Unlike related work that avoids faithful game-theoretic updates such as two-timescale updates, the paper argues these are feasible for LLM alignment given a shared foundation-model worldview.The paper says prior work avoids these updates to sidestep complex hyperparameter tuning and unstable performance.Source: §1 · p. 1
• The method differs from Munos et al. (2023) by using modified Mirror Descent with KL regularization relative to a reference policy, while the proposed method is reference-policy-free.The passage states both the distinction and the reference-policy-free property.Source: §1 · p. 1
• The paper distinguishes its alignment-training approach from Chen et al. (2024), which proposed self-play for supervised fine-tuning rather than alignment training.Source: §1 · p. 1

2. Language Alignment via Nash-learning and Adaptive feedback

• Mirror ascent with an improved opponent defines a sequence of policies for a two-player zero-sum game, using the opponent reward and a KL-based Bregman divergence in the update.The update also uses a learning rate.Source: §2 · p. 2

2.1. Sampling from improved Opponent ˜π

• Improved opponents may take forms including greedy policies, best responses, Monte Carlo tree search, or extra-gradient methods.Source: §2.1 · p. 2
• For a prompt, the policy samples two responses and queries the opponent with an evaluation prompt; the opponent's output-token probabilities determine which answer is preferred.Each player treats the preferred answer as an improved-opponent sample and the rejected answer as its own, then aims to maximize expected win rate.Source: §2.1 · p. 2
• The setup can produce noisy outcomes, including an opponent choosing incorrectly, and its preference measure for a response can change as model parameters update.Source: §2.1 · p. 2
• The procedure samples prompts and response pairs, optimizes each policy with SGD using a log-probability-ratio loss, and repeats until convergence or compute runs out.Source: §2.1 · p. 2
• With a correctly chosen proxy reward and slower learning dynamics, the game is reported to converge to a better policy.Source: §2.1 · p. 2

• Rewards are evaluated adaptively at each time step using the players’ policies.Source: §3 · p. 2
• Plugging adaptive feedback into the optimization yields the Language Alignment via Nash-learning and Adaptive feedback algorithm, or LANA.Source: §3 · p. 2
• LANA can also be viewed as an online, two-player, reference-free and sigma-free DPO variant that needs no human-annotated preference dataset.Source: §3 · p. 2

4.1.1. DATA

• The experiments used 3K randomly selected prompts, using prompts rather than generated responses, and tested with UltraFeedback and task categories from Huang et al.Source: §4.1.1 · p. 2

4.1.2. BASE MODEL

• Most experiments used Phi-3-mini-4k-instruct, a 3.8B-parameter mini model chosen to support faster experimentation amid resource constraints.Source: §4.1.2 · pp. 2–3

4.1.3. HYPERPARAMETERS

• Responses were generated at temperature 0.1 with a 128-token maximum, prompts were limited to 256 tokens, SGD used a 0.0003 learning rate and batch size 4, and both players used rank-16 LoRA.The generation and prompt limits were due to resource limitations.Source: §4.1.3 · p. 3

4.2. Results

• LANA showed noticeable improvement on Alpaca evaluation after training on 3K prompts without a human-annotated preference dataset.Source: §4.2 · p. 3, Table 1 · p. 3
• In MT-Bench, LANA seemed to perform better on GSM8K and match GPT-3.5-turbo despite using a 3.8B-parameter mini model, without a noticeable reasoning-task drop.The reported evaluation also includes the LLM-Evaluation Harness and MT-Bench.Source: §4.2 · p. 3, Table 2 · p. 3
• LANA improved win rates across all tested task categories, with larger gains in riddles, theory of mind, and planning.Source: §4.2 · p. 3

4.3. Ablation study

• The ablation found negligible improvements for Mistral-7B-Instruct-v0.1 and Gemma-2b-it; the authors suspect base-model choice may matter.They suggest noisier self-evaluations from lower-quality training data could cancel alignment progress, while noting other explanations cannot be ruled out.Source: §4.3 · p. 3

5.1. LANA loss derivation

• For the KL-divergence case, the mirror map is the gradient of a convex function, and LANA performs gradient steps in policy log-likelihood space rather than LLM-weight space.The derivation combines the mirror-map formulation with the reward and minimizes an upper bound by omitting the entropy term; the cross-entropy term remains in the objective.Source: §5.1 · pp. 3–4

5.2. Convergence

• MT-Bench results show no noticeable drop in reasoning tasks and improved GSM8K performance matching GPT-3.5-turbo, while other tasks are slightly affected.The caption reports this comparison for Figure 2.Source: Figure 2 · p. 5
• Under a bounded-norm assumption, the analysis establishes average convergence to a Nash equilibrium, but does not guarantee last-iterate convergence unless DKL(π∗, ˜πc) = 0.The bounded-norm condition can be ensured, for example, by keeping policy probabilities from having zero support.Source: §5.2 · p. 5

5.3. Future works

• Further tests across different and larger models are needed, and limited compute restricted training to 3K prompts, leaving the gains from longer instruction training unknown.The authors also note that sampling during training is highly inefficient and PagedAttention is not applicable during training.Source: §5.3 · p. 5