PaperlayerView exact revision on arXiv

Source-linked AI summary

Latent Adversarial Regularization for Offline Preference Optimization

Enyi Jiang, Yibo Jacky Zhang, Yinglun Xu, Andreas Haupt, Nancy Amato, Sanmi Koyejo

arXiv:2601.22083v2cs.LGcs.AI

TL;DR

Offline preference optimization commonly regularizes policies in token space, although token-level similarity can diverge from semantic or behavioral similarity. The paper introduces GANPO, which uses adversarial discrimination between policy and reference latent representations as a plug-in regularizer. Across architectures and tasks, GANPO reports consistent improvements, stronger robustness to noise and distributional shift, and comparable downstream performance with modest overhead.

  • Problem

    Token-space regularizers can poorly reflect semantic or behavioral similarity, motivating structural alignment in latent space.

  • Method

    GANPO adds adversarial latent-space regularization to offline preference objectives by discriminating policy and reference internal representations.

  • Results

    GANPO consistently improves OPO-style methods across architectures and tasks, better preserves representation geometry, and maintains comparable downstream performance with modest overhead.

  • Takeaways & Limitations

    Latent adversarial feedback provides structural alignment signals and greater robustness to stochastic sampling noise and distributional shifts than token-level constraints.

  • Takeaways & Limitations

    GANPO requires maintaining and tuning a discriminator, and reliance on an SFT-defined target manifold can inherit defects in a misaligned reference structure.

Abstract

from arXiv · show

Learning from human feedback typically relies on preference optimization that constrains policy updates through token-level regularization. However, preference optimization for language models is particularly challenging because token-space similarity does not imply semantic or behavioral similarity. To address this challenge, we leverage latent-space regularization for language model preference optimization. We introduce GANPO, which achieves latent-space regularization by penalizing divergence between the internal representations of a policy model and a reference model. Given that latent representations are not associated with explicit probability densities, we adopt an adversarial approach inspired by GANs to minimize latent-space divergence. We integrate GANPO as a regularizer into existing offline preference optimization objectives. Experiments across multiple model architectures and tasks show consistent improvements from latent-space regularization. Further, by comparing GANPO-induced inferential biases with those from token-level regularization, we find that GANPO provides more robust structural feedback under distributional shift and noise while maintaining comparable downstream performance with minor computational overhead.

1. Introduction

Token-level regularization can misrepresent semantic or behavioral similarity, motivating latent-space regularization for offline preference optimization. GANPO adds adversarial latent alignment to existing objectives, with experiments reporting consistent improvements and structural robustness.

  • Motivation: Token-space divergences can assign large distances to semantically similar outputs and small distances to semantically different outputs.This makes token-space regularization a coarse proxy for behavioral similarity.
  • Approach: GANPO penalizes divergence between policy and reference latent-representation distributions using a latent-space discriminator.The discriminator supplies adversarial regularization alongside the preference objective.
  • Approach: GANPO extends the adversarial formulation with two discriminators that jointly distinguish high-quality and low-quality representations.The quad representation framework retains the original offline preference optimization objective.
  • Results: GANPO produces consistent improvements across model architectures and tasks when plugged into OPO-style methods on AlpacaEval-2.0.The contribution statement reports this pattern across diverse architectures and tasks.
  • Results: GANPO better preserves internal-representation geometry, improving robustness to stochastic sampling noise and distributional shifts while maintaining comparable downstream performance.The reported computational cost is modest.

3. Latent Adversarial Regularization

Latent adversarial regularization replaces intractable density-based divergence estimation with discriminator-based comparison of policy and reference representations. GANPO implements this regularization using variational and relativistic-average GAN formulations.

  • Latent-space formulation: GANPO applies regularization to final-layer hidden representations rather than token probabilities.The policy and reference produce representation distributions pθ and pref from prompt-response pairs.
  • Latent-space formulation: Token-space KL divergence can be a coarse behavioral-similarity proxy, whereas sequence-level representations provide global semantic structure for alignment.The discriminator operates on the representation of the entire sequence.
  • Adversarial formulation: Latent KL divergence is generally intractable because latent representations typically lack explicit probability densities.This motivates a variational adversarial formulation.
  • Adversarial formulation: Relativistic-average GANs estimate whether a real sample is more realistic than the current batch’s average fake sample.This formulation is adopted to improve optimization stability over the standard GAN objective.
  • Adversarial formulation: The discriminator distinguishes reference representations from policy representations, converting latent divergence estimation into a binary cross-entropy classification objective.Reference representations receive label 1 and policy representations label 0.

4. GAN Preference Optimization

GANPO uses paired chosen and rejected responses to construct a quadruple of latent representations and trains comparative discriminators for good and bad representations. The policy then minimizes the resulting adversarial latent regularization.

  • Representation construction: GANPO extracts last-layer hidden representations from chosen and rejected responses under both the policy and reference models.The resulting tuple includes reference-good, reference-bad, and policy-good representations.
  • Representation construction: Reference positive and negative representations serve as latent-space anchors for comparative discrimination.The relativistic-average discriminator compares representations rather than evaluating them in isolation.
  • Dual-contrastive objective: Two discriminators model good and bad latent-representation distributions.This extends binary real-fake discrimination to preference-structured comparisons.
  • Policy optimization: The policy minimizes latent-space regularization against trained positive and negative discriminators.The generator-optimization step uses the adversarial losses associated with both representation types.
  • Algorithm: GANPO is implemented with a preference dataset, policy, reference model, two discriminators, adversarial weight, and moving-average decay hyperparameter.Algorithm 1 initializes positive and negative global running means before training iterations.

1. Data Sampling

The supplied passage shows a sampled preference batch, but does not provide enough context to explain the complete data-sampling procedure.

  • Data Sampling: A training step samples a batch B of prompt, chosen-response, and rejected-response triplets from D.The passage gives the batch notation but no further sampling details.

2. Feature Extraction (Latent Space)

GANPO extracts latent representations from both the policy and frozen reference models as inputs to its latent-space regularization procedure.

  • The procedure obtains last hidden states from the policy model with gradients and from the frozen reference model.

3. Discriminator Optimization (Relativistic)

GANPO trains discriminators to distinguish policy and reference representations, then alternates discriminator updates with policy optimization using offline preference and adversarial losses.

  • Raw logits are computed from all four representations, and global running means are updated with moving averages.
  • Positive and negative discriminator losses are computed and used to update both discriminators.
  • The policy combines the offline preference-optimization loss with a weighted adversarial loss before updating its parameters.
  • The generator and discriminators are optimized alternately in the adversarial training procedure.
  • The latent discriminator supplies dense structural feedback and constrains the policy toward the reference manifold under distributional shift.
  • Reference-anchored training promotes manifold overlap, encouraging discriminators to learn semantic distinctions and provide informative gradients.
  • Using the reference model avoids the expense of sampling an external teacher and supports a fully offline training loop with small additional overhead.

5. Experiment

Experiments evaluate GANPO across architectures, model scales, tasks, decoding temperatures, and discriminator designs, finding stronger preference alignment and robustness with modest overhead but reference-dependent limitations.

  • Experimental setup: GANPO is evaluated on instruction following, robustness across architectures and scales, stochastic decoding, downstream tasks, and discriminator effectiveness.
  • Preference alignment: +1.41% LC-Win over DPO and +0.71% LC-Win over SimPO are reported for Gemma2-2B-it on AlpacaEval-2.0.
  • Preference alignment: GANPO consistently improves alignment over non-adversarial counterparts across model scales while maintaining comparable response lengths.
  • Stochastic decoding: At T ≥1.0, GANPO retains better quality under stochastic sampling, while DPO becomes increasingly brittle as entropy rises.
  • Stochastic decoding: DPO’s IFEval strict accuracy drops nearly 20% from T = 0.0 to T = 1.0, whereas GANPO retains good accuracy under stochastic sampling.
  • Stochastic decoding: GANPO maintains preference alignment and constraint adherence when generation trajectories deviate from the optimal path.
  • Downstream evaluation: GANPO does not degrade and sometimes improves math, reasoning, and factuality performance beyond preference-alignment benchmarks.
  • Discriminator architecture: Transformer-based discriminators consistently outperform fixed MSE critics and shallow MLPs across the experiments.

6. Related Work

Prior alignment methods span RLHF, offline preference optimization, GAN-based distribution matching, and latent-space optimization. GANPO combines offline preference learning with adversarial latent-space regularization to improve structural robustness and alignment.

  • Preference optimization: Offline preference optimization methods seek simpler alternatives to complex online methods, with GANPO compared against DPO and SimPO.GANPO is evaluated as an offline alignment method with modest additional computational cost.
  • GAN-based methods: GANPO introduces a GAN-style adversarial regularizer operating in latent space to complement offline preference optimization and mitigate exposure bias and structural degradation.This approach differs from prior GAN work while retaining its adversarial distribution-matching framework.
  • RLHF: RLHF commonly uses supervised fine-tuning, reward modeling, and policy optimization, whereas GANPO is fully offline and orthogonal to RLHF.The cited passage also notes unintended verbosity and length-based reward hacking in RLHF.
  • GANPO: GANPO augments preference learning with adversarial regularization, using latent-space alignment and a dual-contrastive objective to guide policies toward high-quality structural modes.Reference-anchored training is described as stabilizing and computationally efficient.
  • Conclusion: GANPO improves offline alignment performance and robustness while introducing no anticipated societal or ethical impacts beyond those commonly associated with LLM training and alignment.The paper reports improvements through adversarial structural regularization.

A. Relativistic Average Divergence

The relativistic average divergence is presented as a well-defined divergence, with its discriminator-based term defined in the main paper and justified through concavity and sigmoid identities.

  • Definition: A statistical divergence is a nonnegative function that equals zero exactly when the two distributions are identical.The definition applies to distributions sharing support X.
  • Divergence construction: The relativistic average divergence is introduced through a supremum over discriminator functions applied to two distributions.The appendix states that the resulting quantity is well-defined under the proposition’s conditions.
  • Main-paper term: Equation 5 uses a discriminator scalar logit, latent representation h, the sigmoid function, and average baselines to define the divergence term.The discriminator is parameterized by ϕ and operates over the representation space.
  • Well-definedness: When the discriminator ranges over all functions from representation space H to R, the proposed term is a divergence between pθ and pref.This claim is stated as Proposition A.3.
  • Proof: The proof verifies strict concavity, differentiability, normalization at zero, and the required supremum properties for the chosen function.The sigmoid identity 1 − σ(x) = σ(−x) is used to replace the log-sigmoid function by f.

B. Implementation Details and Hyperparameters

Implementation uses fixed preference-optimization settings, model-specific generator learning rates, lightweight Transformer discriminators, and experiments on specified GPU configurations.

  • Training settings: Preference optimization uses batch size 128, one training epoch, maximum sequence length 2048, and cosine learning-rate scheduling with 10% warmup.These settings apply to the preference-optimization dataset.
  • Learning rates: Generator learning rates are 5.0e−7 for Gemma2-2B-it and 1e−6 for Llama-3-8B-Instruct.The passage reports separate rates for the two model families.
  • Discriminator: The discriminator uses a Transformer architecture with two layers for Gemma2-2B-it and four layers for Llama3-8B-Instruct.It processes continuous policy latent representations projected through a spectrally normalized linear layer.
  • Hardware: Training experiments were conducted on either 2×H200 or 4xA100 GPUs.These configurations define the reported computation environment.

C. Computational Cost Analysis

GANPO adds modest computational overhead to DPO- and SimPO-style pipelines while producing stronger preference margins and more stable win rates across response lengths.

  • Training cost: GANPO increases Gemma2-2B-it DPO training time by less than 4% on identical hardware and remains within the same GPU budget for SimPO.It requires neither additional rollout generation nor external teacher queries.
  • Win rates: GANPO maintains stable, consistently higher win rates than DPO for medium and long responses, where DPO degrades as response length increases.The passage links this pattern to reduced verbosity bias and improved preference alignment.
  • Preference margins: GANPO achieves larger preference margins than DPO and SimPO across models and objectives, with margins increasing steadily during optimization.The comparisons cover Gemma2-2B-it and Llama-3-8B-Instruct through Figures 6–9.
  • Implementation: The discriminator is implemented with a Transformer architecture for latent representations.Figure 10 presents the PyTorch implementation.
Loading 2601.22083v2…