A Zeroth-Order Paradigm for LLM Preference Alignment

Peter Chen; Xi Chen; Wotao Yin; Tianyi Lin

Source-linked AI summary

A Zeroth-Order Paradigm for LLM Preference Alignment

Peter Chen, Xi Chen, Wotao Yin, Tianyi Lin

arXiv:2609.19144v1Sep 16, 2026cs.CLcs.AIcs.LG

TL;DR

Direct alignment methods are efficient but can mishandle preference pairs with small likelihood margins through likelihood displacement. ComPO uses comparison-oracle updates, with offline convergence and online reverse-KL coverage guarantees, and experiments report improvements across several model families and direct alignment baselines. The paper’s practical online heuristics remain outside the basic theorem’s guarantees.

Problem
Likelihood displacement motivates alternatives to differentiable preference losses for extracting useful information from small-margin preference pairs.
Method
ComPO perturbs the current policy, uses comparison oracles to estimate directional updates from preference pairs, and extends this mechanism with unlabeled generations for reverse-KL control.
Results
Experiments across Mistral, Llama, Gemma-2, Qwen3, and Gemma-3 report improvements over existing direct alignment methods, including length-controlled win rates, with diagnostics consistent with mitigating likelihood displacement.
Takeaways & Limitations
Small-margin preference pairs can be used as comparative alignment signals rather than directly optimized by a differentiable preference loss.
Takeaways & Limitations
The practical online heuristics are not covered by the basic constrained theorem because they use a length-normalized quantity and do not enforce its hard policy constraint.

Abstract

from arXiv · show

Direct preference alignment methods are widely used to align large language models (LLMs) with human preferences because of their computational and memory efficiency. However, likelihood displacement motivates alternative ways to extract information from preference pairs with small likelihood margins. In this paper, we propose and analyze Comparison-based Preference Optimization (ComPO), a zeroth-order alignment method based on comparison oracles. ComPO extracts directional information from these pairs without directly optimizing a differentiable preference loss on them. We establish a convergence guarantee for its basic offline scheme under smoothness, gradient sparsity, and compatibility between the oracle and a latent objective. We further introduce online ComPO, which retains the offline comparison mechanism and uses unlabeled policy generations for reverse-KL control relative to a reference policy. Following the coverage perspective of preference fine-tuning, we establish a performance guarantee for a basic constrained scheme under local coverage and in-distribution pairwise reward accuracy. Experiments on Mistral, Llama, Gemma-2, Qwen3, and Gemma-3 models demonstrate improvements over existing direct alignment methods, including length-controlled win rates, with pair-level diagnostics providing evidence consistent with mitigating likelihood displacement.

1 Introduction

Direct alignment is efficient and stable but can mishandle small-margin preference pairs, motivating ComPO’s comparison-based alternative. The paper combines offline and online schemes with theoretical guarantees and evaluates them across multiple model families and diagnostics.

Motivation: Likelihood displacement can reduce the absolute likelihood of preferred responses despite increasing their relative margin, potentially shifting probability toward unsafe responses.Small-margin pairs are especially problematic for fixed differentiable preference losses, although they may still contain useful comparative information.
ComPO: ComPO treats noisy preference pairs as comparison signals and estimates update directions from policy perturbations rather than directly optimizing a differentiable preference loss.The oracle checks whether a perturbation raises preferred-response likelihood and lowers dispreferred-response likelihood, then aggregates one-bit signals.
Theory and extensions: Offline ComPO has a best-iterate convergence guarantee under smoothness, gradient sparsity, and oracle compatibility, while online ComPO has a reverse-KL-constrained performance guarantee under local coverage.The online extension uses unlabeled current-policy generations for regularization while retaining offline preference pairs for comparison directions.
Experiments: Experiments cover Mistral, Llama, Gemma-2, Qwen3, and Gemma-3 models, assessing compatibility with direct alignment methods, design choices, damping, replay, and pair-level likelihood diagnostics.The supplied introduction identifies the evaluation scope but does not provide benchmark cell values.
Scope: The journal version extends the preliminary offline work with online ComPO, coverage-based analysis, additional model families, and online regularization and replay experiments.The extension preserves offline preference supervision while adding unlabeled online generations for reverse-KL control.

2 Preliminaries

The preliminaries formalize direct preference alignment, comparison oracles, zeroth-order gradient estimation, and reverse-KL coverage. These definitions explain how ComPO extracts local directional information and constrains online policy updates.

2.1 Direct preference alignment: Direct preference alignment optimizes a policy on preferred and dispreferred response pairs without learning a reward model, typically using a DPO contrastive loss.DPO compares responses through their log-likelihood margin relative to a reference policy.
2.1 Direct preference alignment: DPO increases the relative likelihood margin but may decrease preferred-response likelihood, making fixed margin losses poorly suited to small-margin or similar preference pairs.Noisy pairs can still contain useful information even when direct optimization is problematic.
2.2 Comparison oracles and zeroth-order methods: A comparison oracle returns +1 when a queried policy is no worse under a latent function and −1 when its function value is lower.ComPO uses preference-pair likelihood changes to instantiate this comparison mechanism without observing the latent objective directly.
2.2 Comparison oracles and zeroth-order methods: Zeroth-order gradient estimation uses random perturbations and one-bit comparison labels, with sparse and normalized constraints restricting the estimated direction.For small perturbations, each comparison label approximates a one-bit measurement of the local gradient; sparsity mitigates dimension dependence.
2.3 Reverse KL and local coverage: Online ComPO keeps offline preference pairs for comparison directions and uses unlabeled current-policy generations to estimate reverse-KL regularization relative to a reference policy.The analysis considers policies in a reverse-KL neighborhood and measures reward accuracy through in-distribution pairwise error under local coverage.
2.3 Reverse KL and local coverage: Local reverse-KL coverage requires bounded density-ratio control for policies near the reference, but the reverse-KL constraint alone does not guarantee that coverage condition.Thus, restricting policies to the reverse-KL neighborhood limits where coverage is required without supplying coverage automatically.

3 Main Results

ComPO learns from low-margin preference pairs through comparison oracles rather than direct differentiable preference-loss optimization, with offline and constrained online schemes.

3.1 Offline preference alignment: ComPO replaces first-order preference-loss updates with zeroth-order comparison directions, applying direct alignment to clean pairs and ComPO to noisy pairs.The practical offline scheme perturbs only output-layer weights and uses clipped signed perturbations.
3.1 Offline preference alignment: The preference oracle compares nearby policies by whether preferred-response likelihood rises while dispreferred-response likelihood falls, and aggregates one-bit outcomes into a sparse update direction.The oracle can operate on individual pairs or mini-batches using average likelihood changes.
3.1 Offline preference alignment: Under smoothness, gradient sparsity, and oracle compatibility, the basic offline scheme has a best-iterate convergence guarantee and logarithmic ambient-dimension dependence in comparison queries for fixed sparsity.The theorem assumes an ℓ-smooth objective bounded below and exact solution of the sparse estimator.
3.2 Online ComPO: Online ComPO preserves the comparison direction while using unlabeled current-policy generations to control reverse KL relative to a reference policy.The basic constrained scheme accepts candidates only when they remain feasible, retaining the previous policy otherwise.
3.2 Online ComPO: Under exact reverse-KL evaluation and local coverage, online iterates remain feasible and their performance gap is bounded by in-distribution pairwise reward error.The guarantee requires coverage only inside the reverse-KL neighborhood, and smaller pairwise reward error yields a tighter bound.
3.2 Online ComPO: The practical online scheme is heuristic because its length-normalized divergence and damping rule do not enforce the exact reverse-KL constraint covered by the theorem.Its empirical effects are evaluated separately with replay and online damping.

4 Experiments

Experiments evaluate ComPO as an offline augmentation to DPO and SimPO, examine design and scaling choices, and test online variants. Across models and benchmarks, ComPO improves direct-alignment methods, with diagnostics indicating desired pairwise likelihood changes and practical sparse-update efficiency.

4.1 Offline training for augmenting DPO and SimPO: DPOclean alone does not uniformly outperform DPO, whereas ComPO extracts useful information from filtered low-margin pairs and yields especially consistent AlpacaEval 2 LC gains.The margin is inexpensive to compute but appears to be an imperfect proxy for pair ambiguity.
4.1 Offline training for augmenting DPO and SimPO: Pairwise diagnostics show the oracle-based update makes preferred responses more likely and dispreferred responses less likely, or leaves those likelihoods unchanged, across reported trials.The analysis covers three independent trials for γ ∈ {0.1, 1} on Llama-3-Instruct-8B and Gemma-2-9B-it.
4.1 Offline training for augmenting DPO and SimPO: ComPO improves both AlpacaEval 2 metrics for all three evaluated SimPO models, improves Arena-Hard for two, matches the third, and slightly raises MT-Bench averages.These gains occur without changing SimPO’s original training objective.
4.2 Ablation studies: Increasing perturbations from 800 to 5400 improves mean WR and LC with diminishing gains, while computation time rises and peak memory remains unchanged.Memory stays constant because ComPO accumulates a running average rather than storing all perturbation vectors.
4.2 Ablation studies: Perturbing MLP layers 30–31 alongside the output layer improves all three reported metrics, increasing peak memory from 16.3 GB to 16.7 GB and 600-perturbation time from 50 to 60 seconds.This multi-layer experiment uses Mistral-7B-Instruct and GPT-4.1 as the Arena-Hard judge.
4.2 Ablation studies: The strongest gradient-threshold results retain approximately 1%–6% of entries, while adding noisy pairs from 100 to 300 improves mean AlpacaEval 2 and Arena-Hard performance.Retaining many small entries or filtering almost all entries produces lower performance.
4.2 Ablation studies: ComPO can refine existing DPO checkpoints directly, achieving gains comparable to DPOclean+ComPO while using sparse output-layer updates and modest GPU memory.This supports refining publicly available aligned models with potentially noisy task-specific preference data.
Online scheme: Online damping improves all three reported metrics over offline ComPO for Qwen3-4B-Base, Llama-3.2-3B-Instruct, and Gemma-3-4B-it, while replay further improves every metric for each model.Reported damping gains include 1.23, 1.47, and 0.6 percentage points for Qwen3 across AlpacaEval 2 LC, WR, and Arena-Hard WR.

5 Conclusion

ComPO uses comparison oracles to extract alignment signals from low-margin preference pairs without directly optimizing a preference loss on them. The paper reports improvements over direct alignment methods and develops an online extension with reverse-KL regularization and coverage-based guarantees.

ComPO improves LLMs using noisy preference pairs whose preferred and dispreferred responses have similar reference-policy likelihoods.It treats these pairs as comparison signals rather than directly optimizing a preference loss.
Experiments across multiple models and benchmarks show improvements over existing direct alignment methods, with pair-level diagnostics consistent with mitigating likelihood displacement.
Online ComPO keeps offline pairs for comparison directions and uses unlabeled current-policy generations for reverse-KL regularization.The paper establishes feasibility and a coverage-based performance bound for the basic constrained scheme, and evaluates damping and replay practically.

Appendix A. Further Related Work

The related-work appendix situates ComPO among preference learning, preference-method analysis, zeroth-order optimization, likelihood-displacement studies, and learning from noisy preference data. It emphasizes that ComPO targets low-margin pairs and complements rather than replaces existing methods.

Preference learning methods: Prior preference-learning work includes DPO extensions, preference-data augmentation, token-level formulations, and methods addressing offline-pair quality.
Analysis of preference learning methods: Analyses of preference learning study contextual-bandit formulations, KL-regularized exploration, and sample-complexity guarantees for online methods.
Zeroth-order optimization methods: Zeroth-order optimization commonly estimates gradients with one-point or two-point estimators, with guarantees spanning convex, nonsmooth convex, and smooth nonconvex settings.
Likelihood displacement: Likelihood displacement has been attributed to similar preferred and dispreferred responses under several metrics, alongside other proposed factors such as the initial SFT model and multiple training samples.
Learning from noisy preference data: ComPO addresses low-margin pairs rather than corrupted preference labels and is intended as a complementary modular component that enhances robustness.

Appendix B. Missing Proofs

The appendix supplies technical lemmas used to prove the paper’s main theoretical results.

Several technical lemmas are presented and then used to prove Theorem 3.2 and Theorem 3.4.

B.1 Technical lemmas

The technical-lemma appendix establishes concentration and descent ingredients for offline comparison-based estimation and convergence. Its results combine sparse-gradient geometry, oracle compatibility, and iterative descent bounds.

The offline technical assumptions include smoothness, gradient sparsity, oracle compatibility, and normalized-gradient constructions.
Under oracle compatibility, one-bit labels match the sign of the latent objective change, enabling comparison-based direction estimation.
The lemmas establish that sufficiently distant feasible vectors have lower empirical objective, yielding a best-direction estimation bound.
The proof controls directional-error terms through moment and tail bounds, including concentration that remains valid even when labels depend on perturbations.
The estimator concentrates around the target direction using spherical perturbations, sparse-support nets, block decompositions, and union bounds.
For iterative updates, the descent analysis bounds the minimum gradient norm across T iterations by an objective decrease term and a step-size error term.The guarantee is stated with probability at least 1−Λ under the lemma’s prescribed perturbation and parameter conditions.

B.2 Proof of Theorem 3.2

The proof derives a bound on the minimum gradient norm over T iterations, using a case distinction and a high-probability lemma. This establishes the stated result when the iteration and step-size terms are sufficiently small.

When the relevant case condition holds, Lemma B.3 provides the needed statement with probability at least 1 −Λ.
The proof bounds the minimum gradient norm by 2(f(θ1)−f(θT +1))/(ηT) + ℓη.
The argument concludes by combining the cases and completing the proof.

B.3 Proof of Theorem 3.4

The proof preserves feasibility of all iterates in Πτ and relates the relevant second moment to the pairwise reward error. Under local coverage, sufficiently small error yields the target guarantee.

All iterates remain in Πτ because Eq. (12) either accepts a feasible policy or retains the previous one.
The proof evaluates reward error using a joint distribution pairing each candidate policy with the current policy.
Local coverage compares this joint distribution with reference-policy pairs to bound the second moment by err(πθt).
Whenever err(πθt) ≤ϵ, the resulting bound holds for every t = 1, . . . , T + 1.

Appendix C. Additional Case Studies

The case studies compare response presentation across alignment methods and their ComPO refinements. The examples show framing and organization changes, but do not establish systematic gains in safety, factual accuracy, or mathematical ability.

Overview: The qualitative examples illustrate response presentation rather than systematic improvements in safety, factual accuracy, or mathematical ability.
Harmful-content example: ComPO adds a cautionary preface in the harmful-content example, changing framing without establishing safer behavior.
Helpful-content example: DPOclean+ComPO organizes the helpful library recommendation into explicit pros and cons.
Helpful-content example: Additional detail in the library response does not by itself establish factual correctness.
Budget example: Both budget responses preserve the same relation and acknowledge that the available information does not determine unique numerical amounts.