PaperlayerView exact revision on arXiv

Source-linked AI summary

PLC-DPO: Posterior Label Correction in Noisy and Ambiguous Preference Optimization

Boryeong Cho, Sumyeong Ahn, Se-Young Yun

arXiv:2608.30597v1cs.LGcs.CL

TL;DR

DPO assumes observed pairwise preferences are reliable, although real data can contain reversed, weak, or ambiguous labels that produce harmful updates. PLC-DPO uses a calibrated policy-reference margin to route each pair as clean, flip, or tie and apply the corresponding correction. Across 57 dataset–model–benchmark cells, it achieves the highest mean win rate among robust baselines, while stress tests and diagnostics report stable routing and distinct treatment of reversed and weakly directional pairs.

  • Problem

    DPO assumes every observed preference is reliable and directional, despite annotator disagreement, judge bias, superficial cues, and weakly informative pairs.

  • Method

    PLC-DPO models each observed pair as a latent clean, flip, or tie state and routes training through forward, reversed, or tie-regularizing losses using a calibrated policy-reference margin.

  • Results

    Across 57 dataset–model–benchmark cells, PLC-DPO achieves the highest mean win rate and the second-highest worst-cell result among robust baselines.

  • Takeaways & Limitations

    PLC-DPO corrects supervision direction and strength rather than merely filtering suspicious pairs, distinguishing likely flips from weakly directional cases.

  • Takeaways & Limitations

    PLC-DPO may be less informative when data are extremely small, models are severely under-trained, routing entropy remains high, or policy and reference share systematic errors.

Abstract

from arXiv · show

Direct Preference Optimization (DPO) simplifies alignment through pairwise comparisons but assumes all observed preferences are reliable. Real data often violates this assumption, leading to reversed, weak, or ambiguous labels that cause harmful policy updates. To address this, we propose Posterior Label Correction DPO (PLC-DPO) to robustly optimize preferences by routing each pair's training signal as a clean, flip, or tie case. The key idea is to use the calibrated policy-reference margin as online evidence to take appropriate correction actions. This reframes noisy preference learning as actively correcting supervision direction and strength rather than merely filtering suspicious examples. Across 57 dataset-model-benchmark cells, PLC-DPO obtains the best mean win rate against DPO (60.5 vs. 55.5 for the next-best method). Injected-noise and tie stress tests, human disagreement analysis, and self-confirmation diagnostics further show that the routing remains stable and distinguishes flipped from weakly directional pairs.

1 Introduction

DPO offers a simple, scalable preference objective but assumes observed pairwise labels are reliable and directional. PLC-DPO addresses noisy supervision by routing each pair as clean, flipped, or tie and mixing corresponding corrective losses.

  • Motivation: DPO directly fits a Bradley-Terry preference model using the policy-reference log-ratio, avoiding a separate reward model.Its closed-form pairwise loss is simple, scalable, and suited to large preference datasets.
  • Motivation: Annotator disagreement, judge bias, and superficial cues can make observed preference labels unreliable.
  • Motivation: Non-directional pairs can be similarly good, similarly flawed, or too marginally different to support a stable binary signal.Treating these labels as absolute truth can reinforce incorrect optimization directions.
  • Related approaches: Existing robust-loss, filtering, and latent-quality methods address uncertainty indirectly through global corrections, data removal, or absolute response scores.Filtering can discard useful signals that might instead support label recalibration.
  • Approach: PLC-DPO models each observed label as a latent clean, flip, or tie state and estimates routing from the calibrated policy-reference preference margin.Clean pairs are reinforced, flipped pairs reversed, and ties prevented from producing strong directional gradients.
  • Evaluation: 57 dataset–model–benchmark cells show PLC-DPO achieves the highest mean win rate and second-highest worst-cell result among robust baselines.The training recipe combines EMA calibration, warm-up, routing, and confidence-gated mixing to avoid early instability.

2 Related Work

Related work improves preference optimization objectives and robustness to noisy labels, but many methods retain observed pair directions or remove uncertain data. PLC-DPO instead supplies an explicit training procedure for routing and correcting pairwise supervision.

  • Preference optimization: RLHF trains an explicit reward model before optimizing the policy with PPO, whereas DPO uses a closed-form loss based on the policy-reference log-ratio.
  • Preference optimization: KTO, SimPO, and RSO modify supervision, reward parameterization, or sampling, but objective-level advances typically keep the observed pair direction fixed.
  • Robust preference optimization: Robust preference methods address noisy, weak, or biased labels through label smoothing, robust losses, distributional robustness, filtering, dynamic margins, or EM-style reweighting.
  • PLC-DPO procedure: PLC-DPO training takes policy and reference log-probabilities as inputs, calibrates the resulting margin with batch statistics and an EMA state, and computes routing evidence.
  • PLC-DPO procedure: The routing weights mix clean, flip, and tie state losses with standard DPO, while warm-up increases correction strength only after an initial period.

3 Posterior Label Correction DPO

PLC-DPO treats each observed preference direction as potentially clean, flipped, or tied, then uses calibrated policy-reference margins to route and confidence-weight the training signal. It blends corrected and standard DPO losses with warm-up and achieves strong win-rate performance across reported evaluations.

  • Latent pair states: PLC-DPO reframes noisy preference optimization as an online latent-label problem over clean, flip, and tie states.The states determine whether to reinforce the observed ordering, reverse it, or avoid a strong directional gradient.
  • Margin-based routing: The method routes each pair using a detached, online-calibrated version of the policy-reference margin used by DPO.Large positive margins support the observed label, negative margins support a flip, and values near zero provide weak directional evidence.
  • State-conditional objectives: Clean, flip, and tie losses respectively reinforce the observed direction, reverse it, or discourage large directional margins.These state-conditional losses are combined according to the routing distribution.
  • Warm-up and mixing: PLC-DPO begins with standard DPO during warm-up, then increases correction strength and blends the routing-corrected loss with the DPO loss.Warm-up addresses initially weak margins and uncalibrated EMA statistics.
  • Confidence gating: Routing confidence gates how strongly corrections affect training, with low-confidence assignments deferred and dominant actions receiving greater weight.The maxweight confidence gate is small for near-uniform routing and large when one latent state dominates.

4 Experiments

Experiments evaluate PLC-DPO across alignment benchmarks, models, datasets, injected label noise, and routing diagnostics. Results show strong average performance, robustness to flipped labels, and distinct routing behavior for directional reversals versus weak preferences.

  • Main Alignment Results: PLC-DPO performs strongly across models and evaluation sets, with particularly large gains for Qwen2.5-7B.The reported gains are especially notable on AlpacaEval 2, Vicuna, Evol-Instruct, and HH-RLHF, and improvements are larger for larger models.
  • Generalization Across Models and Datasets: PLC-DPO achieves the highest overall mean win rate across 57 dataset–model–benchmark cells.Its mean win rate is 60.5, compared with 55.5 for the next-best method, rDPO.
  • Robustness to Injected Label Noise: PLC-DPO remains strongest at every tested injected flip rate, including η = 0.30.The Vicuna evaluation uses Qwen2.5-1.5B and measures pairwise win rate against the clean one-epoch DPO baseline.
  • Routing Diagnostics: As hard-pair corruption increases, cumulative qclean decreases from 0.676 to 0.495 while qflip increases from 0.265 to 0.470.qtie remains low because the intervention creates directional reversals rather than weak-gap ambiguity.
  • Tie-State Selectivity: Tie-state mass is significantly higher for the weakest-gap held-out pairs than for the strongest-gap pairs.The corresponding margin magnitude |m| follows the opposite trend, indicating stronger tie selectivity for weak preferences.
  • Tie-State Selectivity: Under injected tie rates from 0% to 30%, PLC-DPO’s margin over same-data DPO increases from +8.3 to +18.5 points.On the independent MultiPref dataset, mean qtie rises from 0.0811 for unanimous pairs to 0.0997 for tie-majority pairs.

5 Discussion

PLC-DPO corrects preference supervision by routing each pair as clean, flipped, or tied, rather than filtering uncertain examples. Its calibration and gating stabilize this routing while preserving useful signals from reversed and ambiguous pairs.

  • Correction rather than selection: The clean/flip/tie decomposition assigns distinct gradient actions to reliable, reversed, and ambiguous pairwise preferences.Clean pairs are reinforced, flipped pairs reversed, and tied pairs prevented from inducing strong directional updates.
  • Correction rather than selection: PLC-DPO retains useful information by reversing likely flipped pairs and neutralizing ambiguous pairs instead of discarding both.A confidence gate defers low-confidence examples while reverse DPO and tie losses target distinct failure modes.
  • Routing stabilization: EMA calibration, stop-gradient routing, warm-up, and confidence gating stabilize the model-dependent, nonstationary policy-reference margin.These choices prevent routing weights from changing with the objective and keep weak early evidence from dominating optimization.
  • Routing diagnostics: Under 20% corruption, the final flip-marker AUROC is 0.731, while qflip predicts independent reward-model disagreement at 0.779 AUROC on clean pairs.With a 30% warm start, final flip-marker AUROC is 0.734.
  • Interpretation: Routing weights indicate the currently supported training action, not a model-independent audit of whether labels are clean, flipped, or tied.A high qtie denotes weak directional evidence under the calibrated policy-reference margin and energy scores.
  • Relation to response-level latent quality: PLC-DPO is narrower than response-level latent-quality routing because it predicts whether the observed pair direction should be reinforced, reversed, or neutralized.This choice connects directly to the pairwise gradient consumed by DPO without requiring an absolute response-quality signal.
  • Fallback behavior: PLC-DPO safely reverts to standard DPO when data, training, or routing confidence is insufficient for informative correction.The largest gains occur when training signals make the policy-reference margin highly informative.

6 Conclusion

PLC-DPO treats each observed preference label as a latent clean, flip, or tie state and uses calibrated routing to mix directional and tie-regularizing DPO objectives. Across 57 dataset–model–benchmark cells, it achieves the highest overall win rate among the robust baselines.

  • PLC-DPO treats each observed preference pair label as a latent clean, flip, or tie state.
  • The method estimates routing from the EMA-calibrated policy-reference margin, stops routing gradients, and mixes forward, reversed, and tie-regularizing losses.Warm-up and confidence gating blend correction with standard DPO.
  • Across 57 dataset–model–benchmark cells, PLC-DPO achieves the highest overall win rate and the second-highest worst-cell result among robust baselines.

Limitations

PLC-DPO’s routing depends on a current policy–reference margin that can remain imperfect when both models share systematic errors. The appendix provides additional diagnostics and implementation details, while reported comparisons use matched preference data without benchmark-specific configurations.

  • The policy–reference margin may be imperfect when the policy and frozen reference share systematic errors.Performance under distribution shift may benefit from longer warm-up or more adaptive calibration.
  • The paper identifies routing error, EMA adaptation, and gating under jointly evolving policy and routing dynamics as areas needing further theoretical characterization.Establishing guarantees would deepen understanding of the mechanism’s robustness.
  • The appendix contains additional cross-model, cross-dataset, tie-state, self-confirmation, sensitivity, runtime, and reproducibility details.
  • Unless otherwise specified, results report win rates against DPO trained on the same preference data without benchmark-specific configuration for each method.

A.1 Dataset Transfer and Seed Stability

PLC-DPO transfers across several preference datasets and evaluation settings, including comparatively clean and naturally noisy data. Across 57 cells it obtains the highest overall win rate, with especially strong aggregate performance on three datasets.

  • Dataset transfer: HH-Golden evaluates PLC-DPO on higher-quality preferred responses, while HH-RLHF exposes methods to naturally noisy, older responses.Nectar-60k and ORPO-mix-40k test whether correction harms comparatively clean preference data.
  • HH-Golden: 22.7 points, 23.1 points, and 30.4 points are PLC-DPO’s HH-Golden improvements over DPO for Qwen2.5-7B, Llama-3-8B, and Mistral-7B, respectively.For Mistral-7B, PLC-DPO reaches 80.4 versus 78.7 for rDPO.
  • HH-RLHF: At roughly 45% of HH-RLHF training, PLC-DPO’s in-domain win rate remains at 87.8, compared with 62.0–73.8 for baselines.PLC-DPO degrades substantially later, and it is the only method whose best-checkpoint in-domain reward remains above the SFT starting point.
  • Evaluation coverage: The cell-level tables cover UltraFeedback, HH-Golden, Nectar-60k, and ORPO-mix-40k across evaluated model–benchmark pairs.The reported aggregate means are summarized over these 57 cells.
  • Aggregate results: 60.5 is PLC-DPO’s highest overall win rate across 57 cells, followed by rDPO at 55.5.PLC-DPO is strongest on UltraFeedback, Nectar-60k, and ORPO-mix-40k, while rDPO is strongest on HH-Golden.

A.3 Cross-Dataset Label-Noise Stress Tests

PLC-DPO’s correction advantage grows under directional corruption and remains selective for ties and flips. Diagnostics also indicate stable routing, limited early correction, and modest sensitivity to recipe choices and runtime.

  • Label-flip stress tests: 72.3 and 69.5 are PLC-DPO’s win rates at 30% noise on Nectar-60k and ORPO-mix-40k, respectively.On clean data, performance is comparable to DPO; the advantage becomes more pronounced as corruption increases.
  • Tie stress tests: +8.3 to +18.5 points is PLC-DPO’s increasing margin over DPO across injected-tie rates.At 5% tie injection, qclean − qflip on injected pairs falls by a factor of 4.5.
  • Human disagreement: Human-disagreement routing is graded: qtie is highest for tie-majority pairs, intermediate for split preferences, and lowest for unanimous preferences.The tie channel transfers beyond the weak-gap proxy used in the main paper.
  • Routing diagnostics: 0.731 is the final cumulative flip-marker AUROC under 20% label-flip corruption, with mean qflip 0.473 on flips versus 0.197 on unmodified pairs.On 8,000 held-out clean UltraFeedback pairs, qflip predicts independent reward-model disagreement with AUROC 0.779 [0.766, 0.791].
  • Training stability: 1.9% of total correction weight occurs in the first 20% of training, while mean correction weight is 0.026 early versus 0.286 overall.Extending warm-up from 7% to 30% keeps flip-marker AUROC comparable, 0.734 versus 0.731, while improving the three-benchmark mean from 71.0 to 73.6.
  • Recipe sensitivity: 59.3 is the overall mean across 24 retraining configurations, all outperforming the DPO reference of 50.0.The fixed default configuration reaches 58.3, while τdir and γmax cause the largest performance variation.

B.1 Cross-Benchmark Injected-Noise Radar

The cross-benchmark radar evaluates PLC-DPO under injected label noise across evaluation sets, using fixed recipes and DPO-relative win rates. Supporting materials specify the datasets, corruption procedure, judge audit, and recipe comparisons.

  • Radar design: PLC-DPO uses the fixed default recipe in Figure 4 rather than selecting a best recipe per noise rate or evaluation set.The figure expands the selected Vicuna view to all evaluation sets for Qwen2.5-1.5B.
  • Radar design: Each Figure 4 radar axis is an evaluation set, each panel is a label-flip rate, and values are win rates against the clean one-epoch DPO baseline.Axis-specific rings scale from the smallest to largest plotted method value for each dataset and noise rate.
  • Recipe comparison: Figure 6 compares balanced, aggressive, and conservative PLC-DPO recipes with DPO, rDPO, and Dr.DPO across evaluation sets.DPO is fixed at 50 because all win rates are measured against the corresponding one-epoch DPO baseline.
  • Evaluation audit: AlpacaEval 2 quality scores use a single-answer LLM-as-a-judge prompt that requests a rationale and bracketed 1–10 rating parsed into per-id scalar scores.These scalar diagnostics complement pairwise benchmark comparisons.
  • Injected noise: Noise rates are η ∈ {0, 0.05, 0.10, 0.20, 0.30}, with each pair independently swapped at rate η and the random seed shared across methods.This procedure creates matched corrupted datasets for direct comparisons.

C.4 Exact-Tie Injection Dataset

The exact-tie stress test replaces selected preference pairs with equal-score responses while holding training data, optimization settings, and evaluation conditions controlled. The study also evaluates transfer to human disagreement and compares recipe variants.

  • Tie-injection construction: At 30% tie injection, the dataset contains 12,012 analysis pairs: 3,567 injected ties and 8,445 unmodified pairs.The 5–30% corruption sets are nested, and DPO, RE-PO, and PLC-DPO train from scratch on the same split at each rate.
  • Human-disagreement evaluation: MultiPref contributes 10,461 independent evaluation pairs partitioned into unanimous, divergent, and tie-majority groups.Pairs are scored without further training while preserving multiannotator labels.
  • Controlled training: All methods use one preference-optimization epoch with matched optimizer, scheduler, adapter rank, and effective batch size.This controlled setup isolates the preference objective from training-budget and adaptation-capacity changes.
  • Hyperparameter settings: The main experiments use fixed common hyperparameters, while PLC-DPO’s aggressive recipe is the main-results setting.The recipe table distinguishes objective-level controls from calibration and scheduling settings.
  • Baseline comparison: Baseline runs preserve the shared data, model, optimizer, scheduler, adapter capacity, and epoch budget while retaining each objective’s method-specific settings.KTO-Pair is the only listed baseline that changes the supervision format by converting pairs into pointwise feedback.

D.4 Evaluation Protocol

The evaluation protocol standardizes decoding and judging across methods while combining several established benchmarks and a controlled compute environment. The paper also documents reproducibility materials and warns that deployed models retain substantial safety risks.

  • Common protocol: Decoding parameters and judge configurations are strictly identical across all evaluated methods.This standardization is intended to ensure fair comparisons.
  • Benchmarks: AlpacaEval and AlpacaEval 2 each contain 805 test samples, with commercial-judge evaluations using a fixed first-200 subset and reporting length-controlled and raw win rates where available.The official evaluation protocol is followed.
  • Benchmarks: UltraFeedback uses all 2,000 test samples, Vicuna uses 80 prompts, HH-RLHF uses an 805-instance earliest-ID subset, and MT-Bench uses 80 two-turn prompts.MT-Bench retains the same judge configuration across methods.
  • Compute: Direct comparisons match device type, hardware allocation, parallelism strategy, and evaluation environment; most training uses two RTX A6000 GPUs.The runtime comparison uses eight NVIDIA B200 GPUs, and 7B LoRA fine-tuning on UltraFeedback takes about 8 hours under two A6000s.
  • Reproducibility: The study reports model and dataset licenses, software versions, and AI-assistance disclosures for reproducibility and transparency.The authors state that core scientific ideas, experimental designs, interpretations, and final text revisions remain their original work.
  • Potential risks: The authors identify hallucination, inherited bias, toxic language, and dual-use risks, recommending robust safety guardrails and continuous human evaluation before real-world deployment.These risks are stated as limitations of the fine-tuned language models and accessible tuning methods.
Loading 2608.30597v1…