PaperlayerView exact revision on arXiv

Source-linked AI summary

Direct Preference Density Alignment for Conversational Audio Equalization

Ioannis Stylianou, Sven Ewan Shepstone, Jon Francombe, Pablo Martinez Nuevo, Zheng-Hua Tan

arXiv:2609.12607v1cs.SDcs.AI

TL;DR

Learned reward models add memory costs and instability to alignment, while offline preference optimization gives up online exploration. The paper builds a nonparametric preference reward surface from user data to combine online GRPO with offline DPO; the hybrid outperforms either alone and a 1.5B model reaches perceptual parity with GPT-4o mini in blind listening.

  • Problem

    Learned reward models increase training memory and are vulnerable to reward hacking, while offline DPO loses active exploration and can permit format collapse in bounded continuous spaces.

    Source: §1 · p. 1, §4.1 · p. 5

  • Method

    The framework uses population data to build nonparametric preference surfaces, enabling proxy-free online reinforcement learning and a GRPO+DPO hybrid.

    Source: §1 · p. 2, §8 · p. 10

  • Results

    The GRPO+DPO pipeline outperforms either algorithm alone, and a 1.5B-parameter model achieves perceptual parity with GPT-4o mini in a blind A/B listening test.

    Source: §1 · p. 2, §8 · p. 10

  • Takeaways & Limitations

    A continuous empirical reward surface supports both GRPO’s online exploration and DPO’s targeted offline refinement without a learned proxy reward model.

    Source: §8 · p. 10

    The system conditions only on text intent, limiting context awareness because it does not analyze the input audio.

    Source: §7 · p. 9

Abstract

from arXiv · show

Large Language Model alignment typically relies on learned proxy reward models, which significantly increase the memory footprint during training and are notoriously prone to instability and reward hacking. While offline methods like Direct Preference Optimization (DPO) bypass the reward model, they lose the ability to perform online exploration. If no optimization constraints are applied, this can lead to format collapse in bounded, continuous spaces. To resolve this, we propose Direct Preference Density Alignment: An alternative framework that removes the need for a learned proxy reward model while strictly preserving the benefits of online reinforcement learning. We leverage large-scale user data (approximately 90,000 samples) to construct non-parametric preference density maps, establishing an empirical reward surface. In addition to removing the reward model, Direct Preference Density Alignment enables the combination of the online structural grounding of Group Relative Policy Optimization (GRPO) with the targeted offline refinement of DPO. We show that this GRPO+DPO combination achieves the highest performance, and in a blind audio equalization listening test, enables a 1.5B-parameter model to achieve perceptual parity with a carefully prompt-engineered GPT-4o mini baseline, using only a fraction of the inference compute.

1 Introduction

LLM alignment for subjective continuous-control tasks is challenged by the cost and instability of learned reward models, while offline preference methods sacrifice online exploration. The paper proposes using population preference data as an empirical reward surface for proxy-free online alignment.

1 Introduction

  • PPO-based RLHF uses learned reward models that are vulnerable to reward hacking, and PPO training requires four separate LLMs.Offline DPO removes the reward model but loses active exploration, which can lead to format collapse in compact spaces.Source: §1 · p. 1
  • The framework uses population data to construct static, nonparametric preference surfaces instead of training a proxy reward model.The model explores the surface to find regions of high human satisfaction in an audio equalization task.Source: §1 · p. 2
  • The GRPO+DPO combination outperforms either method alone, as well as SFT and ICL baselines.The authors also report perceptual parity between a 1.5B-parameter model and GPT-4o mini in a blind A/B listening test.Source: §1 · p. 2

2 Related Work

Prior approaches to preference alignment trade off compute, active exploration, and the ability to represent continuous preferences. This work adapts hybrid online exploration and offline refinement to continuous parameter estimation without learned reward models.

2 Related Work

  • Regression-based audio systems favor a single parameter realization, while prior distributional modeling used only N = 11 examples per prompt; this work scales to N ≈ 3,000 and uses RL.Earlier systems used fixed vocabulary-to-parameter mappings, while recent LLM methods estimate EQ and reverb parameters [7] [8]; distributional audio descriptors were proposed in.Source: §2 · p. 2
  • PPO requires a separate value network, while direct-alignment methods such as DPO, IPO [1], and KTO [10] optimize from preference data without an explicit reward function.These direct methods do not account for probability mass outside the preference dataset, which can leave malformed or constraint-violating outputs unpenalized.Source: §2 · p. 2
  • GRPO removes the value network and uses average group reward as its baseline, generating data online to explore format constraints.The study evaluates vanilla DPO and GRPO without constrained decoding, custom validity penalties, or algorithmic extensions.Source: §2 · pp. 2–3
  • Prior hybrid pipelines sequence online exploration with offline refinement [19] [34] [33]; this work adapts that strategy to continuous parameter estimation without learned verifiers or reward models.Source: §2 · p. 3

3 Preliminaries and reward surface construction

The section defines a two-dimensional continuous equalization control space and constructs a preference-based reward surface from user interactions. Reflective density estimation, normalization, and prompt aggregation support its use for RL evaluation and training.

3.1 The Beosonic control space

  • The Beosonic interface defines a continuous two-dimensional control plane with coordinates in [−6, 6]², where each parameter controls a distinct EQ filter.Horizontal movement yields a smile curve, while vertical movement applies a linear adjustment.Source: §3.1 · p. 3, Figure 1 · p. 3

3.2 Dataset and motivation

  • T2B received 14,087 positive selections, the highest count among the reported models, motivating optimization for both preference peaks and space coverage.The dataset contains approximately 90,000 interaction events; the prior distributional models trained on N = 11 data points per prompt did not generalize effectively.Source: §3.2 · p. 3, Figure 2 · p. 3

3.3 Probability of selection

  • The reward estimates selection probability from preferred-point density relative to the density of all offered coordinates, avoiding bias from how generating models sampled the space.The method estimates separate densities for all offered points and user-preferred regions; a stability term is included in the ratio.Source: §3.3 · p. 4

3.4 Reflective kernel density estimation

  • Reflective-KDE mirrors each point into eight neighboring squares to address boundary bias, using a Gaussian kernel with bandwidth h = 0.25 · hScott.The bandwidth scaling accounts for added reflected point mass and Scott’s Rule oversmoothing non-convex spaces.Source: §3.4 · p. 4

3.5 Global normalization and augmentation

  • Global normalization scales reward values to [−1, 1], and synonymous prompts are aggregated onto the same reward surface under an identical-preference-manifold assumption.Figure 3 depicts sampling bias, total density, preferred density, and the final normalized reward surface.Source: §3.5 · p. 4, Figure 3 · p. 4

3.6 Evaluation splits

  • Evaluation uses approximately 80% of prompts for policy updates, 10% for same-distribution validation, and a manually isolated 10% OOD set of unseen “Vocal Clarity” prompts.The OOD set tests whether the model extrapolates its learned preference geometry to novel semantic inputs.Source: §3.6 · p. 5

4 Alignment strategies in continuous spaces

The section frames continuous alignment as maximizing policy return on a non-parametric reward surface, while maintaining valid coordinate outputs. It proposes combining online structural grounding with offline preference refinement.

4.1 The challenge of continuous formatting

  • Online GRPO feedback penalizes malformed outputs with R = −1.0, grounding the policy in valid coordinate syntax that static DPO examples may not cover.As the policy shifts, DPO can produce novel malformed syntax absent from its fixed dataset.Source: §4.1 · p. 5

4.2 Baseline 1: direct preference optimization

  • DPO widens the implicit-reward margin between preferred and dispreferred responses, using reward-map-ranked coordinate pairs and a KL constraint controlled by β.The dataset pairs coordinates xw and xl when R(xw|p) > R(xl|p).Source: §4.2 · p. 5

4.3 Baseline 2: group relative policy optimization

  • GRPO removes PPO’s value model and normalizes rewards within groups; invalid-format outputs receive −1.0, shaping their relative advantage.For each prompt, the policy generates a group of outputs and optimizes a clipped objective.Source: §4.3 · p. 5

4.4 Proposed method: Hybrid GRPO+DPO pipeline

  • The hybrid addresses GRPO’s possible failure to find preference-density maxima by using GRPO for structural alignment and DPO for refinement.It mines on-policy completions into positive and negative pools, injects grid-searched top-K peaks when needed, then refines from the GRPO reference.Source: §4.4 · p. 6

5 Experiments and results

The experiments evaluate alignment strategies on unseen semantic concepts, measuring preference optimality and coordinate-format success. The hybrid combines the strongest reported GMRR outcomes with format compliance at 1.5B parameters.

5.1 Experimental setup

  • Evaluation uses Qwen2.5 0.5B and 1.5B models on OOD concepts, comparing GMRR and coordinate-format success against ICL baselines.The ICL prompts include expert semantic-to-coordinate mappings, while RL-tuned models use a minimal prompt.Source: §5 · p. 6, §5.1 · p. 6

5.3 Scaling exploration

  • Increasing GRPO rollouts consistently improves GMRR, with gains saturating between G = 8 and G = 16; even G = 2 outperforms sophisticated ICL counterparts.The authors describe the improvement from larger groups as relatively small given their computational burden.Source: §5.3 · p. 7, Table 2 · p. 7

5.2 The stability gap: Online vs. offline

  • GRPO maintains a 100% format success rate even at β = 0.0, while excessively large β values can over-constrain the model and hinder its ability to follow strict structure.Offline DPO focuses on two examples per preference-dataset entry, while probabilities of all other completions are updated without supervision.Source: §5.2 · p. 7

5.4 Hybrid pipeline performance

  • 0.56 GMRR for the 0.5B hybrid matches the GPT-4o baseline at 97% Parse Rate; the 1.5B hybrid reaches 0.60 GMRR without format-compliance loss.Across four independent seeds, the optimal configuration achieved a consistent mean GMRR of 0.58.Source: §5.4 · p. 7, Table 3 · p. 8
  • 0.45 GMRR for SFT on training-prompt peaks falls below the base model’s 0.49 ICL score, which the authors attribute to OOD overfitting.The comparison is reported for Qwen2.5-1.5B.Source: §5.4 · p. 8

6 Subjective evaluation

A blind listening study compared the hybrid RL model with GPT-4o mini on listener-generated requests. The 1.5B model achieved comparable preference ratings while requiring far less inference compute.

6.1 Unconstrained data collection and protocol

  • Eleven participants rated outputs for 30 retained prompts in randomized, blinded comparisons, using a five-point scale after unconstrained prompt elicitation.Both systems controlled the same equalization system; the 12-person elicitation phase was designed to avoid priming participants about system capabilities.Source: §6 · p. 8, §6.1 · pp. 8–9

6.2 Quantitative results

  • Mean Preference Score was 3.04 for the 1.5B RL model versus 2.92 for GPT-4o mini; RL was preferred or equivalent in 77.9% of trials, supporting parity.The mixed-effects model found no significant main effect, and a subsequent equivalence test supported perceptual parity.Source: Table 5 · p. 9, §6.2 · p. 9
  • The Density-Based RL pipeline distilled audio control into a model orders of magnitude smaller than its competitor, without expert knowledge or large inference compute.The authors report equivalent user satisfaction without proprietary APIs.Source: §6.2 · p. 9

7 Discussion and limitations

The discussion presents non-parametric preference density maps as an alternative to learned proxy reward models, while identifying limits around action-space size, audio context, and personalization.

7 Discussion and limitations

  • Non-parametric density maps offer a way to align LLMs using population preferences rather than training unstable proxy reward models.The authors frame this as leveraging the “wisdom of the crowd” directly.Source: §7 · p. 9
  • Reflective-KDE works well in low-dimensional spaces, but required data grows exponentially with action-space dimensions, likely necessitating latent compression.The example low-dimensional setting is the 2D Beosonic interface.Source: §7 · p. 9
  • Text-only conditioning does not analyze input audio, limiting context awareness when the same requested adjustment may depend on audio content.The authors give “warm” adjustments for podcasts versus techno as an example.Source: §7 · p. 9
  • Population-mode seeking models shared preferences but does not adapt settings to individual users’ hearing profiles.Conditioned density estimation is proposed as a future direction.Source: §7 · p. 9

8 Conclusion

The conclusion presents a proxy-free online RL framework built from large-scale user preferences. Its hybrid GRPO+DPO pipeline achieved perceptual parity with GPT-4o mini in a blind listening test.

8 Conclusion

  • Reflective Kernel Density Estimation applied to approximately 90,000 user events creates non-parametric preference maps as an empirical reward surface for online exploration.The framework avoids learned proxy reward models and their memory overhead and reward-hacking vulnerabilities.Source: §8 · p. 10
  • The continuous reward surface supports combining GRPO exploration with DPO refinement; the hybrid aligned a 1.5B model that achieved perceptual parity with GPT-4o mini.Parity was reported in a blind A/B listening test against a heavily prompt-engineered baseline.Source: §8 · p. 10

A Extended statistical analysis of perceptual parity

The extended analysis models listener ratings with crossed participant and prompt effects, then tests equivalence against a predefined margin. It also documents the evaluation interface’s two phases.

A Extended statistical analysis of perceptual parity

  • 660 randomized ratings were analyzed with crossed participant and prompt effects; model variation had no significant main effect (p = 0.095).The analysis modeled subjective score as the dependent variable and model variation as the fixed effect.Source: Table 6 · p. 13, A Extended statistical analysis of… · p. 13

A.1 Two One-Sided Tests (TOST) for Equivalence

  • The 90% CI [+0.002, +0.229] falls within the ±0.25 bounds, supporting statistical equivalence between the 1.5B RL model and GPT-4o mini.The bounds were the predefined margin of practical equivalence on the five-point Likert scale.Source: §A.1 · p. 13

B Subjective evaluation user interface

C Ablation on KDE bandwidth selection

The KDE ablation selects a bandwidth scale of 0.25 to preserve multimodal preference structure without fitting sparse noise; Table 7 lists settings for reproducibility.

C Ablation on KDE bandwidth selection

  • 0.25 scaling balances local-mode preservation against sparse-noise overfitting, while 0.1 fragments the surface and 1.0 merges distinct modes.Unscaled Scott’s Rule assumes a unimodal Gaussian distribution, so it over-smooths multimodal preferences and destroys local topological features.Source: Figure 6 · p. 16, C Ablation on KDE bandwidth selection · p. 16
  • Figure 6’s raw-data heatmap uses cell color for aggregated preference scores and in-cell integers for user-evaluation counts.Source: Figure 6 · p. 16

D Experimental Details

E Multi-seed experiments

Across four independent seeds, the hybrid model at G=16 achieved a mean GMRR of 0.58 ± 0.016 and a 100% parse rate.

E Multi-seed experiments

  • 0.58 ± 0.016 mean GMRR and a 100% parse rate were achieved across four seeds with G=16.Source: E Multi-seed experiments · p. 17
  • The best configuration, G=16, was tested across four independent training seeds to distinguish algorithmic instability from hyperparameter sensitivity.Source: E Multi-seed experiments · p. 17
  • Table 8 reports multi-seed performance but excludes the main model’s GMRR of 0.60.Source: Table 8 · p. 18

Related papers

Loading 2609.12607v1…