PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Mixtures of Plackett-Luce Models for Multi-Objective Alignment

Dongyue Li, Ziniu Zhang, Lu Wang, Hongyang R. Zhang

arXiv:2608.25200v1cs.LGcs.AIcs.CL

TL;DR

The paper addresses learning mixtures of ranking models from heterogeneous preference data under ranking-length unidentifiability. It augments rankings and uses gradient-based estimation in an expectation-maximization algorithm, improving clustering and ranking accuracy over baselines.

  • Problem

    The paper studies identifying clusters and learning mixture ranking models from preference data drawn from heterogeneous subpopulations, where short rankings can make mixtures unidentifiable.

  • Method

    The method augments rankings with generated responses and uses first-order gradient-based output estimation to efficiently train mixtures of Plackett-Luce models.

  • Results

    MoPLEx improves clustering accuracy by 43.7% and ranking accuracy by 15.2% over mixture-of-Bradley-Terry baselines, while gradient estimation remains under 5% error.

  • Takeaways & Limitations

    The evaluations support MoPLEx as an effective and efficient approach for learning mixtures of Plackett-Luce models from heterogeneous multi-way rankings.

  • Takeaways & Limitations

    The current approach assumes a fixed number of mixture components and evaluates single-turn conversational data.

Abstract

from arXiv · show

We consider the problem of learning a mixture of $k$ Plackett-Luce models given multi-way ranking responses from annotators that may represent heterogeneous underlying preferences. This problem has many applications in AI alignment and preference optimization. Prior work has studied mixtures of Bradley-Terry models from pairwise comparisons. However, uncovering mixture models is theoretically unidentifiable when $k$ exceeds $m/2$, where $m$ is the length of a ranking. We propose an efficient implementation to address this limitation, which involves first augmenting the rankings to a larger size by generating new responses from a base language model, followed by a gradient-based estimation to reduce inference cost in the input embedding space. Based on this procedure, we then design an expectation-maximization algorithm with these two steps to fit a mixture of Plackett-Luce models, called MoPLEx. Extensive experiments are conducted to verify this approach. First, we show that the gradient-based approximation estimates true probabilities with less than 5% error on models with up to 34 billion parameters. Second, we show that MoPLEx improves clustering and ranking accuracy by an average of 43.7% and 15.2% over baselines using single ranking and mixtures of Bradley-Terry models, on preference optimization datasets. These results demonstrate the effectiveness of MoPLEx for tackling multi-way rankings from heterogeneous preferences through measuring alignment between gradients.

1 Introduction

The paper studies mixtures of ranking models for heterogeneous preference data, where a single model cannot capture divergent rankings. It addresses short-ranking unidentifiability and computational cost with ranking augmentation and gradient-based estimation.

  • Heterogeneous annotator populations and evaluation criteria can produce different rankings that a single ranking model cannot accurately capture.
  • Mixtures of Bradley-Terry models handle pairwise comparisons, but learning mixtures on longer rankings remains open.
  • When m ≤2k −1, distinct mixtures of Plackett-Luce models can induce identical ranking distributions, preventing identification of true clusters.
  • MoPLEx augments rankings with base-model responses and uses first-order input-embedding approximations to reduce computation from O(km) to O(ka).The approximation computes exact outputs and gradients for anchor responses, then estimates the remaining outputs.
  • Under 5% approximation error was observed on models up to 34 billion parameters.
  • 43.7% and 15.2% improvements in clustering and ranking accuracy, respectively, were reported against single-model and mixture-of-Bradley-Terry baselines.

2 Preliminaries

The paper formulates multiway response rankings and models heterogeneous preferences with a mixture of Plackett-Luce components. Training uses expectation-maximization, with each component implemented through a language-model adapter.

  • Each dataset instance contains a prompt, m candidate responses, and a permutation defining their multiway ranking.
  • Diverse annotators or evaluation criteria can yield rankings that a single model cannot adequately capture.
  • The model represents the data with k Plackett-Luce components, each corresponding to a preference cluster.
  • Expectation-maximization alternates between posterior-based cluster assignment and parameter updates maximizing the mixture likelihood.
  • The mixture can be implemented as k low-rank adapters on a base language model and trained through fine-tuning.

3 Unidentifiability of Mixture Models

Mixture Plackett-Luce models are unidentifiable for sufficiently short rankings: different mixtures can produce the same observed distributions. Synthetic experiments show that increasing ranking length enables cluster recovery.

  • For two clusters over four responses, rankings of length m ≤3 cannot distinguish two different underlying clusterings.
  • With rankings of length m = 4, the two candidate clusterings yield distinct distributions and the correct clusters can be identified.
  • When m ≤2k −1, a mixture of k Plackett-Luce models is non-identifiable because distinct parameters can induce identical m-way ranking distributions.
  • Synthetic experiments vary ranking size m from 2 to 8 and cluster count k from 2 to 4 to measure held-out clustering accuracy.
  • Pairwise comparisons collapse to random guessing for k = 3 and k = 4, whereas longer rankings enable recovery of the true clusters.

4 Our Approach

MoPLEx expands short rankings with generated responses, estimates scores using input-embedding gradients, and fits the resulting mixture with expectation-maximization. Its identifiability guarantee and efficiency rely on assumptions about augmented-response scores and anchor-based approximation.

  • 4.1 Augmenting the ranking: The method expands each ranking to m′ candidates by generating m′ −m responses with a base language model.
  • 4.1 Augmenting the ranking: Generated responses are appended as unranked candidates below existing responses, treating that ordering as an approximation.
  • 4.1 Augmenting the ranking: Over 81% of generated responses scored below existing responses, and none ranked in the top two positions on UltraFeedback.
  • 4.1 Augmenting the ranking: If augmented responses have distinct scores, m′−m ≥k and m ≥2, the mixture is identifiable over the expanded partial-ranking space.
  • 4.2 Learning mixtures of Plackett-Luce models: Gradient-based estimation reduces the per-iteration cost from O(km) to O(ka) by computing exact outputs and gradients for anchor responses.
  • 4.2 Learning mixtures of Plackett-Luce models: The first-order approximation estimates remaining response scores from anchor embeddings, gradients, and an approximation error term.
  • 4.2 Learning mixtures of Plackett-Luce models: MoPLEx then trains the mixture with expectation-maximization, alternating posterior computation with weighted likelihood optimization.

5 Experiments

Experiments evaluate MoPLEx on preference datasets with heterogeneous criteria and demographics, measuring clustering, ranking, efficiency, and component effects. MoPLEx improves accuracy over baselines while reducing computational costs and preserving performance under full PL computation.

  • Experimental setup: MoPLEx is evaluated on UltraFeedback and Persona, covering multiple evaluation criteria and diverse user demographics.UltraFeedback uses four criteria and Persona uses 12 demographic profiles as clusters.
  • Main results: 43.7% improvement in clustering accuracy and 15.2% relative improvement in ranking accuracy are achieved over mixture-of-BT baselines.The comparison includes MiCRo, MaxMin-RLHF, and EM-DPO across the UltraFeedback and Persona datasets.
  • Efficiency: MoPLEx reduces runtime and memory overhead by up to 3× while matching the performance of fully computed PL models.With two anchors, runtime and memory decrease by 2× and 3×, respectively, while performance remains comparable.
  • Efficiency: With six anchors, MoPLEx scales to ranking length m = 32 and outperforms the best fully trained PL models at m = 20 by 4.6%.This setting uses 1.6× less runtime and 1.9× less memory than the comparison.
  • Ablation studies: Ablations show that removing PL models, generated responses, or mixture models worsens performance, while gradient estimation halves GPU memory relative to full computation.The ablation is conducted on UltraFeedback and compares ranking accuracy across algorithm variants.
  • Ablation studies: Sampling temperature around 2.0 benefits ranking accuracy, whereas higher temperatures reduce accuracy by degrading generated-response quality.The study varies temperature among 0.5, 1.0, 2.0, and 4.0.
  • Hyperparameters: Increasing augmented ranking size m′ improves performance until gains plateau after m′ = 26, while increasing anchors beyond a = 6 provides no obvious gain.Increasing k beyond 12 also yields no significant clustering or ranking gains on Persona.

6 Related Work

Related work spans ranking models, mixture learning, influence estimation, and learning from multiple data distributions. This paper applies mixture-ranking ideas to text preference datasets and uses gradient-based methods for efficient language-model alignment.

  • Mixture ranking models: Prior work establishes identifiability conditions and polynomial-time methods for learning mixtures of PL, Mallows, and related ranking models.These methods primarily study voting or other ranking data rather than text preference datasets.
  • Positioning: This work differs by learning mixtures of ranking models on language-model preference datasets using efficient gradient-based algorithms.Its setting includes heterogeneous annotator subpopulations and multiple evaluation criteria.
  • Preference optimization: Preference-optimization research extends beyond BT models through latent-space, surrogate-loss, and regression-based approaches.These approaches address intransitive rankings, multiway ranking metrics, or multiple evaluation criteria.
  • Gradient-based methods: Influence functions and function vectors represent model behavior or training-data effects, while gradient features support efficient influence estimation and task grouping.Related methods use these representations to model task relationships and train weighted adapter ensembles.
  • Multiple data distributions: Kernel methods and kernel mean embeddings have been used to learn across related data distributions and compare contextual task similarities.These studies include predictors for test distributions and reward prediction across contextual-bandit arms.

7 Conclusion

The paper studies mixture learning for distinct ranking distributions and identifies an unidentifiability limit. It proposes an efficient scalable algorithm and reports empirical performance and efficiency advantages over existing approaches.

  • Conclusion: The paper identifies the unidentifiability limit of mixture ranking models.This addresses learning from datasets containing multiple distinct ranking distributions.
  • Conclusion: It proposes an efficient algorithm intended to resolve unidentifiability and scale to large models.The paper supports this proposal through extensive evaluations of performance and efficiency.

Limitations and Future Works

The paper identifies approximation, model-selection, and conversational-scope limitations, and points to extensions involving ranking estimation, adaptive mixture size, and multiturn preferences.

  • Approximation: Generated responses are assumed to rank below existing dataset responses, although exceptions can occur in practice.Computing all possible rankings for new responses is described as computationally expensive.
  • Model scope: The current method learns mixtures with a fixed number of components rather than adapting that number in continual learning settings.Dynamic component adaptation remains an open problem.
  • Conversation scope: The method evaluates single-turn conversational data, leaving multiturn settings with dynamically evolving preferences for future work.The paper presents its current method as a foundation for such extensions.

Potential Risks

The paper identifies risks from clustering preferences into distinct groups, especially the possibility of learning spurious correlations between preferences and specific demographics. It proposes examining group-level errors and incorporating group distributionally robust optimization as future work.

  • Clustering preferences into distinct groups may produce spurious correlations between preferences and specific demographics.
  • The paper focuses on learning mixtures of Plackett-Luce models to identify clusters of rankings.
  • The proposed future direction is to examine misclassification errors within each group.
  • The authors also propose incorporating group distributionally robust optimization into training.

A.3 Proof for Proposition 3.1

The proof constructs a symmetric family of Plackett-Luce components whose probability matrix has rank at most m, establishing non-identifiability when m ≤ 2k − 1. A nontrivial null space then yields two distinct mixtures with identical ranking distributions.

  • The proof constructs 2k distinct clusters where y1 has a cluster-specific score and all other alternatives share an identical score.
  • Under this construction, each ranking probability depends only on y1’s rank position and the cluster.
  • The probability matrix F therefore has only m unique row types and factors through an m-dimensional representation.
  • rank(F) = min{m, 2k}.
  • When m ≤ 2k − 1, F has a nontrivial null space containing a nonzero coefficient vector β.
  • The positive and negative entries of β define two disjoint mixtures with identical marginal distributions over m-way rankings.

A.4 Proof for Proposition 4.1

The proof shows that augmenting rankings with additional responses having distinct scores prevents the rank collapse responsible for non-identifiability. The resulting augmented probability matrix has full column rank 2k under the proposition’s conditions.

  • The proof assumes generated responses have distinct scores and studies partial rankings over the augmented response set.
  • Distinct scores make denominators depend on the subset of unranked items, preventing the rank-deficient factorization used in Proposition 3.1.
  • Because the transformed matrix is obtained from F aug by row combinations and F aug has only 2k columns, rank(F aug) = 2k.
  • At least k additional alternatives are selected from the augmented set to construct a 2k × 2k transformed probability matrix.
  • The transformed matrix contains marginal top-ranking probabilities for selected augmented responses and joint probabilities involving y1 ranked second.
  • A polynomial contradiction shows that the transformed matrix has rank 2k when the augmented scores are pairwise distinct.

B.1 Synthetic data

The synthetic experiments test whether mixture models can recover underlying clusters as ranking length, sample count, and cluster count vary. Short rankings, especially pairwise comparisons, hinder recovery, while larger rankings improve identification; the paper also reports approximation accuracy and downstream evaluation procedures.

  • Synthetic data: More underlying clusters require more training samples to identify the clusters.The experiments vary training rankings n from 1,000 to 100,000 and observe higher sample requirements as cluster count increases.
  • Gradient estimation: The gradient estimation error ranges from 0.1% at the closest distances to 5.0%–5.4% at the largest distance bucket across models from 0.6B to 34B parameters.The evaluation uses five language models on HelpSteer2, including Qwen-0.6B, Gemma-2B, DeepSeek-7B, Llama-13B, and CodeLlama-34B.
  • Fine-tuned generation: The fine-tuned method achieves a 58% average win rate, exceeding single-ranking baselines by 7% and mixture-of-BT baselines by 4.3%.Win rate measures how often generated responses receive higher reward scores than the baseline responses across evaluated clusters.
  • Computation: Response generation consumes roughly 16% of total runtime, while training requires substantially more GPU time on both evaluated datasets.On one A6000 GPU, generation takes 1.1 and 1.3 GPU hours, compared with 7.3 and 7.2 GPU hours for training.
  • Model parameters: Using k = 4 adds 13.5% parameters on UltraFeedback, while k = 12 adds 40.5% on Persona through separate LoRA adapter sets.Each PL component uses a distinct adapter set on top of the base model.
Loading 2608.25200v1…