PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Pairwise Feedback: Listwise Vision-Language Supervision for Preference-Based Reward Learning

Srivalli Katkuri, Maxwell Kawada, Juan Wachs

arXiv:2608.25350v1cs.LGcs.RO

TL;DR

Existing VLM-based preference reward learning mainly uses pairwise comparisons, although VLMs can rank multiple outcomes and listwise methods may use that information more efficiently. The paper introduces a Plackett-Luce framework trained from VLM-generated rankings and evaluates it on Meta-World robotic manipulation tasks. PL performs competitively with pairwise and K-wise baselines while allowing ranking sizes K ∈ {3, 4, 5}, but the evaluation is limited to simulated rigid-body tasks and depends on proprietary hosted VLMs.

  • Problem

    VLM preference supervision has largely retained pairwise comparisons despite VLMs being able to rank multiple outcomes, leaving listwise Plackett-Luce reward learning underexplored with VLM-generated preferences.

  • Method

    The paper trains Plackett-Luce reward models from VLM-generated rankings of K ∈ {3, 4, 5} observations and compares them with Bradley-Terry and RL-VLM-F baselines in Meta-World.

  • Results

    Across three simulated robotic environments, PL learned rewards competitively with BT-Kwise, BT-Pairwise, and RL-VLM-F, with at least one PL ranking size competitive or best in each environment.

  • Takeaways & Limitations

    Listwise VLM supervision is a viable and flexible alternative to pairwise preference feedback because PL supports varying the number of jointly ranked observations.

  • Takeaways & Limitations

    The evaluation covers only simulated rigid-body tasks, and reproducing the configuration requires inference costs from proprietary hosted VLMs.

Abstract

from arXiv · show

Vision-language models (VLMs) have emerged as a powerful source of supervision for reinforcement learning, enabling agents to leverage rich semantic knowledge during training. Inspired by the success of preference-based reward learning (PbRL) in reinforcement learning from human feedback (RLHF), vision-language model generated image-based preferences provide an effective source for learning reward functions. This can be done by visually comparing two outcomes through the Bradley-Terry (BT) model. However, this pairwise formulation utilizes only two observations at a time, despite VLMs being capable of ranking multiple candidates. The Plackett-Luce (PL) formulation can shape a reward model with listwise rankings as opposed to pairwise preferences, allowing for a more suited use of a VLM based ranking. In this work, to our knowledge, we introduce the first framework that combines VLM-generated preferences with the Plackett-Luce model for reward learning. We evaluate our approach on Meta-World manipulation tasks and show that Plackett-Luce (PL) reward models can train robotic policies from VLM-generated rankings as effectively as pairwise Bradley-Terry, $K$-wise Bradley-Terry, and RL-VLM-F baselines. Across all environments, at least one PL ranking size ($K \in \{3,4,5\}$) consistently performs with or outperforms other methods in mean success rate. Unlike pairwise methods, which are restricted to $K=2$, PL supports different ranking sizes and can therefore be adapted to the environment and desired feedback format. Our best PL configuration achieves an 86% mean final success rate and matches the Oracle baseline on Drawer Open. Overall, these results demonstrate that listwise VLM preference supervision is a competitive and flexible approach to reward learning for reinforcement learning.

I. INTRODUCTION

Preference-based reinforcement learning reduces the burden of hand-designed rewards and human labeling, while VLMs can provide task-conditioned visual supervision. This work introduces listwise VLM rankings with Plackett-Luce reward learning and evaluates their flexibility and competitiveness against pairwise baselines.

  • Hand-designed reward functions are difficult for goals that resist formalization, and approximate rewards may be optimized in unintended ways.
  • VLMs can provide task-conditioned reward signals by visually inspecting and contextually reasoning about outcomes, reducing reliance on human annotators.Earlier CLIP-style approaches offered more limited image-text or image-image similarity signals.
  • VLMs can rank K outcomes in one query without human fatigue, making listwise supervision more scalable than pairwise feedback.Listwise Plackett-Luce learning is theoretically more statistically efficient than decomposing rankings into pairwise comparisons.
  • The proposed framework trains a Plackett-Luce reward model from VLM-generated rankings of K ∈ {3, 4, 5} observations in Meta-World environments.It compares PL with pairwise Bradley-Terry, K-wise Bradley-Terry, and RL-VLM-F.
  • At least one PL ranking size performs competitively or best in each environment, while pairwise methods remain restricted to K = 2.The authors present this as the first methodology combining VLM-generated listwise rankings with Plackett-Luce reward learning.
  • The study evaluates PL across multiple ranking group sizes and includes an ablation of the number of feedback groups M per iteration.

C. Comparison of Bradley-Terry and Plackett-Luce

Bradley-Terry models pairwise preferences, whereas Plackett-Luce generalizes the comparison to complete rankings and preserves their joint structure. Both reward models are trained from preference data and used alternately with policy optimization.

  • Training loop: The reward model is fit to preference labels, and the policy is optimized against the learned reward using a standard reinforcement-learning algorithm.The paper uses SAC for policy learning in its framework.
  • Model formulations: Bradley-Terry models preferences between two segments, while Plackett-Luce models a complete ranking of K segments.For K = 2, PL reduces exactly to BT.
  • Model formulations: PL models a ranking by repeatedly selecting the best remaining segment, producing a joint likelihood over the ordering.
  • Model formulations: Rank-breaking converts a K-wise ranking into implied pairwise comparisons for BT, retaining the data but discarding the ranking’s joint structure.Rankings of three, four, and five observations yield three, six, and ten implied pairwise relations, respectively.

IV. METHODS

The method follows a simulated preference-based reinforcement-learning loop in which SAC collects trajectories, a replay buffer supplies observations, and learned rewards guide subsequent policy updates. An unsupervised warm-up provides initial behavioral coverage before VLM preference learning begins.

  • Policy optimization: SAC trains on replay-buffer transitions and updates the policy against rewards assigned by the learned reward model.
  • Data collection: Each iteration resets the simulated environment, collects one trajectory of at most 500 steps, and adds it to the replay buffer.The protocol contributes exactly one newly executed trajectory per interaction round.
  • Warm-up: Unsupervised warm-up iterations populate the buffer before VLM feedback, preventing the first query from relying on observations from approximately one trajectory.
  • Warm-up: The warm-up is retained for simulation experiments but leaves open whether this initialization can create undesirable exploratory behavior in physical settings.
  • Replay buffer: Preference learning samples observations from a fixed-capacity mixed replay buffer, with uniform selection and no trajectory- or timestep-based organization.The fixed capacity limits memory, storage, and reward-processing costs while retaining experience for SAC.

B. Listwise comparison protocol

The listwise protocol uniformly samples K distinct observations and asks a VLM to produce a complete ranking, which is then modeled either jointly with PL or through BT rank-breaking. The study varies ranking size and also tests whether one structured query can replace multiple pairwise queries.

  • Listwise sampling: For K ∈ {3, 4, 5}, each listwise query uniformly samples K distinct replay-buffer observations and requests their complete visual-progress ranking from the VLM.
  • Reward-model training: PL stores the full ranking and fits its Plackett-Luce likelihood, whereas BT-Kwise decomposes the same ranking into implied pairs and fits Bradley-Terry loss.
  • Reward-model training: Both PL and BT-Kwise require M VLM calls per iteration and use the same ranking size, sampling procedure, and complete-ranking request.
  • Ranking-size tradeoff: Increasing K exposes relationships among more observations but may make subtle relative progress harder for the VLM to distinguish.
  • Ranking-size tradeoff: Ranking sizes of three, four, and five observations create three, six, and ten implied pairwise relations, respectively.
  • Query-budget ablation: In the M = 1 ablation, PL uses one VLM request per iteration versus four for BT-Pairwise and eight for RL-VLM-F.This tests whether one structured ranking can substitute for multiple pairwise queries.

C. Pairwise baselines

The evaluation compares listwise PL and BT-Kwise feedback with pairwise BT-Pairwise and RL-VLM-F baselines in Meta-World simulated manipulation tasks. All configurations use uniform replay-buffer sampling, while the tasks are selected for visually interpretable completion.

  • Pairwise feedback: RL-VLM-F uses two VLM calls per pair—free-form analysis followed by preference labeling—whereas BT-Pairwise uses one query.
  • Sampling control: All preference-learning configurations use uniform replay-buffer sampling, avoiding advantages from uncertainty-, disagreement-, or diversity-based acquisition.The protocol tests listwise feedback without a query-optimization stage.
  • Environments and tasks: The benchmark uses Meta-World, where a Sawyer robotic arm performs object-manipulation tasks in simulation.Meta-World provides 50 tasks spanning object interaction and motor-control objectives.
  • Pairwise feedback: Both pairwise baselines sample two distinct observations uniformly and train the reward model with Bradley-Terry loss over retained preference labels.
  • Environments and tasks: The selected tasks require visually interpretable completion markers, including opening a drawer, closing a door, and pressing a button.Custom object-focused camera views and a transparent robot model help keep task completion visible to the VLM.

B. Primary Setup

Primary comparisons use equal feedback-group budgets, with PL and BT-Kwise varying ranking size while pairwise baselines remain limited to K=2. Each run uses 100,000 environment steps after a warm-up phase, with uniformly sampled replay-buffer observations evaluated by a task-conditioned VLM.

  • Feedback and ranking setup: PL and BT-Kwise receive K images per feedback group, while pairwise methods are restricted to K = 2 images.BT-Pairwise uses one VLM request per pair, whereas RL-VLM-F uses two-stage analysis and labeling.
  • Feedback and ranking setup: The primary setup fixes the post-warm-up feedback budget at M = 4 independently sampled groups.PL and BT-Kwise require one VLM request per group for a complete ranking; RL-VLM-F requires 2M = 8 requests.
  • Training schedule: Each run spans 100,000 environment steps, including 18 warm-up iterations without VLM queries or ground-truth rewards.The remaining 182 iterations use uniform sampling from a persistent replay buffer containing up to 100,000 images.
  • VLM supervision: GPT-5.6 Luna receives task-specific descriptions directing preferences toward visual indicators of task progress.Examples include drawer displacement, door angle, and button depression.

C. Ablation Setup

The ablation and primary comparison protocol evaluates feedback-budget and ranking choices under deterministic, five-seed Meta-World evaluation. The supplied results show PL reaching 86% on Drawer Open with K = 4 and 54% on Door Close with K = 5.

  • Ablation setup: The feedback-budget ablation compares M = 4 with M = 1 on Drawer Open while fixing K = 3 and other hyperparameters.The reduced-budget condition queries the VLM one quarter as often as the main setup.
  • Evaluation protocol: Policies are evaluated every 10,000 environment steps through the final 100,000-step checkpoint using 20 deterministic episodes.Success rate is the number of successful episodes out of 20 under Meta-World’s ground-truth criterion.
  • Statistical analysis: Matched PL and BT-Kwise comparisons use two-sided paired t-tests across five seeds with Holm correction.The seed is the statistical unit for comparisons at the same ranking size.
  • Drawer Open results: 86% was the Drawer Open mean success rate for PL with K = 4, exceeding PL with K = 3 at 74% and K = 5 at 81%.At K = 4, PL achieved 86% versus 84% for BT-Kwise, while BT-Pairwise and RL-VLM-F achieved 89% and 92%.
  • Door Close results: 54% was the Door Close mean success rate for PL with K = 5, compared with 48% for K = 4 and 40% for K = 3.At K = 5, PL exceeded BT-Kwise by 7 percentage points, 54% versus 47%.

3) Button Press:

On Button Press, PL performed best at K = 4, while matched statistical comparisons found a corrected-significance caveat for the K = 5 advantage over BT-Kwise. A low-feedback Drawer Open ablation also showed substantial seed dependence.

  • Button Press: 41% was PL’s highest Button Press mean final success rate, achieved at K = 4 across five seeds.PL averaged 22% at K = 3 and 26% at K = 5.
  • Statistical significance: At K = 5, PL achieved 26% on Button Press versus 7% for BT-Kwise.This was the only matched comparison with a nominally significant uncorrected difference, but the Holm-corrected p-value was 0.405.
  • Statistical significance: No statistically significant PL-versus-BT-Kwise difference was observed for Drawer Open or Door Close at any tested ranking size.Button Press also showed no significant difference at K = 3 or K = 4.
  • Feedback-budget ablation: 49% was PL’s Drawer Open mean success rate with M = 1, versus 55% for BT-Kwise across five seeds.Both methods failed completely on at least one seed while achieving high or near-perfect success on others.
  • Feedback-budget ablation: 55% was Bradley-Terry’s aggregate final success rate after 100,000 steps, compared with 49% for Plackett-Luce.The ablation aggregates five seeds with standard error of the mean.
  • Feedback-budget ablation: The M = 1 PL comparison did not establish a statistically significant advantage for either M = 4 pairwise baseline.Paired-test p-values were 0.093 against BT-Pairwise and 0.116 against RL-VLM-F.
  • Feedback-budget ablation: 182 VLM calls enabled PL with M = 1 to match or exceed RL-VLM-F on two of five Drawer Open seeds.PL used fewer calls than BT-Pairwise at 728 and RL-VLM-F at 1,456.

1) Primary:

Plackett-Luce reward learning remained competitive with pairwise baselines across Meta-World environments, while performance depended on ranking size and feedback configuration. The experiments also identify limitations in robustness, task coverage, and generalization beyond simulation.

  • Primary: 26% versus 7%: PL at K = 5 outperformed BT-Kwise on Button Press, although the small sample and multiple comparisons warrant caution.The uncorrected test gave t(4) = 2.881 and p = 0.045.
  • Primary: 54% versus 39%: PL at K = 5 exceeded both RL-VLM-F and single-stage BT-Pairwise on Door Close.
  • Primary: 41% versus 28% and 38%: PL at K = 4 exceeded RL-VLM-F and slightly exceeded BT-Pairwise on Button Press.
  • Feedback Group Ablation: One uniformly sampled K = 3 ranking matched or exceeded RL-VLM-F on two of five seeds while using one-eighth as many VLM calls.The result suggests that a single broad comparison can sometimes provide sufficient supervision without extensive query curation.
  • Feedback Group Ablation: M = 1 had lower mean performance and greater seed sensitivity than higher-feedback configurations, preventing it from being a robust replacement.
  • Limitations: The evaluation covered relatively few tasks because VLMs were sensitive to camera settings and struggled particularly with complex grasping scenarios.The experiments were conducted in simulated rigid-body environments, and future work is proposed for broader benchmarks and real-world robotic tasks.
  • Conclusions: The paper concludes that listwise VLM supervision is a viable alternative to pairwise feedback, with flexibility through ranking size K and group count M.

APPENDIX A QUERY EFFICIENCY AND PRICING

The appendix examines query efficiency, sampling strategy, and feedback-group budgets for PL reward learning. State-diverse sampling and larger feedback budgets can improve outcomes, but they require additional supervision or query design.

  • Query Efficiency and Pricing: The appendix includes figures for API queries, input tokens, and estimated run costs, but the supplied passages provide no plotted values.
  • Sampling Strategy: The primary experiments deliberately used uniform replay-buffer sampling to test PL without active or diversity-aware query selection.State-diverse sampling was examined as an initial alternative for constructing groups spanning different task stages.
  • State-Diverse Sampling: 93% ± 7% versus 27%±20%: state-diverse sampling substantially exceeded uniform sampling on Drawer Open under M = 1, K = 3.The comparison used three matched seeds.
  • Feedback Groups: 25% to 62%: increasing M from one to five raised Door Close mean success with K = 3 and state-diverse sampling.The number of collected rankings increased from 182 to 910 across the matched settings.
  • Feedback Groups: Increasing M can improve performance, but it requires more feedback queries, making the primary M = 4 experiments conservative relative to methods using larger preference budgets.RL-VLM-F is described as sampling 20 pairs per iteration.

APPENDIX C ALTERNATIVE RUN CONFIGURATION

The alternative configuration reports an initial CartPole sanity test comparing PL and BT-Kwise under matched ranking and feedback settings. Both models reached perfect final success in this preliminary evaluation.

  • Alternative Run Configuration: The alternative configuration changed the reward-inference batch size from 512 to 128.
  • Alternative Run Configuration: 100% final success: both PL and BT-Kwise achieved perfect performance in the two CartPole test runs.
  • Alternative Run Configuration: The CartPole tests used K = 4 and M = 3, with 7,500 warm-up steps followed by 42,500 VLM PbRL steps.
  • Alternative Run Configuration: The appendix provides a Plackett-Luce CartPole success-rate figure and a Bradley-Terry-Kwise CartPole success-rate figure.The supplied figure captions do not report additional numerical values.
  • Experimental Configuration: The primary experiments held common hyperparameters constant across methods unless otherwise specified and document the CNN reward-model architecture.

A. Drawer Open

The task-specific prompting defines how images should be ranked for Drawer Open, Door Close, and Button Press. The prompts require complete rankings or pairwise judgments based on task-relevant visual progress.

  • Task Prompts: Drawer Open is judged by whether the drawer is visibly farther open.
  • Task Prompts: Door Close is judged by a smaller opening and a door more nearly flush with its cabinet frame, prioritizing door angle over gripper position.
  • Task Prompts: Button Press is ranked primarily by exposed shaft length: a shorter shaft indicates a more fully pressed button.Fingertip position is considered only when shaft lengths are visually tied.
  • Task Prompts: PL prompts require a complete best-to-worst ranking of the images even when differences are subtle.
  • Task Prompts: RL-VLM-F uses the same task descriptions but applies separate prompts for its two-stage process.
Loading 2608.25350v1…