PaperlayerView exact revision on arXiv

Source-linked AI summary

SPEAR: Selective Preference Elicitation with Adaptive Rewards for VLM-Guided Robotic Manipulation

Zewen Wu, Ruizhe Liu, Pei Zhou, Feng Chen, Yanchao Yang

arXiv:2610.21621v1cs.RO

TL;DR

VLM preference oracles could automate preference labeling for robotic manipulation, but costly queries and abstentions on visually challenging comparisons limit their practicality. SPEAR selects informative, distinguishable comparisons and bootstraps exploration with VLM-synthesized rewards before preference refinement. It achieves 88.9% average success, 23.3 percentage points above the strongest VLM-based baseline, and is demonstrated online on real hardware.

  • Problem

    VLM preference queries are costly and frequently abstain on visually challenging comparisons, limiting their practicality for robotic manipulation.

    Source: §I · p. 1

  • Method

    SPEAR combines distinguishability-aware query selection with an adaptive transition from VLM-synthesized warm-up rewards to preference-based learning.

    Source: §I · p. 1

  • Results

    88.9% average success across six Meta-World tasks, 23.3 percentage points above the strongest VLM-based baseline, with fewer wasted queries and reduced wall-clock training time.

    Source: §I · p. 1

  • Takeaways & Limitations

    SPEAR learns three manipulation tasks through fully online interaction on a real Mobile ALOHA robot.

    Source: 3) Results · p. 7

Abstract

from arXiv · show

Recent Vision-Language Models (VLMs) enable automated preference labeling for robotic manipulation, yet their deployment remains impractical since each query incurs substantial inference cost, while VLMs frequently abstain on visually challenging comparisons. We propose SPEAR, a framework that transforms VLMs into efficient preference oracles via two synergistic mechanisms. First, we design distinguishability-aware query selection to reduce abstention rates by targeting only informative and distinguishable comparisons. Second, we deploy an adaptive reward transition that bootstraps exploration with VLM-synthesized guidance before preference-based refinement. Experiments on the manipulation benchmarks demonstrate that SPEAR achieves 88.9% average success, outperforming baselines by 23.3 percentage points, while significantly reducing abstention and wall-clock training time on contact-rich tasks. SPEAR learns three Mobile ALOHA manipulation tasks fully online on real hardware, demonstrating end-to-end real-world applicability. By addressing query efficiency and exploration challenges, SPEAR establishes VLMs as practical preference oracles for scalable robot learning without manual reward engineering.

I. INTRODUCTION

SPEAR targets costly VLM preference queries that often abstain on visually indistinguishable trajectories, while early random exploration produces uninformative comparisons. It combines distinguishability-aware query selection with VLM-reward warm-up and reports stronger benchmark performance and real-robot learning.

I. INTRODUCTION

  • Expensive VLM queries can abstain on visually indistinguishable trajectories, while aimless initial policies produce task-unrelated comparisons and little learning signal.Higher visual distinguishability is associated with lower abstention and higher preference accuracy.Source: §I · p. 1
  • Distinguishability-aware selection predicts VLM answerability from multi-scale visual features and weights information gain to prioritize answerable, informative comparisons.The approach uses online-adapted modeling to reduce wasted queries.Source: §I · pp. 1–2
  • Adaptive reward transition uses VLM-synthesized rewards to guide initial exploration before switching to preference-based learning.The synthesized rewards guide exploration toward task-relevant regions despite lacking precision for task completion.Source: §I · pp. 1–2
  • 88.9% average success across six Meta-World tasks, 23.3 percentage points above the strongest VLM-based baseline, with fewer wasted queries and reduced wall-clock training time.The full online pipeline was also run on a real Mobile ALOHA robot.Source: §I · p. 1

A. VLM-guided Robot Policy Learning.

  • SPEAR builds on VLMs as automated preference oracles rather than using them for direct reward specification or manually generated reward programs.This preference-based paradigm seeks to avoid ongoing human labeling and manual reward engineering.Source: §I · p. 1, A. VLM-guided Robot Policy Learning · p. 2

B. Preference-based Reinforcement Learning.

Preference-based reinforcement learning learns rewards from pairwise trajectory comparisons, but human labeling is costly and scripted oracles may rely on privileged reward information. This section formulates image-based VLM preference learning and fits a surrogate reward model from non-abstaining labels.

B. Preference-based Reinforcement Learning.

  • Human preference labeling is costly and slow, while scripted teachers with privileged access to ground-truth rewards create an unrealistic training regime.VLMs offer an automated oracle option, but querying costly, fallible models efficiently remains challenging.Source: B. Preference-based Reinforcement… · p. 2

1) Problem Formulation:

  • The manipulation problem is modeled as a POMDP with image observations and policies acting on observations, without access to a reward function.Policies are learned from VLM preference feedback while training a surrogate reward model.Source: 1) Problem Formulation · p. 2

2) VLM Preference Acquisition:

  • For a task description, the VLM compares two observation sub-trajectories and returns a label indicating which is preferred or abstains when evidence is insufficient.Abstention denotes no clear progress difference or insufficient evidence.Source: 2) VLM Preference Acquisition · p. 2

3) Reward Modeling:

  • A neural reward model uses Bradley–Terry modeling to assign likelihoods to trajectory preferences and is trained by maximizing log-likelihood on labeled pairs.Abstentions are excluded from the preference dataset used for optimization.Source: 3) Reward Modeling · p. 2

4) The General PbRL Learning Framework:

The general PbRL loop alternates policy exploration, VLM preference queries, and reward-model updates. SPEAR modifies query selection to account for answerability as well as information gain, addressing costly abstentions.

4) The General PbRL Learning Framework:

  • The learning loop repeatedly collects trajectories, selects pairs for VLM preference elicitation, augments the preference dataset, and updates the reward model until convergence or a budget limit.Query selection matters because VLM calls are costly and may abstain.Source: 4) The General PbRL Learning Framework · p. 2

IV. SPEAR: SELECTIVE PREFERENCE ELICITATION WITH ADAPTIVE REWARDS

  • SPEAR combines distinguishability-aware query selection with adaptive reward transition, using a coarse warm-up reward before preference-based refinement.The two mechanisms address costly abstentions and early exploration that lacks task progress.Source: §IV · p. 3, Figure 1 · p. 3

A. Distinguishability-Aware Query Selection

  • Classical acquisition functions assume effective labels, whereas VLMs may abstain on low-distinctiveness pairs; SPEAR predicts response likelihood and combines it with information gain to choose answerable, informative pairs.The method includes multi-scale visual features and a distinguishability model.Source: A. Distinguishability-Aware Query… · p. 3

1) Multi-Scale Visual Feature Extraction:

SPEAR represents trajectory-pair distinguishability with complementary global semantic and local spatial features. The combination is intended to capture differences at both scales, including subtle manipulation progress that global embeddings may miss.

1) Multi-Scale Visual Feature Extraction:

2) Distinguishability Prediction Model:

SPEAR predicts whether the VLM will answer a comparison and combines that estimate with preference-model information gain. It diversifies selected batches to avoid redundant, near-duplicate queries.

2) Distinguishability Prediction Model:

  • A lightweight predictor estimates whether the VLM will return a label rather than abstain, and adapts online as the policy explores new regions.It is trained on accumulated VLM responses and periodically recalibrated on a sliding window of recent queries.Source: 2) Distinguishability Prediction Model · p. 4

3) Integrated Query Acquisition:

  • SPEAR weights expected information gain by predicted distinguishability, favoring uncertain pairs the VLM is likely to distinguish.Source: 3) Integrated Query Acquisition · p. 4
  • Facility-location optimization diversifies high-scoring query batches in VLM embedding space so selected pairs cover varied progress states rather than near-duplicates.Source: 3) Integrated Query Acquisition · p. 4

4) VLM Query Protocol:

SPEAR uses a conservative, decomposed VLM comparison protocol and an adaptive reward transition to address preference-label errors and early exploration difficulties. Warm-up rewards guide task-related behavior before preference-based refinement, with relay exploration providing additional coverage after the transition.

4) VLM Query Protocol:

  • The decomposed, conservative VLM protocol addresses positional bias and cross-image attribution errors, substantially improving label quality and enabling learning where the original prompt fails.Source: 4) VLM Query Protocol · p. 4

B. Bootstrapped Exploration through Adaptive Reward Transition

1) Warm-up Reward Synthesis:

  • Warm-up rewards use realistically measurable geometric cues rather than simulator-privileged signals, encouraging coarse progress such as approach, alignment, and initial contact.Source: 1) Warm-up Reward Synthesis · p. 4

2) Adaptive Reward Transition:

  • Training switches from the warm-up reward to the learned preference-based reward when warm-up return stops improving for Ktrans consecutive episodes.Source: 2) Adaptive Reward Transition · p. 4

3) Relay Exploration for Coverage:

V. EXPERIMENTS

Across six Meta-World manipulation tasks, SPEAR outperforms the RL-VLM-F baseline at final evaluation. The experiments use image observations, a fixed query schedule, and comparisons against four representative VLM-guided RL approaches.

1) Tasks and Environment:

  • Evaluation covers six Meta-World tasks using image observations for VLM queries and reward-model training, with policies trained via SAC.Source: 1) Tasks and Environment · p. 4

1) Task Performance:

2) Baselines:

  • The four comparison approaches are VLM-RM, Text2Reward, PEBBLE with a scripted oracle, and RL-VLM-F with random query selection.Source: 2) Baselines · p. 5

3) Evaluation Schemes:

  • Each method uses five random seeds; evaluations occur every 10K environment steps, with 40 comparisons collected on the same fixed cadence and no replacement queries for abstentions.Policies are evaluated over 50 episodes, and experiments use 8 NVIDIA A800 GPUs.Source: 3) Evaluation Schemes · p. 5

1) Task Performance:

  • Text2Reward achieves 87.7% versus SPEAR's 84.9% final success rate on window-open, while the reported overall comparison favors SPEAR.Source: 1) Task Performance · p. 5

2) Query Efficiency Analysis:

SPEAR reduces query waste by selecting comparisons that are both informative and answerable, while VLM preference accuracy improves as visual distinguishability increases.

2) Query Efficiency Analysis:

  • At matched 30% success, SPEAR used 960 queries with 29.6% abstention and 9.1 hours, versus RL-VLM-F’s 1640 queries with 55.5% abstention and 13.6 hours.The table reports query efficiency and wall-clock cost at this success level.Source: 2) Query Efficiency Analysis · p. 5, Table III · p. 6
  • SPEAR’s learning curves consistently outperform baselines across six Meta-World tasks in mid-range and final performance.Shaded regions show standard deviation over five seeds.Source: Figure 2 · p. 6

3) VLM Reliability Diagnostics:

  • Across five VLM backbones, abstention decreased and accuracy increased with visual distinguishability, reaching 100% accuracy at sufficiently large frame gaps.With conservative prompting, average preference accuracy was 92.3%; the diagnostic demos were used offline, not for training or query collection.Source: 3) VLM Reliability Diagnostics · pp. 5–6, Table IV · p. 6

C. Ablation Studies

The ablations evaluate a simulated VLM preference oracle, compare query-selection criteria, and test distinguishability-predictor features and calibration.

1) Validity of Simulated Preference Oracle:

  • The simulated oracle fits abstention and correctness probabilities from five-backbone diagnostics, using reward difference as a heuristic distinguishability proxy, and is used only for ablations.A binomial GAM with B-spline smooths models the probabilities; the oracle matches the diagnostic trend, but the proxy is heuristic.Source: 1) Validity of Simulated Preference… · p. 6, Figure 3 · p. 7

2) Query Acquisition Mechanisms:

  • 80.7% final success came from integrated query weighting, compared with 69.7% for distinguishability-only, 21.0% for entropy-only, and 14.0% for random selection on drawer-open.Entropy-only selection can favor uncertain but unanswerable pairs, while distinguishability-only can favor easy pairs with little information gain.Source: 2) Query Acquisition Mechanisms · p. 6, Table V · p. 7

3) Distinguishability Model Validation:

  • Multi-scale features achieved the lowest prediction error (Brier score 0.1508, MAE 0.2950), while temperature scaling improved calibration and reduced both errors.The reported combination of global semantic understanding and local spatial discrimination supports more effective query selection and lower online abstention.Source: 3) Distinguishability Model Validation · p. 7, Table VII · p. 7

4) Exploration Mechanisms:

The exploration ablation tests warm-up reward guidance alongside relay exploration, and the paper also evaluates the full online pipeline on a physical robot.

4) Exploration Mechanisms:

  • 92.7% mid-training and 88.3% final success were achieved by combining warm-up rewards with relay exploration.Warm-up reward transition alone raised mid-training success to 39.8% from the 25.6% final success of random initialization with standard ϵ-greedy exploration; relay alone could not replace directional guidance.Source: 4) Exploration Mechanisms · p. 7, Table VI · p. 7

D. Real-Robot Experiments

  • SPEAR ran its full online exploration and preference-learning pipeline directly on Mobile ALOHA, without sim-to-real transfer.The real-robot settings included drawer-open, window-open, and soccer, as shown in Figure 4.Source: D. Real-Robot Experiments · p. 7, Figure 4 · p. 7

1) Platform and Tasks:

The real-robot evaluation uses a fixed side-view camera and online interaction for three manipulation tasks; the paper reports task convergence and addresses unsafe contact during exploration.

1) Platform and Tasks:

  • The platform used Mobile ALOHA’s left arm, a fixed side-view RGB camera, Cartesian end-effector increments, and a gripper open/close command.Each episode comprised 50 exploration steps; objects were manually reset between episodes.Source: 1) Platform and Tasks · p. 7

3) Results:

  • SPEAR learned drawer-open, window-open, and soccer directly on the physical robot, converging in 18.2K, 16.6K, and 12.6K steps, respectively.Under the fixed query cadence, these corresponded to 720, 660, and 500 preference comparisons.Source: 3) Results · p. 7

4) Safety and failure modes:

  • Unsafe contact is a primary failure mode in unconstrained exploration; the study restricts the workspace and penalizes commanded motions blocked by collisions or environmental constraints.The workspace restriction is intended to preserve sufficient learning coverage while maintaining a safe operating region.Source: 4) Safety and failure modes · pp. 7–8

VI. CONCLUSION

  • The conclusion presents SPEAR as a practical VLM-guided preference-learning framework for online manipulation without manual reward engineering or human annotation.It reports improved sample efficiency and final performance across diverse tasks.Source: §VI · p. 8
Loading 2610.21621v1…