PaperlayerView exact revision on arXiv

Source-linked AI summary

PreferenceFlow: Test-Time Guidance of Flow-Matching Robot Policies from Human Interventions

Yiqi Tang, Diyuan Shi, Runze Li, Donglin Wang

arXiv:2609.36872v1cs.RO

TL;DR

Pretrained flow policies can make local deployment errors, and reward signals for real-world robot manipulation may be difficult to specify or obtain. PreferenceFlow trains a preference model from paired human interventions and robot action chunks, then uses its gradient to guide a frozen flow policy at test time. Across four real-world precision insertion tasks, it achieves 90.5% mean success versus 69% for the frozen policy.

  • Problem

    Pretrained flow policies remain susceptible to local errors under deployment distribution shift, while reward signals for real-world manipulation can be difficult to specify or obtain.

    Source: §I · p. 1

  • Method

    PreferenceFlow learns state-conditioned preferences from human–robot action-chunk comparisons and uses the preference gradient in QGF’s sampling update without updating the base policy.

    Source: §I · p. 1

  • Results

    90.5% mean success across four real-world precision insertion tasks, compared with 69% for the frozen policy.

    Source: D. Main Results · p. 6

  • Takeaways & Limitations

    The evaluated results support human interventions as local preference supervision for guiding frozen generative robot policies without environment rewards.

    Source: §VII · p. 7

    PreferenceFlow has no formal guarantee of long-horizon improvement, and evaluation covers four visually guided insertion tasks on one Franka robot.

    Source: §VI · p. 7

Abstract

from arXiv · show

Flow-matching policies can represent complex robot behaviors but remain susceptible to local errors under distribution shift at deployment. Many reinforcement learning approaches to policy improvement require reward signals that are difficult to specify or obtain in real-world manipulation. We present PreferenceFlow, a framework for improving a pretrained flow policy at test time without environment rewards or updates to the base policy. Human intervention chunks are paired with robot chunks generated from the same initial conditioning state to train a preference model. During infer- ence, we adopt the QGF sampling update, replacing its value gradient with the preference gradient evaluated at an estimated clean action. A gradient-cap loss penalizes excessive gradients on intervention pairs, while a zero-gradient loss discourages guidance near actions from expert demonstrations. On four real-world precision insertion tasks with a Franka robot, Pref- erenceFlow achieves a mean success rate of 90.5%, compared with 69% for the frozen policy. Ablations support the roles of gradient regularization and correctly ordered preference labels in the evaluated settings. These results demonstrate the utility of human interventions as local preference supervision for guiding frozen generative robot policies.

I. INTRODUCTION

Pretrained robot policies can make local execution errors, while updating their parameters is costly and may affect pretrained capabilities; reward-based guidance can also be difficult to support with informative rewards or critics. PreferenceFlow learns preferences from human–robot action-chunk comparisons and uses them to guide a frozen flow policy, with gradient regularization and real-world insertion-task results.

I. INTRODUCTION

  • Human intervention chunks are treated as preferred to robot candidates regenerated from the same window-start state, providing local supervision where correction is needed.Relative judgments can be easier to provide than explicit reward functions.Source: §I · p. 1
  • PreferenceFlow learns a state-conditioned preference score from human–robot action-chunk comparisons using a Bradley–Terry objective, then substitutes that score for QGF’s learned Q-function [5].At each sampling step, it evaluates the preference gradient at an estimated clean action and adds it to the frozen velocity field.Source: §I · p. 1
  • A gradient-cap loss penalizes excessive gradients on intervention pairs, while a zero-gradient loss suppresses guidance near demonstrated routine actions.The framework uses intervention comparisons without environment rewards or updates to the base policy.Source: §I · p. 1
  • 90.5% mean success across four real-world precision insertion tasks, exceeding the frozen policy by 21.5 percentage points.Ablations examine gradient regularization, architecture, intervention-data quantity, and preference-label ordering.Source: §I · p. 1

A. Generative Robot Policies and Robot Foundation Models

Generative robot policies use iterative sampling to produce action chunks conditioned on observations, and pretrained robot foundation models combine action generation with broad pretraining. Prior improvement methods update policies, train auxiliary modules, or steer sampling; this paper's related work situates preference guidance among methods using human input.

A. Generative Robot Policies and Robot Foundation Models

  • Diffusion policies iteratively denoise action sequences, while flow matching learns a velocity field that transports noise to actions; both support test-time guidance.Robot policies such as π0 generate continuous action chunks conditioned on visual and language inputs.Source: A. Generative Robot Policies and Robot… · pp. 1–2

B. Improving Pretrained Robot Policies

  • Policy-improvement approaches include parameter adaptation, auxiliary controllers, critic-gradient flow guidance, external action steering, and runtime constraints.The described methods use reward or value information or map external action guidance back to initial noise; PreferenceFlow instead uses local preferences without environment rewards or base-policy updates.Source: B. Improving Pretrained Robot Policies · pp. 2–3

C. Human Preferences and Interactive Robot Learning

  • Preference-learning work includes ranked demonstrations, preference queries, and direct policy optimization –, while interactive imitation learning collects expert supervision at learner-visited states –[29].Here, intervention-derived action-chunk comparisons train a preference model whose action gradient guides a frozen flow sampler.Source: C. Human Preferences and Interactive… · p. 3

III. PROBLEM FORMULATION

The formulation treats the policy input as a conditioning state and its output as an action chunk, generated by integrating a flow velocity field from noise. Preference supervision compares human and robot chunks sharing an initial conditioning state, although their subsequent state trajectories may differ.

III. PROBLEM FORMULATION

  • An action chunk is represented as a sequence with horizon H and action dimension D, conditioned on policy input s.Flow matching uses a linear conditional path from noise z to action a, while the inference trajectory need not follow an individual training path.Source: §III · p. 3
  • At inference, the flow sampler starts from Gaussian noise and applies T Euler steps using the velocity field, with step size δ = 1/T.The pretrained parameters remain frozen during adaptation.Source: §III · p. 3
  • For each intervention window, a human action chunk is compared with a robot chunk generated from the same initial conditioning state.The human actions are collected under closed-loop execution, so sharing an initial state does not imply identical state trajectories.Source: §III · p. 3
  • The preference model uses candidate action tokens to attend to the frozen policy’s conditioning representation, and its action gradient supplies the flow-sampling guidance signal.The schematic omits robot-state inputs and per-step score heads followed by mean aggregation.Source: Figure 2 · p. 3

IV. PREFERENCE-GUIDED FLOW POLICIES

The section is represented by a pointer to the overview of PreferenceFlow’s data construction, preference learning, and guided flow inference.

IV. PREFERENCE-GUIDED FLOW POLICIES

  • Figure 1 summarizes preference-data construction, preference learning, and guided flow inference in PreferenceFlow.The overview links these stages in the method pipeline.Source: §IV · p. 3

A. Preference Model Architecture

The preference model scores action chunks using frozen policy conditioning features and independent action-token scoring. Training combines intervention-derived preference ranking with gradient regularization on intervention pairs and near expert demonstrations.

A. Preference Model Architecture

  • Each action step independently cross-attends to frozen visual, language, and robot-state tokens, and a score head averages per-step scores.The model scores the first K ≤ H steps of each action chunk; its score head receives cross-attention outputs without a direct action-only residual.Source: A. Preference Model Architecture · p. 3

B. Preference Model Training

  • Bradley–Terry training ranks intervention-derived action pairs, supervising relative preferences without calibrating scores to expected returns.The score model is trained from intervention-derived pairs using a Bradley–Terry likelihood.Source: B. Preference Model Training · p. 4
  • The training objective adds a soft gradient-cap penalty to preference ranking on intervention pairs, discouraging excessive action gradients while retaining smaller corrective gradients.The penalty acts only above gmax, with actions and gradients evaluated in normalized coordinates.Source: B. Preference Model Training · p. 4
  • A zero-gradient objective penalizes preference gradients near expert-demonstrated action chunks, complementing the intervention-pair gradient cap.Intervention pairs supply preference and gradient-cap losses, while expert demonstrations supply the zero-gradient loss; only preference-model parameters are optimized.Source: B. Preference Model Training · p. 4

C. Preference-Guided Flow Sampling

Preference-guided sampling uses a KL-regularized preference objective to motivate reweighting the frozen policy toward higher-scoring actions. At inference, it replaces QGF’s Q-function with the learned preference score in a flow-guidance update.

C. Preference-Guided Flow Sampling

  • The KL-regularized objective yields an exponential tilt of the frozen policy toward higher-scoring actions within its support, assuming a finite partition function.The parameter β controls deviation from the base distribution; this objective motivates preference-based improvement but does not itself determine a flow velocity field.Source: C. Preference-Guided Flow Sampling · p. 4
  • At each sampling step, the method estimates a clean endpoint with one Euler extrapolation because the preference model is trained on clean actions.The preference gradient is evaluated at this estimated endpoint without differentiating through the frozen velocity field.Source: C. Preference-Guided Flow Sampling · p. 4
  • The guided Euler update adds the preference gradient to the frozen velocity field, with a tunable guidance weight and zero direct gradient for chunk steps beyond the first K.The gradient is recomputed at every sampling step; setting the guidance weight to zero recovers the base sampler, and both models remain fixed during inference.Source: C. Preference-Guided Flow Sampling · p. 4

V. EXPERIMENTAL EVALUATION

The evaluation asks whether PreferenceFlow improves real-robot success relative to the frozen policy and other adaptation baselines. It also examines how regularization, data quantity, architecture, and preference-label ordering relate to task success.

V. EXPERIMENTAL EVALUATION

  • The evaluation tests whether PreferenceFlow improves real-robot task success over the frozen policy and how it compares with intervention-, value-, and inversion-based baselines.This is the first of the four stated experimental questions.Source: §V · p. 4
  • The experiments examine the effects of gradient-cap and zero-gradient objectives at a shared guidance weight and after adjustment for stable deployment.This is the second stated experimental question.Source: §V · p. 4
  • The evaluation asks how intervention-data quantity and preference-model architecture affect task success.This is the third stated experimental question.Source: §V · p. 4
  • The experiments test how randomized preference supervision affects task success and whether the learned score supports sample-based action reranking.This is the fourth stated experimental question.Source: §V · p. 4

A. Tasks and Experimental Setup

The study evaluates precision insertion on a single Franka robot and describes the intervention data, implementation, and adaptation baselines used. Each method is assessed over 50 trials per task, with success rate defined as the fraction of successful trials.

A. Tasks and Experimental Setup

  • Four precision insertion tasks require visual connector and port localization followed by precise position and orientation alignment under tight geometric tolerances.Small localization or pose errors can prevent insertion; the setup uses a wrist-mounted camera and two fixed cameras at 30 Hz, with 50 trials per method per task.Source: A. Tasks and Experimental Setup · pp. 4–5

B. Data and Implementation Details

  • Preference pairs use complete 10-step human intervention windows paired with the frozen policy’s first 10 actions from the same window-start state.The checkpoint is fine-tuned on 50 expert demonstrations and then frozen; expert demonstrations also supervise the zero-gradient objective.Source: B. Data and Implementation Details · p. 5

C. Baselines

  • The comparison includes a frozen task-adapted policy, Intervention BC, QGF, and FlowDAgger alongside PreferenceFlow.Intervention BC fine-tunes on takeover windows; QGF uses a critic trained on failed base-policy rollouts and successful expert demonstrations, while FlowDAgger learns structured initial noise from 50 intervention episodes per task. PreferenceFlow trains on the same 50 episodes, whose intervention durations total approximately 2–5 seconds.Source: C. Baselines · pp. 5–6

D. Main Results

PreferenceFlow achieves the strongest reported mean success among the compared methods, including the frozen policy and adaptation baselines. Gradient-objective ablations show lower success without either regularizer, including after guidance weights are adjusted for stable deployment.

D. Main Results

  • 90.5% mean success exceeded the frozen policy’s 69%, a 21.5-point gain representing 181 versus 138 successes in 200 trials.Gains were 24, 22, 24, and 16 percentage points on Insert Ethernet, USB, SATA, and Charger, respectively.Source: D. Main Results · p. 6
  • PreferenceFlow’s 90.5% mean exceeded Intervention BC’s 84.5%, QGF’s 76%, and FlowDAgger’s 78.5%.FlowDAgger’s use of complete 50-step chunks versus PreferenceFlow’s 10-step windows may affect usable supervision, so this comparison does not isolate that contribution.Source: D. Main Results · p. 6

E. Gradient Objective Ablation

  • At the full model’s guidance weight, removing the gradient-cap or zero-gradient loss reduced mean success from 90.5% to 15% or 5%, respectively; the preference-only variant was unstable.Removing the zero-gradient loss also removes demonstration-based anchors, so that ablation tests both the supervision and its regularization objective.Source: E. Gradient Objective Ablation · p. 6, Table II · p. 6
  • After guidance weights were reduced for stable deployment, mean success was 81% without gradient-cap, 74.5% without zero-gradient, and 69% with preference-only guidance.These results remained 9.5, 16, and 21.5 percentage points below the full objective, respectively.Source: E. Gradient Objective Ablation · p. 6, Table II · p. 6

F. Human-Intervention Data Ablation

Mean success rose as intervention episodes per task increased from zero to 50, while each successive block of ten episodes added less than the previous block. Expert demonstrations and training settings were held fixed.

F. Human-Intervention Data Ablation

  • Intervention episodes per task were varied while expert demonstrations and training settings remained fixed.The comparison therefore varies intervention-data quantity under those fixed conditions.Source: F. Human-Intervention Data Ablation · p. 6
  • Mean success increased from 69% with zero episodes to 78.5%, 83.5%, 87%, 89%, and 90.5% with 10, 20, 30, 40, and 50 episodes per task.The zero-episode condition is the frozen policy without preference guidance.Source: F. Human-Intervention Data Ablation · p. 7, Figure 4 · p. 7
  • Each additional block of 10 intervention episodes added 5, 3.5, 2, and 1.5 percentage points, respectively, showing diminishing gains over the tested range.The observed gains remained positive at every tested increment.Source: F. Human-Intervention Data Ablation · p. 7

G. Preference Model Architecture Ablation

PreferenceFlow’s evaluated results vary with preference-model architecture and label ordering, and its evidence is bounded by local guidance on four insertion tasks with one robot.

G. Preference Model Architecture Ablation

  • Adding temporal self-attention reduces mean success from 90.5% to 77%, a difference of 13.5 percentage points.The proposed explanation—that attention may capture human control style rather than task-relevant preferences—remains a hypothesis, not an established mechanism.Source: G. Preference Model Architecture… · p. 7

H. Score-Based and Negative Controls

  • 92% mean success from preference reranking exceeds gradient guidance’s 90.5% by 1.5 percentage points.Reranking uses five complete samples per decision, while gradient guidance uses one; reversing preference labels yields 63.5%, 27 points below PreferenceFlow and 5.5 below the frozen policy.Source: H. Score-Based and Negative Controls · p. 7, Table IV · p. 7

VI. LIMITATIONS AND DISCUSSION

  • The evaluation covers four visually guided insertion tasks on one Franka robot, and local preferences provide no formal guarantee of long-horizon improvement.The approach assumes the base policy reaches states from which local corrections can complete the task; broader applicability requires additional tasks, robots, and operators.Source: §VI · p. 7

VII. CONCLUSION

  • 90.5% mean success exceeds the frozen policy’s 69% and Intervention BC’s 84.5% on four real-world insertion tasks.The conclusion reports these findings alongside support for the training objectives, independent action-token scoring, and correctly ordered labels in the evaluated settings.Source: §VII · p. 7

Related papers

Loading 2609.36872v1…