PaperlayerView exact revision on arXiv

Source-linked AI summary

Steering Generative Robot Policies with Lexicographic Preferences

Yixuan Jia, Jonathan P. How

arXiv:2609.15014v1cs.ROcs.AI

TL;DR

Pretrained generative robot policies can miss deployment-specific requirements and preferences, which may also have an explicit priority order. This paper steers frozen policies at inference time with barrier guidance and lexicographic candidate selection. It reports improved navigation performance and compliance, transfer to manipulation without reduced task success, and more robust parameter sensitivity than a fixed weight in a controlled study.

  • Problem

    Pretrained policies may not represent deployment-specific requirements and preferences, and fixed weighted sums cannot guarantee a priority order when objective ranges vary across deployments.

    Source: §I · p. 1

  • Method

    The method applies dynamic-barrier guidance to sampled trajectories and lexicographically filters candidates at inference time without changing policy weights.

    Source: §VII · pp. 10–11

  • Results

    The method improves navigation success, preference compliance, and traversability over the frozen policy, transfers to LIBERO without reducing task success, and maintains comparable best performance across a wider parameter range than a fixed weight in D3IL.

    Source: C. Summary · p. 10, §VII · p. 11

  • Takeaways & Limitations

    When cost scales vary between episodes, the navigation results favor explicit ordering, while manipulation results show that a dynamic barrier can avoid finding and retuning a narrowly effective fixed weight.

    Source: C. Summary · p. 10

    Steering is local to the policy's proposals and cannot recover a goal the frozen policy cannot represent or provide missing global guidance.

    Source: §VII · p. 11

Abstract

from arXiv · show

Pretrained generative robot policies can produce effective behaviors across diverse environments, but deployment can lead to requirements and preferences that may not have been represented during training. Furthermore, at deployment, an operator, user, or application may assign these requirements and preferences a priority order that can vary across deployments. For example, embodiment-specific feasibility constraints may need to be satisfied first, while user-specific preferences guide behavior among the feasible options. We show that a frozen generative robot policy---based on either diffusion or flow matching---can be steered at inference time to respect such lexicographically ordered deployment objectives. To achieve this, we introduce two modifications to the sampler. First, we apply dynamic-barrier guidance to sampled trajectories, constraining lower-priority updates so that higher-priority costs do not increase (up to first order). Second, we select the executed sample using a cascade that successively filters candidate samples according to each priority level. The policy weights remain unchanged. On a navigation benchmark, we demonstrate that our method improves success, traversability, and preference compliance over the frozen policy, and achieves substantially better compliance than tuned weighted-sum baselines. The same method transfers to a flow-matching manipulation policy on LIBERO, where it improves compliance without reducing task success. A controlled manipulation study further shows that, in settings where a fixed weight can match the desired ordering, the dynamic barrier reaches comparable best performance over a substantially wider range of parameter settings.

I. INTRODUCTION

Pretrained policies can miss deployment-specific feasibility constraints and preferences, while deployments may impose an explicit priority order among them. The paper steers frozen policies at inference time to respect that order rather than relying on fixed tradeoffs.

I. INTRODUCTION

  • Deployment adaptation must preserve the pretrained policy while ensuring that lower-priority improvements do not compensate for higher-priority violations.Fixed weights can fail when objective ranges vary across deployment instances.Source: §I · pp. 1–2
  • The method combines dynamic-barrier guidance for individual sampled trajectories with lexicographic filtering to select the candidate to execute.Both mechanisms operate on trajectories, leaving policy weights unchanged and supporting diffusion and flow-matching samplers.Source: §I · p. 2

B. Prioritized and Lexicographic Objectives

  • The paper contributes a sampler-agnostic approach, a last-mile delivery benchmark, and evaluations on LIBERO and D3IL that examine compliance and tuning sensitivity.The reported results include better navigation compliance than tuned weighted-sum baselines and a wider range of parameter settings with comparable best performance in D3IL.Source: §I · p. 2

A. Inference-Time Steering of Generative Policies

  • Prior steering methods generally use weighted costs or a single reward, while the proposed approach handles explicitly ordered runtime costs within a frozen policy's sampling process.Unlike Pareto Navigation Gradient Descent, which treats objectives as an unordered set, this method combines prioritized gradient steering with lexicographic candidate selection.Source: A. Inference-Time Steering of… · p. 2, B. Prioritized and Lexicographic… · p. 2

III. METHOD

The method applies ordered runtime costs to workspace trajectories represented by samples from a frozen generative policy. It imposes the priority ordering both while editing individual samples and when choosing which candidate to execute.

III. METHOD

  • A frozen policy iteratively refines multiple candidate action sequences, each represented as a sample over a finite action horizon.Candidates start from noise and are refined through learned sampling steps.Source: A. Problem Setting · p. 2
  • Runtime costs are evaluated on decoded workspace trajectories, with the decoder mapping samples to the paths they represent.The decoder integrates sampled waypoint displacements in navigation and maps action chunks to predicted end-effector paths in manipulation.Source: A. Problem Setting · pp. 2–3
  • At deployment, the method adds differentiable runtime costs in a specified lexicographic order while keeping policy weights, sampling schedule, and observations fixed.It applies the ordering through barrier guidance on individual samples and lexicographic selection among candidates.Source: A. Problem Setting · p. 3

B. Lexicographic Barrier Guidance

Lexicographic barrier guidance edits each sampled trajectory to improve a lower-priority preference while preserving higher-priority costs to first order. The update is the smallest correction to ordinary preference descent that satisfies a dynamic barrier condition.

B. Lexicographic Barrier Guidance

  • In navigation, the method edits a seven-control-point B-spline trajectory, while manipulation edits the trajectory directly.The implementation refines completed samples using repeated barrier updates and reconstructs the sample after editing the trajectory parameterization.Source: B. Lexicographic Barrier Guidance · p. 3
  • The dynamic barrier requires a positive decrease in the prioritized cost when that cost is away from its desired value.Its required decrease grows with cost but is capped by the local gradient's capacity to support progress.Source: B. Lexicographic Barrier Guidance · p. 4
  • The steering direction stays as close as possible to preference descent while satisfying a barrier on the higher-priority feasibility cost.The intervention edits the sample between sampler steps rather than modifying the network output, so it has the same form for diffusion and flow-matching samplers.Source: B. Lexicographic Barrier Guidance · pp. 3–4
  • A dynamic multiplier adds only the smallest correction needed when ordinary preference descent would violate the feasibility barrier.The multiplier depends on the current cost, its gradient magnitude, and alignment with the preference gradient; it vanishes when preference descent already respects the barrier.Source: B. Lexicographic Barrier Guidance · p. 4
  • The barrier construction extends to deeper hierarchies by assigning a barrier requirement to each higher-priority cost and enforcing compatible conditions together.When conditions conflict, the general formulation relaxes lower-priority conditions only after higher-priority conditions are satisfied as well as possible.Source: B. Lexicographic Barrier Guidance · p. 4
  • The two-cost experimental case has one prioritized feasibility cost and one preference objective, reducing to a single-barrier quadratic program when the feasibility gradient is nonzero.If the feasibility gradient vanishes, the barrier provides no local information to alter the trajectory.Source: B. Lexicographic Barrier Guidance · p. 4

C. Lexicographic Candidate Selection

Lexicographic candidate selection filters the policy's candidate trajectories one priority level at a time. Lower-priority objectives distinguish candidates that are approximately equivalent on higher-priority costs but cannot compensate for worse higher-priority costs.

C. Lexicographic Candidate Selection

  • The selection cascade filters candidates sequentially by ordered costs, considering each priority level only among candidates surviving the previous levels.This prevents a candidate rejected on a higher-priority cost from returning through better performance on a lower-priority objective.Source: C. Lexicographic Candidate Selection · pp. 4–5
  • At each level, the cost threshold is measured from the minimum among the current candidates and uses a tolerance in that cost's units.The minimum-cost candidate always survives each filtering stage.Source: C. Lexicographic Candidate Selection · p. 5
  • After prioritized costs, the cascade filters by task progress and uses a policy value head only as a final tie-break when available.Higher progress scores indicate better progress.Source: C. Lexicographic Candidate Selection · p. 5
  • Barrier guidance and candidate selection impose the same priority order at complementary stages: trajectory modification and final candidate choice.The selection cascade prevents the final choice from reintroducing a trade between prioritized costs.Source: C. Lexicographic Candidate Selection · p. 5

IV. LAST-MILE DELIVERY WITH A DIFFUSION POLICY

The last-mile delivery benchmark tests whether a robot can reach residential doors while respecting both what it can traverse and where it should drive. It treats avoiding lawn as lower priority than avoiding an untraversable curb.

IV. LAST-MILE DELIVERY WITH A DIFFUSION POLICY

  • Residential lawns are physically traversable but undesirable, whereas a tall curb may be physically impassable for the deployed robot.The benchmark focuses on reaching different residences’ front doors.Source: §IV · p. 5
  • The robot must not trade a traversable route for avoiding lawn: embodiment feasibility takes priority over preference.The benchmark includes routes where the direct path encounters both kinds of constraint.Source: §IV · p. 5
  • Direct routes often encounter both types of constraint, creating conflicts between feasibility and preference.Source: §IV · p. 5

A. Simulation Setup

The simulation uses a Scout 2.0 in a residential town and evaluates 60 sidewalk start–goal pairs selected to expose lawn and step constraints while retaining feasible pavement routes.

A. Simulation Setup

  • The benchmark uses NVIDIA Isaac Sim in a residential town taken from, which comes from CARLA Town01.Source: A. Simulation Setup · p. 5
  • The AgileX Scout 2.0 has an RGB-D camera and LiDAR; step-climbing tests support 0.10 m as a conservative upper bound on traversable step height.Source: A. Simulation Setup · p. 5
  • The 60 start–goal pairs span 9.2–66 m, and every pair has a feasible pavement-only route no more than 1.61× its free-space route.Median route length is 23.4 m; the pavement-only route ratio has a median of 1.04×.Source: A. Simulation Setup · p. 5
  • Of the 60 pairs, 55 direct routes cross lawn and 43 encounter a step ≥ 0.10 m; success means reaching within 1 m in 300 s.Reported measures include lawn distance, zero-lawn episodes, traversability failures, curb episodes, and driven distance.Source: A. Simulation Setup · p. 5

B. Implementation

The implementation refines NavDP’s sampled trajectories using workspace costs, then applies a lexicographic cascade to select an executed trajectory. Goal re-encoding preserves bearing for goals ahead, while a behind-robot goal remains unrepresentable by the frozen policy’s interface.

B. Implementation

  • NavDP [4] samples 16 trajectories of 24 waypoints over 10 DDPM steps and executes the selected trajectory with a model-predictive controller.Source: B. Implementation · p. 5
  • Each path is formed in world coordinates, fit with a clamped cubic B-spline for refinement, then mapped back to waypoint displacements with yaw unchanged.Source: B. Implementation · p. 5
  • The preference field ramps over 0.35 m around lawn boundaries; the online LiDAR feasibility field marks steps ≥ 0.10 m and occluded cells as high and unknown risk.Risk is aggregated as the mean of the worst 20% within a 0.4 m footprint disc.Source: B. Implementation · pp. 5–6
  • In eight selected delivery episodes, the method drives 0.0 m on lawn and reaches the goal in all eight; some routes are longer than needed because policy and steering are local.The baselines drive up to 39 m on lawn; two lawn-free baseline routes end 41 m and 18 m short.Source: Figure 3 · p. 6
  • Barrier refinement follows the final DDPM step for 15 iterations, using seven spline control points with 0.15 m trust regions and a pinned first point.Updates are projected perpendicular to the local path tangent.Source: B. Implementation · p. 6
  • The lexicographic cascade uses feasibility, preference, and progress tolerances of 0.1, 0.05, and 1.5 m, respectively, with the value head used only at the final stage.Feasibility cost is horizon-discounted to avoid making candidates indistinguishable because of unknown risk in the far tail.Source: B. Implementation · p. 6
  • Goals farther than 10 m are rescaled to preserve bearing, while the original goal is retained for progress evaluation; goals behind the robot remain clipped into the forward range.The residual interface defect belongs to the frozen policy, not the steering method.Source: B. Implementation · p. 6

C. Baselines

The benchmark compares the frozen policy, tuned weighted-sum guidance, and lexicographic selection without gradient steering. Each reported configuration is run three times over the same 60 episodes.

C. Baselines

  • Every reported configuration is repeated three times on the same 60 episodes.Source: C. Baselines · p. 6
  • The unsteered baseline is frozen NavDP with only the goal re-encoding applied.Source: C. Baselines · p. 6
  • The weighted-sum baseline steers with ∇(wt g + wp f), fixes wt=1, and searches preference weight and step size on a 12-point grid.The search uses seven tuning episodes, keeps settings that lose no episode against unsteered NavDP, and ranks them by mean lawn distance.Source: C. Baselines · p. 6
  • Selection Only applies the lexicographic cascade to NavDP’s original candidates without gradient steering, isolating ordered selection from trajectory modification.Source: C. Baselines · p. 7

D. Results

On navigation, lexicographic steering improves compliance and traversability over weighted-sum guidance, though paths are longer; failure analysis identifies cases local steering cannot resolve. In manipulation, the method transfers to LIBERO without reducing task success, and a controlled study finds comparable best performance across a wider range of settings.

1) Baseline comparisons:

  • 56.0 successful episodes versus 49.7–51.0; on-lawn distance is 2.60m versus 6.34–6.84m, and traversability failures are 2.0 versus 5.7–8.3 per repeat.The method drives approximately 9m farther per episode than weighted-sum guidance.Source: 1) Baseline comparisons · p. 7
  • Cascade-only selection raises success from 44.3 to 48.0 episodes, lowers mean on-lawn distance from 9.24 to 7.01m, and reduces traversability failures from 12.0 to 9.0 per repeat.Driven distance stays at 27.2m, while the larger gains of the full method come from gradient steering creating new trajectories.Source: 1) Baseline comparisons · p. 7

2) Failure analysis:

  • Local steering can lead to dead ends because runtime costs do not represent downstream reachability; avoiding these traps requires longer-horizon or global guidance.The analysis also reports goal-behind-robot wanders that goal re-encoding cannot repair.Source: 2) Failure analysis · p. 7

V. MANIPULATION WITH A FLOW-MATCHING POLICY

  • In the controlled manipulation study, both methods reach the same best performance, but the weighted sum does so only in a narrow weight band while the proposed method does so across a wide range.The manipulation setting has shorter horizons and more limited cost magnitudes, where a well-tuned fixed weight can more readily preserve the desired ordering.Source: §V · p. 7

A. Transfer to a Flow-Matching Policy

B. How Much Tuning Does Each Method Need?

In the controlled manipulation study, both methods can reach the same best performance, but the dynamic barrier maintains it across a wider parameter range and changing step sizes. The fixed-weight optimum shifts with the operating point, while the barrier’s performance is comparatively insensitive to its parameter.

B. How Much Tuning Does Each Method Need?

  • The controlled D3IL avoiding task uses a rod, six similar-scale obstacles, opposed feasibility and route-preference costs, and compares success against gate agreement across each method’s parameter sweep.The sweep uses the same step size and compares success at equal preference compliance because the methods’ parameters are not equivalent quantities.Source: B. How Much Tuning Does Each Method… · pp. 8–9
  • Full success holds for every αb ≥0.1 through 10^3, whereas the weighted sum reaches its best values only near w=1.The weighted sum loses performance on either side of w=1.Source: B. How Much Tuning Does Each Method… · p. 9, Table III · p. 9, Figure 5 · p. 10
  • The barrier multiplier cannot overshoot its target decrease, so raising αb beyond what is needed changes nothing; αb=0 still cancels updates that would increase feasibility cost.Source: B. How Much Tuning Does Each Method… · p. 9
  • At η=8, w=1 is best, but at η=32 it drops to 55.6%; the proposed method uses the same αb and matches or exceeds the best fixed weight at every step size.Across repeats, the proposed method’s standard deviation is 0.0 at every setting, while the fixed weight reaches 8.2 points.Source: Table IV · p. 9, B. How Much Tuning Does Each Method… · p. 10
  • The authors do not claim a significant mean-success advantage; the measured difference is in sensitivity and repeatability, not best achievable score.Source: B. How Much Tuning Does Each Method… · p. 10

C. Summary

The manipulation experiments show that similar cost scales let a fixed weight preserve the desired priority, while the barrier avoids searching for a weight that works across operating points. Navigation represents the contrasting case, where changing cost scales defeat a single fixed weight.

C. Summary

  • When cost scales are similar across a task, a fixed weight can preserve the desired priority order.Source: C. Summary · p. 10
  • In that regime, any barrier constant of 0.1 or above works without retuning as steering strength changes, while the fixed weight’s narrow working range shifts with the operating point.Source: C. Summary · p. 10
  • In navigation, cost scales vary between episodes, so no single weight serves them all.Source: C. Summary · p. 10

VI. TOY EXAMPLE WITH DEEPER HIERARCHY

The deeper-hierarchy toy example applies the method to ordered wall, lawn, and path-length objectives, and compares it with weighted sums. The conclusion reports transfer to diffusion and flow-matching policies while noting that steering remains local to the frozen policy’s proposals.

VI. TOY EXAMPLE WITH DEEPER HIERARCHY

  • At each Euler step, dynamic-barrier guidance pursues shorter paths while protecting wall and lawn costs to first order, and the cascade selects across wall, lawn, then path-length priorities.The QP multipliers for wall and lawn change as paths change, unlike fixed weighted-sum weights.Source: §VI · p. 10, Figure 6 · p. 10, Figure 7 · p. 10
  • The proposed method has no wall collisions exceeding 1 cm, reaches the goal in 98.5% of episodes, and averages 2.24 m on lawn; 66.5% of unsteered paths enter a wall by more than 1 cm.Source: §VI · p. 10
  • Only five of 20 weighted-sum settings avoid all wall collisions; the best safe setting has similar lawn travel, while another enters a wall in 189/200 episodes.The best safe setting is (w1, w2) = (30, 3), with 2.19 m lawn travel versus 2.24 m for the proposed method; (30, 10) produces the reported collisions.Source: §VI · p. 10

VII. CONCLUSIONS

  • The same sampled-trajectory mechanisms apply to diffusion and flow-matching policies without retraining, and the paper reports improved navigation outcomes, LIBERO transfer, and broader parameter robustness in controlled manipulation.Source: §VII · pp. 10–11
  • Steering is local: it reshapes trajectories near the policy’s proposals and cannot recover an unrepresentable goal or supply missing global guidance.Source: §VII · p. 11
Loading 2609.15014v1…