PaperlayerView exact revision on arXiv

Source-linked AI summary

Inference-Time Policy Steering through Human Interactions

Yanwei Wang, Lirui Wang, Yilun Du, Balakumar Sundaralingam, Xuning Yang, Yu-Wei Chao, Claudia Perez-D'Arpino, Dieter Fox, Julie Shah

arXiv:2411.16627v2cs.ROcs.AIcs.HCcs.LG

TL;DR

Direct task-space interactions can steer frozen policies without fine-tuning, but interventions can exacerbate distribution shift. ITPS uses inference-time conditional sampling to align generated behavior with human objectives; the authors report an alignment–constraint trade-off and present stochastic diffusion sampling as improving alignment while maintaining constraints.

  • Problem

    Existing approaches to mitigating distribution shift largely focus on single-task settings, limiting their applicability to multi-task policies.

    Source: §I · p. 1

  • Method

    ITPS frames policy steering as conditional sampling from a frozen generative policy, combining likelihood constraints with inference-time objectives to align trajectories with user intent.

    Source: §I · p. 1

  • Results

    Across steering methods, improved alignment tends to bring more constraint violations and task failures; stochastic diffusion sampling improves alignment while maintaining constraints within the data manifold.

    Source: §I · pp. 1–2

  • Takeaways & Limitations

    The authors demonstrate steering policies through human interaction in simulation and real-world experiments.

    Source: §V · p. 6

    Producing behaviors aligned with human intent relies on an expensive sampling procedure.

    Source: §V · p. 6

Abstract

from arXiv · show

Generative policies trained with human demonstrations can autonomously accomplish multimodal, long-horizon tasks. However, during inference, humans are often removed from the policy execution loop, limiting the ability to guide a pre-trained policy towards a specific sub-goal or trajectory shape among multiple predictions. Naive human intervention may inadvertently exacerbate distribution shift, leading to constraint violations or execution failures. To better align policy output with human intent without inducing out-of-distribution errors, we propose an Inference-Time Policy Steering (ITPS) framework that leverages human interactions to bias the generative sampling process, rather than fine-tuning the policy on interaction data. We evaluate ITPS across three simulated and real-world benchmarks, testing three forms of human interaction and associated alignment distance metrics. Among six sampling strategies, our proposed stochastic sampling with diffusion policy achieves the best trade-off between alignment and distribution shift. Videos are available at https://yanweiw.github.io/itps/.

I. INTRODUCTION

ITPS steers frozen generative policies at inference using direct human interactions rather than fine-tuning. The framework aims to align behavior with user intent while accounting for the distribution-shift risk of intervention.

I. INTRODUCTION

  • Direct inference-time interventions can exacerbate distribution shift, while existing policies offer few ways to correct behavior when it misaligns with user intent.The paper links distribution shift in behavior cloning to execution failures.Source: §I · p. 1
  • The framework accepts task-space preferences expressed as point goals, trajectory sketches, or physical corrections, avoiding additional interaction-data training.The paper notes that language may not capture low-level, continuous intent as well as these direct interaction forms.Source: §I · p. 1
  • ITPS frames steering as conditional sampling from a learned generative policy, using likelihood constraints to produce valid actions aligned with user objectives without modifying the pretrained policy.This composes the pretrained policy with inference-time objectives rather than requiring policy fine-tuning.Source: §I · p. 1
  • Across steering methods, improved alignment tends to bring more constraint violations and task failures; stochastic diffusion sampling improves alignment while maintaining constraints within the data manifold.The paper frames this as an alignment–constraint satisfaction trade-off and proposes an MCMC procedure for diffusion policies to alleviate distribution shift.Source: §I · pp. 1–2

A. Steering Towards User Intent

The section frames steering as aligning frozen generative-policy trajectories with user intent while maintaining constraint satisfaction. It describes point, sketch, and physical-correction inputs and compares guided diffusion with stochastic sampling.

A. Steering Towards User Intent

  • Steering is defined as increasing Task Alignment for discrete preferences or Motion Alignment for continuous preferences while maximizing Constraint Satisfaction.Task Alignment is the percentage of predicted skills that execute intended tasks; Motion Alignment is the negative L2 distance to target trajectories, and Constraint Satisfaction measures plans satisfying physical constraints.Source: A. Steering Towards User Intent · p. 2
  • Point input maps a user-selected image coordinate to a 3D target state and minimizes the average L2 distance between trajectory states and that target.This lets users specify which objects to manipulate in a real-world kitchen environment.Source: A. Steering Towards User Intent · p. 2
  • Sketch input lets users specify continuous motion intent as a partial trajectory, with the sketch resampled to match the generated trajectory’s temporal dimension when needed.The directional path expresses trajectory-shape preferences in the robot’s workspace.Source: A. Steering Towards User Intent · p. 2
  • Physical corrections intervene directly in motion execution by overwriting the beginning of a generated trajectory with the user’s nudge.The overwritten portion can be the first k steps.Source: A. Steering Towards User Intent · p. 2
  • Diffusion Policy with stochastic sampling achieves the best alignment–constraint-satisfaction trade-off; multimodal DP with PR improves alignment without significant distribution shift.Source: Figure 4 · p. 3
  • When a target point misses every distribution mode, guided diffusion’s sum-based sampling introduces distribution shift, while stochastic sampling’s product-based sampling selects the closest in-distribution mode.Source: Figure 3 · p. 3

B. Inference-Time Interaction-Conditioned Sampling

The section presents six ways to condition generative trajectory sampling on user objectives, ranging from unchanged sampling and post-hoc edits or selection to diffusion-based steering. Stochastic Sampling uses annealed MCMC to combine the policy distribution with the alignment objective while retaining valid trajectories.

Sampling baselines

  • Random Sampling leaves the pretrained policy unchanged, while Output Perturbation modifies a sampled trajectory toward the user objective before resampling its remainder.Output Perturbation does not guarantee constraint-satisfying trajectories from the perturbed state.Source: B. Inference-Time… · p. 3

Post-Hoc Ranking

  • Post-Hoc Ranking selects the batch trajectory that minimizes the alignment objective, but cannot find an intended plan absent from the initial batch.The method can work when at least one generated sample is already close to the input.Source: B. Inference-Time… · p. 3

Biased Initialization

  • Biased Initialization starts reverse diffusion from a noise-corrupted user input, though sampling may still deviate from that input.The approach brings the process closer to the desired mode at initialization.Source: B. Inference-Time… · p. 3

Guided Diffusion

  • Guided Diffusion (GD) steers reverse diffusion with the alignment gradient, but its weighted sum of denoising and alignment gradients approximates a sum rather than a product distribution and can yield out-of-distribution samples.The guide ratio β_i controls the alignment gradient’s influence.Source: B. Inference-Time… · p. 3

Stochastic Sampling

  • Stochastic Sampling (SS) uses annealed MCMC to sequentially sample the policy and objective distributions, targeting trajectories that are both valid and minimize the user-specified objective.The method starts from a noisy trajectory and adds multiple reverse-sampling steps at a fixed noise level to better approximate the product distribution.Source: B. Inference-Time… · pp. 3–4

Qualitative comparisons

  • Figure 5 compares ACT and DP trajectories against a sketch: time is color-coded, thickness indicates ranked sketch similarity, and collision samples are white; SS preserves collision-free constraints while aligning with intent.Figure 6 is identified as a comparison of ACT/DP robustness in Maze2D.Source: Figure 5 · p. 4, Figure 6 · p. 4

III. EXPERIMENTS

The experiments assess steering for continuous motion alignment in Maze2D and discrete task alignment in Block Stacking and Real World Kitchen Rearrangement, while also measuring constraint satisfaction.

III. EXPERIMENTS

  • The evaluation covers continuous Motion Alignment in Maze2D, discrete Task Alignment in Block Stacking and Real World Kitchen Rearrangement, and Constraint Satisfaction among generated samples.Source: §III · p. 4
  • The experiments assess whether steering improves alignment and how it affects Constraint Satisfaction.Source: §III · p. 4
  • The evaluated alignment measures distinguish continuous motion alignment from discrete task alignment.Source: §III · p. 4

A. Maze2D - Continuous Motion Alignment (MA)

Maze2D tests continuous sketch alignment while checking whether frozen policies remain on the collision-free motion manifold. Steering exposes an alignment–constraint trade-off, with Stochastic Sampling achieving the best reported Motion Alignment and Constraint Satisfaction trade-off.

A. Maze2D - Continuous Motion Alignment (MA)

  • The Maze2D test uses frozen ACT and diffusion policies trained on four million navigation steps, then steers them with sketches that may violate collision constraints.The policies are trained on collision-free navigation demonstrations without goal-oriented objectives.Source: A. Maze2D - Continuous Motion… · p. 4
  • Stochastic Sampling achieves the best Motion Alignment and Constraint Satisfaction trade-off among the reported Maze2D steering methods.Table I reports mean collision rate and sketch L2 distance to the closest sample and to all samples per batch; its caption describes SS as achieving the best alignment with minimal collisions.Source: A. Maze2D - Continuous Motion… · p. 4, Table I · p. 4
  • Aggressive steering improves Motion Alignment but reduces Constraint Satisfaction and increases collisions.Diffusion-specific methods can transform constraint-violating sketches into the nearest collision-free samples on the data manifold.Source: A. Maze2D - Continuous Motion… · p. 4
  • Post-Hoc Ranking improves alignment for multimodal DP when the intended plan appears in its initial batch, but cannot discover an absent plan; unimodal ACT does not improve alignment with ranking.If a policy lacks robustness, Output Perturbation can introduce significant distribution shift.Source: A. Maze2D - Continuous Motion… · p. 4

B. Block Stacking - Discrete Task Alignment (TA)

The block-stacking study tests whether users can steer a multimodal policy toward a preferred four-block stacking sequence. GD recovers intended plans missed by unconditional sampling, while steering strength and timing expose a trade-off between sketch alignment and adherence to the training distribution.

B. Block Stacking - Discrete Task Alignment (TA)

  • Unconditional sampling can miss intended plans and PR cannot recover them, whereas GD can; later steering steps trade sketch similarity against training-distribution adherence.The study also reports that higher task alignment correlates with lower constraint satisfaction.Source: Figure 7 · p. 5, B. Block Stacking - Discrete Task… · p. 5
  • The domain uses four randomly positioned blocks and a diffusion policy trained on five million steps of CuRobo-generated pick-and-place trajectories.The policy learns a multimodal motion manifold without goal-oriented behavior.Source: B. Block Stacking - Discrete Task… · pp. 4–5
  • Users sketch a desired stacking sequence in VR; a trial aligns when the closest conditional sample targets the intended block, and satisfies constraints when execution succeeds.The study reports task alignment and constraint satisfaction for PR and GD with βi≤N = 25.Source: B. Block Stacking - Discrete Task… · p. 5
  • For GD, turning steering off after early diffusion steps retrieves a straight-line training trajectory with the correct discrete alignment instead of the sketch-shaped curved path.The modified strategy uses I = 50, aligning low-frequency components early and reverting to unconditional sampling afterward.Source: B. Block Stacking - Discrete Task… · p. 5

C. Real World Kitchen - Discrete Task Alignment (TA)

In the real-world kitchen, users steer a multimodal diffusion policy toward preferred sequences of bowl-placement skills. Stochastic sampling improves aligned success, while stronger alignment comes with lower execution success overall.

C. Real World Kitchen - Discrete Task Alignment (TA)

  • The kitchen policy is trained on 60 demonstrations for each of two tasks: placing a bowl in the microwave or in the sink.The combined dataset is used to train a diffusion policy for 40,000 steps, learning distinct multimodal skills.Source: C. Real World Kitchen - Discrete Task… · p. 5
  • Constraint satisfaction includes whether a multiskill sequence is valid, since combining task datasets permits infeasible orders such as placing a bowl in a closed microwave.The kitchen sequence task requires at least six steps to reach the terminal state.Source: C. Real World Kitchen - Discrete Task… · p. 5
  • Pixel inputs steer users toward a preferred skill, while physical end-effector corrections can trigger switches but often lead to execution failures.GD and SS use pixel inputs; OP serves as an aggressive physical-correction baseline, and higher GD guide ratios disrupt diffusion without improving alignment.Source: C. Real World Kitchen - Discrete Task… · p. 5
  • SS improves Aligned Success by 21% without fine-tuning, although its failure rate is higher than random rollout.Across the evaluated steering methods, task alignment increases as constraint satisfaction decreases.Source: C. Real World Kitchen - Discrete Task… · p. 5

IV. RELATED WORKS

Prior work studies human-robot interfaces, demonstration learning, and inference-time behavior composition, but real-time human input is often absent from policy rollouts. This paper builds on multimodal generative policies to examine physical interactions that steer multiple behavior modes at inference time.

Learning for Human-Robot Interaction

  • Human-robot interfaces and inputs such as language, sketches, and goals have been studied, while real-time human input remains often absent during policy rollouts.The paper frames this as a gap in directly conveying user intent during deployment.Source: §IV · p. 5
  • The framework repurposes pre-trained generative policies for real-time interaction and focuses on physical inputs that provide grounding information complementary to language prompts.This approach is inspired by prior work on adapting pre-trained policies to human-robot interaction.Source: §IV · p. 6

Learning from Human Demonstrations

  • Diffusion models capture multimodal demonstrations and can compose with external probability distributions, while prior latent-plan work focuses on a single high-quality behavior mode.This paper instead focuses on multiple behavior modes for adapting to inference-time inputs.Source: §IV · p. 6

Inference-Time Behavior Synthesis

  • Inference-time composition methods use guidance, learned costs, value-based reranking, or gradient-based policy composition for structured behavior generation.The paper investigates how different real-time physical interactions steer policies at inference time.Source: §IV · p. 6

V. CONCLUSION

The conclusion presents ITPS as a way to steer frozen policies through real-time human interaction without explicit policy training. Its sampling cost remains a limitation, motivating future distillation and user-study validation.

V. CONCLUSION

  • ITPS integrates real-time human interactions to control policy behavior during inference without explicit policy training.The framework is demonstrated and benchmarked in simulation and real-world experiments.Source: §V · p. 6
  • The approach relies on an expensive sampling procedure to produce behaviors aligned with human intent.Source: §V · p. 6
  • Future work aims to distill steering into an interaction-conditioned policy for faster responses and to conduct a user study to further validate steerability.Source: §V · p. 6
Loading 2411.16627v2…