PaperlayerView exact revision on arXiv

Source-linked AI summary

DynaGuide: Steering Diffusion Polices with Active Dynamic Guidance

Maximilian Du, Shuran Song

arXiv:2506.13922v2cs.RO

TL;DR

Complex robot policies need steering for specific situations, but common goal-conditioning approaches require training with test-time objectives in mind. DynaGuide guides pretrained diffusion policies with an external dynamics model during action denoising. Across simulated and real experiments it reports up to 70% guidance success, and its current guidance conditions are limited to observations of desired or undesired outcomes.

  • Problem

    Steering complex robot policies for specific situations ideally should not require retraining, excessive sampling, or anticipating all possible test-time needs.

    Source: §1 · p. 1

  • Method

    DynaGuide separates a dynamics model from the base policy and uses model-derived guidance during diffusion action denoising.

    Source: §1 · p. 2

  • Results

    DynaGuide guides the base policy successfully up to 70% of the time across simulated and real experiments, and outperforms goal conditioning by 5.4x on lower-quality objectives.

    Source: §1 · p. 2

  • Takeaways & Limitations

    DynaGuide demonstrated steering pretrained policies toward particular objectives, multiple behaviors, and away from undesirable behaviors in simulated and real experiments.

    Source: §5 · p. 9

    Current guidance conditions are observations of desired or undesired outcomes, making it difficult to specify how an objective should be achieved; language and kinesthetic demonstrations remain future modalities.

    Source: §5.1 · pp. 9–10

Abstract

from arXiv · show

Deploying large, complex policies in the real world requires the ability to steer them to fit the needs of a situation. Most common steering approaches, like goal-conditioning, require training the robot policy with a distribution of test-time objectives in mind. To overcome this limitation, we present DynaGuide, a steering method for diffusion policies using guidance from an external dynamics model during the diffusion denoising process. DynaGuide separates the dynamics model from the base policy, which gives it multiple advantages, including the ability to steer towards multiple objectives, enhance underrepresented base policy behaviors, and maintain robustness on low-quality objectives. The separate guidance signal also allows DynaGuide to work with off-the-shelf pretrained diffusion policies. We demonstrate the performance and features of DynaGuide against other steering approaches in a series of simulated and real experiments, showing an average steering success of 70% on a set of articulated CALVIN tasks and outperforming goal-conditioning by 5.4x when steered with low-quality objectives. We also successfully steer an off-the-shelf real robot policy to express preference for particular objects and even create novel behavior. Videos and more can be found on the project website: https://dynaguide.github.io

1 Introduction

DynaGuide addresses the need to steer complex robot policies for specific situations without retraining or anticipating every test-time objective. It separates an external dynamics-based steering signal from a pretrained diffusion policy and applies that signal during action denoising.

1 Introduction

  • Complex deployed policies need steering that fits a situation without retraining, excessive sampling, or anticipating all possible test-time needs.Goal-distribution conditioning requires foresight about future steering needs [46].Source: §1 · pp. 1–2
  • DynaGuide combines an external dynamics-model guidance signal with a pretrained policy’s action prior, modifying the diffusion denoising process rather than selecting sampled actions.The model predicts far-future visual outcomes, which are compared with desired or undesired outcomes to form a differentiable guidance metric.Source: §1 · p. 2
  • DynaGuide accepts multiple positive and negative objectives, unlike goal-conditioned policies that typically accept a single goal condition.The guidance conditions can be changed during inference.Source: §1 · p. 2
  • The separated guidance and DinoV2 embedder are described as avoiding out-of-distribution components for new or lower-quality objectives, while the guidance metric remains meaningful and can support novel behavior.The authors attribute the meaningful signal in part to the metric’s averaging effect.Source: §1 · p. 2
  • Separating guidance from the base policy enables inference-time steering of off-the-shelf policies and direct denoising guidance toward behaviors underrepresented in the base policy.The approach uses one denoising sequence, unlike sample-based methods that require many policy inferences per step.Source: §1 · p. 2
  • Across simulated and real experiments, DynaGuide guides the base policy successfully up to 70% of the time and reaches up to 80% guidance success on an existing real-robot policy.It outperforms goal conditioning by 5.4x on lower-quality objectives and creates a novel behavior on the real robot policy.Source: §1 · p. 2

2 Related Works

Related work includes goal-conditioned policies, dynamics-based planning and policy filtering, and inference-time steering of diffusion policies. DynaGuide instead uses a dynamics model to guide a trained policy directly through action diffusion.

2 Related Works

  • Goal-conditioned policies require a policy trained to accept goal inputs, while DynaGuide’s external dynamics model works on top of diffusion policies and supports desirable and undesirable outcomes.Observation-based goal conditioning typically supports only one desirable outcome.Source: §2 · p. 3
  • Latent dynamics models support reinforcement learning, planning, and policy-sample filtering; DynaGuide instead steers the trained base policy directly through action diffusion.Related approaches filter policy samples using predicted value, VLM feedback, or engineered reward [35, 53, 40].Source: §2 · p. 3
  • The method comparison identifies untrained goals as a limitation for goal conditioning, rare behaviors as challenging for sampling, and precise-coordinate requirements as a limitation of position guidance.DynaGuide is described as guiding complicated objectives without precise coordinate input.Source: Table 1 · p. 3
  • Inference-time steering includes noise-skewing, sampling, classifier-free guidance, and classifier guidance; DynaGuide uses a trained dynamics model as feedback for classifier guidance.Classifier-free guidance still requires training on supplied conditions.Source: §2 · p. 3

3 DynaGuide Method

DynaGuide uses a learned visual dynamics model to compare predicted outcomes with guidance conditions, then applies the resulting metric’s gradient during diffusion-policy action denoising. Its objective supports desired and undesired outcomes, and stochastic sampling stabilizes guidance during denoising.

3 DynaGuide Method

  • During action denoising, DynaGuide combines the pretrained policy’s denoising signal with a dynamics-derived guidance gradient to steer the generated action sequence.The dynamics model predicts future outcomes and compares them with desired or undesired conditions.Source: §3 · p. 4, Figure 2 · p. 4

3.1 A Dynamics Model Capable of Guidance

  • A transformer predicts a future latent observation from the current observation latent and action sequence, using frozen DinoV2 patch embeddings and a regression objective.The dynamics model is trained on diverse robot interaction data, with Gaussian noise added to actions to match inference-time noise.Source: §3.1 · pp. 4–5
  • The guidance metric uses log-sum-exp as a soft maximum over outcome likelihoods, a design the authors identify as useful for diverse objectives and lower-quality guidance conditions.The metric compares predicted outcomes with guidance conditions.Source: §3.1 · pp. 4–5

3.2 Guiding the Action Diffusion Process

  • DynaGuide modifies the diffusion denoising signal with the metric gradient; repeating each denoising step through stochastic sampling stabilizes guidance and permits higher guidance strength without sacrificing trajectory validity.Higher guidance strength alone can produce less smooth or incoherent trajectories as the gradient pulls noise predictions out of distribution.Source: §3.2 · p. 5

4 Experiments

Experiments compare DynaGuide with sampling, goal-conditioning, position guidance, and a base policy across simulated and real-robot settings. They test fully specified and underspecified objectives, multiple objectives, underrepresented behaviors, and steering an off-the-shelf policy.

4.1 Experiments 1 & 2: Steering in Complex 3D Environments

  • 70% average target behavior success on ArticulatedParts, where DynaGuide boosts base-policy behavior 8.7x and outperforms position guidance across tasks.In MovableObjects, steering performance drops; sampling falls to base-policy performance, while randomized cube positions create outcome mismatches that affect goal-conditioning more than DynaGuide.Source: §4.1 · pp. 6–7, Figure 4 · p. 7

4.2 Experiment 3: Robustness to Lower Quality Guidance Conditions

  • Under lower-quality guidance, goal-conditioned policy average success fell below 10%, while DynaGuide and DynaGuide-Sampling outperformed it by 5.4x on average; DynaGuide was more consistent than sampling.The combined latent-space guidance metric can cancel some noise among individually noisy, inaccurate latent distances and yield meaningful guidance.Source: §4.2 · p. 7

4.3 Experiment 4: Steering with Multi-Objectives

  • 80% average success with nearly no full behavior failures for multiple objectives, and nearly perfect avoidance of undesired behaviors while executing alternatives.GPC also supports multiple objectives, but has lower desired-behavior success and more undesired behaviors and failures, especially when avoiding outcomes.Source: §4.3 · p. 8, Figure 5 · p. 8

4.4 Experiment 5: Enhancing Underrepresented Behaviors

  • 40% Switch-On success with only 1% of the original behavior in the base-policy training data, outperforming sampling in lower-data regimes.The experiment reduces Switch-On representation in the training set; both the base policy and GPC offer more of that behavior as its representation increases.Source: §4.4 · p. 8, Figure 5 · p. 8

4.5 Experiment 6: Steering Off-The-Shelf Policies with a Real Robot

  • 72.5% average target-behavior success for cup-color preference and 80% red-cup retrieval in HiddenCup; DynaGuide doubled interactions with a novel mouse object.These experiments steer a publicly available pretrained policy using a separately trained dynamics model.Source: §4.5 · p. 9, Figure 6 · p. 9

5 Conclusion and Discussion

DynaGuide steers pretrained diffusion policies with a separately trained latent dynamics model, enhancing target behaviors across simulated and real experiments. Its current guidance conditions represent desired or undesired outcomes, leaving richer modalities for future work.

5 Conclusion and Discussion

  • DynaGuide uses a separately trained latent dynamics model to steer pretrained diffusion policies, enhancing target behavior across simulated and real experiments.The authors also report multiple behaviors, avoidance of undesirable behaviors, and enhancement of behaviors underrepresented in base-policy training.Source: §5 · p. 9

5.1 Limitations and Future Work

  • The method currently makes it difficult to specify how an objective should be achieved.The authors attribute this limitation to the current form of guidance conditions.Source: §5.1 · p. 9
  • Current guidance conditions represent desired or undesired outcomes as observations; language and kinesthetic demonstrations are identified as future modalities.The authors hope multimodal conditions will enable finer-grained guidance across the trajectory.Source: §5.1 · p. 10

NeurIPS Paper Checklist

The checklist records the paper's claims, experimental reporting, and reproducibility disclosures, alongside responses on ethics and societal impacts.

NeurIPS Paper Checklist

  • The checklist points readers to the experimental sections for the paper's main claims and says limitations are discussed in Section 5 and the results sections.It also instructs that claims should match the evidence and reflect the expected scope of generalization.Source: NeurIPS Paper Checklist · pp. 14–15
  • The authors report that experiment setups and reproducibility information are provided in the main paper and appendices, with simulated data, real-world policy and data, and code made available or slated for release.The checklist identifies some real-world data as additionally collected in the experimental environment.Source: NeurIPS Paper Checklist · p. 16
  • The checklist reports that training and test details are described in the paper and appendix, simulation plots include confidence intervals, and real-robot results lack intervals because they are resource-intensive.The authors refer to Appendix B for detailed settings and interval information.Source: NeurIPS Paper Checklist · p. 17
  • The authors report discussing positive and negative societal impacts and state that the dynamics model has no easy path to misuse or dual use.The checklist also records that no crowdsourcing or human participants were used.Source: NeurIPS Paper Checklist · pp. 18–20

A.1 Steering in a Simple 2D Block Task

In BlockTouch, DynaGuide steers a navigation policy toward a selected square color and can guide it through tasks beyond the base policy's training behavior.

A.1 Steering in a Simple 2D Block Task

  • In the synthetic BlockTouch setup, the base policy and dynamics model learn navigation to squares at varied locations and colors, then guidance steers toward one color.Figure 7 illustrates the agent, targets, and the guided color preference.Source: §A.1 · p. 22, Figure 7 · p. 22
  • During early denoising, the policy and guidance signals can compete over action direction; later, they cooperate to produce a sequence toward the guided square.The example contrasts the policy's pull toward a nearby yellow square with guidance toward a blue square.Source: §A.1 · p. 22
  • The dynamics model guides the base policy through novel tasks, including precise navigation to a tight cluster and passing close squares to reach a farther target.Both the base policy and dynamics model were trained on random navigation behavior; the base policy tended to choose nearer squares in the far-target setup.Source: §A.1 · p. 22, Figure 7 · p. 22

A.2 Additional Goal-Conditioning Baselines

Additional tests find that stronger goal-conditioning baselines improve selected steering cases but remain below DynaGuide on the reported comparison; a real-robot comparison favors classifier guidance over sampling.

A.2 Additional Goal-Conditioning Baselines

  • In the CupPreference comparison, DynaGuide-Sampling improves over baseline, while DynaGuide achieves the best guidance.The table caption identifies this as an additional result for the GuideRed condition.Source: Table 2 · p. 22
  • On UnderspecifiedObjectives, goal conditioning strengthened with additional data or hindsight replay still performs worse than DynaGuide.The additions improve steering for some targets or tasks, but not uniformly.Source: Table 3 · p. 23, §A.2 · p. 23

A.3 Additional Real Robot Baseline

  • On one real-robot task, diffusion-based action sampling improves success over baseline, but DynaGuide's classifier guidance achieves higher success.The comparison is reported for the target task in Table 2.Source: §A.3 · p. 23

A.4 Ablations and Hyperparameter Investigation

Ablations identify noisy-action pretraining as essential and show that reducing dynamics-model training data lowers steering success, while reasonable hyperparameter changes are generally tolerated.

A.4 Ablations and Hyperparameter Investigation

  • The guidance strength s and σ are robust to reasonable changes, but very low s weakens guidance and excessive s creates instability.Very low σ also lowers success because guidance conditions are too far away to provide a meaningful signal.Source: §A.4 · p. 23
  • Increasing stochastic sampling steps M generally improves guidance performance at added computational cost, and comparable guidance is achieved with one condition.The experiments use M = 4 as a balance of stability and computational efficiency.Source: §A.4 · p. 23
  • Noised-action pretraining is essential for performance; removing it greatly decreases the dynamics model’s ability to steer action diffusion.The authors attribute this to the noise otherwise being out of distribution.Source: §A.4 · p. 23
  • Reducing dynamics-model training data reduces steering success, though performance remains above the baseline.This ablation uses a model trained only on Calvin’s D-split for ArticulatedParts.Source: §A.4 · pp. 23–24, Table 5 · p. 24

A.5 Classifier-Free Guidance Baseline

Classifier-Free Guidance can steer the policy toward multiple objectives and away from behaviors, but its positive-guidance success is lower than DynaGuide’s and avoidance can bring more failures.

A.5 Classifier-Free Guidance Baseline

  • CFG successfully guides multiple objectives, including performing multiple behaviors and avoiding others, but its total positive-guidance success is lower than DynaGuide’s.The comparison is between CFG results in Table 6 and DynaGuide results in Figure 5.Source: §A.5 · p. 24, Table 6 · p. 24
  • For undesired behaviors, CFG and DynaGuide have success rates within one standard deviation, but CFG fails to execute a behavior more than 10% of the time in all three tested conditions.DynaGuide nearly always offers a valid alternative behavior.Source: §A.5 · p. 24, Table 6 · p. 24
  • The authors attribute CFG’s failures to equal weighting of guidance signals, which can let contradictory signals produce poorly formed actions.DynaGuide’s log-sum-exp latent selection instead favors relevant guidance and limits contradictory influences.Source: §A.5 · p. 24

B.1 DynaGuide Implementation Details

DynaGuide combines a pretrained diffusion policy with a separately trained latent dynamics model whose action gradients steer denoising; the baselines use goal conditioning, target-position guidance, sampling, and classifier-free guidance.

B.1 DynaGuide Implementation Details

  • The base diffusion policy predicts 16-action chunks from two-step visual and proprioceptive histories using a ResNet-18 encoder and a U-Net noise predictor.It has around 18 million parameters; 14 actions are executed open-loop before the policy is queried again.Source: §B.1 · pp. 24–25
  • The dynamics model uses DinoV2 patch embeddings and a Transformer to predict future or final visual latents from the current observation and actions.For CALVIN, it predicts the trajectory’s final latent; for the real-world experiment, it predicts the latent after the proposed action chunk.Source: §B.1 · p. 25
  • Half of dynamics-model training actions are noised with the inference DDIM scheduler, enabling the model to handle noisy actions during guidance.The noising step is sampled geometrically with an expected step of 20; Figure 8 lists the hyperparameter settings used in sweeps and ablations.Source: §B.1 · p. 25, Figure 8 · p. 25
  • At inference, gradients of the objective are backpropagated through the dynamics model to the actions, and DDIM sampling reduces denoising from 100 steps to 10.Stochastic sampling repeats denoising steps within the denoising loop.Source: §B.1 · p. 25

B.2 Implementation of Baselines

  • The baselines use goal-conditioned state input, a manually estimated target position, five policy samples ranked by the guidance metric, or classifier-free guidance conditions.The classifier-free baseline randomly zeros the goal condition during training and uses the same conditions as the goal-conditioned baseline at inference.Source: §B.2 · p. 26

B.3 Distance Metric: Derivation and Design Choices

The distance metric combines probabilistic evidence for desirable and undesirable outcomes using log-sum-exp, with design choices shaped by achievable targets and the observed latent space.

B.3 Distance Metric: Derivation and Design Choices

  • The derivation treats latent states as Gaussian with diagonal variance, an approximation also implicit in squared latent-distance methods.The authors motivate log-sum-exp probabilistically rather than assuming a Euclidean metric.Source: §B.3 · p. 26
  • Log-sum-exp adds probabilities across desired outcomes, acting as a soft maximum that emphasizes closer outcomes; σ controls its sharpness.Adding log-probabilities instead would multiply probabilities, making any single unachievable or mutually exclusive target overly influential.Source: §B.3 · pp. 26–27
  • The final metric divides desirable-outcome probability by undesirable-outcome probability, creating a stronger gradient incentive to reduce the latter than subtraction would.The derivation absorbs a constant into σ and discards another constant in the final expression.Source: §B.3 · p. 27
  • Empirically, Euclidean distance works best in Eq. 2 instead of squared L2 distance, which the authors attribute to DinoV2 latents not being fully Gaussian.This differs from the squared-distance form implied by the Gaussian derivation.Source: §B.3 · p. 27
  • Guidance can target a future state or a final state: CALVIN uses final-state prediction, while the real-world setting uses a horizon of H = 48 steps.Future-state prediction is easier to train but requires an achievable guidance target; final-state prediction simplifies target selection but can be difficult for long horizons.Source: §B.3 · p. 27

B.4 Experimental Setup: Simulation

The simulation setup uses CALVIN data to train a base diffusion policy and a separate dynamics model, then evaluates several steering conditions with task-specific success criteria.

B.4 Experimental Setup: Simulation

  • The base policy uses CALVIN-D demonstrations with third-person and wrist-camera observations plus full proprioception, while the dynamics model uses CALVIN-ABCD data and predicts final task observations.Adding non-relevant environments to dynamics-model training improved convergence and reduced overfitting.Source: §B.4 · p. 28
  • Evaluation samples 20 trajectories per desired behavior for guidance conditions and runs 50 trials per evaluation with a 400-step horizon.Trials stop when an object is sufficiently articulated or moved; otherwise, a 400-step rollout counts as failure.Source: §B.4 · p. 28
  • Simulation figures report 1-sigma uncertainty, with standard errors based on six policies trained on separate CALVIN train-validation splits.Each policy’s success rate averages 100 trials; the line-plot shaded regions use the same standard-error calculation.Source: §B.4 · p. 28
  • Simulation success requires fully pressing or flipping buttons and switches, moving drawers and doors past halfway, or touching and slightly displacing a colored cube.A cube may be displaced by lifting or nudging it.Source: §B.4 · p. 28
  • For underspecified objectives, the target object is set to its desired pose while other object states and the robot start pose are sampled from validation trajectories.Unlike the other conditions, the target object is not shown directly interacting with the robot.Source: §B.4 · p. 28
  • The multiple-objective experiment seeks comparable frequencies of desired behaviors while avoiding undesired ones, and reports the full task distribution rather than a single success rate.Rollouts terminate after one behavior; the experiment does not test chaining behaviors.Source: §B.4 · p. 28
  • The underrepresented-behavior setup removes Switch-On demonstrations from base-policy training, retaining between 1% and 80% of the original Switch-On data.The tested retained-data levels are 1%, 2%, 5%, 10%, 20%, 40%, 60%, and 80%.Source: §B.4 · p. 29
  • BlockTouch trains the dynamics model to classify the destination square color and guides actions with cross-entropy against the desired color’s one-hot vector.Test layouts vary whether the agent must decide early, choose among nearby squares, or pass other squares to reach the blue square.Source: §B.4 · p. 29

B.5 Experimental Setup: Real Robot

The real-robot setup steers a pretrained policy with separately trained dynamics guidance; the section also discusses broader impacts, risks, and code and asset licenses.

B.5 Experimental Setup: Real Robot

  • The real-robot experiments leave a publicly available Universal Manipulation Interface diffusion policy unchanged and train dynamics guidance on cup-placement demonstrations, plus mouse-rearrangement data for novel behavior.The policy was trained on about 2,000 cup grasping, reorientation, and saucer-placement trajectories; the dynamics data includes 500 collected demonstrations and 3,648 public mouse trajectories for the novel-behavior experiment.Source: §B.5 · p. 29
  • Real-robot evaluations use an ARX5 arm and 20 trials per base or guided policy, without confidence intervals.Cup preference tests full cup-pickup and saucer-placement success, while hidden-cup and novel-object tests count a grasp as success.Source: §B.5 · pp. 29–30

C Broader Impact

  • Steering pretrained policies without retraining may reduce retraining energy needs, broaden access for labs, and help remove unwanted or problematic policy biases.The dynamics model uses publicly available or easy-to-collect data described as containing no private or sensitive information.Source: C Broader Impact · p. 30
  • Steering off-the-shelf policies carries a risk of malicious use; the authors suggest excluding malicious behaviors from base-policy training or training policies to resist such guidance.The paper presents these as possible mitigations.Source: C Broader Impact · p. 30

D Code, Assets, and Licenses

  • The paper reports using public data and codebases or assets under MIT or Apache licenses, and says its code will be released under MIT.The named assets include CALVIN, Robomimic, DinoV2, and the Universal Manipulation Interface.Source: D Code, Assets, and Licenses · p. 30
Loading 2506.13922v2…