Source-linked AI summary
Temporal Logic Guidance for Action-Only Diffusion Policies with World Models
Moritz Zoellner, Anastasios Manganaris, Rohan Paleja
TL;DR
Diffusion policies can represent multimodal robot behavior but offer limited inference-time control over which behavior mode is selected, while prior STL guidance requires jointly generated actions and states. The paper uses a separate differentiable world model to evaluate STL robustness and inject its gradient into action-only diffusion denoising without retraining. On Robomimic’s Can Transport task, the method improves constraint adherence while preserving task success.
Problem
Diffusion policies offer limited inference-time control over behavior modes, and prior STL-guidance methods require joint action-state generation with increased complexity and training cost.
Method
A separately trained differentiable world model predicts future states from action sequences, enabling STL-robustness gradients to guide action-only diffusion policies during denoising.
Results
On Robomimic’s Can Transport task, the method improves constraint satisfaction over the base diffusion policy and sample-and-rank baseline while maintaining task success.
Takeaways & Limitations
The approach enables runtime STL-based steering of existing action-only diffusion policies without retraining.
Abstract
from arXiv · showhide
Diffusion policies enable multimodal robot behavior but offer limited ability to choose among behavior modes at inference time, even though such control is desirable in human-robot settings. Prior solutions to this lack of control have utilized Signal Temporal Logic (STL) to express human intentions and provide corresponding guidance for diffusion policy inference. However, these approaches can only guide diffusion policies that jointly generate future actions and states, increasing both complexity and runtime. We propose a novel guidance method for action-only diffusion policies that uses a separate learned world model to enable differentiable evaluation of STL robustness, with its gradient then injected into the diffusion process. This steers behavior toward constraint satisfaction without retraining, improving constraint adherence while preserving task performance. On the Can Transport task from Robomimic, our method maintains 100% task success while reducing constraint violations from over 80% for baseline methods to 4%. We also discuss extensions toward improved robustness and more complex constraints.
I. INTRODUCTION
The paper addresses inference-time selection among diverse robot behaviors for differing user preferences without retraining. It uses STL guidance with a separate world model to steer action-only diffusion policies.
- I. INTRODUCTION: Inference-time guidance is needed because diffusion policies can represent multiple valid behaviors but cannot reliably select a user-preferred mode without retraining.This matters for robots operating across users and situations with differing preferences.
- I. INTRODUCTION: Prior STL-guidance methods evaluate constraints using predicted states during diffusion, requiring joint action-state generation with greater complexity and training cost.
- I. INTRODUCTION: STL provides a structured way to express complex tasks and safety constraints, and prior work has demonstrated mappings from language to STL specifications –.
- I. INTRODUCTION: The proposed method uses a separately trained differentiable world model to predict future states from proposed actions, enabling STL guidance for action-only diffusion policies.The STL-evaluation horizon can differ from the policy’s action-prediction horizon.
II. METHOD
The method injects STL-robustness gradients into diffusion denoising to guide action sequences without retraining. A differentiable world model supplies the predicted state trajectory needed for robustness evaluation.
- II. METHOD: The method starts from noisy action sequences and adds the gradient of a guidance objective to the diffusion denoising update.The policy generates an action sequence conditioned on the current state, while the guidance objective is based on STL robustness.
- II. METHOD: The guidance strength λ controls the update, and additional gradient-ascent steps on the final action sequence improve constraint satisfaction.
- II. METHOD: The guidance objective uses the STL robustness measure, which quantifies how strongly a predicted trajectory satisfies the specification.
- II. METHOD: A separately trained world model iteratively predicts future states induced by the proposed actions, making STL robustness differentiable with respect to the action sequence.This gradient is injected into denoising to steer the policy toward constraint-compliant behavior at inference time.
III. RESULTS
On Robomimic’s Can Transport task, the method steers undesirable multimodal rollouts toward upright-can trajectories while preserving task success. It outperforms the base diffusion policy and sample-and-rank baseline on constraint satisfaction.
- III. RESULTS: The Can Transport evaluation constrains the robot to keep the can upright using G(Rzz > cos(5°)), while the learned policy exhibits multiple behaviors from the same initial state.Some policy trajectories violate the uprightness constraint, and the world model accurately predicts rollouts over the action horizon.
- III. RESULTS: Figure 2 shows that guidance can steer an undesirable policy mode toward a trajectory satisfying the constraint.
- III. RESULTS: The method outperforms the base diffusion policy and sample-and-rank baseline on constraint satisfaction while maintaining task success over 50 rollouts.The comparison uses the same robustness objective for all methods.
IV. FUTURE WORK
The authors identify limited generality and long-horizon STL constraints as key directions for extending the approach. They propose specification-level world-model prediction through STL automaton states to support more complex task specifications.
- Long-horizon STL constraints remain a major limitation of existing STL-guidance approaches and a central direction for extending this method.
- The authors plan broader evaluation across tasks, environments, and STL specifications to assess the approach’s generality.
- They propose improving guidance stability or efficiency with evolutionary or second-order optimization methods.
- A separate world model could predict STL automaton states from environment states, automaton states, and policy-selected action modes.
- This abstraction is intended to enable longer-term, specification-relevant prediction for diffusion policies with arbitrarily complex task specifications.