Source-linked AI summary
Learning Robot Policies from Sparse Success Signals via STL-Guided Stein Variational Policy Gradient
Hongrui Zheng, Cristian Ioan Vasile, Antonio Loquercio, Rahul Mangharam
TL;DR
Sparse-success robot tasks make useful action feedback difficult to obtain, especially when completion requires coordinated or temporally constrained behavior. STL-SVPG uses trajectory-level STL robustness to train a population of feedback policies. It achieves the highest mean success on five of six benchmarks, with simulation-trained temporal and contact behaviors also transferring to hardware.
Problem
Sparse success signals make it difficult to distinguish useful actions in tasks requiring coordinated movements or ordering, timing, and simultaneous constraints.
Source: §I · p. 1Method
Results
STL-SVPG achieves the highest mean task success on five of six benchmarks, and simulation-trained policies transfer temporal and contact behaviors to real-world hardware.
Source: A. Learning With Sparse Success Signals · p. 6, E. Sim-to-Real Transfer on Crazyflie · p. 7Takeaways & Limitations
The results suggest combining quantitative task-specification feedback with population-based policy optimization is promising when local progress does not ensure overall success.
Source: §VIII · p. 8The approach depends on informative quantitative specifications and sufficiently accurate differentiable dynamics.
Source: §VIII · p. 8
Abstract
from arXiv · showhide
Learning robot policies for tasks with sparse success signals is challenging when completion depends on coordinated actions, precise contact outcomes, or satisfying several conditions together. Intricate physical interactions with the world further complicate these requirements. Prior work using conventional reward shaping mechanisms provides dense feedback but local progress might not translate into eventual task completion. We present Signal Temporal Logic-guided Stein Variational Policy Gradient (STL-SVPG), a population-based method that uses smooth STL robustness as a trajectory-level training objective. Differentiating this objective through the dynamics assigns credit to policy actions according to their effect on the complete task specification, rather than local progress alone. We evaluate the approach on six quadcopter and manipulator tasks that involves event-triggered responses, strictly ordered behavior, responses within specified deadlines, and physical interaction with the world. STL-SVPG achieves the highest mean success rate among the compared methods on five of six benchmarks. Simulation-trained policies trained in simulation transfer temporal and contact task behavior to the real world.
I. INTRODUCTION
The introduction frames sparse-success robot control as a task-specification and credit-assignment problem, then motivates STL-guided population learning as an alternative to handcrafted rewards and exploration-dependent feedback.
I. INTRODUCTION
- Sparse success signals leave few clues about useful actions, while coordinated movements and ordering, timing, or simultaneous constraints narrow the successful trajectories.STL robustness can provide quantitative feedback from unsuccessful trajectories rather than waiting for rare successful exploration.Source: §I · p. 1
- Reward shaping can require iterative reward redesign, while model-based methods may miss long-horizon outcomes or accumulate prediction errors in longer rollouts.Prior STL-robustness policy methods also leave temporal credit assignment to exploration or rely on online prediction and backup mechanisms.Source: §I · p. 1
- STL-SVPG differentiates smooth, trajectory-level STL robustness through robot dynamics and combines policy-improvement directions with population diversity.Kernel-weighted updates share directions among similar policies, while repulsive forces encourage behavioral diversity.Source: §I · pp. 1–2
- STL-SVPG achieves the highest mean success among reward-shaped reinforcement-learning baselines on five of six benchmarks, and simulation-trained policies transfer zero-shot to hardware.The paper also reports validation of simulation-only trained policies on two real-world tasks.Source: §I · p. 2
II. RELATED WORK
- Reward-design work includes potential-based shaping, coding-agent reward improvement, and Hindsight Experience Replay, which learns from unsuccessful attempts by relabeling goals.The paper contrasts these approaches with STL robustness as a smooth surrogate of the task specification.Source: §II · p. 2
- Task-progress methods use reward machines, or LTL progression, while quantitative-specification work includes robustness-reward learning,, and differentiable STL approaches,,,.STL-SVPG is presented as avoiding explicit task decomposition and using population-based optimization.Source: §II · p. 2
III. BACKGROUND
The background introduces STL as a way to express temporal requirements and score trajectories, then describes differentiable robustness and SVGD's population updates.
A. Signal Temporal Logic (STL)
- STL expresses trajectory requirements with predicates and logical and temporal operators; for example, it can require reaching a goal within five seconds while remaining safe throughout.The syntax includes conjunction, negation, and until, with eventually and always defined using these primitives.Source: A. Signal Temporal Logic (STL) · pp. 2–3
- STL robustness assigns a trajectory-level score whose sign indicates satisfaction or violation, combining predicate margins through logical and temporal operators.Conjunction and always use the smallest relevant margin, whereas disjunction and eventually use the largest.Source: A. Signal Temporal Logic (STL) · p. 3
- Smooth max/min approximations and computation-graph evaluation make STL robustness differentiable with respect to trajectory states and, with differentiable dynamics, policy actions.This supplies gradients from robustness to actions.Source: A. Signal Temporal Logic (STL) · p. 3
B. Stein Variational Gradient Descent
- SVGD transports a set of parameter particles toward a target distribution using updates that share improvement information while repelling particles to prevent mode collapse.The update combines attraction toward higher expected scores with a repulsion force.Source: B. Stein Variational Gradient Descent · p. 3
IV. PROBLEM FORMULATION
The section casts policy learning as variational inference over policy parameters, with expected STL robustness defining the task objective. Its variational objective balances robustness with entropy in policy-parameter space.
IV. PROBLEM FORMULATION
- The policy objective is expected STL robustness over feedback-policy trajectories induced by randomized initial conditions and task inputs.The task-conditioned Gibbs target replaces expected return with expected STL robustness.Source: §IV · p. 3
- The Gibbs target assigns policy-parameter density according to expected robustness, with temperature α controlling its sharpness.Z is a normalizing factor that makes the target a valid distribution.Source: §IV · p. 3
- Minimizing variational free energy balances expected robustness and entropy in policy-parameter space, rather than action space.The entropy term is defined over policy parameters.Source: §IV · p. 3
V. STL-GUIDED STEIN VARIATIONAL POLICY GRADIENT
STL-SVPG optimizes a population of policy parameters toward a robustness-weighted target by differentiating smooth trajectory robustness through simulated dynamics. Local gradients are combined with Stein updates to share improving directions while maintaining particle diversity.
V. STL-GUIDED STEIN VARIATIONAL POLICY GRADIENT
- STL-SVPG represents the variational approximation with a population of policy-parameter particles and updates them using a Stein direction.The update uses the expected robustness gradient scaled by 1/α; its kernel-weighted score shares task-improving directions and its kernel derivative repels particles in parameter space.Source: §V · pp. 3–4
- LogSumExp smooth maximum and minimum operators provide a differentiable surrogate for exact trajectory robustness during optimization.β controls approximation sharpness: larger β more closely approximates exact extrema, while smaller β distributes gradient weight more broadly; evaluation uses exact robustness.Source: §V · pp. 3–4
- Differentiating smooth trajectory robustness through simulated dynamics yields action gradients that can guide improvement before a rollout satisfies the specification.The local policy-gradient estimate multiplies action gradients by policy action Jacobians, sums over time, and averages across rollouts.Source: §V · p. 4
- The stored action gradients produce a local approximation, not the full closed-loop policy derivative, because observations and sampled policy noise are held constant.Storing them lets contributions accumulate through randomized transition minibatches without repeatedly differentiating the simulator.Source: §V · p. 4
- Policy behavioral differences are measured by comparing actions on a shared random observation batch, rather than by parameter distance.The kernel uses action similarities on those shared observations.Source: §V · p. 4
VI. EXPERIMENTAL SETUP
The experiments test STL-SVPG on sparse-success tasks across quadcopter and manipulator embodiments, combining continuous control with temporal requirements and physical interaction.
VI. EXPERIMENTAL SETUP
- The benchmark tasks test ordered actions, event-triggered responses, and contact-based outcomes under sparse success signals, mainly on Crazyflie and with generalization to a Franka Emika Panda manipulator.The scenarios assess whether the algorithm can learn successful physical behavior while satisfying the logical and temporal specifications.Source: §VI · p. 4
A. Robot Simulations
The simulation setup covers quadcopter and manipulator models, while the training procedure differentiates smooth robustness through simulated trajectories and updates a policy population.
A. Robot Simulations
- Crazyflie simulation uses rigid-body dynamics, a cascaded body-rate PID controller, first-order motor dynamics, and rotor-dependent aerodynamic drag.Normalized thrust and body-rate commands are issued at 100 Hz while control and physics run at 500 Hz.Source: A. Robot Simulations · p. 4
- The simulation procedure collects rollouts, differentiates smooth trajectory robustness through the simulator, and stores action gradients with transitions for policy-gradient estimation.It computes action Jacobians, accumulates local policy gradients over randomized transition minibatches, and uses shared observations to calculate the kernel and Stein direction.Source: A. Robot Simulations · p. 4
- The algorithm updates all policy particles iteratively and selects a policy using separate validation cases.The update applies the computed Stein direction to each particle.Source: A. Robot Simulations · p. 4
- The Panda simulation uses a seven-joint model with bounded joint-velocity control, integrating positions every 5 ms and holding each action for a 25 ms control interval.Source: A. Robot Simulations · p. 4
B. Tasks
The task suite specifies timed and ordered behavior alongside contact-rich physical interactions. Each task is expressed with safety constraints and episode-bounded temporal requirements.
B. Tasks
- Unlock requires a 0.6-second dwell in A, entry into Q within 2 seconds afterward, and a return home within 10 seconds; Q is forbidden before activation.Safety requires staying within world bounds.Source: B. Tasks · p. 5
- For Random Cue, the robot must avoid the randomly selected region first, then remain inside it for 0.5 seconds by the 5-second deadline.The cue is visible for 1 second; safety constraints are also specified.Source: B. Tasks · p. 5
- In Crazyflie Ping-pong, a valid strike must redirect the incoming ball into a target volume within 6 seconds; Panda Ping-pong scores target entry alone.The Crazyflie specification enforces post-contact events.Source: B. Tasks · p. 5
- Pillar Pull requires removing any two of five supporting pillars within 10 seconds while preserving the plate and block; both pillars must be out simultaneously.The task does not specify which pair of pillars to remove.Source: B. Tasks · p. 6
- In Branching, the Crazyflie may follow A→C or B→D, reaching the first region by 2 seconds and its corresponding goal within another 3 seconds.Each route avoids the alternative first region throughout the episode; the disjunction tests population diversity across valid routes.Source: B. Tasks · p. 6
C. Evaluation Protocol
Evaluation measures complete task success across independent training seeds and held-out initial conditions.
C. Evaluation Protocol
- A rollout succeeds only when the full task specification has positive exact robustness; each main benchmark uses six training seeds and 1,000 shared held-out initial conditions.The report gives mean success rate and sample standard deviation across seeds; policy evaluation uses deterministic actions, while TD-MPC2 uses its evaluation-mode planner.Source: C. Evaluation Protocol · p. 6
D. Baselines
The study compares STL-SVPG with three reinforcement-learning baselines, using task-specific dense rewards for the baseline methods.
D. Baselines
- PPO and SAC receive interaction budgets matched to the full STL-SVPG population, while TD-MPC2 matches GPU training time and uses fewer simulation steps.Baseline rewards combine progress shaping, completion bonuses, and control and safety penalties; STL-SVPG derives its objective from the task specification alone.Source: D. Baselines · p. 6
VII. RESULTS
STL-SVPG has the highest mean success on five of six benchmarks, while objective and population ablations test the contributions of direct STL optimization and multiple policy particles. The results section also frames questions about sparse-signal learning, multimodal solutions, and transfer to real robots.
VII. RESULTS
- The results section asks whether STL-SVPG can learn from extremely sparse success signals and how direct robustness optimization, policy populations, multimodal solutions, and simulation-to-real transfer contribute.These are the stated questions guiding the results analysis.Source: §VII · p. 6
A. Learning With Sparse Success Signals
- STL-SVPG achieves the highest mean task success on five of six benchmarks across the Crazyflie drone and Panda arm.The only tasks with comparable baseline performance are the two Random Cue tasks; STL-SVPG reaches 87.67% on Unlock, versus zero for PPO and SAC and 0.02% for TD-MPC2.Source: A. Learning With Sparse Success Signals · p. 6, Table I · p. 7
B. Benefits of Direct STL Optimization
- Unlock success rises from 45.17% with the dense-reward objective to 87.67% with direct STL optimization; the STL objective also enables Crazyflie Ping-Pong completion where dense return achieves zero.The comparison holds the SVPG procedure constant while changing the policy-gradient objective.Source: B. Benefits of Direct STL Optimization · p. 6, Table II · p. 6
C. Benefits of Population-Based Learning
- Increasing the population from one to 64 particles raises Unlock success from zero to 87.67% and Panda Ping-Pong success from 7.78% to 35.78%.The full method improves performance across both embodiments and discovers successes on tasks where the single-particle variant fails.Source: C. Benefits of Population-Based Learning · p. 7, Table III · p. 7
D. Diverse Successful Policy Populations
STL-SVPG retained both Branching routes across six runs and transferred temporal-task behavior to hardware, while Ping-pong transfer produced contact and occasional target delivery but not reliable full-task completion. The conclusion identifies informative specifications and sufficiently accurate differentiable dynamics as dependencies, with contact-model uncertainty and credit assignment remaining challenges.
D. Diverse Successful Policy Populations
- Across six runs, STL-SVPG retained both Branching routes, and the final chosen policy achieved 99.65% mean success on 128 held-out initial conditions.The population averaged 34 A–C and 30 B–D policies out of 64.Source: D. Diverse Successful Policy Populations · p. 7
E. Sim-to-Real Transfer on Crazyflie
- Fig. 3 shows 64 policies on one Branching reset: 30 A–C successes, 29 B–D successes, and five failures.Hardware deployment policies used new scene and drone-cage geometries and brushless-motor dynamics; Ping-pong additionally required top-face target entry.Source: Figure 3 · p. 7, E. Sim-to-Real Transfer on Crazyflie · p. 7
- Random Cue succeeded in 37 of 60 hardware episodes (61.67%), while Ping-pong achieved a 1.67% real-platform success rate after 0.78% in simulation.Ping-pong contact and valid-strike rates were 33.33% and 6.67% on real hardware, versus 25% and 3.91% in simulation; two episodes sent the ball into the target, one ending in a crash.Source: E. Sim-to-Real Transfer on Crazyflie · pp. 7–8
VIII. LIMITATIONS AND CONCLUSION
- The approach depends on informative quantitative specifications and sufficiently accurate differentiable dynamics; closing the hardware gap calls for better contact-uncertainty treatment, domain randomization, and credit assignment across events.Random Cue transferred temporal-task success, whereas Ping-pong transferred contact and occasional valid strikes but not complete target delivery.Source: §VIII · p. 8