Source-linked AI summary
STeP: Signal Temporal Logic for Precise Specifications for Action Generation with Vision Language Models
Kasra Torshizi, Anukriti Singh, Sidharth Mathur, Khuzema Habib, Leo Du, Pratap Tokekar
TL;DR
VLA models can struggle to represent and enforce precise spatial, temporal, and logical requirements in language instructions. STEP uses STL as a shared interface for hierarchical task decomposition, execution-mode selection, validation, monitoring, and replanning. On real-world tabletop tasks, the framework improves constraint satisfaction and supports switching and targeted recovery, while retaining limitations in learned-policy control, plan expressiveness, clutter handling, and specification accuracy.
Problem
VLA models do not inherently expose mechanisms for representing, monitoring, or enforcing the precise spatial, temporal, and logical constraints that define successful task completion.
Method
STEP has a VLM decompose instructions into grounded subtasks and STL specifications, then uses learned policies or STL-guided MPC with runtime validation, monitoring, and replanning.
Results
Across real-world manipulator tasks, STEP improves constraint satisfaction, supports switching between execution modes, and enables more targeted recovery after intermediate failures.
Takeaways & Limitations
An explicit STL specification layer can connect high-level VLM reasoning with low-level control instead of relying on one learned policy to satisfy all requirements implicitly.
Takeaways & Limitations
Learned policies are monitored by STL but do not yet use specifications as inference-time control signals, while sequential plans limit branching and cyclic behaviors and the VLM can produce incorrect specifications.
Abstract
from arXiv · showhide
Vision-language-action (VLA) models have shown impressive generalization, but often lack interpretability and can struggle to follow precise natural language instructions that encode spatial, temporal, and logical requirements. We propose a hierarchical framework that uses Signal Temporal Logic (STL) as a shared representation connecting high-level language understanding with low-level robot execution. A high-level policy leverages a VLM to decompose language instructions into high-level subtasks, generate STL specifications for each subtask, and choose a low-level policy for executing each subtask. The STL specifications translate language-derived intent into precise constraints, and the low-level policy selection determines whether those constraints are enforced directly through STL-guided model-predictive control or monitored during execution of a learned policy for perceptually complex, or contact-rich behaviors. By integrating STL into plan validation, low-level policy, subtask monitoring, and replanning, our framework enables language-derived plans to be checked, optimized, and revised at runtime using a common formal structure. We evaluate the approach on a real-world tabletop domain, demonstrating how formal specifications can improve the precision, reliability, and interpretability of language-conditioned robot planning.
1 Introduction
STEP uses STL to formalize precise spatial, temporal, and logical requirements that VLA policies may not represent or enforce reliably. Its hierarchical architecture decomposes instructions, selects execution modes, and uses STL for validation, monitoring, and replanning.
- VLA policies can satisfy an instruction’s broad intent while violating the spatial, temporal, ordering, or safety conditions required for correct completion.
- STEP translates language-derived requirements into STL specifications that connect high-level VLM reasoning with low-level execution, monitoring, and replanning.STL provides quantitative robustness for evaluating progress, detecting violations, and optimizing trajectories.
- The framework combines learned policies for expressive or contact-rich behaviors with STL-guided MPC when precise constraint satisfaction is required.MPC directly optimizes explicit constraints, whereas learned policies are better suited to perceptually complex or contact-rich settings.
- STEP integrates STL across plan validation, robustness-shaped control, execution monitoring, and replanning after failures.
- The framework evaluates precise language following, subtask-level switching between learned and model-based execution, and few-shot replanning grounded in the original instruction.
2 Related Works
Prior work uses language models for embodied task planning and temporal logic for robot planning and control, establishing the two research directions that STEP combines.
- Language-model planning methods ground plans in robot affordances, generate executable programs or policy code, construct optimization-based spatial representations, and support long-horizon decomposition and replanning [13] [14] [16].
- Temporal logic has long been applied to robot motion planning and control, including LTL-based mission planning and reactive synthesis.
3 Background
STL specifies spatial and temporal properties over task-relevant signals using predicates and temporal operators. Its quantitative robustness semantics distinguish satisfaction from violation and can be smoothed for optimization.
- 3.1 Signal Temporal Logic (STL): STL formulas combine predicates with Boolean operators and temporal operators for eventually, always, and until over specified intervals.The implication operator is included explicitly because conditional structure occurs naturally in language instructions.
- 3.2 Signals and Predicates: A task signal is computed from state and observation, while the language model selects predicates from a predefined library rather than generating arbitrary signal functions.Predicates used in gradient-based optimization must be differentiable.
- 3.3 Robustness: STL robustness provides a signed measure of satisfaction: positive values indicate satisfaction, negative values indicate violation, and magnitude measures the margin from the boundary.
- 3.3 Robustness: STEP uses STL within a two-system architecture in which System 2 formalizes instructions and System 1 executes, monitors, and replans subtasks.
- 3.3 Robustness: The framework uses softmax to smoothly approximate MAX/MIN operations so quantitative robustness can support gradient-based optimization.β controls the approximation’s sharpness.
4 Method
STEP separates language-to-STL planning from low-level execution while using STL as the shared interface for validation, control, monitoring, and replanning. System 2 compiles grounded subtasks into executable specifications, and System 1 selects MPC or learned policies according to task demands.
- 4.1 System 2: Language-to-STL Task Planning: System 2 uses the instruction, observation, skill and predicate libraries, and recall context to produce grounded subtasks with skills, parameters, constraints, and descriptions.
- 4.1 System 2: Language-to-STL Task Planning: Static checks verify schema validity, skill availability, parameter types, scene references, and constraint support before execution.Failed specifications are returned to the VLM with an error message.
- 4.1 System 2: Language-to-STL Task Planning: A deterministic compiler maps grounded skill templates and constraints into STL formulas, converts time intervals into controller timesteps, and produces executable subtask specifications.An optional LLM consistency check flags obvious contradictions or mismatches between language and compiled structure.
- 4.2 System 1: Low-Level Execution: System 1 selects STL-guided MPC for explicit geometric, temporal, or safety constraints and learned policies for behaviors requiring expressive perceptual control or difficult-to-model contact dynamics.
- 4.2 System 1: Low-Level Execution: STL monitoring measures progress and satisfaction, triggers recall when robustness falls below a threshold, and advances subtasks when formulas are satisfied.The history log supplies recent observations, states, descriptions, executors, robustness values, and completion or failure events for replanning.
- 4.2.2 Model-Predictive Control (MPC): MPC predicts state evolution with a dynamics model, optimizes over a finite horizon, executes only the first action, and replans at the next timestep.Future signals are computed from predicted states and the current observation because the dynamics model does not predict future observations.
5 Evaluation
STEP is evaluated on real-world tabletop manipulation through precise constraint following, runtime switching between learned policies and MPC, and few-shot recovery from failures. Across these studies, STL improves safe success, enables executor switching, and provides structured feedback for replanning.
- Evaluation Setup: The evaluation uses long-horizon tabletop manipulation requiring monitoring, disturbance recovery, and switching between learned and model-based executors.The real-world platform uses a UR3e robotic arm and compares STEP against an ablated VLM-MPC-COST baseline without STL components.
- RQ1: Precise Language Following: Higher safe success rates were achieved across all nine task categories compared with VLM-MPC-COST, with the largest gains on temporal constraints.Tasks covered logical, spatial, logical + spatial, and temporal requirements on a UR3e platform.
- RQ2: Learned Policy/Motion Planning Switching: Across 30 trials, STL-guided switching detected collisions in 29 cases, rerouted via MPC in 25, and completed the grasp in 22.The learned policy handled the contact-rich grasp, while MPC handled obstacle avoidance after robustness dropped below a threshold.
- RQ2: Learned Policy/Motion Planning Switching: Without switching, the behavioral-cloning policy collided in 28 of 30 trials, whereas MPC alone failed at the grasp in 24 of 30 trials.The comparison supports assigning different subtask phases to the executor suited to each behavior.
- RQ3: Few-Shot Replanning: With STL violation feedback, 12 of 16 initial stacking failures were corrected on the second attempt and 14 within three attempts.The feedback included the violated constraint, robustness margin, violation timestep, and an updated scene snapshot; VLM-MPC-COST corrected only 7 of 18 initial failures within the reported window.
6 Conclusion
STEP uses STL as a shared representation between language-based task reasoning and low-level robot execution. Real-world results indicate that this interface improves constraint satisfaction, supports executor switching, and enables targeted recovery after failures.
- 6 Conclusion: STEP converts spatial, temporal, and logical requirements into STL specifications that support validation, MPC optimization, execution monitoring, and replanning.The framework combines STL-guided MPC for constraint-sensitive subtasks with learned policies for contact-rich behaviors.
- 6 Conclusion: Across real-world manipulator tasks, the formal interface improves constraint satisfaction, supports switching between execution modes, and enables more targeted recovery after intermediate failures.STL robustness detects failures and provides structured feedback for replanning.
7 Limitations
STEP has limitations in learned-policy control, plan expressiveness, implementation complexity, and VLM specification accuracy.
- 7 Limitations: Only MPC-based skills directly optimize actions against STL specifications, while learned policies are monitored but lack STL-based inference-time control.The high-level plan is also limited to a sequence of subtasks rather than a full automaton, and simple system components struggle in cluttered environments.
- 7 Limitations: The VLM can produce incomplete or incorrect task specifications, especially when fine spatial details are missing from the scene description.This limits specification reliability even when the execution framework is otherwise available.
8 Appendix
This appendix situates STEP across language planning, temporal-logic control, and STL generation while detailing its two-module execution pipeline and perception-language interfaces.
- STEP Architecture: System 2 converts language, scene context, skill definitions, and recall context into an STL-annotated task specification.
- STEP Architecture: System 1 executes each subtask using STL-guided MPC or a learned policy, with monitoring signals for completion, violations, and recall.
- Related Works: Prior work separately studies language-model task planning, temporal-logic motion planning, and natural-language-to-STL translation, whereas STEP maintains STL across the full control pipeline.The pipeline includes decomposition, policy selection, low-level control, monitoring, and replanning.
- STEP Architecture: STEP uses System 2 for task interpretation, STL compilation, validation, and recall-based revision, while System 1 executes subtasks and monitors robustness.The two systems connect formal task specifications to low-level execution and replanning.
- Real-World Implementation: The perception module builds a typed world representation from RGB-D segmentation, VLM labels, calibrated depth, entity geometry, and refreshed world-frame poses.