PaperlayerView exact revision on arXiv

Source-linked AI summary

Tractable Reinforcement Learning for Full Class of Signal Temporal Logic Specifications Using Spatiotemporal Tube Reward

Vaishnavi Jagabathula, P Sangeerth, Pushpak Jagtap

arXiv:2609.28396v1cs.ROcs.SYeess.SY

TL;DR

Robotic controllers must satisfy complex spatial and temporal requirements even when dynamics are unknown and actuator limits are strict, while STL learning can require trajectory history. This paper maps STL tasks to time-varying spatiotemporal tubes and trains a time-aware SAC policy with geometry-aware rewards. The framework is validated on three robotic platforms, and the paper identifies formal guarantees against safety violations as future work.

  • Problem

    STL reward design is challenging because temporal satisfaction usually depends on trajectory history, while analytical tube controllers can impose restrictive system assumptions and struggle with actuator limits.

    Source: §I · p. 1

  • Method

    The method maps full-class STL specifications to time-varying spatiotemporal tubes and trains a time-aware SAC agent using a continuous distance-based reward and bounded actor outputs.

    Source: §I · p. 1, C. Construction of Reward Functions… · p. 4, D. Time-Aware Soft Actor-Critic · p. 4

  • Results

    The framework was validated on differential-drive, cartpole, and spacecraft case studies, with reported STL satisfaction and enforcement of the spacecraft input constraint ∥u∥≤700 Nm.

    Source: §I · p. 1, A. Differential drive robot · p. 5, B. Cart-Pole · p. 5

  • Takeaways & Limitations

    The paper presents a history-free approach for full-class STL control that handles input-constrained, underactuated, and non-holonomic systems across its reported robotic case studies.

    Source: §I · p. 1, §V · p. 6

    Formal guarantees to preclude safety violations remain future work.

    Source: §V · p. 6

Abstract

from arXiv · show

This paper addresses the control problem for robotic systems, including non-holonomic and underactuated platforms operating under unknown dynamics and strict actuator limits to satisfy complex high-level specifications. We denote these high-level specifications using Signal Temporal Logic (STL) and propose a novel time-aware Reinforcement Learning (RL) framework that leverages the geometric properties of Spatiotemporal Tubes (STTs). While traditional analytical STT controllers often struggle to enforce input constraints, and existing RL approaches rely on memory-intensive state history, our method natively overcomes both limitations. By mapping the logical and temporal complexities of the full class of STL into time-varying geometric boundaries, we directly constrain the multidimensional system state without relying on scalar robustness metrics. Augmenting the state space with time, we train a time-aware Soft Actor-Critic (SAC) agent using a continuous, geometry-aware reward function that eliminates the need to explicitly evaluate complex logical semantics during execution. The proposed framework offers a history-free, computationally efficient approach to learn continuous control policies that ensure robust satisfaction of specifications while strictly adhering to system input constraints.

I. INTRODUCTION

The paper targets robotic tasks whose spatial and temporal requirements call for expressive specifications, but learning controllers for STL tasks is difficult when satisfaction depends on trajectory history. It proposes an RL-based controller that uses spatiotemporal tubes to handle the full class of STL specifications across systems including underactuated and non-holonomic systems with input constraints.

I. INTRODUCTION

  • STL expresses spatial and temporal task requirements and supports quantitative evaluation of task performance through robustness metrics.The paper motivates it with missions such as sequential goal-reaching while avoiding obstacles.Source: §I · p. 1
  • RL can synthesize controllers without restrictive assumptions on the system model, but STL reward design is difficult because satisfaction usually depends on past trajectories.History-based methods can incur curse-of-dimensionality costs, while other approaches enforce only fragments of STL.Source: §I · p. 1
  • Analytical STT controllers often require control-affine or fully actuated systems and can struggle with disturbances and strict actuator limits.These assumptions exclude underactuated and non-holonomic systems.Source: §I · p. 1
  • The proposed RL controller maps full-class STL specifications to time-varying tubes and uses a time-aware SAC agent with bounded outputs to handle input constraints without storing trajectory history.The paper validates the framework on differential-drive, cartpole, and spacecraft case studies.Source: §I · p. 1

A. System Dynamics

The paper models the plant as an unknown discrete-time system with measurable state and defines STL formulas over signal predicates and temporal operators. It also introduces robustness as a quantitative measure of satisfaction.

A. System Dynamics

  • The system is discrete-time, has an unknown transition map, and is assumed to have a fully measurable state vector.The state and input belong to their respective vector spaces and the input is selected from a physical input set.Source: A. System Dynamics · p. 2

B. Signal Temporal Logic (STL)

  • STL formulas are built recursively from predicates, Boolean operators, and temporal operators over signals.A predicate is determined by whether its function value is nonnegative.Source: B. Signal Temporal Logic (STL) · p. 2
  • The bounded-until operator requires the second formula to hold at some time in the specified interval, while conjunction requires both formulas to hold.The interval is measured relative to the time at which satisfaction is evaluated.Source: B. Signal Temporal Logic (STL) · p. 2
  • STL robustness quantifies the degree of satisfaction, with positive robustness indicating that the signal satisfies the formula.Its magnitude reflects the strength of satisfaction or violation.Source: B. Signal Temporal Logic (STL) · p. 2

C. Markov Decision Process (MDP) Formulation

The paper formulates learning for the unknown system as an MDP and aligns its states and actions with the physical system. The agent learns a continuous-control policy from interactions with the system.

C. Markov Decision Process (MDP) Formulation

  • The MDP formulation represents sequential decision-making with state space, action space, transition distribution, reward, and discount factor.The paper uses this formulation to synthesize a controller for an unknown discrete-time system.Source: C. Markov Decision Process (MDP)… · p. 2
  • MDP states map directly to system states, and the action space equals the physical input constraint set.The state space is restricted to a compact domain.Source: C. Markov Decision Process (MDP)… · p. 2
  • Because the transition map is unknown, the agent learns a policy by interacting with the system and collecting state-action-next-state samples.The policy selects continuous control inputs to maximize cumulative expected reward.Source: C. Markov Decision Process (MDP)… · p. 2

D. Objective

The objective is a history-free RL policy for satisfying full-class STL specifications on an unknown system. The proposed approach uses time-varying spatiotemporal tubes, whose confinement condition implies STL satisfaction.

D. Objective

  • The objective is to synthesize a history-free RL policy that satisfies the full class of STL specifications for an unknown discrete-time system.The paper approaches this objective using spatiotemporal tubes.Source: D. Objective · p. 2

III. PROPOSED METHODOLOGY

  • The method uses spatiotemporal tubes to construct time-varying rewards, with hyperrectangular and spherical forms among the available geometric parameterizations.A spherical tube is specified by a center trajectory and a strictly positive radius.Source: §III · p. 3, A. Spatiotemporal Tubes · p. 3

A. Spatiotemporal Tubes

  • A valid spatiotemporal tube is a time-varying compact set with nonempty interior whose confined trajectories have positive STL robustness.The tube is defined over the task interval.Source: A. Spatiotemporal Tubes · p. 3
  • Confining a trajectory within a valid spatiotemporal tube guarantees satisfaction of the associated STL specification.The lemma states this implication directly.Source: A. Spatiotemporal Tubes · p. 3

B. Soft Actor-Critic (SAC)

The paper adopts Soft Actor-Critic (SAC) for robotic systems’ continuous action spaces and uses its maximum-entropy objective to balance return and policy entropy.

B. Soft Actor-Critic (SAC)

  • SAC is selected to handle continuous robotic action spaces without discretization, which the paper associates with poor scalability and unstable learning.The paper contrasts SAC with value-based methods such as DQN, which require discrete actions.Source: B. Soft Actor-Critic (SAC) · p. 3
  • SAC maximizes expected discounted return together with policy entropy, unlike conventional RL’s reward-only objective.The temperature parameter α controls the trade-off between reward maximization and exploration.Source: B. Soft Actor-Critic (SAC) · p. 3
  • SAC uses neural networks for the policy actor and soft action-value critic, updated through a soft Bellman equation and entropy-regularized return.The paper refers readers to for further SAC details.Source: B. Soft Actor-Critic (SAC) · p. 3

C. Construction of Reward Functions using STTs

The reward functions encode STL constraints through STT geometry, rewarding states within the tube and penalizing violations rather than relying on scalar robustness values.

C. Construction of Reward Functions using STTs

  • Hyperrectangular STTs use independent lower and upper bounds for each state dimension, with user-defined scaling gains in the reward.The supplied passage describes state and input at time t, but does not provide the complete reward expression.Source: C. Construction of Reward Functions… · p. 3
  • Spherical STTs are characterized by a time-varying center and radius, with the reward based on Euclidean distance.The displayed spherical reward subtracts the distance from the center from the radius, scaled by k.Source: C. Construction of Reward Functions… · p. 4
  • The reward provides positive reinforcement inside STT boundaries, penalizes violations, and reaches its maximum when the state tracks the tube center.The continuous distance-based penalty is intended to support stable learning and avoid gradient masking.Source: C. Construction of Reward Functions… · pp. 3–4
  • Instead of constraining scalar robustness, the method directly constrains the multidimensional state within the tube geometry, which abstracts STL’s temporal and logical complexity.The passage says this avoids computing complex robustness metrics or storing state history.Source: C. Construction of Reward Functions… · p. 4

D. Time-Aware Soft Actor-Critic

The time-aware SAC formulation makes the reward and policy depend on the current time while retaining Markov transitions; the algorithm supplies time alongside state to its networks.

D. Time-Aware Soft Actor-Critic

  • The modified MDP uses a time-varying reward and policy, while transitions remain Markov because the next state depends on the current state and action.The soft action-value function is also explicitly time-dependent.Source: D. Time-Aware Soft Actor-Critic · p. 4
  • The actor and critic receive an augmented state formed by concatenating the time step with the system state.The actor optimizes the time-dependent soft Q-value.Source: D. Time-Aware Soft Actor-Critic · p. 4
  • During each episode, the algorithm samples actions, observes transitions under unknown dynamics, computes the time-dependent reward, and stores transitions for network updates.The listed updates use replay-buffer mini-batches for the critic and actor, with target-network updates.Source: D. Time-Aware Soft Actor-Critic · p. 4

IV. RESULTS AND DISCUSSION

The framework is validated in three case studies: Turtlebot, Cartpole, and spacecraft.

IV. RESULTS AND DISCUSSION

  • The framework is validated on Turtlebot, Cartpole, and spacecraft case studies.Training and simulations used a computer equipped with an AMD Ryzen 9 5950x processor and 128 GB RAM.Source: §IV · p. 4

A. Differential drive robot

The Turtlebot case study applies the time-aware SAC framework to a sequential STL task with timed goals, obstacle avoidance, and bounded inputs. The policy demonstrates satisfaction of the task under non-holonomic dynamics and input constraints.

A. Differential drive robot

  • The task keeps the robot in workspace S, visits G1 during [15, 20] seconds, reaches G2 during [33, 35] seconds, and avoids O1 and O2.The velocity and angular velocity inputs are bounded by A = [−15, 15] × [−15, 15].Source: A. Differential drive robot · p. 4
  • The trained policy tries to keep trajectories inside a circular STT while reaching G1 and then G2 within their specified time windows.Degree-6 polynomial tubes took approximately 1 minute to generate, and RL training took approximately 1 hour.Source: Figure 1 · p. 5, A. Differential drive robot · p. 5
  • The Turtlebot policy satisfies the sequential STL task under non-holonomic dynamics and strict input constraints, without history storage.The discussion contrasts this with prior methods that struggle to meet these requirements simultaneously.Source: A. Differential drive robot · p. 5

B. Cart-Pole

The cart-pole case study applies time-aware SAC to a swing-up, balancing, and sequential goal-reaching task with bounded force. The policy can recover from a slight STT excursion, although formal hard safety guarantees remain future work.

B. Cart-Pole

  • The cart-pole must swing up from near θ ≈ π, balance within 15° after 5 s, visit G1 during [12] s, then G2 during [18, 20] s, and remain within [−6, 6].The applied force is bounded by F ∈ [−20, 20] N.Source: B. Cart-Pole · p. 5
  • Degree-6 polynomial tubes took approximately 25 seconds to generate, and RL training took approximately 15 minutes.The mission horizon is 20 seconds.Source: B. Cart-Pole · p. 5
  • After a slight cart-position excursion beyond the STT during [12] s, the policy recovers toward the tube center while satisfying the overall specification.Unlike analytical approaches that can encounter singularities or solver infeasibility after boundary violations, its reward gradient remains continuous; formal hard safety guarantees remain future work.Source: B. Cart-Pole · p. 5

C. Spacecraft model

The spacecraft case study tests sequential rendezvous with timed targets and obstacle avoidance. Its trained policy follows the STT boundaries while respecting an actuator limit that the analytical comparison method does not respect; the conclusion identifies formal safety guarantees and dynamic multi-agent settings as future work.

C. Spacecraft model

  • Starting in S, the spacecraft must reach T1 or T2 during s, then G during [14] s, while avoiding O throughout the 15-second horizon.Degree-5 polynomial tubes took 0.91 s to generate, while RL training took approximately 15 minutes.Source: C. Spacecraft model · pp. 5–6
  • The RL policy follows the spacecraft STT boundaries and enforces ∥u∥≤700 Nm, whereas the analytical method in fails to respect the actuator limit.Its trajectories closely resemble the baseline's geometric response.Source: C. Spacecraft model · p. 6, Figure 3 · p. 6, Figure 4 · p. 6

V. CONCLUSION AND FUTURE WORK

  • The paper concludes that its history-free, time-aware STT-based RL framework was validated across diverse robotic platforms for full-class STL specifications and input-constrained, underactuated systems.Future work includes formal guarantees against safety violations and extension to multi-agent settings in dynamic environments.Source: §V · p. 6

Related papers

Loading 2609.28396v1…