PaperlayerView exact revision on arXiv

Source-linked AI summary

TGPO: Temporal Grounded Policy Optimization for Signal Temporal Logic Tasks

Yue Meng, Fei Chen, Chuchu Fan

arXiv:2510.00225v1cs.ROcs.AIcs.LGcs.LO

TL;DR

Standard RL struggles with STL's history dependence and sparse long-horizon satisfaction rewards. TGPO decomposes STL into timed subgoals and trains a time-conditioned policy with dense stage-wise rewards, while critic-guided temporal search selects promising schedules. Across five environments, TGPO* has the highest overall task success rate among baselines, with especially significant gains in high-dimensional and long-horizon tasks.

  • Problem

    STL's history-dependent specifications and sparse long-horizon satisfaction rewards make them difficult for standard RL algorithms.

    Source: §1 · p. 1

  • Method

    TGPO decomposes STL into timed subgoals and invariant constraints, using high-level temporal grounding and a time-conditioned policy trained with dense, stage-wise rewards.

    Source: §1 · p. 2, §4.1 · p. 4

  • Results

    TGPO* achieves the highest overall task success rate among baselines across five environments, with especially significant gains in high-dimensional systems and long-horizon tasks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    TGPO's decomposition, temporal sampling, state augmentation, and reward design are reported to handle general and complex STL tasks across robotic environments and formulas.

    Source: §6 · p. 10

    TGPO lacks formal guarantees of convergence to a global optimum, may not efficiently handle disjunctions or infinite-horizon Always-Eventually tasks, and has tested scalability only up to five time variables.

    Source: §5.6 · p. 9

Abstract

from arXiv · show

Learning control policies for complex, long-horizon tasks is a central challenge in robotics and autonomous systems. Signal Temporal Logic (STL) offers a powerful and expressive language for specifying such tasks, but its non-Markovian nature and inherent sparse reward make it difficult to be solved via standard Reinforcement Learning (RL) algorithms. Prior RL approaches focus only on limited STL fragments or use STL robustness scores as sparse terminal rewards. In this paper, we propose TGPO, Temporal Grounded Policy Optimization, to solve general STL tasks. TGPO decomposes STL into timed subgoals and invariant constraints and provides a hierarchical framework to tackle the problem. The high-level component of TGPO proposes concrete time allocations for these subgoals, and the low-level time-conditioned policy learns to achieve the sequenced subgoals using a dense, stage-wise reward signal. During inference, we sample various time allocations and select the most promising assignment for the policy network to rollout the solution trajectory. To foster efficient policy learning for complex STL with multiple subgoals, we leverage the learned critic to guide the high-level temporal search via Metropolis-Hastings sampling, focusing exploration on temporally feasible solutions. We conduct experiments on five environments, ranging from low-dimensional navigation to manipulation, drone, and quadrupedal locomotion. Under a wide range of STL tasks, TGPO significantly outperforms state-of-the-art baselines (especially for high-dimensional and long-horizon cases), with an average of 31.6% improvement in task success rate compared to the best baseline. The code will be available at https://github.com/mengyuest/TGPO

1 INTRODUCTION

STL tasks are difficult for standard RL because STL is history-dependent, satisfaction rewards are sparse over long horizons, and existing approaches cover limited formula fragments or face challenging model learning and planning. TGPO grounds temporal variables into sequenced subgoals, enabling dense stage-wise rewards; experiments report its strongest advantages in high-dimensional and long-horizon settings.

Motivation

  • STL's history dependence breaks the Markovian assumption of common RL algorithms, while trajectory-level satisfaction yields extremely sparse rewards on long-horizon tasks.The difficulty is especially consequential for systems with complex dynamics and long task horizons.Source: §1 · p. 1
  • Existing model-free approaches handle only limited STL fragments, while model-based approaches face difficult dynamics learning, accumulated long-horizon estimation error, and potentially intractable high-dimensional planning.The model-free methods described encode histories or formula progress with state augmentation and reward shaping.Source: §1 · p. 1

Approach

  • TGPO assigns concrete values to STL time variables, then trains a low-level time-conditioned policy on sequenced subgoals using dense, stage-wise rewards.A critic guides high-level temporal search with Metropolis–Hastings sampling; inference evaluates sampled assignments and executes the most promising schedule.Source: §1 · p. 2

Experiments

  • TGPO* achieves the highest overall task success rate among baselines across five environments, with especially significant gains in high-dimensional systems and long-horizon tasks.The time-conditioned design also provides critic-based interpretability of promising temporal plans and diverse behaviors for one STL specification.Source: §1 · p. 2

Contributions

  • The paper presents TGPO as a hierarchical model-free RL framework for general nested STL tasks over long horizons, with subgoal decomposition and critic-guided temporal grounding.Its contributions also include an interpretable learning process and reported strong performance on complex dynamics.Source: §1 · p. 2

2 RELATED WORK

The related work situates STL among specification and planning methods, then reviews temporal-logic RL and its reward-design challenges. Existing STL approaches have limitations in formula structures and system settings, motivating a method for more general tasks.

2.1 SIGNAL TEMPORAL LOGIC TASKS

  • STL specifies robotic tasks over continuous signals with time intervals, and prior approaches include sampling-based methods, mixed-integer programming, trajectory optimization, and learning-based methods.Learning-based examples include differentiable policy learning, imitation learning, and reinforcement learning.Source: §2.1 · p. 2

2.2 REINFORCEMENT LEARNING FOR TEMPORAL LOGIC TASKS

  • Temporal-logic RL addresses reward design; STL adds explicit time constraints and real-valued predicates, while existing STL methods often remain restricted in formula structure or system setting.The reviewed methods include state augmentation, robustness-based reward approximations, and model-based learning with evolutionary strategies or Monte-Carlo Tree Search.Source: §2.2 · pp. 2–3
  • TGPO is presented as handling more general STL formulas while efficiently designing augmented states and dense, stage-wise rewards.This contrasts with prior methods constrained by limited temporal nesting, fixed-size time windows, or grid-like environments.Source: §2.2 · p. 3

3 PRELIMINARIES

The preliminaries define STL over discrete-time signals, describe its temporal operators and robustness semantics, and formulate control as maximizing STL satisfaction probability. Because STL policies depend on history and satisfaction is checked over full trajectories, the problem requires state augmentation and dense reward design.

3.1 SIGNAL TEMPORAL LOGIC (STL)

  • STL formulas use predicates and logical or temporal operators, with eventually and always quantified over bounded intervals and robustness scores indicating satisfaction when nonnegative.The until operator requires one formula to hold until another becomes true within its interval.Source: §3.1 · p. 3

3.2 MARKOV DECISION PROCESS

  • An MDP describes states, actions, transition probabilities, rewards, and discounting, with a policy mapping states to action distributions.Its objective is an optimal policy maximizing expected discounted cumulative reward.Source: §3.2 · p. 3

3.3 PROBLEM FORMULATION

  • The control problem is to formulate an MDP and learn a policy that maximizes the probability of satisfying a given STL formula.The system is a discrete-time process whose controls generate a signal from an initial state.Source: §3.3 · p. 3, §3.1 · p. 3
  • An STL policy depends on trajectory history, so the control state alone is not Markov; full-trajectory satisfaction also makes dense reward design difficult.The formulation therefore calls for state augmentation to retain history and dense rewards under the augmented state space.Source: §3.3 · pp. 3–4

4 METHODOLOGY

TGPO decomposes STL into timed subgoals and invariant constraints, then combines temporal grounding with policy optimization to guide task execution.

4 METHODOLOGY

  • TGPO combines STL decomposition and critic-guided temporal grounding to produce subgoals and invariant constraints for dense-reward policy optimization.Its framework is hierarchical: temporal assignments guide the policy's subgoal sequence.Source: §4 · p. 4, Figure 1 · p. 4

4.1 STL SUBGOAL DECOMPOSITION

  • Nested STL formulas are flattened top-down into timed reachability subgoals and invariance constraints, with subgoal and avoidance tasks tracked separately.The decomposition carries ancestor time variables into subformulas and represents atomic propositions or their negations; disjunctions and G(F ...) structures are not considered here.Source: §4.1 · pp. 4–5, Figure 2 · p. 5

4.2 TEMPORAL GROUNDED STATE AUGMENTATION AND REWARD DESIGN

  • For each concrete time assignment, the augmented state tracks task execution so rewards can encourage timed subgoal completion while penalizing invariant violations.The reward combines distance shaping, progress and success bonuses, invariant-violation penalties, and a final-time robustness signal.Source: §4.2 · p. 5

4.3 CRITIC-GUIDED BAYESIAN TIME ALLOCATION

  • A hybrid temporal sampler combines critic-guided Metropolis–Hastings, uniform samples, and elite assignments to search for promising time allocations.The critic scores assignments through its value estimate, while the elite buffer retains assignments with the highest STL robustness scores.Source: §4.3 · p. 6

5 EXPERIMENTS

Experiments compare TGPO with several baselines across five simulation environments and varied STL tasks. TGPO* leads overall, retains performance at longer horizons, and its ablations and visualizations examine sampling, reward design, and policy behavior; the authors also state scope limitations.

5.1 IMPLEMENTATION DETAILS

  • The five benchmarks span a 2D linear system, 4D Unicycle, 7-DoF Franka Panda, 12D Quadrotor, and 29D Ant; each has 10 STL tasks.Five tasks per benchmark are two-layer STLs, and the rest are deeper nested tasks that F-MDP cannot solve.Source: §5.1 · pp. 6–7, Figure 3 · p. 7

5.2 MAIN RESULTS

  • TGPO* achieves the highest overall success rate; on Quadrotor and Ant it reaches 86.46% and 61.57%, while most baselines stay below 10%.The experiments span five environments, and the advantage is especially clear in higher-dimensional or more difficult settings.Source: Figure 4 · p. 7, §5.2 · pp. 7–8

5.3 SOLVING STL WITH DIFFERENT HORIZON-LENGTHS

  • Across horizons of 50, 200, 300, 800, and 1000, TGPO and TGPO* maintain high success rates, while τ-MDP, F-MDP, and RNN drop as horizons increase.CEM and Grad maintain performance as the horizon expands tenfold; the authors suggest their trajectory-optimization formulation may explain this.Source: Figure 6 · p. 8, §5.3 · p. 8

5.4 ABLATION STUDIES

  • In the Linear-environment sampling ablation, random sampling achieves 80.33%, individual Bayesian or elite-buffer sampling hurts performance, and combining sources makes Ours* best.The authors suggest early myopic exploration may explain the individual strategies’ weaker performance.Source: Table 1 · p. 8, §5.4 · p. 9
  • In the Linear-environment reward and state ablation, robustness-only reward yields 11.73%; adding reward terms improves performance to 88.99%, while removing flags drops success by 41.48%.Removing the time-index counter as well yields 11.43%.Source: Table 1 · p. 8, §5.4 · p. 9

5.5 VISUALIZATION FOR INTERPRETABILITY AND MULTI-MODAL BEHAVIOR

  • For an Ant task requiring visits to A1 and A2 while avoiding B, two high-value time-assignment regions correspond to two distinct behaviors that satisfy the STL.The critic assigns low values to regions associated with dynamically infeasible or too-rapid subgoal visits.Source: Figure 7 · p. 9, §5.5 · p. 9

5.6 LIMITATIONS

  • TGPO lacks formal guarantees of convergence to a global optimum; its tested scope includes five time variables, and scaling to more complex STLs remains open.The authors also flag possible inefficiency for disjunctions and infinite-horizon requirements such as G(F).Source: §5.6 · p. 9

6 CONCLUSION

TGPO is an RL framework for long-horizon STL tasks that uses STL decomposition, time-variable sampling, state augmentation, and reward design. Experiments show improved performance across robotic environments and STL formulas; future work targets broader STL coverage and scalability.

6 CONCLUSION

  • TGPO combines STL decomposition, time-variable sampling, state augmentation, and reward design to address long-horizon, complex STL tasks.The paper presents TGPO as a reinforcement-learning framework.Source: §6 · p. 10
  • Experiments show TGPO significantly outperforms existing baselines across robotic environments and STL formulas.Source: §6 · p. 10
  • Future work will extend TGPO to a broader class of STL formulas and improve its scalability.Source: §6 · p. 10

A.1 ALGORITHM HYPERPARAMETERS

The appendix identifies the main training hyperparameters and points to Table 2 for the assignments used to train TGPO*.

A.1 ALGORITHM HYPERPARAMETERS

  • The appendix identifies the main hyperparameters used during training.Source: §A.1 · p. 13
  • Table 2 presents the hyperparameter assignments used for training TGPO*.Source: Table 2 · p. 13
  • The table's stated scope is the training configuration for TGPO*, not a comparison of algorithms.Source: Table 2 · p. 13

A.2 SIMULATION ENVIRONMENT DETAILS

Experiments use five simulation environments; the first four have JAX-written system dynamics, while Ant uses a MuJoCo JAX implementation.

A.2 SIMULATION ENVIRONMENT DETAILS

  • The five simulation environments are Linear, Unicycle, Franka Panda, Quadrotor, and Ant.Source: §A.2 · p. 13
  • Linear, Unicycle, Franka Panda, and Quadrotor use implementations written in plain JAX with system dynamics specified directly.Source: §A.2 · p. 13
  • Ant uses the MuJoCo JAX implementation.Source: §A.2 · p. 13

A.2.1 LINEAR

The Linear environment uses single-integrator dynamics, with position coordinates as its state and two directional velocities as its control input.

A.2.1 LINEAR

  • The Linear environment uses a single-integrator dynamics model.Source: §A.2.1 · p. 13
  • Its two-dimensional state (x, y)^T represents coordinates on the xy-plane.Source: §A.2.1 · p. 13
  • Its two-dimensional control input (v, w)^T represents velocities in the two directions.Source: §A.2.1 · p. 13

A.2.2 UNICYCLE

The unicycle section specifies a car-like robot model with a four-dimensional state and two control inputs. It gives the discrete-time dynamics and the simulation step, actuation limits, and scene bounds.

A.2.2 UNICYCLE

  • The model state is (x, y, θ, v), representing planar position, heading, and velocity; its inputs (ω, a) are angular velocity and acceleration.Source: §A.2.2 · p. 13
  • The discrete-time dynamics update position from velocity and heading, heading from angular velocity, and velocity from acceleration.Source: §A.2.2 · p. 13
  • The simulation uses Δt = 0.2s, limits actuation to [−1rad/s, +1rad/s] × [−4m/s2, +4m/s2], and bounds the scene to [−5m, +5m] × [−5m, +5m].Source: §A.2.2 · p. 14

A.2.3 FRANKA PANDA

This appendix section details TGPO’s experimental methodology and benchmark setups, including training-time comparison, critic validation, temporal sampling, baseline implementations, and STL tasks with temporal goals and safety constraints across environments.

A.2.3 FRANKA PANDA; A.2.4 QUADROTOR

  • The Panda setup uses a 7-DoF single-integrator joint model at Δt = 0.05 s, while the quadrotor uses full dynamics with a 12D state, four motor inputs, and Δt = 0.10 s.The quadrotor mass is 0.2 kg, with gravity set to 9.81 m/s² and diagonal inertia (0.01, 0.01, 0.02) kg·m².Source: §A.2.3 · p. 14, §A.2.4 · p. 14

A.2.5 ANT

  • For Ant, the authors first train a goal-reaching policy, then formulate STL-task execution as waypoint planning for an 8-DoF quadruped with 29D observations.The quadruped’s complex dynamics are implemented in Brax, and its original controls are torques for eight joints.Source: §A.2.5 · p. 14

A.3.1 CEM; A.3.3 F -MDP; A.3.4 RNN; A.3.5 GRAD

  • The comparison reimplements τ-MDP and F-MDP with PPO, uses a history-encoding RNN with terminal robustness reward, and applies deterministic differentiable rollouts with BPTT for Grad.CEM samples 512 policy candidates per iteration, retains 32 elites, and uses 1,000, 2,000, or 4,000 iterations depending on environment; τ-MDP augments the trajectory into the state and uses terminal robustness reward.Source: §A.3.1 · p. 15, §A.3.3 · p. 15, §A.3.4 · p. 15

A.4 TEMPORAL SAMPLING ALGORITHM DETAILS

  • TGPO* samples temporal variables with discrete Metropolis-Hastings, using exp(Vψ(s0, t)) as an unnormalized proxy and accepting proposals according to critic-value differences.It runs 512 chains for 500 iterations with 200 warm-up steps, proposing a ±1 move on a uniformly selected time-variable dimension and choosing each chain’s sampled assignment with the highest critic value.Source: §A.4 · p. 16

A.5 TRAINING TIME COMPARISON

  • TGPO and TGPO* have runtime similar to τ-MDP, F-MDP, and Grad, while CEM is 20.8%∼35.8% higher than TGPO* and TGPO* is 1.96X∼6.11X faster than RNN.The comparison averages results over 10 STL tasks and 7 random seeds per method and environment; the text says TGPO can achieve higher task success rate while remaining as scalable as other top RL baselines in training time.Source: Figure 8 · p. 16, §A.5 · pp. 16–17

A.6 CORRELATION BETWEEN THE CRITIC AND THE STL ROBUSTNESS SCORE

  • In Linear, 4096 sampled critic-time assignments showed that higher critic values generally corresponded to higher STL robustness and success probability, with STL-09 as a noted exception.In most cases, the cumulative success probability increased or plateaued at 100% as the critic threshold rose; this supports using the critic to identify promising time assignments.Source: §A.6 · p. 17, Figure 9 · p. 17, Figure 10 · p. 17

A.7 STL TASK DETAILS; A.7.1 STLS IN “LINEAR” ENVIRONMENT; A.7.2 STLS IN “UNICYCLE” ENVIRONMENT; A.7.3 STLS IN “FRANKA PANDA” ENVIRONMENT; A.7.4 STLS IN “QUADROTOR” ENVIRONMENT

  • Each simulation environment uses 10 STL formulas across two-level and multi-level categories, restricted to reach, stay, and avoid predicates.The supplied Linear, Unicycle, Franka Panda, and Quadrotor task listings specify timed subgoals and invariants, with obstacle-avoidance constraints in the formulas.Source: §A.7 · p. 17, §A.7.1 · p. 18, §A.7.2 · p. 18

A.7.4 STLS IN “QUADROTOR” ENVIRONMENT

  • Quadrotor tasks 07–10 are multi-layer STL specifications combining timed goals, nested temporal conditions, or until constraints with obstacle and wall avoidance.The task formulas use horizons up to 100; task 10 includes three until-goal clauses and a later sustained goal.Source: §A.7.4 · pp. 24–25

A.7.5 STLS IN “ANT” ENVIRONMENT

  • Ant tasks 01–05 are two-layer specifications that add timed goals and, in later tasks, additional goals or invariants alongside obstacle-avoidance constraints.Their formulas include goals A, C, D, F, and H at differing time windows; tasks 04–05 also impose invariants over B and E, with task 05 adding G.Source: §A.7.5 · p. 25
  • Ant tasks 06–10 are multi-layer specifications, progressing from timed goals with obstacle avoidance to nested goals, until constraints, and a sustained terminal goal.Tasks 06–08 use one or more timed goals; task 09 nests four goals, and task 10 combines three until clauses with a sustained goal near the horizon end.Source: §A.7.5 · p. 26

Related papers

Loading 2510.00225v1…