Source-linked AI summary
From LLM-Generated Specifications to Learned Quadruped Locomotion
Merve Atasever, Keyan Azbijari, Cagan Bakirci, Alfredo Reina Corona, Tolga Izdas, Richard Yang, Erdem Biyik, Jyotirmoy V. Deshmukh
TL;DR
Hand-engineered quadruped rewards require substantial tuning, while writing explicit STL specifications also leaves a manual design burden. This paper uses LLM-proposed parametric STL structures calibrated and filtered with expert trajectories; results vary by gait setting, with Qwen-MG succeeding at high-speed commands where Text2Reward-MG terminates.
Problem
Quadruped RL rewards require substantial manual engineering, and manually designed parametric STL specifications leave specification design unresolved.
Source: §I · pp. 1–2Method
GPT-5.5 and Qwen 3.6 propose parametric STL structures whose parameters are fitted and filtered using expert trajectories, then used as PPO rewards.
Source: §VI · p. 6Results
Qwen-MG achieves 100% survival and command success at 1.9, 2.0, and 2.1 m/s, while Text2Reward-MG terminates in every rollout at those commands.
Source: §V · p. 5Takeaways & Limitations
Performance depends on the control setting, and successful command tracking does not guarantee recovery of all intended gait structures.
Source: §V · p. 5Evaluations are confined to MJX simulation, and transferability of the learned multi-gait behaviors to a physical quadruped has not been established.
Source: §VI · p. 6
Abstract
from arXiv · showhide
Quadruped robot locomotion policies are often trained using reinforcement learning, which in turn relies heavily on hand-crafted reward functions. Designing reward functions requires substantial manual engineering, and it is often unclear which local rewards will induce the desired global behavior. Shaped rewards from formal specifications in languages like Signal Temporal Logic (STL) can make rewards more interpretable, but writing STL specifications itself still requires domain expertise. We study whether large language models (LLMs) can fill this gap by generating Parametric Signal Temporal Logic (PSTL) specifications that are subsequently used for policy learning. Given a natural language locomotion objective and a constrained specification grammar, GPT-5.5 and Qwen 3.6 independently propose STL templates for command tracking, safety, and gait structure. We instantiate the parameters of the generated PSTL templates using expert trajectories and retain only specifications that are consistent with demonstrated expert behavior. The resulting specifications are then transformed into smooth, finite-history reward functions and used to train a quadruped locomotion policy with Proximal Policy Optimization (PPO) in MuJoCo XLA (MJX). We evaluate both \emph{gait-aware} and \emph{gait-agnostic} settings. The former specifies walking-trot, trot, and bound regimes, while the latter allows contact patterns to emerge from the task objective. We compare against hand-engineered rewards, Text2Reward-style LLM-generated reward code, and an expert-switching oracle. Gait-aware Qwen 3.6 specifications achieved 100\% survival and command success across all tested speeds (0.3--2.1 m/s) and matched the target gait at high speeds, whereas Text2Reward achieved 0\% for both metrics at $\geq 1.9$ m/s. Videos: https://stl-locomotion.github.io/
I. INTRODUCTION
Quadruped RL performance depends heavily on hand-engineered rewards, while their local terms do not explicitly specify the intended global temporal behavior. The paper proposes using LLM-generated symbolic STL structures, calibrated and filtered with expert trajectories, as a route to interpretable locomotion rewards.
I. INTRODUCTION
- Conventional locomotion rewards require substantial domain knowledge and empirical tuning, and their local terms do not explicitly describe intended global temporal behavior.This challenge is especially acute for multi-gait locomotion, where walking, trotting, and bounding differ in contact timing and support patterns.Source: §I · p. 1
- Prior quadruped logic-guided gait learning has largely used discrete contact rules or reward machines rather than continuous temporal-logic formulations.STL can express bounded-horizon properties over real-valued trajectories and use quantitative robustness as a dense RL signal.Source: §I · pp. 1–2
- The authors’ earlier parametric STL work improved interpretability and gait control but left manual specification design unresolved.Source: §I · p. 2
- The proposed pipeline asks LLMs for symbolic temporal specifications, then fits their parameters from expert trajectories and retains candidates consistent with demonstrated behavior.The retained specifications are converted into dense learning signals through STL robustness rather than having the LLM generate numerical reward code directly.Source: §I · p. 2
- The study evaluates gait-aware and gait-agnostic formulations and benchmarks them against a heuristic reward, Text2Reward, and an expert-switching oracle.It also filters out any generated specification whose median robustness on expert data is negative.Source: §I · p. 2
II. BACKGROUND AND PROBLEM SETTING
The problem setting uses Barkour vb in MJX to learn command-tracking policies that remain stable and energy efficient. The background defines PSTL and evaluates temporal specifications over a finite trailing window.
II. BACKGROUND AND PROBLEM SETTING
- Barkour vb is simulated in MJX, where the policy receives velocity commands, proprioception, its previous action, and a short observation history, then outputs 12 normalized joint-position commands.A PD controller maps those commands to actuator torques.Source: §II · p. 2
- The policy objective is to track commanded velocities while remaining stable and energy efficient.Source: §II · p. 2
- PSTL replaces numerical constants in STL predicates or temporal intervals with parameters that can be fitted from data; positive robustness denotes satisfaction and negative robustness denotes violation.Source: §II · p. 2
- Each temporal specification is evaluated over the trailing window from t − H through t, and its robustness values are later combined using smooth aggregation.Source: §II · p. 2
A. Specification Generation
The LLMs propose symbolic STL template structures while expert trajectories supply numerical parameters and help select gait-aware regimes. The specifications cover shared locomotion signals and, when applicable, gait-specific contact patterns.
A. Specification Generation
- GPT-5.5 and Qwen 3.6 receive the locomotion description and allowed STL grammar, and propose template structures with thresholds and temporal constants left symbolic.The numerical parameters are estimated from expert trajectories rather than generated by the LLM.Source: A. Specification Generation · pp. 2–3
- Generated formulas can combine shared tracking and safety signals with regime-specific contact statistics.Example template families address torque limits, torso orientation, center-of-mass height, support contacts, and linear-velocity tracking.Source: A. Specification Generation · pp. 2–3
- The robustness horizon H defines a trailing evaluation window, and a globally constrained predicate takes its robustness from the worst predicate margin within that window.Source: A. Specification Generation · pp. 2–3
- Gait-aware templates use walking-trot, trot, and bound regimes, with transitions set at 0.7 m/s and 1.7 m/s using reference Froude numbers and Barkour vb’s 0.41 m leg length.The gait-agnostic setting prescribes no target gait, while the multi-gait setting seeks one policy that adapts across speed regimes.Source: A. Specification Generation · p. 3
B. Robustness Reward
The reward construction aggregates robustness values from active retained specifications and normalizes grouped robustness before producing the scalar reward. This decomposition also supports inspection of individual formula robustness during evaluation.
B. Robustness Reward
- Active retained specification robustness terms are combined using a soft-min aggregation.Source: B. Robustness Reward · p. 3
- Each grouped robustness is normalized with tanh to the range [−1, 1], preventing large robustness magnitudes from dominating the reward.Source: B. Robustness Reward · p. 3
- The reward decomposition allows evaluation to report both the scalar RL reward and each retained formula’s robustness.Source: B. Robustness Reward · p. 3
C. Data Grounding and Expert-Consistency Filtering
Expert trajectories ground the numerical parameters of generated PSTL templates and filter out specifications that conflict with demonstrated behavior.
C. Data Grounding and Expert-Consistency Filtering
- 50 expert trajectories per regime, each with 500 time steps, supply the signals used to instantiate generated templates.Parameters are estimated from expert-data statistics, using upper quantiles for error ceilings and lower quantiles for safety floors.Source: C. Data Grounding and… · p. 3
- Expert trajectories are used to filter LLM suggestions that may be technically sound but incompatible or overly strict.Source: C. Data Grounding and… · p. 3
- Specifications with negative median robustness across expert evaluation points are discarded by expert-consistency filtering.Equivalently, a formula violated on more than half of those points is not used to construct the reward.Source: C. Data Grounding and… · p. 3
A. Training and Baselines
The training setup holds the Barkour MJX environment and learning configuration constant across methods while varying the reward source and comparing against several baselines.
A. Training and Baselines
- All learned policies share the Barkour MJX environment, policy architecture, PPO implementation, command distribution, and domain randomization; only the reward source changes.Training uses friction randomization in [0.6, 1.4] and actuator-gain perturbations, with new velocity commands sampled every 10 s.Source: A. Training and Baselines · p. 3
- The forward-velocity curriculum expands to 1.9 m/s as training progresses toward higher speeds.The initial command ranges include lateral velocity and yaw rate in [−0.2, 0.2].Source: A. Training and Baselines · p. 4
- The evaluation compares hand-engineered and human-STL rewards, GPT-STL and Qwen-STL, Text2Reward code, and an expert-switching oracle.The oracle selects a specialized expert for the commanded speed regime and is a reference, not a learned unified controller.Source: A. Training and Baselines · p. 4
B. Evaluation
Evaluation measures locomotion across commanded forward speeds using tracking, survival, energy-efficiency, and gait-match metrics.
B. Evaluation
- Nine forward commands from 0.3 to 2.1 m/s are tested with 20 rollouts of 500 steps per command, excluding the first 50 steps for tracking metrics.Source: B. Evaluation · p. 4
- The four evaluation metrics are cost of transportation (CoT), survival rate, success rate, and gait match.Lower CoT indicates better transport efficiency; it is total energy consumed divided by m g d, with d the planar distance traveled.Source: B. Evaluation · p. 4
- Success requires average post-warm-up forward speed to stay within ±15% of the commanded velocity, while survival means avoiding early termination.Source: B. Evaluation · p. 4
- Gait match is labeled Success, Partial, or Fail according to whether the dominant contact pattern matches the prescribed gait.Source: B. Evaluation · p. 4
- Each LLM’s robustness horizon is selected by validation performance from H values of 1, 5, 10, 20, and 30.Source: B. Evaluation · p. 4
V. RESULTS AND DISCUSSION
Across settings, outcomes depend on whether gait structure is prescribed: Qwen-MG outperforms Text2Reward-MG at high speeds, while gait mismatches and horizon sensitivity qualify success. Gait-agnostic command scores can also conceal unnatural motion.
- Qwen-GA terminates from 1.6 m/s onward, whereas Qwen-MG tracks across 0.3–2.1 m/s; GPT-MG fails command success at 1.9 m/s and above.The relative performance of the two LLM-generated specification sets depends on whether gait structure is prescribed.Source: §V · p. 5
- The Text2Reward comparison tests whether exposing LLM output as symbolic temporal constraints changes learned behavior relative to directly generating numerical reward code.The pipeline fits thresholds from expert data after generating the symbolic specification.Source: §V · p. 5
- At 1.9, 2.0, and 2.1 m/s, Qwen-MG achieves 100% survival and command success while Text2Reward-MG terminates in every rollout.Qwen-MG exhibits the desired bound contact pattern but does not reproduce several prescribed lower-speed gaits.Source: §V · p. 5
- GPT-STL and Text2Reward succeed across the full range gait-agnostically, but their multi-gait versions fail at high speeds; Qwen-STL shows the opposite trend.The benefit of adding gait-specific structure depends on the reward or specification produced by the language model.Source: §V · p. 5
- Gait-agnostic policies can score strongly yet use a bound-like contact pattern across speeds; at 0.3 m/s, 100% velocity-tracking success coincides with unnatural, oscillatory motion.Table I reports gait-agnostic results across commanded forward velocities, with CoT, survival, and success measures over 20 rollouts.Source: §V · p. 4, Table I · p. 5
- GPT-GA success falls to 44% at H = 20 after 100% success and CoT ≤1.00 for H ≤10; Qwen-MG drops to 27.8% at H = 30 from 100% success and CoT 1.28 at H = 1.Longer windows can retain past violations in the reward and make credit assignment harder, so a longer robustness horizon is not always beneficial.Source: §V · pp. 5–6
VI. CONCLUSION
The paper presents an LLM-assisted formal reward-design pipeline for quadruped locomotion, supporting both gait-aware multi-gait control and gait-agnostic locomotion. Its evaluations are confined to simulation, and transfer to a physical quadruped remains unestablished.
VI. CONCLUSION
- The pipeline uses LLM-proposed parametric STL structures, expert trajectories to set parameters and filter violated suggestions, and smooth STL robustness as the PPO reward.It supports both gait-aware multi-gait control and a gait-agnostic setting where locomotion style can emerge.Source: §VI · p. 6
- The evaluations compare gait-aware multi-gait control with a gait-agnostic setting in which locomotion style is allowed to emerge.The comparisons study what is gained and what remains difficult when temporal logic mediates between language-model generation and policy learning.Source: §VI · p. 6
- Evaluations are confined to MJX simulation, so empirical sim-to-real transfer of the learned multi-gait behaviors on a physical quadruped remains unestablished.The study uses domain randomization over friction and actuation parameters.Source: §VI · p. 6