Video2STL: Grounding VLM-Generated Temporal Specifications for Robot Learning

1 INTRODUCTION

• Scalar rewards can hide how a task unfolds over time, while reward code can entangle temporal structure with thresholds, APIs, and implementation choices.These issues matter in cross-embodiment settings where pixel or pose correspondence is unreliable and transferable content is often semantic rather than kinematic.Source: §1 · pp. 1–2
• Video2STL maps a video to an embodiment-independent semantic trace and parametric STL bank, then grounds thresholds and temporal bounds using successful robot trajectories.The VLM specifies task structure; robot data supplies embodiment-specific numerical quantities.Source: §1 · pp. 2–3
• Short-horizon formulas provide dense local rewards through rolling-window robustness, while a causal prefix monitor gives one-time rewards for valid stages of long-horizon tasks.The split keeps dense feedback local while preserving event ordering and deadlines.Source: Figure 1 · p. 2, §1 · p. 3
• Sign-preserving Boltzmann reductions smooth STL robustness while preserving the hard min/max satisfaction boundary.The smoothing retains the hard formula's sign while providing smoother within-region magnitudes for reward construction.Source: §1 · p. 3
• The framework uses one reusable ontology across manipulation tasks and supports cross-embodiment transfer while keeping the representation inspectable through policy training.The expert trajectories calibrate and validate specifications but do not provide action supervision or determine symbolic structure.Source: §1 · p. 3

2 RELATED WORK

• Video methods transfer visual features, translated demonstrations, or adapted policies, while pretrained visual models also produce embedding-based rewards or VLM-estimated task progress.Earlier approaches include context translation, viewpoint-invariant representations, and domain-adaptive meta-learning.Source: §2 · pp. 3–4
• Foundation models have generated or refined executable rewards from language or visual feedback, including Text2Reward, Eureka, Video2Reward, and ROSETTA.Other approaches use proxy reward functions, visual-language similarity, or VLM preference comparisons.Source: §2 · p. 4
• Lang2LTL begins with language commands and uses logic for planning, whereas Video2STL extracts temporal task structure from video and checks it against embodiment-specific robot data.Temporal-logic rewards, learned specifications, and formal planning for legged robots are also established research directions (Kapoor et al., 2020; Atasever et al., 2026).Source: §2 · p. 4
• Video2STL transfers video-derived temporal specifications into grounded rewards and causal monitoring, a more structured interface than scalar visual rewards and a more constrained one than free-form reward code.Explicit satisfaction boundaries can be checked on held-out expert trajectories, with different credit-assignment mechanisms for short- and long-horizon structure.Source: §2 · p. 4

3.1 PROBLEM SETTING

• The setting assumes a successful behavior video and successful robot trajectories, without requiring them to come from the same embodiment.The video supplies semantic events and temporal relationships; robot trajectories ground and filter the numerical parameters.Source: §3.1 · pp. 4–5

3.2 SIGNAL TEMPORAL LOGIC (STL) & QUANTITATIVE ROBUSTNESS

• STL expresses real-valued signal properties using predicates, Boolean connectives, and bounded temporal operators; quantitative robustness is positive for satisfaction and negative for violation.Parametric STL allows predicate constants and temporal intervals to be parameters, enabling data-driven instantiation from expert trajectories.Source: §3.2 · p. 5

4.1 VIDEO TO AN EMBODIMENT-INDEPENDENT SEMANTIC TRACE

• The video-processing stage extracts semantic events, entity roles, and temporal relations using a shared ontology, while reporting visually important concepts missing from that ontology.The trace identifies event classes and relations such as before, overlaps, and alternates_with rather than inventing unsupported predicates.Source: §4.1 · p. 5

4.2 SEMANTIC TRACE TO A PARAMETRIC STL BANK

• A second VLM stage converts the trace, not the original video, into a variable-size bank of symbolic STL formulas with no numerical thresholds or time constants.A bank allows requirements at different temporal scales; the PushCube example contains seven formulas.Source: §4.2 · pp. 5–6

4.3 TARGET-EMBODIMENT GROUNDING

• Predicate thresholds and temporal bounds are fitted from successful grounding trajectories using empirical quantiles rather than hand-tuned values or a single trajectory.High but non-extreme quantiles for error quantities reduce sensitivity to brief deviations under always operators; lower quantiles are used for floor-type quantities.Source: §4.3 · p. 6

4.4 EXPERT-CONSISTENCY FILTERING

• A formula is retained when the alpha-quantile of its held-out full-trajectory robustness is nonnegative; all seven PushCube formulas pass the Q0.10 test.In this example, filtering serves as held-out validation rather than aggressive pruning.Source: §4.4 · p. 6

4.5 QUANTITATIVE STL AND SIGN-PRESERVING SMOOTH SEMANTICS

• Sign-preserving smooth min/max reductions retain the satisfaction boundary while smoothing robustness magnitudes for reward construction.The operators preserve the corresponding hard min/max sign.Source: §4.5 · p. 6

4.6.1 LOCAL ROLLING-WINDOW ROBUSTNESS

• Retained formulas whose temporal span fits the local horizon are evaluated on a trailing window, normalized, and averaged into dense local feedback.Each formula keeps its temporal semantics and satisfaction boundary; longer-span specifications are excluded from this term.Source: §4.6.1 · p. 7

4.6.2 LONG-HORIZON CAUSAL PROGRESS

• For manipulation, a causal monitor tracks the furthest valid ordered stage reached in the long-horizon backbone and rewards only newly reached stages.This sparse progress signal complements dense local robustness; quadruped locomotion uses only local robustness because it is periodic rather than a fixed stage progression.Source: §4.6.2 · p. 7

5 EXPERIMENTS

• The evaluation spans periodic quadruped locomotion and four staged manipulation tasks, and all methods share the policy architecture, optimization, training budget, environment, and evaluation protocol.Only reward formulation changes in the main comparison.Source: §5 · pp. 7–8, §5.2 · p. 9

5.1 QUADRUPED LOCOMOTION

• 100% success across 0.3–2.1 m/s for Video2STL-Qwen, Video2STL-GPT-5.6, and Text2Reward; the heuristic reward falls to 5% at 2.0 m/s and 0% at 2.1 m/s.All methods achieve 100% survival, so survival alone does not distinguish the controllers.Source: Table 1 · p. 8, §5.1 · p. 8
• At 1.9, 2.0, and 2.1 m/s, Video2STL-Qwen CoT is 0.98, 1.00, and 1.02, versus Text2Reward’s 1.00, 1.07, and 1.14.Text2Reward has the lowest CoT over most low- and mid-speed commands; Video2STL-Qwen is increasingly competitive at higher speeds.Source: §5.1 · p. 8, Table 1 · p. 8

5.2 ROBOT MANIPULATION

• 85.8% success-once and 67.0% success-at-end averaged over four manipulation tasks, versus 81.5%/59.5% for native dense PPO and 65.0%/42.3% for Text2Reward.Video2STL’s largest gain is on PlaceSphere; LiftPegUpright remains difficult to retain, partly because its grounded Upright tolerance exceeds the peg’s tipping angle.Source: Figure 2 · p. 9, §5.2 · p. 9
• Manipulation success-once counts episodes reaching native success at least once, while success-at-end counts episodes satisfying it in the final state; results average 128 episodes.The four ManiSkill3 tasks cover pushing, grasping and stacking, orientation-dependent manipulation, and placement.Source: §5.2 · p. 9

• Video2STL converts videos into interpretable temporal task specifications and uses them to guide robot learning across different embodiments.Source: §6 · p. 9
• Across quadruped locomotion and four manipulation tasks, video-derived temporal structure provides effective learning signals without manually engineered dense rewards.Source: §6 · p. 9
• The authors report using generative AI for parts of development and writing, while stating that it did not fabricate experimental data or quantitative results.AI-generated code and figures were reviewed, and manuscript claims were checked by the authors.Source: AI USE STATEMENT · pp. 9–10

A.1.1 COMPARISON OF VLM CONFIGURATIONS

• Table 2 compares quadruped locomotion performance using STL specifications generated by different VLMs.Source: Table 2 · p. 13
• The table reports survival and success as percentages, with higher values better, and CoT, with lower values better.Source: Table 2 · p. 13
• The table’s scope is locomotion performance across VLM-generated specifications; its caption does not state which configuration performs best.Source: Table 2 · p. 13, §A.1.1 · p. 13

A.1.2 REWARD CONSTRUCTION

• The locomotion video shows a dog running on a treadmill, from which the VLM extracts contact, swing, touchdown, liftoff, support, trunk-stability, and periodic-leg-motion semantics.Source: §A.1.2 · p. 13
• GPT-5.6 generated 11 candidates; four passed a pooled fifth-percentile robustness filter on 50 trot-expert trajectories, while φ11 was not implemented because its predicate was ungrounded.Source: §A.1.2 · p. 13
• Qwen generated 15 candidates; after rejecting φ11 for disjunction and filtering on 50 trot-expert trajectories, seven candidates remained.Source: §A.1.2 · p. 13
• Gemini’s six candidates and torque-safety and forward-tracking specifications passed expert-rollout screening; lateral and yaw tracking were excluded.Source: §A.1.2 · p. 13

A.1.3 HYPERPARAMETERS AND DOMAIN RANDOMIZATION

• All reward variants use the same policy architecture and PPO settings, with training lasting 400M environment steps under the listed optimization parameters.The settings include 8192 parallel environments, a batch size of 256, and a learning rate of 1.5 × 10^-4.Source: §A.1.3 · p. 13
• Table 3 reports GPT-5.6 specifications and final weights; gray formulas were filtered out, φ11 was rejected before training, and all tanh scales are 1.Source: Table 3 · p. 14
• Table 4 lists GPT-5.6 grounding and reward parameters, with values fixed in the implementation except the VLM-specified support count and filtering rule.Parameter triples are ordered walk/trot/bound, and ∆t = 0.02 s.Source: Table 4 · p. 14
• Friction is sampled from (0.6, 1.4), while actuator gain and bias perturbations are sampled from (−5, 5).Source: §A.1.3 · p. 14

A.1.4 SOURCES VIDEOS & VLM PROMPTS FOR QUADRUPED LOCOMOTION

• The locomotion workflow uses two sequential VLM prompts: one extracts an embodiment-independent semantic trace from an animal video, and the next converts only that trace into candidate PSTL specifications.The resulting specifications are intended for a quadruped robot with different morphology, body dimensions, and controllers.Source: §A.1.4 · p. 14, Stage 1: Video to Semantic Locomotion… · p. 16
• The trace prompt prioritizes repeated stance, swing, touchdown, liftoff, inter-leg timing, and persistent body regularities, while recording ambiguity rather than guessing obscured events.It excludes robot-specific metrics and numerical thresholds, and warns against interpreting treadmill-stationary body coordinates as zero forward velocity.Source: Stage 1: Video to Semantic Locomotion… · p. 17
• The specification prompt reasons only from the semantic trace, forbids unsupported predicates or relations, and leaves temporal constants symbolic for later numerical grounding during training.Its allowed grammar covers conjunction, bounded eventuality and persistence, repeated-event and response requirements, and repeated paired events.Source: Stage 2: Semantic Trace to PSTL… · p. 18

A.2.1 REWARD CONSTRUCTION

• For PUSHCUBE, the VLM generated eight candidate specifications covering approach and engagement, displacement, goal progress and attainment, support, and post-goal stationarity.These are raw candidates before numerical grounding and expert-consistency filtering, so not all are necessarily retained in the final reward.Source: PushCube-v1 · p. 20, §A.2.1 · p. 20
• For STACKCUBE, seven candidates describe grasping and transport, supported placement and release, and persistent terminal stability.Temporal bounds are symbolic at this stage and are subsequently grounded from expert trajectories.Source: StackCube-v1 · p. 20, §A.2.1 · p. 20
• For LIFTPEGUPRIGHT, seven candidates characterize peg interaction and displacement, progress toward an upright configuration, supported placement, release, and persistence of terminal upright and supported states.Source: LiftPegUpright-v1 · p. 21
• For PLACESPHERE, eight candidates represent approaching and grasping the object, transporting it to the receptacle, placing and releasing it, and maintaining containment and stationarity.Source: PlaceSphere-v1 · p. 21
• PushCube reward robustness is defined for predicates including Near, TowardGoal, Displaced, AtGoal, OnSupport, Static, and Engaged, with task parameters for predicate thresholds and temporal bounds.The listed parameters include δnear, δengaged, δdisplaced, δprogress, δgoal, δz, δstatic and Htask, hengage, hprogress, hgoal, hsettle.Source: PlaceSphere-v1 · p. 21

A.2.2 HYPERPARAMETERS

• All four tasks use the standard ManiSkill PPO implementation without changes to its optimization procedure, with learning rate 3 × 10^-4, γ = 0.8, λ = 0.9, and 8 update epochs per batch.The setup also uses a 0.2 clipping coefficient, 0.5 value-function coefficient, target KL divergence 0.1, 32 minibatches, and three-layer actor and critic MLPs with 256 hidden units and tanh activations.Source: §A.2.2 · p. 21
• Training budgets are 2M environment steps for PushCube, 30M for StackCube, 25M for LiftPegUpright, and 10M for PlaceSphere.PushCube uses 2048 parallel environments and rollout length 20; each other task uses 1024 environments and rollout length 50.Source: §A.2.2 · p. 22

A.2.3 SOURCE VIDEOS & VLM PROMPTS FOR MANIPULATION

• Each manipulation task uses one human demonstration and the same two sequential, task-agnostic VLM prompts: one extracts a semantic event trace and the other maps it to parametric STL specifications.The prompts are applied to all four tasks without task-specific modification.Source: §A.2.3 · p. 22
• Stage 1 extracts embodiment-independent entities and temporal relationships without forcing absent roles or copying human-specific variables, kinematics, or timing.The prompt directs the VLM to describe semantic task structure rather than the demonstrator’s exact trajectory or incidental movements.Source: Stage 1: Human Video to Semantic… · pp. 22–23
• Stage 2 builds candidate PSTL formulas only from the supplied trace, keeps temporal bounds symbolic for later grounding, and permits ordered-event and terminal-stability forms.The allowed grammar also includes conjunctions, bounded eventuality and persistence, and response requirements.Source: Stage 2: Semantic Manipulation Trace… · pp. 24–25