Source-linked AI summary
SPECOLA: A Tool for Specification Driven Robot Evaluation
Immanuel Ampomah Mensah, Hadas Kress-Gazit
TL;DR
Simulation benchmarks commonly use success rate, but benchmark thresholds can skew conclusions about policy performance. SPECOLA discovers simulator signals, supports STL specification writing, and integrates monitors into policy evaluation; its case studies show how robustness can qualify benchmark success and guide rollout analysis.
Problem
Benchmark success decisions depend on hand-designed thresholds that can skew conclusions about robot policy performance.
Source: B. Failures that should be successes · p. 5Method
SPECOLA discovers simulator signals, supports writing STL specifications, and connects monitors to policy execution through an API and adapters.
Source: §I · p. 1, B. Writing specifications · p. 3, C. Running Monitors · p. 4Results
Case studies show robustness can qualify benchmark success: the goal held for a full second in only 16 of 47 rerun successful rollouts.
Source: D. Unstable successes · p. 7Takeaways & Limitations
Robustness scores support more nuanced policy assessment, evaluation of requirements beyond benchmark success criteria, and selection of rollouts for visualization.
Source: §VI · p. 8
Abstract
from arXiv · showhide
Simulation benchmarks are a useful way to measure progress in robotics; typically they provide success rate as the dominant measure of performance. However, success rate reports only whether a rollout completed the task, hiding by what margin it succeeded, how sensitive the success is to goal parameter changes, requirements it may have violated, and whether success persisted. Signal Temporal Logic (STL) provides a formal language for capturing specifications; it can be used to define requirements, and measure how close a robot behavior is to satisfying or violating them. Using STL monitors in a simulator, however, requires exposing simulator state and mapping it to specification-relevant signals. We present Specola, a tool for evaluating robot behaviors in simulation with respect to STL specifications. Specola provides an interface for writing specifications over simulator signals, and creates and manages the relevant monitors while a policy executes. We provide a Python API and command-line interface for integrating monitoring into existing policy execution loops. We provide an adapter interface for connecting new simulators, and provide adapters for nine simulation environments and benchmarks, including LIBERO, RoboCasa, Isaac Lab, and RoboDojo. We provide case studies that illustrate the nuanced evaluation that Specola enables. Code and adapters are available at https://github.com/specola-eval/specola.
I. INTRODUCTION
Success rate compresses robot behavior into a thresholded binary outcome, while STL robustness measures how close a rollout comes to satisfying or violating a requirement. SPECOLA connects these specifications to simulator signals and monitors them during policy execution.
I. INTRODUCTION
- Success rate can hide how close failures came to success, how close successes are to failure, and how benchmark thresholds affect policy comparisons.The paper situates this evaluation problem in the use of simulation benchmarks for rapid progress in robot learning.Source: §I · p. 1
- STL expresses requirements over continuous robot and environment signals and assigns rollouts robustness scores indicating satisfaction and distance from violation.The sign indicates whether a specification holds; the score’s magnitude indicates how close the signal is to satisfying or violating it.Source: §I · p. 1
- SPECOLA discovers monitorable signals, supports specification writing, integrates monitors into policy loops through Python and command-line interfaces, and provides adapters for nine simulators.The paper also provides instructions for writing adapters for other simulators.Source: §I · p. 1
A. Evaluation of Learned Robotic Policies
- Prior policy-evaluation work measures partial completion or statistically compares bounded scores, while other tools evaluate safety with STL or hand-coded checks.MANGO, SafeManip, SENTINEL, and LabGuard require predicates or entities in advance; SPECOLA automatically discovers entities and monitorable signals. XPolicyLab standardizes policy-environment interfaces, a different evaluation connection.Source: A. Evaluation of Learned Robotic… · pp. 1–2
B. STL Monitoring Tools and Semantics
Online STL monitors update robustness as signal samples arrive, but applying them in simulation requires mapping simulator state to specification signals. SPECOLA supplies that integration through reusable adapters and a discovery interface.
B. STL Monitoring Tools and Semantics
- Applying online STL monitors to simulators requires mapping simulator state to the signals referenced by the specification.SPECOLA provides this connection through reusable adapters and a discovery interface.Source: B. STL Monitoring Tools and Semantics · p. 2
III. PRELIMINARIES
- The monitored past-time STL fragment uses discrete-time signals and temporal operators over bounded trailing windows.Historically requires a formula to hold throughout its window, Once requires it to hold at some point, and Since combines a prior condition with a condition holding afterward through the current step.Source: §III · p. 2
- Boolean conjunction requires both formulas to hold, while Historically, Once, and Since encode their respective window conditions.Quantitative robustness has the same satisfaction sign as Boolean semantics, and its absolute value indicates distance from satisfaction or violation.Source: §III · p. 2
IV. SPECOLA WORKFLOW
SPECOLA’s workflow discovers simulator entities and signals, supports writing specifications over them, and monitors them within the policy evaluation loop. Discovery organizes scene elements and exposes their signal properties and metadata.
IV. SPECOLA WORKFLOW
- The workflow example monitors bowl position, drawer pose, gripper state, and end-effector force in a LIBERO task.The task is to put a black bowl in the cabinet’s bottom drawer and close it.Source: §IV · p. 2
A. Discovering signals
- The discover interface lists entities by class and role, with signals, units, and reference frames where available.In LIBERO, objects can be grouped as targets or distractors when the adapter supplies role information.Source: A. Discovering signals · p. 2
- Entities expose named scalar properties as signals; vector properties can include component and reference-frame metadata, and contact entities may appear only as interactions occur.The drawer-task example selects a robot wrist torque command and opens its signal metadata panel.Source: A. Discovering signals · pp. 2–3, Figure 2 · p. 3
IV. SPECOLA WORKFLOW
- The workflow proceeds from an existing evaluation loop through signal discovery and specification writing to monitoring while the policy runs.It also supports connecting other simulators through adapters, with adapters provided for nine environments and benchmarks.Source: §IV · p. 2, Figure 1 · p. 3
B. Writing specifications
SPECOLA lets users write or compose specifications over discovered signals, validates formulas against the grammar and connected scene, and expands selectors into monitors for matching entities or properties.
B. Writing specifications
- SPECOLA specifications can be written in YAML or composed interactively, and the grammar’s STL operators have corresponding SPECOLA syntax.The operators and their SPECOLA forms are summarized in Table I.Source: B. Writing specifications · p. 3, Table I · p. 3
- For a wrist torque command u(t) and permitted magnitude τmax, the specification |u(t)| ≤ τmax expresses the torque bound.The paper presents the expression in SPECOLA YAML using supported arithmetic and comparison symbols.Source: B. Writing specifications · p. 3, Figure 3 · p. 3
- A selector of the form class_entity[properties] references signals by class, entity, and property, with wildcards or quantity filters supporting broader selection.A numeric entity name selects one entity; an asterisk selects every entity of the class, and property selections can name properties or use a wildcard.Source: B. Writing specifications · pp. 3–4
- The example requirements cover drawer closure, robot contact force below 140 N, and wrist torque limits over a one-second window.The wrist requirement can cover both wrist commands; a separate example sets the torque limit to 12 N m.Source: B. Writing specifications · pp. 3–4
- Selectors and requirements expand into monitors for matching scene entities, contacts, or robot properties.The drawer requirement yields one monitor, the contact requirement one per robot-body contact, and each torque requirement one monitor for the robot.Source: B. Writing specifications · p. 4
- Interactive specification composition validates formulas against the grammar and connected scene before saving, and manually written files can be checked the same way.The check resolves entity references and signal names against the scene.Source: B. Writing specifications · p. 4
C. Running Monitors
SPECOLA integrates online monitoring into policy execution loops and also supports standalone rollout evaluation, reporting, and replay through command-line tools.
C. Running Monitors
- SPECOLA adds monitoring to existing policy-evaluation code through an import and four API calls.The example attaches a session after environment setup, then evaluates the initial state and each executed action, including when a policy returns action chunks.Source: C. Running Monitors · p. 4
- The monitor discovers scene signals, compiles and binds specifications, and returns robustness scores and truth values at each step.At rollout end, the session saves the benchmark success flag with the results.Source: C. Running Monitors · p. 4
- The standalone run command streams robustness values and saves traces, while report summarizes runs and view replays them without the simulator.Run can apply actions in the simulator or replay a recorded demonstration; report groups statistics and plots by task and policy, and view includes video and a robustness trace.Source: C. Running Monitors · pp. 4–5
D. Adding New Simulators
A simulator adapter maps simulator state into SPECOLA observations, with required hooks for timing, reset, and observation and optional rendering-related hooks.
D. Adding New Simulators
- An adapter supplies the monitoring period, spatial dimensions, a reset hook, and an observation hook that returns scene entities and their properties.Each observation includes a step index, time, and entities with a class, episode-stable identifier, optional position and velocity, and named scalar properties.Source: D. Adding New Simulators · p. 5
- Rendering, camera, and scene-geometry hooks are optional, and adapters register under a simulator name from a package or the repository.For MuJoCo-based simulators, a shared adapter handles most work; new adapters provide entity-classification rules.Source: D. Adding New Simulators · p. 5
- For MuJoCo-based simulators, a new adapter can reuse the shared adapter and supply only rules for classifying entities.The documentation provides an implementation template, and a coding-agent skill guides adapter writing.Source: D. Adding New Simulators · p. 5
V. CASE STUDIES
The case studies use robustness scores to provide policy-evaluation insight beyond success rate, including false failures, failure modes, and unstable successes.
V. CASE STUDIES
- Robustness scores can help detect false failures that may distort success-rate-based policy comparisons and identify failure modes.The paper also examines unstable successes that might become failures.Source: §V · p. 5
- The case studies evaluate behaviors beyond the benchmark’s success criteria.Source: §V · p. 5
A. Experimental Setup
The experiments evaluate policy rollouts in LIBERO and RoboDojo, with benchmark-mirroring STL goals and a separate measurement of monitoring overhead.
A. Experimental Setup
- LIBERO evaluation covered four policy systems on 40 tasks across the Spatial, Object, Goal, and Long suites, with 50 runs per policy under shared initial conditions.RoboDojo evaluation used three policies on three-bowl stacking across 25 initial conditions.Source: A. Experimental Setup · p. 5
- For each task, the goal specification conjoined the predicates checked by the benchmark, and each predicate was also monitored individually.Source: A. Experimental Setup · p. 5
- Full monitoring lengthened the control loop by 17.6% ± 7.8%, while all 36 LIBERO tasks produced byte-identical actions, states, termination flags, and outcomes across monitoring conditions.Per step, state extraction cost 3.3 ± 1.1 ms and specification evaluation 0.49 ± 0.10 ms; memory grew by 34 ± 29 MB.Source: A. Experimental Setup · p. 5
B. Failures that should be successes
Robustness scores reveal when benchmark thresholds classify near-identical or task-appropriate outcomes differently. In the two-mug task, inspecting near-threshold failures changes the apparent comparison between policies.
B. Failures that should be successes
- Two pudding placements that differed by less than the pudding’s width received opposite labels: one succeeded by 0.056 mm and the other failed by 14.9 mm.The benchmark’s hand-designed state thresholds can skew success-rate conclusions.Source: B. Failures that should be successes · p. 5
- The pudding plot shows goal robustness for all 50 rollouts per policy, marking VLA-Adapter’s highest and lowest successes and OpenVLA’s lowest and highest failures.The marked near-boundary cases contrast with a placement well inside the basket at +57.9 mm and a juice carton placed instead of the pudding at −448.5 mm.Source: B. Failures that should be successes · pp. 5–6, Figure 4 · p. 6
- Five π0.5 failures were within 4 mm of the threshold, and counting them as successes raises its rate from 90% to 100%, matching VLA-Adapter.Both mugs rested on their plates in all five rollouts.Source: B. Failures that should be successes · p. 6
C. Failure modes
Predicate-level robustness distinguishes how policies fail bowl stacking, while temporal monitoring tests whether a benchmark success persists after it is declared. These views expose differences that binary outcomes alone do not show.
C. Failure modes
- π0.5, MolmoAct2, and StarVLA PI-v3 succeeded on 19, 11, and 13 of 25 layouts, respectively; π0.5 failures included opposite failures—arms not returned or bowls tipped.The benchmark assigned all six π0.5 failures the same process score of 15, while predicate robustness showed all policies were nearly 100% successful at keeping bowls upright and π0.5 was most robust at stacking.Source: C. Failure modes · p. 6
D. Unstable successes
- A goal-state check can miss an immediate later violation, whereas temporal specifications allow analysis of whether success persists.The rerun monitoring checked whether the goal held throughout the past second.Source: D. Unstable successes · pp. 6–7
C. Failure modes
- The bowl-stacking plot shows final robustness for six requirements across all 25 rollouts per policy, with colors identifying policies and markers identifying π0.5 layouts 7 and 18.Those two layouts illustrate opposite failures: bowls stacked but arms did not return, versus arms returned but bowls tipped over.Source: C. Failure modes · p. 6, Figure 6 · p. 7
D. Unstable successes
- Only 16 of OpenVLA-OFT’s 47 successful two-mug rollouts held the goal for the full second after success; some benchmark success determinations may therefore be premature.In trial 37, the mug was still held by the gripper at first success, then lifted again by 26.5 mm.Source: D. Unstable successes · p. 7
E. Beyond “success”
SPECOLA’s specifications reveal distinctions that task success alone misses, including contact-force violations and whether requirements persist during successful rollouts. The discussion highlights robustness-based evaluation and rollout selection, while noting that current monitoring is limited to privileged simulator state at a fixed control period.
E. Beyond “success”
- 42 of OpenVLA-OFT’s 49 successes and 47 of π0.5’s 50 exceeded the 140 N contact limit while pressing the stove button.Most rises were brief, while others were sustained presses.Source: E. Beyond “success” · p. 7
- The benchmark check tracks whether the goal continues to hold at each control step across all 47 successful rollouts.The figure marks goal-holding steps in green and violations in orange, illustrating persistence after first success.Source: Figure 7 · p. 7
- 15 of OpenVLA-OFT’s 49 successful rollouts and 12 of π0.5’s 50 violated the refined force-and-recovery specification.It requires force below 280 N and any force above 140 N to return below that threshold within 0.5 s; violations included sustained presses and single-step spikes.Source: Figure 8 · p. 7, E. Beyond “success” · p. 7
VI. DISCUSSION
- SPECOLA supports evaluating robustness and requirements beyond task success, including specifications with timing and conditions on simulator signals.Examples include gripper-width constraints and detecting periods when the robot freezes.Source: §VI · p. 8
- Robustness scores can help select a small set of rollouts for visualization, rather than viewing hundreds of initial-condition runs.The case studies use rollouts chosen based on robustness to gain further insight.Source: §VI · p. 8
- Current monitoring uses privileged simulator state with one observation per action at a fixed control period.Extending to physical robots requires addressing asynchronous, delayed sensor observations and uncertainty, including perception failures.Source: §VI · p. 8