Scenic: A Language for Scenario Specification and Data Generation

1 Introduction

• Real-world data can be slow and expensive to collect and label, while random scene generation can yield unrealistic or physically impossible configurations instead of scenarios useful for training or testing.Synthetic data offers bulk generation with correct labels and control over its distribution.Source: §1 · p. 2
• Scenic specifies distributions over scenes and dynamic-agent behaviors, combining imperative object construction with declarative hard and soft constraints and concise spatial and temporal constructs.Its syntax also supports object properties that default based on other properties, and composing or generalizing scenarios.Source: §1 · pp. 3, 5
• Scenic scenarios range from individual test cases to broad classes, developed top-down by adding constraints or bottom-up by generalizing a failure, with structured scenarios between those extremes.The paper presents the three approaches as useful for different training, testing, and debugging tasks.Source: §1 · p. 3
• Scenic's domain-specific syntax improves clarity and concision, while restrictions on its programs enable sampling algorithms that exploit distribution structure to prune the sample space.The paper contrasts this with implementing Scenic as a library or translating it into general-purpose probabilistic programming languages.Source: §1 · p. 4
• The case study uses Scenic for condition-specific testing, rare-case training, and debugging; the approach also applies beyond visual perception and GTAV to robotics and CARLA falsification.Scenic can generate different data types through simulator interfaces, including images, LIDAR point clouds, and trajectories.Source: §1 · pp. 4–5
• With an equal-size training set, Scenic-generated hard cases significantly improved car detection; generalizing a known failure also fixed its root cause and improved original-test performance without enlarging training data.The case study used SqueezeDet, a convolutional neural network for autonomous-car object detection.Source: §1 · p. 4
• This paper extends an earlier version with dynamic scenarios and spatiotemporal pruning, scenario composition, and integration with VerifAI.The earlier version appeared as a technical report and a PLDI conference paper.Source: §1 · p. 5

General Procedure

• Probabilistic programs encode operation scenarios, which are sampled into environment configurations and rendered by a simulator as images or other sensor data for training and testing.Generated data can supplement real-world data; synthetic data can also support testing and debugging systems trained on real data.Source: §2 · pp. 5–6

Testing under Different Conditions

• For each operating condition, scenario-generated test sets let designers assess system performance, including under rare conditions that can be sampled repeatedly.When conditions are unspecified, a general scenario such as an autonomous vehicle’s Operational Design Domain can support falsification against a system specification.Source: §2 · p. 6

Training on Rare Events

• Scenarios for rare or difficult cases can generate examples to augment or replace training data, emphasizing hard cases while aiming to preserve typical-case performance.The section illustrates this with partially overlapping cars in images.Source: §2 · p. 6
• Increasing overlapping-car images in a fixed-size training set dramatically improves performance on overlaps without harming the generic dataset.The comparison is against a state-of-the-art synthetic dataset generated by randomly driving in GTAV, on which performance on overlapping images is significantly worse.Source: §2 · p. 7

Debugging Failures

• Encoding a failure configuration as a scenario allows nearby variations to be explored, helping identify relevant scene features and eventually the root cause.The root cause can then be encoded as a generalizing scenario for retraining; the described case study replaces part of the training data and improves performance on the original test set.Source: §2 · p. 7

3.1 Basic Scenarios

• Scenic’s probabilistic scene properties can encode real-world variation, helping generate training data that matches the conditions a system will encounter.For example, Scenic supports distributions for features such as distances between cars.Source: §3.1 · p. 8
• Scenic’s order-independent specifiers compose spatial relationships and object properties, while resolving their dependencies to determine values such as position and orientation.Local coordinate systems let specifications be relative to objects or vector fields, and dependencies can include a car’s model and width when positioning it beside a curb.Source: §3.1 · pp. 8–10
• Built-in requirements keep objects within the workspace, non-intersecting, and visible, while hard and soft custom requirements restrict or balance scenario conditions.Soft requirements can specify a minimum probability, such as requiring at least 90% of generated images to show a car driving on the right side of the road.Source: §3.1 · p. 11
• Scenic supports adding variation to existing scenes through mutation and reusing simulator-agnostic world models to make scenarios portable across compatible simulators.The paper illustrates mutation by adding Gaussian noise to an object’s position and heading, and describes the same driving scenario code running in CARLA and LGSVL.Source: §3.1 · pp. 12–13

3.2 Dynamic Scenarios

• Dynamic agents are Scenic objects assigned behaviors that issue actions over discrete time steps, running in parallel while the simulator advances and returns updated world state.Behaviors can query the current state to decide when to wait or act.Source: §3.2 · pp. 13–14
• Behaviors support runtime-sampled parameters and interrupts, allowing probabilistic decisions during simulation and reuse of an existing behavior with added complications.For example, a behavior can sample a threshold once at startup and braking strength at each braking step, or interrupt lane following to brake for an obstacle.Source: §3.2 · p. 15
• Scenic behaviors support hierarchical interrupts that suspend and resume execution, with priority rules, aborts, and condition- or time-based shorthand.Nested interrupts of outer statements take precedence; local variables can also preserve behavior state across interruptions.Source: §3.2 · pp. 15–16
• Behaviors can preserve local state to schedule repeated actions, such as stopping for five seconds after randomly selected driving intervals.The example draws each interval from 15–30 seconds and resets its timer after each stop.Source: §3.2 · p. 17
• Temporal requirements can constrain conditions at every time step or eventually, while monitors run in parallel to enforce more complex temporal properties.A monitor can track whether two cars have entered an intersection and reject a simulation if a third enters first.Source: §3.2 · pp. 17–18
• Behavior preconditions and invariants express assumptions about starting and ongoing states; violations reject simulations by default, but exceptions can enable recovery actions.Invariants are checked at every time step, including after an interrupted behavior resumes.Source: §3.2 · pp. 18–19
• Scenarios run indefinitely by default but can end when a condition holds or when a terminate statement executes inside a behavior or monitor.Examples terminate after the ego travels a specified distance or spends a specified number of time steps in an intersection.Source: §3.2 · pp. 19–20

3.3 Compositional Scenarios

• Reusable scenarios with parameters and setup blocks can be composed in parallel or sequence, with objects and scenario properties reused across scenarios.A compose block supports control flow such as running a scenario until a condition holds.Source: §3.3 · pp. 20–21
• Compose blocks can interrupt one scenario to run another, override an object's behavior for a sub-scenario, and distinguish an initial scenario from one invoked within another.The override is reverted when the sub-scenario terminates.Source: §3.3 · pp. 21–22
• Scenic can choose among scenarios whose preconditions hold, using uniform or weighted selection, or shuffle them into a random execution order.If no candidate scenario is enabled, Scenic rejects the simulation.Source: §3.3 · pp. 22–23

4 Syntax of Scenic

• Scenic restricts conditional branching based on random variables to behaviors, while its novel syntax concisely expresses spatiotemporal relationships.The language otherwise uses standard imperative constructs, including conditionals, loops, functions, and methods.Source: §4 · p. 23

4.1 Data Types

• Primitive types include booleans, scalars, vectors, headings, vector fields, and regions; object classes provide default values for their properties.Vectors represent positions and offsets, while vector fields associate orientations with points in space.Source: §4.1 · p. 23

4.2 Expressions

• Expressions include arithmetic, boolean, and geometric operators, while distributions produce samples and object definitions create objects in the generated scene.Scenic also permits custom distributions beyond its built-in set.Source: §4.2 · pp. 25, 27

4.3 Specifiers

• Object specifiers assign properties from dependency values, with optional assignments overridable by other specifiers; conflicting assignments or cyclic dependencies make a scenario ill-formed.Specifier dependencies determine evaluation order, and examples express relative position and orientation.Source: §4.3 · pp. 27–28

4.4 Statements

• Statements support modular scenarios, behaviors and monitors, and hard, probabilistic, and temporal requirements for controlling scenario execution.A scenario can have setup and compose blocks, while requirements can apply always or eventually over time.Source: §4.4 · pp. 29–30

5.1 Semantics of Scenic

• Scenic semantics define a distribution over scenarios, discarding runs that violate requirements and renormalizing the remaining probabilities.A generated scenario consists of an initial scene and, for dynamic programs, a policy mapping past scenes to agents’ actions.Source: §5.1 · p. 31
• Dynamic execution checks scenario composition, requirements, monitors, and agent behaviors step by step before returning the agents’ selected actions.Failed requirements discard the simulation; termination also depends on satisfying any eventual requirements.Source: §5.1 · pp. 32–33

5.2 Domain-Specific Sampling Techniques

• Geometric constraints enable sound pruning of initial scene regions, including for containment, relative orientation and distance, configuration size, and reachability.Pruning can combine spatial and temporal constraints to rule out initial scenes that force a later requirement violation.Source: §5.2 · pp. 33–34
• After pruning, rejection sampling produces the desired distribution, but may require many samples or fail to terminate when requirements have zero probability.In tested reasonable scenarios, sampling took at most several hundred iterations; pruning reduced sample counts by a factor of 3 or more. Dynamic-agent physics is an unknown black-box function that complicates general PPL techniques.Source: §5.2 · pp. 33–34

6.1 Experimental Setup

• The main case study rendered Scenic-generated GTAV scenes for SqueezeDet, using 13 car models, 14 weather types, and standard precision and recall metrics.Models were trained for 10,000 iterations unless noted, on images resized to 1248 × 384.Source: §6.1 · p. 35

6.2.1 Testing a Perception Module

• For perception testing, the researchers generated 4,000 generic training images and specialized good- and bad-weather test sets from scenarios of one to four cars.The generic test set contained 200 images; good- and bad-weather scenarios fixed time and weather conditions.Source: §6.2.1 · p. 36
• The detector achieved 85.7% precision and 94.3% recall on good-weather tests, versus 72.8% precision and 92.8% recall on bad-weather tests.On the generic test set, precision was 83.1% and recall was 92.6%; the authors note rain was less likely than shine in the default weather distribution.Source: §6.2.1 · p. 36

6.2.2 Falsifying a Dynamic Closed-Loop System

• Safety-specification violations occurred in approximately 21% of Town05 tests and 17% of Town03 tests, with failures often at traffic-light trigger distances of 15–25 m.The trigger distance was the dominant factor controlling failures across the two maps.Source: Figure 11 · p. 37, §6.2.2 · p. 37
• Four-lane intersection legs with turning radii around 6.5 m most often caused failures, when the agent and crossing car merged into the same lane simultaneously.Scenic helped identify scenarios in which the closed-loop system violated its specification.Source: §6.2.2 · p. 37

6.3 Training on Rare Events

• Replacing 5% of the training set with Scenic-generated overlap images significantly improved performance on overlapping cars without reducing performance on the original test set.The total training-set size remained fixed, and results were averaged over eight training runs.Source: Table 6 · p. 39, §6.3 · p. 39

6.4 Debugging Failures

• Failure-variation tests implicated car model, color, camera distance, and view angle, while background position mattered least.The first round found model and color most relevant; later tests refined the role of proximity and angle.Source: §6.4 · pp. 40–41
• Retraining with specialized close-car images reduced false positives, while classical augmentation hurt performance and shallow-angle training helped less.The authors suggest classical augmentation overfit slight variants of one image and shallow-angle training overfit the restricted angle range.Source: Table 8 · p. 41, §6.4 · p. 41

Data Generation and Testing for ML

• Prior synthetic-data and domain-adaptation work showed that synthetic-only training can yield acceptable real-world performance; Scenic adds systematic, language-based data generation for cyber-physical systems.Applications cited include text recognition, autonomous driving, and semantic image segmentation.Source: §7 · pp. 41–42

Probabilistic Programming Languages

• Unlike GANs, Scenic needs only a simulator and supports declarative constraints; unlike related scene-generation PPLs, its domain-specific design supports geometry, dynamic behavior, and specialized sampling.Picture was used for inverse rendering rather than data generation, while Quicksand and Picture lack specialized syntax or semantics for geometry or dynamic behaviors.Source: §7 · p. 42

Model-Based Test Generation

• Scenic combines fine-grained control, global-property enforcement, probability distributions, and simple imperative syntax, unlike example-based or grammar-based test generation approaches.Example-based methods lack fine-grained control, while grammars do not easily enforce global properties.Source: §7 · p. 42

Scenario Description Languages for Autonomous Driving

• Compared with Paracosm and M-SDL, Scenic is probabilistic by design, offers higher-level geometric constraints, and is not limited to autonomous driving.M-SDL provides declarative constraints and compositional features, but distributions are optional.Source: §7 · p. 43

8 Conclusion

• Scenic specifies distributions over physical configurations and agent behaviors, supporting specialized testing, falsification, hard-case training, and failure generalization.The car detector improved beyond prior synthetic-data methods not based on probabilistic programming, with training-set size held fixed; broader domain experiments and language extensions remain future work.Source: §8 · p. 43

A.1 The scenic.simulators.gta.model Module

• The GTA world model defines Car properties, including road-relative heading through roadDeviation, and provides convenience subclasses such as EgoCar.A car facing 10 degrees relative to roadDirection is equivalent to one with roadDeviation 10 degrees.Source: §A.1 · p. 48

A.2 The Simplest Possible Scenario

• The simplest example creates one car without specifying properties, relying on reasonable defaults.The appendix shows scenes generated from this scenario.Source: §A.2 · p. 49, Figure 15 · p. 49

A.3 A Single Car

• A single-car scenario permits up to 10° deviation from the road direction and requires visibility, a constraint Scenic already builds in.The example assigns the ego car and another car road deviations within the same wiggle range.Source: §A.3 · p. 50, Figure 16 · p. 50

A.4 A Badly-Parked Car

• The badly-parked-car scenario places a car half a unit left of a visible curb point and gives it a randomly signed 10–20° deviation from the road direction.The appendix illustrates scenes generated from this scenario.Source: §A.4 · p. 51, Figure 17 · p. 51

A.5 An Oncoming Car

• The oncoming-car scenario places a second car 20–40 m ahead and requires it to be visible to the ego car within a 30° view angle.With the default road-aligned heading, the visibility requirement places its orientation within 15° of facing the camera.Source: §A.5 · p. 52, Figure 18 · p. 52

A.6 Adding Noise to a Scene

• The scenario uses Scenic’s mutation feature to add noise to an otherwise fully specified scene in the Sec. 6.4 experiment.Removing the mutate statement exactly reproduces the original scene.Source: §A.6 · p. 53
• Fig. 19 shows scenes generated by adding noise to the original misclassified image shown in Fig. 20.Source: Figure 19 · p. 53, Figure 20 · p. 54

A.7 Two Cars

• The generic two-car scenario is used in the experiments in Secs. 6.2 and 6.3.Source: §A.7 · p. 55
• The scenario sets the ego car’s road deviation to a wiggle from -10 to 10 degrees and resamples that wiggle for each of two visible cars.Source: §A.7 · p. 55
• Fig. 21 depicts two cars facing close to the direction of the road.Source: Figure 21 · p. 55

A.8 Two Overlapping Cars

• The scenario produces images of two partially overlapping cars for the experiment in Sec. 6.3.Source: §A.8 · p. 56
• A visible car is placed beyond c by leftRight, defined as Uniform(1.0, -1.0) × (1.25, 2.75), at distance range (4, 10), and receives roadDeviation resample(wiggle).wiggle = (-10 deg, 10 deg); the ego car receives roadDeviation wiggle directly, while c and the placed car receive resample(wiggle).Source: §A.8 · p. 56
• Fig. 22 depicts two cars, with one partially occluding the other.Source: Figure 22 · p. 56

A.9 Four Cars, in Poor Driving Conditions; A.12 Robot Motion Planning with a Bottleneck

• The examples range from four-car scenes in rain to a robot-planning rubble field whose bottleneck forces the planner to consider climbing over a rock.The robot scenario places a rock near the route to the goal, pipes that constrain passage, and additional obstacles; generated workspaces are shown in Webots.Source: §A.9 · p. 57, Figure 23 · p. 57, §A.12 · p. 61

B.2 Semantics of Expressions

• Object construction resolves specifier dependencies in an evaluation order, rejects conflicting or cyclic specifications, and accumulates probabilities when specifiers introduce randomness.Properties are determined by non-optional specifiers, optional specifiers, or defaults, in that order.Source: §B.2 · pp. 63–64

B.3 Semantics of Statements

• Hard requirements discard executions that fail their conditions, soft requirements enforce conditions with their stated probability, and mutations add sampled Gaussian noise to object positions and headings.After mutations, execution terminates only if Scenic’s built-in hard requirements are satisfied.Source: §B.3 · pp. 65–66

B.4 Semantics of a Scenic Program

• A Scenic program defines a distribution over scenes by combining probabilistic execution with requirements that condition the distribution on satisfying constraints.Discarding non-terminating traces that violate hard requirements and normalizing the remaining trace probabilities yields the scenario’s distribution over scenes.Source: §B.4 · pp. 66–67

B.5 Sampling Algorithms

• Scenic’s domain-specific sampling algorithms prune candidate map regions by relative orientation and minimum width, using distance and heading-deviation bounds.The width-based algorithm uses a subroutine to find polygons thinner than the minimum-width bound.Source: §B.5 · p. 67

C Detailed Semantics of Specifiers and Operators

• The appendix specifies position and heading specifiers and value-type-specific operators, with object construction evaluating property-dependent specifiers only after their dependencies are available.Its notation also defines vector-field following using a forward Euler approximation with N = 4 steps.Source: §C.2 · p. 68, §C.1 · p. 68, §C.3 · p. 70

C.5 Operators

• Figures 38–41 present four operator categories: Heading, Vector, Region, and OrientedPoint operators, respectively, with each caption identifying the corresponding figure’s operator type.Source: Figure 38 · p. 72, Figure 39 · p. 73, Figure 40 · p. 73
• Table 9 presents average-precision results for the experiments in Table 6.The caption provides no numerical results.Source: Table 9 · p. 74

• The additional experiment uses the generic two-car Scenic scenario as a baseline for an experiment analogous to Sec. 6.3.Source: D Additional Experiments · p. 74

• Because GTAV lacks an explicit map, the authors approximated one from a bird’s-eye schematic and manually filtered generated images with cars on sidewalks or medians.They note this manual filtering is unnecessary with a real simulator such as Webots.Source: Additional Details on Experimental Setup · p. 74
• Average precision and recall are used to evaluate model performance over a collection of test-set images.Source: Additional Details on Experimental Setup · p. 74

Overlapping Scenario Experiments

• Using a mixture of training images significantly improves performance on overlapping images while leaving performance on the original dataset unchanged.The comparison is reported in AP for the experiment replacing part of the Driving in the Matrix training set with overlapping-car images.Source: Overlapping Scenario Experiments · p. 74
• Intersection over Union measures overlap between predicted and ground-truth boxes, with detections counted when IoU exceeds 0.5.Precision and recall are defined from true positives, false positives, and false negatives.Source: Overlapping Scenario Experiments · p. 74
• The generic two-car and overlapping scenarios each generated 1,000 training images, and each scenario supplied a 400-image test set.Models were evaluated on both test sets.Source: Overlapping Scenario Experiments · pp. 74–75
• Average overlap is much lower in generic two-car training images than in overlapping-scenario images, making overlap atypical in the generic distribution.Fig. 42 shows the IoU distributions on a log scale.Source: Overlapping Scenario Experiments · p. 74, Figure 42 · p. 75
• Table 10 reports performance for models trained on mixtures and tested on both sets, averaged over eight training runs.The caption defines 90/10 as a 9:1 mixture.Source: Table 10 · p. 75
• Training runs used the maximum precision and recall from 4,000 to 5,000 steps, sampled every 250 steps, with eight repetitions per mixture.Each repetition used an independently randomized choice of images for the mixture.Source: Overlapping Scenario Experiments · p. 75