PaperlayerView exact revision on arXiv

Source-linked AI summary

Scenic: A Language for Scenario Specification and Scene Generation

Daniel J. Fremont, Tommaso Dreossi, Shromona Ghosh, Xiangyu Yue, Alberto L. Sangiovanni-Vincentelli, Sanjit A. Seshia

arXiv:1809.09310v2cs.PLcs.CVcs.LG

TL;DR

ML perception systems need methods for rare-event training, condition-specific testing, and failure debugging, while realistic environment modeling remains difficult. Scenic addresses these needs with a domain-specific probabilistic programming language that specifies scene distributions, constraints, and sampling procedures. In a car-detection case study, Scenic improved performance beyond prior synthetic-data methods at fixed training-set size.

  • Problem

    ML perception systems need to handle rare events, varied conditions, and failures, but collecting labeled real-world data is slow and expensive and naive synthetic sampling can produce unrealistic scenes.

  • Method

    Scenic specifies distributions over scenes using probabilistic programming, declarative constraints, and domain-specific sampling techniques for generating training and test data.

  • Results

    Scenic improved a car detector beyond prior synthetic data generation methods while keeping the training set size fixed.

  • Takeaways & Limitations

    Scenic supports specialized test-set generation, rare-case training, and generalization from individual failures to broader retraining scenarios.

  • Takeaways & Limitations

    The paper identifies extending Scenic to additional domains, applications, simulators, 3D scenes, and dynamic scenarios as future work.

Abstract

from arXiv · show

We propose a new probabilistic programming language for the design and analysis of perception systems, especially those based on machine learning. Specifically, we consider the problems of training a perception system to handle rare events, testing its performance under different conditions, and debugging failures. We show how a probabilistic programming language can help address these problems by specifying distributions encoding interesting types of inputs and sampling these to generate specialized training and test sets. More generally, such languages can be used for cyber-physical systems and robotics to write environment models, an essential prerequisite to any formal analysis. In this paper, we focus on systems like autonomous cars and robots, whose environment is a "scene", a configuration of physical objects and agents. We design a domain-specific language, Scenic, for describing "scenarios" that are distributions over scenes. As a probabilistic programming language, Scenic allows assigning distributions to features of the scene, as well as declaratively imposing hard and soft constraints over the scene. We develop specialized techniques for sampling from the resulting distribution, taking advantage of the structure provided by Scenic's domain-specific syntax. Finally, we apply Scenic in a case study on a convolutional neural network designed to detect cars in road images, improving its performance beyond that achieved by state-of-the-art synthetic data generation methods.

1 Introduction

Scenic addresses the difficulty of designing, testing, and debugging ML perception systems by specifying distributions over realistic scenes and sampling specialized data. The paper presents Scenic as a domain-specific probabilistic programming language with specialized sampling techniques and reports improved car-detector performance.

  • ML perception systems need higher assurance because they increasingly serve safety-critical applications.
  • Real-world data collection is slow and expensive, while naive synthetic sampling produces unrealistic or physically impossible scenes.
  • Probabilistic programming languages specify input distributions and sample concrete inputs for training and testing, while also supporting formal environment models.
  • Scenic defines distributions over scenes and combines imperative object construction with declarative hard and soft constraints and concise geometric relationships.
  • Scenic was applied to a GTAV-rendered car-detection case study, including specialized training and test sets and debugging of network failures.
  • The paper contributes domain-specific sampling algorithms and a methodology for applying probabilistic programming to ML perception systems.

2 Using PPLs to Design and Analyze Perception Systems

The proposed workflow formalizes operating scenarios, samples concrete configurations, and uses a simulator to produce data for perception-system training and testing. Scenic also supports rare-event training and failure debugging by varying scenario features around known cases.

  • The workflow uses probabilistic programs to formalize scenarios, sample environment configurations, and obtain sensor data through simulation.
  • Testing under Different Conditions: Condition-specific scenarios can generate test sets for evaluating system accuracy under conditions such as good or bad weather.
  • Training on Rare Events: Rare-event scenarios can augment or replace training data so difficult cases receive greater emphasis.
  • Training on Rare Events: With a fixed training-set size, increasing overlapping-car images dramatically improves performance on overlaps without harming performance on the original generic dataset.
  • Debugging Failures: Encoding a failure configuration as a scenario enables systematic exploration of nearby scenes while fixing some features and varying others.

3 The Scenic Language

Scenic combines probabilistic scene generation with concise geometric specifications for constructing realistic, varied scenarios. Its object-oriented specifiers resolve dependencies among properties while supporting multiple domains and simulators.

  • Language overview: Scenic scenarios generate scenes for perception-system data by combining object-oriented modeling, geometry, and probability distributions.The language supports scenarios involving physical objects and agents, with random choices modeling real-world stochasticity.
  • Language overview: Scenic programs use default object properties and local geometric relationships to construct scenes without specifying every property explicitly.Cars can inherit default positions and headings from class definitions, while operators such as offset by and relative to express local relationships.
  • Specifiers: Specifier dependencies let Scenic determine evaluation order when properties such as position, width, and model interact.For example, a car’s position can depend on its width, whose default depends on the selected model, without sampling the model outside the object definition.
  • Specifiers: Scenic’s independent specifiers avoid proliferating constructors for different combinations of object properties while preserving readable scenario descriptions.The same object can be described with separate specifiers such as left of X by D and with model M.
  • Geometric specification: Specifier syntax supports multiple geometric relationships, including visible-region placement and orientations relative to vector fields or other objects.A parked car can be sampled from a visible part of a curb and assigned a heading offset from the road direction.

4 Syntax of Scenic

Scenic is an object-oriented probabilistic programming language whose syntax combines standard imperative constructs with geometric operators, distributions, and dependency-aware specifiers for defining scenes.

  • Language structure: Scenic programs use standard imperative constructs, but restrict conditional branching from depending on random variables to support efficient sampling.The language includes conditionals, loops, functions, and methods while imposing this domain-specific restriction.
  • Data types: Scenic provides booleans, scalars, vectors, headings, vector fields, regions, and objects as primitive data types for representing spatial scenes.Scalars can be sampled from distributions; vectors use meter-based coordinates, while headings represent orientations.
  • Data types: Objects inherit properties through Point, OrientedPoint, and Object classes, with default values reevaluated independently whenever objects are created.Object adds bounding-box properties such as width and height to position and heading.
  • Expressions: Geometric operators express visibility, local-coordinate offsets, relative placement, and visible regions using points, oriented points, vector fields, and regions.For example, X relative to Y interprets X as an offset in Y’s local coordinate system, while visible region refers to the portion visible from the ego object.
  • Expressions: Distributions evaluate to samples, while resample(D) supports independent repeated sampling from a distribution.Scenic distinguishes sampled distributions from deterministic geometric expressions, including vector construction such as x @ x.
  • Specifiers: Specifiers map dependency properties to one or more object properties and are combined in dependency order, with optional and default values resolving remaining properties.Scenic rejects cycles or duplicate property specifications and allows optional heading values to be overridden explicitly.

5 Semantics and Scene Generation

Scenic programs define distributions over scenes through random choices and requirements, and Scenic exploits geometric structure to sample valid scenes more efficiently. Its implementation combines domain-specific pruning with rejection sampling while preserving the desired distribution.

  • Semantics of Scenic: A Scenic program produces a scene assigning values to object properties and global parameters, with its semantics defined as a distribution over possible scenes.Random choices during interpretation give runs probabilities, and each run violating a require statement is discarded and the remaining probabilities normalized.
  • Semantics of Scenic: Soft requirements are checked probabilistically, ensuring their conditions hold with at least the specified probability in the induced distribution.The statement require[p] B behaves as require B with probability p and as a no-op otherwise.
  • Semantics of Scenic: Scenic object definitions combine specifiers and inherited defaults, raising errors for duplicate specifications and resolving properties through dependency-aware construction.Default, optional, and non-optional specifiers are assigned precedence when completing an object definition.
  • Domain-Specific Sampling Techniques: Scenic’s geometric constraints and restriction against conditional random branching enable sampling methods inspired by robotic path planning and configuration-space reasoning.These methods prune regions where objects cannot fit or satisfy containment, orientation, or size constraints.
  • Domain-Specific Sampling Techniques: The pruning techniques restrict candidate regions using containment erosion, orientation-based dilation, and size-based elimination of narrow polygons.The methods remove portions of the sample space that cannot satisfy the scenario’s geometric requirements.
  • Scene Generation: Rejection sampling then generates scenes until all requirements are satisfied, producing the exact target distribution but potentially requiring many iterations.In experiments, reasonable scenarios needed at most several hundred iterations, and pruning reduced the required samples by a factor of 3 or more.

6 Experiments

The experiments use Scenic with GTAV to test, train, and debug an autonomous-driving car detector under specified scene conditions. Specialized scenarios expose weaknesses, generate difficult training examples, and improve performance without increasing training-set size.

  • Testing under Different Conditions: Scenic scenarios generated GTAV scenes and test sets for evaluating car detection under specified operation regimes.The setup interfaced Scenic with GTAV and measured detection using precision and recall.
  • Testing under Different Conditions: 83.1%, 85.7%, and 72.8% precision was obtained on Tgeneric, Tgood, and Tbad, respectively, while recall was 92.6%, 94.3%, and 92.8%.The specialized sets fixed time and weather to represent good and bad road conditions.
  • Training on Rare Events: Scenic generated 250 overlapping-car training images and 200 test images to represent a difficult occlusion case.One car was placed behind another from the camera view while remaining at least partially visible.
  • Training on Rare Events: Replacing 5% of Xmatrix with Scenic overlap images significantly improved Toverlap performance without reducing performance on Tmatrix.The overall training-set size stayed constant, and results were averaged over eight runs.
  • Debugging Failures: Variant scenarios identified car model, color, angle, and proximity to the camera as relevant factors in a misclassification.The experiments found that model and color mattered most initially, while later comparisons highlighted angle and closeness to the camera.
  • Debugging Failures: Replacing training data with specialized near-camera scenes reduced false positives, whereas classical augmentation hurt performance.The authors interpret scenario generalization as broadening individual failures enough to reduce overfitting during retraining.

7 Related Work

Related work covers synthetic data generation, domain adaptation, model-based test generation, and probabilistic programming. Scenic differs by combining language-based scenario control with geometry-aware syntax and specialized sampling for data generation.

  • Data Generation and Testing for ML: Prior synthetic-data and domain-adaptation methods have supported training models for tasks including detection and segmentation.The cited applications include text recognition, grasping, autonomous driving, 3D object detection, pedestrian detection, and semantic segmentation.
  • Data Generation and Testing for ML: Scenic differs from GAN-based augmentation because it does not require an initial training set or pretrained model and supports declarative constraints.The comparison is made against GANs used to augment training sets.
  • Model-Based Test Generation: Example-based and grammar-based model-driven generation offer different trade-offs between ease of use and fine-grained control.The related approaches include mutational and generative fuzz testing, example-based scene synthesis, and procedural generation from shape grammars.
  • Probabilistic Programming Languages: Unlike typical probabilistic programming applications focused on inference, Scenic uses probabilistic programming for test generation.The authors state that probabilistic programming languages had not previously been used for this purpose to their knowledge.
  • Probabilistic Programming Languages: Scenic’s domain-specific design represents complex scenarios concisely and enables specialized sampling techniques.This is presented as its main advantage over closely related object-and-scene generative languages such as Quicksand and Picture.

8 Conclusion

The paper concludes that Scenic specifies distributions over physical scenes and supports specialized synthetic-data generation for perception-system analysis. Its case study applies this to testing, training, and debugging, while future work extends domains, simulators, scene dimensionality, and dynamics.

  • 8 Conclusion: Scenic specifies distributions over configurations of physical objects and agents.The language is presented as a probabilistic programming language for scene specification.
  • 8 Conclusion: Scenic generated specialized test sets, emphasized difficult training cases, and generalized individual failures into broader retraining scenarios.These applications cover testing, training, and debugging of perception systems.
  • 8 Conclusion: Training on Scenic-generated hard cases significantly boosted a car detector’s performance beyond prior synthetic-data methods at fixed training-set size.The comparison concerns prior methods not based on probabilistic programming languages.
  • 8 Conclusion: Future work targets additional domains, applications, and simulators, plus user-defined specifiers, 3D scenes, and dynamic scenarios.Examples include VerifAI, X-Plane, CARLA, and analysis of complex dynamic behaviors.
  • Appendix Scenario Examples: The autonomous-car case study uses GTAV-specific libraries and Scenic code defining roads, cars, headings, colors, and default properties.The Car abstraction derives heading from the road direction plus roadDeviation and provides convenience subclasses such as EgoCar.

A.4 A Badly-Parked Car

The badly-parked-car scenario generates a car near the curb with a controlled deviation from the road direction. Related examples constrain car placement, visibility, and camera-facing orientation in generated scenes.

  • A.4 A Badly-Parked Car: The badly-parked-car scenario places a car near the curb with a randomly signed 10–20° deviation from the road direction.The car is positioned left of a spot by 0.5 and faces at the sampled badAngle relative to roadDirection.
  • Rendered Scenes: The figures show generated scenes for badly parked cars and cars facing roughly toward the camera.These images illustrate the corresponding Scenic scenario outputs.
  • Related Scenario Examples: A separate scenario creates a car 20–40 m ahead and requires it to face within 15° of the camera.The 30° view cone implies the camera-facing requirement.

A.6 Adding Noise to a Scene

Scenic’s mutation feature automatically adds noise to an otherwise completely specified scenario, while removing the mutation reproduces the original scene shown in Fig. 14. The resulting noisy scenes are shown in Fig. 13 and were used in the Sec. 6.4 experiment.

  • Scenic’s mutation feature automatically adds noise to an otherwise completely specified scenario.Removing the mutate statement exactly reproduces the original scene.
  • The scenario specifies noon, EXTRASUNNY weather, and an ego-car position at -628.7878 @ -540.6067.
  • The original misclassified image used in Sec. 6.4 is shown in Fig. 14.
  • The noisy scenes generated from this scenario are shown in Fig. 13.

A.7 Two Cars

The appendix illustrates Scenic scenarios for structured traffic scenes and explains how their probabilistic, constraint-based semantics produces valid scene distributions.

  • Two Cars: A generic two-car scenario samples each car’s road deviation independently from a shared wiggle range.
  • Two Cars: A partially overlapping two-car scenario places one car beyond another with a sampled lateral offset and gap.
  • Four Cars: A four-car scenario combines sampled road deviations with rain and midnight settings to represent poor driving conditions.
  • Structured Traffic: Helper functions let Scenic construct structured configurations such as car platoons and three lanes of traffic with specified gaps.The examples use createPlatoonAt and carAheadOfCar to build larger traffic structures.
  • Other Domain: Scenic also models rubble-field workspaces in Webots, including a bottleneck that forces a planner to consider climbing over a rock.
  • Semantics: A Scenic program induces a distribution over scenes through probabilistic execution, while hard requirements condition that distribution on satisfied constraints.

D Additional Experiments

Additional experiments compare the generic two-car Scenic scenario against a baseline in an experiment analogous to the earlier partially overlapping-car study.

  • The experiment uses the generic two-car Scenic scenario as a baseline.
  • It is designed to be analogous to the experiment in Sec. 6.3.

Additional Details on Experimental Setup

The experimental setup approximates the GTAV road map and evaluates object-detection models using IoU-based detection, average precision, and average recall.

  • Map Construction: The GTAV map is approximated from a bird’s-eye schematic by extracting roads, detecting curbs, and estimating nominal traffic direction.
  • Detection Metrics: Intersection over Union measures overlap between predicted and ground-truth bounding boxes, with IoU > 0.5 counted as a detection.
  • Detection Metrics: Average precision and average recall evaluate model performance across the collection of test images.

Overlapping Scenario Experiments

The experiments test whether emphasizing overlapping-car scenes in training improves performance on difficult overlap cases without sacrificing performance on generic two-car images.

  • Training-set construction: Overlapping-car images are highly untypical of generic two-car images because their average overlap is much higher.The comparison used 1,000 generated images for each training set, with overlap distributions shown on a log scale.
  • Experimental procedure: The experiment averaged results over 8 training runs and varied mixture composition through independent random choices of training examples.Evaluation used the maximum precision and recall from 4,000 through 5,000 steps in increments of 250.
  • Baseline performance: Models trained only on generic two-car images have high precision and recall on Ttwocar but drastically worse recall on Toverlap.
  • Mixture results: Devoting more training data to overlapping cars substantially improves recall on Toverlap while leaving performance on Ttwocar essentially unchanged.
  • Conclusion: Using Scenic to increase representation of overlapping cars improves network performance on difficult corner cases.
Loading 1809.09310v2…