Source-linked AI summary
Meta-Sim: Learning to Generate Synthetic Datasets
Amlan Kar, Aayush Prakash, Ming-Yu Liu, Eric Cameracci, Justin Yuan, Matt Rusiniak, David Acuna, Antonio Torralba, Sanja Fidler
TL;DR
Expensive labeled-data collection motivates methods that can synthesize datasets relevant to downstream tasks. Meta-Sim learns to transform scene-graph attributes and renders the resulting scenes and ground truth through a graphics engine, optionally optimizing downstream task performance. Experiments show improved content generation quality over a human-engineered probabilistic scene grammar, including greater than 99% classification accuracy in a toy setting.
Problem
Expensive labeled-data collection and the synthetic-to-real content gap limit the usefulness of synthetic training data for downstream tasks.
Method
Meta-Sim learns to transform attributes of probabilistic-grammar scene graphs, renders images and ground truth through a graphics engine, and can optimize a downstream Task Network objective.
Results
Meta-Sim generates qualitatively and quantitatively better samples than the baseline across two toy tasks and one real task, achieving greater than 99% classification accuracy in one toy setting.
Takeaways & Limitations
The results support dataset- and task-specific synthetic content generation that adapts probabilistic scene grammars to target real-world distributions.
Abstract
from arXiv · showhide
Training models to high-end performance requires availability of large labeled datasets, which are expensive to get. The goal of our work is to automatically synthesize labeled datasets that are relevant for a downstream task. We propose Meta-Sim, which learns a generative model of synthetic scenes, and obtain images as well as its corresponding ground-truth via a graphics engine. We parametrize our dataset generator with a neural network, which learns to modify attributes of scene graphs obtained from probabilistic scene grammars, so as to minimize the distribution gap between its rendered outputs and target data. If the real dataset comes with a small labeled validation set, we additionally aim to optimize a meta-objective, i.e. downstream task performance. Experiments show that the proposed method can greatly improve content generation quality over a human-engineered probabilistic scene grammar, both qualitatively and quantitatively as measured by performance on a downstream task.
1. Introduction
Meta-Sim addresses the cost and domain-gap barriers of synthetic data by learning to generate labeled scenes whose content matches target real-world imagery and supports downstream performance. It transforms scene-graph attributes while retaining valid scene structure and uses graphics-engine rendering for labels.
- Motivation: Labeled-data collection is costly and time-consuming, while synthetic generation provides ground truth through a graphics engine but suffers from a synthetic-to-real domain gap.The gap limits performance when models trained on synthetic data are applied to real-world data.
- Motivation: The domain gap includes a content gap: synthetic scenes may lack the layout, object diversity, and distributional variation found in target real-world imagery.The target environment and downstream task can change, requiring more adaptable synthetic worlds.
- Approach: Meta-Sim learns a generative model that modifies attributes of scene graphs sampled from probabilistic scene grammars to match desired real-world content distributions.The method changes locations, poses, and other attributes while preserving the grammar-defined scene structure.
- Approach: Meta-Sim can additionally optimize a meta-objective for downstream real-world task performance using a Task Network trained on generated datasets.The learning framework uses approximated gradients through a non-differentiable renderer.
- Results: Meta-Sim improves content generation quality over a human-engineered probabilistic scene grammar in two toy simulators and a real self-driving task.The paper reports qualitative improvements and quantitative gains measured by sim-to-real performance.
2. Related Work
Prior approaches either manually specify probabilistic scene-grammar parameters or focus on the appearance gap between synthetic and real images. Meta-Sim instead learns simulator generation processes and aligns synthetic content with real distributions while optimizing downstream task performance.
- Synthetic Content Generation and Simulation: High-quality synthetic environments can require substantial expert effort, while Domain Randomization cheaply creates diverse scenes whose distributions differ from real-world scenes.Meta-Sim seeks alignment through direct optimization of scene attributes and a downstream-task meta-objective.
- Procedural Modeling: Probabilistic scene grammars generate large and diverse worlds, but their distribution-controlling parameters are manually specified, making the process tedious and error-prone.Manual settings do not guarantee faithful real-world distributions.
- Procedural Modeling: Meta-Sim aims to learn the generative process itself from real data rather than relying only on manually specified generation parameters.This reframes simulator construction as learning from the target data distribution.
- Domain Adaptation: Most synthetic-to-real domain-adaptation methods address the appearance gap, whereas Meta-Sim targets content differences such as layout.The related-work distinction separates style transformation from content-distribution alignment.
- Optimizing Simulators: Compared with simulator optimization using a variational GAN-like objective, Meta-Sim uses MMD and also optimizes a meta-objective for downstream-task suitability.The comparison concerns both the distribution-matching method and the optimization goal.
3. Meta-Sim
Meta-Sim parametrizes synthetic dataset generation by transforming attributes of probabilistic-grammar scene graphs, rendering images and ground truth, and optimizing similarity to real imagery and downstream task performance.
- Meta-Sim defines a generator D(θ) that synthesizes task-specific images X(θ) and corresponding ground truth Y(θ) from real imagery and an optional labeled validation set.The validation set supports an additional meta-objective based on downstream task performance.
- 3.1. Parametrizing Synthetic Scenes: Scene graphs represent 3D worlds hierarchically, and probabilistic scene grammars generate valid structures while sampling their attributes from tunable distributions.The renderer converts transformed scene graphs into images and pixel-level annotations.
- 3.1. Parametrizing Synthetic Scenes: The method preserves the scene-graph structure sampled from a probabilistic grammar while transforming node attributes such as locations, poses, colors, and asset ids.This turns the model into a conditional generative model, called a Distribution Transformer.
- 3.1. Parametrizing Synthetic Scenes: A graph convolutional network predicts transformed attributes per node, while mutable-attribute selection limits which properties the transformer changes.Attributes excluded from the mutable subset remain as generated by the grammar.
- 3.2. Learning Framework: Training proceeds through identity pretraining, distribution matching, and meta-training, with objectives illustrated through forward and backward control flow.The framework uses approximated gradients through a non-differentiable renderer and reduces estimator variance with a moving-average score baseline.
- 3.2.2 Distribution Matching: Meta-Sim uses feature-space MMD, or KID with InceptionV3 features, to bring rendered-image distributions closer to real imagery while favoring training stability.During MMD training, categorical attributes are immutable and continuous predictions are passed as samples; gradients are backpropagated through the renderer using approximations.
4. Experiments
Experiments evaluate Meta-Sim from controlled MNIST distribution shifts to aerial segmentation and 3D driving scenes. Across these settings, Meta-Sim learns transformations that improve target-task performance and synthetic content quality over probabilistic-grammar baselines.
- Experimental setup: Meta-Sim is evaluated on three target datasets and tasks, progressing from controlled experiments to aerial semantic segmentation and driving-scene detection.The controlled experiments probe increasingly complex distribution shifts, while later experiments assess realistic scene adaptation.
- MNIST: Training directly on upright, centered MNIST grammar samples yields just above random performance, whereas Meta-Sim recovers the target transformation and exceeds 99% classification accuracy.The target distributions include 90-degree rotation, with the harder setting also adding translation to the bottom-left corner.
- MNIST: Meta-Sim learns to transform MNIST inputs to replicate target distributions qualitatively in both rotation-only and rotation-plus-translation experiments.The generated examples are reported to corroborate the quantitative classification results.
- Aerial Views (2D): On aerial semantic segmentation, Meta-Sim improves mean IoU by reducing the car-performance gap and improving road performance in a more complex setup.The experiment uses a U-Net task network and compares generated outputs with the probabilistic grammar’s scenes.
- Driving Scenes (3D): Training task networks with Meta-Sim data beats the probabilistic-grammar baseline for KITTI car detection, with fewer false positives and negatives and better recall and ground-truth overlap.Online training during transformation learning and offline training on final generated data produce similar final detection performance.
- Driving Scenes (3D): In driving scenes, Meta-Sim generates cars better aligned with roads and meaningful inter-car distances, while also adapting camera and contextual elements.A reported failure case contains collided cars, and some scenes show overlapping cars that remain visually indistinguishable from a single car.
5. Conclusion
Meta-Sim generates synthetic data that matches real content distributions while optimizing downstream real-task performance. Experiments across two toy and one real task produced quantitatively better and noticeably higher-quality samples than the probabilistic-grammar baseline, within limitations on modelable scenes.
- Meta-Sim transforms probabilistic-grammar scenes to generate synthetic data matching real content distributions while optimizing downstream real-task performance.
- Experiments on two toy and one real task showed quantitatively better and noticeably higher-quality samples than the baseline.
- Meta-Sim relies on valid scene structures from a grammar, limiting the kinds of scenes it can model.Future directions include inferring grammar rules from real images, learning scene structure generation, and introducing multimodality.
6. Appendix
The appendix supplies further methodological details and additional experimental results for Meta-Sim.
- The appendix provides additional details about Meta-Sim.
- The appendix includes more results beyond those presented earlier.
- Its purpose is to supplement the main presentation with methodological and experimental material.
6.1. Grammar and Attribute Sets
The appendix describes probabilistic grammars and the attribute sets used to represent scenes. Meta-Sim’s Distribution Transformer modifies selected scene attributes while preserving grammar-defined scene structure.
- Grammar and Attribute Sets: Probabilistic grammars define scene structure and sample scene graphs that serve as inputs to Meta-Sim.
- Grammar and Attribute Sets: MNIST scenes sample background and digit textures, then sample digit rotation and location for placement.
- Grammar and Attribute Sets: MNIST and Aerial 2D node attributes include class, rotation, locationX, locationY, and size.Classes are represented as one-hot vectors, while geometric attributes are normalized to appropriate ranges.
- Grammar and Attribute Sets: The Aerial 2D grammar samples grass, a road, two constrained-on-road cars, a tree, and a house.
- Grammar and Attribute Sets: Driving Scene 3D attributes include class, rotation, distance, and offset, with only selected graph nodes kept mutable.Distance and offset specify placement along and across the parent spline; the driving grammar uses fixed global weather and sky parameters and omits side-streets and parking lanes.
6.2. Training Details
Training uses a graph-based Distribution Transformer together with distribution and task objectives, with experiment-specific optimization schedules. The appendix also documents iterative task-network training and qualitative driving-scene outcomes and failures.
- Training Details: The Distribution Transformer uses graph convolutional encoder and decoder layers, with experiment-specific intermediate feature sizes.The MNIST and Aerial2D architecture is Encoder (a_in -> 16 -> 10) and Decoder (10 -> 16 -> a_in); 3D driving uses feature sizes 28 and 18.
- Training Details: Successful driving generations align objects, adjust camera position, and move or densify context elements such as buildings and trees.
- Training Details: Failure cases include unresolved dense scenes causing collisions, unrealistic car colours, and cars placed very close to the ego-car.
- Training Details: Task-network training follows each generated-data epoch and is resumed from the previous checkpoint rather than restarted.This iterative procedure is described for the general training setup and for the driving task network.
- Training Details: Meta-Sim-generated driving data yields fewer false positives and negatives than probabilistic-grammar data in Mask-RCNN results.
6.3. Additional Results
Additional results show Meta-Sim can transform object layouts and attributes to improve scene compatibility with target driving imagery, while still producing failures in dense or geometrically complex scenes.
- Aerial 2D: Meta-Sim transforms objects and scene context to improve semantic segmentation on Aerial-2D validation data.The network uses convolutional translation equivariance, including object translations that accommodate the task network’s limited receptive field.
- Driving Scenes (3D): In driving scenes, Meta-Sim rotates cars toward KITTI appearances and places objects to avoid collisions, even in crowded scenes.It can also move cars outside the camera view when required, although some cases fail.
- Driving Scenes (3D): The model slightly changes camera height and moves context elements, usually densifying scenes.
- Failure Cases: Failure cases include unresolved occlusions, intersecting or colliding cars, unrealistic car proximity, and overlapping buildings.The authors note that object size was omitted from Distribution Transformer features, which may contribute to these errors, especially when objects are densely sampled.