PaperlayerView exact revision on arXiv

Source-linked AI summary

Infinite Photorealistic Worlds using Procedural Generation

Alexander Raistrick, Lahav Lipson, Zeyu Ma, Lingjie Mei, Mingzhe Wang, Yiming Zuo, Karhan Kayan, Hongyu Wen, Beining Han, Yihan Wang, Alejandro Newell, Hei Law, Ankit Goyal, Kaiyu Yang, Jia Deng

arXiv:2306.09310v2cs.CV

TL;DR

Computer vision needs more high-quality labeled data, especially for 3D vision and natural-world content. Infinigen addresses this gap with a procedural generator that creates photorealistic 3D scenes from scratch, offering broad natural-world coverage and real geometry. Experiments suggest its synthetic data is useful, particularly for bridging natural-object coverage gaps.

  • Problem

    High-quality labeled data is limited for important vision tasks, especially 3D vision, while existing synthetic datasets provide narrow coverage of natural-world objects and scenes.

  • Method

    Infinigen procedurally generates photorealistic 3D natural-world scenes, including fine geometric and textural details, using randomized rules and Blender-based procedural tools.

  • Results

    Experiments suggest Infinigen data is useful, especially for bridging gaps in the coverage of natural objects.

  • Takeaways & Limitations

    Infinigen provides broad, customizable synthetic data with accurate geometric ground truth for computer vision tasks including 3D reconstruction.

Abstract

from arXiv · show

We introduce Infinigen, a procedural generator of photorealistic 3D scenes of the natural world. Infinigen is entirely procedural: every asset, from shape to texture, is generated from scratch via randomized mathematical rules, using no external source and allowing infinite variation and composition. Infinigen offers broad coverage of objects and scenes in the natural world including plants, animals, terrains, and natural phenomena such as fire, cloud, rain, and snow. Infinigen can be used to generate unlimited, diverse training data for a wide range of computer vision tasks including object detection, semantic segmentation, optical flow, and 3D reconstruction. We expect Infinigen to be a useful resource for computer vision research and beyond. Please visit https://infinigen.org for videos, code and pre-generated data.

1. Introduction

Infinigen addresses limited, especially natural-world, synthetic-data coverage by procedurally generating photorealistic 3D scenes with infinite variation, broad content, real geometry, and open access.

  • High-quality labeled data remains scarce for important vision tasks, particularly 3D vision because accurate real-image 3D ground truth is difficult to acquire.
  • Existing synthetic datasets often cover narrow object and shape categories, motivating broader coverage of natural-world objects and scenes.
  • Infinigen generates infinitely many distinct shapes, textures, materials, and scene compositions from randomized mathematical rules without external assets.
  • Its natural-world coverage includes plants, animals, terrains, and phenomena such as fire, clouds, rain, and snow.
  • Infinigen procedurally generates coarse and fine geometric and textural details, while keeping all geometric details real for accurate 3D reconstruction ground truth.
  • Built on Blender and released under the BSD license, Infinigen provides unlimited customizable assets, renders, and task-specific ground truth for many vision tasks.
  • Experiments suggest Infinigen data is useful, especially for bridging gaps in natural-object coverage, and the paper analyzes generation-pipeline computational costs.

2. Related Work

Existing synthetic-data work is concentrated in built environments and often depends on static asset libraries or limited procedural components. Infinigen targets underrepresented natural objects and scenes with fully procedural, accessible generation.

  • Infinigen generates natural-world objects and scenes procedurally rather than relying on external assets, producing infinite original assets and compositions.
  • Existing synthetic datasets predominantly cover built environments, especially indoor and urban scenes, while natural objects and scenes receive limited coverage.
  • Many existing systems use static external or in-house asset libraries, making procedural variation narrower and potentially easier to overfit.
  • Infinigen is maximally accessible through free code under the BSD license, whereas many existing works restrict assets or do not release code.

3. Method

Infinigen combines Blender-based procedural rules, artist-friendly authoring tools, detailed geometric modeling, natural-scene generators, and task-oriented rendering utilities. Its pipeline produces photorealistic scenes with controllable variation and dense ground truth.

  • Procedural framework: Procedural generation models structures through generalized rules, while Blender provides primitives, node graphs, and Python automation for implementing those rules.
  • Procedural framework: The Node Transpiler converts Blender node graphs into Python code, enabling more general procedural rules and contributions from non-programmers.
  • Natural-world generators: Generators cover terrains, plants, underwater objects, creatures, materials, and natural phenomena using randomized parameters, simulations, and specialized asset structures.
  • Geometric detail: Infinigen models accurate geometry directly, excluding bump mapping, Phong interpolation, and image-texture shortcuts that cannot be represented as meshes.
  • Rendering: Dynamic resolution scaling evaluates assets so each rendered face is below 1px, preserving final-image appearance across camera distances while varying mesh resolution.
  • Rendering: The system renders procedural scenes and extracts depth, normals, occlusion boundaries, optical flow, categories, bounding boxes, and instance segmentation from geometry and metadata.

4. Experiments

The experiment tests whether Infinigen-generated stereo data can train a model that generalizes to natural photographs and the Middlebury benchmark.

  • The authors generated 30K stereo image pairs with ground truth, trained RAFT-Stereo from scratch, and evaluated it on Middlebury validation and test sets.

5. Contributions & Acknowledgements

The contribution section records the authors’ roles and support, while supplementary evidence presents randomly selected samples and directs readers to additional resources.

  • Alexander Raistrick, Lahav Lipson, and Zeyu Ma contributed equally, with responsibilities spanning coordination, creatures, training, annotations, rendering, terrain, and camera selection.
  • The work acknowledges Zachary Teed and support from the Office of Naval Research and National Science Foundation.
  • The supplementary material includes 576 randomly selected JPEG images and states that the system can generate infinitely many 1080P PNG images with accompanying ground truth.
  • Additional terrain samples are grouped by scene type, and the project website provides videos, code, and extended high-resolution random samples.

B. Experiments

Infinigen is evaluated for stereo matching on synthetic natural scenes, real natural images, and the indoor Middlebury benchmark. Models trained on Infinigen perform strongly on natural scenes but struggle more with exclusively artificial indoor content.

  • 30K image pairs train RAFT-Stereo from scratch, with comparisons against models trained on other synthetic datasets.The training uses 200k steps and the same hyperparameters as prior work.
  • Infinigen-trained models perform well zero-shot on real natural-scene images and qualitatively better than models trained on existing datasets.Evaluation is qualitative because 3D ground truth for real natural scenes is difficult to obtain.
  • On Middlebury, the model struggles with exclusively artificial objects but performs well on the natural-object Jadeplant image.The benchmark contains cluttered indoor scenes with textureless surfaces and thin structures.
  • 400 independent Infinigen evaluation scenes contain no assets shared with the training scenes.
  • Infinigen-trained models achieve significantly lower error than models trained on other datasets on independent synthetic natural scenes.The authors interpret this as evidence that Infinigen provides a useful supplement with a distribution distinct from existing datasets.

C.1. Image Rendering

Infinigen renders photorealistic images with Blender’s physically based Cycles renderer and provides broad visual samples. The rendering setup uses high resolution and extensive per-pixel sampling, while the examples demonstrate randomized, non-cherry-picked outputs.

  • Cycles uses physically based path tracing to simulate diffuse and specular reflection, transparent refraction, and volumetric effects.Images are rendered at 1920 × 1080 with 10,000 random samples per pixel.
  • The system produces 576 randomly generated, non-cherry-picked images in each displayed sample set.The samples are compressed due to space constraints.
  • Infinigen helps models generalize to images with natural objects, including the Middlebury Jadeplant example.Models trained exclusively on natural objects can generalize less well on indoor datasets containing planar or textureless surfaces.

C.2. Ground Truth Extraction

Infinigen extracts dense ground truth directly from mesh geometry rather than relying on Blender render passes. This supports accurate, configurable annotations and extensions for task-specific labels.

  • Prior datasets rely on Blender render passes that can be incorrect for translucent surfaces, volumetric effects, blur, or sampling noise.
  • OpenGL code extracts surface normals, depth, segmentation masks, and occlusion boundaries directly from mesh geometry.The method provides exact occlusion boundaries, which Blender does not natively produce.
  • Surface normals are computed by fitting a plane to the local depth map, with configurable plane size and occlusion-aware sampling.Planes with fewer than 3 samples are marked invalid.
  • Because the system is controllable and open-source, users can extend it to generate task-specific ground truth beyond the provided annotations.

C.3. Runtime

Generating Infinigen image pairs is computationally intensive but can be accelerated with additional CPUs and faster processors. The benchmark reports average wall time and memory usage across many trials.

  • 3.5 hours is the average wall time to produce a pair of 1080p images across 1000 independent trials.About one hour uses a GPU for rendering specifically.
  • The system uses about 24Gb of memory on average.More CPUs per image pair and faster CPUs can significantly reduce wall time.

D. Interpretable Degrees of Freedom

Infinigen’s procedural system combines interpretable parameter counting with view-consistent scene generation and a spherical terrain-meshing method designed to allocate detail where cameras need it.

  • The parameter count excludes trivial transformations and treats meaningful relative dimensions, such as leg-to-body ratios, as parameters.
  • 1070 distinct interpretable parameters span 182 procedural asset generators, providing a conservative estimate of Infinigen’s expressive capacity.The count includes only human-understandable parameters that can be randomized to produce noticeably different, photorealistic assets.
  • Scenes are expansive, view-consistent worlds from which users can select arbitrary camera poses or pose sequences for video and multi-view data.Scene composition begins with a sampled ground surface and Poisson-disc placement to reduce asset intersections.
  • Spherical Marching Cubes generates dense terrain geometry near the camera while avoiding unnecessary resolution in distant regions.It adapts marching cubes to spherical coordinates, improving performance over fixed-interval sampling that misallocates geometry with depth.
  • The terrain pipeline searches visible blocks with low-resolution SDF queries before producing a finalized mesh from dense pixel-size queries.The process uses spherical blocks, visible-block refinement, and high-resolution marching cubes, with iterative passes available to prevent holes.

G. Asset Implementation Details

Infinigen implements natural assets and materials procedurally using SDF-based terrain composition, shader and geometry generators, and detailed real-geometry constructions for plants and animals.

  • Materials combine procedural shaders with local geometry templates to generate realistic appearance and fine geometric detail.Shaders control color, roughness, specularity, metallicity, subsurface scattering, and translucence, while geometry templates displace mesh vertices.
  • Terrain materials use combinations of Perlin noise, wave textures, layered masks, and joint color-displacement-roughness variation.Mud generates puddles and slick ground, while Sandstone models layered sedimentary rock using noise and world-height arithmetic.
  • The system provides specialized procedural materials for lava, fire, smoke, water, plants, fish, birds, mammals, insects, bone, slime, and metal.These generators combine domain-specific textures, volumetric or physically based shaders, and geometric or color variation.
  • Terrain elements are represented with signed distance functions that support arbitrary precision, parallel evaluation, and flexible Boolean composition.SDF minimum and maximum operations allow elements such as cellular rocks to be composed with mountains.
  • Compared with common tree generators, Infinigen creates leaves and bark as dense real geometry, supporting high-quality segmentation and depth ground truth.

G.3.6 Corals

Infinigen generates diverse coral forms through multiple procedural templates, including reaction-diffusion, differential-growth, tree-skeleton, extrusion, and crystal-growth methods.

  • Eight coral templates cover distinct forms such as brain, honeycomb, leather, bush, twig, tube, table, and cauliflower corals.The library is designed to generate different classes of corals for underwater scenes.
  • Leather and table corals use differential growth, varying force parameters and stopping conditions to create folded or flat boundary forms.Leather coral grows until 1k faces, whereas table coral stops after 400 faces.
  • Brain and honeycomb corals use Gray-Scott reaction-diffusion with parameter choices that produce grooves or honeycomb-shaped holes.The simulated field is mapped onto geometry or used to displace a mesh surface.
  • Bush and twig corals reuse a tree-skeleton generator with different branch configurations to control directionality and height.Twig coral uses parameters that make it more low-lying and less directional than bush coral.
  • Tube coral is built from a deformed icosphere’s dual mesh, repeated extrusion, and solidification to form hollow tubes.
  • Coral bodies receive additional high-frequency surface noise and distributed tentacles generated from the tree system.Tentacles are placed on selected regions such as top-facing or outermost surfaces.

G.3.7 Other sea plants

Infinigen procedurally generates diverse marine and terrestrial natural assets by modeling their geometry, materials, and spatial distribution with specialized rules. Its creature system further supports realistic templates, random part combinations, and genome interpolation.

  • Other sea plants: Kelps and seaweeds are generated as distinct marine algae assets, with both responding to the scene’s global oceanic current.Kelps use procedurally shaped leaves and stems, while seaweeds use differential growth and current-directed bending.
  • Other sea plants: Moss, lichen, slime mold, and pine needles are procedurally distributed over surfaces using geometry, textures, growth paths, or terrain noise.These generators model both asset form and placement conditions, such as boulder orientation, shortest paths, and height-dependent needle colors.
  • Other sea plants: Mollusk generators cover multiple snail and shell classes by modeling shared growth behavior and class-specific shell or body shapes.Snails include conch, auger, volute, and nautilus forms; shells include scallop, clam, and mussel forms.
  • Other sea plants: Mollusk materials use surface coordinates aligned with growth direction to produce changing texture patterns.The U coordinate follows growth, while V runs orthogonally along the relevant surface boundary.
  • Creature generators: Creature generation represents each animal as a parameter tree whose nodes define parts and whose edges define attachments, supporting realistic templates and diverse recombinations.The system includes realistic Carnivore, Herbivore, Bird, Beetle, and Fish templates, while modular parts can be randomly combined or interpolated.

G.6.4 Creature Parts

Infinigen’s creature-part system combines procedural shape parameterization, reusable node-graph components, and extensive natural variation. The surrounding system provides exact geometric ground truth, dynamic-resolution meshes, and broad asset and scene coverage.

  • Creature Parts: NURBS provide a parameterized basis for many creature body and head templates, with closed surfaces formed from controlled knot vectors and handles.The representation supports sharp edges through densely spaced or coincident handles rather than directly randomizing an unstructured high-dimensional handle space.
  • Creature Parts: Creature parts include procedurally varied horns, hooves, and beaks, each exposing shape parameters and multiple animal-specific parameter sets.Horns use spiral geometry, hooves use NURBS, and beaks use paired upper and lower mandible models with adjustable curvature and bulging.
  • Creature Parts: Node-graph templates form additional creature parts from reusable parameterized components, including tubes for muscular legs, arms, and heads.The library is designed to ease construction of parts such as tiger heads and quadruped legs.
  • Generated scenes: 144 randomly generated terrain images demonstrate the system’s terrain output, while comparison panels include Infinigen and several existing datasets.The supplied comparison labels include Li et al., Sceneflow, TartanAir, and Falling Things.
  • Evaluation: RAFT-Stereo trained with Infinigen data generalizes to natural stereo photographs and Middlebury images containing natural objects.The qualitative figures use natural stereo photographs and Plant and Australia Middlebury test images.
  • Resolution: Dynamic-resolution scaling adjusts mesh face sizes with camera distance so rendered faces are approximately one pixel across.The figures relate face area, camera distance, and projected pixel area to the scaling strategy.
  • Ground truth: Ground truth is computed directly from underlying geometry, avoiding noisy render-pass depth for volumetric effects such as water, fog, smoke, and semitransparent objects.The system also provides exact geometric outputs including depth, surface normals, occlusion boundaries, and instance segmentation.
  • Asset coverage: The system includes procedural variation across bushes, cacti, ferns, corals, kelps, seaweeds, boulders, mollusks, and fantastical creature combinations.The creature examples are described as a random, non-cherry-picked sample with diverse body and limb plans and realistic materials.
Loading 2306.09310v2…