ORRB -- OpenAI Remote Rendering Backend

• Visual domain randomization is presented as a simple, fast way to generate abundant data without relying on photorealism or real training data.Source: §1 · p. 2
• ORRB exposes appearance controls and supports randomization at different levels of granularity.Source: §1 · p. 2
• Its modular architecture supports rapid prototyping of new randomization and augmentation techniques.Source: §1 · p. 2
• The system targets large-scale, high-throughput batch rendering in distributed, headless environments.Source: §1 · p. 2
• ORRB is designed for seamless configuration with machine-learning libraries.Source: §1 · p. 2
• ORRB successfully trained Dactyl, a dexterous robotic hand, purely from simulated, synthetic data.Source: §1 · p. 2

• Game environments have supported major machine-learning results in Atari [8], Dota 2, and Starcraft II.Source: §2 · p. 2
• Unity and Unreal have been used to build autonomous drone [12] and car simulations.Source: §2 · p. 2
• Custom environments based on Doom [7], Quake III Arena, Minecraft [1], and Unity [6] [5] provide testbeds for basic machine-learning research.Source: §2 · p. 2

• Modern game engines combine advanced graphics and physical simulation with broad hardware and operating-system support.Source: §3 · p. 2
• Conventional game engines target 30–60 images per second at approximately 1920 × 1080 resolution and preserve temporal coherence for human perception.Source: §3 · p. 2
• Robotics reinforcement-learning workloads prioritize total rendering throughput over low latency and can exploit batch parallelism.Source: §3 · p. 3
• At low image resolutions, rendering bottlenecks shift from GPU rasterization and shading toward CPU scene setup.Source: §3 · p. 3
• Single Scene Multiple States shares geometry, textures, materials, and kinematic descriptions while separating randomized parameters and lightweight state.Source: §3 · p. 3
• ORRB's design effort focuses on adapting Unity's interactive operation for high-throughput, batch, SSMS rendering.Source: §3 · p. 3

4.1 Renderer as a Service

• ORRB loads a scene XML and assets, reads transformation and randomization configuration, maps state to the kinematic hierarchy, and awaits update and batch-render requests.Source: §4.1 · p. 3

• MuJoCo XML scene descriptions specify bodies, geometry, joints, cameras, sites, materials, inheritance, includes, and STL mesh assets.Source: §4.2 · p. 3
• Human-readable YAML protocol buffers describe transformations and randomizations, while binary protocol buffers update renderer configuration over RPC.Source: §4.2 · p. 3
• On-the-fly configuration updates during training make adaptive randomization and related techniques possible.Source: §4.2 · p. 3

4.3 Component Manager

• The Component Manager instantiates, runs, and updates modular components attached to scene entities or kinematic subtrees, with filtering and auxiliary outputs.Source: §4.3 · p. 3
• Randomization is reproducible through per-batch or per-state seeds and consistent execution order, although OpenGL and GPU techniques leave low-level pixel discrepancies.Source: §4.3 · pp. 3–4

4.3.1 Randomizer Components

• Ready-made randomizers control materials, cameras, joints, lights, and post-processing parameters across configurable ranges and modes.Source: §4.3.1 · p. 4

4.3.2 Utility Components

• Utility components provide tracking outputs, entity transforms, visibility control, target-oriented views, and programmatic light setup.Source: §4.3.2 · p. 5

4.3.3 Camera Calibrator

• The Camera Calibrator interactively aligns scene cameras with physical footage by adjusting position, rotation, and viewing angle.Source: §4.3.3 · p. 5

• Preallocated render and destination texture pools, bulk DMA, and asynchronous GPU finalization reduce stalls and support pipelined batch capture.Source: §4.4 · p. 5
• Round-robin render-texture reuse allows consequent frames without data dependencies to be finalized asynchronously on the GPU.Source: §4.4 · p. 5
• The pipeline supports RGB images with optional transparency, segmentation maps, depth maps, and screen-space normal-vector maps.Source: §4.4 · p. 5

• Any language with gRPC support can communicate with the ORRB render service.Source: §4.5 · p. 5
• Python utilities provide server lifecycle management and a parallel batch render executor.Source: §4.5 · p. 5
• A Python suite includes demos, benchmarks, and unit tests for ORRB.Source: §4.5 · p. 5

• ORRB runs on OSX and Ubuntu local machines and on Kubernetes, Microsoft Azure, and Google Cloud Platform virtual machines.Source: §4.6 · p. 5
• True headless rendering is not supported in Unity, so ORRB starts an Xorg instance on Nvidia virtual devices for datacenter rendering.Source: §4.6 · p. 5
• Multiple virtual screens configured in xorg.conf control multi-GPU rendering efficiently.Source: §4.6 · p. 5

5 Performance

• The benchmark uses batches of 64 samples, producing 192 images per call at 200 × 200-pixel resolution with 3 × 8-bit RGB color depth.Source: §5 · p. 5
• The evaluation scene contains a Shadow Dexterous Hand holding a colorful block, three cameras, three soft-shadow spot lights, and multiple visual randomizers.Source: §5 · p. 5
• Measurements run on a Google Cloud Platform n1-standard-96 VM with 96 virtual cores and 8 Nvidia V100 GPUs using Ubuntu 16.04, Nvidia 410.48 drivers, and Unity 2018.3.Source: §5 · pp. 5–6
• ORRB scales to 3 GPUs by using multiple separate client processes, while further scaling is limited by insufficient CPU processing power.A single-threaded Python client throttles performance as render-server counts increase; approximately one client process per 8 to 12 render-server instances counters this bottleneck.Source: §5 · p. 6, Table 2 · p. 6, Table 3 · p. 6
• 16 to 24 render servers running in parallel are required to fully saturate a V100 GPU.Source: §5 · p. 6