PaperlayerView exact revision on arXiv

Source-linked AI summary

Building Pretraining Data for World Models: An Unreal Engine-Based Pipeline for Action-Conditioned Video Generation

Haoyu Wang, Songchun Zhang, Haoran Li, Haoyang Huang, Zeyue Xue, Nan Duan

arXiv:2609.03557v1cs.CVcs.GR

TL;DR

Action-conditioned video models need large-scale video paired with temporally aligned control signals, which ordinary video rarely provides directly. This paper presents a two-stage Unreal Engine pipeline and distributed production system for generating and curating synchronized synthetic data, producing thousands of hours across retained levels and characters while exposing limitations of perceptual filtering.

  • Problem

    Ordinary real-world video rarely directly observes the actions and camera motion that caused visual changes, complicating construction of temporally aligned action-conditioned training data.

  • Method

    The pipeline runs physics-based trajectory generation in PIE, records per-frame states, and replays them for MRQ offline rendering within a distributed, cache-aware production workflow.

  • Results

    The system produced approximately 2,700 hours of 1080p video and 6,000 hours of 720p video from 429 production levels, 40 characters, and five synchronized camera views.

  • Takeaways & Limitations

    The report demonstrates an upstream synthetic-data production component that supplies synchronized video, action, character, and camera trajectories for world-model pretraining.

  • Takeaways & Limitations

    Perceptual quality proxies remove obvious visual failures but are not necessarily equivalent to world-model utility, and their thresholds have not been calibrated against downstream performance.

Abstract

from arXiv · show

Action-conditioned video models require large-scale visual data paired with control signals that are temporally aligned with the resulting scene transitions. Such supervision is difficult to obtain from ordinary real-world video because the actions that caused each visual change are typically unknown. We present a large-scale synthetic data production pipeline built on Unreal Engine for generating action-conditioned, multi-view video. To accommodate the different execution requirements of real-time physics and high-quality offline rendering, the pipeline executes trajectory generation and final rendering in two stages: Stage I runs real physics in PIE and records per-frame character states, control inputs, and camera states into an intermediate trajectory representation; Stage II replays those trajectories in a new engine process and renders them offline with Movie Render Queue (MRQ). Around this core, we develop a distributed production system with cache-aware task partitioning, node-local slot scheduling, automated scene screening, aesthetic and luminance filtering, partial-output recovery, asynchronous upload, and continuous cluster health monitoring. The production cluster contains 25 servers with eight NVIDIA RTX 5090 GPUs per server. From 2,384 asset packs, 429 levels were retained for production together with a pool of 40 humanoid characters. The pipeline has produced 2,691 hours of 1080p video and 6,076 hours of 720p video. We describe the system architecture, the implementation decisions that emerged from production failures, and the limitations of using perceptual quality proxies for world-model data curation. The pipeline described in this report constitutes the Unreal Engine synthetic-data production component used in EchoWM.

1 Introduction

Action-conditioned world models need temporally aligned video, actions, and camera motion, but ordinary video rarely provides these signals directly. This report presents a scalable Unreal Engine pipeline that generates and curates synchronized synthetic data at cluster scale.

  • Real-time physics execution and high-quality offline rendering are separated into trajectory generation in PIE and replay-based MRQ rendering.Stage I records motion and camera states; Stage II replays them in a separate Unreal Engine process.
  • 2,691 hours of 1080p video and 6,076 hours of 720p video were produced from 429 retained levels and 40 humanoid characters.The system runs on 25 nodes with eight NVIDIA RTX 5090 GPUs per node and provides five synchronized camera views with frame-aligned metadata.
  • The pipeline generates synchronized multi-view video with frame-level actions, character states, and camera poses from heterogeneous Unreal Engine environments.
  • The distributed architecture combines persistent scene pools, node-local rendering slots, cache-aware scheduling, and multi-process GPU execution across a 200-GPU cluster.
  • The production workflow integrates asset screening, aesthetic and luminance filtering, partial-output recovery, asynchronous upload, failure snapshots, and cluster monitoring.
  • The report focuses on synthetic-data production infrastructure rather than introducing a new world-model architecture.

2 Related Work

Interactive world models increasingly depend on large-scale data with reliable actions, camera geometry, and temporal alignment. Related systems combine synthetic environments, gameplay, Internet video, and spatial annotation, while this report focuses on synthetic Unreal Engine production infrastructure.

  • Interactive world-model progress increases the need for video data with reliable camera geometry, actions, and temporal alignment.
  • Existing data engines combine synthetic environments, gameplay, and Internet video, while other pipelines derive camera supervision and spatial annotations from public or in-the-wild video.
  • This report treats data construction as production infrastructure, covering asset screening, trajectory generation, offline rendering, distributed scheduling, quality control, and failure recovery.
  • The pipeline operates in a fully synthetic control setting where spatial quantities and actions are available from the engine during generation rather than estimated from rendered video.
  • Third-party synthetic scenes introduce distributional biases and require screening for collision validity, renderability, exposure, texture quality, and scene semantics.

3 Data Generation Overview

The data-generation system pairs character actions with synchronized multi-view observations in synthetic Unreal Engine scenes. It uses physics-based trajectory generation followed by deterministic trajectory replay for high-quality offline rendering.

  • 3.1 Action-Conditioned Multi-View Data: Each sample pairs video and action information while five cameras synchronously observe a character moving through a 3D level.
  • 3.1 Action-Conditioned Multi-View Data: Figure 1 arranges top-down, back, first-person, left-side, and right-side views across different environments to show viewpoint variation and scene diversity.
  • 3.1 Action-Conditioned Multi-View Data: The action space contains nine discrete states: {W,A,S,D,WA,WD,SA,SD,IDLE}.
  • 3.1 Action-Conditioned Multi-View Data: A standard trajectory contains 1,800 frames, corresponding to one minute of video, and produces five synchronized camera streams.
  • 3.2 Two-Stage Generation: Stage I runs actual physics in PIE to resolve inputs, collision, gravity, slopes, and geometry contact while recording character and camera states.
  • 3.2 Two-Stage Generation: Stage II replays recorded states frame by frame in a new Unreal Engine process and renders the sequence with MRQ without resampling actions or running a second physics simulation.
  • 3.2 Two-Stage Generation: The intermediate trajectory file serves as the contract between the stage that determines motion and the stage that determines rendering quality.

4 Assets and Dataset Curation

The pipeline curates heterogeneous Unreal Engine assets through node-local preparation, renderability checks, and automated visual-quality filtering. The retained pool spans diverse scene categories and action states rather than concentrating on one scene or simple forward motion.

  • Pre-Production Screening: Renderability screening rejects levels that crash, lack walkable regions, produce character hovering, or cause substantial geometry interpenetration.Each candidate is tested with five 10-second 540p clips before large-scale production.
  • Pre-Production Screening: Automated visual-quality filtering evaluates individual camera views using aesthetic and luminance scores before production.The aesthetic score targets composition, color, and sharpness, while luminance identifies excessively dark or overexposed outputs.
  • Pre-Production Screening: More than 25% of a level’s sampled outputs must pass the automatic criteria for the level to be retained.The procedure evaluates twenty 10-second clips at 540p, excludes tiled previews from scoring, and computes a per-level pass rate.
  • Dataset Scale and Diversity: The four largest scene categories each represent approximately 19–29% of retained production scenes, avoiding dominance by a single scene type.The categories are Historical & Fantasy, Sci-Fi & Futuristic, Nature & Landscape, and Urban & Architecture.
  • Dataset Scale and Diversity: All nine action states are represented, with diagonal, backward, and lateral movement comprising 46.6% of the current action distribution.This distribution is reported as not being dominated by simple forward motion.

5 System Architecture

The production architecture separates global task assignment from node-local execution and transfer. Persistent scene pools preserve cache locality, while independent rendering slots support concurrent Unreal Engine processes with recovery and cleanup.

  • Distributed Production: A central allocator assigns level pools, while each node scheduler manages local rendering slots and an independent uploader transfers completed outputs.The design keeps high-frequency process management local rather than routing slot status through the central service.
  • Cache-Aware Scheduling: Uniform sampling from the global level list performs poorly because first renders can incur more than ten minutes of scene-specific cold-start costs.Shaders, acceleration structures, and textures create local state that subsequent renders can reuse.
  • Cache-Aware Scheduling: Each node is persistently bound to one level pool, repeatedly rendering that subset while reshuffling after exhaustion.Randomized trajectory initialization still produces distinct data when levels are revisited.
  • Cache-Aware Scheduling: Pool-based scheduling reduces per-scene rendering time to less than half of cold-start time for repeatedly visited scenes.Assignments are persisted so allocator restarts do not discard accumulated node-local caches.
  • Slot Scheduling: Each node runs multiple independent Unreal Engine slot processes, with concurrency tuned empirically to balance utilization against memory pressure and interference.Multiple processes may share a GPU because asset loading, shader compilation, and disk I/O leave individual devices underutilized at times.
  • Slot Lifecycle: For each task, a slot copies scene assets, runs Stage I, launches Stage II only for successful trajectories, removes copied assets, and returns to idle.On-demand copying limits disk usage and project-startup scanning costs; headless slots also use independent virtual displays.

6 Trajectory Generation

Trajectory generation uses a frame-driven Unreal Engine state machine to apply sampled actions within real physics and record realized motion. It then constructs synchronized multi-view camera trajectories while handling initialization, dead ends, and occlusions.

  • Frame-Driven Control: A per-frame callback advances trajectory generation without blocking Unreal Engine’s runtime.Each invocation performs only the operation required by the current state, such as checking initialization, issuing an action, reading transforms, or transitioning.
  • Frame-Driven Control: The Stage I state machine initializes the scene and character, warms up, loops through trajectories with respawn handling, then computes cameras.The documented sequence is initialize, enter runtime, prepare, warm up, trajectory loop, exit runtime, compute cameras, and continue.
  • Physics-Based Trajectories: Runtime simulation resolves collision, gravity, landing, and slope traversal while the pipeline reads the resulting character transform every frame.The controller applies inputs and does not analytically integrate character motion.
  • Action Sampling: Each trajectory samples one of nine discrete actions for a random frame duration, trading locally coherent motion against action-transition frequency.Temporal holding avoids high-frequency reversals and visible jitter caused by independent per-frame resampling.
  • Trajectory Guidance: Point-of-interest guidance broadens spatial coverage by gradually turning the character toward sampled targets without changing action-sampling probabilities.The mechanism addresses limited exploration near the initial position while preserving the action distribution.
  • Trajectory Recovery: Dead-end recovery detects negligible displacement despite nonzero input, attempts an escape rotation, and respawns the character if repeated attempts fail.This provides a defined response to blocked or ineffective motion.
  • Multi-View Cameras: Stage I computes five synchronized camera trajectories, including follow, side, elevated, and first-person views based on character-relative offsets.The external cameras’ desired world positions evolve with character orientation; ray casting and inward correction address occlusion.

7 Offline Rendering

Offline rendering separates realized physics from image generation: Stage I records trajectories, and Stage II replays them through MRQ. This improves rendering quality while requiring explicit transform reconstruction and output-management safeguards.

  • Why MRQ: Direct Stage-I capture left visible aliasing because scripted production could not achieve stable high-quality anti-aliasing.The limitation persisted across multiple control paths and console-variable configurations.
  • Why MRQ: MRQ renders offline with spatial and temporal supersampling, providing better anti-aliasing and temporal quality than the Stage-I capture setup.It can spend more computation per frame because it is not constrained by an interactive frame-time budget.
  • Why MRQ: MRQ supports unattended production through warm-up controls, reusable settings, console-variable overrides, and image-sequence output.The pipeline uses these controls to prioritize image quality and produce stable sequences for encoding.
  • Two-Stage Workflow: The two-stage design records physical motion first, then reconstructs it as a deterministic MRQ timeline because offline rendering is incompatible with Stage-I real-time physics execution.The stages therefore assign realized motion to Stage I and rendering quality to Stage II.
  • Trajectory Replay: Stage II creates one character and five camera objects per trajectory and replays recorded transforms without resampling actions or rerunning physics.Translation, rotation, and scale channels are written explicitly to prevent timeline evaluation from resetting character scale.
  • Rendering Robustness: Level sanitization removes built-in cinematic behavior that could seize the active camera and switch output to an unintended showcase view.The scan occurs before trajectory replay begins.
  • Rendering Robustness: Texture-streaming adjustments favor residency and higher-resolution detail to reduce blurred textures during the initial rendered trajectory.The pipeline increases the texture pool and relaxes normal streaming constraints for offline consistency.
  • Output Management: MRQ outputs separate videos for five camera views plus a tiled preview, then deletes raw image sequences after encoding to limit storage use.Raw sequences require roughly an order of magnitude more storage than encoded videos.

8 Reliability and Operations

Reliability and operations are built around loosely coupled progress tracking, asynchronous output handling, health monitoring, and recovery mechanisms for long-running Unreal Engine production.

  • Progress and scheduling: Completed trajectories are recorded through append-only markers that the scheduler counts, avoiding a bidirectional handshake with potentially crashing Unreal Engine processes.The renderer writes fixed-format progress markers, while the scheduler reads them as a one-way communication channel.
  • Output handling: The uploader transfers completed outputs independently and deletes local copies only after successful transfer, while quiet-period protection prevents encoder races.Failed rendering tasks are packaged with engine logs, exit codes, failure reasons, and slot or GPU information for diagnosis.
  • Operations: Cluster management verifies reachability, container availability, and scheduler state before startup, while diagnosed failures span engine, container, hardware, and storage layers.Representative cases are summarized in Appendix D.
  • Monitoring: Production status combines scheduler logs, object-storage listings, and allocator state, each serving different roles and freshness levels.Scheduler logs are freshest, storage listings are authoritative but delayed by upload queueing, and allocator state reflects task-pool balance.
  • Monitoring and recovery: External monitoring handles GPU-access failures and disk pressure because overloaded or failed production nodes may not monitor themselves reliably.A recurring failure occurs when containers lose functional GPU access despite the host still reporting devices; rebuilding the container restores mappings.

9 Limitations and Discussion

The pipeline balances controllability, diversity, physical plausibility, visual quality, and scalability, while leaving important questions about filtering, action coordinates, and action coverage unresolved.

  • Data-production trade-offs: The system makes production compromises because stronger motion guidance, stricter aesthetic filtering, and additional camera views can trade off diversity, useful environments, or cost.These choices are not intended to be globally optimal for every downstream task.
  • Perceptual quality and downstream utility: Aesthetic and luminance proxies remove obvious visual failures but have not been calibrated against downstream world-model performance.Aggressive filtering may discard useful structural diversity, while permissive filtering may retain samples that interfere with training.
  • Action semantics and future extensions: Character-centric and camera-centric controls define different action coordinates, and the report explores their effects without assuming either representation is intrinsically superior.Camera-centric controls define movement relative to the view, whereas character-centric controls define it relative to character orientation.
  • Limited action repertoire: The current action repertoire centers on locomotion and excludes jumping, running, crawling, attacking, and complex interactions such as climbing.The limitation concerns the present action space rather than a claimed impossibility of adding richer behaviors.

10 Conclusion

The report presents an Unreal Engine pipeline that produces synchronized multi-view action-conditioned video through separate physics and rendering stages, then operates it with cluster-scale infrastructure and monitoring.

  • Conclusion: The pipeline converts heterogeneous 3D assets into synchronized video, action, character, and camera trajectories while preserving engine-generated physical motion.Trajectory generation and final rendering are separated through recorded per-frame states.
  • Conclusion: 25 nodes with eight RTX 5090 GPUs each retained 429 production levels from 2,384 asset packs and supported 40 characters and five synchronized camera views.These figures describe the production configuration and retained asset pool.
  • Conclusion: The system produced approximately 2,700 hours of 1080p video and 6,000 hours of 720p video.The report gives these as approximate production totals.
  • Conclusion: Cluster-scale operation integrates screening, cache-aware scheduling, node-local rendering slots, offline MRQ rendering, asynchronous transfer, recovery, and monitoring.The operational system extends beyond rendering alone.
  • Conclusion: The report emphasizes scalable, operationally robust infrastructure rather than improvements to a particular world-model architecture.It identifies downstream evaluation of scene selection, filtering, action semantics, and trajectory distributions as an important next step.

D Representative Production Failure Cases

Appendix D summarizes representative production failures and emphasizes that their causes are case-specific, requiring diagnostic evidence and validated mitigations.

  • Representative production failure cases: Large-scale production exposed failures across Unreal Engine execution, container state, GPU drivers, and local storage.Table 2 summarizes representative cases diagnosed during production.
  • Representative production failure cases: Similar high-level symptoms can arise from different underlying failures, so each case is reported with its diagnostic evidence and effective mitigation.The mapping from symptom to cause is explicitly treated as case-specific.
Loading 2609.03557v1…