Source-linked AI summary
WorldRover: A Scalable Synthetic Video Data Engine for World Exploration with Rich Annotations
Xiaojie Xu, Zhengyuan Lin, Runyi Li, Yihao Liu, Kaipeng Zhang, Yongtao Ge
TL;DR
Existing synthetic video resources rarely preserve rich annotations and controlled viewpoint or appearance changes across the same exploration. WorldRover provides a trajectory-centered rendering engine and WorldRover-10M, a multi-view dataset with task-specific geometric, correspondence, and action annotations. The release contains 21.9M frames across 6,003 sequences from 32 environments.
Problem
Existing resources rarely combine camera motion, metric geometry, temporal correspondence, and action signals on shared frames while preserving controlled variation across observations.
Method
WorldRover treats exploration as a reusable world-space trajectory, rendering it across viewpoints and environmental states while retaining route, timing, scene structure, and frame-aligned annotations.
Results
WorldRover-10M contains 21.9M rendered frames across 6,003 sequences from 32 environments, with task-specific combinations of depth, flow, point tracks, trajectories, and actions.
Takeaways & Limitations
WorldRover provides scalable, controlled supervision for learning coherent explorable-world representations from long-horizon video.
Takeaways & Limitations
The system does not separately control or annotate multiple dynamic scene elements, lacks sensor effects such as noise and rolling shutter, and reports no downstream results.
Abstract
from arXiv · showhide
Learning to generate or reconstruct explorable worlds requires video paired with more than RGB: camera motion, scene geometry, temporal correspondence and, for interactive models, control signals. Real capture can provide some of these signals, but dense geometry and long-range correspondence usually rely on estimation or specialised instrumentation. Rendering provides these quantities directly, yet existing synthetic resources rarely combine them on the same frames while also supporting controlled changes of viewpoint and appearance. We introduce WorldRover, a data engine for generating richly annotated, long-range explorations of artist-built environments. At its core, WorldRover-Engine is an Unreal Engine pipeline that executes and offline-renders minute-scale routes while preserving their full trajectories and scene geometry. The same exploration can be replayed from first-person, third-person, and 360 panoramic cameras under different environmental states. Using WorldRover-Engine, we construct WorldRover-10M, whose sequences pair RGB with metric depth, camera trajectories, and trajectory-derived action signals throughout each exploration. Third-person subsets additionally provide dense optical flow, long-range 2D/3D point tracks with visibility, and a character trajectory distinct from the camera trajectory. The engine can render a traversal from first-person, third-person and 360 panoramic viewpoints, under different environmental states or with a neutral white material, while preserving the route and scene geometry. WorldRover therefore turns long-horizon world exploration into a scalable data-generation problem, providing supervision for models that must build, maintain, and revisit coherent representations of an explorable world.
1 Introduction
WorldRover addresses the difficulty of collecting coherent, richly annotated exploration data by treating an exploration as a reusable world-space trajectory rather than a finalized video. Its engine supports controlled re-observation across viewpoints and environmental states, yielding the WorldRover-10M release with large-scale, task-specific annotations.
- Motivation: Explorable-world models must preserve scene layout and object identity while separating camera motion from world changes, but RGB alone does not expose these causes.These requirements arise in interactive world generation and dynamic 4D reconstruction.
- Problem: Real-video corpora offer broad coverage, yet pose, depth, and correspondence are usually estimated after capture and can drift, fail under occlusion, or require restrictive instrumentation.Rendering exposes these quantities through scene state and image formation, but existing synthetic resources target different goals.
- WorldRover-Engine: WorldRover-Engine prepares artist-built Unreal environments once, generates minute-scale routes, and offline-renders each reusable exploration from multiple cameras or altered environmental states.The same route can be rendered from first-person, third-person, and 360◦ panoramic viewpoints, or as a neutral white model, while retaining shared structure.
- WorldRover-10M: WorldRover-10M contains 6,003 sequences from 32 environments, totaling 21.9M rendered frames, 202.7 hours of video, and 18.7 TB of released data.The name refers to its 10.8M first-person frames; third-person and panoramic subsets add observations of the exploration corpus.
- Annotations: The release pairs observations with subset-specific geometry, motion, correspondence, camera, character, and action annotations under a common schema.WorldRover-Engine also supports streaming annotation extraction and parallel campaign execution over artist-built Unreal environments.
2 WorldRover-Engine
WorldRover-Engine prepares Unreal scenes for headless rendering, generates routes through multiple trajectory strategies, and converts them into smooth camera motion. It then renders configured viewpoints and environmental states through a batch pipeline designed for scalable sequence production.
- Sequence construction: Sparse route samples become dense per-frame trajectories through constant-speed motion, position and heading smoothing, yaw-rate limiting, and occasional view-rotation dwells.The resulting trajectory is rendered from the target subset’s viewpoint, with panoramic runs assigning six cube-face orientations.
- Asset ingest: Asset ingest converts Unreal scene packs into uniform UE 5.5, headless-loadable forms before navigation baking and stylization.Streaming sub-levels or World Partition cells are loaded during ingest so render workers can operate without an editor session.
- Scene stylization: Environmental appearances are authored as scene-side states with fixed lighting, atmospheric effects, optional precipitation or mist, and calibrated fixed exposure.Auto-exposure remains disabled, allowing states such as night to preserve their intended darkness during offline rendering.
- Trajectory generation: Routes are generated with NavMesh planning, reactive exploration, or manual recording, trading automated coverage, scene preparation, and direct route control.NavMesh routes stay on walkable surfaces; reactive agents use short ray casts without a precomputed map; manual recording provides deliberately selected paths and viewpoints.
- Offline rendering: Movie Render Queue performs offline batch rendering with spatial sub-sample accumulation, while SPEAR controls project loading, actor creation, and render-job submission.The batch configuration prioritizes rendering work per frame over interactive latency and reduces aliasing and temporal variation in Lumen illumination relative to the single-sample interactive path.
- Parallel scheduling and recovery: Multiple render instances share each GPU within video-memory limits, reuse instances across sequences, and encode RGB on dedicated hardware to preserve CPU capacity for EXR processing.A watchdog terminates failed instances, supporting recovery during parallel production.
3 WorldRover-10M: Three Camera Viewpoints
WorldRover-10M derives first-person, third-person, and panoramic sequences from a shared recorded route, preserving distinct camera and character trajectories where applicable. The viewpoints support egocentric navigation, articulated-subject observation, and omnidirectional video with metric-preserving rendering.
- First-person: First-person sequences use an eye-height camera whose smoothed, rate-limited heading follows travel direction, making the camera and agent trajectories identical.The trajectory-derived action stream, pose, and imagery share the same motion, matching egocentric video and embodied navigation.
- Third-person: Third-person sequences replay the route with an animated character while a follow camera maintains an offset and independently varying orbit azimuth.The character can move toward, away from, or across the camera, with separate subject and camera trajectories for disentangling camera motion from non-rigid subject motion.
- Third-person: Third-person control semantics distinguish camera-relative character translation from camera yaw around a stationary character.W/A/S/D translate the character relative to the camera, while L/R rotate the camera and keep the character’s image position fixed.
- Panoramic: Panoramic sequences follow the recorded path and assemble 4096 × 2048 equirectangular video from six cube faces rendered simultaneously.Rendering ordinary perspective faces preserves the measured HDR range instead of using Unreal Engine’s panoramic pass, which can collapse highlights during accumulation.
4 WorldRover-10M: Multimodal Data and Annotations
WorldRover-10M packages standardized multimodal sequence outputs, including registered colour, depth, velocity, trajectories, actions, and third-person geometric point tracks. These annotations are rendered or derived from the actual camera and character motion, with explicit visibility and storage conventions for long-range supervision.
- Data format: Each sequence is delivered as a manifest-backed directory whose annotation bundles share coordinate, timing, and metadata conventions.Available outputs vary by subset.
- Raster annotations: A single multi-layer EXR registers colour, metric depth, and screen-space motion at identical resolution, projection, and frame timing.Movie Render Queue writes the three rasters from one configuration with spatial anti-aliasing samples.
- Raster annotations: Depth stores camera-centre radial distance in metres, clipped to 0.1–200 m, logarithmically quantized to 16 bits, and decoded from lossless FFV1 with sidecar parameters.The logarithmic encoding provides finer absolute precision at short range and coarser precision at long range.
- Raster annotations: Third-person subsets provide renderer-derived dense optical flow as lossless FFV1 streams, preserving motion within ±64 pixels per frame by default while clipping larger displacements.Flow is stored in two 16-bit channels with a campaign-wide scale g, configurable beyond the default 64 pixels per frame.
- Point tracks: Third-person point tracks are generated by unprojecting seed pixels into geometry and reprojecting them across frames, with separate character/background cohorts and explicit per-frame visibility.Occluded points retain amodal image positions; the default character binding follows root translation and yaw but not articulated deformation, while an LBS variant is also implemented.
- Trajectories and actions: Per-frame records include the rendered camera pose and intrinsics, the distinct character pose and speed, and trajectory-derived camera-relative actions reconstructed from differenced motion.Panoramas report an equirectangular projection and shared camera-centre pose; third-person action streams remain camera-relative rather than using the follow camera.
5 WorldRover-10M: Diversity and Scale
WorldRover-Engine scales artist-built worlds by recombining environments, states, viewpoints, characters, and routes after one-time authoring. WorldRover-10M releases long, diverse explorations across 32 environments with multiple trajectory strategies and 21.9M rendered frames.
- Scalable generation: A fixed asset library expands through scene composition, environmental states, viewpoints, characters, and routes, requiring authoring once before campaign-based sequence generation.Subsequent rendering uses campaign configurations and compute rather than per-frame manual annotation.
- Scene diversity: The engine comprises 30+ commercially licensed, artist-built environments spanning urban, historical, horror, industrial, interior, fantasy, cyberpunk, mountain, street, and village settings.Scenes range from single apartments to full urban districts and include authored materials, set dressing, and clutter.
- Appearance diversity: Lighting spans day, dawn, evening, sunset, and night, while weather spans clear, overcast, fog, and snow; rerendered routes preserve geometry and motion.Because lighting is simulated rather than baked, indirect illumination, colour bleeding, and shadow softness change with the authored state.
- Appearance diversity: The engine can rerender sequences as white models, preserving geometry and camera motion while replacing surface appearance with a neutral material and shading.This provides a coarse structural input for coarse-to-real generation without editing individual scene components.
- Character diversity: The asset library contains 70+ animated characters, including humanoids, animals, and creatures with varied body plans, scales, gait, articulation, and turning behaviour.Bipeds pivot in place, whereas quadrupeds and birds turn along arcs, making subject articulation a distinct diversity axis.
- Release scale: The release contains 6,003 sequences from 32 environments, totalling 21.9M rendered frames, 202.7 hours of video, and 18.65 TB of data.WorldRover-10M names the 10.8M first-person frames; third-person and panoramic sequences add observations of routes defined by first-person trajectories.
- Route diversity: The dataset includes 3,662 navmesh-planner sequences, 1,996 reactive-explorer sequences, and 345 waypoint trajectories, with median durations of 115 s, 81 s, and 105 s respectively.Route speeds use three settings—1.0, 1.2, and 1.5 m/s—forming three distribution clusters rather than one.
6 Related Work
Related work spans interactive simulators and engines, synthetic video rendered from existing 3D assets, and recordings of finished video games. These efforts provide varied environments, motion, geometry, and tracking annotations, but differ in how they generate and instrument explorations.
- Simulators and engines: Interactive simulators expose agent controls and world state, spanning Unreal, Unity, Isaac Sim, and physically interactive manipulation platforms.UnrealCV opened Unreal Engine to computer vision, while UnrealZoo extends it with more than one hundred worlds.
- Rendered from existing 3D assets: Synthetic video datasets render existing 3D assets, including clothed human motion, animated characters, and long-range tracking sequences.PointOdyssey contains 104 videos averaging 2,000 frames, with deformable characters driven by real motion capture.
- Recorded from video games: Game-recording approaches intercept or instrument finished games to capture RGB alongside depth, cameras, segmentation, body joints, or other state signals.OmniWorld uses a shader injector and recorder to lift synchronised RGB and depth from the screen in one of its four domains.
- Recorded from video games: Recorded-game datasets extend to large-scale sequences with environmental and rendering annotations, including location, weather, camera trajectories, and G-buffer channels.The Generative World Renderer captures 4M synchronised RGB frames with five G-buffer channels, while WildWorld records 108M frames with skeletons and world state.
7 Applications
WorldRover’s annotations support interactive world models, dynamic 4D reconstruction, and generative rendering. Its paired trajectories, correspondences, geometry, and controlled appearance changes provide supervision for these applications.
- Interactive world models: Interactive world models use RGB, rendered camera trajectories, and trajectory-derived actions without a separate pose-estimation step.Third-person sequences distinguish camera from character motion, while matched perspective and panoramic observations provide reference content outside the perspective field of view.
- Dynamic 4D reconstruction: Dynamic 4D reconstruction uses dense optical flow and long-range point tracks with 3D position, 2D projection, and visibility through occlusion.Separate camera and character trajectories help distinguish camera motion from articulated-subject motion.
- Generative rendering: Generative rendering uses white-model and textured pairs for frame-aligned structural control, plus environmental variants that change appearance while fixing geometry and motion.These pairs support coarse-to-real generation and controlled appearance variation.
8 Conclusion and Future Work
WorldRover combines an Unreal Engine rendering pipeline with WorldRover-10M, a release of long-horizon explorations containing richly aligned visual, geometric, camera, tracking, and action annotations. The current system is limited by its treatment of independently moving characters and follow-camera traversability in cluttered scenes.
- Contributions: WorldRover-10M contains 21.9M frames across 6,003 sequences, including 10.8M first-person frames plus third-person and panoramic observations.Task-oriented subsets combine metric depth, optical flow, long-range 2D/3D point tracks, calibrated cameras, and trajectory-derived actions.
- Annotation design: WorldRover specifies how each annotation relates to its accompanying pixels and declares which outputs each sequence carries.Raster annotations share a rendered instant, camera calibration describes the producing camera, and track visibility is determined against rendered depth.
- Limitations: Third-person sequences control and annotate only one independently moving character, while the follow camera may enter geometry in cluttered scenes and cause sequence withholding.The follow camera rides several metres behind the character, and the route provides no traversability guarantee.