Magpie: Real-Time World Renderer for Interactive Games

• Conventional pipelines make polished prototypes costly because they require extensive asset production and runtime optimization, while complex materials, illumination, and environmental effects remain approximations.A prototype may have complete gameplay yet fail to convey the intended experience until its visual assets are finished.Source: §1 · p. 2
• Video generation models can produce realistic appearance and motion across styles, but do not reliably maintain game rules or provide precise state control, debugging, timely responses, history, and scene-structure adherence.Reusing a trained model makes exploring a new appearance less costly than rebuilding a production asset set.Source: §1 · p. 2
• The Game Engine resolves player actions and maintains world state, while the Render Server generates imagery from engine-produced white-box frames and viewpoint-retrieved historical frames.A text prompt and first-frame image initialize the Render Server; gameplay state, object properties, and event signals remain in the engine.Source: §1 · p. 2
• Magpie keeps gameplay rules and state in the Game Engine, where designers can inspect and modify state and reproduce interactions under controlled conditions.Games require actions and states to retain stable meanings under specified rules so mechanics can be tested, design knowledge communicated, and prototypes developed into products.Source: §1 · p. 2
• Magpie supports different visual styles, materials, lighting, environmental details, and secondary motion for the same playable scene, while letting developers evaluate gameplay before completing visual assets.A white-box scene can approach the intended visual effect before detailed modeling, texturing, rigging, lighting, and optimization are complete.Source: §1 · pp. 2–3

2.1 Real-Time Video Generation

• Real-time video work combines causal sampling acceleration, low-precision deployment, and bounded visual memory; Magpie applies these techniques to synthesis while keeping rules and authoritative state in the Game Engine.The approaches include short sampling trajectories, quantized execution, and memory schemes that retain evidence for revisitation without context cost growing with rollout length.Source: §2.1 · p. 3

2.2 Real-Time Interactive Video Generation

• Interactive video systems use keyboard or camera controls, text prompts, persistent scene memory, and runtime optimizations, but these controls do not guarantee compliance with authored gameplay rules.Magpie instead lets the Game Engine resolve actions, collisions, events, and persistent state before producing the white-box observation.Source: §2.2 · p. 3

2.3 Controllable Video Generation

• Controllable video research spans unified multimodal representations, editable geometric controls, rendering from graphics buffers, and translation of coarse simulations into realistic video.Magpie extends coarse-to-real rendering into a closed interactive loop: the engine resolves gameplay and emits white-box video, then the renderer supplies appearance and fine-scale dynamics.Source: §2.3 · p. 4

• The Game Engine executes rules, resolves player actions, maintains world state, and produces white-box observations; the independent Render Server converts those observations into video for the User Client.Designers define the scene and rules in the engine, and the client forwards player input to it.Source: §3 · p. 4
• At initialization, a text prompt and first-frame image establish visual style; during interaction, white-box frames alone condition denoising, while camera poses retrieve relevant visual history.Player actions, object properties, event signals, and hidden gameplay state remain inside the Game Engine.Source: §3 · pp. 4–5
• The Game Engine determines what happens and the Render Server how it is presented, so visual generation errors can reduce feedback clarity but cannot alter collision logic, progression, or later gameplay decisions.Transforming white-box observations directly into final imagery bypasses much of the asset production otherwise needed to make a functional prototype visually representative.Source: §3 · p. 5

4 Magpie Data Engine

• The data engine teaches visual rendering rather than gameplay execution: engine-authored rules and state transitions govern play, while time-aligned video pairs supervise conversion of engine output into generated video.Each capture also includes camera poses and initialization appearance text; operator inputs, collisions, state transitions, and events are preserved as metadata but are not training conditions or targets.Source: §4 · p. 5

4.1 Data Collection by Human Operators

• Operators manually recorded approximately 300 hours of 1920 × 1080, 60-FPS paired gameplay across 30+ Unreal scenes, deliberately covering difficult viewpoints, transitions, interactions, and idle intervals.Individual captures range from tens of minutes to hours and span indoor and outdoor environments, varied layouts, and basic gameplay states.Source: §4.1 · p. 5

4.2 Synchronized White-Box Rendering

• For each target frame, a synchronized white-box view provides geometry, layout, occlusion, and principal motion, while the high-fidelity stream supplies appearance, lighting, materials, detail, and secondary dynamics.The paired representations share spatial occupancy, collision boundaries, principal silhouettes, interaction functions, timeline, character state, and camera configuration; the white-box view omits final textures, materials, complex lighting, and nonessential geometry.Source: §4.2 · p. 5

4.3 Interaction Distribution

• Coverage centers on locomotion, viewpoint changes, vehicle control, simple transitions, and idle behavior, while complex combat, cooperation, and dense object manipulation remain underrepresented.Recorded interactions expose engine-resolved consequences such as wall and floor contact, stair traversal, vehicle motion changes, blocking, sliding, and character-pose changes; stationary characters and environment-only motion are retained to represent inactivity.Source: §4.3 · p. 5

4.4 Time-Aligned Structured Records

• Timestamped records preserve camera pose, operator input, collisions, state transitions, and events for provenance, filtering, and system analysis, but only camera pose currently supports visual-history retrieval.Operator inputs, collisions, state transitions, and event records are not used as training conditions or targets in this version.Source: §4.4 · p. 5, §4 · p. 5

4.5 Appearance Annotation

• Appearance annotation uses scene style and character appearance to establish visual identity without describing mechanics, rules, actions, or outcomes; Qwen3.6-27B generates these fields from representative observations.A fixed prompt produces the two fields separately and excludes gameplay semantics.Source: §4.5 · p. 5

5 Magpie System Architecture

• The distributed runtime uses WebRTC and prioritizes fresh, low-latency frames, discarding stale frames rather than delaying later ones.Explicit message boundaries prevent gameplay state from bypassing the Game Engine or entering the generative model as undeclared conditioning.Source: §5 · p. 6

5.1 Gameplay Design Workflow

• Designers create editable white-box scenes by specifying layouts, traversable regions, interactive rules, objects, physical parameters, triggers, event logic, and initial session state.An edit changes the executable scene first, after which the Game Engine resolves state and produces an updated white-box observation; design metadata is not sent directly to the Render Server.Source: §5.1 · p. 6

5.2 User Client

• The User Client displays generated video, captures keyboard, mouse, or controller input, timestamps it, and forwards it to the Game Engine without locally changing world state.It connects player input and generated output to the runtime loop and sends no controls to the Render Server.Source: §5.2 · p. 6

5.3 Game Engine

• The Game Engine applies designed rules, resolves traversal, collisions, interactions, events, and camera updates, then sends the Render Server a white-box observation and camera pose for each chunk.Object properties, actions, state variables, and event records remain inside the engine, separating stochastic visual synthesis from reproducible gameplay.Source: §5.3 · p. 6

5.4 Render Server

• The Render Server initializes appearance from text and image prompts, then uses each white-box frame as a structural condition and camera pose to retrieve history for visual consistency.Figure 4 describes memory combining retrieved history, an early anchor block, and recent generated chunks; chunk-wise few-step generation uses pyramid distillation, LightTAE, and FP8 execution.Source: §5.4 · p. 7, Figure 4 · p. 7

5.5 Closed-Loop Execution

• Each interaction cycle resolves player actions and their gameplay consequences in the Game Engine before the Render Server generates the visual segment shown to the player.The current implementation exchanges these results at chunk boundaries; the generative model does not predict or redefine rule-level outcomes.Source: §5.5 · p. 7

6 Generative World Renderer

• The renderer uses Wan2.2-TI2V-5B as its backbone and adopts Helios’s bounded multi-scale context, hierarchical generation, and few-step distillation designs for long-form generation at interactive compute throughput.Text and a first-frame style image initialize appearance; white-box frames provide continuing structural control, and camera poses are reserved for historical indexing.Source: §6 · p. 7

6.1 Restricted Conditions

• Text and a first-frame image establish scene appearance, while each chunk’s white-box observation supplies continuing control over layout, geometry, occlusion, character placement, and engine-resolved motion.White-box observations omit final materials, lighting, texture, and nonessential geometric detail; player actions, game state, object properties, and event signals affect generation only indirectly.Source: §6.1 · pp. 7–8, §6 · p. 7
• At 24 FPS, the engine records 20 white-box frames in approximately 830 ms and the Render Server generates the chunk in approximately 620 ms, yielding a first action-aligned visual response after roughly 1550 ms.Subsequent output chunks are displayed over approximately 830 ms; transfer and buffering account for the intervals between stages.Source: Figure 5 · p. 8
• Cross-attention provides the most practical balance of rendering quality and white-box consistency, so Magpie adopts it to inject the white-box condition.Direct latent embedding strongly preserves structure but reduces visual detail and quality; AdaLN conveys the condition within a chunk but causes pronounced boundary discontinuities.Source: §6.1 · p. 8

6.2 Autoregressive History Context

• The bounded history combines a stable early anchor, camera-pose-retrieved observations relevant to the current viewpoint, and recent chunks that preserve local continuity.Camera position, orientation, and frustum are used only for historical retrieval, not as denoising conditions; older recent chunks are evicted as new ones arrive.Source: §6.2 · p. 8

6.3 Hierarchical Few-Step Generation

• Magpie applies Helios’s hierarchical generation and multi-stage distillation, resolving coarse structure and motion before higher-frequency appearance and transferring a multi-step teacher trajectory to a 3-step student.Self forcing reduces the discrepancy between training on clean video and inference on generated context while preserving white-box adherence and ordered history conditioning.Source: §6.3 · p. 9

6.4 Deployment Optimization

• LightTAE and mixed precision reduce runtime cost, while chunk-at-a-time encoding and decoding keep their workloads independent of session duration.FP8 is used where numerical sensitivity permits; normalization, temporal embeddings, and sensitive operators retain higher precision.Source: §6.4 · p. 9

7.1 Stable Chunk-Wise Decoding

• To accommodate Wan decoder temporal causality, Magpie decodes one chunk at a time and bridges adjacent chunks so latency does not grow with session length.The bridge provides the causal context required at chunk boundaries while preserving a fixed chunk-wise decoding schedule.Source: §7.1 · p. 9

7.2 Render Server Throughput

• After warm-up, the Render Server generates at approximately 32.2 FPS from 620 ms per regular 20-frame chunk, with approximately 34 GB peak GPU memory usage.The first chunk produces 17 frames and is excluded from steady-state throughput calculation.Source: §7.2 · p. 9

7.3 End-to-End Interaction Latency

• Approximately 1.55 s elapses from user action to corresponding visual feedback, including 0.83 s of sequence preparation, 0.1 s of transmission and encoding, and 620 ms of rendering.A regular 20-frame sequence takes approximately 0.83 s to collect at 24 FPS; the Render Server renders under stable operation.Source: §7.3 · pp. 9–10

7.4 Qualitative Results

• The qualitative evaluation focuses on long-horizon memory, consistency with the white-box condition, and overall visual quality.The paper presents qualitative results for both the base renderer and the distilled real-time renderer.Source: §7.4 · p. 10

Latency from white-box pre-recording and chunk-wise inference

• At 24 FPS, recording 20 white-box frames takes approximately 830 ms, and chunk-wise transfer, inference, and decoding bring latency to approximately 1.6 s.The proposed remedy is frame-wise streaming so engine execution, condition encoding, generation, and display can overlap.Source: §8 · p. 10

Depth ambiguity in white-box conditions

• A single RGB white-box observation can leave depth, surface orientation, thin structures, and spatial relationships ambiguous, causing geometry errors despite plausible 2D structure.Engine-derived depth, normals, segmentation, or motion vectors could enrich the condition stream without transferring gameplay rules or hidden state.Source: §8 · p. 10

Inconsistency to white-box conditions

• Under rapid motion, viewpoint changes, occlusion, complex interactions, or unfamiliar scenes, generated geometry, character placement, and object boundaries can drift from engine observations.Improving consistency may require stronger condition encoding and injection, structural-deviation penalties, broader transition coverage, and runtime checks.Source: §8 · pp. 10–11

Insufficient high-fidelity appearance supervision

• Paired engine captures provide synchronized structural supervision but insufficiently diverse appearance targets, limiting photorealism and robustness across materials, lighting, weather, motion, and fine effects.Future training could mix real video and production-quality renders, using extra rendering passes, geometric condition construction, pseudo-labeling, or staged training when white-box alignment is absent.Source: §8 · p. 11

Absence of audio generation and synchronization

• Magpie generates visuals but no audio, so a complete experience needs synchronized sound tied to engine events and tighter latency and temporal alignment.The engine can provide authoritative audio events or conventional playback, but generated motion may alter the apparent timing or character of visible events.Source: §8 · p. 11

Lack of persistent three-dimensional visual memory

• Bounded 2D history supports short-term continuity and revisitation but lacks persistent 3D world memory, so appearance may drift across viewpoint changes or long absences.Continuously projecting generated observations into a reusable 3D representation could improve view-consistent appearance and geometry while leaving gameplay state authoritative in the engine.Source: §8 · p. 11

High inference cost and limited edge deployability

• The distilled 5B renderer requires a high-end server GPU and approximately 34 GB peak GPU memory, limiting client and edge deployment and increasing serving costs.Future work targets smaller, more efficient models through architectural improvements, distillation, quantization, and deployment-aware optimization.Source: §8 · p. 11

Frame rate and visual quality

• The reported 32.2 FPS still falls short of game requirements, while few-step generation, low precision, bounded context, and limited training coverage leave a visible quality gap.Outputs can show temporal instability, texture degradation, identity drift, and inconsistency with engine-visible detail.Source: §8 · p. 12

9 Conclusion

• The Game Engine resolves player actions and maintains world state, while an independent Render Server generates visuals from the engine’s white-box observations.The model handles visual presentation; the engine retains gameplay execution and state control.Source: §9 · p. 12
• This separation lets playable white-box scenes support richer materials, lighting, environmental detail, motion, and visual styles before complete production assets are available.Developers can evaluate gameplay and experience design earlier with less dependence on a complete visual-asset pipeline.Source: §9 · p. 12
• Magpie shows how generative models could bridge to game engines without replacing them, enabling more flexible, personalized presentation for players.Source: §9 · p. 12