PaperlayerView exact revision on arXiv

Source-linked AI summary

L3G-World: Real-Time Infinite World Generation with Latent 3D Gaussian Memory

Jinhyeok Choi, JoungBin Lee, Jongmin Lee, Minkyeong Jeon, Jiyoung Kim, Kaeun Han, Yoonjin Yoon, Seungryong Kim

arXiv:2610.40614v1cs.CV

TL;DR

Long-horizon interactive world models face a challenge: maintaining spatial consistency in real time. L3G-World alternates an updater for compact latent 3D Gaussian memory with a video diffusion transformer. It reports state-of-the-art generation quality and long-horizon consistency alongside reduced memory costs and real-time generation.

  • Problem

    Existing temporal-context and explicit 3D-memory approaches scale poorly with stream length, and RGB re-encoding for latent-generator conditioning limits real-time generation.

    Source: §1 · p. 2

  • Method

    L3G-World alternates a feed-forward updater that spawns or refines Gaussians with a video diffusion transformer conditioned by directly rendered Gaussian VAE features.

    Source: §1 · p. 2

  • Results

    L3G-World achieves state-of-the-art generation quality and long-horizon consistency, using 26.7–38.9× fewer primitives, reconstructing memory 3.0–3.2× faster, and rendering VAE features 349–359× faster than prior explicit 3D-memory baselines.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Few-step distillation combined with the memory design supports real-time world generation at 9.02 FPS on one H100 GPU.

    Source: §1 · p. 2

    The latent 3D Gaussian memory is primarily designed for static scenes and does not explicitly model independently moving objects.

    Source: E LIMITATIONS AND FUTURE WORK · p. 24

Abstract

from arXiv · show

Interactive world models generate visual content conditioned on user actions, yet maintaining spatial consistency over long-horizon exploration in real time remains challenging. Existing approaches rely on either temporal context or explicit 3D memory, both of which scale poorly because memory grows with the length of the generated stream. Moreover, explicit 3D memories condition latent video generators by re-encoding RGB renderings, which precludes real-time generation. In this paper, we introduce a Latent 3D Gaussian Memory (L3G) that combines the efficiency of feed-forward reconstruction with a compact, rapidly renderable scene representation. Building on this memory, we propose L3G-World, in which a memory updater and a video diffusion transformer (DiT) alternate autoregressively to generate spatially consistent worlds. The memory updater spawns Gaussians in newly observed regions and refines existing ones, so that the memory grows with scene extent rather than stream length. Each Gaussian further stores a VAE latent feature, allowing the memory to be rendered directly into the DiT's latent space without VAE re-encoding. Experiments show that L3G-World achieves state-of-the-art generation quality and long-horizon spatial consistency, with up to $38.6\times$ fewer 3D primitives, $3.2\times$ faster memory updates, and $359\times$ faster memory rendering relative to the most efficient prior 3D-memory baseline for each metric, enabling real-time world generation at 9 FPS.

1 INTRODUCTION

Interactive world models must preserve spatial consistency over long exploration while generating in real time. L3G-World addresses limits in existing memory approaches with an online, compact 3D Gaussian memory and reports state-of-the-art quality and efficiency.

1 INTRODUCTION

  • Long-horizon interactive world generation must maintain spatial consistency during exploration and loop closures while supporting real-time generation.Source: §1 · p. 2
  • Temporal-context methods help preserve consistency, but their memory and attention costs grow with stream length and resolution, potentially causing out-of-memory failures.Tab. 1 illustrates the scaling issue.Source: §1 · p. 2
  • Explicit 3D memories improve revisited-region consistency and camera control, but feed-forward dense memories accumulate redundant primitives while optimization-based compact memories add substantial latency.Unrefined primitives can also preserve geometric inaccuracies.Source: §1 · p. 2
  • L3G-Mem and L3G-Gen alternate autoregressively: the updater adds Gaussians in uncovered regions, refines existing ones, and renders stored VAE features directly into the generator’s latent space.This avoids redundant accumulation and VAE re-encoding; few-step distillation supports real-time, long-horizon generation.Source: §1 · p. 2
  • L3G-World achieves state-of-the-art long-horizon consistency and generation quality, using 26.7–38.9× fewer primitives, reconstructing memory 3.0–3.2× faster, and rendering VAE features 349–359× faster than prior explicit 3D-memory baselines.With few-step distillation, it reaches 9.02 FPS on one H100 GPU.Source: §1 · p. 2

2 RELATED WORKS

Related work spans latent video diffusion and interactive world models, feed-forward 3D reconstruction, and persistent 3D memories for consistent world generation. L3G-World combines compact adaptive Gaussian memory with direct VAE-latent conditioning.

Video Diffusion and World Models

  • Interactive world models generate environments conditioned on user or agent actions, drawing on progress in latent video diffusion and low-latency few-step generation.The cited approaches include training on model-generated histories and flow-based transitions with on-policy distillation.Source: §2 · p. 3

Feed-forward 3D Reconstruction

  • Feed-forward reconstruction predicts geometry in one pass, commonly as per-pixel pointmaps or Gaussians, while streaming and token-based methods process images online or decode compact primitive sets.Source: §2 · p. 3

Consistent World Generation

  • Explicit point-cloud and Gaussian memories improve spatial consistency and camera controllability, but dense feed-forward memories accumulate primitives while optimization-based scene memories incur substantial latency.L3G-World instead adaptively spawns or refines Gaussians and stores VAE latent features for efficient video-model conditioning.Source: §2 · p. 3

3 METHODOLOGY

L3G-World pairs a selectively updated latent 3D Gaussian memory with an autoregressive video generator. Its memory is rendered directly into the generator’s VAE latent space and updated as each chunk is generated.

3.1 MOTIVATION AND OVERVIEW

  • L3G-World alternates a memory updater and a memory-conditioned video generator, with selective updates designed to keep memory usage largely independent of generated-frame count.The memory representation uses 3D Gaussians for target-view rendering, and each Gaussian stores a VAE latent feature to avoid VAE re-encoding.Source: §3.1 · pp. 3–4

3.2 L3G-MEM: 3D GAUSSIANS AS WORLD MEMORY

  • New Gaussians are spawned only where rendered memory has low opacity or high RGB error, while visible existing Gaussians are retrieved for refinement.Masked geometry features are clustered into spawning queries; spawning and refinement tokens are jointly processed, then decoded into new Gaussians and residual updates.Source: §3.2 · pp. 4–5
  • L3G-Mem lifts VAE features into spatially aligned latent tokens and updates memory by appending spawned Gaussians and applying residuals to retrieved Gaussians.Entries not selected for refinement remain unchanged.Source: §3.2 · p. 5

3.3 L3G-GEN: MEMORY-CONDITIONED AUTOREGRESSIVE VIDEO GENERATION

  • L3G-Gen conditions video diffusion on memory latents rendered from target cameras, concatenated with noisy target latents, while previously generated reference latents remain fixed.Few-step on-policy distillation trains the student on its own autoregressive rollout to reduce train–test mismatch.Source: §3.3 · pp. 6–7

3.4 L3G-WORLD: CONSISTENT WORLD GENERATION FRAMEWORK

  • L3G-World repeatedly renders the previous memory for the next chunk, then passes generated latents to L3G-Mem to update that persistent memory.The corresponding RGB frames are obtained once through the VAE decoder, avoiding redundant latent encoding between the modules.Source: §3.4 · p. 7

4 EXPERIMENTS

Experiments evaluate L3G-World on DL3DV and RealEstate10K, including long-horizon loop-closure trajectories. Results report memory-efficiency gains, strong quality and consistency, and qualitative preservation of scenes on return to previously observed views.

4.1 EXPERIMENTAL SETUP

  • Evaluation uses DL3DV and RealEstate10K, with 481-frame trajectories that explore away from the initial view and retrace the camera path to measure long-horizon consistency.The study also evaluates generation quality, camera controllability, geometric consistency, and memory efficiency against explicit- and implicit-memory baselines.Source: §4.1 · pp. 7–8

4.2 QUANTITATIVE RESULTS

  • 266–417× fewer primitives and 42–67× less memory than NeoVerse, with 349–359× higher VAE-conditioning throughput.L3G-World also reconstructs memory 8.9–9.5× faster than Lyra 2.0 while using 7.8–10.4× less memory.Source: §4.2 · p. 8
  • L3G-World obtains the best PSNR and LPIPS on both datasets, with leading generation-quality metrics and strong camera-control results, though some RealEstate10K baselines score better on pose and MEt3R errors.On DL3DV, L3G-World has the best LPIPS, FID, SC, and BC; on RealEstate10K it also leads those metrics and is second-best in IQ.Source: §4.2 · pp. 8–9, Table 2 · p. 8

4.3 QUALITATIVE RESULTS

  • On loop-closure trajectories, L3G-World preserves scene structure and appearance after exploration, while baselines show drift, blur, or rendering artifacts.The qualitative comparisons start from the same input view, explore unseen regions, and return to the initial camera location.Source: §4.3 · pp. 9–10, Figure 6 · p. 10

4.4 ABLATION STUDY

  • Selective spawning, residual refinement, and coordination between L3G-Mem’s streams each contribute to reconstruction quality; removing them degrades RGB or VAE novel-view synthesis.Without adaptive spawning, the model uses about 1.8× more Gaussians; the full model has the strongest reconstruction quality across most RGB and VAE metrics.Source: §4.4 · pp. 10–11, Table 3 · p. 10
  • Removing memory refinement degrades all evaluated L3G-World metrics, particularly self-consistency, pose accuracy, and 3D consistency.The ablation indicates that the persistent memory must progressively correct reconstruction errors as scenes are revisited.Source: §4.4 · p. 11, Table 4 · p. 11
  • Replacing direct VAE latent rendering with RGB rendering followed by VAE encoding adds encoding overhead and lowers generation quality.The ablation supports rendering stored latent features directly for memory-conditioned video generation.Source: §4.4 · p. 11

5 CONCLUSION

L3G-World combines a compact Gaussian memory with direct VAE-latent rendering to support real-time, spatially consistent world generation. The appendix describes latent video diffusion and the 3D Gaussian representation used in this framework.

5 CONCLUSION

  • L3G-World selectively adds Gaussians in insufficiently covered regions and refines existing ones, while storing VAE features for direct latent-space rendering without re-encoding.The conclusion presents compact latent-space 3D memories as a practical foundation for scalable interactive world models.Source: §5 · p. 11

A.1 VIDEO DIFFUSION MODELS

  • Video diffusion models estimate a pretrained 3D VAE latent from Gaussian noise, with the VAE encoder and decoder mapping between video and latent representations.The latent state follows a probability-flow ODE; its flow map transports states between noise levels using an average velocity.Source: §A.1 · p. 17

A.2 3D GAUSSIAN SPLATTING

  • 3D Gaussian Splatting represents a scene with anisotropic Gaussians whose centers, covariances, opacities, and view-dependent colors are projected and alpha-composited for rendering.Covariance is parameterized by rotation and scale.Source: §A.2 · p. 17

B.1 GAUSSIAN MEMORY UPDATER

The updater selectively spawns and refines Gaussian memory, including VAE features, while the generator uses memory-conditioned diffusion models for chunked world generation. The teacher is trained with rectified flow matching, and the student is distilled for autoregressive inference.

B.1 GAUSSIAN MEMORY UPDATER

  • Low-opacity or high-error patches trigger Gaussian spawning, with the number of queries scaled to the masked-patch fraction and each query producing 32 Gaussians.At the first step, when memory is empty, the updater uses Nmax queries.Source: §B.1 · p. 18
  • The updater retrieves visible Gaussians and predicts gated residuals for their geometry, opacity, color, and VAE features, leaving unretrieved memory unchanged.New Gaussians also receive a 16-dimensional VAE latent feature, rendered with the same geometry and compositing as RGB.Source: §B.1 · pp. 18–19
  • Training rolls the updater across five consecutive chunks and supervises RGB, depth, and VAE-feature renderings, with visibility loss encouraging Gaussians to lie within an observed input-view frustum.Supervision from earlier target views constrains current spawning and refinement outputs despite detaching previous memory states.Source: §B.1 · p. 19

B.2.1 TEACHER MODEL

  • The teacher concatenates rendered memory VAE features, a validity mask, and a conditioning indicator with noisy video latents, and retrieves up to three nearby-pose reference latents.It is trained with rectified flow matching on DL3DV and RealEstate10K, then generates 81-frame clips using 35 denoising steps.Source: §B.2.1 · p. 20

B.2.2 STUDENT MODEL

  • The student converts the bidirectional DiT into a chunk-causal model that attends to compressed long-term history, retrieved references, and recent full-resolution chunks.Keys and values are cached separately for standard self-attention and UCPE attention branches.Source: §B.2.2 · p. 20
  • The student learns flow-map transitions between arbitrary denoising noise levels, then uses on-policy distribution matching and generates three latent frames per step with four denoising steps.At inference, generated latents join the reference pool and periodically update the Gaussian memory used for subsequent reconstruction guidance.Source: §B.2.2 · p. 20

B.3 BASELINES

The evaluation aligns explicit 3D-memory baselines to the metric camera trajectory and adapts several released implementations to the single-image, long-horizon benchmark setting.

B.3 BASELINES

  • Monocular reconstruction creates scale ambiguity against the metric evaluation trajectory, so predicted metric depth is used to align explicit 3D-memory baselines.For NeoVerse and L3G-World, the initially rendered Gaussian depth is additionally aligned with estimated input-image depth.Source: §B.3 · pp. 20–21
  • SPMem and 3DScenePrompt are modified to initialize their point clouds from the benchmark's single input image instead of multiple views or video frames.Their subsequent autoregressive generation chunks retain the official implementations' behavior.Source: §B.3 · p. 21
  • An autoregressive inference pipeline is built for NeoVerse by repeatedly reconstructing and updating its Gaussian representation from generated observations.Yume 1.5 and Matrix-Game 3.0 receive discrete actions converted from the benchmark's camera trajectories.Source: §B.3 · p. 21

B.4 EVALUATION

Evaluation uses long loop-closure trajectories to measure consistency after revisitation, alongside camera controllability and memory efficiency. Qualitative comparisons examine whether methods preserve scene structure and appearance when returning to the initial view.

B.4 EVALUATION

  • 481-frame trajectories explore from the input view and then retrace the route in reverse to return to the initial camera pose.Paired outgoing and returning frames at the same pose are compared using PSNR, SSIM, and LPIPS as self-consistency metrics.Source: §B.4 · p. 21
  • Camera controllability is measured by trajectory translation error after global-scale alignment and rotation error in degrees.The estimated camera trajectory is compared with the ground-truth trajectory used for generation.Source: §B.4 · p. 21
  • Efficiency measurements cover stored primitive count, memory footprint, reconstruction throughput, and RGB and VAE-space rendering throughput on one NVIDIA A100 GPU.VAE-space conditioning for RGB-only memories includes RGB rendering and VAE encoding, whereas L3G-World directly renders stored VAE features.Source: §B.4 · p. 21
  • On DL3DV and RealEstate10K loop-closure examples, L3G-World preserves scene structure and appearance upon return, while baselines show drift, blur, or rendering artifacts.The qualitative figure shows the shared starting view, exploration of unseen regions, and return to the initial camera location.Source: Figure 8 · p. 22

C QUALITATIVE RESULTS

Across long-horizon loop-closure trajectories, L3G-World generates plausible unseen regions and returns to a view spatially aligned with the input. Baselines show distinct failures in return-view consistency, unseen-region generation, or visual quality.

C QUALITATIVE RESULTS

  • L3G-World generates plausible content in unseen regions and returns with structure and appearance spatially aligned to the input view.Its latent 3D Gaussian memory anchors previously generated content while being refined during exploration.Source: C QUALITATIVE RESULTS · p. 23
  • SPMem hallucinates beyond initially observed regions, NeoVerse blurs, GEN3C distorts unseen content, and Lyra 2.0 drifts from the original scene on return.GEN3C returns to the starting scene more reliably, while Lyra returns reasonably close but not fully spatially aligned.Source: C QUALITATIVE RESULTS · p. 23

D COMPARISON WITH ONLINE 3DGS METHODS

The comparison evaluates online 3D Gaussian memories for reconstruction quality and the efficiency needed in repeated world-generation updates and queries. Latent 3D Gaussian memory combines compact storage and direct VAE-space rendering with useful reconstruction fidelity and spatially aligned conditioning for video generation.

D COMPARISON WITH ONLINE 3DGS METHODS

  • The online evaluation processes nine-frame chunks, renders four target viewpoints, and measures RGB/VAE NVS quality alongside memory size, reconstruction FPS, and VAE rendering throughput.The datasets are DL3DV and RealEstate10K.Source: D COMPARISON WITH ONLINE 3DGS METHODS · p. 23, Table 6 · p. 24
  • At target resolution 480 × 832, latent 3D Gaussian memory improves RGB- and VAE-space NVS over ReCoSplat on both datasets and outperforms OF3GS at higher resolution.ReCoSplat at 480 × 480 achieves higher fidelity, consistent with its optimization for high-quality NVS.Source: D COMPARISON WITH ONLINE 3DGS METHODS · p. 23
  • Across two datasets, at 480 × 480, latent 3D Gaussian memory uses approximately 49–53× fewer primitives and 13–14× less memory than ReCoSplat (Cheng et al., 2026), updating 2.5–2.6× faster.It also achieves 245–430× higher VAE rendering throughput by rendering stored VAE features directly rather than re-encoding RGB renderings.Source: D COMPARISON WITH ONLINE 3DGS METHODS · p. 23, Table 6 · p. 24
  • Target-camera VAE conditions rendered from memory remain spatially aligned and preserve previously observed structure, providing persistent scene conditioning for novel video content.Figure 9 shows ground-truth and generated views alongside the rendered VAE condition.Source: D COMPARISON WITH ONLINE 3DGS METHODS · pp. 23–24, Figure 9 · p. 24

E LIMITATIONS AND FUTURE WORK

L3G-World’s current limitations concern dynamic content, scalability under very large exploration, and error propagation during long autoregressive generation.

E LIMITATIONS AND FUTURE WORK

  • The memory targets static scenes and does not explicitly model independently moving objects; more general environments require separating dynamic content from static geometry.Source: E LIMITATIONS AND FUTURE WORK · p. 24
  • Memory grows with explored spatial extent rather than generation length, but can still increase during extremely large-scale exploration.Adaptive pruning or hierarchical memory representations are proposed to improve scalability.Source: E LIMITATIONS AND FUTURE WORK · p. 24
  • Refinement can correct inaccurate Gaussians, yet errors from long-horizon autoregressive generation may still propagate to subsequent memory updates.Source: E LIMITATIONS AND FUTURE WORK · p. 24

Related papers

Loading 2610.40614v1…