Source-linked AI summary
Walking in the Implicit: Interactive World Exploration via Neural Scene Representation
Zhiqi Li, Chengrui Dong, Zhenhua Du, Hangning Zhou, Cong Qiu, Hailong Qin, Mu Yang, Dongxu Wei, Peidong Liu
TL;DR
Interactive video generation often rolls out frame latents, entangling state transition with high-frequency observation synthesis and complicating long-horizon consistency. Walking in the Implicit replaces those rollouts with a fixed-length, renderable NIS state, which NeuWorld evolves under camera and retrieved-history conditions before pose-conditioned rendering. Experiments report strong long-horizon pose and revisit consistency with favorable efficiency, while the evaluation remains scoped to static scenes and locally bounded states.
Problem
Existing camera-controlled systems roll out observations directly, entangling state transition with high-frequency synthesis and making long-horizon consistency increasingly difficult.
Method
NeuWorld factorizes interaction into NIS-state transition and pose-conditioned rendering, using NIS-VAE for scene states and NIS-DiT with unified camera, reference, and history conditioning.
Results
NeuWorld shows strong long-horizon pose and revisit consistency with favorable inference efficiency on static-scene exploration, without pretrained video foundation models or auxiliary 3D reconstructors.
Takeaways & Limitations
A fixed-length, locally anchored NIS provides a scene-centric rollout variable that supports camera-conditioned rendering and revisitation consistency.
Takeaways & Limitations
The evaluation is scoped to static scenes, and the current local, bounded NIS state is re-anchored rather than maintained as a persistent global map.
Abstract
from arXiv · showhide
Interactive video generation systems for camera-controlled world exploration roll out growing sequences of latent video frames, entangling state transition with high-frequency observation synthesis. We propose Walking in the Implicit, a scene-centric paradigm that changes the rollout variable from frame latents to a fixed-length, renderable implicit state, termed Neural Implicit Scene (NIS). This factorizes interactive generation into stochastic transition of a compact scene state and deterministic pose-conditioned rendering given the sampled state. We instantiate this paradigm as NeuWorld: a transformer VAE learns locally anchored NIS from sparse posed frames, and a diffusion transformer evolves NIS conditioned on future camera trajectories and geometry-aware retrieved history. By reusing the VAE encoder as a unified conditioner, NeuWorld maps camera, reference-image, and history cues into the same NIS modality, avoiding external heterogeneous encoders. Trained from scratch on public posed-view data without pretrained video backbones or auxiliary 3D reconstructors, NeuWorld achieves strong long-horizon consistency with favorable inference efficiency.
1 Introduction
Walking in the Implicit reframes camera-controlled exploration around a compact, renderable scene state rather than growing frame-latent rollouts. NeuWorld implements this factorization and reports strong long-horizon consistency and favorable efficiency without pretrained video backbones or auxiliary 3D reconstructors.
- Directly rolling out observations entangles state transition with high-frequency appearance synthesis, making long-horizon consistency increasingly difficult.
- A Neural Implicit Scene (NIS) is a fixed-length latent token set that serves as the interaction state and supports pose-conditioned rendering.The formulation separates generative transition in NIS space from rendering queried views through a decoder.
- NeuWorld uses NIS-VAE to encode posed views and NIS-DiT to sample the next local NIS state under camera control.Camera, reference-image, and retrieved-history cues are mapped into partial NIS or memory NIS tokens through the shared encoder.
- NeuWorld is evaluated on static-scene camera-controlled exploration using models trained from scratch on public posed-view datasets.The evaluation isolates compact scene-state support for local re-anchoring, memory-conditioned rollout, and revisitation consistency.
- Experiments show strong long-horizon pose and revisit consistency with favorable inference efficiency, while ablations support the roles of the proposed rollout and conditioning components.
2 Related Work
Related work spans learned and explicit scene representations for novel view synthesis, camera-conditioned video generation, and interactive world models. These lines of work address rendering, controllability, or iterative consistency through different representations and rollout strategies.
- Latent Scene Representations for NVS: Novel view synthesis uses representations including NeRF variants, voxels, hash grids, point-based structures, Gaussian splats, and learned posed-view models.
- Camera-Controlled Video Generation: Camera-controlled video generation steers video models with motion adapters, camera encoders, synchronization, trajectories, or diffusion-transformer camera knowledge.
- Interactive and Consistent World Models: Interactive world models extend camera or action signals into repeated rollouts and require efficient inference plus consistency when revisiting observed regions.Existing approaches improve long-horizon behavior through faster rollout or reconstruction and geometry modules.
3 Method
NeuWorld represents each interaction step with a locally anchored, fixed-length NIS, samples its evolution with NIS-DiT, and renders queried views through a frozen decoder. Its shared NIS conditioning interface incorporates camera, reference, and retrieved-history evidence, with training and retrieval strategies aimed at reducing long-horizon drift.
- Problem Formulation: Each interaction step uses the current posed view, a future camera segment, and retrieved history to sample a compact local NIS state.The state is re-anchored as the agent moves and covers the upcoming trajectory segment.
- Problem Formulation: NIS rollout factorizes stochastic scene-state sampling from deterministic pose-conditioned rendering of future observations.Multiple queried views are rendered from the same sampled state.
- NIS-VAE: NIS-VAE encodes posed context views into a fixed-length token set and decodes target images conditioned on queried poses.The latent length is independent of the number of target frames to be rendered.
- Unified NIS Conditioning: The frozen NIS encoder converts pose-only, pose-plus-reference, and retrieved-history inputs into partial or memory NIS tokens for NIS-DiT.These conditions share the rollout state modality rather than using separate image, camera, or reconstruction encoders.
- NIS-DiT: NIS-DiT evolves token-set latents with a diffusion transformer whose shared query slots define a canonical interface across clean, noised, and partial NIS states.The denoiser operates directly on NIS token sets without spatial or temporal positional encodings.
- Training Curriculum: A weak-to-strong curriculum introduces pose, reference, and memory conditions progressively while retaining weaker conditions for cold-start robustness.
- Geometry-Consistent Long-Horizon Interaction: Anti-drift augmentation degrades training history and adds latent condition noise so NIS-DiT can adapt to increasingly imperfect generated history during interaction.The augmentation strength is ramped with interaction step at inference.
- Geometry-Consistent Long-Horizon Interaction: Geometry-aware retrieval combines recent frames with globally retrieved frames scored by pose distance, field-of-view overlap, and recency across future trajectory poses.Pose-space diversity filtering forms the history set encoded as memory NIS tokens.
4 Experiments
NeuWorld is evaluated for camera-controlled forward generation and cycle revisitation on Re10K and DL3DV, with emphasis on long-horizon consistency, pose drift, retrieval, and efficiency. Results show strong long-horizon performance, geometry-preserving NIS conditioning, and favorable controlled ablation outcomes.
- Evaluation setup: NeuWorld is evaluated on static-scene camera-controlled exploration using Re10K and DL3DV, with forward trajectories and cycle revisitation as the main protocols.Evaluation reports visual quality, pose-consistency proxies, and runtime over long trajectories.
- Forward trajectory generation: NeuWorld achieves low long-horizon pose drift and strong novel-view generation, ranking best or within the top group across the reported Re10K and DL3DV settings.On Re10K, it is best across all metrics at frame 50 and has the lowest pose errors at frame 200; on DL3DV it remains competitive at frame 80.
- Cycle revisitation: NeuWorld achieves the best Re10K revisit self-consistency and translation consistency while remaining the second fastest overall at 1.14 minutes per forward-and-return trajectory.Cycle revisitation evaluates return-path quality, paired-frame self-consistency, and average runtime.
- Latent representation: Under aligned Stage-1 conditions, NIS improves FVD from 88.03 to 86.20 and rotation error from 4.20° to 3.26°, while slightly increasing translation error from 0.141 to 0.157.The NIS prior also reaches 50k training steps in 17.2 hours versus 78.0 hours for the latent-frame prior.
- NIS capacity: Increasing NIS token length steadily improves view synthesis, whereas increasing channel width gives only marginal gains; L=1024 and D=64 provide the selected compute–quality trade-off.At fixed L=1024, PSNR increases from 26.25 to 26.82 as D grows from 32 to 256.
- Conditioning and retrieval: Unified NIS conditioning and anti-drift augmentation improve pose consistency and long-horizon robustness, while geometry-aware hybrid retrieval outperforms recent-only history on return paths.Removing anti-drift augmentation increases long-horizon Tdist from 0.153 to 0.680, and recent-only retrieval fails badly during revisitation.
5 Conclusion
NeuWorld instantiates Walking in the Implicit by representing rollout states as fixed-length, locally anchored NIS and rendering queried observations with a frozen decoder. Experiments show strong long-horizon pose and revisit consistency, while the evaluation remains scoped to static scenes and local, bounded states.
- NeuWorld represents each rollout state as a fixed-length, locally anchored Neural Implicit Scene and renders queried observations with a frozen decoder.
- The design separates latent scene-state sampling from high-frequency observation synthesis and provides a unified NIS modality for camera, reference-image, and history conditions.
- Experiments on Re10K and DL3DV show strong long-horizon pose and revisit consistency with a favorable accuracy–latency trade-off under cycle revisitation.
- The evaluation is scoped to static scenes under camera control, while the local NIS state is bounded, re-anchored during movement, and covers a trajectory segment rather than a persistent global map.
A.1.1 Visual Evidence from Masked Reconstruction
Masked reconstruction tests whether partial NIS can preserve geometry when most appearance information is removed. The resulting scaffold retains camera-consistent coarse structure while leaving unsupported appearance uncertain, supporting unified conditioning for NIS-DiT.
- Motivation: The masked-reconstruction study tests whether a frozen NIS-VAE encoder produces a useful geometric scaffold when most image content is removed.
- Protocol: Full NIS preserves complete context-view information, whereas pose+reference partial NIS retains camera poses and one reference image while dropping other view pixels.
- Observations: Partial NIS preserves dominant scene structure, coarse depth ordering, and cross-view geometric alignment around the reference anchor despite losing unsupported appearance detail.
- Observations: Partial NIS provides NIS-DiT with a stable scaffold without over-specifying erroneous details, allowing supported appearance propagation and content inference where extrapolation is required.
A.1.2 Latent Interpolation as NIS-Space Smoothness Evidence
Interpolation experiments probe whether NIS changes smoothly under local re-anchoring and across distinct sequences. Decoded reference views evolve continuously, supporting a structured common NIS space for local anchoring and interactive generation.
- Motivation: Interpolation is used as a qualitative probe of local smoothness, while not establishing that the latent manifold is globally well structured.
- Same-sequence reference-shift interpolation: Same-sequence reference-shift interpolation re-canonicalizes identical scene evidence under first- and last-view anchors to test continuity of NIS coordinates.
- Same-sequence reference-shift interpolation: The decoded reference-view image changes smoothly during anchor shifts, with dominant layout and major structures remaining coherent rather than showing abrupt geometric inconsistencies.
- Cross-sequence reference-view interpolation: Cross-sequence interpolation produces gradual evolution of rendered reference-view content instead of immediate collapse into meaningless images, but does not represent physically faithful trajectories between unrelated states.
- Conclusion: Together, the interpolation results support mapping diverse conditioning inputs into a common NIS space with enough continuity for stable local anchoring and interactive generation.
A.2 Additional Implementation Details
The additional implementation-details section states that its settings are shared across the main experiments unless otherwise noted.
- The listed implementation settings are shared across all main experiments unless otherwise stated.
A.2.1 Data Processing and Pose Normalization
The preprocessing pipeline standardizes image geometry and camera coordinates, then anchors each Neural Implicit Scene to a local reference-view neighborhood.
- Frames are center-cropped and resized to 256 × 256, with camera intrinsics adjusted for the changed aspect ratio.
- Camera poses are transformed into a chosen reference-view coordinate system and translations are normalized using scene-extent statistics with random training rescaling.The default scale is the farthest camera distance, and each NIS therefore represents a local scene neighborhood.
- Cross-sequence latent interpolation is used diagnostically to test whether the decoder responds continuously to latent perturbations across distinct sequences.The visualization is not evidence of physically faithful interpolation between unrelated world states.
A.2.2 View Sampling Details
The training and evaluation design progressively exposes NeuWorld to broader camera, appearance, and history conditions, while retrieval selects compact, trajectory-relevant memory under fixed rollout protocols.
- View Sampling Details: Stage 1 and Stage 2 use M=8 context views with temporal gaps g ∼Unif[80, 128], whereas Stage 3 uses M=15 views and g ∼Unif[180, 300].Stage 3 divides the sampled sequence into history and future trajectory segments around a fixed middle reference view.
- Training Curriculum: NIS-DiT follows a weak-to-strong curriculum from pose-only conditioning to reference appearance and retrieved history, with fallback paths preserving weaker conditioning and cold-start capability.The curriculum is intended to prevent shortcut copying from strong appearance conditions during from-scratch learning.
- Regularization: History-side corruption and latent condition noise are applied during Stage 3 to narrow the train-test gap in long rollouts, with the same noise-level interface used at inference.Removing anti-drift augmentation disables both history corruption and latent condition noise.
- Memory Retrieval: History retrieval combines recent and global frames with a fixed budget of Mret=8, using temporal continuity and geometric recall for the next local NIS state.The retrieval budget and other rollout settings remain fixed in retrieval ablations.
- Memory Retrieval: Trajectory-aware retrieval scores candidates using pose similarity, visibility-overlap-based FoV support, and recency, aggregating relevance across sparse query poses rather than only an endpoint.Global candidates additionally use a pose-space diversity constraint before merging with recent indices.
- Evaluation Protocols: Evaluation covers forward novel-view generation and cycle-trajectory revisitation, with default 10-frame subsampling and protocol-specific frame horizons across Re10K and DL3DV.Forward evaluation uses the 50th/200th frames on Re10K and the 20th/80th frames on DL3DV.