PaperlayerView exact revision on arXiv

Source-linked AI summary

AlayaVista: Streaming World Modeling from Panoramic States to Perspective Video

Jiaming Tan, Mingliang Zhai, Zhen Li, Yuwei Wu, Chuanhao Li, Kaipeng Zhang

arXiv:2609.14462v1cs.CV

TL;DR

Interactive world models must preserve broad scene context under camera motion while producing high-fidelity observations efficiently. AlayaVista evolves a camera-conditioned panoramic latent state, renders only the requested perspective viewport, and refines it for display, with streaming support from chunk-autoregressive training and few-step distillation. The paper reports validation across perspective quality, camera control, stability, revisit consistency, and end-to-end streaming efficiency, alongside the MUGEN training dataset.

  • Problem

    Interactive video world models must maintain off-screen scene context during camera motion while producing high-fidelity perspective observations with low latency.

  • Method

    AlayaVista expands one perspective image into a 360° prior, evolves a camera-conditioned panoramic latent state, renders the requested viewport, and refines it through a progressive streaming pipeline.

  • Results

    AlayaVista’s evaluation covers perspective-video quality, camera controllability, long-horizon stability, viewpoint-revisit consistency, and end-to-end streaming efficiency, with the highest Consistency score of 0.9240 and rotation error of 2.132 reported in the comparison.

  • Takeaways & Limitations

    The global-to-local architecture preserves omnidirectional context while concentrating high-fidelity computation on the requested viewport rather than the complete sphere.

Abstract

from arXiv · show

Interactive video world models must maintain broad scene context under camera motion while producing high-fidelity observations with low latency. Existing approaches face a representation trade-off: perspective models operate on local views and must preserve off-screen content over long rollouts, whereas broader spatial coverage is typically obtained by synthesizing full-sphere videos or constructing explicit 3D representations. Motivated by the complementary roles of global context and selective local acuity in visual perception, we present AlayaVista, a camera-controllable streaming video world model that decouples panoramic world evolution from perspective observation synthesis. Given a single perspective image, AlayaVista constructs a 360-degree scene prior using a pretrained panorama expansion model and then evolves the scene as a camera-conditioned panoramic latent state. A latent viewport renderer maps this state to the requested perspective video latents, while a perspective refiner restores details, suppresses artifacts, and performs super-resolution. To support efficient streaming, we adapt the panoramic generator to chunk-autoregressive generation and distill both panoramic generation and perspective refinement into few-step processes. To provide the supervision required by this design, we construct MUGEN, a large-scale real-world panoramic video dataset containing 1,318 hours of videos at resolutions of at least 4K, together with rich semantic and geometric annotations.

1 Introduction

AlayaVista addresses the tension between maintaining broad scene context and producing high-fidelity perspective video by evolving a panoramic latent world state and refining only the requested viewport. It introduces a progressive streaming pipeline and the MUGEN dataset to support camera-controlled, long-horizon generation.

  • Motivation: Perspective-only world models must preserve off-screen content as the camera moves, while panoramic or explicit-3D approaches provide broader context at higher full-scene synthesis or representation cost.The paper frames this as a representation trade-off between local observations and broad spatial coverage.
  • Approach: AlayaVista starts from one perspective image, evolves a camera-conditioned panoramic latent state, and maps the requested viewport into refined perspective video.A panorama expansion model supplies the 360° scene prior; latent rendering and perspective refinement produce the displayed observation.
  • Approach: The design preserves omnidirectional context internally while concentrating high-fidelity computation on the user-selected view rather than the entire sphere.This follows the paper’s global-context/local-acuity decomposition.
  • Training Data: MUGEN provides 1,318 hours of real-world panoramic video at resolutions of at least 4K, with semantic and geometric annotations, plus a 300-hour MUGEN-HQ subset.The annotations include camera trajectories, depth maps, and instance masks.
  • Efficient Streaming: A progressive training pipeline learns long-window camera-conditioned dynamics, adapts them to chunk-autoregressive rollout, separately trains latent rendering, and distills components into few-step processes.These stages target continuous state evolution and efficient end-to-end streaming.
  • Evaluation: Evaluation covers perspective-video quality, camera controllability, long-horizon stability, viewpoint revisits, and streaming efficiency, validating the global-state/local-observation decomposition.The reported evaluation also examines whether panoramic states follow requested trajectories and remain stable during autoregressive rollout.

2 Related Work

Related work spans perspective world models, panoramic video generation, explicit 3D representations, and generative rendering. AlayaVista differs by using panoramic video latents as internal dynamic states and rendering only the requested perspective viewport at high fidelity.

  • Panoramic Video Generation: Panoramic generation has progressed toward controllable, explorable 360° worlds using spherical representations, geometry-aware operations, and camera conditioning.Prior systems address distortion, longitude periodicity, seam continuity, and trajectory-controlled exploration.
  • Panoramic Video Generation: Most panoramic methods retain panoramic RGB video as the principal output, whereas AlayaVista treats panoramic video latents as internal world states and maps only the requested viewport to perspective latents.This distinguishes the method’s output interface from panoramic exploration systems.
  • Perspective World Modeling: Perspective world models directly generate user observations and condition them on camera trajectories, but they operate in local perspective space.AlayaVista instead separates world evolution from perspective observation synthesis.
  • Perspective World Modeling: Explicit 3D approaches build Gaussian splats, meshes, point clouds, or spatial proxies before rendering perspective observations, while AlayaVista maintains a panoramic video latent without an explicit 3D asset.The comparison concerns the internal world representation and rendering interface.
  • Perspective World Modeling: AlayaVista’s perspective output pipeline converts camera-conditioned panoramic states through latent rendering, spatial upsampling, local refinement, and RGB decoding.Figure 2 summarizes the interface from panoramic state to perspective video.

3.1 Overview

AlayaVista separates omnidirectional world evolution from perspective observation synthesis through four functional modules operating primarily in latent space. The requested viewport is rendered and refined before only the final perspective latent is decoded into RGB.

  • 3.1 Overview: The framework contains a panorama initializer, ERP-aware panoramic state generator, geometry-guided latent renderer, and perspective video refiner.The refiner combines latent spatial upsampling with carrier-conditioned generative refinement.
  • 3.1 Overview: Given an input image, camera trajectory, viewport parameters, and optional text condition, the pipeline evolves panoramic states and selects perspective observations through learned rendering.The overview defines the main inputs and the global-to-local transformation.
  • 3.1 Overview: The panoramic state is an omnidirectional representation centered at the evolving camera pose rather than an explicit metric 3D map.The upsampled perspective latent acts as a structural carrier for refinement.
  • 3.1 Overview: Generation, rendering, upsampling, and refinement operate in latent space, and only the final refined perspective latent is decoded into RGB.The fixed decoder converts the final perspective latents into the displayed image sequence.

3.2 Panorama Initialization

Panorama initialization expands the input perspective image into a complete ERP panorama that supplies the scene prior for subsequent camera-conditioned panoramic evolution.

  • 3.2 Panorama Initialization: A pretrained HY-World 2.0 panorama model expands the input perspective image into a complete 2:1 equirectangular panorama.The fixed module synthesizes unobserved surroundings and provides the initial full-sphere scene prior.

3.3 ERP-Aware Panoramic State Generator

The ERP-aware panoramic generator adapts a pretrained video transformer to spherical geometry, camera conditioning, and chunk-causal streaming. It preserves panoramic state across camera-controlled rollouts while reusing cached features across chunks.

  • 3.3 ERP-Aware Panoramic State Generator: The generator retains pretrained Wan conditioning and flow matching while adapting positional encoding, VAE boundary handling, and camera conditioning to panoramic geometry.The panoramic model operates on 48-channel WanVideoVAE latents and disables spatial tiling to avoid artificial internal boundaries.
  • 3.3 ERP-Aware Panoramic State Generator: Spherical positional encoding and longitude-circular operations adapt the pretrained generator to seam-continuous ERP geometry.Higher-frequency channels use periodic longitudinal harmonics, while lower-frequency channels use continuous spherical coordinates; temporal and latitude handling remain unchanged.
  • 3.3 ERP-Aware Panoramic State Generator: Camera conditioning represents each ERP token with a spherical viewing ray and camera-relative local ray frame through a parallel UCPE attention pathway.The pathway maps attended features into query-ray coordinates before residual fusion with native self-attention.
  • 3.3 ERP-Aware Panoramic State Generator: Chunk-causal state evolution uses bidirectional attention within chunks and causal attention across chunks, with aligned KV caches for native and panoramic attention.Generated panoramic features and their camera-relative geometry are reused without recomputing the entire history.

3.4 Latent Render Module

The latent render module converts panoramic latent trajectories into requested perspective observations using geometry-conditioned viewport queries. It approximates decode–project–encode supervision while avoiding full panoramic RGB decoding at deployment.

  • 3.4 Latent Render Module: The renderer maps panoramic latent trajectories to perspective observations using viewport yaw, pitch, and horizontal field-of-view conditions.Analytic viewport geometry selects observation directions within the ERP representation at each RGB timestamp.
  • 3.4 Latent Render Module: A learned latent mapping approximates the decode–project–encode reference operator, combining analytic gnomonic projection with VAE-specific nonlinear corrections.Wκ performs pixel-space gnomonic projection, Dpan decodes ERP latents, and EWan encodes perspective outputs for supervision.
  • 3.4 Latent Render Module: Perspective queries combine geometry-aligned ERP neighbors, fractional offsets, spherical position, and distortion features before cross-attending to ERP tokens.Four viewport pose anchors are retained per latent slice because each non-initial Wan slice represents four RGB frames.
  • 3.4 Latent Render Module: Global-transformer and local-adapter residuals are added to geometry-conditioned latent samples, preserving an explicit sampling prior without full RGB projection at deployment.The residual terms share the shape of the perspective output latent.

3.5 Perspective Video Refiner

The perspective refiner turns low-resolution viewport latents into high-resolution video through deterministic spatial upsampling followed by carrier-conditioned generative refinement. Chunked four-step causal inference supports streaming while preserving structural continuity across chunks.

  • 3.5 Perspective Video Refiner: The refiner combines a deterministic latent spatial upsampler with a carrier-conditioned generative network to produce high-resolution observations.The upsampler establishes the target grid and structural carrier, while the generative network synthesizes appearance details before RGB decoding.
  • 3.5 Perspective Video Refiner: Three-dimensional residual blocks aggregate local spatiotemporal information, while spatial PixelShuffle doubles spatial resolution without changing temporal length.The upsampler is trained specifically in Wan latent space rather than applying an LTX checkpoint directly.
  • 3.5 Perspective Video Refiner: The upsampler increases the latent grid from 18 × 32 to 36 × 64, corresponding to 512 × 288 to 1024 × 576 output resolution.Its fixed nearest-neighbor carrier is augmented by a learned residual in normalized Wan latent units.
  • 3.5 Perspective Video Refiner: Carrier-conditioned refinement processes chunks of four latent slices, appending the clean carrier and using the preceding refined chunk as a causal prefix after the first chunk.Attention is bidirectional within each chunk and causal across chunks, while only current target tokens are decoded.
  • 3.5 Perspective Video Refiner: Four-step causal inference denoises each chunk, emits the refined result, and reuses it as the clean prefix for the next chunk.Long videos are formed by concatenating the refined chunks.

3.6 Training Stages

AlayaVista trains panoramic generation, latent rendering, and perspective refinement as separate stages, progressively adapting the system for chunked streaming and few-step inference.

  • Panoramic Generator Training: The panoramic generator progresses from bidirectional adaptation to chunk-autoregressive training and few-step distillation for efficient rollout.Long-window training precedes causal chunk generation, while distillation reduces the number of sampling steps.
  • Panoramic Generator Training: Chunk generation conditions each current block on cached initialization and preceding chunks while excluding future chunks and exposing training to imperfect rollout histories.Teacher-forced, corrupted, replayed, self-resampled, and teacher-generated histories approximate deployment conditions without propagating gradients through constructed history.
  • Render Module Training: The render module learns a deterministic panoramic-to-perspective latent transformation from varied viewport trajectories using reconstruction, temporal, perceptual, and high-frequency supervision.Deployment retains latent-space geometric sampling and learned correction rather than RGB decode–project–encode operations.
  • Perspective Video Refiner Training: Perspective refinement is trained after upstream components are fixed through latent upsampling, multi-step quality adaptation, and few-step causal distillation.The upsampler learns a deterministic high-resolution carrier, while later refinement restores detail while preserving carrier layout and motion.
  • Perspective Video Refiner Training: Refinement losses anchor low-frequency structure and restore high-frequency target details while also enforcing perceptual quality and temporal consistency.The distilled student rolls out chunk by chunk using detached predictions as subsequent prefixes, reducing long-horizon error accumulation.

4 Training Data

MUGEN is a large-scale panoramic-video corpus built for camera-controllable world modeling, combining high-resolution real-world footage with aligned semantic and geometric supervision. A unified annotation and filtering pipeline produces diverse training data and the MUGEN-HQ subset.

  • 4.2 Hierarchical Annotation: MUGEN and Sekai2 are processed with a shared ERP-based hierarchical annotation pipeline containing panoramic videos, temporally aligned captions, and per-frame camera information.The common schema supports mixing both sources during training without separate text-conditioning formats.
  • 4 Training Data: Training data jointly targets realistic appearance, long temporal evolution, diverse camera motion, and temporally aligned semantic and geometric supervision.MUGEN provides large-scale high-resolution real-world videos, while Sekai2 broadens the scene and trajectory distributions.
  • 4.1 MUGEN Collection and Curation: MUGEN-HQ is a 300-hour subset selected through COVER-based quality ranking and stratified sampling across scenes, camera motion, actions, and weather.The procedure preserves high visual quality while maintaining diversity in scene content and controllable camera motion.
  • 4.2 Hierarchical Annotation: The annotation scheme separates video-level context from segment-level subject motion, environment motion, static scene content, and camera behavior.Camera labels include both continuous per-frame trajectories and a discrete 16-way camera-motion vocabulary, supporting explicit separation of subject dynamics from camera movement.

5 Experiments

Experiments evaluate AlayaVista on single-image camera-controlled perspective video synthesis using matched inputs and trajectories, with quantitative and qualitative analyses. The final perspective videos show strong reference agreement, temporal consistency, and camera-orientation tracking, alongside a motion-quality trade-off.

  • 5.1 Quantitative Evaluation: AlayaVista obtains the highest Consistency score (0.9240), improves Quality to 0.5579, and achieves the lowest rotation error of 2.132.Its translation error is 0.03312, lower than HY-World 2.0’s 0.03885 but higher than MoVerse’s 0.02746.
  • 5.1 Quantitative Evaluation: The lower Dynamic score of 0.9200 suggests a trade-off between attenuated local motion and temporal consistency during latent rendering and refinement.The reported interpretation identifies possible motion attenuation as a limitation of the rendering and refinement process.
  • 5.2 Qualitative Evaluation: Figure 6 pairs input images and target camera trajectories with generated frames to inspect framing changes, object displacement, newly revealed regions, appearance, and temporal continuity.The visualization complements quantitative translation-error evaluation by linking camera conditioning to user-visible observations.
  • 5.2 Qualitative Evaluation: Figure 7 aligns decoded panoramic-state visualizations with refined perspective outputs and marks queried viewports to show their correspondence across representations.The perspective outputs are produced through latent rendering, spatial upsampling, and generative refinement rather than direct ERP-frame cropping.

6 Conclusion

AlayaVista decouples panoramic world evolution from local perspective synthesis, using panoramic latent states for omnidirectional context and refinement for high-fidelity requested views. The system supports streaming through chunk-autoregressive rollout and few-step distillation, while MUGEN supplies large-scale annotated panoramic training data.

  • 6 Conclusion: AlayaVista evolves a 360° camera-conditioned panoramic latent state and converts requested viewports into high-fidelity perspective video through latent rendering and refinement.The architecture treats panoramic video as an internal dynamic state rather than the final output.
  • 6 Conclusion: The global-to-local design preserves omnidirectional scene context while concentrating expensive high-fidelity computation on the user’s displayed view.Chunk-autoregressive rollout and few-step distillation of panoramic generation and perspective refinement enable continuous generation.
  • 6 Conclusion: MUGEN provides 1,318 hours of at-least-4K panoramic video with rich semantic and geometric annotations, plus the 300-hour MUGEN-HQ subset.The complete system is trained using MUGEN and the panoramic subset of Sekai2.
Loading 2609.14462v1…