Source-linked AI summary
MUGEN: Interactive Panoramic World Exploration via Camera Control
Jiaming Tan, Zhen Li, Shuwei Shi, Minggui Teng, Siqi Yang, Yuwei Wu, Bo Zheng, Chuanhao Li, Kaipeng Zhang
TL;DR
Interactive panoramic video generation needs suitable annotated data and models that handle panoramic geometry while following camera trajectories. The paper introduces MUGEN and Wan360, and reports strong video quality, temporal coherence, and trajectory controllability, with comparisons favoring Wan360 on several metrics.
Problem
Existing panoramic datasets are often short or lack camera trajectories, while perspective video-generation models do not directly support panoramic geometry.
Source: §1 · p. 2Method
The paper builds MUGEN, a large annotated panoramic dataset, and fine-tunes a perspective camera-control model as Wan360 with ERP-aware components and panoramic Plücker embeddings.
Source: §1 · p. 2Results
Wan360 achieves strong panoramic video quality, temporal coherence, and camera-trajectory controllability, reducing FVD and LPIPS and improving SSIM, PSNR, and video quality against prior methods.
Source: §1 · p. 2, §5.3.3 · p. 10Takeaways & Limitations
Experiments support MUGEN as reliable data for interactive panoramic exploration and validate Wan360’s generation quality.
Source: §1 · pp. 2–3Wan360 generates fixed-length clips and does not yet support real-time or unbounded long-horizon generation; strict 3D consistency is not guaranteed, particularly in dynamic scenes.
Source: §6.1 · p. 13
Abstract
from arXiv · showhide
Interactive panoramic video generation aims to synthesize immersive 360\textdegree{} videos that remain visually coherent while following user-specified camera trajectories during exploration. However, progress is limited by a coupled data-and-model gap: existing panoramic video datasets are often short, weakly annotated, or lack camera trajectories, while existing camera-controlled video generation models are designed for perspective videos and do not directly support panoramic geometry. In this paper, we introduce MUGEN and Wan360 to address these limitations. MUGEN is a large-scale real-world panoramic video dataset tailored to interactive 360-degree world exploration, comprising over 1,300 hours of at least 4K panoramic videos with rich semantic and geometric annotations. Built on MUGEN, we further present Wan360, a camera-controllable interactive panoramic video generation model. Panoramic videos are commonly represented by EquiRectangular Projection (ERP), which unfolds a spherical 360-degree view into a rectangular frame with cyclic longitude seams and pole distortions. To this end, Wan360 introduces three parameter-free ERP-aware components: periodic longitude RoPE for seam-consistent positional encoding, ERP-aware padding for reducing boundary artifacts, and random roll yaw for consistent learning. For camera control, Wan360 uses a panoramic Pl\"ucker embedding that represents camera motion with ERP rays rather than perspective pinhole rays. Experiments show that MUGEN serves as a data foundation for panoramic world exploration, and that Wan360 enables high-quality, temporally coherent, camera-controllable 360-degree video generation.
1 Introduction
Interactive panoramic exploration needs high-fidelity, temporally and spherically consistent video that follows user camera trajectories, but suitable datasets and model architectures remain insufficient. The paper introduces MUGEN and Wan360 to address the data and model gaps, with experiments validating their annotation and generation quality.
Motivation
- Interactive panoramic exploration requires high-fidelity content, temporal and spherical consistency, and adherence to user-specified camera trajectories.The paper identifies insufficient data and model architectures as reasons the task remains underdeveloped.Source: §1 · p. 2
MUGEN
- MUGEN provides over 1,300 hours of at-least-4K panoramic video with minute-level clips and rich semantic and geometric annotations.Annotations include captions, scene categories, actions, weather, crowd density, locations, camera trajectories, instance masks, and depth; MUGEN-HQ is a 300-hour subset for training and evaluation.Source: §1 · pp. 2–3
Wan360
- Wan360 adapts a perspective camera-control model to ERP panoramas using three parameter-free components and panoramic Plücker embeddings.Periodic longitude RoPE encodes cyclic longitude, ERP-aware padding reduces seam artifacts, and random roll yaw preserves corresponding camera trajectories; the embedding uses ERP rather than perspective pinhole rays.Source: §1 · pp. 2–3
Experiments
- Experiments find that MUGEN’s annotations provide reliable support for exploration and Wan360 achieves strong video quality, temporal coherence, and camera-trajectory controllability.The reported improvements are attributed to both MUGEN’s scale and quality and Wan360’s ERP-aware components.Source: §1 · p. 2
2 Related Work
Prior panoramic datasets tend to serve generation or perception, while panoramic generation and camera-control approaches address different geometry and environment challenges. The related work positions MUGEN and Wan360 in relation to these gaps.
2.1 Panoramic Video Datasets
- MUGEN combines large-scale real-world 360° video, minute-level duration, high visual fidelity, semantic labels, and geometric annotations for interactive exploration.Generation-oriented datasets often lack explicit camera trajectories, while perception-oriented datasets are not designed for world exploration.Source: §2.1 · p. 3
2.2 Panoramic Video Generation
- Wan360 addresses ERP longitude periodicity, boundary artifacts, and yaw-equivalent observations with three parameter-free ERP-aware components.The paper contrasts its exactly cyclic Periodic Longitude RoPE with PanoSplatt3R’s approximate periodicity, its video-VAE padding with PAR’s image-generation padding, and its joint panorama-and-trajectory rotation with PanoDiffusion’s panorama rotation.Source: §2.2 · p. 3
2.3 Camera-Controlled Video Generation
- Existing camera-controlled panoramic methods primarily target static or quasi-static environments, whereas Wan360 targets dynamic real-world exploration under natural, diverse, and unconstrained conditions.The related work also notes that many camera-controlled video-generation methods focus on perspective video.Source: §2.3 · p. 3
3 MUGEN Curation
MUGEN’s curation pipeline collects real-world panoramic video, prepares minute-level clips, adds semantic and geometric annotations, and filters content for quality. The resulting dataset contains 1,318 hours of one-minute clips from 6,446 source videos.
3 MUGEN Curation
- The curation pipeline collects panoramic video, filters format and content, segments it into minute-level clips, annotates it, and samples MUGEN-HQ.MUGEN-HQ is a high-quality subset selected for reliable and diverse annotations.Source: §3 · p. 4
3.1 Data Acquisition and Preprocessing
- YouTube sourcing and format filtering yield 2,893 hours of raw footage from videos longer than one minute, with retained videos at least 4K and 30 FPS.The team manually curates channels and reviews search results to exclude non-real-world scenes; it initially retains 9,544 candidate source videos.Source: §3.1 · p. 4
- Continuous segments are split into non-overlapping 60-second clips, producing 137,209 clips totaling about 2,287 hours before content filtering.Preprocessing trims intros and outros, removes corrupted footage, detects shot boundaries, and transcodes the clips.Source: §3.1 · p. 4
3.2 Annotation
- MUGEN provides semantic annotations for content generation and geometric annotations for interactive camera control.Semantic labels include clip narratives, event-agnostic and interval captions, weather, time of day, crowd density, scene category, action, and location; ERP clips are projected into six cube faces for captioning and attribute prediction.Source: §3.2 · p. 5, §3.2.1 · p. 5
3.2.2 Geometric Annotations
- ViPE is selected for panoramic input and more coherent camera trajectories, and estimates trajectories, instance masks, and depth for each clip.Generating geometric annotations for the full dataset requires over 30,000 GPU hours.Source: §3.2.2 · p. 5
3.3 Quality Filtering
- Four content filters address poor lighting, visual defects, overlays, and unreliable trajectories before clips enter MUGEN.Clips outside mean luma [40, 215] are discarded; COVER scores below 0.7 are removed; six projected views are checked for subtitles, watermarks, and nadir patches; trajectories are screened for invalid paths.Source: §3.3 · p. 5, §3.3.1 · p. 5, §3.3.2 · p. 5
3.3.4 Trajectory Validity
- After filtering for trajectory validity, MUGEN contains 1,318 hours of high-quality one-minute clips from 6,446 unique source videos.Clips with chaotic paths, temporal discontinuities, or abnormal rotations are removed.Source: §3.3.4 · p. 5
3.4 MUGEN-HQ Sampling
MUGEN-HQ is a 300-hour subset selected to retain visual quality while balancing dataset diversity. Dataset analysis describes variation in semantic attributes and camera trajectories across MUGEN.
3.4 MUGEN-HQ Sampling
- MUGEN-HQ retains the top 70% of clips by COVER score, then stratifies sampling across scene categories, camera-motion patterns, actions, and weather.The subset contains clips from 4,023 unique source videos.Source: §3.4 · pp. 5–6
3.5 Dataset Analysis
- MUGEN spans varied weather, times of day, crowd densities, scene categories, and actions, alongside variation in translation, rotation, angular speed, and dominant motion direction.Figure 3 presents semantic attribute distributions on the left and camera-trajectory statistics on the right.Source: §3.5 · p. 6, Figure 3 · p. 6
4 Wan360 Model
Wan360 adapts a perspective camera-control video model to panoramic image-to-video generation conditioned on an input panorama, text prompt, and target camera trajectory. It addresses ERP geometry with parameter-free components and spherical-ray trajectory conditioning.
4 Wan360 Model
- Wan360 fine-tunes a perspective camera-control video baseline to generate ERP videos from an input panorama, text prompt, and user-specified camera trajectory.The baseline’s perspective geometry and pinhole camera rays cannot be directly applied to panoramas.Source: §4 · p. 6
4.1 Component Overview
- Four components adapt the model to panoramic geometry: three parameter-free ERP-aware components and an ERP ray generator for camera control.The components address longitude seams, VAE boundary artifacts, yaw-equivalent training pairs, and trajectory conditioning without camera intrinsics.Source: §4.1 · p. 6
4.2 Periodic Longitude RoPE
- Periodic Longitude RoPE replaces only longitude positional encoding with a cyclic formulation, keeping temporal and latitude encoding unchanged for baseline compatibility.Its positional code repeats across the latent-grid width, making tokens at opposite seam boundaries geometrically consistent and reducing seam-related artifacts.Source: §4.2 · p. 7
4.3 ERP-Aware Padding
- ERP-Aware Padding uses longitude-cyclic padding during VAE encoding and decoding so convolutions crossing either horizontal border sample from the opposite panorama side.This prevents the autoencoder from injecting artificial boundary artifacts into latent features and decoded frames.Source: §4.3 · p. 7
4.4 Random Roll Yaw
- Random Roll Yaw rolls each ERP training video and applies the same yaw rotation to its camera trajectory, producing geometrically aligned training pairs.This exposes the model to diverse trajectory orientations for the same scene.Source: §4.4 · pp. 7–8
4.5 Panoramic Plücker Embedding
- Panoramic Plücker Embedding converts ERP pixels into spherical rays transformed by target camera poses, replacing pinhole-ray conditions with panoramic trajectory conditioning.It reuses the baseline’s ray-based control interface while making conditioning compatible with panoramic geometry.Source: §4.5 · p. 8
5.1 Evaluation of Dataset Quality
A user study assesses MUGEN’s long-form descriptions, category labels, and camera trajectories using independent volunteer reviews of sampled clips. The annotations receive high validity or usability ratings across all three aspects.
5.1 Evaluation of Dataset Quality
- The study sampled 1,000 one-minute clips, reviewed independently by 10 volunteers, and required at least seven positive judgments for validity.Reviews covered descriptions, category labels, and camera trajectories.Source: §5.1 · p. 8
5.1.1 Text Descriptions
- 94.7% of sampled long-form descriptions were judged accurate against clip content, events, and camera motion.Hallucinated or contradictory descriptions counted as invalid.Source: §5.1.1 · p. 8
5.1.2 Category Labels
- 95.3% of sampled category labels were judged valid and supported by the video.Ambiguous or visually unverifiable labels counted as invalid.Source: §5.1.2 · p. 8
5.1.3 Camera Trajectories
- 92.4% of sampled camera trajectories were judged usable and aligned with observed camera motion.Obvious drift or wrong-direction cases counted as unusable.Source: §5.1.3 · p. 8
5.2 Experimental Setup
Wan360 is fine-tuned on MUGEN-HQ, and evaluation uses a held-out test set stratified by camera trajectory patterns and scene categories. The study measures reconstruction, perceptual and temporal quality, and camera control.
5.2.1 Training Details
- Wan360 is fine-tuned on MUGEN-HQ to generate 161 frames at 1920×960 resolution, sampled at 16 FPS.Training uses 8 NVIDIA H200 GPUs for 10 epochs at a learning rate of 1×10−5, starting from Wan2.2-Fun-5B-Control-Camera.Source: §5.2.1 · p. 9
5.2.2 Evaluation Data
- The final test set contains 100 videos stratified by camera trajectory patterns and scene categories.The videos are selected from 200 randomly chosen samples excluded from training.Source: §5.2.2 · p. 9
5.2.3 Metrics
- Evaluation measures reconstruction with FVD, SSIM, LPIPS, and PSNR, perceptual and temporal quality with VBench++, and camera control with TransErr.TransErr compares ViPE-annotated generated trajectories against input trajectories.Source: §5.2.3 · p. 9
5.3 Evaluation of Model Performance
On the held-out test set, Wan360 outperforms representative panoramic video generation methods across video-quality measures and achieves the lowest reported camera-control error. Ablations indicate complementary ERP-aware components and an important contribution from MUGEN-HQ.
5.3.1 Ablation Study
- Removing any of the three ERP-aware components weakens the balance between video quality and camera control; the full model achieves the best overall score.The ablation indicates the components are complementary when adapting a perspective video-generation baseline to panoramic video.Source: §5.3.1 · p. 9, Figure 5 · p. 10
5.3.2 Disentangling Model and Data
- Alternative training datasets yield several metric improvements over their corresponding original methods, but remain behind Wan360 trained on MUGEN-HQ.The comparison suggests model-side gains as well as a contribution from MUGEN-HQ’s scale, diversity, and annotation quality.Source: §5.3.2 · p. 10
5.3.3 Comparison with Existing Methods
- Wan360 reduces FVD and LPIPS, improves SSIM, PSNR, and video quality, and achieves the lowest TransErr among methods with explicit camera-control evaluation.The comparison uses the same held-out test set; the reported pattern indicates better frame-level fidelity, temporal realism, and trajectory following.Source: §5.3.3 · p. 10
5.4 Qualitative Examples
Qualitative examples compare Wan360 with other methods and show its outputs under prescribed camera trajectories. The examples depict coherent panoramic structure and scene appearance across varying camera motions.
5.4 Qualitative Examples
- Compared with 4K4DGen, GenEX, CubeComposer, and OmniRoam, Wan360 better preserves panoramic structure, visual details, and temporally coherent motion.The cited comparisons describe limited apparent motion, artifacts and geometric distortions, less dynamic videos, and static-scene assumptions in the respective alternatives.Source: §5.4 · p. 10, Figure 6 · p. 11
- Under explicit target trajectories, Wan360 follows prescribed motion while maintaining dynamic real-world content, unlike the compared OmniRoam sequences.The figure places the input panorama and target trajectory alongside generated sequences from OmniRoam and Wan360.Source: §5.4 · p. 10, Figure 7 · p. 12
- From the same input panorama, Wan360 generates distinct sequences for different camera trajectories while keeping scene appearance consistent.The examples cover 10 seconds of generation.Source: §5.4 · p. 12, Figure 8 · p. 12
6 Conclusion
MUGEN supplies large-scale real-world panoramic videos with semantic and geometric annotations, while Wan360 adapts a perspective camera-control baseline to ERP panoramas. The paper reports strong visual quality, temporal coherence, and trajectory controllability, while noting limits in 3D consistency, generation horizon, and data domain.
6 Conclusion
- MUGEN and Wan360 provide a practical foundation for camera-controlled interactive panoramic exploration in real-world environments.Wan360 adapts a perspective camera-control baseline to ERP panoramas using ERP-aware components and a panoramic Plücker embedding.Source: §6 · p. 13
6.1 Limitations
- Strict 3D consistency is not guaranteed, particularly in dynamic scenes with independently moving objects.In static scenes, reconstructed structure and relative object positions generally remain stable, with smooth, continuous estimated camera trajectories.Source: §6.1 · p. 13
- Wan360 generates fixed-length clips and does not yet support real-time or unbounded long-horizon generation.Error accumulation in long-horizon generation remains an open challenge.Source: §6.1 · p. 13
- MUGEN’s data primarily comes from the real world; extending it to animation and games is future work.Source: §6.1 · p. 13