PaperlayerView exact revision on arXiv

Source-linked AI summary

RealMe: Immersive Volumetric Video from Minimal Captures

Chenxu Zhou, Zhiyuan Yu, Yudong Jin, Yinji Shentu, Yu Zhang, Jiaming Sun, Xiaowei Zhou

arXiv:2610.30023v1cs.CV

TL;DR

Sparse, low-quality captures limit the practical reach of high-quality volumetric video reconstruction, while independent frame- or view-wise enhancement can leave temporal and multi-view inconsistencies. RealMe adapts high-resolution video diffusion to the capture layout, and reports superior visual quality and 4D consistency on two benchmarks, including reconstruction at 2K resolution.

  • Problem

    High-quality volumetric video reconstruction typically relies on dense, controlled views, while independent frame- or view-wise enhancement ignores temporal and multi-view dependencies.

    Source: §1 · p. 2, §3.3 · p. 4

  • Method

    RealMe enhances limited captures with a high-resolution video diffusion framework using auxiliary videos, layout-aware positional embeddings, factorized spatio-temporal attention, and latent frequency loss.

    Source: §1 · p. 2

  • Results

    RealMe achieves superior visual quality and 4D consistency over existing baselines on RealVV and DNA-Rendering, extending reconstruction to 2K resolution under limited capture conditions.

    Source: §6 · p. 8

  • Takeaways & Limitations

    The framework reduces capture requirements for high-quality volumetric video reconstruction from limited inputs.

    Source: §1 · p. 2

    Training data remains constrained, generalization is limited for large scenes or complex camera settings, and insufficient geometric cues may prevent initialization of a valid coarse 4DGS.

    Source: §6 · p. 8

Abstract

from arXiv · show

This paper addresses the problem of creating immersive volumetric videos from sparse and low-resolution capture resources. Conventional dense multi-view reconstruction methods can produce high-quality volumetric videos, but their reliance on costly capture setups makes them impractical in real-world scenarios. Recent approaches use generative models to relax capture requirements, but still suffer from limited resolution and 4D inconsistency. To overcome these constraints, we propose RealMe, a unified framework built around a high-resolution video diffusion model that enhances limited captures into high-resolution (2K), 4D-consistent videos for volumetric video reconstruction. Specifically, RealMe constructs auxiliary data from capture videos and encodes them into latents along with layout-aware positional embeddings. The video diffusion model adapts factorized spatio-temporal attention to the multi-view capture layout, denoises these latents across viewpoints and time, and decodes them into enhanced videos to refine the 4D representation. This adaptation helps the model capture temporal dynamics and multi-view relationships, leading to more 4D-consistent generation. To reduce the data gap between generated videos and real captures, we apply a latent frequency loss that provides frequency-domain supervision in latent space. Extensive experiments on the public DNA-Rendering benchmark and our self-captured RealVV dataset demonstrate that RealMe is able to create immersive volumetric videos and outperforms state-of-the-art methods in visual fidelity and 4D consistency.

1 Introduction

RealMe targets high-quality volumetric video reconstruction from sparse or low-quality captures, which are impractical inputs for methods that depend on dense, controlled views. It enhances these captures with a layout-aware video diffusion framework and evaluates the resulting approach on DNA-Rendering and RealVV.

Motivation

  • Dense multi-view reconstruction can achieve high quality, but its controlled capture requirements make it impractical when only sparse views or low-quality videos are available.The paper frames matching dense-capture visual quality and 4D consistency from limited captures as its central problem.Source: §1 · p. 2

Approach

  • RealMe enhances limited captures with a high-resolution video diffusion model, using auxiliary videos, layout-aware positional embeddings, and factorized spatio-temporal attention.The generated videos and original captures are used to optimize the scene representation; latent frequency loss guides low-frequency structure and high-frequency detail.Source: §1 · p. 2

Experiments

  • RealMe outperforms state-of-the-art methods in visual fidelity and 4D consistency on DNA-Rendering and RealVV under limited-capture settings.RealVV is a self-captured dataset with varying focal lengths, diverse scenes, actors, performances, and real and synthetic degradations.Source: §1 · p. 2

Contributions

  • The paper adapts a pretrained high-resolution image diffusion model to 4D enhancement and introduces RealVV to evaluate volumetric video under challenging capture conditions.The authors state that they plan to release RealVV to the community.Source: §1 · p. 2

2 Related Work

Prior work addresses dynamic-scene reconstruction, sparse-view recovery, and generative enhancement, but the paper identifies a gap in enhancing limited captured videos while preserving their capture layout for general volumetric reconstruction.

2.1 Dense-View 4D Reconstruction

  • Dense-view 4D reconstruction systems model dynamic scenes with persistent primitives, 4D fields, or spatio-temporal features, but rely on captured frames as direct evidence and degrade with sparse or visually degraded inputs.Methods seek improved temporal scale and motion coverage or lower reconstruction cost; the input limitation motivates an explicit enhancement stage before reconstruction.Source: §2.1 · pp. 2–3

2.2 Sparse-View 4D Reconstruction

  • Sparse-view methods use geometric, learned, or human priors to recover structure or synthesize missing observations, but generally target reconstruction or category-specific rendering rather than layout-preserving enhancement for general pipelines.The surveyed methods include limited-view optimization, four-view capture, temporal undersampling, feed-forward reconstruction, and diffusion-based supervision.Source: §2.2 · p. 3

2.3 Generative Priors for 4D Enhancement

  • Video restoration and super-resolution typically enhance monocular streams, while reconstruction uses of video or image priors target frame rates, rendered outputs, or static scenes rather than joint multi-view layout.Large video-generation models offer spatio-temporal priors but require layout-aware adaptation for multi-view capture enhancement; RealMe conditions video diffusion on the 4D capture layout.Source: §2.3 · p. 3

3 Method

RealMe turns sparse, low-quality captures into auxiliary videos and adapts a high-resolution image diffusion model to enhance them across time and viewpoints. The enhanced videos are combined with the original captures to refine a 4D scene representation for volumetric video reconstruction.

3.2 Auxiliary Data Construction

  • A coarse 4DGS renders novel-view videos, while cropped regions of interest preserve local detail cues; both auxiliary video types are enhanced independently at a unified output resolution.The framework uses novel-view videos as approximate multi-view geometry guidance and ROI videos for local details, which remain limited by camera resolution.Source: §3.2 · p. 3

3.3 4D Enhancement Video Diffusion Model

  • Because frame-wise or view-wise image enhancement ignores temporal and multi-view dependencies, RealMe adapts an image diffusion model into a spatio-temporal video diffusion model.Independent denoising may introduce appearance drift across time and geometry mismatch across viewpoints.Source: §3.3 · p. 4

3.3.1 Model Architecture.

  • Condition latents and text features provide ControlNet signals to DiT blocks, which denoise noisy latents with layout-aware positional embeddings and factorized spatio-temporal attention.A VAE decoder turns the denoised latent into enhanced videos.Source: §3.3.1 · pp. 4–5, Figure 3 · p. 4

3.3.2 Layout-Aware Positional Embedding.

  • Four-dimensional layout-aware IDs distinguish tokens by frame, viewpoint, and latent-grid coordinates, addressing collisions in the original image IDs.The four axes represent temporal and spatial layout plus the two latent-grid dimensions.Source: §3.3.2 · p. 5

3.3.3 Factorized Spatio-Temporal Attention.

  • Factorized attention separates information propagation along temporal and spatial dimensions, with the spatial dimension indexing viewpoints rather than image patches.The reported ablation indicates improved 4D consistency over full attention.Source: §3.3.3 · p. 5, Figure 4 · p. 5

3.4.1 Training Data Curation.

  • RealVV training pairs clean target videos with conditions synthesized through sparse-view or underfit reconstruction, resolution, and appearance or motion degradations.The training split contains 100 sequences from 20 actors and over 1.4 million multi-view images; evaluation sequences are held out.Source: §3.4.1 · p. 5

3.4.2 Training Objectives.

  • Training combines flow matching with latent frequency supervision that targets low-frequency structure and high-frequency detail to reduce the gap between generated videos and real captures.The frequency-loss weight controls the strength of this supervision.Source: §3.4.2 · pp. 5–6

3.1 Overview

  • RealMe uses enhanced videos together with input captures to refine a 4DGS representation for volumetric video reconstruction.The framework adapts a pretrained high-resolution image diffusion model to generate those enhanced videos.Source: §3.1 · p. 3, §3.4.3 · p. 6

4 Implementation Details

RealMe adapts a pretrained high-resolution diffusion model for 2K video enhancement, using a 14B FLUX.1 base model and a fully trained ControlNet branch. Training uses mixed temporal, spatial, and joint sampling, with 4DGS reconstruction supervised by original and enhanced views.

4 Implementation Details

  • Training uses the 14B FLUX.1 model for 96 hours on eight H100 GPUs, with an effective batch size of 8 and a learning rate of 10^-4.The frequency-loss weight is 0.1; target videos are 2560×1440 and condition videos are resized to 1280×720.Source: §4 · p. 6
  • The full ControlNet branch is initialized from pretrained DiT weights and trained during adaptation, using mixed temporal, spatial, and joint sampling.Training also uses bfloat16 mixed precision and gradient checkpointing to reduce memory usage.Source: §4 · p. 6
  • Each final 4DGS is supervised by eight original capture views and 40 generated views enhanced from novel-view and ROI conditions.The 4DGS optimization is based on EasyVolcap and follows FTGS training settings.Source: §4 · p. 6

5 Experiments

Experiments evaluate RealMe on RealVV and DNA-Rendering across novel-view synthesis, 4D-consistent reconstruction, and detail enhancement. Comparisons and ablations assess visual quality, consistency, capture-view requirements, model components, and latent frequency supervision.

5.1.1 Datasets.

  • Evaluation uses seven held-out RealVV sequences at 2560×1440 and four zero-shot DNA-Rendering sequences at 1024×1024, with eight input views per sequence.Each RealVV sequence has 48 views and 180 frames per view; DNA-Rendering sequences have 120 frames per view.Source: §5.1.1 · p. 6

5.1.2 Baselines.

  • Novel-view synthesis is compared with FTGS, DIFIX3D+, 4C4D, and 4K4D; other tasks use Diffuman4D for 4D reconstruction and STAR and FlashVSR for detail enhancement.These baselines represent optimization-based, generation-based, four-camera reconstruction, high-fidelity view synthesis, and video super-resolution approaches.Source: §5.1.2 · p. 6

5.1.3 Metrics.

  • PSNR, SSIM, and LPIPS measure pixel-level, structural, and perceptual fidelity, while consistency metrics evaluate agreement with downstream 4DGS renderings.Temporal Warping Error measures residuals after flow-based warping of adjacent frames over non-occluded regions; it is treated as a stability measure, not reconstruction fidelity.Source: §5.1.3 · p. 6

5.2 Comparison Experiments

  • RealMe achieves the best visual quality among compared methods on both RealVV and DNA-Rendering, while its generated videos align better with input captures than Diffuman4D.Its temporal consistency is comparable to the optimization-based baseline and better than generation-based methods.Source: Table 1 · p. 7, §5.2 · p. 7, Table 3 · p. 7
  • The experiments compare RealMe with optimization- and generation-based methods, including a four-camera RealVV setting, and report finer natural-looking facial, hair, and clothing details at 2K.RealMe also shows stronger reconstruction performance than two 4D reconstruction baselines under the four-camera setting.Source: §5.2 · p. 7

5.3 Ablation Studies

  • Ablations evaluate model adaptations and training objectives, while varying input-view counts to examine the trade-off between capture requirements and reconstruction quality.All ablation studies use RealVV evaluation sequences.Source: §5.3 · p. 7

5.3.1 Ablation on the Number of Input Views.

  • Eight spatially distributed input views provide a practical balance: quality improves substantially from four views, with limited additional benefit at 16.The paper notes that quality-focused studio settings may favor more views, whereas outdoor captures with limited equipment favor fewer.Source: §5.3.1 · pp. 7–8, Table 5 · p. 8

5.3.2 Ablation on Model Components.

  • Removing layout-aware positional embedding lowers PSNR_c, while replacing factorized spatio-temporal attention with full attention reduces visual quality and consistency metrics.The ablations indicate that both components contribute to visual fidelity and downstream 4D consistency.Source: §5.3.2 · p. 8, Table 6 · p. 8

6 Conclusion and Discussion

RealMe enhances sparse, low-quality captures into high-resolution, 4D-consistent videos for volumetric reconstruction. Experiments report improved visual quality and 4D consistency, while training-data scarcity and insufficient geometric cues constrain the method.

6 Conclusion and Discussion

  • Image-space frequency-loss metrics are reported at 1K because 2K training runs out of memory.The ablation distinguishes image-space and latent-space frequency loss and defines LPIPS_c as its consistency metric.Source: Table 7 · p. 8
  • RealMe achieves superior visual quality and 4D consistency over existing baselines, extending reconstruction to 2K under limited capture conditions.The framework adapts a pretrained image diffusion model to temporal and spatial relationships while preserving local details.Source: §6 · p. 8
  • Auxiliary videos condition the diffusion model, while factorized spatio-temporal attention and latent frequency loss support consistent generation and reduce the gap to real captures.The model extends positional IDs with temporal and spatial indices; frequency-domain supervision is applied in latent space.Source: §6 · p. 8
  • Training data remains constrained, generalization is limited for large scenes or complex camera settings, and insufficient geometric cues may prevent valid coarse 4DGS initialization.Scaling training data and accelerating high-resolution diffusion inference remain future directions.Source: §6 · p. 8

Supplementary Material for RealMe: Immersive Volumetric Video from Minimal Captures

The supplementary evaluation uses matched inputs and common output resizing where supported, while adapting each baseline's released implementation to its intended setting.

A Baseline Settings

  • Baseline comparisons use the same input views and frame ranges when supported, official implementations and checkpoints when available, and resized outputs for metric computation.This establishes a common evaluation setup while retaining method-specific configurations.Source: A Baseline Settings · p. 13
  • FTGS optimizes a dynamic Gaussian representation from RealMe's sparse captures, then renders held-out views for novel-view metrics.The implementation uses EasyVolcap and follows the original FTGS training settings.Source: A Baseline Settings · p. 13
  • 4C4D uses four spatially distributed cameras, MASt3R initialization, and its released 30,000-iteration configuration for each sequence.Held-out-camera renders are evaluated through the common novel-view pipeline.Source: A Baseline Settings · p. 13
  • 4K4D is trained per sequence with four cameras and its official 1,600-epoch schedule adapted to RealVV's calibrated frames and camera parameters.Held-out views use the same resizing and metric computation as other methods.Source: A Baseline Settings · p. 13
  • DIFIX3D+ selects the nearest-pose input as each target view's reference, while zero-shot Diffuman4D is evaluated only inside the foreground mask.Both use official code and checkpoints; Diffuman4D is designed for human performance reconstruction.Source: A Baseline Settings · p. 13
  • STAR and FlashVSR use official checkpoints on matched frame ranges, with default-resolution outputs resized to ground truth before PSNR, SSIM, and LPIPS computation.These methods serve as video super-resolution baselines for detail enhancement.Source: A Baseline Settings · p. 13

B Additional Implementation Details

RealMe adapts FLUX.1 with a trainable ControlNet branch and parameter-efficient updates to selected pretrained DiT layers.

B.1 Model Architecture

  • A ControlNet branch injects auxiliary-video conditioning and is initialized from pretrained DiT weights, then trained in full.RealMe's video diffusion model is built on FLUX.1 and implemented with DiffSynth-Studio.Source: §B.1 · p. 13
  • Rank-32 LoRA adapters update selected DiT projections, MLP, and modulation layers while the remaining pretrained weights stay frozen.Each repeated factorized-attention schedule uses three temporal blocks followed by one spatial block, a practical trade-off because temporal modeling is more challenging in this setting.Source: §B.1 · p. 13

B.2 Training Data

Training conditions are synthesized to cover sparse-view geometry, resolution and detail loss, appearance noise, and motion blur.

B.2 Training Data

  • Sparse-view and underfit conditions come from 32, 16, or 8 sampled views or a 3,000-iteration all-view 4DGS, with rendered views used as degraded conditions.The rendered conditions are either remaining views or all views, depending on the construction.Source: §B.2 · p. 13
  • Resolution degradation uses padded face-region crops, blur and stochastic resizing, then bicubic upsampling to 1280 × 720; appearance and motion degradations add noise, compression, defocus, and motion blur.The blur pipeline uses USM sharpening and first-stage kernels from 7 to 21; the degradation set spans geometry artifacts, resolution loss, local detail loss, appearance noise, and motion blur.Source: §B.2 · p. 13

B.3 Training Strategy

The appendix specifies training, latent-frequency supervision, reconstruction, and synchronization procedures, then presents qualitative comparisons of RealMe’s outputs and high-resolution details.

B.3 Training Strategy

  • Training progresses target resolution from 1280×720 to 2560×1440 and samples temporal, spatial, and joint groups with probabilities 0.4, 0.4, and 0.2.The corresponding windows are 4 frames, 4 views, and a 2 × 2 temporal-view group.Source: §B.3 · pp. 13–14

B.4 Latent Frequency Loss

  • The latent-frequency loss uses complementary soft low- and high-frequency masks to compute branch losses, weighted by the diffusion timestep and applied to the predicted clean latent.The two weighted branches are equivalently expressed as a sum of timestep-weighted L1 terms.Source: §B.4 · p. 14

B.5 4DGS Reconstruction

  • The final 4D Gaussian scene is supervised by eight original views and 40 generated views, with enhanced ROI videos mapped back to full-frame coordinates.Novel-view and ROI conditions are enhanced separately; foreground masks are used only for the Diffuman4D-aligned evaluation.Source: §B.5 · p. 14

B.6 Input Synchronization and Efficiency

  • Annotated time offsets let the method associate unsynchronized camera frames with a common scene timeline; diffusion runtime and peak memory are measured on one 4-frame 2560 × 1440 batch.The efficiency setup uses 30 denoising steps on a single NVIDIA H100 GPU.Source: §B.6 · p. 14, Table 8 · p. 14

C.1 Qualitative Comparisons

  • Across RealVV and DNA-Rendering, RealMe preserves foreground motion and scene backgrounds, generalizes to unseen performers and views, and remains closer to ground truth on long-focus views.The comparisons also report that baselines lose scene content or introduce visible appearance artifacts.Source: §C.1 · p. 14

C.2 High-Resolution Detail Comparisons

  • RealVV detail comparisons pair full-frame results with zoomed crops to inspect facial details, clothing textures, and local motion.These qualitative examples complement the paper’s quantitative detail-enhancement metrics.Source: §C.2 · p. 14

C.3 Qualitative Ablation Results

Qualitative ablations show how the proposed components affect detail quality and 4D consistency across time and views.

C.3 Qualitative Ablation Results

  • With latent frequency loss, generated results retain sharper high-frequency structures and reduce visible detail errors.The comparison visualizes RGB results, error maps, and low- and high-frequency latent maps; the high-frequency maps highlight local textures and boundaries.Source: §C.3 · p. 14

D Dataset Visualization

The visual material covers RealVV’s capture diversity alongside qualitative comparisons across datasets, views, and latent-frequency ablations.

D Dataset Visualization

  • RealVV examples show short-focus and long-focus captures, foreground masks, and camera-rig views across varied scenes, actors, actions, baselines, and focal settings.Short-focus views capture performers and surroundings; long-focus views preserve local clothing and facial details.Source: D Dataset Visualization · pp. 14–15, Figure 11 · p. 19
  • Qualitative comparisons cover RealVV and DNA-Rendering, including short- and long-focus views, held-out viewpoints, and two DNA-Rendering evaluation sequences.The long-focus examples compare generated views and RealMe reconstructions with ground truth and competing methods.Source: Figure 7 · p. 15, Figure 8 · p. 16, Figure 9 · p. 17
  • The latent-frequency ablation compares generated RGB results with and without LFL, error maps, and corresponding low- and high-frequency latent visualizations.The latent maps show preservation of silhouettes, facial details, and clothing boundaries with LFL.Source: Figure 10 · p. 18

Related papers

Loading 2610.30023v1…