Source-linked AI summary
VideoTok4D: A 4D-Aware Video Tokenizer for Compact World Representation
Xinyi Chen, Hanxin Zhu, Xijun Wang, Xingrui Wang, Sen Liang, Xin Li, Zhibo Chen
TL;DR
Existing tokenizers largely represent videos as observed 2D sequences or static scenes, leaving dynamic scene structure and coherent motion beyond observed viewpoints unmodeled. VideoTok4D uses separate static and dynamic tokens with track-aware attention, and Co4DGen builds on that token space for conditional 4D generation. The paper reports state-of-the-art dynamic novel-view synthesis with nearly four orders of magnitude less storage than dense 4D representations.
Problem
Existing observation-centric tokenizers represent videos as 2D measurements, while static scene representations do not explicitly model temporal evolution needed to preserve scene structure and coherent object dynamics beyond observed viewpoints.
Source: Introduction · pp. 1–2Method
VideoTok4D factorizes persistent scene content and time-varying object evolution into static and dynamic tokens, uses trajectory-aware dynamic attention, and learns Co4DGen as a diffusion prior over the token space.
Source: Introduction · p. 2Results
VideoTok4D achieves state-of-the-art dynamic novel-view synthesis with nearly four orders of magnitude less storage than dense 4D representations.
Source: Introduction · p. 2Takeaways & Limitations
Co4DGen samples a joint token state once and renders multiple synchronized target-view videos without rerunning the token prior.
Source: Introduction · p. 2
Abstract
from arXiv · showhide
Video tokenizers have emerged as a cornerstone of modern video modeling, underpinning progress in compression, reconstruction and generation by mapping high-dimensional visual signals into compact latent spaces. However, despite this progress, current tokenization paradigms largely remain within the 2D visual domain, treating videos as image sequences rather than observations of an underlying dynamic 3D world. Consequently, the learned tokens inherit this observation-centric bias, limiting their capacity to compactly represent real-world 4D scenes. To mitigate this issue, we propose VideoTok4D, a novel 4D-aware video tokenizer for compact world representation. Specifically, our approach comprises three key designs: 1) a spatiotemporal disentanglement strategy that factorizes videos into static and dynamic tokens for holistic world modeling; 2) a track-aware dynamic attention mechanism that aggregates trajectory-aligned cues to promote cross-view motion consistency; and 3) Co4DGen, a diffusion prior learned over the resulting VideoTok4D token space for efficient 4D scene generation. Extensive experiments have demonstrated that our proposed method achieves state-of-the-art performance while requiring up to 4 orders of magnitude less storage than dense 4D representations. Moreover, the compact token space substantially shortens diffusion sequences, enabling efficient generation.
Introduction
Existing video tokenizers either preserve observed-view motion in view-bound representations or provide compact, view-consistent static scenes without explicitly modeling temporal evolution. VideoTok4D separates static scene content from dynamic object evolution and reports state-of-the-art dynamic novel-view synthesis with nearly four orders of magnitude less storage than dense 4D representations.
Introduction
- Observation-centric tokenizers encode dense image-time grids but remain view-bound, while 3D-aware tokenizers compactly represent static scenes without explicitly modeling temporal evolution.These paradigms therefore leave the joint preservation of scene structure and temporally coherent object dynamics beyond observed viewpoints unresolved.Source: Introduction · pp. 1–2, Figure 1 · p. 1
- VideoTok4D separates persistent scene content into static tokens and time-varying object evolution into dynamic tokens, with trajectory-aware attention for coherent motion under changing viewpoints.Co4DGen samples a joint token state conditioned on a source video and camera trajectories, enabling multiple synchronized camera trajectories to be rendered without rerunning the token prior.Source: Introduction · p. 2
- VideoTok4D achieves state-of-the-art dynamic novel-view synthesis with nearly four orders of magnitude less storage than dense 4D representations.Co4DGen also offers a favorable quality–efficiency trade-off.Source: Introduction · p. 2
Related Work
Prior video tokenizers reduce redundancy but generally encode observed clips as dense spatiotemporal grids rather than reusable scene states. Scene-centric tokenizers capture static 3D scenes, while VideoTok4D extends scene-centric tokenization to dynamic scenes.
Video Tokenization
- Conventional video tokenizers trade compression and reconstruction favorably but produce dense spatiotemporal grids optimized for observed clips, not reusable scene states across camera trajectories.Some methods reduce redundancy through latent spaces, architecture and training changes, or progressive temporal compression.Source: Video Tokenization · p. 2
- Structured-layout tokenizers factorize structure or spatial and temporal information but remain observation-centric; scene-centric LVSM and SceneTok represent static 3D scenes without temporal evolution.VideoTok4D extends this formulation with static tokens for persistent content and dynamic tokens for object evolution, aggregated along trajectories.Source: Video Tokenization · p. 2
4D Scene Reconstruction
4D reconstruction methods represent persistent scene structure and temporal evolution across viewpoints, but their states remain distributed over explicit scene representations. VideoTok4D instead compresses scene content and dynamics into a small latent token set for camera-conditioned reconstruction.
4D Scene Reconstruction
- 4D reconstruction targets persistent scene structure and temporal evolution across viewpoints, using neural fields, Gaussian primitives, per-frame pointmaps, or dense trajectories.These representations support spatially grounded novel-view rendering.Source: 4D Scene Reconstruction · p. 2
- Explicit reconstruction states remain distributed over fields, points, Gaussians, or trajectories, with size tied to resolution, duration, or scene complexity.Source: 4D Scene Reconstruction · p. 2
- VideoTok4D instead compresses scene content and dynamics into a small latent token set for camera-conditioned reconstruction.Source: 4D Scene Reconstruction · p. 2
4D Scene Generation
4D scene generation includes explicit-scene methods whose representations scale with scene detail and duration, and view-space methods that generate videos along camera paths without a reusable compact scene state. Co4DGen samples compact static and dynamic tokens once and reuses the VideoTok4D renderer across camera trajectories.
4D Scene Generation
- Explicit-scene generation methods produce persistent 4D states as neural fields or Gaussian primitives, with representations that scale with scene detail and duration.Source: 4D Scene Generation · p. 2
- View-space methods generate dense videos along specified camera paths, but their view-specific outputs lack a compact scene state reusable across queries.Co4DGen instead samples compact static and dynamic tokens once and reuses the frozen VideoTok4D renderer across camera trajectories.Source: 4D Scene Generation · p. 2
- The VideoTok4D renderer decodes static and dynamic token groups along a target camera trajectory.Source: Figure 2 · p. 3
Method
VideoTok4D learns a compact world representation by separating persistent scene content from object dynamics, then decoding its tokens under target camera trajectories. Track-aware attention and Co4DGen extend this representation to trajectory-aligned motion and conditional 4D generation.
Overview
- VideoTok4D encodes a source video and camera trajectory as static and dynamic tokens, then decodes them to synthesize views along a target camera trajectory.Unlike reconstruction of the input clip alone, its token space is learned through novel-view synthesis.Source: Overview · p. 3, Spatiotemporal Disentanglement · p. 3
Spatiotemporal Disentanglement
- Persistent scene structure and appearance become static tokens, while time-varying object evolution becomes dynamic tokens in complementary spatial and temporal branches.The spatial branch aggregates camera-aware scene context into static queries, while dynamic queries form the dynamic representation; training uses rectified flow matching along a linear path from target-video latents to Gaussian noise.Source: Spatiotemporal Disentanglement · pp. 3–4
Co4DGen: Efficient 4D Scene Generation
- Co4DGen jointly denoises static and dynamic tokens conditioned on a monocular video and anchor camera trajectory, then reuses one sampled token state across query camera trajectories.Static queries attend only within the static stream, while dynamic queries attend to both streams; the frozen renderer renders the sampled state across synchronized trajectories.Source: Figure 3 · p. 4, Co4DGen: Efficient 4D Scene Generation · p. 4
Track-Aware Dynamic Attention
- Camera-motion flow is estimated under a static-scene assumption, and thresholded residual displacement forms a motion mask that focuses the temporal branch on independently moving regions.This suppresses displacement already explained by source-camera motion.Source: Track-Aware Dynamic Attention · p. 4
- Spatial attention is restricted to masked dynamic regions, then flow-propagated tracks provide trajectory-aligned features for per-track temporal attention and dynamic-query updates.Each layer uses self-attention along each trajectory, cross-attention to track features, and an MLP.Source: Track-Aware Dynamic Attention · p. 4
Experiments
VideoTok4D is evaluated for reconstruction, storage, generation, and token correspondence, with strong dynamic novel-view synthesis quality and a compact scene state. The evaluation uses held-out dynamic scenes and controlled baselines.
Experiments
- Evaluation covers reconstruction fidelity and storage, Co4DGen generation quality and efficiency, and static- and dynamic-token correspondence biases.Source: Experiments · p. 5
Experimental Setup
- The MultiCamVideo-Dataset contains 13,600 dynamic scenes from 10 synchronized cameras, with 400 asset-disjoint scenes held out for testing.Source: Experimental Setup · p. 5
- VideoTok4D uses 1,024 static and 512 dynamic tokens of dimension 64, and Co4DGen is trained for 300K iterations on 16 A100 GPUs.VideoTok4D training uses 32-frame, 256 × 256 clips for 100K iterations on 8 A100 GPUs.Source: Experimental Setup · p. 5
- Methods share held-out scenes, source videos, target trajectories, and resolution; conditional-generation claims are limited to comparable-capacity baselines trainable or adaptable on MultiCamVideo.Representation size counts the FP32 per-sequence state for 32 frames, excluding shared model parameters.Source: Experimental Setup · p. 5, Dynamic Novel-View Synthesis · p. 5
Dynamic Novel-View Synthesis
- Nearly four orders of magnitude less storage than MoVieS, while achieving the strongest reconstruction and temporal quality and outperforming capacity-matched SceneTok†.The reported comparisons suggest structured static–dynamic tokenization is more effective than a unified token set at the same capacity.Source: Dynamic Novel-View Synthesis · p. 5
4D Scene Generation
Co4DGen provides conditional 4D scene generation from a compact sampled world state, while probes and ablations show that static and dynamic tokens have distinct but complementary roles. The experiments also support the contributions of disentanglement, motion isolation, and trajectory-aligned aggregation.
4D Scene Generation
- Co4DGen samples one 1,536-token 4D world representation and renders nine unobserved camera trajectories without target-view RGB observations.SceneGen† follows the same one-sample, nine-query protocol, whereas DFoT‡ requires nine independent dense-video diffusion runs.Source: 4D Scene Generation · p. 5
- Co4DGen achieves the best generation gFID/gFVD, Motion Smoothness, and end-to-end time, is 3.40× faster than DFoT‡, and outperforms capacity-matched SceneGen†.Figure 5 reports more consistent scene structure, appearance, and articulated motion than blurred DFoT‡ outputs and less stable SceneGen† motion.Source: 4D Scene Generation · p. 5, Table 2 · p. 6, Figure 5 · p. 7
Analysis of Static–Dynamic Tokens
- EPE on mixed tracks falls from 0.445 for the best individual token group to 0.240 with combined tokens, which achieve the lowest EPE across all subsets.Zs is more accurate on static tracks, while Zd is more accurate on dynamic tracks.Source: Analysis of Static–Dynamic Tokens · p. 6, Table 3 · p. 6
- Removing Zs drops PSNR by over 10 dB and nearly halves Motion Smoothness, while removing Zd causes smaller but consistent drops across metrics.The results support complementary specialization rather than a strict decomposition: neither token group alone provides a complete dynamic-scene representation.Source: Analysis of Static–Dynamic Tokens · p. 6, Table 4 · p. 7
Ablation Studies
- Replacing disentangled tokens with unified tokens reduces PSNR by 2.44 dB; removing motion isolation degrades results, and fixed-grid attention causes a larger drop.The ablations suggest that pose-compensated masking suppresses camera-induced displacement and trajectory-aligned aggregation matters for coherent motion encoding.Source: Ablation Studies · p. 7, Table 5 · p. 7
Conclusion
VideoTok4D compactly represents dynamic worlds beyond observed viewpoints using static and dynamic tokens, with Track-Aware Dynamic Attention to preserve coherent motion under changing viewpoints. Co4DGen samples a joint world state for synchronized target-view generation; experiments report state-of-the-art synthesis with substantially less storage than dense 4D representations.
Conclusion
- VideoTok4D represents dynamic worlds beyond observed viewpoints using a spatiotemporal disentanglement strategy that separates persistent scene content from time-varying object evolution.The Spatial Branch aggregates stable scene information into static tokens, while the Temporal Branch captures object dynamics with dynamic tokens.Source: Conclusion · p. 7
- Track-Aware Dynamic Attention compensates for camera-induced displacement and aggregates dynamic features along estimated trajectories to preserve coherent object motion across viewpoints.Co4DGen samples one joint world state and renders multiple synchronized target-view videos without rerunning the token prior.Source: Conclusion · p. 7
- Experiments report state-of-the-art dynamic novel-view synthesis with up to four orders of magnitude less storage than dense 4D representations, alongside a favorable quality–efficiency trade-off for conditional generation.Source: Conclusion · p. 7