Source-linked AI summary
A Compact Explicit 4D Representation for Dynamic Scenes
Di Yang, Zhihao Li, Yanhai Xiong, Yufei Wang
TL;DR
Dynamic-scene reconstruction needs a compact representation that retains surfaces and appearance for novel views. Sparc4D encodes monocular video as a sparse 4D state with shared static features, temporal slots, surfel geometry, and stored source texture. It reports compact state sizes, competitive reconstruction quality, and transfer to real datasets without fine-tuning.
Problem
The challenge is encoding monocular video into a compact 4D scene state that supports explicit novel-view rendering while retaining fine appearance and avoiding repeated storage of static content.
Source: §1 · p. 1Method
Sparc4D shares static features across a clip, compresses dynamic features into spatially anchored temporal slots, and decodes surfels with stored source pixels re-projected onto geometry.
Source: §3.4 · p. 3Results
Takeaways & Limitations
Temporal slots reduce time-varying feature storage from 871.9K to 249.4K values with nearly unchanged reconstruction quality, and the model transfers to DyCheck and Neu3D without real-data fine-tuning.
Source: §6 · p. 8The decoder renders dynamic occupancy at observed timestamps without transporting surfaces across time, so content hidden from the source camera at a query time is not recovered from other frames.
Source: §5 · p. 8
Abstract
from arXiv · showhide
A compact dynamic-scene representation must retain both the surfaces seen over time and the appearance needed to render them from new viewpoints. We present Sparc4D, a feed-forward autoencoder that encodes a monocular video with known cameras into a sparse 4D scene state. Static features are shared across the clip, while spatially anchored temporal slots compress time-varying features. A sparse decoder produces 2D Gaussian surfels, while stored source pixels preserve fine texture through geometric re-projection. The state includes one full source frame and dynamic-region pixels sampled every fourth frame, alongside learned features and sparse occupancy. For a 32-frame MultiCamVideo clip, it averages 0.95M 32-bit-equivalent values on random windows and 0.92M on the first-32 protocol. On first-32, Sparc4D reaches 21.70\,dB, compared with 20.40\,dB for MoVieS. On randomly placed windows, their PSNR scores are comparable. With stored texture disabled, temporal slots compress the time-varying feature state by a median $4.0\times$ and reduce the mean state from 1.04M to 0.42M values, with essentially unchanged target-view reconstruction quality. Without fine-tuning on real data, Sparc4D transfers to DyCheck and Neu3D, where stored texture improves LPIPS while slightly reducing PSNR.
1. Introduction
Dynamic-scene representations must balance compact storage with retaining appearance for explicit novel-view rendering. Sparc4D addresses this by sharing static features, compressing dynamic features into temporal slots, and pairing learned geometry with stored texture.
Motivation
- The representation question is how to encode monocular video as a compact 4D scene state that supports explicit novel-view rendering.Video tokenizers lack an explicit 3D rendering interface, while storing separate reconstructions repeats static content and coarse latents make fine appearance difficult to retain.Source: §1 · p. 1
Approach
- Static observations share one latent across the clip, while per-frame dynamic observations are compressed into spatially anchored temporal slots.Sparc4D lifts posed video into sparse 3D observations using estimated depth.Source: §1 · p. 1
- A sparse decoder reconstructs 2D Gaussian surfels, with sparse occupancy included in the representation size.A finer appearance code retains texture separately from the coarse latent.Source: §1 · p. 1
Texture storage
- One full source frame and dynamic-region pixels sampled every fourth frame are re-projected onto decoded geometry, and decoding uses only the stored state and cameras.The complete state combines learned spatial features with this sparse texture reservoir, whose cost is included in main comparisons.Source: §1 · p. 1
Contributions
- Sparc4D reports a median 4.0× reduction in time-varying feature storage with negligible reconstruction loss, plus competitive novel-view quality with a much smaller state than MoVieS and 4DGT.Its state size is comparable to C4G.Source: §1 · p. 2
2. Related Work
Prior work spans scene-specific dynamic reconstruction, feed-forward prediction, and structured scene or video compression. Sparc4D instead learns an encoder and decoder shared across scenes and combines sparse geometry with a stored-image texture path.
Dynamic scene reconstruction
- Dynamic radiance-field and Gaussian methods model motion through deformation, scene flow, image-based rendering, or regularized motion, while optimizing a scene-specific representation.Sparc4D instead learns an encoder and decoder shared across scenes.Source: §2 · p. 2
Feed-forward reconstruction
- Feed-forward methods predict 3D representations or dynamic Gaussians; MoVieS, 4DGT, and C4G are comparison methods, with C4G also fine-tuned at its original output budget.The comparison focuses on encoded scene-state size and quality, not only output-primitive count.Source: §2 · p. 2
Scene and video compression
- Video tokenizers encode sequences into spatiotemporal latents, while structured sparse representations encode 3D assets and SceneTok encodes static scenes into tokens.Sparc4D adapts the TRELLIS.2 shape autoencoder to dynamic scenes with a learned-query temporal bottleneck, then decodes surfels without generative refinement or scene completion.Source: §2 · p. 2
Image-based rendering
- Stored-image texture methods synthesize views from reference images and geometry; Sparc4D follows this principle using geometry reconstructed from its stored 4D state rather than transmitted depth maps.The evaluation also distinguishes observed from unobserved regions, following DyCheck.Source: §2 · p. 2
3. Method
Sparc4D encodes a posed monocular clip as a sparse 4D state, sharing static features while compressing dynamic features into temporal slots. It decodes Gaussian surfels for rendering and reprojects stored source pixels to retain fine texture.
3.1. Scene observations
- The 32-frame input is lifted into world-space observations using camera-conditioned depth, then separated into pooled static points and per-frame dynamic points.The method voxelizes observations and stores mean colour, normal, point offset, and octant colour residuals.Source: §3.1 · p. 2
3.2. Shared static features and temporal slots
- Static and coarse latents are stored once per clip, while dynamic features are compressed into two spatially anchored slots per occupied cell.Queries at a cell and time attend to its slots to reconstruct the latent used by the sparse decoder.Source: §3.2 · pp. 2–3
3.3. Appearance and surfel decoding
- A fine 128^3 appearance grid supplies texture features, and each decoded voxel emits four 2D Gaussian surfels for rendering.Dynamic decoding uses occupancy observed at the queried time, so novel views are supported at observed timestamps.Source: §3.3 · p. 3
3.4. Stored source-pixel texture
- The texture reservoir stores the first source frame and dynamic-region pixels every fourth frame, reprojecting valid samples to change colour but not geometry.Depth consistency determines whether a reprojected keyframe colour is used; otherwise, the surfel colour is retained.Source: §3.4 · pp. 3–4
3.5. Training
- Training uses 32-frame windows, one source camera, four supervised timestamps, and renders from the source plus four target cameras per timestamp.Image losses combine L1, SSIM, and LPIPS; depth, normal, temporal, feature-reconstruction, occupancy, and KL losses also supervise training.Source: §3.5 · p. 4
3.6. State size
- The state-size ledger counts stored features, occupancy, environment map, background colour, and texture, with occupancy estimated from octree symbols.The reported expression adds feature-grid terms, occupancy cost, environment texels, background colour, a full 256^2 RGB frame, and additional dynamic pixels.Source: §3.6 · p. 4
4. Experiments
Experiments evaluate Sparc4D’s reconstruction quality, state size, temporal compression, and zero-shot transfer. Results show competitive novel-view quality at compact state sizes, with temporal slots reducing feature storage and stored texture improving perceptual quality.
4.1. Data and protocol
- Evaluation uses 404 held-out MultiCamVideo clips and two 32-frame window protocols, reporting full-image, co-visible, and uniform-fill metrics against released baselines and fine-tuned C4G.The test scenes are excluded from training; released baseline weights are zero-shot references because their training data differ.Source: §4.1 · pp. 4–5
4.2. Novel-view reconstruction
- On random windows, Sparc4D reaches 17.29 dB with a 0.95M-value state, comparable to MoVieS; on first-32, it reaches 21.70 dB versus 20.40 dB for MoVieS.The random-window comparison is in Table 1; the first-32 comparison is in Table 2.Source: §4.2 · p. 5, Table 2 · p. 5, Table 1 · p. 6
4.3. Size–quality comparison
- On first-32, Sparc4D reaches 21.70 dB and 0.200 LPIPS with 0.92M values, about 8× smaller than single-frame MoVieS at 7.1M values.Reduced-frame, lower-resolution, and opacity-pruned MoVieS variants lose more quality at comparable sizes; reduced-frame variants also receive fewer observations than Sparc4D.Source: §4.3 · p. 5, Table 2 · p. 5
4.4. Transfer to real videos
- Without real-data fine-tuning, Sparc4D reaches 13.89 dB mPSNR on DyCheck versus MoVieS at 13.95 dB, while its mLPIPS is higher at 0.544 versus 0.488.On both DyCheck and Neu3D, additional stored texture progressively improves LPIPS while slightly reducing PSNR.Source: §4.4 · p. 5, Table 3 · p. 7
4.5. Representation analysis
- The appearance pathway improves PSNR by 0.18 dB over geometry only, and increasing appearance-code capacity adds 0.06 dB.The motion mask changes PSNR by −0.04 dB; the combined final-backbone configuration differs by only +0.01 dB from the appearance-code branch.Source: §4.5 · p. 5, Table 5 · p. 7
- Stored texture reduces LPIPS from 0.421 to 0.379 with the first frame and to 0.371 with dynamic refreshes; more frequent refreshes add only small gains while increasing state size substantially.The ablation holds the trained backbone fixed; an interval of four is used as the default size–quality tradeoff.Source: §4.5 · p. 6, Table 6 · p. 8
- Temporal slots reduce time-varying feature storage by a median 4.0× with essentially unchanged reconstruction quality; increasing slots from 2 to 10 raises PSNR by 0.04 dB on average.Dynamic-slot storage increases from 21.1K to 105.6K values per clip, motivating the two-slot configuration in the 12k-step setting.Source: §4.5 · pp. 5–6, Table 7 · p. 8
- On first-32, one keyframe raises static-region PSNR from 22.13 to 25.18 dB, while dynamic texture raises dynamic-region PSNR from 18.73 to 19.14 dB and lowers LPIPS from 0.180 to 0.153.Stored-texture benefit decreases later in the window, and source-view voxelization reduces dynamic-pixel PSNR from 27.27 to 18.12 dB before the learned per-frame decoder recovers it to 22.57 dB.Source: §4.5 · pp. 6–8, Table 8 · p. 8
5. Limitations
The decoder renders at observed timestamps using per-frame dynamic occupancy, so it cannot recover content hidden from the source camera at query time. Cross-time texture re-projection assumes locally consistent geometry, and state size varies with scene content.
5. Limitations
- Per-frame dynamic occupancy and no surface transport leave content hidden from the source camera at a query time unrecovered from other frames.Stored dynamic texture does not remove this support limitation.Source: §5 · p. 8
- Cross-time texture re-projection assumes locally consistent geometry, so depth errors and fast motion can produce incorrect texture matches.Source: §5 · p. 8
- State size varies with occupied volume and dynamic image area, and the entropy-based ledger is not an implemented codec.Source: §5 · p. 8
6. Conclusion
Sparc4D encodes monocular video as a sparse explicit 4D state with shared static features and spatially anchored temporal slots. It reduces time-varying feature storage with nearly unchanged reconstruction quality, while combining learned features and stored texture in a compact state that transfers to two datasets without real-data fine-tuning.
6. Conclusion
- 871.9K to 249.4K values reduces time-varying feature storage with nearly unchanged reconstruction quality.The reduction uses shared static features and spatially anchored temporal slots.Source: §6 · p. 8
- 0.95M values on random windows and 0.92M on first-32 are the complete representation sizes in 32-bit-equivalent values.The state combines fine-grid appearance codes and stored source texture.Source: §6 · p. 8
- Without real-data fine-tuning, the model transfers from synthetic training to DyCheck and Neu3D.Source: §6 · p. 8