PaperlayerView exact revision on arXiv

Source-linked AI summary

FL4N: Feed-forward Latent Model for 4D Novel View Synthesis

Eunbeen Hong, Hyeonseo Yu, Honggyu An, Jooho Son, Hyunah Ko, Soowon Son, Chaehyun Kim, Jahyeok Koo, Jisoo Kim, Junhwa Hur, Seungryong Kim

arXiv:2610.35854v1cs.CV

TL;DR

Monocular feed-forward 4D NVS must infer 3D structure and motion from video, while explicit Gaussian representations can produce temporal artifacts. FL4N learns temporal residuals on a frozen static latent model using a lightweight dynamic encoder. It reports state-of-the-art results across three synthesis settings and eight benchmarks, with the fastest runtime.

  • Problem

    Monocular feed-forward 4D NVS must infer entangled 3D structure and object motion, while explicit Gaussian representations can yield temporal artifacts.

    Source: §1 · p. 2

  • Method

    FL4N adds a lightweight dynamic encoder to a frozen pretrained static latent model, learning temporal residuals from image synthesis with binary dynamic-mask supervision.

    Source: §1 · p. 2

  • Results

    Across eight benchmarks, FL4N achieves state-of-the-art rendering quality on novel-view, cross-time, and novel-time synthesis, with the fastest runtime.

    Source: §1 · p. 2

  • Takeaways & Limitations

    FL4N generalizes to real-world videos, disentangles viewpoint from time, and preserves scene geometry.

    Source: §5 · p. 10

    The appendix passages describe qualitative view-time examples on NVIDIA and CamxTime but state no specific limitation.

    Source: G LIMITATIONS · pp. 31–32

Abstract

from arXiv · show

4D novel view synthesis (NVS) aims to render dynamic scenes across viewpoints and timestamps, which requires jointly inferring 3D structure and temporal dynamics. Existing monocular feed-forward methods often rely on explicit 4D Gaussian representations, which are prone to temporal inconsistency and severe rendering artifacts. We present FL4N, the first feed-forward latent 4D NVS model. Instead of relying on explicit 4D primitives, FL4N encodes a 4D scene prior into a unified latent representation across space and time. Our core insight is that learning temporal dynamics as residuals on top of a pretrained static latent prior enables effective 4D modeling with limited dynamic training data. To this end, we introduce a lightweight dynamic encoder that leverages static scene features to disentangle object motion from camera-induced parallax. Trained solely with an image synthesis objective, FL4N achieves state-of-the-art 4D NVS quality while rendering in real time. Its learned latent representation generalizes to real-world videos and disentangles viewpoint from time.

1 INTRODUCTION

FL4N addresses monocular 4D NVS by adding a trainable dynamic encoder to a frozen static latent model. It uses temporal residuals to model dynamics with limited data and reports strong results across three synthesis settings.

1 INTRODUCTION

  • Monocular feed-forward 4D NVS must infer entangled 3D structure and object motion, while explicit Gaussian approaches can produce temporal artifacts and over-smoothed regions.The passage identifies ghosting, floaters, and holes as examples of artifacts.Source: §1 · p. 2
  • Latent static NVS methods avoid explicit 3D representations, but extending them to dynamic scenes would require multi-view dynamic datasets that are much scarcer than static datasets.FL4N's approach is motivated by using pretrained static latent priors to reduce the dynamic-data burden.Source: §1 · p. 2
  • FL4N adds a lightweight trainable dynamic encoder to a frozen pretrained static latent model, learning temporal residuals rather than scene dynamics from scratch.The static model supplies a generalizable 3D prior, allowing the dynamic encoder to focus on temporal changes with limited dynamic data.Source: §1 · p. 2
  • Across eight synthetic and real-world benchmarks, FL4N achieves state-of-the-art rendering quality on novel-view, cross-time, and novel-time synthesis, with the fastest runtime.The evaluation reports state-of-the-art or competitive performance pose-free, and state-of-the-art performance across all three settings when context camera poses are optionally incorporated.Source: §1 · p. 2

2 RELATED WORK

Prior 4D Gaussian methods face trade-offs involving motion consistency, coverage, or restrictive camera setups. Existing latent NVS methods often ignore dynamics or constrain how time and viewpoint are controlled.

Feed-forward Gaussian splatting for 4D novel view synthesis

  • Explicit 4D Gaussian methods can disrupt temporal consistency through incoherent per-Gaussian motion, while query limits can constrain coverage, especially in large outdoor scenes.Timestamp-anchored prediction can be view-biased, and multi-view methods require synchronized static cameras rather than a monocular moving camera.Source: §2 · p. 3

Latent-based novel view synthesis

  • Many latent NVS methods assume static scenes or suppress dynamic content, while DyST keeps the scene latent time-invariant and uses target frames for temporal control.Using target-pose and target-time frames for control codes can limit explicit view and time control and risk appearance leakage.Source: §2 · p. 3

Generative novel view synthesis

  • Generative NVS can extrapolate plausible unseen content, but stochastic sampling can compromise cross-view consistency and scene faithfulness.This work focuses on deterministic NVS and treats generative refinement as orthogonal.Source: §2 · p. 3

3 METHOD

FL4N extends a frozen latent-based static NVS model with a lightweight dynamic encoder that represents temporal changes as residuals. Target-time residuals are combined with the static latent for rendering, and the analysis finds that the residual features concentrate on motion while preserving static scene content.

3.2 FL4N: FEED-FORWARD LATENT MODEL FOR 4D NOVEL VIEW SYNTHESIS

  • FL4N adapts a frozen static latent representation with target-time residuals, then uses the pretrained renderer to synthesize the requested view.The dynamic encoder processes context frames, and its residual is adapted for the target timestamp before being added to the static latent.Source: §3 · p. 3, §3.2 · p. 5

3.1 PRELIMINARIES: LATENT-BASED NOVEL VIEW SYNTHESIS

  • The static NVS baseline encodes multi-view images into a scene latent and decodes it at a query camera pose.Its latent represents a static scene; the renderer conditions the output on the target camera.Source: §3.1 · pp. 3–4

3.2 FL4N: FEED-FORWARD LATENT MODEL FOR 4D NOVEL VIEW SYNTHESIS

  • Static anchoring lets the dynamic encoder use static features to separate camera-induced changes from residual motion in dynamic objects.The dynamic encoder also receives learnable per-frame time tokens; cross-attention uses static features as its conditioning signal.Source: §3.2 · p. 4

3.3 TRAINING

  • Full-image reconstruction and perceptual losses are supplemented with dynamic-region perceptual and gradient losses, addressing weak supervision when moving objects occupy little of an image.A DPT-based mask head predicts dynamic regions and is supervised with pseudo-label masks.Source: §3.3 · p. 5

3.4 ANALYSIS ON RESIDUAL LATENT

  • The static latent suppresses moving content or freezes it at an observed frame, while target-time residuals place it at the queried time and preserve the static background.This analysis describes the residual as supplying the temporal information missing from the time-invariant static latent.Source: §3.4 · p. 5, Figure 3 · p. 6
  • PCA visualizations show camera-ray-like patterns in the static latent and temporal variation focused on moving objects in the residual latent.The residual gate is largely inactive in static regions and activates around moving objects, concentrating temporal modeling capacity where motion occurs.Source: §3.4 · pp. 5–6, Figure 4 · p. 6, Figure 5 · p. 6

4 EXPERIMENTS

FL4N is evaluated across novel-view, cross-time-view, and novel-time synthesis, with additional ablations, runtime comparisons, qualitative results, and latent probes. It outperforms pose-free Gaussian-based baselines across the protocols, while context poses further improve results.

4 EXPERIMENTS

  • Training uses synchronized multi-view videos, with asynchronous view-time inputs and targets sampled from held-out pairs to train separation of temporal dynamics from viewpoint changes.Evaluation includes NVS and CTVS on fixed-camera multi-view datasets and NTS on moving-camera monocular videos; Figure 6 marks context frames in red and evaluation targets in blue.Source: §4 · pp. 6–7, Figure 6 · p. 7

4.1 RESULTS

  • Across the three protocols, FL4N leads pose-free baselines overall and with context poses achieves state-of-the-art performance across all benchmarks; novel-time synthesis shows a large margin over all methods.The pose-free result has an exception for SSIM on NVS of two real-world datasets; the captions report FL4N outperforming Gaussian-based methods across NVS and CTVS benchmarks.Source: Table 1 · p. 7, Table 2 · p. 7, §4.1 · p. 8
  • Static-feature anchoring improves overall accuracy, dynamic-mask loss modestly improves accuracy, and freezing the renderer is crucial to avoid destabilizing the latent-to-image mapping.Without the dynamic-mask loss, the model tends to collapse as training progresses; anchoring helps the residual branch isolate temporal variation.Source: §4.1 · p. 8, Table 4 · p. 8
  • On the NVIDIA dataset at 224 × 224 resolution, FL4N is 1.08× faster than NeoVerse and 565× faster than NeoVerse with pose alignment, while achieving substantially better accuracy.The runtime comparison uses a single RTX 5880 Ada GPU.Source: §4.1 · p. 8, Table 5 · p. 8
  • Gaussian-based methods show holes, object inconsistencies, and blurred details at novel views and timestamps, whereas FL4N produces visually consistent results.Figure 7 presents this qualitative comparison.Source: §4.1 · pp. 8–9, Figure 7 · p. 9

4.2 LATENT PROBING ON DOWNSTREAM TASKS

  • Dynamic latents outperform static latents in zero-shot matching on DAVIS, with the accuracy gap slightly wider on dynamic regions.The evaluation follows the protocol of Nam et al. (2026).Source: §4.2 · p. 9, Table 6 · p. 9
  • Dynamic latents consistently outperform static latents on video depth across full frames and dynamic regions, with larger gains at greater temporal strides.The probing uses DPT predictors trained on Kubric; the authors interpret the gains as preserving geometry-aware information while adding temporal cues.Source: §4.2 · pp. 9–10, Table 7 · p. 9

5 CONCLUSION

FL4N renders dynamic scenes from monocular video at novel views and timestamps by learning temporal residuals over a frozen static latent model. Its dynamic encoder accesses static features, while a time module conditions the residual for the queried timestamp.

5 CONCLUSION

  • FL4N renders dynamic scenes from monocular video at novel views and timestamps, learning a 4D representation from limited dynamic data with image and binary-mask supervision.Across eight benchmarks and three protocols, it outperforms prior feed-forward Gaussian-based methods, generalizes to real-world videos, and its latent space disentangles view from time while preserving scene geometry.Source: §5 · p. 10

A.1 ARCHITECTURE DETAILS

  • The frozen static prior uses a 24-layer transformer encoder and a 12-layer transformer renderer, both with hidden size 768 and 12 heads on 8 × 8 patches.Both components remain frozen during training.Source: §A.1 · p. 16
  • The 8-layer dynamic encoder cross-attends to static features, while a time module conditions the residual latent for the target timestamp.The dynamic encoder has one-third the depth of the static encoder; its DPT decoder predicts a binary dynamic mask. The time module uses a learnable per-view time token and a zero-initialized residual gate.Source: §A.1 · p. 16

A.2 TRAINING DETAILS

FL4N is trained on synchronized multi-view video datasets using sampled view-time pairs, pose dropout, image-based losses, and pseudo dynamic masks derived from optical flow.

A.2 TRAINING DETAILS

  • Training runs for 32k steps on four RTX 5880 Ada GPUs with batch size 8, using 8 conditioning views and 12 target views at 336 × 512 resolution.AdamW uses learning rate 3.5 × 10^-5, weight decay 0.05, and 3k warmup steps; the static encoder and renderer remain frozen.Source: §A.2 · p. 16
  • Kubric4D and CamxTime provide synchronized multi-view videos captured by static cameras; to disentangle camera and object motion, the model reconstructs target frames at view–time pairs absent from the input set.During training, conditioning camera tokens are randomly zeroed with probability 0.4 to support pose-free conditioning at inference.Source: §A.2 · p. 16
  • Pseudo dynamic masks come from thresholding optical-flow magnitude between frames from the same static camera, where camera-induced motion and parallax are absent.Dynamic-region losses use mask-selected moving pixels on 128 × 128 crops, and the mask loss is computed only for context frames; off-the-shelf segmentation models can replace flow-based masks.Source: §A.2 · p. 16

B.1 EVALUATION DATASETS

The evaluation uses eight datasets across novel-view, cross-time-view, and novel-time synthesis, with held-out scenes or unseen datasets testing generalization. For static-camera multi-view datasets, moving monocular video is simulated by selecting diagonal camera-time frames.

B.1 EVALUATION DATASETS

  • The benchmark uses five datasets for NVS and CTVS and three for NTS; several evaluate cross-dataset generalization on unseen training datasets.Kubric4D and CamxTime use held-out scenes, while the remaining datasets are unseen during training.Source: §B.1 · p. 17
  • Static-camera multi-view datasets simulate moving monocular video by selecting diagonal frames, pairing camera i with time i.The Charge cameras are reordered along a continuous path before uniform sampling; CTVS uses the first eight cameras and NVS holds out C02.Source: §B.1 · p. 17
  • Kubric4D evaluation uses 100 test scenes, 12 horizontal-ring cameras, and a frame interval of 8; CamxTime uses Trajectory 01 and holds out the last 10% of scenes.Kubric4D excludes four top-down cameras; CamxTime has four trajectories of 120 cameras and 120 frames per camera.Source: §B.1 · p. 17

B.2 EVALUATION PROTOCOLS

The protocols separately test novel-view, cross-time-view, and novel-time synthesis. The model-specific setup uses predicted camera poses for real-world data and gives pose-free baselines test-time pose alignment, unlike FL4N.

B.2 EVALUATION PROTOCOLS

  • NVS holds out the fifth of nine frames as a target and uses the other eight as input, with the target viewpoint queried at all conditioning times.For N3DV, cam0 is the target camera.Source: §B.2 · p. 17
  • CTVS evaluates unseen viewpoint–timestamp combinations by targeting off-diagonal cells after providing diagonal frames as input.N is 12 for NVIDIA, 17–20 for N3DV, and 8 for Kubric4D, CamxTime, and Charge.Source: §B.2 · p. 17
  • NTS follows the C4G protocol, using one moving camera for context and targets and evaluating 15 interleaved frames sampled from 16 frames at stride 6.The protocol specifies scenes from each dataset.Source: §B.2 · p. 17
  • The static latent primarily encodes camera pose, while residual features grow more pronounced for dynamic objects as their positions diverge from the anchor time.The static-latent visualization shows similar features for similar viewing directions; the residual visualization indicates larger feature variation with larger object motion.Source: §B.2 · p. 18

B.3 MODEL SPECIFIC CONFIGURATIONS

  • Pose-free baselines receive 2,000-step test-time pose alignment, while FL4N does not; real-world FL4N evaluations use VGGT-predicted camera poses.For NVIDIA, estimated intrinsics are used; for N3DV, estimated intrinsics and extrinsics are used.Source: §B.3 · p. 18

C LATENT ANALYSIS

The analysis links static and residual latents to viewpoint and timestamp-dependent scene variation, and reports stronger dynamic-region results and qualitative consistency. Additional analyses examine dynamic masks and generalization to longer context sequences.

C LATENT ANALYSIS

  • Static-latent features vary smoothly with camera motion, while residual features increase with timestamp-dependent object displacement from the anchor frame.Similar static features correspond to similar viewing directions; larger residual variations correspond to larger object motion.Source: C LATENT ANALYSIS · p. 18

D.3 ADDITIONAL QUALITATIVE RESULTS

  • Qualitative comparisons show more coherent renderings than Gaussian-based methods and stronger temporal consistency with fewer visual artifacts.The comparisons include 4D NVS and cross-time view grids.Source: Figure C · p. 19, §D.3 · p. 20

D.1 QUANTITATIVE COMPARISON ON DYNAMIC REGIONS

  • FL4N consistently outperforms baselines on dynamic-region metrics for NVS and CTVS on Kubric4D and Charge.These metrics are computed only on moving pixels over the same target subset; Table A reports the dynamic-region results.Source: §D.1 · p. 19, C LATENT ANALYSIS · p. 19

D.2 ADDITIONAL ANALYSIS ON QUANTITATIVE RESULTS

  • NeoVerse performs poorly on Kubric4D, where large camera gaps and weak temporal information leave unseen regions, including background, largely unmodeled.The analysis attributes the difficulty to gaps spanning nearly the full viewing range.Source: §D.2 · p. 19

D.4 DYNAMIC MASK

  • Dynamic-mask evaluation reports substantially stronger segmentation results than other 3D or 4D geometry methods, with cleaner boundaries and greater robustness on unseen real-world datasets.The reported metrics include IoU and Boundary F-measure, plus their corresponding recall metrics; Figure D compares mask predictions with VGGT4D.Source: §D.2 · p. 20, §D.4 · p. 20, Figure D · p. 20

D.5 CONTEXT FRAME NUMBER ABLATION

  • Rendering quality remains on a flat plateau as inference context grows from the 8-frame training configuration to 32 or more frames.The CamxTime evaluation varies the number of conditioning frames while covering the same camera and temporal extent.Source: §D.5 · pp. 20–21

D.6 ARCHITECTURAL ABLATION

Architectural ablations favor preserving the pretrained renderer and static-feature anchoring, and report the 8-layer dynamic encoder as the best tested depth. Additional probes cover video depth, point tracking, and zero-shot matching.

D.6 ARCHITECTURAL ABLATION

  • The 8-layer dynamic encoder achieves the best overall tested performance; unfreezing the renderer, using a single branch, removing dynamic-mask loss or static anchoring, and sinusoidal time embeddings underperform the full model.The ablation spans temporal strides 1, 2, 4, and 8; unfreezing the renderer degrades performance.Source: §D.6 · p. 21
  • Initializing the dynamic encoder from static encoder weights consistently improves performance across temporal strides, while random initialization remains competitive.Table C compares encoder depth and initialization.Source: §D.6 · p. 21
  • On DAVIS, the dynamic latent has better zero-shot matching accuracy than the static latent at all reported thresholds.The appendix also reports point-tracking performance across temporal strides and video-depth probing on Kubric, with larger gains at larger temporal gaps.Source: Table D · p. 22, §D.6 · p. 22

E.1 FULL PROBING RESULTS

The probing, tracking, depth, and pose-alignment analyses examine what FL4N’s dynamic features capture and how pose alignment affects comparisons. Dynamic features improve point tracking and depth prediction, while FL4N without alignment remains ahead of both aligned baselines on every reported metric.

E.1 FULL PROBING RESULTS

  • Tables D and E contain the full probing results; the main paper reports a subset of matching thresholds and depth across temporal strides.Source: §E.1 · p. 22

E.2 POINT TRACKING

  • Dynamic features substantially improve moving-point tracking over static features, with the gap widening at larger temporal strides.The analysis uses a CoTracker3 head trained on Kubric and evaluated on DAVIS; the authors interpret the widening gap as evidence that residuals capture temporal information useful under larger object displacement.Source: §E.2 · p. 22, §E.3 · p. 23

E.3 VIDEO DEPTH ESTIMATION

  • Dynamic features yield sharper boundaries, more coherent within-object depth, and better background depth estimates than static features.The qualitative comparisons show noisier depth maps from static features.Source: §E.3 · pp. 22–23

F TEST-TIME POSE ALIGNMENT FOR LATENT MODELS

  • Pose-free models can have scale misalignment, so evaluation aligns target poses; for Gaussian baselines, the same Gaussians remain fixed while poses are optimized.FL4N’s cached scene latent is likewise held fixed while the renderer re-renders under updated poses; alignment uses 2,000 Adam steps with pixel and perceptual losses.Source: F TEST-TIME POSE ALIGNMENT FOR LATENT… · pp. 23–24
  • 5.94 dB on ADT and 5.13 dB on TUM for C4G, versus 3.59 dB and 2.55 dB for FL4N, are the gains from pose alignment; unaligned FL4N remains ahead of both aligned baselines on every metric.Pose alignment fits each target pose to its ground-truth target image; the authors note that the recovered gain measures camera error rather than reconstruction quality.Source: F TEST-TIME POSE ALIGNMENT FOR LATENT… · p. 24

G LIMITATIONS

FL4N inherits the static latent backbone’s difficulty with unseen extrapolated viewpoints, and its residual formulation has limited capacity when the scene representation or camera pose is substantially incorrect. Qualitative comparisons also report rendering-quality and view-time disentanglement advantages over other methods across several datasets.

G LIMITATIONS

  • Unseen extrapolated viewpoints can produce blurry or collapsed renderings because latent NVS models cannot synthesize unseen content.The paper contrasts this with explicit Gaussian representations, where observed scene portions remain usable even when extrapolation leaves holes in unseen regions.Source: G LIMITATIONS · p. 24
  • Residual correction is naturally limited when the underlying scene representation or camera pose is substantially incorrect.The authors state that stronger static NVS models with broader scene coverage and more robust viewpoint understanding can alleviate these limitations.Source: G LIMITATIONS · p. 24
  • Qualitative comparisons report better rendering quality than 3D-Gaussian methods on NVIDIA, N3DV, ADT, and Charge, and better view-time-grid rendering with clear disentanglement on NVIDIA, N3DV, and Charge.Source: G LIMITATIONS · pp. 25, 28, 29, Figure I · p. 25, Figure L · p. 27
Loading 2610.35854v1…