PaperlayerView exact revision on arXiv

Source-linked AI summary

3DTV: A Feedforward Interpolation Network for Real-Time View Synthesis

Stefan Schulz, Fernando Edelstein, Hannah Dröge, Matthias B. Hullin, Markus Plack

arXiv:2604.11211v1cs.CVcs.LGcs.MM

TL;DR

Real-time free-viewpoint rendering must balance capture redundancy, fidelity, and latency. 3DTV addresses this with geometric camera-triplet selection and a feedforward depth-guided network, reporting strong quality-efficiency performance and real-world generalization while retaining explicit scope limitations.

  • Problem

    Real-time multi-view rendering must reduce redundant capture data while preserving fidelity under the latency constraints of interactive applications.

  • Method

    3DTV combines Delaunay-based camera-triplet selection with coarse-to-fine depth-guided feature projection and feedforward fusion without scene-specific retraining.

  • Results

    3DTV achieves a strong balance between synthesis quality and computational efficiency, outperforming recent real-time sparse-view baselines and generalizing to real-world captures.

  • Takeaways & Limitations

    The framework supports low-latency sparse-view interpolation from three cameras without per-scene optimization.

  • Takeaways & Limitations

    True real-time performance above 60 FPS requires further optimization, and the model is limited to three-view indoor-stage inputs with reduced applicability to large-scale scenes.

Abstract

from arXiv · show

Real-time free-viewpoint rendering requires balancing multi-camera redundancy with the latency constraints of interactive applications. We address this challenge by combining lightweight geometry with learning and propose 3DTV, a feedforward network for real-time sparse-view interpolation. A Delaunay-based triplet selection ensures angular coverage for each target view. Building on this, we introduce a pose-aware depth module that estimates a coarse-to-fine depth pyramid, enabling efficient feature reprojection and occlusion-aware blending. Unlike methods that require scene-specific optimization, 3DTV runs feedforward without retraining, making it practical for AR/VR, telepresence, and interactive applications. Our experiments on challenging multi-view video datasets demonstrate that 3DTV consistently achieves a strong balance of quality and efficiency, outperforming recent real-time novel-view baselines. Crucially, 3DTV avoids explicit proxies, enabling robust rendering across diverse scenes. This makes it a practical solution for low-latency multi-view streaming and interactive rendering. Project Page: https://stefanmschulz.github.io/3DTV_webpage/

1 Introduction

3DTV targets low-latency sparse-view synthesis by combining geometrically selected camera triplets with depth-guided learning, avoiding scene-specific retraining.

  • Motivation: Real-time multi-view rendering must reduce massive capture redundancy while preserving fidelity under stringent latency constraints.The challenge is especially acute for AR/VR, telepresence, and interactive video editing.
  • Motivation: Existing neural rendering methods often incur heavy computation, require per-scene retraining, and struggle with real-time or streaming scenarios.Classical image-based rendering suggests that geometrically meaningful view subsets can support faithful interpolation.
  • Method: Delaunay triangulation selects geometrically consistent camera triplets with balanced angular coverage for sparse-view interpolation.The strategy addresses limitations of heuristic nearest-neighbor selection while using only three input cameras.
  • Method: A coarse-to-fine depth pyramid progressively refines depth hypotheses, enabling geometry-aware feature projection with reduced computational overhead.The architecture adapts depth estimation to viewpoint changes and supports efficient synthesis.
  • Contribution: 3DTV combines geometric view selection, depth-guided projection, and hierarchical refinement in a feedforward architecture for real-time synthesis without retraining.The framework is designed to operate from a small set of input cameras under real-time constraints.
  • Results: Experiments report strong generalization to real-world data and competitive quality and runtime.The supplied introduction identifies this as an empirical outcome without specifying numerical values.

2 Related Work

Related work spans frame interpolation, multi-view stereo, and neural novel-view synthesis; 3DTV instead uses triangulation-driven selection and a lightweight depth-oriented network for real-time interpolation.

  • Frame Interpolation: Frame interpolation generates intermediate frames from surrounding frames to produce smooth and coherent video transitions.Prior approaches use optical flow, phase shifts, CNNs, transformers, feature correlation, and diffusion-based generation.
  • Frame Interpolation: Free-viewpoint interpolation can be treated as frame interpolation, but depth estimation replaces central optical flow to model viewpoint changes.The distinction is between temporal motion and geometric variation across camera viewpoints.
  • Multi-View Stereo: Multi-view stereo estimates dense 3D geometry from calibrated images using photoconsistency, with traditional and learned volumetric, point-based, and depth-based methods.Learning-based examples construct cost volumes or infer depth and structure directly from images.
  • Multi-View Stereo: 3DTV differs from transformer-based MVS by using triangulation-driven view selection and a lightweight depth-oriented network for real-time interpolation.Its design emphasizes computational efficiency rather than long-range feature evaluation during reconstruction.
  • Novel View Synthesis: Neural novel-view synthesis includes NeRF, accelerated variants, and 3D Gaussian Splatting, which improve reconstruction or rendering efficiency but do not uniformly satisfy real-time requirements.The related-work discussion identifies real-time capability as a distinguishing concern.

3 Method

3DTV combines Delaunay-based three-camera selection with lightweight multi-scale features, coarse-to-fine depth estimation, and depth-guided fusion for sparse-view interpolation.

  • Geometric View Selection: Delaunay triangulation selects three supporting cameras whose enclosing triangle brackets each query view and provides balanced angular coverage.The method uses cylindrical normalization and perspective mapping before identifying the enclosing triangle.
  • Efficient Feature Extraction Backbone: A Ghost-based hierarchical backbone extracts seven-level feature pyramids while reducing projection overhead through low channel counts and lightweight operations.Each level halves spatial resolution, and channel attention enhances global context with negligible computational cost.
  • Depth Estimation and Refinement: Plane-sweep stereo initializes 32 coarse depth hypotheses and recursively narrows finer-level searches around upsampled predictions.At finer levels, local windows use residual offsets whose scale is defined by ε_l, enabling sub-pixel depth accuracy with fewer planes.
  • Depth Estimation and Refinement: Depth-projected source features are compared with grouped correlations and processed with shared encoder–decoder refinement using proposed depths and propagated latents.The feature channels are partitioned into groups, and the decoder predicts depth residuals and opacity maps.
  • Hierarchical Feature Fusion and Image Synthesis: The fusion stage warps source features into the target frame, predicts per-view confidence weights, and aggregates them for hierarchical RGB synthesis.Element-wise weighting supports occlusion- and view-dependent fusion before the hierarchical decoder synthesizes the target image.
  • Loss Functions: Training includes perceptual and style supervision, with the style loss comparing Gram matrices of predicted and ground-truth VGG features.The supplied method passage also describes geometric losses for depth consistency and residual regularization.

4 Experiments

3DTV is evaluated across diverse multi-view benchmarks against offline optimization and feed-forward baselines, emphasizing the trade-off between synthesis quality, sparse inputs, and runtime efficiency. It provides competitive sparse-view quality while achieving practical real-time performance, with reduced detail on out-of-distribution scenes.

  • Evaluation setup: Evaluation covers six benchmarks spanning diverse lighting, motion, outfits, human captures, and non-human scenes, including LLFF as an out-of-distribution stress test.Methods requiring per-scene optimization are evaluated on a representative subset, and experiments generally use 1024^2 resolution.
  • Qualitative comparisons: 3DTV produces more stable geometry and preserves fine details more reliably than 2-view systems under wide-baseline interpolation.The comparison highlights fewer floating structures, duplicated geometry, and temporal-instability artifacts when synthesis uses a geometrically consistent 3-camera triplet with depth-guided projection.
  • Quantitative comparisons: Among sparse-view methods using 2–3 inputs, 3DTV achieves competitive or superior quality on human-centric datasets such as MVHumanNet and ZJUMoCap.The reported comparison indicates robustness to imperfect foreground masks and real capture artifacts.
  • Out-of-distribution robustness: On LLFF, 3DTV captures coarse geometry but lacks high-frequency detail because large depth ranges, unbounded environments, and imperfect masks fall outside its training assumptions.The method is robust enough to render these scenes but is not optimized for this setting.
  • Runtime and memory: Per-scene optimization methods achieve the strongest metrics but require multi-minute reconstruction or heavy per-scene training, limiting interactive use.This establishes the paper’s central quality–efficiency trade-off between optimized reconstruction and feed-forward rendering.
  • Runtime and memory: 40 FPS at 1024×1024 with 2.2 GB peak memory demonstrates the efficiency of the TensorRT-optimized model.OursRT outperforms GPS-Gaussian and RIFTCast in quality-to-memory ratio, with the full deployment configuration provided for reproducibility.

RIFTCast

The paper contrasts real-time view-synthesis designs through their computational structure, depth-pyramid choices, residual connections, and viewpoint behavior. 3DTV maintains stability across novel views and resolutions, while its practical scope remains bounded by input and scene constraints.

  • Architecture: 3DTV’s 7-level depth pyramid is important for wide-baseline interpolation because shallower pyramids reduce depth stability and increase memory usage.Coarse scales help the model search over larger depth ranges before refinement.
  • Architecture: Removing residual depth and feature connections causes unstable training and visible shimmering artifacts.Residual updates focus prediction on local corrections and propagate coarse geometric structure toward finer levels.
  • Generalization and viewpoint stability: Geometry-aware projection maintains high viewpoint stability and scales from 1024^2 to 2048^2 without finetuning.The reported behavior contrasts with popping or blurring outside trained bounds and preserves sharp textures at higher resolution.
  • Runtime and memory: Table 3 compares memory and speed on an NVIDIA RTX 4090 using official implementations, including TensorRT-optimized OursRT for real-time usage.The table is intended to compare computational demands across the tested approaches.
  • Limitations: The method is constrained to three-view inputs and indoor capture stages with limited applicability to large-scale scenes.The LLFF stress test reflects this boundary because its large scene scales exceed the training distribution.
  • Limitations: 3DTV targets bounded interpolation around the selected camera triplet, leaving large extrapolation beyond the camera hull as future work.Complex regions can also remain blurry, especially around high-frequency details and disocclusions.

5 Conclusion

3DTV combines geometric camera selection with depth-guided fusion for feedforward, real-time sparse-view synthesis. Experiments show a strong quality–efficiency balance, low latency, and robust generalization from synthetic training to real-world captures.

  • 3DTV enables stable wide-baseline interpolation from three input cameras without per-scene optimization.It combines Delaunay-selected camera triplets with coarse-to-fine target-view depth estimation.
  • 3DTV achieves a strong balance between synthesis quality and computational efficiency while maintaining low latency.
  • The model generalizes robustly to real-world captures and diverse scene configurations despite training solely on synthetic data.
  • Feedforward neural synthesis is presented as a promising direction for scalable real-time free-viewpoint video rendering.

Supplementary

The supplementary material expands the paper’s technical, training, dataset, baseline, and analysis documentation. It provides implementation details and additional qualitative and quantitative evidence supporting the main claims.

  • The supplement derives the Delaunay triangulation procedure and explains the network architecture.
  • It documents the training regime, including optimizer settings, loss weights, training duration, and data augmentation.
  • Dataset documentation covers source resolutions and the preprocessing pipeline.
  • Two additional baseline models are evaluated in extended comparisons, with results provided to justify their exclusion from the main manuscript.
  • Expanded visual comparisons and ablation studies provide qualitative and quantitative analysis against the primary baselines.

A Delaunay Triangulation Ablation

The triangulation ablation constructs a 3D camera mesh by projecting camera positions to a plane, applying Delaunay triangulation, and back-projecting the faces. The setup supports coverage across capture-stage geometries, but excessive projection distances create artifacts.

  • Camera positions are projected from a fitted cylinder onto a parallel 2D plane before triangulation.A ray from an origin through each camera location defines the projection.
  • Delaunay faces are back-projected into 3D and identify the source-camera triangle for each novel view.
  • The approach avoids rendering images from the floor while supporting consistently covered top-down views.
  • The same triangulation pipeline generalizes to cuboid setups by replacing the fitted cylinder with a cuboid.
  • Positioning the origin 1m below and the projection plane 1m above the cylinder yields the optimal capture-stage triangulation.
  • Excessive origin or projection-plane distances clip bottom triangles and produce thin, elongated triangles that destabilize interpolation.

B.1 Dataset Curation

The dataset and training pipeline use synthetic rendered scenes with ground-truth depth, progressive resolution training, a multi-task objective, and augmentations designed to bridge synthetic and real-world conditions. Deployment uses TensorRT optimization for real-time inference.

  • Because stable training and faster convergence require ground-truth depth, training uses synthetic data rather than real-world datasets.The synthetic set contains 24,753 samples generated from 357 3D assets and curated Blender scenes.
  • Training proceeds progressively from 512×512 for 100 epochs to 1024 × 1024 for 25 fine-tuning epochs.The two phases require approximately four days and 36 additional hours on a single NVIDIA A40 GPU.
  • The training objective combines L1, RGB, depth, offset, alpha, VGG, and style losses with phase-dependent weighting.
  • Augmentations apply photometric jitter, exposure and gamma changes, depth-conditioned background noise, blur, quantization, and stochastic scaling.
  • Real-time deployment exports PyTorch weights through ONNX and compiles a BF16 TensorRT engine for target hardware.

C Dataset and Evaluation Details

The evaluation uses standardized 1024 × 1024 inputs across multiple datasets and view configurations, alongside comparisons with alternative reconstruction methods. Excluded baselines were omitted because of artifacts, resolution constraints, or reproducibility issues.

  • Evaluation setup: Evaluations standardize all methods at 1024 × 1024 pixels, downsampling higher-resolution datasets and scaling camera intrinsics accordingly.This preserves geometric consistency after resizing.
  • Datasets: RIFTCast contains 32 dynamic scenes captured with a 35-camera rig, including complex multi-actor interactions and sequences lasting 2–20 seconds at 25 FPS.All 34 available sequences are evaluated for real-time methods, with one camera reserved for evaluation.
  • Datasets: DNA Rendering evaluation follows the RIFTCast selection protocol, using 67 scenes from Part 2 with 2448 × 2048 images downscaled to 1024 × 1024.The 5-megapixel subset uses specified cameras and holds out a test view.
  • Datasets: Additional datasets include LLFF, MVHuman, THuman2.1, and ZJUMoCap, each evaluated with dataset-specific camera, sequence, or object selections.The configurations include two- and three-view setups with held-out test cameras where specified.
  • Excluded baselines: Snap-Snap was excluded after persistent ghosting and duplication artifacts remained across diverse scenes despite camera-configuration adjustments.Limited availability of official pretrained weights also affected reproducibility.
  • Excluded baselines: FWD was excluded from the main comparison because it produced aspect-ratio distortion and blur, while adapting its 640 × 480 design to 1024 × 1024 required infeasible retraining.The final comparison therefore focused on baselines with more reliable high-resolution performance.

E More qualitative evaluations

This section presents additional visual comparisons with baseline methods and qualitative ablation studies. The evaluations examine both reconstruction outputs and the influence of individual network components.

  • Qualitative comparisons: Additional qualitative results compare the proposed method with baseline reconstruction methods.
  • Baseline comparisons: The visual comparisons include evaluations against baseline methods as part of the supplementary experiments.
  • Ablation studies: Qualitative ablations examine how individual network components influence the overall model.

F Additional Quantitative Evaluations

Additional experiments report ablations on RIFTCast and qualitative analyses of view selection, baseline failures, and architectural influences. The results support consistent module contributions across datasets.

  • Quantitative ablations: Ablation results remain consistent across datasets, with removing residual depth or prior-guided depth causing significant drops in PSNR and LPIPS.These modules are therefore associated with maintaining high-fidelity results at 1024 × 1024 resolution.
  • Quantitative ablations: RIFTCast ablations evaluate model variants with TensorRT optimization at 1024 × 1024 resolution.The table examines trade-offs involving channel depth, pyramid levels, and proposed depth modules.
  • View selection: The supplementary figures evaluate hyperparameter choices and their influence on the resulting Delaunay triangulation.
  • Baseline analysis: Qualitative figures document reconstruction failures including Snap-Snap ghosting and duplication and FWD distortion and blur.
  • Qualitative comparisons: Additional qualitative comparisons present the proposed method alongside online and offline reconstruction methods across multiple figure parts.
Loading 2604.11211v1…