PaperlayerView exact revision on arXiv

Source-linked AI summary

3DTV: A Feedforward Interpolation Network for Real-Time View Synthesis

Stefan Schulz, Fernando Edelstein, Hannah Dröge, Matthias B. Hullin, Markus Plack

arXiv:2604.11211v1cs.CVcs.LGcs.MM

TL;DR

Real-time multi-view rendering must balance redundant captures against fidelity and efficiency under low-latency demands. 3DTV combines Delaunay-based camera-triplet selection with coarse-to-fine depth-guided synthesis for feedforward sparse-view interpolation. Across multiple benchmarks, it balances quality and computational efficiency, outperforms recent real-time sparse-view baselines, and generalizes to real-world captures.

  • Problem

    Real-time multi-view synthesis must balance redundant captures against fidelity and efficiency, while dense-view methods often impose computational costs or per-scene retraining that limit streaming use.

    Source: §1 · p. 2

  • Method

    3DTV selects geometrically consistent camera triplets with Delaunay triangulation and uses coarse-to-fine depth-guided feature projection and fusion for feedforward interpolation.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    Across multiple multi-view benchmarks, 3DTV balances synthesis quality and computational efficiency, outperforms recent real-time sparse-view baselines, and generalizes robustly to real-world captures.

    Source: §5 · p. 10

  • Takeaways & Limitations

    These results position 3DTV as a low-latency option for sparse-view synthesis across real-world captures and diverse scene configurations.

    Source: §5 · p. 10

    The model is currently limited to three-view inputs and indoor capture stages, with limited applicability to large-scale scenes.

    Source: §4.4 · p. 10

Abstract

from arXiv · show

Real-time free-viewpoint rendering requires balancing multi-camera redundancy with the latency constraints of interactive applications. We address this challenge by combining lightweight geometry with learning and propose 3DTV, a feedforward network for real-time sparse-view interpolation. A Delaunay-based triplet selection ensures angular coverage for each target view. Building on this, we introduce a pose-aware depth module that estimates a coarse-to-fine depth pyramid, enabling efficient feature reprojection and occlusion-aware blending. Unlike methods that require scene-specific optimization, 3DTV runs feedforward without retraining, making it practical for AR/VR, telepresence, and interactive applications. Our experiments on challenging multi-view video datasets demonstrate that 3DTV consistently achieves a strong balance of quality and efficiency, outperforming recent real-time novel-view baselines. Crucially, 3DTV avoids explicit proxies, enabling robust rendering across diverse scenes. This makes it a practical solution for low-latency multi-view streaming and interactive rendering. Project Page: https://stefanmschulz.github.io/3DTV_webpage/

1 Introduction

3DTV addresses the tension between high-fidelity multi-view rendering and low-latency streaming by synthesizing novel views from a sparse, geometrically selected set of cameras. It combines Delaunay-based view selection with coarse-to-fine depth-guided feature projection in a feedforward framework that requires no scene-specific retraining.

1 Introduction

  • Dense neural rendering methods can be computationally costly and require per-scene retraining, while selecting a geometrically meaningful subset of views can support faithful interpolation.The challenge is especially important for AR/VR, telepresence, and interactive editing, where low latency matters.Source: §1 · p. 2
  • Delaunay triangulation selects a geometrically consistent triplet of source cameras for each novel viewpoint, providing balanced angular coverage with only three input cameras.This addresses limitations of heuristic nearest-neighbor selection and reduces redundancy.Source: §1 · p. 2
  • A coarse-to-fine depth pyramid progressively refines depth hypotheses, enabling geometry-aware feature projection and occlusion handling with reduced computational overhead.The approach replaces motion-centric flow reasoning with geometry-aware depth estimation to address correspondence and occlusion challenges under wide baselines and complex geometry.Source: §1 · p. 2
  • 3DTV combines geometric view selection, depth-guided feature projection, and hierarchical refinement in a feedforward system for real-time novel-view synthesis without scene-specific retraining.The introduction describes the method as achieving high-quality results under real-time constraints and reports competitive quality and runtime with generalization to real-world data.Source: §1 · p. 2

2 Related Work

The related work spans frame interpolation, multi-view stereo, and novel view synthesis. 3DTV draws on these areas while using target-camera depth and triangulation-driven view selection for feedforward real-time synthesis without scene-specific retraining or optimization.

2.1 Frame Interpolation

  • Frame interpolation has progressed from explicit motion estimation to learned CNN, transformer, and diffusion approaches, while optical flow remains common for modeling temporal progression.Classical methods use optical flow or phase shifts; CNN methods include [17] [18] [19] [21], and later work addresses large gaps or motions, spatial awareness and cost, non-uniform motion, and lighting changes or blur [27]. Diffusion approaches include [29], while optical flow is used in [18].Source: §2.1 · pp. 2–3
  • 3DTV treats free-viewpoint interpolation as frame interpolation but replaces central optical flow with target-camera depth to model viewpoint rather than temporal motion.This framing is inspired by Reda et al..Source: §2.1 · p. 3

2.2 Multi-View Stereo

  • Multi-view stereo evolved from volumetric, point-based, and depth-map methods to learned cost-volume and transformer approaches for depth or 3D reconstruction.MVSNet uses deep-feature cost volumes and 3D CNN regularization; follow-ups reduce memory or use coarse-to-fine volumes. Transformer methods use attention and epipolar constraints, while other work addresses robustness and non-Lambertian surfaces.Source: §2.2 · p. 3
  • Unlike transformer-based MVS focused on long-range feature evaluation, 3DTV uses triangulation-driven view selection and a lightweight depth-oriented network for real-time interpolation.The approach combines principled geometric view selection with a lightweight depth-guided feedforward network.Source: §2.2 · p. 3, §2.3 · p. 4

2.3 Novel View Synthesis

  • Novel view synthesis spans NeRF extensions, faster Instant-NGP, 3D Gaussian Splatting, sparse- or single-input Gaussian methods, and diffusion-based approaches.NeRF [2] uses neural implicit functions; Instant-NGP accelerates training and inference, and 3D Gaussian Splatting [4] enables real-time rendering. Gaussian methods address sparse inputs [59] [1], single inputs, and speed [59] [11]; diffusion methods include [66] and single- or multi-view approaches [67] [68] [6].Source: §2.3 · p. 3
  • For real-time NVS without scene-specific retraining, PixelNeRF and IBRNet remain slower than real time, while FWD, Guo et al., GPSGaussian, and GPSGaussian+ target real-time rendering.Snap-Snap produces high-quality reconstructions but lacks real-time capability. 3DTV combines principled geometric view selection with a lightweight depth-guided feedforward network and avoids scene-specific retraining or optimization.Source: §2.3 · pp. 3–4

3 Method

3DTV combines Delaunay-based camera selection with a lightweight, depth-guided network that estimates target-view depth and alpha coarse to fine, then projects and fuses features to synthesize the image. Its method also specifies hierarchical feature extraction and synthesis, multi-component supervision, and synthetic-data training.

3.1 Viewpoint Selection via Projected Delaunay Triangulation

  • Delaunay triangulation selects the three supporting cameras whose enclosing triangle brackets the query view, providing a complete angular basis for interpolation.For inward-facing cameras, centers are projected onto a fitted cylinder and then onto a plane before triangulation.Source: §3.1 · p. 4

3.2 Efficient Feature Extraction Backbone

  • At each of seven pyramid levels, a Ghost-based backbone extracts low-channel features, with channel attention and an L-ASPP module adding context.Ghost modules generate some feature maps with standard convolution and others with inexpensive depthwise operations; L-ASPP aggregates multi-scale context at the deepest level. The design follows GhostNet and GhostNetV2, with L-ASPP inspired by MobileNetV3 and DeepLab.Source: §3.2 · pp. 4–5

3.3 Depth Estimation and Refinement

  • Plane-sweep depth estimation initializes 32 hypotheses uniformly over [0.5, 8.5] meters at the coarsest level, then refines locally around upsampled depth predictions; grouped correlations and propagated latents condition depth-residual and opacity prediction.Source features are warped into the target camera frustum for each depth hypothesis; the grouped correlation volume follows.Source: §3.3 · pp. 5–6

3.4 Hierarchical Feature Fusion and Image Synthesis

  • The synthesis stage projects source features using estimated depth, predicts per-view confidence from warped features and camera geometry, and decodes fused features hierarchically into RGB.The decoder uses depth, opacity, and upsampled coarser latents at each level; its coarse-to-fine structure lets lower-resolution global structure regularize finer detail.Source: §3.4 · p. 6

3.5 Loss Functions

  • The seven-term objective combines image reconstruction, geometric depth and alpha supervision, and VGG and style perceptual losses applied hierarchically.Depth supervision is masked by the foreground mask; the VGG loss follows Johnson et al., and the style loss follows Gatys et al..Source: §3.5 · pp. 6–7

3.6 Implementation Details

  • Training uses 24,753 synthetic samples from 357 3D assets and curated Blender scenes, with target views sampled barycentrically and jittered by ±20cm along the forward axis.The model is trained on one NVIDIA A40 and targets NVIDIA RTX 4090 inference.Source: §3.6 · p. 7

4 Experiments

The experiments compare 3DTV with offline optimization and feed-forward baselines across six benchmarks, including LLFF as an out-of-distribution stress test. They assess synthesis quality, runtime, memory, and architectural trade-offs.

4 Experiments

  • Evaluation spans DNA Rendering, LLFF, MVHumanNet, RIFTCast, THuman2.1, and ZJUMoCap, using PSNR, SSIM, and LPIPS against offline and feed-forward baselines.Per-scene optimization methods are evaluated on a representative subset; experiments use 1024² resolution unless otherwise stated. Baselines include Nerfacto-big, Splatfacto-big, FrugalNeRF, ENeRF, GPS-Gaussian, GPS-Gaussian+, and RIFTCast.Source: §4 · p. 7

4.1 Qualitative and Quantitative Comparisons

  • With a geometrically consistent three-camera triplet and depth-guided projection, 3DTV preserves faces and extremities more reliably than two-view systems and remains consistent using fewer views than RIFTCast.The passage attributes two-view failures under wide baselines to depth ambiguity, floating structures, duplicated geometry, and temporal instability; RIFTCast processes all available views and full-view masks to construct a visual hull.Source: §4.1 · p. 7
  • With only 2–3 inputs, our approach achieves competitive or superior quality to prior feed-forward baselines on human-centric datasets; geometric triplet selection and lightweight depth-guided fusion improve stability and fidelity in sparse, wide-baseline regimes.Source: §4.1 · pp. 7–8
  • On out-of-distribution LLFF scenes, trained on synthetic indoor scenes with limited depth ranges, 3DTV captures coarse geometry but lacks high-frequency detail amid large depth ranges, unbounded environments, and imperfect masks.The passage presents this as a robustness test, while noting the method is not optimized for these settings.Source: §4 · p. 7, §4.1 · p. 8

4.2 Runtime and Memory Analysis

  • At 1024×1024 resolution, OursRT achieves 40 FPS with 2.2 GB peak memory and outperforms GPS-Gaussian and RIFTCast in quality-to-memory ratio.The TensorRT deployment configuration is provided for reproducibility; optimization-based methods require heavy per-scene training and do not meet real-time requirements.Source: §4.2 · p. 8

4.3 Ablations

  • The full ablation configuration provides the best balance for interactive performance; halving channel width increases speed but degrades high-frequency detail, while doubling it yields minor gains at substantially greater cost.The ablation evaluates model complexity, synthesis quality, and runtime on a held-out test set.Source: §4.3 · p. 8

RIFTCast

3DTV’s geometry-aware, depth-guided projection supports stable novel-view rendering across resolutions, while its current use is bounded by input, scene-scale, and extrapolation limitations.

THuman2.1

  • RIFTCast [9] requires all views and isolated scene content, while the other compared methods synthesize novel views from 2 to 3 views.This is a qualitative comparison across human-capture and non-human-containing scenes.Source: Figure 4 · p. 9
  • A 7-level pyramid is important for wide-baseline interpolation; shallower pyramids increase memory use and reduce depth stability by requiring larger coarse-scale depth searches.Source: THuman2.1 · p. 9
  • Residual depth updates and feature residuals focus learning on local corrections; removing them causes unstable training and shimmering, while the hierarchy propagates coarse geometry to finer levels.Source: THuman2.1 · p. 9
  • Scaling from 1024^2 to 2048^2 without finetuning, depth-guided projection preserves sharp textures where coordinate-based networks can blur during upsampling.The passage describes this projection as resolution-agnostic.Source: THuman2.1 · p. 9
  • The geometry-aware projection maintains viewpoint stability on novel views, unlike Gaussian Splatting, which can pop or blur outside trained bounds.The passage also states that the model generalizes to higher resolutions.Source: THuman2.1 · p. 9
  • Table 3 compares memory and speed on an NVIDIA RTX 4090 using official implementations, including TensorRT-optimized OursRT; Table 4 evaluates TensorRT model variants at 1024 × 1024 on DNA Rendering.Source: Table 3 · p. 10, Table 4 · p. 10

4.4 Limitations

  • True real-time performance above 60 FPS requires further optimization such as weight quantization; the model is limited to three-view inputs and indoor capture stages, with large LLFF scenes exceeding its training distribution.Source: §4.4 · p. 10
  • Interpolation is bounded around the selected camera triplet, leaving extrapolation beyond the camera hull as future work; complex-region blur may require denser features or temporal propagation.Source: §4.4 · p. 10

5 Conclusion

3DTV combines geometric camera-triplet selection with coarse-to-fine depth-guided fusion for feedforward sparse-view synthesis. Experiments report a balance of quality and efficiency, with robust generalization from synthetic training to real-world captures.

5 Conclusion

  • Delaunay triangulation selects geometrically consistent camera triplets, while a coarse-to-fine target-view depth pyramid enables stable wide-baseline interpolation from three input cameras without per-scene optimization.These components form 3DTV, a feedforward framework combining geometric view selection with efficient depth-guided fusion.Source: §5 · p. 10
  • Across multiple multi-view benchmarks, 3DTV balances synthesis quality and computational efficiency, outperforming recent real-time sparse-view baselines while maintaining low latency.Despite training solely on synthetic data, it generalizes to real-world captures and diverse scene configurations.Source: §5 · p. 10
  • Despite synthetic-only training, 3DTV predicts coherent depth and stable synthesis across camera depths and scales to higher output resolutions without retraining.Figure 5 illustrates generalization across geometry, viewpoint, and resolution.Source: Figure 5 · p. 11
  • The results suggest that combining lightweight geometric reasoning with feedforward neural synthesis is promising for scalable real-time free-viewpoint video rendering.3DTV is presented as a direction for real-time rendering under low-latency constraints.Source: §5 · p. 11

Supplementary

The supplement provides additional technical details and expanded experimental material supporting the main paper. It covers implementation, training, datasets, additional baselines, and qualitative and quantitative comparisons.

Technical Implementation

  • The supplement details Delaunay-triangulation mesh generation and the network architecture.It provides technical derivations and an architecture deep dive.Source: Supplementary · p. 17

Training Protocols

  • The training protocol specifies the optimizer, loss weightings, training duration, and data augmentation strategies.These details describe the training regime used.Source: Supplementary · p. 17

Dataset Characteristics

  • The dataset description reports source resolutions and the preprocessing pipeline.It breaks down characteristics of the datasets used.Source: Supplementary · p. 17

Extended Baseline Comparisons

  • The supplement evaluates two additional baseline models and gives results to justify their exclusion from the main manuscript.The comparisons are included for completeness.Source: Supplementary · p. 17

Qualitative & Quantitative Analysis

  • Expanded visual results compare the method with primary baselines and examine multiple ablation studies.The supplement includes qualitative and quantitative analysis.Source: Supplementary · p. 17

A Delaunay Triangulation Ablation · B Training Details

The triangulation pipeline projects camera positions onto a cylinder, triangulates their 2D projections, and back-projects the faces to select source-camera triangles. The ablation identifies optimal projection offsets and shows that excessive distances produce thin, elongated triangles and clipping artifacts.

A Delaunay Triangulation Ablation

  • Camera positions are projected onto a fitted cylinder, then rays from an Origin through each camera location reach a parallel 2D plane for triangulation.The Origin is defined as the center of the elongated base; the figure shifts it for clarity.Source: A Delaunay Triangulation Ablation · p. 17
  • Standard Delaunay faces on the 2D plane are back-projected into 3D and used to identify the source-camera triangle for each novel view.Source: A Delaunay Triangulation Ablation · p. 17
  • The triangulation avoids rendering floor images while consistently covering top-down views.Source: A Delaunay Triangulation Ablation · p. 17
  • The pipeline generalizes to cuboid camera setups by fitting a cuboid instead of a cylinder and leaving the remaining steps unchanged.Source: A Delaunay Triangulation Ablation · p. 17
  • An Origin 1m below the cylinder and a projection plane 1m above it yield the optimal triangulation for capture stages.The ablation evaluates the Origin offset and projection-plane placement.Source: A Delaunay Triangulation Ablation · p. 17
  • Excessive Origin or projection-plane distances cause bottom triangles to be clipped and produce thin, elongated triangles instead of near-equilateral ones.The resulting mesh tends to follow vertical lines, creating triangles unsuitable for stable interpolation.Source: A Delaunay Triangulation Ablation · p. 17

B.1 Dataset Curation … B.4 Deployment and Inference Optimization

Because stable training requires ground-truth depth, 3DTV is trained on synthetic scenes; the appendix describes its progressive training, robustness augmentations, and BF16 TensorRT deployment. The training and inference procedures are designed to support detail refinement and real-time use.

B.1 Dataset Curation

  • Ground-truth depth requirements led the authors to train on 24,753 synthetic samples generated from 357 3D assets and curated Blender scenes.The scenes include varied objects, human poses, custom cube arrangements, and environment-map lighting; training samples use three source views and one target view.Source: §B.1 · pp. 17–18

B.2 Optimization and Training Schedule

  • Optimization uses Adam with an initial learning rate of 10−5 and a OneCycleLR schedule that reaches 10−3 before linear annealing, with a final division factor of 50.Source: §B.2 · p. 18
  • Training used a two-stage resolution progression: 100 epochs at 512×512 took about 4 days, followed by 25 epochs at 1024 × 1024 taking about 36 hours on one NVIDIA A40 GPU.Both phases used batch size 2; the second phase refines high-fidelity textures and interpolation boundaries.Source: §B.2 · p. 18
  • The multi-task training objective combines L1, RGB, depth, offset, alpha, VGG, and style losses, with weights adjusted across phases to move from structural convergence to perceptual refinement.The specific loss-weight schedule is referenced in Table 5, but its values are not included in the supplied passages.Source: §B.2 · p. 18

B.3 Data Augmentation Pipeline

  • To bridge synthetic training and noisy real-world conditions, augmentations apply color jitter and, with 30% probability, exposure and gamma changes.Jitter settings are brightness 0.3, contrast 0.3, saturation 0.2, and hue 0.2; gamma is sampled from U(0.7, 1.3) and exposure from U(0.8, 1.0).Source: §B.3 · p. 18
  • Depth-conditioned background noise injects Gaussian noise only where depth is zero, simulating uncertainty in distant regions or missing geometry without perturbing other pixels.The motivation is to prevent overfitting to the renderings’ black backgrounds.Source: §B.3 · pp. 18–19
  • Capture artifacts are simulated with probabilistic Gaussian blur and color quantization, while stochastic scaling and cropping provide additional input variation.Blur uses a 3×3 kernel with σ ∈[0.01, 0.5] at 50% probability; quantization occurs at 30% probability, and scaling/cropping at 20%.Source: §B.3 · p. 19

B.4 Deployment and Inference Optimization

  • For inference, PyTorch weights are exported to ONNX and compiled into a BF16 TensorRT engine using trtexec’s exhaustive --best search; the model is trained in BF16 to limit precision-related performance loss.Throughput is averaged over 100 inference cycles after 10 warm-up cycles.Source: §B.4 · p. 19

C Dataset and Evaluation Details … D.2 FWD [71]

The evaluation standardizes image resolution and documents dataset-specific data and view selections. Snap-Snap and FWD are excluded from the main comparison because of persistent rendering problems or infeasible adaptation to the benchmark resolution.

C Dataset and Evaluation Details

  • All methods were evaluated at 1024 × 1024 pixels, with higher-resolution images bilinearly downsampled and camera intrinsics scaled in both dimensions.The intrinsic-matrix scaling preserves geometric consistency when image resolution changes.Source: C Dataset and Evaluation Details · p. 19

C.1 RIFTCast [9]

  • For real-time methods, evaluation covered all 34 available sequences; one camera was reserved strictly for evaluation, while offline baseline training used the first set of images from the first 10 scenes.The data use RGB images, binary masks, and background matting.Source: §C.1 · p. 19

C.2 DNA Rendering [85]

  • DNA Rendering [85] evaluation used the first 67 Part 2 scenes, with offline methods trained on the first 10 scenes using available views except test cameras.The 5-megapixel camera subset at frame 80 excluded cameras 3, 8, 19, and 40; original 2448 × 2048 images were downscaled to 1024 × 1024.Source: §C.2 · p. 19

C.3 LLFF (Local Light Field Fusion)

  • For LLFF, the evaluation used six standard scenes and selected single-level horizontal camera poses for 2-view and 3-view real-time setups.Blurry frames 4030 and 4037 from fern and 2971 from flower were removed; test views were held out, and offline baselines used all remaining views.Source: §C.3 · p. 20, Table 6 · p. 21

C.4 MVHuman [87]

  • MVHuman [87] evaluation used Part 1 sequences 100001–100010, with cameras A055 and A005 for 2-view tests, A037 added for 3-view tests, and A029 held out.The 1500 × 2048 images were downscaled to 1024 × 1024.Source: §C.4 · p. 20

C.5 THuman2.1 [88]; C.6 ZJUMoCap [89]

  • THuman2.1 [88] used the first 500 rendered objects for general evaluation, while ZJUMoCap [89] used eight specified sequences and the first 500 frames per sequence.THuman2.1 models were rendered with virtual camera extrinsics mimicking the Bonn Telebench rig and downscaled from 2664 × 2304 to 1024 × 1024; offline comparison was limited to its first 10 examples. ZJUMoCap used cameras 1 and 5 for 2-view, added camera 4 for 3-view, and held out camera 3.Source: §C.5 · p. 20, §C.6 · p. 20

D.1 Snap-Snap [73]

  • Snap-Snap [73] produced persistent ghosting and subject duplication on RIFTCast and DNA Rendering, including when rendering from input views, so it was excluded from the main comparison.Camera-configuration changes did not resolve the artifacts across scenes and subjects; limited availability of official pretrained weights also hindered a fair, stable evaluation.Source: §D.1 · p. 20

D.2 FWD [71]

  • FWD [71] showed aspect-ratio distortion and blurry results in 3-view tests, and adapting its 640 × 480 model to 1024 × 1024 required infeasible retraining.The authors therefore focused the comparison on baselines with more reliable high-resolution performance.Source: §D.2 · pp. 20–21

E More qualitative evaluations · E.1 Main Evaluations · E.2 Ablation Evaluations

This section presents additional visual results, comparisons with baseline methods, and ablation studies.

E.1 Main Evaluations

E.2 Ablation Evaluations

F Additional Quantitative Evaluations · MVHuman

The supplementary evaluations report RIFTCast ablations, triangulation hyperparameter and reconstruction-failure visuals, and MVHuman qualitative comparisons with online and offline methods.

F Additional Quantitative Evaluations

  • At 1024 × 1024, excluding residual depth or prior-guided depth significantly lowers PSNR and LPIPS, underscoring their importance for high-fidelity RIFTCast results.Table 7 evaluates variants with TensorRT optimization and highlights trade-offs involving channel depth, pyramid levels, and the proposed depth modules.Source: F Additional Quantitative Evaluations · p. 21, Table 7 · p. 22
  • Figures 7 and 8 evaluate how hyperparameter choices influence the resulting triangulation.Source: Figure 7 · p. 23, Figure 8 · p. 24
  • Snap-Snap produces ghosting and duplication when rendering a frontal RIFTCast view, and fails to fuse front and back views in DNA Rendering, duplicating geometry.Source: Figure 9 · p. 25, Figure 10 · p. 25
  • FWD fails to maintain proper object proportions and exhibits significant blurring compared with the ground truth.Source: Figure 11 · p. 25

MVHuman

Related papers

Loading 2604.11211v1…