PaperlayerView exact revision on arXiv

Source-linked AI summary

Bi-FlowGS: Bridging Generative View Completion and Gaussian Geometry through Bidirectional Flow Co-Refinement

Yuetong Wang, Jinsheng Quan, Yi Yang, Yawei Luo

arXiv:2609.17039v1cs.CV

TL;DR

Sparse-view 3DGS reconstruction is underconstrained, and existing approaches leave a gap between unseen-view completion and explicit geometry supervision. Bi-FlowGS uses optical flow to connect restored videos and Gaussian geometry in a bidirectional refinement loop. It reports consistent gains in rendering quality and geometric consistency, including improved results across datasets and sparsity levels.

  • Problem

    Sparse-view reconstruction methods either remain largely limited to observed-view geometry regularization or use completed views mainly as RGB supervision, leaving video temporal priors underused for explicit geometry supervision.

    Source: §1 · p. 2

  • Method

    Bi-FlowGS aligns restored-video teacher flow with differentiable geometry-induced flow for geometry supervision, and feeds 3DGS-derived flow and geometric cues back into video restoration.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    Bi-FlowGS ranks first in PSNR across nine settings on Tanks and Temples, DL3DV-Benchmark, and CO3D, with an average gain of 0.71 dB, and achieves best SSIM and LPIPS in eight of nine settings each.

    Source: §4.2 · p. 6

  • Takeaways & Limitations

    Experiments show consistent gains in rendering quality and geometric consistency, with V2G generalizing across reconstruction frameworks.

    Source: §5 · p. 9

Abstract

from arXiv · show

Sparse-view 3D scene reconstruction with 3D Gaussian Splatting (3DGS) is inherently underconstrained. Plausible renderings can also coexist with erroneous Gaussian geometry, as errors in positions or depths may be concealed by opacity, scale, and appearance; we term this failure mode Geometry Cheating. Existing regularization methods constrain geometry but remain limited to observed views, while video-diffusion-based methods complete unseen views yet mainly use them as RGB pseudo-supervision, underusing motion and temporal priors and lacking explicit geometry supervision. We present Bi-FlowGS, which uses optical flow to bridge generative view completion and Gaussian geometry regularization. Our plug-and-play Video-to-Geometry Flow Distillation (V2G) distills temporal correspondence priors from restored videos into Gaussian geometry to alleviate Geometry Cheating. Conversely, Geometry-to-Video Flow-Guided Restoration (G2V) uses the current 3DGS geometry to guide temporally consistent video restoration, providing more reliable generative supervision. Together, V2G and G2V form an implicit bidirectional co-refinement process, enabling restored videos and the optimized 3DGS scene to iteratively improve each other. Experiments demonstrate improved rendering quality and geometric consistency across wide-baseline and unbounded 360{\deg} benchmarks.

1 INTRODUCTION

Sparse-view 3DGS can produce plausible renderings despite incorrect geometry, while existing regularization and generative completion leave a gap between geometry supervision and unseen-view completion. Bi-FlowGS bridges these directions through optical flow and bidirectional co-refinement.

Motivation

  • Sparse-view regularization remains tied largely to observed views, while generative methods mainly use completed views as RGB pseudo-supervision and leave temporal priors underused.Plausible RGB can also mask inaccurate Gaussian positions or depths through opacity, scale, and appearance compensation, a failure mode termed Geometry Cheating.Source: §1 · p. 2

Contributions

  • V2G aligns restored-video teacher flow with differentiable geometry-induced flow, distilling temporal-motion and implicit geometric priors into Gaussian geometry.This provides explicit geometry-optimization supervision to alleviate Geometry Cheating.Source: §1 · p. 2
  • G2V injects cross-view flow and geometric cues from the current 3DGS scene into video diffusion, while reference features preserve fine-grained texture details.Together, V2G and G2V form an implicit feedback loop in which video supervision and geometric guidance mutually strengthen over iterations.Source: §1 · p. 2

2 RELATED WORK

Prior sparse-view reconstruction work regularizes observed inputs or synthesizes unseen views, but these directions do not fully combine unseen-view supervision with explicit geometric constraints. Bi-FlowGS is situated between these approaches.

Conventional Sparse-View 3D Reconstruction

  • Sparse-view reconstruction methods use semantic, frequency, depth, structural, or Gaussian regularization, or predict Gaussian representations directly, but do not explicitly complete unseen viewpoints for additional supervision.These approaches improve reconstruction under sparse observations.Source: §2 · p. 3

Diffusion-Based Generative View Completion

  • Diffusion-based methods synthesize or refine unseen views, and video-based approaches alternate generation or restoration with 3D reconstruction, but generated views typically serve as RGB or pseudo-view supervision.Their temporal correspondences remain underexploited for geometry optimization.Source: §2 · p. 3

Direct Geometric Constraints for Sparse-View 3DGS

  • Direct geometric-constraint methods strengthen 3DGS supervision through novel-view feature matching or pretrained matching flow, but remain tied to observed-view correspondences or matching priors.These methods do not exploit generatively completed views.Source: §2 · p. 3

3 METHOD

Bi-FlowGS couples video restoration and Gaussian geometry optimization through optical flow. Restored videos supervise geometry with pseudo-views and teacher flow, while the optimized scene guides subsequent restoration.

3 METHOD

  • Restored-video frames provide pseudo-view supervision, and their extracted optical flow serves as teacher flow aligned with differentiable geometry-induced flow to supervise Gaussian geometry.Repeating the two directions lets the restored video and 3DGS scene iteratively improve each other.Source: §3 · p. 3

3.1 VIDEO-TO-GEOMETRY FLOW DISTILLATION

  • V2G aligns teacher flow from restored videos with differentiable geometry-induced flow, distilling temporal correspondence priors into Gaussian geometry.The reliability-aware loss weights pixels using teacher-flow validity, geometric validity, and teacher-flow confidence.Source: Figure 2 · p. 4, §3.1 · p. 4

3.2 GEOMETRY-TO-VIDEO FLOW-GUIDED RESTORATION

  • G2V uses rendered-sequence flow features for motion-consistent propagation and flow-derived correspondence biases for reference retrieval, alongside endpoint DINOv2 features.The design combines global appearance consistency with flow-guided local correspondence and strengthens reference conditioning in geometrically unreliable regions.Source: §3.2 · p. 5

3.3 RESTORED-VIDEO FEEDBACK OPTIMIZATION

  • The feedback objective trains on real and restored views, while restored videos provide pseudo-view and teacher-flow supervision and optimized 3DGS supplies more accurate flow guidance for subsequent restoration.This forms an implicit bidirectional co-refinement loop.Source: §3.3 · p. 5

4 EXPERIMENTS

Experiments evaluate Bi-FlowGS across sparse-view benchmarks and report rendering, geometric-consistency, and ablation results, including cross-framework V2G gains and evidence of bidirectional co-refinement.

4.1 EXPERIMENTAL SETUP

  • Fine-tuning uses 800 DL3DV-10K scenes and 49-frame clips; evaluation spans 18 DL3DV-Benchmark, 9 Mip-NeRF 360, 6 Tanks and Temples, and 16 CO3D scenes.Reconstructions are optimized from 3, 6, or 9 views; training uses aligned real videos and rendered clips.Source: §4.1 · pp. 5–6

4.2 COMPARISON WITH STATE-OF-THE-ART METHODS

  • Shared COLMAP-track reprojection reveals that plausible renderings can conceal geometry errors, while qualitative comparisons show better preservation of scene details across three benchmarks.Figure 3 pairs novel-view renderings with SfM-track reprojection maps, where lower error indicates better geometric consistency.Source: Figure 3 · p. 7, Figure 4 · p. 7, §4.2 · p. 7

4.3 ABLATION STUDIES

  • 10.70%–47.77% relative geometry gains are reported for V2G across GenFusion, ViewCrafter, and GSFixer; the full model also reports gains for 3, 6, and 9 views.The full-model PSNR gains are 0.26, 0.35, and 0.29 dB, with an additional cost of 21.71–26.51 ms/step.Source: §4.3 · pp. 7–8, Table 3 · p. 8
  • Removing V2G drops PSNR by 0.35 dB and SSIM by 0.018; DINO and FlowCond contribute texture-preservation and motion-guidance gains, while joint V2G and FlowCond achieves the best AbsRel and PSNR on Bonsai.On Ignatius, V2G improves both F-score and PSNR; the full model also shows sharper details and lower SfM-track reprojection errors.Source: §4.3 · pp. 8–9, Figure 6 · p. 9, Figure 7 · p. 9
  • λV2G = 0.014 gives the best overall rendering quality, improving PSNR by 0.14 and 0.10 dB over weights 0.007 and 0.028, respectively.A larger weight slightly improves geometry but degrades PSNR and SSIM; 0.014 is used in all experiments.Source: §4.3 · p. 9
  • Against FDS on six Tanks and Temples scenes, V2G achieves better rendering metrics and Surface F-score; Surface F-score is reported for the four scenes with ground-truth point clouds.The comparison attributes the differing supervision sources to FDS's observed-view correspondences versus V2G's restored-video motion and temporal correspondence priors.Source: Figure 8 · p. 9, §4.3 · p. 9

5 CONCLUSION

Bi-FlowGS couples video restoration and Gaussian geometry optimization through optical flow, using restored videos to supervise geometry and the optimized scene to guide subsequent restoration. The supplied details describe the G2V conditioning, trajectory sampling, and V2G supervision used in this process.

5 CONCLUSION

  • The implicit co-refinement loop uses restored videos for pseudo-view and teacher-flow supervision, while optimized 3DGS geometry guides later video restoration.The paper reports gains in rendering quality and geometric consistency, with V2G generalizing across reconstruction frameworks.Source: §5 · p. 9

A DETAILS OF THE GEOMETRY-TO-VIDEO FLOW-GUIDED RESTORATION MODEL

  • G2V combines rendered-video, Flow-Geometry, endpoint-reference, and text conditions; its flow cues encode depth consistency, depth gradients, and adjacent-frame motion.Endpoint DINOv2 features provide semantic and appearance anchors, while flow guides local feature retrieval.Source: A DETAILS OF THE GEOMETRY-TO-VIDEO… · pp. 13–14
  • The G2V model is trained on 800 DL3DV-10K scenes for 10,000 steps, unfreezing selected Transformer layers after the first 2,000 steps.The VAE and diffusion Transformer are frozen in the first stage; self-attention and cross-normalization layers are additionally unfrozen in the second.Source: A DETAILS OF THE GEOMETRY-TO-VIDEO… · p. 14

A.1 VIDEO SAMPLING TRAJECTORIES

  • G2V uses elliptical trajectories for Mip-NeRF 360 and Tanks and Temples, and bounded circular trajectories for CO3D and DL3DV-Benchmark.Trajectories stay within observed camera coverage for CO3D and DL3DV-Benchmark; phase varies across rounds to sample complementary viewpoints.Source: §A.1 · p. 14

B ADDITIONAL DETAILS OF VIDEO-TO-GEOMETRY DISTILLATION

  • V2G samples adjacent restored-video frame pairs and uses forward flow as teacher supervision, filtering unreliable correspondences with validity and cycle-consistency checks.The V2G weight is warmed up over 100 valid updates and capped at 0.9 times the current base reconstruction loss.Source: B ADDITIONAL DETAILS OF… · pp. 14–15
  • The V2G loss compares teacher and geometry-induced flow with Smooth-L1, while target depth masks invalid, out-of-bound, and occluded correspondences.The stated settings are γ = 1.5 and λV2G = 0.014.Source: B ADDITIONAL DETAILS OF… · p. 15

B.1 SPARSE-DEPTH GEOMETRY METRICS AND REPROJECTION VISUALIZATION

The section evaluates sparse-depth accuracy and cross-view consistency using COLMAP/SfM points, with shared tracks and validity rules intended to keep method comparisons comparable. It also specifies a reprojection visualization and a relative-improvement aggregate across eight geometry metrics.

B.1 SPARSE-DEPTH GEOMETRY METRICS AND REPROJECTION VISUALIZATION

  • Mip-NeRF 360 geometry is evaluated against COLMAP/SfM sparse-point depths, using projected camera-space depths as references and bilinearly sampled rendered depths as predictions.The same SfM tracks and color scales are used across methods in the reprojection visualization.Source: §B.1 · pp. 15–16
  • Evaluation retains the intersection of valid points across methods and excludes the farthest 15% by reference depth, focusing on near- to midrange geometry.A point must have finite positive depths and rendered opacity above 0.05; distant points may have weak parallax and large depth outliers.Source: §B.1 · p. 15
  • Geo. Avg. averages relative improvements over eight geometry metrics, reversing the improvement direction for lower-is-better versus higher-is-better metrics.The lower-is-better metrics are AbsRel, SqRel, RMSE, RMSE-log, and MedianRel; the higher-is-better metrics are δ1, δ2, and δ3.Source: §B.1 · p. 15

C COMPARISON PROTOCOL

  • Baselines and V2G variants share sparse views, poses, video trajectories, frame-pair poses, and test views, with V2G added without changing baseline pipelines or optimization settings.ViewCrafter is adapted for iterative 3D Gaussian Splatting optimization with the given camera poses.Source: C COMPARISON PROTOCOL · p. 16

D ADDITIONAL QUALITATIVE RESULTS

The additional qualitative figures show restoration and reconstruction renderings across datasets, alongside comparisons with other methods and ground-truth views.

D ADDITIONAL QUALITATIVE RESULTS

  • Figure 9 compares rendered video before and after restoration, then shows the final reconstruction and ground-truth views across datasets.Each row group presents these views in sequence.Source: Figure 9 · p. 17
  • Figures 10 and 11 use the same row-group sequence: rendered video before and after restoration, followed by final reconstruction and ground-truth views.Both figures show results across multiple datasets.Source: Figure 10 · p. 18, Figure 11 · p. 19
  • Figure 12 compares ViewCrafter, GenFusion, Difix3D+, GSFixer, Bi-FlowGS, and ground-truth views across additional examples.Source: Figure 12 · p. 20

E ADDITIONAL ABLATION TRAINING-PROCESS VISUALIZATIONS

The section describes two geometry F-score evaluations and their training-process visualizations. One measures reconstructed surfaces, while the other compares Gaussian centers directly with the reference geometry.

E ADDITIONAL ABLATION TRAINING-PROCESS VISUALIZATIONS

F ADDITIONAL QUANTITATIVE RESULTS

The appendix reports scene-wise rendering results for complete Bi-FlowGS across benchmarks and input-view settings, including Mip-NeRF 360 results at three view counts.

F ADDITIONAL QUANTITATIVE RESULTS

F.1 MIP-NERF 360

  • Table 5 reports per-scene rendering results on Mip-NeRF 360 with 3, 6, and 9 input views.Source: Table 5 · p. 22

F.2 PLUG-AND-PLAY V2G PER-SCENE RESULTS

The appendix evaluates plug-and-play V2G and provides per-scene rendering results for Tanks and Temples and CO3D; the supplied passages describe evaluation scope, not numerical outcomes.

F.2 PLUG-AND-PLAY V2G PER-SCENE RESULTS

  • V2G augments GenFusion, ViewCrafter, and GSFixer without changing their reconstruction or optimization settings, across nine Mip-NeRF 360 scenes and 3-, 6-, and 9-view settings.The evaluation reports PSNR, SSIM, LPIPS, and sparse-depth metrics on shared COLMAP/SfM tracks; dense geometry ground truth is unavailable.Source: §F.2 · p. 22
  • Tables 6–8 present per-scene rendering and sparse-depth geometry results for the three Mip-NeRF 360 input-view settings.Geometry is pooled over shared points rather than reported as a row-wise mean.Source: §F.2 · p. 22, Table 6 · p. 23, Table 7 · p. 24

F.3 TANKS AND TEMPLES

  • Table 9 reports per-scene rendering results on Tanks and Temples with 4, 6, and 9 input views.Source: Table 9 · p. 26

F.4 CO3D

F.5 DL3DV-BENCHMARK

The supplied DL3DV-Benchmark passages identify per-scene rendering tables for three input-view settings but provide no numerical outcomes.

F.5 DL3DV-BENCHMARK

  • Table 12 reports per-scene DL3DV-Benchmark rendering results with 3 input views.Source: Table 13 · p. 27
  • Table 13 reports per-scene DL3DV-Benchmark rendering results with 6 input views.Source: Table 14 · p. 28
  • Table 14 reports per-scene DL3DV-Benchmark rendering results with 9 input views.Source: Table 15 · p. 28
Loading 2609.17039v1…