PaperlayerView exact revision on arXiv

Source-linked AI summary

Do Flat Minima Improve Sparse Novel View Synthesis?

Youngsik Yun, Dongjun Gu, Youngjung Uh

arXiv:2511.17918v2cs.CV

TL;DR

Sparse-view novel view synthesis struggles to generalize from limited training views, and the role of loss sharpness in this setting is underexplored. The paper introduces structure-aware sharpness regularization, adapting sharpness and its weight to local image structure. It reports consistent improvements across reconstruction frameworks and settings while retaining sharpness for fine details.

  • Problem

    Sparse-view synthesis overfits limited training views, and the relationship between loss sharpness and generalization in novel view synthesis remains underexplored.

    Source: §1 · pp. 1–2

  • Method

    Structure-Aware Sharpness Regularization adapts sharpness estimation and its regularization weight to local image structure.

    Source: §1 · pp. 2–3, §3 · p. 6

  • Results

    The method consistently improves baselines across 3DGS frameworks, implicit and explicit representations, and static and dynamic scenarios.

    Source: §1 · p. 3, §5 · p. 15

  • Takeaways & Limitations

    The findings suggest that both flat and sharp minima are needed for accurate novel-view reconstruction, with desirable sharpness varying by local image structure.

    Source: §1 · p. 2, §5 · p. 14

    The authors acknowledge that the work has limitations.

    Source: H Discussion · p. 30

Abstract

from arXiv · show

Despite the success of recent novel view synthesis methods, they tend to struggle in sparse-view settings. This poor generalization to unseen viewpoints is an inherent challenge when training with limited data. To address this, we investigate the relationship between loss sharpness and generalization in novel view synthesis-an underexplored direction. Interestingly, while pursuing flatter minima is widely known to improve generalization in deep learning, reducing loss sharpness is not always beneficial in novel view synthesis. We demonstrate that this difference arises because high-detail regions inherently require a sharp loss landscape for accurate reconstruction, whereas low-detail regions benefit from a flat loss landscape for improving generalization. Based on this insight, we introduce structure-aware sharpness, defined within structure-adaptive neighborhoods, and propose to adaptively adjust the sharpness regularization weight according to the local image structure. This strategy encourages flatter minima for generalization while preserving the loss sharpness necessary to reconstruct fine details. Across various datasets and configurations, our strategy consistently improves a wide range of baselines. Code is available at https://bbangsik13.github.io/FASR.

1 Introduction

Sparse-view synthesis overfits limited training views, and the paper argues that the useful degree of loss sharpness depends on local image detail. It proposes structure-aware sharpness regularization and reports improved generalization across reconstruction settings.

1 Introduction

  • Sparse-view models can reconstruct training views yet produce unresolved details and floating artifacts in unseen views because limited data leads to overfitting.Improving the optimization procedure of scene representations themselves has received less attention than adding geometric priors or correspondence cues.Source: §1 · pp. 1–2
  • SAM reaches the lowest loss sharpness but has higher test error than the proposed method, showing that flatter minima do not always generalize better in novel view synthesis.The comparison uses eight training views and 800 interpolated test views from Blender’s Lego scene; plotted values are means and standard deviations over ten runs.Source: Figure 1 · p. 2
  • High-detail regions such as edges benefit from sharp minima for accurate reconstruction, whereas low-detail regions favor flat minima for better generalization.Uniformly pursuing flatness can over-penalize high-detail regions and under-penalize low-detail regions.Source: §1 · p. 2, Figure 2 · p. 3
  • Structure-Aware Sharpness Regularization adapts sharpness estimation and its regularization to local image structure, reducing both in high-detail regions and increasing both in low-detail regions.This preserves sharpness needed for fine details while encouraging flatter minima for generalization.Source: §1 · pp. 2–3
  • The method narrows the generalization gap and suppresses floating artifacts across 3DGS frameworks, implicit NeRF, flat-minima methods, and static and dynamic scenes.The authors describe it as a general principle for novel view synthesis, rather than a narrow implementation.Source: §1 · p. 3
  • The approach is presented as a broadly applicable optimization principle that improves reconstruction pipelines without requiring a method-specific adaptation.The introduction identifies this generality as a contribution.Source: §1 · p. 3

2 Related Work

Prior sparse-view synthesis work adds priors or uses perturbations, while research on flatness and generalization has examined both supporting evidence and counterexamples. This paper situates its approach as local-structure-aware sharpness regularization.

2.1 Sparse Novel View Synthesis

  • Sparse-view methods use geometric, depth, or frequency constraints for NeRF and external priors for 3DGS; Sparfels freezes Gaussian means and uses raywise color variance, unlike direct worst-case optimization over all Gaussian parameters.The authors characterize their optimization strategy as complementary to approaches that alter representations or add priors.Source: §2.1 · p. 4

2.2 Flatness and Generalization

  • Prior work links sharp minima with poor generalization and studies sharpness-reducing methods, but also reports parameter-rescaling sensitivity and cases where sharper models generalize well.The paper hypothesizes that optimal sharpness and its estimation radius vary with local image detail, motivating sharp minima where they benefit reconstruction.Source: §2.2 · pp. 4–5

2.3 Reconstructing with Random Noise

  • Random perturbations have been used for robustness or uncertainty estimation; the paper frames random neighborhood sampling as an inefficient proxy for worst-case loss.The cited approaches include perturbing Gaussian parameters or query locations, while some methods estimate uncertainty from sampled model parameters.Source: §2.3 · p. 5

3 Method

The method integrates structure-aware sharpness into SAM and 3DGS, using local image structure to adapt perturbations and regularization. Its table reports improvements over all baselines on LLFF and Mip-NeRF360 across all evaluation metrics.

3.1 Preliminaries

3.2 Geometric Tolerance Map

  • The geometric tolerance map quantifies local image structure by measuring each pixel’s distance to the nearest high-detail structure.Pixels in low-detail areas have larger distances, while those near or within high-detail areas have smaller distances.Source: §3.2 · pp. 6–7

3.3 Structure-Aware Sharpness

  • Structure-aware sharpness adapts each Gaussian attribute’s perturbation neighborhood using its geometric tolerance, converted from image space to 3D using depth and focal length.The formulation produces smaller perturbations in high-detail regions and stronger perturbations in low-detail regions.Source: §3.3 · p. 7
  • The method consistently improves all baselines on LLFF and Mip-NeRF360 across all evaluation metrics.Source: Table 1 · p. 8

3.4 Structure-Aware Regularization Weighting

  • The structure-aware weighting scheme reduces the sharpness penalty in high-detail regions and strengthens it in low-detail regions.When the regularization weight approaches zero, the objective reduces to empirical loss; near 0.95, the sharpness term is emphasized.Source: §3.4 · p. 8

4 Experiments

Across the evaluated baselines, datasets, and metrics, the method improves reconstruction quality without architectural changes or additional priors. Analyses and extensions examine how its effects relate to local detail, view sparsity, and other optimization methods and scene representations.

4 Experiments

  • Experiments use 3 input views for LLFF and 12 for MipNeRF-360, with inputs downsampled by 8×; evaluation uses PSNR, SSIM, LPIPS, and AVGE.The authors report ten LLFF runs and five MipNeRF-360 runs to account for 3DGS randomness.Source: §4 · pp. 8–9

4.1 Reconstruction Quality

  • The method improves all reported metrics across datasets and baselines, while qualitative comparisons show fewer floating artifacts and corrected geometric inaccuracies in novel viewpoints.The gains come from the optimization algorithm, without architectural changes or additional priors.Source: Figure 3 · p. 9, §4.1 · p. 10

4.2 Analysis

  • Structure-aware regularization weighting preserves high-detail sharpness, while structure-aware sharpness adapts perturbation magnitude; using both balances sharpness reduction and detail preservation.Applying either component alone improves performance, but removing either from the full method degrades it.Source: Ablation study · pp. 10–11, Figure 4 · p. 10
  • Our method narrows the generalization gap to 0.0805, outperforming SAM at 0.0809 despite converging to less flat minima than SAM.Compared with 3DGS, our method’s measured loss sharpness is 1.01× smaller and its test loss is 0.0141 lower.Source: Loss landscape visualization · p. 11
  • The method’s improvement over 3DGS grows as covisibility decreases, consistent with stronger gains in regions observed by fewer training views.Covisibility maps are computed following CoMapGS using MASt3R.Source: Performance improvement by… · p. 11, Table 3 · p. 11

4.3 Generality of Our Insight

5 Conclusion

The conclusion argues that optimal loss sharpness in sparse novel view synthesis varies with local image structure, so both flat and sharp minima matter. It presents an adaptive optimization strategy and reports improvements across varied methods, representations, and scenarios.

5 Conclusion

  • In sparse novel view synthesis, optimal loss sharpness is correlated with local image structure, making both flat and sharp minima important for accurate reconstruction.This challenges the perspective that flatter minima always generalize better.Source: §5 · p. 14
  • Structure-Aware Sharpness Regularization adapts the neighborhood radius for sharpness calculation and the regularization weight to local image structure.The approach aims to correct geometric inaccuracies and suppress floating artifacts.Source: §5 · pp. 14–15
  • The method consistently improves results across 3DGS frameworks, flat-minima optimizers, implicit and explicit representations, and static and dynamic scenarios.The authors present it as a general practical principle rather than a narrow implementation.Source: §5 · p. 15

Supplementary for Do Flat Minima Improve

The supplementary material reports a convergence-speed comparison.

Supplementary for Do Flat Minima Improve

  • At five-second intervals, the method achieves higher test PSNR than the baseline within the same training time.The figure presents the comparison as a convergence-speed evaluation.Source: Figure A1 · p. 22

A Computational Cost

Explicit flat-minima optimization adds per-step computation, while a late-training variant reduces that overhead with a smaller reported gain. The supplementary analysis also varies how long the method is applied.

A Computational Cost

  • Explicit flat-minima methods such as SAM compute the loss gradient twice per iteration, doubling training time per step to estimate sharpness.The method nevertheless achieves higher reconstruction quality than baseline 3DGS within the same wall-clock time.Source: A Computational Cost · p. 22
  • Ours-L reduces training-time overhead versus 3DGS from 2.77× to 1.15×, with an AVGE improvement of 0.0095 versus 0.0132 for the full method.The authors describe this as a trade-off between performance and training speed.Source: A Computational Cost · p. 22, Table A1 · p. 23
  • Applying the method in late iterations is most effective, but early iterations remain influential; the application-duration ablation varies duration in 10% increments.The authors infer that densification, which dynamically adjusts the number of parameters, may explain the influence of early iterations.Source: A Computational Cost · p. 22, Figure A2 · p. 23

B Analysis of Structure-Aware Flatness

The analysis links a Gaussian’s flat-neighborhood size to local image structure, then evaluates attribute contributions and practical choices for setting the regularizer. The LoG implementation assigns smaller tolerances near detail and larger ones in smoother regions.

B Analysis of Structure-Aware Flatness

C Effect of Individual Gaussian Attributes

  • The Gaussian mean is the dominant contributor to performance gains, and applying the method only to the mean could be a simplified, effective alternative.The analysis reports that all attributes contribute positively; the mean-only option avoids perturbing all attributes simultaneously.Source: C Effect of Individual Gaussian… · p. 25, Table A2 · p. 25

D Choice of Hyperparameters

  • The Gaussian-mean neighborhood radius is replaced by geometric tolerance and fixed at 1, while the radius for simultaneous perturbation is selected through a second search among smaller candidates.The maximum regularization weight is selected by grid search, with performance reported as robust across a broad range of values.Source: D Choice of Hyperparameters · p. 26, Table A3 · p. 26

E Design Choice of Geometric Tolerance Map

  • The practical geometric-tolerance map uses multi-scale Laplacian-of-Gaussian responses, assigning smaller values near detail and larger values in low-detail regions.At each pixel, the procedure detects the first significant response increase and assigns the preceding scale; if there is no increase, it uses the maximum scale.Source: E Design Choice of Geometric Tolerance… · pp. 26–27

F Implementation detail

The implementation adapts baseline handling and gradient updates, and includes supplementary evaluations on OMMO scene 15.

F Implementation detail

  • The official 3DGS implementation is refactored for sparse novel view synthesis using CoR-GS and DropGaussian as references.For MipNeRF-360, NexusGS lacks optical flow, so FlowFormer++ [53] is used to compute it.Source: F Implementation detail · p. 27
  • The supplementary evaluation covers scene 15 of OMMO, with Fig. A5 showing qualitative results across a wide depth range and Tab. A5 reporting quantitative results.The text describes the method as effective in this large-scale ambiguity environment.Source: Figure A5 · p. 28, Table A5 · p. 28
  • When a baseline uses regularization, its gradient is backpropagated after the proposed method’s gradient; deformed Gaussians are differentiated and gradients propagated back.For Yun et al., the residual map is updated during ascent and Gaussians during descent.Source: F Implementation detail · p. 28

G More results

The supplementary evaluations report effectiveness across a large-scale OMMO scene, extrapolated viewpoints, different input-view counts, and post-optimization of a feed-forward model.

G.1 Large Scene Scenario

  • The method is reported effective on OMMO scene 15 in a large-scale ambiguity environment.The evaluation includes qualitative results in Fig. A5 and quantitative results in Tab. A5.Source: §G.1 · p. 28

G.2 Test Views on Extreme Viewpoints

  • On extrapolated LLFF viewpoints, the method remains effective, but artifacts persist in regions with no training-view overlap.The authors state that improving those non-overlapping regions requires additional priors and leave this for future work.Source: §G.2 · p. 29, Figure A6 · p. 29

G.3 Various Number of Input Views

  • The method yields greater performance advantages in sparser input-view settings across the evaluated 2-, 3-, 6-, and 9-view configurations.The reported quantitative comparison is in Tab. A6.Source: §G.3 · p. 29, Table A6 · p. 29

G.4 Leveraging Feedforward Priors

  • Applying the method during AnySplat post-optimization on 3-view DTU improves reconstruction quality, demonstrating its use with feed-forward priors.The quantitative comparison is reported in Tab. A7.Source: §G.4 · p. 30, Table A7 · p. 30

G.5 Per-Scene Results

Per-scene results report that the method outperforms the baseline in most cases. The discussion notes remaining tuning and scope limitations.

G.5 Per-Scene Results

  • The method outperforms the baseline in most per-scene comparisons on LLFF and MipNeRF-360.Tab. A8 reports per-scene AVGE, for which lower is better.Source: §G.5 · p. 30

H Discussion

  • Gaussian scale and rotation still require manual perturbation-radius tuning, while opacity and spherical harmonics need new strategies for challenging specular or transparent environments.The authors say Gaussian-mean-only regularization already performs competitively, so further gains from tuning other Gaussian attributes may be marginal.Source: H Discussion · p. 31
  • Applications to feed-forward reconstruction models and generative priors remain underexplored.The authors leave these applications as a future direction.Source: H Discussion · p. 31
Loading 2511.17918v2…