Source-linked AI summary
FixAnything: 3D-Consistent Rendering Refinement via Video Generative Priors
Khiem Vuong, Deva Ramanan, Srinivasa Narasimhan
TL;DR
Sparse or distant-view rendering produces artifacts across 3D representations, while existing generative refinements are often specialized. FixAnything repurposes a pretrained video diffusion model with mask-aware conditioning, lightweight finetuning, and pose-rewarded DPO; it improves rendering across four representations while matching or exceeding specialist methods.
Problem
Sparse input views and distant novel viewpoints cause artifacts across 3DGS, NeRF, meshes, and point clouds, while existing generative-prior pipelines are representation-specific.
Method
FixAnything cleans rendering videos with a pretrained video model using mask-aware latent conditioning, lightweight LoRA adaptation, and camera-pose-based Flow-DPO.
Results
FixAnything improves rendering quality across four 3D representations, matches or exceeds specialist methods, and updates less than 1% of parameters.
Takeaways & Limitations
A single generalist video prior can replace multiple specialist refinement pipelines and can be applied to newer video foundation models through a new lightweight LoRA training run.
Abstract
from arXiv · showhide
Rendering views using 3D scene representations such as Gaussian Splatting (3DGS), Neural Radiance Fields (NeRF), meshes, or even point clouds produces artifacts when input views are sparse or target views lie far from the input. Recent work mitigates these artifacts using diffusion-based generative priors, but is specialized to individual representations and require custom architectures or extensive retraining. We present FixAnything, a single model for fixing a wide range of rendering artifacts. It does so by repurposing a pretrained video generative model, leveraging its implicit multi-view priors with only minimal modification and lightweight finetuning. Our key insight is that even noisily-rendered sequences preserve camera motion and coarse scene structure, allowing cleanup to be formulated as video-to-video translation. To control what scene structure should be preserved, we introduce a binary mask denoting the clean pixels, enabling the model to anchor its output to high-quality inputs (e.g. training views) while refining the rest. To encourage FixAnything to produce 3D-consistent renderings that support downstream reconstruction, we use camera pose accuracy (recovered via structure-from-motion) as a reward signal for direct preference optimization (DPO). Across four distinct 3D representations, FixAnything consistently improves rendering quality with lightweight finetuning, demonstrating that a single generalist video prior can replace multiple specialist refinement pipelines. The simplicity of the framework enables immediate adoption of stronger future video models without architectural redesign.
1 Introduction
FixAnything uses a pretrained video model as a generalist renderer-refinement pipeline across multiple 3D representations. It combines mask-aware conditioning, lightweight adaptation, and pose-based preference optimization to improve visual and geometric consistency.
- Motivation: Sparse inputs and distant novel viewpoints cause representation-specific artifacts that can make rendered views unusable for applications such as content creation and robotics.Examples include floaters in 3DGS, foggy geometry in NeRF, texture distortions in meshes, and holes in point clouds.
- Generalist approach: FixAnything translates degraded rendering videos onto the natural-video manifold while preserving camera trajectories and coarse scene layout.The method treats artifacts across representations as sharing this structure, enabling one generalist model instead of specialist pipelines.
- Framework: The framework adapts Wan2.1 with latent-space conditioning, minor architectural changes, and lightweight LoRA finetuning across 3DGS, NeRF, meshes, and sparse point clouds.A per-frame binary mask identifies clean reference frames to preserve and degraded frames to refine.
- Results: Sparse point-cloud renders achieve comparable cleanup quality to other representations and can expose the camera path without teaching the model explicit SE(3) coordinates.This suggests some intermediate reconstruction representations may be unnecessary when the generative model is sufficiently powerful.
- Geometric consistency: Flow-DPO uses structure-from-motion camera-pose accuracy as a reward to steer outputs toward geometric consistency, improving AUC@5° pose estimation by 7.2% without extra inference cost.This addresses the risk that individually plausible generated frames remain inconsistent across views.
- Results: FixAnything matches or exceeds specialist methods across representations while updating less than 1% of parameters and requiring only a new LoRA run for newer video models.The adaptation is designed for limited, academic-scale compute and does not require architectural changes.
2 Related Work
Prior work improves sparse-view reconstruction and novel-view synthesis through representation-specific regularization, generative priors, video models, and preference optimization. These approaches differ in where geometric or visual priors are introduced and how consistency is encouraged.
- Sparse-view novel view synthesis: Sparse-view NeRF and 3DGS methods add regularization or depth cues because limited multi-view supervision causes quality to degrade sharply.Examples include patch, frequency, occlusion, and geometric constraints for stabilizing unobserved-view rendering.
- Generative priors for novel view synthesis and 3D enhancement: Generative-prior methods enhance 3D reconstructions by regularizing NeRF geometry or generating additional views conditioned on nearby-camera features.Nerfbusters targets floater artifacts in 3D space, while ReconFusion uses generated views to regularize NeRF training.
- Generative priors for novel view synthesis and 3D enhancement: Video-diffusion approaches synthesize or restore view-consistent renderings using camera-controlled generation, rendered point clouds, or custom spatial-temporal decoders.Some methods further finetune 3DGS on enhanced views or recast sparse-view synthesis as test-time video processing.
- Preference optimization for diffusion models: DPO methods for diffusion models optimize generation toward preferences such as visual quality, text alignment, or geometric consistency.Flow-DPO applies preference losses to rectified-flow video generators, while concurrent methods use epipolar geometry or reconstruction priors for 3D consistency.
3 Method
FixAnything formulates representation-agnostic rendering cleanup as video-to-video translation, using a pretrained video diffusion model with mask-aware conditioning and lightweight adaptation. Supervised finetuning and geometric preference optimization produce temporally and geometrically coherent cleaned renderings across four 3D representations.
- 3.1 Representation-Agnostic Rendering Cleanup: FixAnything processes entire rendering videos rather than independent frames, using temporal context to clean degraded views while preserving scene content and camera motion.The formulation applies to NeRF, 3DGS, meshes, and point clouds because their renderings retain camera trajectory and coarse scene layout.
- 3.2 Lightweight Adaptation of a Pretrained Video Model: The method adapts Wan2.1-I2V-14B by concatenating the rendering video as latent conditioning and training only a lightweight LoRA adapter without changing the architecture.The base model and VAE remain frozen during adaptation.
- 3.4 Geometry-Aware Preference Optimization: Flow-DPO ranks multiple outputs by COLMAP-recovered camera-pose accuracy and steers the model toward geometrically consistent renderings after supervised finetuning.This addresses hallucinated structures that may look plausible per frame but shift across views and corrupt structure-from-motion.
- 3.3 Training Data: Training jointly covers NeRF blur and fog, 3DGS floaters, mesh holes, and sparse-point-cloud patches while retaining clean training views along sampled trajectories.Paired degraded and clean videos are rendered from four representation types using trajectories that pass through at least two training views.
- 3.2 Lightweight Adaptation of a Pretrained Video Model: A binary mask marks clean training-view frames as trusted anchors and degraded frames for refinement, allowing clean content and neighboring appearance, lighting, and structure to be preserved.The mask is injected alongside the noised target latent and degraded-video latent through channel concatenation.
- 3.3 Training Data: 20 paired videos already produce effective cleanup, while scaling to 500 yields further improvements and remains substantially smaller than prior data requirements.The pretrained video model supplies strong priors, so finetuning mainly teaches conditioning on degraded inputs.
4 Experiments
Experiments evaluate FixAnything on held-out DL3DV scenes under sparse-view settings against reconstruction and post-hoc enhancement baselines. The method performs comparably across four input representations, while Flow-DPO improves geometric consistency and mask conditioning improves image quality.
- 4.1 Experimental Setup: Evaluation uses 20 held-out DL3DV-10K scenes with 3, 6, or 9 training views, reporting PSNR, SSIM, LPIPS, and camera-pose accuracy metrics.Query frames are sampled from the remaining views after excluding training frames.
- 4.2 Comparison with Prior Methods: FixAnything also generalizes to MipNeRF-360 and LLFF, achieving comparable performance to state-of-the-art methods with a notable LPIPS improvement.The reported supplementary evaluation uses 3DGS input.
- 4.2 Comparison with Prior Methods: FixAnything uses one model across NeRF, 3DGS, mesh, and sparse SfM-point inputs, achieving comparable cleanup quality across all four representations.Sparse point-cloud renderings perform on par with or better than 3DGS and NeRF inputs despite containing mainly scattered keypoints between clean training views.
- 4.3 Geometry-Aware Preference Optimization: Flow-DPO improves pose AUC@5◦by 7.2% over SFT-only training, while image-quality metrics improve modestly and inference cost remains unchanged.The geometric prior is incorporated into the LoRA adapter during training rather than added at inference.
- 4.4 Ablation Studies: Mask-aware conditioning improves PSNR by 1.3 dB because removing the mask causes hallucination over clean frames that should be preserved.The mask explicitly distinguishes trusted frames from frames requiring repair.
- 4.4 Ablation Studies: 20 paired videos already provide effective cleanup, and reducing denoising steps from 50 to 5 preserves comparable quality while yielding a 10× speedup.A 61-frame clip at 480×832 resolution takes 31 seconds on one H100 under the reported accelerated setting.
5 Discussion: Hallucination and Uncertainty
FixAnything treats hallucination as useful for filling unobserved regions unless generated content contradicts observations. A five-sample uncertainty estimate identifies ambiguous areas and correlates with reconstruction error.
- Hallucination is useful when it plausibly fills unobserved regions, becoming a failure only when it contradicts existing observations.
- Five inference runs with different random seeds provide a training-free per-pixel uncertainty estimate based on output standard deviation.The estimate is lower in sky- and ground-like regions and higher where multiple completions are plausible, such as buildings.
- 25.7 dB versus 14.4 dB PSNR separates the most-confident 25% from the least-confident 25% of pixels on DL3DV with 6 views.
6 Conclusion
The conclusion presents FixAnything as a single generative cleanup model that works across several 3D representations and suggests simplifying reconstruction workflows. It further identifies weakly observed regions as the main challenge and proposes combining reliable geometry with generative completion.
- 6 Conclusion: FixAnything cleans 3DGS, NeRF, meshes, and point clouds using a pretrained video diffusion model finetuned on a modest set of paired videos.
- 6 Conclusion: Comparable results can be obtained by rendering sparse COLMAP point clouds, optionally meshing them, and applying generative cleanup directly.
- 6 Conclusion: The main difficulty in novel-view rendering arises in weakly observed or unobserved regions, where the task becomes plausible completion rather than geometric inference.