PaperlayerView exact revision on arXiv

Source-linked AI summary

DReSG: Diffusion Residuals for Stylized Gaussian Splatting

Zhongliang Liu, Wenjie Liu, Yang Li

arXiv:2608.29048v2cs.CVcs.GR

TL;DR

Reference-guided 3DGS stylization must combine expressive reference appearance with preserved structure and cross-view consistency, but existing feature and diffusion guidance approaches have complementary limitations. DReSG converts attention-guided diffusion proposals into render-relative residual targets, fits them into a shared Gaussian scene with controlled multi-view feedback, and reports a better balance among stylization, structure preservation, and cross-view persistence. Its performance remains bounded by proposal quality and input-3DGS quality.

  • Problem

    Existing VGG-based methods provide stable optimization but under-represent expressive style cues, while direct diffusion guidance can cause view drift, artifacts, and poorly controlled appearance updates.

  • Method

    DReSG converts attention-guided diffusion proposals into render-relative residual targets and progressively fits them into a shared Gaussian scene using controlled multi-view feedback.

  • Results

    DReSG improves the balance among reference-specific stylization, structure preservation, and cross-view persistence compared with representative 3D stylization baselines.

  • Takeaways & Limitations

    DReSG provides a 3D-grounded way to use expressive diffusion proposals while retaining shared-scene rendering and cross-view stability.

  • Takeaways & Limitations

    Performance is limited by the quality and resolution of diffusion proposals and by input-3DGS artifacts, missing geometry, or weakly anchored regions.

Abstract

from arXiv · show

Reference-guided stylization of scenes represented by 3D Gaussian Splatting (3DGS) is important for efficient and controllable 3D content creation. Existing VGG-feature-based 3D stylization methods provide stable rendered-view optimization, but often under-represent expressive reference style cues; diffusion models offer stronger image priors, yet direct per-view or score-based diffusion guidance can lead to view drift, local artifacts, and hard-to-control appearance updates. We present DReSG, a 3D-grounded residual-feedback framework for stylized Gaussian splatting. DReSG represents attention-guided diffusion proposals as residual targets relative to the current render, and progressively absorbs these residuals into a shared Gaussian scene through multi-view Gaussian feedback. To make this feedback stable and controllable, DReSG modulates residual strength during target construction and combines coverage-aware view selection with conflict-filtered color updates during multi-view fitting. Extensive experiments demonstrate that DReSG achieves competitive reference-guided stylization while better preserving scene structure and cross-view stability. Our project page is available at https://vpx-ecnu.github.io/DReSG-website/.

1. Introduction

Reference-guided 3D stylization must transfer expressive appearance while preserving scene structure, semantic content, and view-consistent 3D anchoring. DReSG addresses the gap between stable but limited feature supervision and expressive yet unstable diffusion guidance with render-relative residual feedback.

  • Rendered-view optimization provides stable multi-view constraints, but VGG-feature objectives mainly transfer global colors and related feature-level appearance cues.
  • Diffusion models offer richer semantic, structural, and local appearance changes than VGG feature losses.
  • Per-view diffusion stylization can cause inconsistent views, while score-based 3D optimization lacks explicit render-space targets and may produce artifacts or uncontrolled updates.
  • DReSG converts attention-guided diffusion proposals into render-relative residual targets and fits them into a shared Gaussian scene through multi-view rendering.
  • The framework combines RGB-logit target scaling, SNR-balanced residual modulation, coverage-aware active views, and conflict-filtered updates to balance stylization, content preservation, and cross-view stability.

2. Related Work

Related work spans 2D style transfer, NeRF and Gaussian scene stylization, and diffusion-based 3D editing. DReSG targets the remaining gap between expressive style signals and persistent, scene-level appearance by fitting proposal-render residuals through a shared Gaussian representation.

  • 2D Reference-Guided Stylization: Classical 2D neural style transfer separates content and style in VGG feature space, while later methods improve efficiency and generality with perceptual objectives, adaptive normalization, or feature transforms.
  • 3D Scene Stylization: 3D scene stylization must transfer reference appearance to a renderable representation while maintaining consistency under camera motion.
  • 3D Gaussian Splatting Stylization: Recent 3DGS methods use VGG-feature losses, patch matching, texture transfer, geometry-aware constraints, CLIP or prompt guidance, and diffusion-related supervision.
  • Limitations of Existing Supervision: Existing supervision forms leave a gap between strong style signals and persistent scene-level appearance because feature losses are indirect, prompt guidance is global, and per-view diffusion details can drift.
  • Diffusion-Based 3D Editing: Diffusion-based 3D methods either optimize representations with score-based or multi-view priors or propagate prompt-guided image edits through multi-view constraints and scene optimization.
  • DReSG Positioning: DReSG instead forms observable proposal-render residuals and fits them with the shared Gaussian scene, providing an explicit bridge from diffusion guidance to 3D rendering.

3. Preliminaries

The preliminaries define Gaussian scene attributes, rendered-view notation, feedback-stage indices, and diffusion scheduler quantities used by DReSG. They also outline the iterative optimization procedure that generates proposals, constructs targets, and updates the shared scene.

  • Gaussian Rendering: A Gaussian scene comprises primitives with mean, scale, rotation, opacity, and color attributes, rendered by projecting visible Gaussians and compositing them front to back.
  • Gaussian Rendering: For each view, the rendered color is formed from depth-ordered Gaussian contributions weighted by projected opacity and accumulated transmittance.
  • Notation: Superscripts identify current renders, content guidance, style images, residual targets, and proposal-render residuals, while n indexes feedback stages and k indexes diffusion scheduler states.
  • Optimization Loop: Algorithm 1 alternates attention-guided latent proposal updates, residual decoding, target construction, multi-view fitting, conflict-gradient projection, and Gaussian-scene re-rendering.
  • Diffusion Scheduler: Diffusion scheduler states trade off signal and noise, and DReSG uses scheduler quantities for residual-strength modulation rather than treating Gaussian scene parameters as diffusion variables.

4. Method

DReSG uses attention-guided diffusion to propose render-relative style changes, then progressively fits those residual targets into a shared Gaussian scene. Its feedback loop controls residual strength and stabilizes multi-view fitting through coverage-aware view selection and conflict filtering.

  • Attention-Guided Proposal Residuals: DReSG repeatedly renders active views, optimizes render latents with frozen diffusion features, and converts the resulting proposals into residual targets for Gaussian feedback.The current render, content view, and style image guide proposal generation while preserving scene layout and reference appearance.
  • Attention-Guided Proposal Residuals: Content queries preserve layout while style keys and values guide reference appearance during attention-based render-latent optimization.The attention operator extracts compact Q/K/V features from selected frozen U-Net layers.
  • SNR-Modulated Residual Targets: DReSG constructs bounded fitting targets by amplifying proposal-render residuals in RGB-logit space with an SNR-dependent scale.The scale uses γ_k = 1 + p_k with p_k ∈ [0,1], retaining the unamplified proposal as its baseline.
  • SNR-Modulated Residual Targets: The residual scale lies in [1,2], peaks at SNR(k) = 1, and returns to γ_k = 1 when signal or noise dominates.The schedule is symmetric under SNR 7→1/SNR and depends only on scheduler state.
  • SNR-Modulated Residual Targets: Targets remain relative to the current render, so absorbed style changes produce smaller later residuals while view-specific details must be reconciled through the shared scene.This construction progressively transfers appearance without treating a fixed stylized image as the optimization target.
  • Multi-View Gaussian Feedback: The selected view targets are fitted through differentiable multi-view Gaussian rendering, while conflicting color updates are filtered before fusion into the shared scene.This feedback makes proposal appearance persistent in one Gaussian scene rather than remaining isolated across views.
  • Multi-View Gaussian Feedback: DReSG selects a compact active-view set offline using visibility-based Gaussian coverage and keeps it fixed during optimization.Greedy selection stops when the coverage threshold or marginal-gain condition is reached, and the selected views determine where proposal targets are generated.

5. Experiments

DReSG is evaluated against representative 3D stylization methods using quality, temporal-consistency, efficiency, ablation, and user-study analyses. Results indicate strong content preservation and reduced view-dependent drift, with style and efficiency trade-offs examined across controlled comparisons.

  • Experimental setup: Evaluation covers 48 LLFF scene-style pairs, qualitative comparisons on LLFF and Tanks and Temples, ablations, and a user study.The benchmark reports style, content, cross-view consistency, and resource measurements.
  • Baselines: DReSG is compared with NeRF-, Gaussian-, VGG-feature-, CLIP-, and diffusion-based stylization methods under matched inputs and evaluation cameras.Gaussian methods share the same base reconstruction, while baselines use official implementations or default hyperparameters when available.
  • Main results: DReSG attains the highest DINO-C and lowest short- and long-term drift, while its CLIP-S remains below the highest reported value.These results indicate strong content preservation and persistent stylized appearance under viewpoint changes.
  • Main results: DReSG achieves the highest reported inference FPS and lowest inference memory, while requiring additional time for diffusion proposal construction and residual fitting.The final stylized scene remains compact and fast to render despite this optimization cost.
  • Qualitative comparison: Qualitatively, DReSG preserves object boundaries, occlusion relationships, and foreground-background separation while retaining recognizable reference-style cues.Compared with baselines, it avoids some local texture fragmentation, oversaturation, content overwrite, and boundary-detached details.
  • Ablation studies: SNR-balanced residual scheduling gives the highest DINO-C among schedules and outperforms fixed-scale baselines on both CLIP-S and DINO-C.Removing residual feedback weakens style and increases drift, while removing color-gradient projection increases local style fragmentation despite similar aggregate metrics; RGB-logit scaling avoids localized color artifacts.
  • User study: In the user study, DReSG wins style-match comparisons against every baseline and receives more than half of overall-preference votes against each baseline.Style-match preference ranges from 68.33% to 89.44%, while overall preference ranges from 61.11% to 81.67%.

6. Conclusion and Limitations

DReSG uses residual-feedback stylization to balance reference-specific appearance, structure preservation, and cross-view persistence in Gaussian scenes. The paper also identifies proposal quality, input reconstruction quality, sparse active views, and iterative optimization as limitations.

  • Conclusion: DReSG converts attention-guided diffusion proposals into bounded residual targets and fits them into a shared Gaussian scene through multi-view rendering.Residual strength is modulated with an SNR-balanced schedule during target construction.
  • Limitations: Diffusion proposal quality and resolution constrain the residual targets and therefore the final stylized 3D scene.The paper suggests stronger high-resolution, multi-scale, or multi-view diffusion models as a possible improvement.
  • Limitations: DReSG depends on input 3DGS quality, while reconstruction artifacts, missing geometry, weakly anchored regions, and sparse active views can limit stylization stability or supervision.The paper leaves geometry reconstruction or repair and adaptive view expansion for future work.
  • Limitations: DReSG is not feed-forward: each scene-style pair still requires iterative diffusion guidance and 3DGS optimization.The paper identifies distillation into a scene/style-conditioned predictor as a future direction.
Loading 2608.29048v2…