PaperlayerView exact revision on arXiv

Source-linked AI summary

PixSDS: Why Latent SDS Makes Noisy Pixels

Vsevolod Skorokhodov

arXiv:2608.12997v1cs.CV

TL;DR

Latent SDS can produce structured pixel artifacts because VAE-induced pixel drift is weakly constrained by clean latent representations. The paper introduces PixSDS, which decodes a latent SDS look-ahead step into a clean pixel-space direction and substantially reduces artifacts while preserving semantic content.

  • Problem

    Latent SDS can let optimized pixels drift in VAE-inconsistent directions, causing structured artifacts even when the latent representation remains clean and semantically meaningful.

  • Method

    PixSDS repairs latent SDS updates by decoding a latent-space SDS look-ahead step and using the decoded image as a clean pixel-space direction.

  • Results

    PixSDS substantially reduces structured artifacts in controlled 2D optimization and text-to-3D generation while preserving semantic content.

  • Takeaways & Limitations

    Artifact reduction can guide pixel-space updates toward directions consistent with the VAE latent space rather than merely suppressing large gradients.

Abstract

from arXiv · show

Score Distillation Sampling (SDS) enables text-to-3D generation by optimizing rendered images with a pretrained diffusion prior, but latent SDS often produces structured color artifacts and high-frequency texture noise. We identify a failure mode of latent SDS caused by VAE-induced pixel drift: the optimized image can move along pixel-space directions that are weakly constrained by the VAE encoder, so its latent representation remains clean and semantically meaningful while the image itself accumulates visible artifacts. We support this diagnosis with controlled 2D SDS experiments, VAE-only optimization, and a simplified analysis showing that encoder-like latent objectives can amplify image-space noise when the inverse mapping to pixels is underconstrained. Motivated by this observation, we propose PixSDS, a lightweight VAE-consistent gradient repair method. PixSDS decodes a latent SDS lookahead step and uses the decoded image as a clean direction for pixel-space optimization, reducing motion in VAE-inconsistent directions without retraining the diffusion model, changing the renderer, or replacing the SDS objective. Experiments in 2D optimization and text-to-3D generation show that PixSDS substantially reduces structured artifacts while preserving semantic content. Code is publicly available at https://sevashasla.github.io/pixsds-webpage/.

1 Introduction

The paper diagnoses structured artifacts in latent SDS as VAE-induced pixel drift and introduces PixSDS, which repairs pixel-space updates using VAE-consistent directions while preserving semantic content.

  • Diagnosis: Controlled 2D experiments, VAE-only optimization, and simplified analysis show that underconstrained inverse mappings can amplify image-space noise.The artifacts arise without requiring 3D rendering or texture extraction.
  • Failure mode: Latent SDS can leave the latent code clean and semantically valid while optimized pixels accumulate structured high-frequency artifacts.The paper identifies this mismatch as VAE-induced pixel drift.
  • Method: PixSDS decodes the latent-space SDS update and uses the decoded image as a clean direction for pixel-space optimization.This is a lightweight VAE-consistent gradient repair method.
  • Results: PixSDS reduces structured artifacts in controlled 2D SDS optimization and improves visual cleanliness in existing text-to-3D pipelines.It preserves semantic content without retraining the diffusion model or changing the renderer.

2 Related Works

The paper situates PixSDS within SDS research by distinguishing its VAE-mapping diagnosis from prior objective, gradient, and renderer-focused explanations of artifacts.

  • SDS: SDS optimizes a 3D representation using a pretrained 2D diffusion prior.The framework supports text-to-3D optimization without a dedicated 3D generative model.
  • Prior methods: Prior methods modify distillation objectives, derive alternative directions, combine latent and pixel guidance, or clip decoded pixel-wise gradients.These approaches target different aspects of SDS guidance and artifact formation.
  • Paper distinction: The paper argues that latent SDS artifacts can arise from the VAE mapping even when the corresponding latent representation remains clean.PixSDS repairs the pixel-space update rather than introducing a new score distillation objective.
  • Related explanations: Prior work attributes artifacts to mechanisms including texture extraction, mipmap sampling, and large decoded pixel-wise gradient outliers.The paper presents its VAE-based explanation as complementary to these accounts.
  • Controlled comparisons: Low-resolution latent SDS still produces structured artifacts, whereas pixel-space SDS with a small diffusion model produces realistic images without the same artifacts.These controlled comparisons suggest tensor shape and pixel-space optimization alone do not explain the failure.

3 Diagnosing SDS Artifacts

Controlled experiments isolate the VAE mapping as a sufficient mechanism for latent SDS artifacts: pixel images can become structured and noisy while their latent representations remain clean. Underconstrained inverse mappings allow encoder-like objectives to amplify image-space noise.

  • Controlled diagnosis: VAE mapping is sufficient to produce latent SDS artifacts, whereas latent tensor shape alone does not explain them.Artifacts persist at 128×128 latent resolution, while pixel-space SDS with a CIFAR-10 diffusion model does not show the same structured patterns.
  • VAE mapping: The VAE creates an underconstrained inverse problem in which visually different images can map to similar latent codes.Pixel-space updates can therefore move in weakly constrained directions while preserving a clean latent representation.
  • VAE-only optimization: Optimizing only through the VAE encoder produces structured noise in images even without a diffusion model, while decoded latents remain clean.The experiment starts from a random image and matches a target latent using the VAE encoder.
  • Simplified analysis: A toy objective demonstrates that gradient descent can converge to a noisier minimizer when the objective admits multiple solutions.From (1, 1), the objective converges to (0.4, −0.2), although (0, 0) is also a minimizer.
  • Simplified analysis: For an encoder-like convolutional objective, gradient descent can increase solution noise even when the input noise is initially below any chosen bound C.The analysis establishes this possibility for simple one-dimensional convolutions.
  • Diagnosis: Together, the experiments show that latent objectives can keep encoded representations clean while optimized images drift into noisy pixel-space directions.This separates the failure mode from latent tensor shape, direct pixel-space SDS, and renderer-specific effects.

4 Method

PixSDS repairs latent SDS updates by decoding a latent lookahead step and using the resulting image as a VAE-consistent pixel-space direction. Per-pixel normalization balances this clean direction with the original SDS update without changing the diffusion model, renderer, or SDS objective.

  • Motivation: PixSDS uses the decoded latent SDS update as a VAE-consistent direction to repair pixel-space optimization.The method is motivated by clean latents and decoded latents despite noisy optimized images.
  • Gradient repair: PixSDS pulls optimized images toward decoded latent updates, reducing motion in pixel-space directions unsupported by the VAE representation.It avoids an explicit reconstruction loss toward the current decoded latent, which could pull the image backward.
  • Clean direction: At each iteration, PixSDS decodes a latent lookahead step and defines the clean direction as the decoded image minus the current image.The lookahead is Z-hat = dec(enc(Z) − βg_sds^latent), and g_clean = Z-hat − Z.
  • Algorithm overview: PixSDS pseudocode computes the standard SDS gradient and adds a cleanup direction obtained by decoding the SDS-updated latent.The update is illustrated as moving first toward noisy images and then toward clean images.
  • Gradient repair: The repaired update adds the normalized clean direction to the original SDS direction, matching its per-pixel channel magnitude.Normalization preserves the spatial scale of the SDS update while replacing part of its noisy motion.
  • Compatibility: The method modifies the update after SDS computation, so it requires no diffusion-model retraining, renderer change, or replacement of the underlying SDS objective.This makes PixSDS compatible with other SDS-style objectives.

5 Experiments

PixSDS is evaluated in controlled 2D optimization and text-to-3D pipelines, where it reduces structured artifacts while preserving semantic guidance and local detail.

  • Evaluation settings: Experiments use controlled 2D optimization and text-to-3D pipelines, including DreamGaussian and LucidDreamer.The 2D setting removes renderer-specific effects; the 3D tests integrate PixSDS into existing optimization stages.
  • 2D quantitative results: PixSDS achieves the best FID, BRISQUE, and CLIP-IQA Noisiness results while remaining competitive in CLIP Score and CLIP-IQA Quality.The evaluation samples 100 MS-COCO captions and reports standard, quality, and noise-related metrics.
  • 2D qualitative results: PixSDS reduces structured artifacts in 2D comparisons while preserving semantic content and more local detail than oversmoothed alternatives.Examples include blender buttons and chair structures retained in PixSDS outputs.
  • DreamGaussian: In DreamGaussian, PixSDS produces cleaner textures and fewer noisy texture patterns than SDS.The method is applied in the pipeline’s second optimization stage, with the same Gaussian smoothing used for the baseline.
  • LucidDreamer: In LucidDreamer, PixSDS consistently reduces noisy artifacts, including noisy Gaussians inside objects and floating artifacts around assets.The white-hair result is cleaner around the head, although the attribute is not fully preserved and may require further tuning.
  • Ablation: The ablation indicates that the full update combines the original SDS direction with a normalized clean direction to preserve semantic guidance while reducing artifacts.Using only the clean direction harms composition, while removing channel normalization can fail in 3D because the SDS direction may dominate.

6 Conclusion

The paper attributes latent SDS artifacts to VAE-induced pixel drift and introduces PixSDS to repair updates with a VAE-consistent pixel-space direction. Across 2D and text-to-3D experiments, PixSDS reduces structured artifacts while preserving semantic guidance without changing the diffusion model or renderer.

  • Diagnosis: VAE-induced pixel drift lets images accumulate structured artifacts along pixel-space directions weakly constrained by the encoder while latents remain clean and semantically meaningful.The conclusion identifies the latent-to-pixel mapping geometry as an important factor in latent SDS optimization.
  • Method: PixSDS decodes a latent SDS look-ahead step and combines its clean pixel-space direction with the original SDS update using per-pixel normalization.The method is designed as a lightweight VAE-consistent gradient repair.
  • Findings: Experiments in 2D optimization and text-to-3D pipelines show that PixSDS reduces structured artifacts while preserving semantic guidance.The approach does so without retraining the diffusion model or changing the renderer.

A Noise Proof

The analysis defines a first-order noise functional and proves that gradient descent on an underconstrained encoder-like objective can increase image-space noise, even when the optimization reaches a minimizer.

  • Noise functional: The noise functional N(x) measures first-order variation through the difference operator D.D uses adjacent-coordinate differences, so N(x) is based on ||Dx||^2.
  • Theorem: For sufficiently small α, a direction z with x^T D^T D z ≠ 0 can reduce N(x + αz) relative to N(x).The linear term in the noise difference dominates the quadratic term when α is sufficiently small with the appropriate sign.
  • Theorem: Theorem 1 states that, under a multi-dimensional kernel and a nondegeneracy condition, gradient descent can increase the noise level.The theorem assumes dim ker A ≥ 2 and that nonzero kernel vectors are not annihilated by D^T D.
  • Theorem: Gradient descent removes non-kernel components while preserving kernel components, allowing a low-noise initialization to converge to a solution with arbitrarily large noise.For every C > 0, the proof constructs an initialization with N(x) < N(x*) and N(x*) ≥ C.
  • Convolution corollary: The result extends to one-dimensional convolution operators whose matrix representation has a kernel of dimension at least two when the filter width is at least three.The rank bound gives dim ker A ≥ m − 1, so m ≥ 3 implies the theorem’s kernel-dimension condition.

B VAE Optimization Loss Landscape

The VAE-only loss landscape contains structured noisy solutions that match the original image’s latent representation, creating spurious basins that can trap gradient-based optimization.

  • Visualization setup: The loss is visualized on a two-dimensional affine subspace centered at the clean image X.Points are parameterized as X + α_1v_1 + α_2v_2 with both coefficients in [-1, 1].
  • Specified directions: Specified directions toward VAE-optimized solutions reveal local minima away from the clean image.These solutions satisfy f(Z_i) ≈ f(X) while remaining noisy in pixel space.
  • Random directions: Random directions do not show the same spurious minima, suggesting that the problematic perturbations have structure aligned with weakly constrained VAE directions.The comparison distinguishes structured VAE-inconsistent solutions from arbitrary image perturbations.
  • Interpretation: The clean image remains the global optimum, but undesirable noisy basins can trap gradient-based optimization and hinder recovery.The basins correspond to structured solutions Z for which f(Z) ≈ f(X).

C β Hyperparameter Study

PixSDS remains realistic and visually clean across a broad range of β values, with especially stable behavior for β ∈ [0.075, 1.0].

  • Experimental setup: The sensitivity study varies β from 0 to 10 while keeping the other 2D-generation settings fixed.The experiment isolates the effect of the look-ahead hyperparameter.
  • Results: PixSDS produces realistic and visually clean images particularly for β ∈ [0.075, 1.0].The results indicate limited sensitivity to the exact β choice within this interval.
  • Interpretation: Very small β limits the clean-direction correction, while excessively large β can overemphasize the look-ahead direction and make updates less stable.The authors therefore use β values within the identified stable range.

D PixSDS for Stable Diffusion 3

PixSDS also produces clean and realistic images with Stable Diffusion 3, suggesting that the correction can extend beyond the standard latent diffusion setting to rectified-flow-based models.

  • Setup: The evaluation tests PixSDS with Stable Diffusion 3, which is based on Rectified Flow.The experiment uses the same hyperparameters as the 2D-generation experiments except for the optimization-step count.
  • Setup: The Stable Diffusion 3 experiment uses N = 500 optimization steps.This is the stated deviation from the standard 2D-generation settings.
  • Result: PixSDS produces clean and realistic images with Stable Diffusion 3.The result is presented as evidence that the correction is not specific to one diffusion backbone.
  • Implication: The authors suggest that PixSDS can be integrated with different generative formulations, including rectified-flow-based models.This conclusion is limited to the reported Stable Diffusion 3 evaluation.

E Timestep Strategy

The proposed timestep strategy produces more balanced colors, fewer oversaturated regions, and more moderate late-stage image-space gradients than linear annealing. These moderated gradients help stabilize optimization and improve generation quality, while qualitative results suggest applicability beyond standard latent diffusion models.

  • Experimental setup: Across 10 MS-COCO prompts, the proposed schedule was compared with linear annealing while keeping other hyperparameters fixed.The study recorded latent-space and image-space gradient norms throughout optimization and aggregated them across prompts.
  • Qualitative comparison: The proposed timestep strategy produces more balanced colors and fewer oversaturated regions than linear annealing.Linear annealing often creates stronger color biases, including overemphasized green backgrounds.
  • Gradient behavior: After approximately step 600, linear annealing produces substantially larger image-space gradient norms than the proposed strategy.Step 600 corresponds to diffusion timestep t = 400 under a 1000-step schedule.
  • Broader applicability: PixSDS with Stable Diffusion 3 produces clean and realistic qualitative examples, suggesting applicability beyond standard latent diffusion models.The figure presents qualitative generations using PixSDS with Stable Diffusion 3.
  • Optimization stability: Moderating late-stage gradient magnitudes helps preserve visual stability and improve generation quality.Large gradients can make optimization less stable because a single step may introduce excessive image-space changes.
Loading 2608.12997v1…