PaperlayerView exact revision on arXiv

Source-linked AI summary

NeuralRemaster: Phase-Preserving Diffusion for Structure-Aligned Generation

Yu Zeng, Charles Ochoa, Mingyuan Zhou, Vishal M. Patel, Vitor Guizilini, Rowan McAllister

arXiv:2512.05106v3cs.CVcs.GRcs.LGcs.RO

TL;DR

Structure-aligned generation needs to change appearance while retaining spatial layout, but standard diffusion corrupts phase, and architectural control methods add complexity. ϕ-PD preserves input phase while randomizing magnitude, with FSS noise offering cutoff-based control of alignment. Across re-rendering and simulation enhancement, it reports improved aligned outputs and a 50% planner-transfer improvement on CARLA simulations.

  • Problem

    Structure-aligned generation requires changing appearance while keeping spatial layout fixed, but standard diffusion destroys phase and existing structural controls add architectural complexity.

    Source: §1 · p. 2

  • Method

    ϕ-PD preserves input phase while randomizing magnitude, and FSS noise uses a single frequency cutoff to control structural rigidity.

    Source: §1 · p. 2, §3.2 · p. 6

  • Results

    Across re-rendering and simulation enhancement, ϕ-PD maintains geometry alignment and outperforms prior methods; on CARLA simulations it improves planner transfer to the Waymo Open Dataset by 50%.

    Source: §1 · p. 3, §4.2 · pp. 12–13

  • Takeaways & Limitations

    The phase-preserving reformulation requires no architectural changes or inference-time overhead and applies to image and video diffusion models.

    Source: §6 · pp. 14–15

    ϕ-PD assumes image-like inputs; depth or normal modalities may require a lightweight prior to produce an initial image representation.

    Source: §6 · p. 15

Abstract

from arXiv · show

Standard diffusion corrupts data using Gaussian noise whose Fourier coefficients have random magnitudes and random phases. While effective for unconditional or text-to-image generation, corrupting phase components destroys spatial structure, making it ill-suited for tasks requiring geometric consistency, such as re-rendering, simulation enhancement, and image-to-image translation. We introduce Phase-Preserving Diffusion (\phi-PD), a model-agnostic reformulation of the diffusion process that preserves input phase while randomizing magnitude, enabling structure-aligned generation without architectural changes or additional parameters. We further propose Frequency-Selective Structured (FSS) noise, which provides continuous control over structural rigidity via a single frequency-cutoff parameter. \phi-PD adds no inference-time cost and is compatible with any diffusion model for images or videos. Across photorealistic and stylized re-rendering, as well as sim-to-real enhancement for driving planners, \phi-PD produces controllable, spatially aligned results. When applied to the CARLA simulator, \phi-PD significantly improves sim-to-real planner transfer performance. The method is complementary to existing conditioning approaches and broadly applicable to image-to-image and video-to-video generation. Videos, additional examples, and code are available on our \href{https://yuzeng-at-tri.github.io/ppd-page/}{project page}.

1 Introduction

Structure-aligned generation changes appearance while keeping spatial layout fixed, but standard diffusion corrupts phase and existing structural controls add architectural complexity. ϕ-PD preserves input phase while randomizing magnitude, and FSS noise controls structural rigidity with a frequency cutoff.

1 Introduction

  • Structure-aligned generation modifies appearance while keeping spatial layout, including object boundaries, geometry, and scene structure, fixed.Examples include neural rendering, stylization, and sim-to-real transfer for autonomous driving or robotics simulation.Source: §1 · p. 2
  • Existing approaches such as ControlNet and T2I-Adapter add auxiliary branches and parameters, increasing computational cost for structure-aligned generation.The paper attributes this complexity to diffusion corrupting both magnitude and phase, even though phase encodes structure and magnitude encodes texture.Source: §1 · p. 2
  • ϕ-PD preserves input phase while randomizing frequency-domain magnitude, maintaining spatial alignment without architectural changes or extra parameters.The method is compatible with DDPM and flow-matching models for images or videos and adds no inference-time overhead.Source: §1 · pp. 2–3
  • FSS noise uses a single frequency-cutoff parameter to control the trade-off between strict structural alignment and creative flexibility.It interpolates between input phase and pure Gaussian noise.Source: §1 · pp. 2–3
  • ϕ-PD preserves structure through phase consistency while remaining lightweight and model-agnostic.The figure contrasts this with prior approaches that modify architectures and add overhead.Source: Figure 2 · p. 3
  • ϕ-PD improves planner transfer to the Waymo Open Dataset by 50% when enhancing CARLA simulations.Across photorealistic and stylized re-rendering and simulation enhancement, it maintains geometry alignment and outperforms prior methods on quantitative and qualitative metrics.Source: §1 · p. 3

2 Related Work

Prior work explores frequency-domain operations, architectural controls, and training-free guidance for diffusion-based generation. ϕ-PD instead provides spatial control through the diffusion process without additional inference overhead.

Diffusion Models

  • Diffusion models progressively corrupt data into Gaussian noise and learn to reverse that process through iterative denoising.The cited work describes applications across image, video, audio, reinforcement learning, and robotics [3] [4] [17] [29] [32], [14] [26] [37], [1] [5].Source: §2 · p. 3

Frequency-Domain Manipulation for Diffusion

  • Frequency-domain diffusion research studies unequal corruption rates, feature manipulation, frequency truncation, phase modulation, and spectral noise optimization.The cited approaches include FreeDiff and phase-based style transfer.Source: §2 · p. 4

Structure-Aligned Generation with Diffusion

  • Architectural-control methods add modules to incorporate structural signals, with approaches trading off computation, control precision, or training cost.Examples range from ControlNet's copied UNet encoder branch to lightweight adapters and selective fine-tuning; ϕ-PD instead enables spatial control without added computation or parameters.Source: §2 · p. 4

Training-Free Guidance Methods

  • Training-free guidance methods avoid training costs but may add test-time overhead through external models, inversion, or repeated inference.ϕ-PD achieves spatial control without additional inference-time overhead.Source: §2 · pp. 4–5

3 Method

The method uses Fourier phase to preserve spatial structure while randomizing magnitude, then adds frequency-selective control over how much input structure is retained. It trains with structured noise in diffusion or flow-matching objectives and extends the approach to video through a two-stage pipeline.

3.1 Frequency Domain Fundamentals

  • Phase primarily determines spatial structure, while magnitude largely controls texture; mixing images’ components yields layouts that follow the phase source.This phase–structure correspondence motivates preserving input phase during diffusion.Source: §3.1 · p. 5, Figure 3 · p. 5

3.2 Phase-Preserving Diffusion

  • ϕ-PD replaces Gaussian noise with structured noise that preserves the input phase while randomizing magnitude, maintaining spatial alignment during corruption.The structured noise pairs the input phase with a random magnitude derived from Gaussian noise’s Fourier transform.Source: §3.2 · p. 6
  • FSS noise preserves input phase within a cutoff radius and uses noise phase elsewhere, controlling the trade-off between structural alignment and flexibility.As cutoff radius r increases, the noise becomes more structured and generated images align more tightly with the input.Source: §3.2 · pp. 6–7

3.3 Training Objective

  • The training objectives replace Gaussian noise with structured noise; flow matching interpolates between structured noise and the target image.Although structured noise is non-Gaussian and formally violates standard DDPM assumptions, standard samplers produce high-quality, structure-aligned outputs after finetuning.Source: §3.3 · pp. 7–8

3.4 Extension to Videos

  • For video, ϕ-PD constructs structured noise frame by frame, generates an initial frame with image-based ϕ-PD, then extends it with first-frame-conditioned video ϕ-PD.The video pipeline requires no architectural changes.Source: §3.4 · p. 8

4 Experiments

The experiments test ϕ-PD in photorealistic and stylized re-rendering and CARLA simulation enhancement, using image and video diffusion models. Results report improved structure alignment and a 50% zero-shot planner generalization boost, with no added parameters or FLOPs.

4 Experiments

  • At inference, structured SDEdit starts from a source image with added structured noise, while the from-noise mode encodes structure entirely in phase.Structured SDEdit generally aligns more strongly; starting from noise allows larger appearance changes.Source: §4 · p. 9

4.1 Implementation Details

  • The experiments use an approximately 5,000-image, 200-scene UnrealCV set, 29 ImageNetR images, and 5.5 hours of CARLA driving videos.The CARLA-trained planner is evaluated on Waymo's WOD-E2E validation set.Source: §4.1 · pp. 9–10

4 Experiments

  • Evaluations span SD 1.5, FLUX-dev, and Wan2.2-14B, covering image DDPM, image flow matching, and video flow matching.The models are finetuned with phase-preserving noise without architectural changes.Source: §4 · p. 9, Model architecture · p. 10

4.2 Results

  • At matched t=0.5, structured noise yields near 90% improvements on LPIPS over Gaussian noise while maintaining text-prompt alignment.The quantitative comparison uses SD 1.5-based models on UnrealCV.Source: Table 1 · p. 11, §4.2 · p. 11

4.1 Implementation Details

  • Training samples cutoff radius r from an exponential distribution with minimum r0; experiments set λ=0.1 and transition bandwidth σ=2.SD 1.5 is fully finetuned, while FLUX-dev and Wan2.2-14B use LoRA finetuning due to computational constraints.Source: Training and Inference Details · p. 11

4.2 Results

  • ϕ-PD produces coherent stylizations with object boundaries and spatial consistency, while prior methods often distort geometry or misalign texture.The figure presents representative stylized re-rendering results.Source: Figure 7 · p. 10, Table 2 · p. 12, §4.2 · p. 12
  • ϕ-PD adds no extra parameters or FLOPs relative to the base model.The efficiency comparison also reports wall-clock processing time; the passage does not give its values.Source: Table 3 · p. 12, §4.2 · p. 13
  • ϕ-PD boosts planner generalization by 50% in the zero-shot setting on Waymo validation, with the planner-error figure showing lower is better.The CARLA video examples also show maintained road boundaries, vehicle shapes, and spatial layout across frames.Source: Figure 8 · p. 12, §4.2 · p. 13

5 Ablation Studies

The ablations examine how the FSS cutoff affects alignment and whether phase–structure correspondence holds in VAE latent space. Larger inference cutoffs improve structural alignment, while latent swaps provide empirical support for the method's phase-based premise.

5.1 Cutoff Radius

  • Increasing inference cutoff r lowers LPIPS from 0.44–0.45 to 0.30–0.38, while CLIP scores stay around 0.30–0.32.A higher training cutoff r0 favors larger inference radii; a lower r0 favors smaller ones.Source: §5.1 · p. 14

5.2 Phase Preservation in Latent Space.

  • The latent-space correspondence is empirically validated, while a complete theoretical analysis of its transfer through nonlinear encoders is outside the work's scope.The authors apply phase preservation directly to VAE latents.Source: §5.2 · p. 14
  • Swapping VAE latent phase and magnitude yields decoded images whose structure follows the source of the phase.The examples preserve the dog silhouette or the road, horizon, and vehicles, depending on which latent supplies phase.Source: §5.2 · p. 14

6 Conclusion

The conclusion presents ϕ-PD as a diffusion reformulation that preserves phase while randomizing magnitude, and FSS noise as a way to tune structural rigidity. It also states an input-modality boundary for the method.

6 Conclusion

  • ϕ-PD replaces Gaussian noise with structured noise that preserves image phase while randomizing frequency-domain magnitude.The change retains spatial alignment without architectural or training-objective changes or inference-time overhead.Source: §6 · pp. 14–15
  • FSS noise controls structural-alignment rigidity continuously through a single frequency-cutoff parameter.The conclusion describes the framework as broadly applicable to different applications.Source: §6 · p. 15
  • ϕ-PD assumes image-like inputs; depth or normal modalities may require a lightweight prior to produce an initial image representation.Source: §6 · p. 15

Related papers

Loading 2512.05106v3…