PaperlayerView exact revision on arXiv

Source-linked AI summary

Domain-adaptive Zero-Shot Image Enhancement via Locality-Constrained Diffusion Guidance

Theresa Neubauer, Dimitrios Lenis, Astrid Berg, Maria Wimmer, Gaia Romana De Paolis, Philip Matthias Winter, David Major, Johannes Novotny, Ariharasudhan Muthusami, Katja Bühler

arXiv:2609.35289v1cs.CV

TL;DR

Image enhancement across domains must balance photorealistic target-domain appearance with preservation of relevant source features. LocDiff adds locality-constrained guidance to pre-trained diffusion models, and achieves competitive realism-faithfulness trade-offs on art painting and ultrasound rendering tasks. The paper also identifies limits when domain shifts are strong or faces are unnatural, noisy, or distorted.

  • Problem

    Image-enhancement methods struggle to balance target-domain realism with preserving relevant source features, especially when images are shifted from the photorealistic diffusion prior.

    Source: §1 · p. 3, §3.1 · p. 9

  • Method

    LocDiff guides reverse diffusion with region-specific constraints, and can use auxiliary prior alignment without target-domain data.

    Source: §1 · p. 4, §5 · p. 27

  • Results

    LocDiff performs competitively with state-of-the-art methods, achieving effective realism-faithfulness trade-offs on art painting and ultrasound rendering datasets.

    Source: §1 · p. 4, §5 · p. 27

  • Takeaways & Limitations

    The method supports fully zero-shot art-to-photo translation and ultrasound enhancement with auxiliary prior alignment while keeping target-domain inference zero-shot.

    Source: §1 · p. 3, §5 · p. 27

    Enhancement quality depends on the pre-trained diffusion model and is limited for unnatural face shapes, including surrealistic paintings or statues, and highly noisy or distorted images.

    Source: §4.6 · p. 26

Abstract

from arXiv · show

Denoising Diffusion Probabilistic Models have shown remarkable performance in unconditional image generation. In order to generate images with desired semantics, recent works have restricted the solution space by using guidance constraints in the diffusion sampling process. However, for image enhancement across different domains, these methods struggle to balance two main requirements: looking realistic in the target domain (photorealistic images) and preserving relevant features of the source domain, e.g., low-quality renderings or art paintings. Here, small local changes can alter the fidelity of the image completely, while large changes in other regions might be insignificant. We introduce LocDiff, a locality-constrained guidance method for image enhancement, which serves as a zero-shot extension to pre-trained diffusion models, ensuring the preservation of critical features during domain adaptation. In this way, we retain important local features, while allowing less critical regions to remain unconstrained and not interfere with the guidance process for relevant regions. We evaluate our method on two different domain-shift tasks: For art-to-photo translation, we apply the method in a fully zero-shot setting, preserving facial identity from paintings while generating photorealistic details. For enhancing low-quality fetal ultrasound renderings, we demonstrate zero-shot inference with auxiliary prior alignment. Here, the objective is to artificially add high-resolution characteristics and produce photorealistic ultrasound renderings, a target domain for which no ground truth distribution exists. Our experimental results demonstrate that LocDiff achieves favorable realism-faithfulness trade-offs compared to state-of-the-art methods, enabling controllable cross-domain enhancement.

1. Introduction

Image enhancement across domains must balance photorealistic detail with fidelity to source features, a trade-off existing approaches struggle to manage. LocDiff applies region-specific guidance to preserve critical features; it performs competitively across art-to-photo and rendering-to-photo tasks.

1. Introduction

  • Existing enhancement methods struggle to balance realistic details with preserving source identity, while GAN-based approaches have limited control over source semantics and often require retraining under domain shifts.Diffusion guidance methods also primarily address inputs aligned with the trained prior.Source: §1 · p. 3
  • LocDiff uses locality-constrained guidance so important local features can be preserved while less critical regions remain unconstrained.This targets the mismatch between uniform image-level guidance and regions that differ in importance.Source: §1 · pp. 3–4
  • The method extends reverse diffusion with flexible conditioning regions and an adaptive sampling schedule to selectively preserve critical details during domain adaptation.It is presented as a zero-shot extension to pre-trained diffusion models.Source: §1 · p. 4
  • Ablations indicate that guidance can be independently modulated to target specific image regions during reverse diffusion.Source: §1 · p. 4
  • LocDiff performs competitively with state-of-the-art methods on art-to-photo and rendering-to-photo, achieving effective realism-faithfulness trade-offs on both datasets.The evaluation covers art painting and ultrasound rendering.Source: §1 · p. 4
  • LocDiff supports fully zero-shot art-to-photo enhancement and ultrasound inference with auxiliary prior alignment, without target-domain training data.The ultrasound setting uses semantic-space alignment.Source: §1 · p. 4

2. Related Work

Prior diffusion guidance and image-restoration methods adapt pre-trained models through varied constraints, but commonly apply guidance uniformly or offer limited control over regional structure. LocDiff addresses this by varying guidance strength across regions.

Guidance Strategies for Diffusion Models

  • Diffusion guidance adapts pre-trained models to downstream tasks, while image-enhancement methods guide sampling with conditions or intermediate-step initialization.Starting at an intermediate step can require more steps for stronger degradation or domain shift.Source: §2 · pp. 5–6
  • Mask-based methods such as MultiDiffusion and RePaint constrain regions all-or-nothing and do not preserve structural consistency within guided regions.MultiDiffusion starts from noise for text-to-image generation, whereas RePaint replaces selected regions during inpainting.Source: §2 · p. 5

Diffusion Prior for Image Enhancement with Domain Shifts

  • Existing guidance strategies apply constraints uniformly, although domain adaptation requires different constraint strengths for regions with different relevance.LocDiff adapts constraints region by region to preserve small local features while leaving less critical areas unconstrained.Source: §2 · p. 6

Image Enhancement for Faces

  • Face-enhancement methods use facial priors and constraints, but GAN-based restoration can fail under unseen degradations and may need retraining when the target domain is not photorealistic.Diffusion priors are more robust to small shifts, while increased noise for larger shifts can weaken anatomical correspondence.Source: §2 · pp. 6–7

3. Method

LocDiff enhances shifted-domain images toward a pre-trained photorealistic distribution while aiming to preserve source characteristics. Its reverse-diffusion guidance combines posterior sampling for realism with localized constraints for faithfulness.

3. Method

  • LocDiff applies region-specific guidance strengths to shifted-domain images, preserving critical facial features while allowing less important areas to be enhanced toward photorealism.The method uses disjoint binary masks, with stronger constraints for identity-critical regions such as eyes and weaker constraints for backgrounds.Source: §3 · p. 7
  • The enhancement objective maps low-quality samples toward the target distribution while preserving low-frequency source features and adding high-frequency target characteristics.The setup assumes semantic continuity despite a clear distribution shift and does not require corresponding high-quality ground truth.Source: §3 · pp. 7–8
  • The diffusion model supplies a learned target-domain prior through reverse denoising, with its score network estimating the score used to form a clean-data prediction.The forward process adds noise; the reverse process learns to denoise toward samples sharing the training distribution's characteristics.Source: §3 · p. 8

3.1. Locality-Constrained Guidance

  • Locality-constrained guidance combines posterior sampling for realism with a localized gradient that constrains solutions toward local manifolds aligned with the input.The guidance gradient is computed separately for each conditioning region and updates only that region.Source: §3.1 · pp. 8–9, Figure 2 · p. 9
  • Attribution regions, distance measures, and selected guidance timesteps determine where and how strongly LocDiff constrains the denoised estimate.The sum of region-specific gradients is masked to each region; Figure 2 illustrates these independent regional updates.Source: Figure 2 · p. 9, §3.1 · p. 10
  • Guidance is scheduled around the timestep when noisy input aligns with noisy target-distribution samples; larger manifold gaps require more steps, and semantic adaptation is applied at later timesteps.If the gap is too large, aligning the diffusion prior on auxiliary data closer to the input domain can improve enhancement.Source: §3.1 · p. 10

Algorithm 1 Locality-Constrained Sampling Process

The sampling procedure starts from a noised input and iterates through reverse-diffusion steps, applying localized guidance at selected timesteps.

Algorithm 1 Locality-Constrained Sampling Process

  • The algorithm starts by sampling x_Tstart from the noised input, then iterates through timesteps while estimating x_0|t and sampling the reverse update.The listed inputs include a distance function, noise-schedule parameters, guidance strength, and conditioning regions with their guidance timesteps.Source: Algorithm 1 Locality-Constrained… · p. 11
  • At selected guidance timesteps, the algorithm computes a masked distance gradient for each region and applies it to the current reverse-sampling state.The masks are specified as binary and pairwise disjoint.Source: Algorithm 1 Locality-Constrained… · p. 11

4. Experiments

The experiments test LocDiff on face enhancement across art-to-photo and fetal ultrasound rendering-to-photo domain shifts. Results examine realism-faithfulness trade-offs, regional guidance, and robustness to segmentation errors.

4.2. Datasets, Implementation Details, Metrics

  • Experiments focus on face images and use region masks, a Gaussian-blur distance function, and scheduled guidance; identity, similarity, and realism are assessed with face metrics and FID.Conditioning regions can come from segmentation or manual specification; guidance strength is reduced by limiting its sampling steps.Source: §4 · p. 11, §4.2 · pp. 12–13

4.1. Diffusion Prior for Domain-Adaptive Zero-Shot Tasks

  • Art portraits use the FFHQ prior without fine-tuning, while ultrasound enhancement uses auxiliary alignment on 100 baby portraits without observing the nonexistent target domain.The ultrasound task is described as zero-shot inference with auxiliary prior alignment.Source: §4.1 · pp. 11–12

4.3. Ablation on Locality-Constrained Guidance

  • Multi-region guidance improves identity preservation while retaining better image quality than full-image guidance, and region-specific guidance preserves details without visible boundary artifacts.Reduced background guidance supports high-frequency hair detail; increasing Tstart improves photorealism but can reduce structural fidelity.Source: §4.3 · pp. 14–16, Table 1 · p. 15, Figure 3 · p. 16

4.4. Comparison with State-of-the-Art Methods

  • For fetal ultrasound rendering-to-photo, LocDiff achieves superior facial detail preservation while maintaining FID quality on par with baselines.The comparison uses a fine-tuned baby-face prior; LocDiff preserves facial characteristics better in the qualitative comparison.Source: §4.4 · p. 19, Table 3 · p. 20, Figure 7 · p. 21
  • On WikiArt-Faces and MetFaces, LocDiff offers a favorable identity-preservation and image-quality trade-off, with second-best LMD, SSIM-F, and IDS metrics.DifFace and PGDiff produce photorealistic outputs but fail to preserve identity, while DiffBIR preserves identity but retains painting-like artifacts and has high FID.Source: §4.4 · p. 19, Table 4 · p. 20, Figure 8 · p. 23

4.5. Robustness to Segmentation Mask Quality

  • Face parsing agreement is strong on artistic images but lower on ultrasound, where eye-detection failures reduce overall agreement; FaRL (LaPa) generalizes best across datasets.Excluding eye classes, ultrasound parsing agreement remains high.Source: §4.5 · pp. 23–24, Table 6 · p. 25
  • LocDiff remains robust to spatial misalignment, moderate noise, and small morphological mask perturbations, with metrics degrading by less than 10% in most cases.Missing eyes cause the largest degradation, although enhancement quality remains acceptable and moderate parsing inaccuracies do not lead to artifacts.Source: §4.5 · pp. 25–26, Table 7 · p. 26

4.6. Limitations

  • Performance is limited for unnatural face shapes, including surrealistic paintings or statues, and for images with high noise or distortion.The degree of realism depends on the power of the pre-trained diffusion model.Source: §4.6 · p. 26

5. Conclusion and Future Work

LocDiff uses locality-constrained guidance to balance realism and faithfulness across domain shifts, with competitive trade-offs on art-painting and ultrasound-rendering datasets.

5. Conclusion and Future Work

  • LocDiff provides region-adaptive control of pre-trained diffusion models for fully zero-shot settings and settings with lightweight prior alignment.Source: §5 · pp. 26–27
  • Locality-constrained guidance preserves selected source-domain features while allowing remaining features to adapt globally to the target domain.Users can adjust the local image region through the attribution map and change the guidance strength.Source: §5 · p. 27
  • LocDiff achieves competitive realism-faithfulness trade-offs on art-painting and ultrasound-rendering datasets, addressing cross-domain enhancement rather than in-domain restoration.Source: §5 · p. 27

Appendix A.1. Datasets

The appendix describes face-image alignment and a 100-image baby-portrait dataset collected for fine-tuning the diffusion prior on the rendering-to-photo task.

Appendix A.1. Datasets

  • Dataset images were aligned to the FFHQ template face and resized to 512 × 512 pixels.Source: Appendix A.1 · p. 27
  • The rendering-to-photo task used 100 high-resolution baby portraits collected to fine-tune the diffusion prior.Images were manually selected from Pexels under a CC0 license, with visible faces, minimal occlusion, balanced head poses, and high image quality.Source: Appendix A.1 · p. 28
  • The baby portraits were aligned using the same protocol described for the other datasets.Source: Appendix A.1 · pp. 27–28

Appendix A.2. Hyperparameter & Baseline Implementation

The appendix reports a 20% hyperparameter-search split and evaluates baselines using the best-tested ControlNet tile configuration.

Appendix A.2. Hyperparameter & Baseline Implementation

  • 20% of each dataset was randomly selected for hyperparameter search, and metrics were reported on the remaining 80%.Source: Appendix A.2 · p. 28
  • After hyperparameter search on 20% of each dataset, ControlNet baseline results on the remaining 80% used tile-based control with Realistic Vision 5.1, control scale 1.0, and guidance scale 5.Source: Appendix A.2 · p. 28
  • The authors tested tile, canny, and soft-boundary controls with SD 1.5 and Realistic Vision 5.1 backbones, then reported the best setting.Source: Appendix A.2 · p. 28

Appendix A.3. Metrics

The appendix specifies identity and landmark metrics, image-quality metrics, and the hardware conditions used to measure peak inference memory.

Appendix A.3. Metrics

  • IDS measures angular distance between ArcFace embeddings of ground-truth and reference images, while LMD measures distances between selected facial landmarks.SPIGA predicts 65 landmarks covering eyebrows, eyes, nose, and mouth.Source: Appendix A.3 · p. 28
  • LPIPS, PSNR, and SSIM were computed with torchmetrics, and FID used the implementation of.SSIM-F is the mean SSIM over the eye, nose, and mouth regions.Source: Appendix A.3 · pp. 28–29
  • Peak inference GPU memory was measured for batch size 1 on 512 × 512 inputs using an Nvidia A100.Source: Appendix A.3 · p. 29

Appendix A.4. Implementation Details: Art-to-photo

For art-to-photo enhancement, guidance stops once image semantics have largely stabilized, while the starting timestep is set where metrics remain stable.

Appendix A.4. Implementation Details: Art-to-photo

  • At step 60, the predicted image x0|60 has semantics comparable to the final prediction x0|0.Source: Appendix A.4 · p. 29
  • Guidance terminates at step 60 because later steps have minimal semantic impact and are designed to develop high-frequency details.Source: Appendix A.4 · p. 29
  • Metrics remain stable beyond 140 steps, so the starting time step is set to Tstart = 140.Source: Appendix A.4 · p. 29

Appendix A.5. Implementation Details: Rendering-to-photo

Rendering-to-photo uses moderate enhancement to preserve ultrasound appearance while adding photorealistic details. Appendix visuals also examine mask quality and parsing disagreement.

Appendix A.5. Implementation Details: Rendering-to-photo

  • Rendering-to-photo starts guidance at Tstart = 80 and stops at tmin = 40, with weak eye guidance and strong background constraints.The weak eye guidance encourages plausible details where ultrasound eye features are barely visible; background constraints keep surrounding tissue close to the input.Source: Appendix A.5 · p. 29

Appendix A.6. Additional Visual Results

  • Figures A.10–A.13 support the mask-quality ablation, while Figure A.14 presents failure cases discussed in Section 4.6.Source: Appendix A.6 · p. 30
  • In the lowest-cross-model-mIoU test sample (0.709), stronger mouth/skin conditioning preserves the moustache better than weak background conditioning, despite parsing disagreement.The enhanced images show no artifacts despite the incorrect parsing.Source: Figure A.10 · p. 30

Related papers

Loading 2609.35289v1…