PaperlayerView exact revision on arXiv

Source-linked AI summary

Deformable One-shot Face Stylization via DINO Semantic Guidance

Yang Zhou, Zichong Chen, Hui Huang

arXiv:2403.00459v2cs.CV

TL;DR

One-shot face stylization methods have often emphasized appearance while leaving structural deformation underexplored. This paper fine-tunes a deformation-aware StyleGAN using DINO guidance and a single real-style pair. The authors report that it surpasses existing one-shot methods in qualitative and quantitative comparisons.

  • Problem

    One-shot stylization methods mainly transfer color and texture, with less exploration of the structural deformation that contributes to artistic exaggeration.

    Source: §1 · pp. 1–2

  • Method

    The method fine-tunes a deformation-aware StyleGAN on one real-style pair, using DINO-guided deformation constraints and style-mixing for color alignment.

    Source: §1 · p. 2

  • Results

    Qualitative and quantitative comparisons show that the method surpasses existing one-shot methods while providing high-quality appearance transfer and convincing structural deformation.

    Source: §6 · p. 8

  • Takeaways & Limitations

    The framework stylizes faces with strong exaggerations in appearance and structure while maintaining a faithful identity to the input.

    Source: §1 · p. 2

    The paired references used are mainly produced by existing generative models trained on large datasets, and are treated as sufficiently representative real pairs.

    Source: §5 · p. 6

Abstract

from arXiv · show

This paper addresses the complex issue of one-shot face stylization, focusing on the simultaneous consideration of appearance and structure, where previous methods have fallen short. We explore deformation-aware face stylization that diverges from traditional single-image style reference, opting for a real-style image pair instead. The cornerstone of our method is the utilization of a self-supervised vision transformer, specifically DINO-ViT, to establish a robust and consistent facial structure representation across both real and style domains. Our stylization process begins by adapting the StyleGAN generator to be deformation-aware through the integration of spatial transformers (STN). We then introduce two innovative constraints for generator fine-tuning under the guidance of DINO semantics: i) a directional deformation loss that regulates directional vectors in DINO space, and ii) a relative structural consistency constraint based on DINO token self-similarities, ensuring diverse generation. Additionally, style-mixing is employed to align the color generation with the reference, minimizing inconsistent correspondences. This framework delivers enhanced deformability for general one-shot face stylization, achieving notable efficiency with a fine-tuning duration of approximately 10 minutes. Extensive qualitative and quantitative comparisons demonstrate our superiority over state-of-the-art one-shot face stylization methods. Code is available at https://github.com/zichongc/DoesFS

1. Introduction

One-shot face stylization has often transferred appearance while leaving structural deformation underexplored, though exaggeration is an important part of artistic style. The paper uses a single real-style pair and DINO-guided deformation-aware StyleGAN to address both appearance and structure.

1. Introduction

  • Prior one-shot methods focus mainly on color and texture, while the structural exaggeration important to artistic style remains less explored.With only one style example, training must also guard against overfitting and mode collapse.Source: §1 · p. 1
  • Inverting an artistic reference can provide misleading deformation guidance because current GAN inversion techniques do not faithfully map unseen domains.The paper instead proposes using a real-style image pair for cross-domain guidance.Source: §1 · p. 2, Figure 2 · p. 2
  • A deformation-aware StyleGAN trained on one real-style pair uses DINO-guided losses for structural deformation and style-mixing for color alignment.The method combines directional deformation and relative structural consistency constraints, with spatial transformers appended to the generator.Source: §1 · p. 2
  • Qualitative and quantitative comparisons are reported to demonstrate the framework’s effectiveness and superiority over existing state-of-the-art methods.The introduction describes its target as strong appearance change and structural deformation while maintaining input identity.Source: §1 · p. 2

2. Related Work

Earlier face stylization work ranges from neural style transfer to GAN-based translation and fine-tuning, while data-intensive approaches are poorly suited to styles represented by very few examples. One-shot methods stylize appearance but tend to preserve face structure, motivating deformable stylization from a single paired example.

2. Related Work

  • Neural style-transfer methods and image-translation GANs are established approaches, while some translation methods fine-tune pretrained StyleGANs on hundreds to thousands of artistic faces.Some GAN-based approaches model deformation for caricatures, but their data requirements limit use with extremely few examples.Source: §2 · p. 2
  • Few-shot domain-adaptation work uses regularization, and face-stylization methods [6] [21] [40] [45] extend this setting to a single style example.These one-shot methods can produce well-stylized appearances but tend to preserve face structure and overlook non-local deformations.Source: §2 · p. 2
  • CLIP is widely used for text-guided image editing, generation, and stylization, while DINO [5] has been used to capture high-level semantics and disentangle appearance from structure.The authors identify DINO as a candidate representation for deformable face stylization.Source: §2 · pp. 2–3

3. DINO Semantic Guidance

The paper investigates DINO-ViT features as cross-domain structural guidance for face stylization, comparing them with CLIP and FaRL. It reports that DINO features provide more consistent facial semantics across real and artistic domains and uses DINO tokens as its feature representation.

3. DINO Semantic Guidance

  • DINO is a self-distilled ViT in which a student learns output similarity across two transformed inputs and an exponential moving average updates the teacher.The authors expect its high-level semantics at high spatial resolution to support guidance across the real-style domain gap.Source: §3 · p. 3
  • DINO features show cleaner, more precise, and more consistent high-level facial segments across real and artistic images than CLIP and FaRL.The comparison visualizes ViT-B/16 features at layers 3, 6, and 12, representing low, middle, and high levels.Source: §3 · p. 3, Figure 3 · p. 3
  • The authors attribute DINO’s cross-domain facial semantics to self-distillation and paired input augmentation, which encourages attention to structural semantics.They contrast this with text-guided ViTs, whose semantic precision may be reduced by natural-language ambiguity.Source: §3 · p. 3
  • The method uses DINO tokens rather than keys alone because keys can miss structural semantics in an overfitting test.The paper names this token-based feature representation DINO semantic guidance.Source: §3 · p. 3

4. Method

The method fine-tunes a deformation-aware StyleGAN using a single real-style pair, combining DINO-guided structural constraints with color alignment and adversarial style transfer.

4.1. Deformation-aware generator

  • TPS-STN warps StyleGAN feature maps, while basic STNs provide translation, rotation, and scaling at the 32 × 32 and 64 × 64 resolutions.The Transform modules are inserted after synthesis blocks; the TPS warping field is regularized for smoothness.Source: §4.1 · p. 4

4.2. DINO-based domain adaption

  • DINO-space directional alignment guides cross-domain deformation, while relative token self-similarities preserve structural diversity and patch-level adversarial training targets local texture, color, and fidelity.The directional loss aligns generated deformation with the paired-reference direction; the consistency loss compares structural diversity across domains.Source: §4.2 · p. 5

4.3. Color alignment

  • StyleGAN inversion and fine-level style mixing align a random W+ sample’s colors with a paired reference: swapping codes 9–18 yields aligned colors while preserving the random sample’s structure and identity.Source: §4.3 · pp. 5–6

5. Experiments

Experiments compare the method with style-only and paired one-shot baselines, and ablate the loss terms, STN blocks, and color alignment. The reported comparisons favor the method on most metrics, especially style perception and deformation, while the ablations show distinct contributions from these components.

5. Experiments

  • The framework uses StyleGANv2 initialized from FFHQ and paired references mainly obtained from generative models trained on large datasets.Evaluation compares against MTG, JoJoGAN, DiFa, and OneshotCLIP, and also includes paired variants of MTG and JoJoGAN.Source: §5 · p. 6, §5.1 · p. 6

5.1. Evaluation and comparisons

  • Against style-only methods, our results show plausible exaggeration and correct color transfer, while competitors miss exaggerated components such as face contours.Against paired MTG and JoJoGAN variants, the method has better color style and finer geometry deformation; those variants remain limited by imprecise cross-domain structure guidance.Source: §5.1 · p. 6, Figure 7 · p. 6, Figure 8 · p. 7
  • The method leads on most metrics against paired variants, especially style perception and structure deformation, and shows an advantage in deformation and identity over style-only methods.The evaluation covers three styles; Table 1 separates style-only methods from paired variants.Source: §5.1 · p. 6, Table 1 · p. 7

5.2. Ablation Analysis

  • The full loss terms jointly support deformation and style: directional guidance can overfit, structural consistency can introduce color variation, and the adversarial loss stabilizes style quality.The ablation reports that adding structural consistency improves diversity, while adding the adversarial loss corrects color style.Source: §5.2 · p. 7, Figure 9 · p. 8
  • STN blocks produce more accurate deformation of exaggerations in the style reference, including eye contraction and cheek deformation missed without them.The paper identifies exaggeration as a key component of artistic styles.Source: §5.2 · p. 7, Figure 11 · p. 8
  • When paired references have a large color gap, color alignment preserves consistent DINO semantics and alleviates structural mismatching.Without color alignment, the lower lips are affected.Source: §5.2 · p. 7, Figure 12 · p. 8

5.3. Facial deformation control

  • Interpolating the TPS-STN warping field controls deformation degree, with α=0 and α=1 producing results without and with warping, respectively.The STN blocks are plug-ins that warp intermediate StyleGAN features.Source: §5.3 · pp. 7–8, Figure 10 · p. 8

6. Conclusion

The framework fine-tunes a deformation-aware generator under DINO semantic guidance using a single paired reference. It produces stylized faces with high-quality appearance transfer and convincing structure deformation, and surpasses existing one-shot methods.

6. Conclusion

  • The method surpasses existing one-shot methods in qualitative and quantitative comparisons.The conclusion reports this as an overall comparison.Source: §6 · p. 8
  • Fine-tuning a deformation-aware generator under DINO semantic guidance enables stylization from a single paired reference.Source: §6 · p. 8
  • The framework combines high-quality appearance transfer with convincing facial structure deformation.Source: §6 · p. 8
Loading 2403.00459v2…