PaperlayerView exact revision on arXiv

Source-linked AI summary

Pastiche Master: Exemplar-Based High-Resolution Portrait Style Transfer

Shuai Yang, Liming Jiang, Ziwei Liu, Chen Change Loy

arXiv:2203.13248v1cs.CVcs.LG

TL;DR

Exemplar-based high-resolution portrait style transfer asks how to apply an artistic portrait’s style to a target face, beyond an overall domain translation. DualStyleGAN uses intrinsic and extrinsic style paths with progressive fine-tuning. The paper reports high-quality pastiches, flexible color and structural control, and superior performance over state-of-the-art methods.

  • Problem

    StyleGAN fine-tuning translates the overall portrait distribution but does not provide exemplar-based style transfer to a preferred artwork.

    Source: §1 · p. 2

  • Method

    DualStyleGAN combines an intrinsic StyleGAN path with an extrinsic target-domain style path and progressively fine-tunes the model toward that domain.

    Source: §1 · p. 2

  • Results

    DualStyleGAN provides high-quality, high-resolution pastiches and flexible color and structural control, with superior performance over state-of-the-art methods in high-quality and diverse artistic portrait generation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The model extends StyleGAN to new domains while preserving original-domain style control, with a friendly data requirement.

    Source: §5 · p. 8

    The method can fail on under-represented styles, including curly hair without bangs and extremely large eyes, under strong dataset bias.

    Source: §4.4 · p. 8

Abstract

from arXiv · show

Recent studies on StyleGAN show high performance on artistic portrait generation by transfer learning with limited data. In this paper, we explore more challenging exemplar-based high-resolution portrait style transfer by introducing a novel DualStyleGAN with flexible control of dual styles of the original face domain and the extended artistic portrait domain. Different from StyleGAN, DualStyleGAN provides a natural way of style transfer by characterizing the content and style of a portrait with an intrinsic style path and a new extrinsic style path, respectively. The delicately designed extrinsic style path enables our model to modulate both the color and complex structural styles hierarchically to precisely pastiche the style example. Furthermore, a novel progressive fine-tuning scheme is introduced to smoothly transform the generative space of the model to the target domain, even with the above modifications on the network architecture. Experiments demonstrate the superiority of DualStyleGAN over state-of-the-art methods in high-quality portrait style transfer and flexible style control.

1. Introduction

The paper targets exemplar-based portrait style transfer, which requires more than StyleGAN’s overall domain translation. DualStyleGAN adds a second style path and progressive training to support high-resolution pastiches with flexible color and structural control.

1. Introduction

  • StyleGAN fine-tuning translates the face domain to artistic portraits with hundreds of images and hours of training, but learns only an overall distribution translation rather than exemplar-based style transfer.A fixed caricature model can map a laughing face to one with an exaggerated mouth, without allowing the user to pastiche a preferred artwork.Source: §1 · p. 2
  • DualStyleGAN retains StyleGAN’s intrinsic style path and adds an extrinsic path for target-domain style, with hierarchical modulation of structural styles in coarse layers and color styles in fine layers.The two paths correspond naturally to content and style paths in standard style transfer.Source: §1 · p. 2
  • Progressive fine-tuning initializes the extrinsic path to retain StyleGAN’s generative space, then increases task difficulty to translate that space toward the target domain.The approach introduces the extrinsic style residually into convolution layers to approximate fine-tuning effects; facial destylization supplies face-portrait supervision to promote diverse styles and avoid mode collapse.Source: §1 · p. 2
  • DualStyleGAN offers high-quality, high-resolution pastiches with flexible and diverse control of color and complex structural styles.The paper reports superior performance over state-of-the-art methods in high-quality and diverse artistic portrait generation, requiring only a few hundred style examples.Source: §1 · p. 2

2. Related Work

Related work includes StyleGAN fine-tuning for portrait generation and image-to-image translation for face stylization. The paper distinguishes DualStyleGAN by its explicit exemplar-based style control and high-resolution generation.

2. Related Work

  • Earlier StyleGAN fine-tuning methods translate the overall distribution but lack exemplar-based control; feature swapping can fail as domain alignment weakens, limiting structural-style control.The paper contrasts this with DualStyleGAN’s explicit extrinsic style path and facial destylization supervision for learning diverse styles.Source: §2 · p. 2
  • Figure 2 frames the comparison as unconditional StyleGAN fine-tuning versus conditional DualStyleGAN fine-tuning.The related-work discussion notes that unconditional fine-tuning translates the generative space as a whole, leading to loss of captured-style diversity.Source: §2 · p. 2, Figure 2 · p. 3
  • Image-to-image translation can stylize faces through bidirectional domain mappings, but learning these mappings from scratch limits resolution and requires long training time.DualStyleGAN follows StyleGAN fine-tuning for high-resolution portraits and hierarchical style control.Source: §2 · p. 3

3. Portrait Style Transfer via DualStyleGAN

DualStyleGAN preserves the original face generator while learning an extrinsic path for exemplar styles, using face–portrait supervision and progressive fine-tuning to model target-domain styles. Its pipeline includes facial destylization, hierarchical style control, staged transfer, and exemplar-code optimization and sampling.

3. Portrait Style Transfer via DualStyleGAN

  • DualStyleGAN is trained from a pre-trained StyleGAN to characterize styles from both the original and extended domains, using supervision and a progressive strategy.The approach addresses the loss of captured style diversity associated with unconditional fine-tuning.Source: §3 · p. 3

3.1. Facial Destylization

  • Facial destylization embeds an artistic portrait, optimizes its latent code, then re-embeds the reconstruction to obtain a realistic face as structural supervision.The resulting face-portrait pairs provide supervision for how facial structures deform or abstract to imitate the portrait.Source: §3.1 · p. 3

3.2. DualStyleGAN

  • The fixed intrinsic StyleGAN path preserves original styles, while an extrinsic path controls target-domain styles hierarchically across color and structure.Residual blocks simulate fine-tuning changes to convolution layers; the two paths can be mixed with weight w.Source: §3.2 · pp. 3–4

3.3. Progressive Fine-Tuning

  • Progressive fine-tuning moves from source-domain color transfer to structure transfer and then target-domain style transfer.During structure transfer, the style-mixing boundary decreases from layers 7 to 5, increasing the structural style supplied by the second code.Source: §3.3 · pp. 4–5

3.4. Latent Optimization and Sampling

  • Exemplar-code optimization fits each extrinsic style code to its portrait while the generator is fixed, refining colors; a sampling network learns the optimized-code distribution.Structure and color codes are sampled separately and concatenated because the two are well disentangled.Source: §3.4 · p. 5

4. Experiments

The experiments compare DualStyleGAN with prior methods, test its training and design choices, and analyze style control across familiar and unseen styles. They also identify failure cases tied to non-facial detail and data imbalance.

4. Experiments

  • Experiments use cartoon, caricature, and anime datasets of 317, 199, and 174 images, respectively, and test intrinsic styles on CelebA-HQ.Testing takes about 0.13 seconds per image.Source: §4 · pp. 5–6

4.1. Comparison with State-of-the-Art Methods

  • DualStyleGAN transfers exemplar styles best in both color and complex structure among the compared methods.The comparison includes image-to-image translation, StyleGAN-based, and few-shot adaptation methods.Source: §4.1 · p. 6, Figure 8 · p. 6

4.2. Ablation Study

  • Without paired face-portrait supervision, the model overfits portraits; removing regularization overfits example hairstyles, while excessive λreg over-preserves face shape.The study uses λreg = 0.005 as a trade-off; all three progressive fine-tuning stages are needed for accurate extrinsic-style transfer.Source: §4.2 · p. 7, Figure 10 · p. 7
  • Extrinsic-path layers act at different scales: initial layers adjust overall face shape, 16 × 16 layers exaggerate components such as mouths, and 32 × 32 layers target local shapes such as wrinkles.The analysis focuses on structural modulation in coarse-resolution layers.Source: §4.2 · p. 7, Figure 11 · p. 7

4.3. Further Analysis

  • Color can be preserved by setting wc = 0 to deactivate color-related layers in the extrinsic style path or by replacing extrinsic style codes with intrinsic codes in the last 11 layers; structure preservation uses ws < 1.The intrinsic codes also pass through color-transform blocks, making final color more aligned with the target domain.Source: §4.3 · p. 7
  • Interpolating intrinsic and/or extrinsic codes blends styles smoothly; the method also performs well on Pixar, Comic, and Slam Dunk styles, but unseen-style optimization introduces artifacts.The additional datasets contain 122, 101, and 120 images, respectively; unseen-style robustness is left for future work.Source: §4.3 · p. 8, Figure 13 · p. 8, Figure 14 · p. 8

4.4. Limitations

  • Typical failures include lost hat and background detail, unnatural anime noses when preserving photo color, and poor results on underrepresented styles such as curly hair without bangs or extremely large eyes.The authors warn that severe data imbalance can yield unsatisfactory results on underrepresented data.Source: §4.4 · p. 8, Figure 16 · p. 8

5. Conclusion and Future Work

The conclusion presents DualStyleGAN as an extension of StyleGAN that accepts styles from new domains while retaining original-domain style control. It identifies future work on style-image recommendations and data bias.

5. Conclusion and Future Work

  • DualStyleGAN adds a style path to model and modulate intrinsic and extrinsic styles for flexible, diverse artistic portrait generation.The paper reports high-resolution exemplar-based portrait transfer with a friendly data requirement.Source: §5 · p. 8
  • The authors report valid transfer learning with specialized architecture design and a progressive training strategy.They suggest the model-extension idea may apply to broader image-to-image translation and knowledge distillation tasks.Source: §5 · p. 8
  • Future work will explore recommending a suitable style image and weight vector for each input photo, and using data augmentation to alleviate data bias.Source: §5 · p. 8

A.1. Dataset

The dataset appendix describes the source and size of the cartoon, caricature, and anime portrait datasets, along with their face-alignment procedures.

A.1. Dataset

  • The Cartoon dataset has 317 images, while the Caricature and Anime datasets contain 199 WebCaricature images and 174 Danbooru Portraits images, respectively.Source: §A.1 · p. 11
  • Caricature and anime faces are aligned following FFHQ using facial landmarks; anime landmarks were manually labeled by the authors.WebCaricature includes the caricature-face landmarks.Source: §A.1 · p. 11
  • The Cartoon dataset is provided by Toonify, and the Caricature and Anime datasets are assembled from WebCaricature and Danbooru Portraits, respectively.Source: §A.1 · p. 11

A.2. Network Architecture

The architecture appendix specifies the extrinsic style path's structural and color-modulation blocks and their correspondence to StyleGAN layers.

A.2. Network Architecture

  • The extrinsic path has seven ModRes blocks corresponding to one 4 × 4, two 8 × 8, two 16 × 16, and two 32 × 32 StyleGAN convolution layers.Source: §A.2 · p. 11
  • Each ModRes combines a ResBlock with an AdaIN block; each ResBlock uses two 3 × 3 convolutions, and AdaIN's style code has dimension 512.Source: §A.2 · p. 11
  • The structure transform block Ts has two linear layers, while 11 color transform blocks Tc correspond to StyleGAN's 64 × 64 through 1024 × 1024 convolution layers and ToRGB layer.Each Tc is made up of a linear layer.Source: §A.2 · p. 11

A.3. Network Training

The training appendix specifies compute, iteration schedules, and post-processing settings for destylization, progressive fine-tuning, and latent refinement.

A.3. Network Training

  • Cartoon, caricature, and anime training uses 1,400, 1,000, and 2,100 iterations, respectively, and averages 0.75 hour.Stage III uses 8 NVIDIA Tesla V100 GPUs, batch size 4 per GPU, and the reported adversarial, perceptual, identity, regularization, cross-correlation, and feature-matching weights.Source: §A.3 · p. 11
  • Progressive fine-tuning Stage II trains at layers 7, 6, and 5 for 300, 300, and 3,000 iterations, taking about 1.8 hours.Destylization Stage II optimizes each latent for 300 iterations and processes about 45 images per hour; the trained model can be applied to any style dataset.Source: §A.3 · p. 11
  • Post-processing uses Adam for 100 latent-optimization iterations, taking about 0.46 hour per 100 artistic portrait images.The first seven z+ rows use learning rates of 0.005, 0.005, and 0.0005 for cartoon, caricature, and anime; the last 11 use 0.1, 0.01, and 0.005 for color refinement.Source: §A.3 · p. 11
  • Training the two sampling networks takes about 0.13 hour.Source: §A.3 · p. 11

B. Appendix: Experimental Details of Simulating Fine-Tuning Behavior

The appendix motivates residual paths as a way to simulate fine-tuning, while later experiments show arbitrary-code generation, structural-style advantages over StyleCariGAN, and flexible style manipulation.

B. Appendix: Experimental Details of Simulating Fine-Tuning Behavior

  • Residual blocks most closely reproduce fine-tuned StyleGAN outputs among the tested residual-path modules, so the paper uses them in its design.The simulation compares ResBlocks, channel-wise AdaIN, and spatial-wise DAT while keeping the original StyleGAN fixed and training residual paths on Cartoon data.Source: B. Appendix: Experimental Details of… · pp. 11–12, Figure 17 · p. 12

C.1. Arbitrary Artistic Portrait Generation

C.2. Comparison with StyleCariGAN

  • DualStyleGAN surpasses StyleCariGAN in transferring diverse structure styles, although both render diverse, plausible textures and colors.The comparison uses StyleCariGAN’s official style palette and DualStyleGAN’s Caricature-dataset codes; StyleCariGAN degrades when using example-image codes.Source: §C.2 · pp. 12–13

C.3. Style Manipulation

  • Style manipulation can vary structural qualities such as cuteness or maturity while retaining a cute color style.The example also adjusts structure to better match the hair and face shape in the photo.Source: §C.3 · p. 13

C.4. Comparison with State-of-the-Art Methods

The comparison evaluates portrait-transfer methods through user selections and visual examples; the proposed method receives notable user preference across the reported groups.

C.4. Comparison with State-of-the-Art Methods

  • Twenty-seven subjects made 810 selections across 30 groups, choosing the best of four methods for content preservation and reference-style consistency.Source: §C.4 · p. 13
  • The proposed method receives notable user preference across the 30 result groups.Source: §C.4 · p. 14
  • Full visual comparisons used in the user study are shown for cartoon, caricature, and anime portrait transfer.Source: §C.4 · p. 14, Figure 23 · p. 14, Figure 24 · p. 15

Related papers

Loading 2203.13248v1…