Source-linked AI summary
Separating Style and Content for Generalized Style Transfer
Yexun Zhang, Ya Zhang, Wenbin Cai, Jie Chang
TL;DR
Existing style-transfer methods often require retraining for new styles, limiting generalization. The paper proposes EMD, which separates style and content representations, combines them with a bilinear mixer, and generates target images; experiments on Chinese typeface transfer demonstrate effectiveness and robustness, while future work remains on learning-to-learn integration.
Problem
Existing style-transfer methods generally learn transformations for training styles and require retraining to handle new styles.
Method
EMD uses separate style and content encoders, a bilinear mixer, and a decoder to generate images from small style and content reference sets.
Results
Experiments on Chinese typeface transfer show that EMD generates novel styles with 10 style reference images and outperforms several baselines while approaching stronger zi-to-zi models.
Takeaways & Limitations
Separating style and content supports simultaneous transfer among multiple styles and generalization to novel but related styles and contents.
Takeaways & Limitations
The authors leave further exploration of learning-to-learn and its integration with the framework for future work.
Abstract
from arXiv · showhide
Neural style transfer has drawn broad attention in recent years. However, most existing methods aim to explicitly model the transformation between different styles, and the learned model is thus not generalizable to new styles. We here attempt to separate the representations for styles and contents, and propose a generalized style transfer network consisting of style encoder, content encoder, mixer and decoder. The style encoder and content encoder are used to extract the style and content factors from the style reference images and content reference images, respectively. The mixer employs a bilinear model to integrate the above two factors and finally feeds it into a decoder to generate images with target style and content. To separate the style features and content features, we leverage the conditional dependence of styles and contents given an image. During training, the encoder network learns to extract styles and contents from two sets of reference images in limited size, one with shared style and the other with shared content. This learning framework allows simultaneous style transfer among multiple styles and can be deemed as a special `multi-task' learning scenario. The encoders are expected to capture the underlying features for different styles and contents which is generalizable to new styles and contents. For validation, we applied the proposed algorithm to the Chinese Typeface transfer problem. Extensive experiment results on character generation have demonstrated the effectiveness and robustness of our method.
1. Introduction
Existing style-transfer models generally require retraining for new styles, while this paper proposes separating style and content representations to generalize from small reference sets. The resulting framework mixes the two representations to generate multiple style-content combinations.
- Motivation: Existing methods explicitly learn transformations between styles, so new styles require time-consuming retraining.
- Proposed approach: The proposed network represents styles and contents separately using small sets of reference images, then mixes the corresponding representations.
- Proposed approach: The framework consists of a style encoder, content encoder, bilinear mixer, and decoder that generates images from combined representations.
- Expected capability: Separating style and content enables simultaneous transfer among multiple styles and supports generation for unseen style-content combinations.
2. Related Work
Existing style-transfer methods include optimization-based, feed-forward, GAN-based, and character-specific approaches, but many require retraining for new styles. EMD instead separates style and content representations and combines them through an encoder-decoder architecture.
- Neural Style Transfer: Neural style transfer methods use iterative optimization, feed-forward generators, perceptual or other losses, and increasingly adversarial training.These approaches target image generation with specified styles and contents.
- Character Style Transfer: Character style-transfer systems include Rewrite, zi-to-zi, From A to Z, and AEGN, using supervised labels or specialized generative architectures.The reviewed methods address typographic or calligraphic character synthesis.
- Generalization: Most reviewed methods transfer only styles seen during training and require retraining for new styles, whereas EMD uses small reference sets for novel styles or contents.The paper frames EMD as a generalized alternative to fixed style-transfer mappings.
- Proposed Model: EMD contains style and content encoders, a bilinear mixer, and a decoder that generates images from combined style-content representations.The encoders extract factors from reference images, the mixer integrates them, and the decoder produces the target image.
- Encoder Network: The encoders receive reference sets sharing style or content, concatenated across channels to capture common characteristics.Style reference sets vary in content, while content reference sets vary in style.
- Mixer and Decoder: The bilinear mixer combines style and content features, and the decoder maps the resulting representation to an image with the target style and content.The bilinear formulation is motivated by separability and generalization to unfamiliar factor classes.
4. Experiments
Experiments evaluate EMD on Chinese Typeface transfer across known and novel styles and contents, varying training data, reference-set size, skip-connections, and style-content separation. Comparisons show that EMD generalizes to novel styles and contents from few references and performs competitively with stronger baselines.
- Data Set: The Chinese Typeface dataset contains 832 fonts, 1,732 characters per font, and four subsets spanning known and novel styles and contents.D1 contains known styles and contents; D2 known styles with novel contents; D3 known contents with novel styles; D4 both novel.
- Training Set Size: Larger training sets improve generation, with Nt=300k and Nt=500k clearly outperforming smaller settings while producing similar results.The authors conclude that performance tends to saturate and Nt=300k is sufficient for good performance.
- Reference Set Size: More reference images improve character detail, but r=10 and r=15 perform similarly; r=5 is already adequate for generalizing to novel styles.The authors use r=10 in subsequent experiments and report that r > 1 supports colearning of same-style or same-content images.
- Skip-connection: Skip-connections matter most for novel contents: outputs improve in detail, whereas models with and without them perform closely for known contents.Without the content skip-connection, novel characters may resemble previously seen characters or become blurry or incorrect because structural information is lost during down-sampling.
- Style and Content Separation: Using disjoint reference sets, similar outputs support separate style and content extraction by the Style Encoder and Content Encoder.Figure 7 varies style references while holding content references fixed; analogous results for content references are reported in Figure 8.
- Comparison with Baseline Methods: For known styles with novel contents, EMD performs much better than pix2pix, AEGN, and CycleGAN, and close to or slightly better than zi-to-zi.The comparison attributes weaker baseline performance partly to sample requirements and, for CycleGAN, to learning domain mappings without domain knowledge.
- Comparison with Baseline Methods: EMD can use images of any styles because it learns feature representations rather than transformations between specific style pairs.The authors contrast this with baselines that require many source and target style images and retraining for new styles.
5. Conclusion and Future Work
The paper concludes that EMD transfers learned style and content representations to new styles and contents using few references. Future work will connect this framework more deeply with learning-to-learn.
- Conclusion: EMD extracts style and content representations, mixes them, and generates images with new styles and contents from a few reference images.The framework supports simultaneous transfer among multiple styles and treats the shared encoders and mixer as transferable knowledge.
- Future Work: The learning process targets novel but related image-generation tasks by learning high-level feature representations, resembling a learning-to-learn program.The authors identify integrating learning-to-learn more fully with their framework as future work.
Supplemental Materials: Separating Style and Content for Generalized Style Transfer
The supplement reports additional experiments covering style morphing, factors affecting model performance, quantitative and qualitative results, and neural style transfer.
- Supplemental Materials: Supplementary experiments examine morphing between two styles, model-performance factors, quantitative and qualitative results, and neural style transfer.These experiments are presented to further validate the proposed method.
1. Morphing
The model morphs between two novel styles by linearly interpolating their extracted style representations while keeping content representations available for image generation. The resulting images show a transition from style S1 to style S2.
- Morphing: A new style is synthesized as SNew = (1 −λ) × S1 + λ × S2 for 0 ≤λ ≤1.Changing λ controls the interpolation between the two style representations.
- Morphing: The experiment extracts style features from reference sets RS1 and RS2, combines them with varying λ, and generates images using a content feature.All styles and contents used in the experiment are novel.
- Morphing: The generated images display a changing process from style S1 to style S2 as λ varies from 0.0 to 1.0.Figures 1 and 2 present outputs at λ values from 0.0 through 1.0.
2. Influence of the Training Set Size
Larger training sets improve both quantitative and qualitative performance, but gains eventually saturate as training-set size increases.
- Larger training sets produce better quantitative and qualitative performance.
- Performance saturates as the training-set size continues to increase.
- Figure 3 compares outputs for Nt=20k, 50k, 100k, 300k, and 500k across D1–D4.
3. Influence of Reference Set Size
Increasing the reference-set size improves generation quality, while performance becomes similar for sufficiently large reference sets.
- r=2 performs worst, while r=10 and r=15 perform similarly.
- Performance saturates as the number of reference images increases.
- Figure 4 compares outputs for r=2, 5, 10, 15, and 50 with Nt=300k.
4. Effect of the Weighted Loss
Weighted L1 loss improves the generation of thin and light Chinese characters compared with standard L1 loss.
- Weighted L1 loss generates thin and light characters better than standard L1 loss.
- Table 3 compares models trained with L1 loss and weighted L1 loss.
- Figure 5 shows target images alongside outputs generated with L1 and weighted L1 losses.
5. Results of One Reference Image
Using ten reference images performs similarly to using one reference image, while the ten-reference model is more time efficient during training.
- The r=10 and r=1 models perform similarly, but r=10 is more time efficient.
- The r=10 model learns r^2 style-content pairs simultaneously, whereas r=1 splits each triplet into 100 triplets.
- Figure 6 compares outputs from models using r=10 and r=1.
6. Experiment for Neural Style Transfer
The experiment evaluates neural style transfer on a dataset of artistic Photoshop filters, with results reported in Figure 7. The authors state that their method works well for neural images.
- The dataset contains 106 artistic Photoshop styles, with 781 images of different contents per style.
- Figure 7 presents the experiment results for neural style transfer on this dataset.
- The reported results indicate that the method works well for neural images.