PaperlayerView exact revision on arXiv

Source-linked AI summary

DG-Font: Deformable Generative Networks for Unsupervised Font Generation

Yangchen Xie, Xinyuan Chen, Li Sun, Yue Lu

arXiv:2104.03064v2cs.CV

TL;DR

Font generation is difficult for writing systems with many characters because existing methods often require labor-intensive paired data, while conventional image translation does not capture geometric font style. DG-Font uses unsupervised style-content generation with deformable feature transformations, and experiments report comparable state-of-the-art results plus extension to unseen styles.

  • Problem

    Existing font-generation methods often require large amounts of paired training data, while image-to-image translation’s texture-and-color style representation does not directly fit font generation.

  • Method

    DG-Font separates style and content, uses FDSC to deform low-level content features with predicted offsets, and combines them through a mixer.

  • Results

    DG-Font achieves comparable results to state-of-the-art font-generation methods and can generate characters in unseen styles.

  • Takeaways & Limitations

    The model generates realistic characters without paired images and extends to unseen fonts, according to the authors’ conclusion.

  • Takeaways & Limitations

    The method constrains deformable-convolution offsets because same-colored regions create non-unique solutions that are difficult to optimize.

Abstract

from arXiv · show

Font generation is a challenging problem especially for some writing systems that consist of a large number of characters and has attracted a lot of attention in recent years. However, existing methods for font generation are often in supervised learning. They require a large number of paired data, which is labor-intensive and expensive to collect. Besides, common image-to-image translation models often define style as the set of textures and colors, which cannot be directly applied to font generation. To address these problems, we propose novel deformable generative networks for unsupervised font generation (DGFont). We introduce a feature deformation skip connection (FDSC) which predicts pairs of displacement maps and employs the predicted maps to apply deformable convolution to the low-level feature maps from the content encoder. The outputs of FDSC are fed into a mixer to generate the final results. Taking advantage of FDSC, the mixer outputs a high-quality character with a complete structure. To further improve the quality of generated images, we use three deformable convolution layers in the content encoder to learn style-invariant feature representations. Experiments demonstrate that our model generates characters in higher quality than state-of-art methods. The source code is available at https://github.com/ecnuycxie/DG-Font.

1. Introduction

Font generation supports applications from font-library creation to handwriting imitation, but existing approaches often require costly paired data or auxiliary annotations. DG-Font addresses this with unsupervised deformable generation, and experiments report comparable results to state-of-the-art methods while extending to unseen styles.

  • Font generation is important for font libraries, personalized handwriting, historical handwriting imitation, and OCR or handwriting-identification data augmentation.
  • Chinese, Japanese, and Korean writing systems contain more than 60,000, 50,000, and 11,172 characters, respectively, making manual glyph creation expensive and labor-intensive.
  • Existing font-generation methods often rely on supervised learning with large amounts of paired training samples, while others use auxiliary stroke or radical annotations.
  • DG-Font separates style and content, applies deformable convolution to content features through FDSC, and feeds the transformed outputs into a mixer.
  • Font style involves geometric transformation, stroke thickness, stroke tips, and joined-up writing rather than only texture and color, motivating spatial deformation.
  • Experiments report comparable results to state-of-the-art font-generation methods and the ability to generate characters in unseen styles.

2. Related works

Related work spans image-to-image translation, supervised and unsupervised font generation, auxiliary structural annotations, and deformable convolution. These approaches provide context for DG-Font’s use of unsupervised style-content translation with spatial deformation.

  • Image-to-Image Translation: Image-to-image translation learns mappings from source-domain images to target-domain images and includes supervised, cycle-consistent, and multi-style approaches.
  • Proposed Method: DG-Font’s overview combines style and content encoders, FDSC modules, a mixer, and a discriminator for unsupervised font generation.
  • Font Generation: Font-generation methods such as Zi2zi and Rewrite use GANs with thousands of paired character examples for strong supervision.
  • Font Generation: Other font-generation methods use stroke or radical decomposition to provide structural guidance and improve generation quality.
  • Deformable Convolution: Deformable convolution augments regular convolution with offsets that change spatial sampling locations, improving modeling of geometric transformations.
  • Deformable Convolution: Prior generation studies apply deformable convolution for tasks including video super-resolution and novel-view synthesis, while DG-Font applies it to font generation.

3. Methods

DG-Font performs unsupervised font generation by combining style and content representations with deformable feature transformations. Its FDSC modules deform low-level content features using style guidance, while multiple losses preserve realism, content consistency, reconstruction, and controlled offsets.

  • Network architecture: The network encodes style into Zs, content into spatial features Zc, and mixes them to generate the content character in the target font.The content encoder uses three deformable convolution layers to produce style-invariant features, followed by residual blocks.
  • Feature deformation skip connection: FDSC predicts sampling parameters from concatenated content and style-guidance maps, then applies deformable convolution to transform low-level content features.The deformed feature is transferred to the mixer and concatenated with the style-guidance representation through a skip connection.
  • Feature deformation skip connection: The deformable convolution uses offsets and masks over a regular 3×3 kernel grid to sample content features at geometrically transformed locations.The offsets may be fractional, with bilinear interpolation used for the irregular sampling positions.
  • Feature deformation skip connection: FDSC operates on feature maps after the first and second convolution layers because low-level features retain more spatial information.The model uses two FDSC modules at these levels.
  • Loss function: The model combines adversarial, content-consistency, image-reconstruction, and deformation-offset normalization losses for unsupervised training.These losses respectively target realistic images, preserved content features, reconstruction of the source image in its origin style, and controlled deformation offsets.
  • Loss function: Offset normalization constrains deformation because same-content fonts place corresponding strokes close together while large same-color regions create non-unique sampling solutions.The constraint focuses deformation on stroke relationships such as thickness and tips.

4. Experiments

Experiments evaluate DG-Font on Chinese font generation across seen and unseen fonts, compare it with existing methods, and analyze its components. DG-Font performs strongly in perceptual metrics and benefits from deformable convolution, FDSC modules, offset normalization, and learned feature deformation.

  • Dataset: The dataset contains 410 fonts and 990 commonly used Chinese characters per font, with 400 fonts used for training.Testing includes unseen characters from training fonts and 10 entirely unseen fonts.
  • Quantitative comparison: DG-Font is comparable to existing methods on pixel-level metrics, including L1 loss, RMSE, and SSIM.These metrics measure pixel-wise similarity between generated images and ground-truth images.
  • Quantitative comparison: DG-Font outperforms compared methods on FID and LPIPS for both seen and unseen fonts.These perceptual-level metrics are described as closer to human perception than pixel-level metrics.
  • Qualitative comparison: Compared with other methods, DG-Font better preserves complete character structures while transforming strokes, skeletons, and joined-up writing patterns.The qualitative comparison covers relatively simple fonts and more challenging fonts with rich details and cursive writing.
  • Ablation study: Replacing the first three content-encoder convolutions with deformable convolutions improves L1 loss, RMSE, and SSIM.The ablation study uses a baseline with normal convolutions and no FDSC modules.
  • Ablation study: Adding FDSC preserves more structural information, offset normalization makes style more target-like, and a second FDSC module yields more detail, less noise, and better quantitative results.The full model successively adds deformable convolution, FDSC-1, normalization, and FDSC-2.
  • Skip-connection comparison: Models with FDSC modules outperform versions using U-Net skip-connections.The comparison replaces both FDSC modules with standard skip-connections.
  • Feature analysis: Visualized FDSC offsets mainly affect character regions, while background offsets tend toward zero.The paper uses optical flow and character flow to visualize the learned offsets.

5. Conclusion

The paper proposes an unsupervised font generation model that generates realistic characters without paired images and extends well to unseen fonts. It uses FDSC to transfer deformable low-level spatial information and deformable convolutions to learn style-invariant content representations.

  • DG-Font generates realistic characters without paired images and extends well to unseen fonts.
  • FDSC transfers deformable low-level spatial information to the mixer to preserve generated character integration.
  • Deformable convolution layers in the content encoder learn style-invariant feature representations.
  • Extensive experiments on Chinese font generation verify the effectiveness of the proposed model.

A.1 Training Strategy

The training strategy initializes convolutional and linear layers with specified distributions, uses separate optimizers for style and content-related components, and trains with adversarial loss and regularization.

  • Convolutional layers use He initialization, while linear-layer weights are sampled from N(0, 0.01) and biases are set to zero.
  • Adam optimizes the style encoder, while RMSprop optimizes the content encoder and mixer.
  • The framework is trained for 200K iterations with learning rate 0.0001 and weight decay 0.0001.
  • Training uses a hinge-version adversarial loss with R1 regularization.

A.2 Network Architecture

The proposed model is an encoder-decoder network with separate style and content encoders, a mixer, and discriminators. Its symmetric content encoder and mixer architectures help preserve domain-invariant content information.

  • The model is an encoder-decoder network with a VGG-11-based style encoder.
  • The style encoder extracts style information from input images.
  • The content encoder and mixer have symmetric architectures that help preserve domain-invariant content information.
  • Detailed architectures for the encoders, mixer, and discriminator are provided in Tables 5 and 6.

B. Ablation Study

The ablation study evaluates loss weights and the number of FDSC modules on 400 seen fonts and 187 handwritten fonts. Results identify useful reconstruction and offset-normalization weights, while showing diminishing or negative returns from additional FDSC modules.

  • The evaluation of objective-loss hyperparameters uses metrics computed on 400 seen fonts.
  • λcnt = 0.5 provides a good trade-off, whereas large values degrade performance and removing content reconstruction performs worse.
  • λimg = 0.1 performs well; excessive image reconstruction weighting harms generation of new characters by emphasizing input reconstruction.
  • λoffset = 0.5 significantly improves generation quality in deformable offset normalization.
  • The FDSC-module ablation varies N, the number of FDSC modules, as defined in the Table 3 caption.
  • Adding one FDSC module for deformed low-level features significantly improves performance, but higher-level modules provide little or negative benefit.The second module yields no obvious improvement, while the third reduces performance.

D. More Results

The method is evaluated on 24 calligraphy and handwriting fonts spanning diverse geometric and stroke styles. It produces high-quality generated characters, with Figure 8 contrasting input style fonts and corresponding imitation results.

  • The evaluation uses 24 calligraphy and handwriting fonts with varied geometric transformations, stroke thicknesses, tips, and joined-up writing patterns.
  • The reported results show that the proposed method outputs high-quality characters across these diverse font styles.
  • Table 4 presents the impact of the selected hyper-parameters.
  • Table 5 specifies the generative network architecture and defines BN, IN, AdaIN, and FC as normalization and fully connected components.
  • Table 6 specifies the discriminator architecture, with AvgPool denoting average pooling and the LeakyReLU slope set to 0.2.
  • Figure 8 places the ten style-font columns on the left and the ten corresponding generated imitation-result columns on the right for each case.
Loading 2104.03064v2…