PaperlayerView exact revision on arXiv

Source-linked AI summary

Few-shot Compositional Font Generation with Dual Memory

Junbum Cha, Sanghyuk Chun, Gayoung Lee, Bado Lee, Seonghyeon Kim, Hwalsuk Lee

arXiv:2005.10510v2cs.CVcs.GRcs.LG

TL;DR

Font creation for glyph-rich scripts is labor-intensive, and existing methods either need many references or struggle with unseen styles. DM-Font uses compositional supervision and dual memories to generate fonts from few samples, and consistently outperforms existing few-shot methods on Korean and Thai letters.

  • Problem

    Font creation for glyph-rich scripts is labor-intensive, while existing methods require many references or fail to generate high-quality fonts for unseen styles.

    Source: §1 · p. 1

  • Method

    DM-Font combines persistent and dynamic memories with weakly supervised component labels, requiring neither component bounding boxes nor masks.

    Source: §4 · p. 4, §4.1 · pp. 5–6

  • Results

    DM-Font consistently outperforms existing few-shot font-generation methods on Korean and Thai letters, while prior methods fail to stylize unseen fonts.

    Source: §6 · p. 14

  • Takeaways & Limitations

    The framework generates high-quality full font libraries from 28 Korean or 44 Thai reference samples.

    Source: §4 · p. 4

    Training assumes different font labels represent different styles and encodes only a core component subset for efficiency.

    Source: §4.2 · p. 6

Abstract

from arXiv · show

Generating a new font library is a very labor-intensive and time-consuming job for glyph-rich scripts. Despite the remarkable success of existing font generation methods, they have significant drawbacks; they require a large number of reference images to generate a new font set, or they fail to capture detailed styles with only a few samples. In this paper, we focus on compositional scripts, a widely used letter system in the world, where each glyph can be decomposed by several components. By utilizing the compositionality of compositional scripts, we propose a novel font generation framework, named Dual Memory-augmented Font Generation Network (DM-Font), which enables us to generate a high-quality font library with only a few samples. We employ memory components and global-context awareness in the generator to take advantage of the compositionality. In the experiments on Korean-handwriting fonts and Thai-printing fonts, we observe that our method generates a significantly better quality of samples with faithful stylization compared to the state-of-the-art generation methods quantitatively and qualitatively. Source code is available at https://github.com/clovaai/dmfont.

1 Introduction

Generating fonts for glyph-rich scripts is labor-intensive, while existing methods require many references or struggle with unseen styles. DM-Font uses component supervision and dual memories to generate Korean and Thai fonts from few samples.

1 Introduction

  • Manual font-library design is costly for glyph-rich scripts, including Chinese (>50,000 glyphs), Korean (11,172), and Thai (11,088).Source: §1 · p. 1
  • Many-shot GAN methods require 775 samples and target-font fine-tuning, limiting practicality when collecting target glyphs is costly.Source: §1 · pp. 1–2
  • Existing few-shot methods often fail to produce high-quality font libraries for unseen styles, despite succeeding on training styles.The paper attributes its approach to using glyph characteristics rather than relying only on end-to-end data-driven methods.Source: §1 · p. 2
  • In compositional scripts, component shapes and positions vary with their combination, so a complete component inventory alone does not yield a full font set.Source: §1 · p. 2
  • DM-Font uses weak component-label supervision and dual memories to generate high-quality Korean and Thai font libraries from 28 and 44 samples, respectively.Its experiments also report better quantitative metrics and user-study preference.Source: §1 · p. 2

2 Related Works

Prior font-generation work includes few-shot image translation and methods tailored to automatic font generation. The paper distinguishes these approaches by their sample requirements, script coverage, and ability to render unseen styles.

2.1 Few-shot image-to-image translation

  • Few-shot image-to-image translation addresses scarce target samples and can be applied to font generation as translation between reference and target fonts.Source: §2.1 · p. 3
  • Style-transfer methods are not used as font-generation baselines because their texture-and-color notion of style differs from fonts' discriminative local properties.Source: §2.1 · p. 3

2.2 Automatic font generation

  • Many-shot methods fine-tune on large target-font sets, while few-shot methods avoid that requirement but may be limited to small scripts or struggle with unseen styles.Whole-font-set single-pass methods need large model capacity and may not suit glyph-rich scripts; EMD and AGIS-Net apply to general scripts but synthesize unseen styles less well, while SA-VAE uses character-wise rather than component-wise features.Source: §2.2 · p. 3

3 Preliminary: Complete Compositional Scripts

Complete compositional scripts form glyphs from fixed sets of components, but component appearance can vary with its combination. Korean and Thai illustrate how relatively small component sets cover many valid characters.

3 Preliminary: Complete Compositional Scripts

  • The same Korean sub-glyph can change shape and position depending on the glyph combination.Source: Figure 2 · p. 4
  • Complete compositional scripts decompose each glyph into a fixed number of sub-glyphs, whereas incomplete scripts can require varying numbers.Korean uses three sub-glyphs per glyph; Thai uses four components.Source: §3 · p. 4
  • Korean has 19 × 21 × 28 = 11,172 valid characters from three component types, and 28 glyphs suffice to cover the full sub-glyph set.Thai represents 11,088 characters through 44 × 7 × 9 × 4 components, with 44 characters needed to cover its full sub-glyph sets.Source: §3 · p. 4

4 Dual Memory-augmented Font Generation Network

DM-Font uses compositionality and two memory modules to generate full glyph libraries from very few references. Its architecture combines component-level style features with script-wide composition information, while training uses component labels and several complementary losses.

4 Dual Memory-augmented Font Generation Network

  • DM-Font separates global composition information from local styles in persistent and dynamic memories to generate full glyph libraries from very few references.The paper gives 28 Korean or 44 Thai samples as examples.Source: §4 · p. 4

4.1 Architecture overview

  • Persistent memory stores learned component embeddings for intrinsic shape and script-global information, while dynamic memory stores encoded features from the reference glyphs.Persistent memory is fixed after training; dynamic memory adapts to the reference style samples.Source: §4.1 · p. 5
  • The generator uses encoder self-attention for component relationships and a decoder hourglass block for global context while preserving locality; component labels, not bounding boxes or masks, supervise compositionality.A multitask discriminator handles font and character conditions, and a component classifier adds generator supervision.Source: §4.1 · p. 6

4.2 Learning

  • Training treats font labels as style labels and encodes only the core components needed to compose each target glyph, rather than the full component set.For Korean, the full set has 68 components, but a single character requires only three.Source: §4.2 · pp. 6–7
  • Adversarial loss encourages plausible images, L1 loss supervises against target glyphs, and discriminator feature matching is used to improve training stability.The font discriminator is conditioned on font and the character discriminator classifies character identity.Source: §4.2 · p. 7
  • A component-classification loss trains encoded features against decomposition labels, and the final objective combines it with adversarial, L1, and feature-matching losses.The component labels are derived from the target character’s decomposition.Source: §4.2 · p. 7

5 Experiments

Experiments evaluate DM-Font on Korean-handwriting, Thai-printing, and unrefined Korean data using quantitative, visual, and human-preference measures. DM-Font outperforms comparison methods on most metrics, particularly style-aware measures, and shows better generalization to unseen characters.

Korean-handwriting dataset; Thai-printing dataset; Korean-unrefined dataset; Comparison methods; Evaluation metrics

Quantitative evaluation

  • DM-Font outperforms comparison methods on most metrics, especially style-aware benchmarks, with style-aware accuracy of 62.6% for Korean unseen characters and 50.6% for Thai unseen characters.Other methods reach about 5% on those unseen character sets.Source: Quantitative evaluation · pp. 9–10, Table 1 · p. 9, Table 2 · p. 10
  • DM-Font generalizes better to unseen characters in both datasets, while AGIS-Net’s Korean content-aware accuracy, PD, and mFID worsen from 98.7 to 98.3, 0.018 to 0.019, and 23.9 to 25.9.The authors attribute DM-Font’s generalization to component-level glyph interpretation and memory-based reuse of learned component features.Source: Quantitative evaluation · p. 10

Qualitative comparison

  • Visual comparisons show more faithful target-style transfer by DM-Font, while baselines can miss detailed component styles or lose content in unseen Thai characters.The paper attributes preservation of component details to the dual memory’s reuse of learned details when generating new glyphs.Source: Qualitative comparison · pp. 10–11, Figure 4 · p. 11, Figure 5 · p. 12

User study

5.4 More analysis

  • Adding dynamic memory, persistent memory, and the compositional generator improves overall performance; memory modules raise style scores while preserving content scores.The ablation baseline, similar to FUNIT, has content and style accuracies of 93.9 and 5.4.Source: §5.4 · p. 12, Table 4 · p. 13
  • Removing L1 and feature-matching losses slightly degrades performance, while component-classification loss is the most important objective for successful training.Component-classification loss enforces compositionality in the model.Source: §5.4 · p. 12, Table 4 · p. 13

Style overfitting of baselines; Component-wise style mixing

6 Conclusions

DM-Font targets few-shot generation for compositional scripts, using two external memories and weak compositionality supervision without component boxes or masks; it outperforms existing few-shot methods on Korean and Thai letters. Its memory addressor maps character labels to component labels using language-specific Unicode-based decomposition.

6 Conclusions

  • DM-Font consistently outperforms existing few-shot font-generation methods on Korean and Thai letters, while prior methods fail to stylize unseen fonts.The conclusion attributes its high-quality few-shot samples to using compositionality.Source: §6 · p. 14
  • Weak compositionality supervision requires component labels but no component-wise bounding boxes or masks during training.The framework uses dynamic and persistent memories to incorporate compositional-script prior knowledge.Source: §6 · p. 14

A.1 Memory addressors

  • The memory addressor maps a character label to component labels using language-specific Unicode-based decomposition functions.For Korean, the function decomposes characters by a predefined rule; for Thai, each constituent Unicode is itself a component label, and the function determines its component type.Source: §A.1 · p. 18

A.2 Network architecture

The architecture combines global-context awareness with local-style preservation: it reasons across components while retaining source-glyph style through multi-level features and dual memory. Its generator and discriminator use distinct blocks and classification outputs.

A.2 Network architecture

  • Global-context awareness supports relational reasoning between components, while local-style preservation reflects the source glyph’s local style in the target.The architecture uses multi-level features within the dual-memory framework to preserve local style.Source: §A.2 · p. 18
  • The encoder uses GCBlock and SABlock, and the decoder uses HGBlock to extend the receptive field globally while preserving locality.The SABlock uses multi-head self-attention and position-wise feed-forward; the HGBlock reduces the feature map to 1 × 1 and restores its original size.Source: §A.2 · pp. 18–19
  • The encoder has multiple heads according to the number of component types T.Figure A.1 also depicts the spatial size of each block.Source: Figure A.1 · p. 19
  • The discriminator feeds a spatially squeezed feature map to separate font and character discriminators, each producing binary outputs for its target classes.The discriminator consists of a convolution block followed by residual blocks; its output is squeezed to 1×1.Source: §A.2 · p. 19
  • Persistent memory is sized to the encoder’s high-level features, refined by three convolution blocks, and added to dynamic-memory features before decoding.A component classifier identifies the class of high-level component features from dynamic memory.Source: §A.2 · p. 19

B.1 DM-Font implementation details

Training uses Adam, hinge GAN loss, and generator exponential moving averages, with distinct learning rates and iteration counts for Korean and Thai experiments.

B.1 DM-Font implementation details

  • The Korean experiments train for 200K iterations using generator and discriminator learning rates of 0.0002 and 0.0008, respectively.The component classifier uses the generator’s learning rate; training uses Adam with the two time-scale update rule and hinge GAN loss.Source: §B.1 · p. 20
  • Thai-printing experiments use generator and discriminator learning rates of 0.00005 and 0.0001, respectively, for 250K iterations.Other settings are the same as in the Korean experiments.Source: §B.1 · p. 20
  • Training applies spectral normalization to the discriminator and exponential moving averages to the generator.The model is trained with hinge GAN loss.Source: §B.1 · p. 20

B.2 Evaluation classifier implementation details

Evaluation uses separately trained content and style classifiers, a Korean user study, and repeated reference-sample selections. The reported results show high classifier validation accuracy and robustness to reference-sample selection.

B.2 Evaluation classifier implementation details

  • The evaluation classifiers achieve 97.9% Korean content, 96.0% Korean style, 99.6% Thai content, and 99.99% Thai style validation accuracy.They are separately trained ResNet-50 classifiers and are used only for evaluation, not DM-Font training.Source: §B.2 · p. 20

B.3 User study details

  • The Korean user study collected 3,420 responses from 38 Korean natives comparing four methods and ground truth across content, style, and preference.It used 30 styles and eight randomly selected characters per style.Source: §B.3 · p. 20

C.1 Reference set sensitivity

  • Eight Korean-handwriting runs with different reference samples support that DM-Font is robust to reference-sample selection.Table C.1 is identified as reporting reference-sample sensitivity across those runs.Source: §C.1 · p. 20, Table C.1 · p. 21

C.2 Results on the Korean-unrefined dataset

On the Korean-unrefined dataset, DM-Font performs consistently across the reported evaluation metrics, though accuracy could not be measured.

C.2 Results on the Korean-unrefined dataset

  • DM-Font consistently shows strong performance across all reported evaluation metrics on the Korean-unrefined dataset.The evaluation uses classifiers trained on the Korean-handwriting dataset, so it reports perceptual distance and mFID but not accuracy.Source: §C.2 · p. 21
  • Accuracy is not measurable with the classifiers used for this dataset.The classifiers were trained on the Korean-handwriting dataset.Source: §C.2 · p. 21
  • Figure C.1 presents generated samples from the Korean-unrefined dataset used in the user study.Source: §C.2 · p. 21

C.3 Ablation study

The appendix reports improvements from the proposed components and objectives, illustrates failure cases and component-shape variation, and compares style generalization between seen and unseen fonts.

C.3 Ablation study

  • The proposed components and objective functions significantly improve generation quality across the reported ablation metrics.Table C.3 reports ablations on the Korean-handwriting dataset; higher is better except for perceptual distance and mFID.Source: §C.3 · p. 21, Table C.3 · p. 22

C.4 Failure cases

  • Complex glyphs can produce incorrect content, while observing only one sample per component leaves multimodal styles ill-posed.The authors suggest content-failure detection and user-guided correction as possible responses.Source: §C.4 · p. 24
  • Damaged reference glyphs make style and content difficult to disentangle, and the model may ignore the damaged style in favor of memorized local styles.Source: §C.4 · p. 24

C.5 Examples of various component shape in generated glyphs

  • Generated component shapes vary with sub-glyph composition, supporting the claim that DM-Font combines local styles with global composition rather than copying references.The authors conclude that the model disentangles local style and global composition using only a few references.Source: §C.5 · pp. 24–25

C.6 Generalization gap between seen and unseen fonts

  • DM-Font shows a significantly better style generalization gap than comparison methods on seen versus unseen fonts.The comparison methods perform well on seen training fonts but fail to generalize to unseen validation styles; the evaluation uses unseen characters, and smaller gaps indicate better generalization.Source: §C.6 · p. 25, Table C.4 · p. 25
Loading 2005.10510v2…