PaperlayerView exact revision on arXiv

Source-linked AI summary

Multiple Heads are Better than One: Few-shot Font Generation with Multiple Localized Experts

Song Park, Sanghyuk Chun, Junbum Cha, Bado Lee, Hyunjung Shim

arXiv:2104.00887v1cs.CV

TL;DR

Few-shot font generation must preserve target-character structure while expressing diverse local reference styles, but prior approaches struggle with localized styles or unseen components. MX-Font uses multiple localized experts with weak component supervision and content-style disentanglement. It outperforms prior state-of-the-art methods in in-domain Chinese and zero-shot Chinese-to-Korean generation.

  • Problem

    FFG must preserve global character structure and diverse local styles, while prior methods struggle with localized styles or generalizing to unseen components and languages.

    Source: §1 · p. 1

  • Method

    MX-Font uses multiple localized experts, weakly guided by component labels but not explicitly mapped to components, to learn local features and disentangle content from style.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    MX-Font outperforms prior state-of-the-art FFG methods in both in-domain Chinese generation and zero-shot Chinese-to-Korean generation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The method generates plausible font images that preserve reference local style and source-character precision in the reported settings.

    Source: §5 · p. 9

Abstract

from arXiv · show

A few-shot font generation (FFG) method has to satisfy two objectives: the generated images should preserve the underlying global structure of the target character and present the diverse local reference style. Existing FFG methods aim to disentangle content and style either by extracting a universal representation style or extracting multiple component-wise style representations. However, previous methods either fail to capture diverse local styles or cannot be generalized to a character with unseen components, e.g., unseen language systems. To mitigate the issues, we propose a novel FFG method, named Multiple Localized Experts Few-shot Font Generation Network (MX-Font). MX-Font extracts multiple style features not explicitly conditioned on component labels, but automatically by multiple experts to represent different local concepts, e.g., left-side sub-glyph. Owing to the multiple experts, MX-Font can capture diverse local concepts and show the generalizability to unseen languages. During training, we utilize component labels as weak supervision to guide each expert to be specialized for different local concepts. We formulate the component assign problem to each expert as the graph matching problem, and solve it by the Hungarian algorithm. We also employ the independence loss and the content-style adversarial loss to impose the content-style disentanglement. In our experiments, MX-Font outperforms previous state-of-the-art FFG methods in the Chinese generation and cross-lingual, e.g., Chinese to Korean, generation. Source code is available at https://github.com/clovaai/mxfont.

1. Introduction

Few-shot font generation must preserve a target glyph’s global structure while transferring diverse local reference styles. MX-Font uses multiple localized experts to address limitations in prior methods and reports gains in Chinese and cross-lingual generation.

1. Introduction

  • FFG must preserve the target character’s detailed structure while capturing diverse local styles such as stroke, thickness, size, and serif-ness.The paper notes that damage to a local Chinese-glyph component can hurt the character’s meaning.Source: §1 · p. 1
  • Existing methods either struggle to capture diverse local styles or depend on component labels that hinder generalization to unseen components and languages.Universal-style methods use one style representation; component-conditioned methods extract localized features but remain tied to target-language component labels.Source: §1 · p. 2
  • MX-Font uses multiple experts to learn different local concepts without explicitly mapping experts to specific components.Component and style classifiers provide weak supervision, and a graph-matching assignment solved by the Hungarian algorithm discourages experts from learning the same component.Source: §1 · p. 2
  • MX-Font outperforms prior state-of-the-art methods in both in-domain Chinese generation and zero-shot Chinese-to-Korean generation.With four references, Figure 1 illustrates generation of unseen Korean glyphs using the Chinese font generation model.Source: Figure 1 · p. 1, §1 · p. 2
  • Independence and content-style adversarial losses are used to enforce disentanglement within each localized expert.The paper reports that experts specialize in local areas with image-level, rather than pixel-level, component supervision.Source: §1 · p. 2

2. Related Works

The related work distinguishes style transfer from image-to-image translation and contrasts many-shot generation with few-shot font generation. Within FFG, MX-Font uses localized style representations without requiring explicit component labels at test time.

2. Related Works

  • Style-transfer methods treat texture or color as style, whereas font generation defines style through local shape; image-to-image translation learns domain mappings instead.FFG methods often use image-to-image translation, including FUNIT’s transfer of reference style while preserving content.Source: §2 · p. 2
  • Many-shot methods may require fine-tuning on hundreds of reference glyphs, while FFG targets unseen fonts without costly fine-tuning or large reference sets.The passage gives 775 glyphs as an example of a many-shot method’s reference count.Source: §2 · p. 3
  • Universal-style FFG methods use one style vector, while component-conditioned methods use localized styles but require target-character component labels even at test time.MX-Font instead uses multiple expert features; its generator combines style and content features to produce the target glyph.Source: §2 · p. 3, Figure 3 · p. 3

3. Method

MX-Font uses multiple localized experts to extract local features, weakly guides their specialization with component labels, and combines source content with reference style for few-shot glyph synthesis.

3.1. Model architecture

  • Each expert encodes a glyph into a spatial local feature split into content and style, while training-only classifiers supervise style and component-label information.The classifiers are independent of inference.Source: §3.1 · p. 3

3.2. Learning multiple localized experts with weak local component supervision

  • Image-level component labels guide experts without locating components; prediction-weighted assignments use Hungarian matching, and experts empirically attend distinct areas without pixel-level supervision.The allocation is formulated as weighted bipartite matching.Source: §3.2 · pp. 4–5

3.3. Content and style disentanglement

  • Content-style adversarial and independence losses encourage each local content and style representation to retain its domain information while excluding the other.The adversarial objective trains features to fool the classifier for the opposite domain; the independence loss is applied between local content and style features.Source: §3.3 · p. 5

3.4. Training

  • Training combines adversarial, feature-matching, and reconstruction objectives with the content-style and independence losses in an end-to-end weighted sum.The model is trained for 650k iterations.Source: §3.4 · p. 5

3.5. Few-shot generation

  • At generation time, MX-Font averages each expert’s style features across the reference glyphs and combines them with content features from the source glyph.The combined expertwise features are used to generate an unseen-style glyph.Source: §3.5 · p. 6

4. Experiments

The experiments compare MX-Font with prior few-shot font-generation methods on Chinese and Korean generation, then analyze its localized experts and training objectives. Results favor MX-Font on most metrics, particularly in cross-lingual generation, while analyses examine expert specialization, supervision, and loss contributions.

4.1. Comparison methods

  • Among the compared methods, universal-style approaches use one style representation, while component-conditioned methods rely on explicit components and may not handle unseen components.DM-Font cannot generate characters with unseen components; modified LF-Font is included in comparisons despite the same limitation.Source: §4.1 · p. 6

4.2. Evaluation protocols

  • Evaluation covers Chinese in-domain transfer and Chinese-to-Korean zero-shot cross-lingual generation, using style/content classifiers, user study, LPIPS for Chinese, and FID(H).The test set uses 28 styles, 214 Chinese characters, and 245 Korean characters.Source: §4.2 · pp. 6–7

4.3. Experimental results

  • MX-Font outperformed competitors on most metrics in Chinese and Korean generation, with especially strong cross-lingual performance and better in-domain classification accuracy and user-study results.The comparison averages 50 runs with four reference images per style.Source: Table 1 · p. 7, §4.3 · p. 7
  • MX-Font performed worse than other methods on Chinese FID, which the authors note is sensitive to noisy or blurred images.The authors report better user-study performance across all criteria.Source: §4.3 · p. 7
  • MX-Font produces more detailed content and style structures; in cross-lingual samples, competing models often generate unrecognizable characters while MX-Font remains plausible and recognizable.The figure contrasts generated samples with references, source images, and available ground truth.Source: §4.3 · pp. 7–8, Figure 6 · p. 7

4.4. Analyses

  • CAM visualizations show experts attending to different local areas without explicit pixel-level supervision; six experts also preserve local details better than a single expert.The single-expert samples omit important strokes, while multi-expert samples capture local detail.Source: §4.4 · p. 8, Figure 7 · p. 8, Table 2 · p. 8
  • Replacing component supervision with character supervision incurs mode collapse; the authors suggest the larger character-label set and lack of an allocation constraint may contribute.The comparison uses the component classifier versus the character classifier as weak supervision.Source: §4.4 · p. 8, Table 3 · p. 8
  • Ablations on Korean handwriting show that the content-style adversarial loss, maximizing-entropy term, and independence loss each contribute to overall performance.The table compares models trained and tested on the Korean handwriting dataset.Source: Table 4 · p. 8, §4.4 · p. 9

5. Conclusion

MX-Font uses weak local-component supervision with a multi-headed encoder to learn localized features for disentangling style and content. The paper reports better results than existing methods in both Chinese in-domain and zero-shot cross-lingual generation.

5. Conclusion

  • MX-Font combines weak local-component supervision and a multi-headed encoder to learn localized features, generating plausible fonts that preserve reference style details and source-character structure.The conclusion describes this as targeting both rich local-detail representation and generalization to unseen components and languages.Source: §5 · p. 9

Appendix

  • The appendix provides additional experimental results, implementation details, and evaluation protocols.Source: Appendix · p. 11

A. Additional experimental results

Additional experiments show readable cross-lingual generations and examine how the number of experts affects in-domain Chinese transfer. Performance improves as the expert count rises to six, while eight experts perform slightly worse.

A.1. More visual examples

  • MX-Font produces readable cross-lingual glyphs, while competing methods often produce inconsistent results that are difficult to understand.The examples also show synthesis of strokes, dots, thickness, and size.Source: §A.1 · p. 11

A.2. Impact of the number of experts

  • In-domain Chinese transfer improves as the number of experts increases through k = 6, but k = 8 performs slightly worse than k = 6.The authors attribute the drop to insufficient training examples with at least eight components; larger k also increases parameters and runtime, so they use k = 6.Source: §A.2 · p. 11

B.1. Network architecture

The supplementary material describes MX-Font’s expert and generator architecture, its component-allocation formulation, and its training objectives. The objectives include feature independence and adversarial discrimination for content and style.

B.1. Network architecture

  • Each localized expert has 11 layers, and the experts share their first five blocks; the classifiers share their first two residual blocks, while the generator uses convolutional and residual blocks.The expert blocks include convolution, residual, global-context, and CBAM components.Source: §B.1 · p. 11

B.2. Component allocation problem to weighted bipartite B-matching problem

  • The component-allocation problem is represented as weighted bipartite B-matching, maximizing selected edge weights subject to vertex budgets, and is solved with the Hungarian algorithm in O(|V|^3).The supplementary figure provides additional generation samples using four reference glyphs.Source: §B.2 · p. 11, Figure A.2 · p. 12

B.3. HSIC Formulation

  • HSIC is used to measure independence between content and style feature variables, with an unbiased finite-sample estimator computed from mini-batch samples.The formulation uses RBF kernels; with these kernels, HSIC is zero if and only if the variables are independent.Source: §B.3 · p. 13

B.4. GAN objective details

  • A multitask discriminator predicts style and content labels, while hinge, feature-matching, and reconstruction losses support generation and training.The discriminator’s layer outputs are used in the feature-matching and reconstruction formulations.Source: §B.4 · p. 13

B.5. Training details

Training uses different schedules for Chinese and Korean generation, with expert counts and objectives adjusted for the scripts. The Korean model omits the expert-independence loss because Korean characters decompose into a fixed number of components.

B.5. Training details

  • The Chinese model uses k = 6 and trains for 650k iterations, while the Korean model uses k = 3 and trains for 200k iterations.For Korean generation, training omits the expert-independence loss because the script has a fixed number of components, such as three.Source: §B.5 · p. 13
  • Training batches use a target glyph plus three same-style glyphs and three same-character glyphs, with the target excluded from both selections.The model is optimized with Adam; the discriminator learning rate is 0.001 and the remaining modules use 0.0002.Source: §B.5 · p. 13

C.1. Classifiers

The supplementary material specifies classifier training, describes a test-time adjustment to LF-Font for unseen components, and shows the images used in the user study. The user-study examples include reference, source, and generated images.

C.1. Classifiers

  • The style and Chinese-character classifiers use a Chinese dataset of 209 fonts and 6,428 characters per font, while the Korean-character classifier uses the Korean dataset provided by DM-Font.The classifiers use ResNet-50 and are trained for 20 epochs with AdamP; CutMix augmentation and batch size 64 are also specified.Source: §C.1 · pp. 13–14

C.3. User study examples

  • User-study examples contain reference, source, and generated images, and five methods are randomly displayed to users for each query.The methods are EMD, AGIS-Net, FUNIT, LF-Font, and MX-Font.Source: Figure C.1 · p. 14, §C.3 · p. 14

C.2. LF-Font modification

  • For cross-lingual testing, LF-Font’s component-condition block is skipped only at test time when an unseen component is given; its original structure is retained for training.Source: §C.2 · p. 14

C.4. FID

This section reports style-aware and content-aware FID measurements for Chinese and cross-lingual generation, and notes that Chinese FID trends do not align with other quality assessments.

  • For cross-lingual generation, style-aware FID compares outputs with available rendered glyphs of the same style because ground-truth glyphs are unavailable.Source: §C.4 · p. 14
  • The appendix reports style-aware (S), content-aware (C), and harmonic-mean (H) FIDs, with H identical to main-table values.Source: Table C.1 · p. 14
  • Chinese-generation FID shows slight degradation for MX-Font, unlike the user-study and qualitative evaluations.Source: §C.4 · p. 14
Loading 2104.00887v1…