PaperlayerView exact revision on arXiv

Source-linked AI summary

Auto-Encoder Guided GAN for Chinese Calligraphy Synthesis

Pengyuan Lyu, Xiang Bai, Cong Yao, Zhen Zhu, Tengteng Huang, Wenyu Liu

arXiv:1706.08789v1cs.CV

TL;DR

The paper addresses Chinese calligraphy synthesis from standard-font images, where stroke-based methods are complex and limited by difficult extraction. It proposes a two-subnet image-to-image translation model with an autoencoder-based supervise network, and reports the best results on a benchmark spanning four calligraphy styles.

  • Problem

    Chinese calligraphy synthesis must handle diverse styles, stroke-layout deformations, and the limitations of stroke extraction and assembly methods.

  • Method

    A transfer encoder-decoder and an autoencoder-based supervise network are trained end-to-end to generate specified-style calligraphy directly from standard-font images.

  • Results

    The proposed method achieves the best results among the evaluated baselines on the benchmark subsets.

  • Takeaways & Limitations

    Direct image-to-image translation can synthesize calligraphy without extracting strokes, while the supervise network supplies detailed stroke information during training.

Abstract

from arXiv · show

In this paper, we investigate the Chinese calligraphy synthesis problem: synthesizing Chinese calligraphy images with specified style from standard font(eg. Hei font) images (Fig. 1(a)). Recent works mostly follow the stroke extraction and assemble pipeline which is complex in the process and limited by the effect of stroke extraction. We treat the calligraphy synthesis problem as an image-to-image translation problem and propose a deep neural network based model which can generate calligraphy images from standard font images directly. Besides, we also construct a large scale benchmark that contains various styles for Chinese calligraphy synthesis. We evaluate our method as well as some baseline methods on the proposed dataset, and the experimental results demonstrate the effectiveness of our proposed model.

I. INTRODUCTION

The paper frames Chinese calligraphy synthesis as direct image-to-image translation from standard fonts, addressing stroke-extraction limits and deformation between input and target images with a supervised two-subnet network and a multi-style benchmark.

  • I. INTRODUCTION: Chinese calligraphy synthesis is challenging because styles vary widely and standard-font characters differ from calligraphy in stroke layout and style.The two image types retain similar radical layout but can have substantially different stroke arrangements.
  • I. INTRODUCTION: The proposed model directly generates specified-style calligraphy from standard font images, replacing stroke extraction and assembly with image-to-image translation.The approach targets cases where complex or cursive strokes are difficult to separate.
  • I. INTRODUCTION: The proposed architecture combines a transfer encoder-decoder with an autoencoder-based supervise network that guides low-level decoder features during end-to-end training.The architecture is described as jointly trained, with the supervise network providing stroke-detail information.
  • I. INTRODUCTION: The paper establishes a benchmark with four calligraphy styles and about 7,000 images per style for evaluating synthesis methods.The benchmark is collected from the Internet and is used alongside baseline comparisons.
  • I. INTRODUCTION: The proposed method outperforms the compared baselines qualitatively, producing desired results where UNet performs poorly.The cited comparison includes standard-font input, UNet output, the proposed output, and the target image.

II. RELATED WORK

Prior work includes stroke-based calligraphy synthesis and broader image-to-image translation, but this task differs because standard-font and calligraphy images can have different stroke positions and styles.

  • II. RELATED WORK: Earlier calligraphy methods extract and assemble stroke shapes or character topologies from existing calligraphic examples.These approaches are described as generating new artwork or transferring font styles through parameterized stroke and topology choices.
  • II. RELATED WORK: Image-to-image translation covers tasks such as contour extraction, semantic segmentation, style transfer, and colorization.The paper distinguishes these common applications from its deformation setting.
  • II. RELATED WORK: Unlike typical pixel-to-pixel translation, calligraphy synthesis maps inputs and targets that share relative radical layout but differ in stroke position and style.This deformation characterizes the specific translation scenario studied by the paper.

3) Generative Adversarial Networks:

The proposed method uses two related encoder-decoder networks and adversarial training to synthesize realistic calligraphy. A supervise network reconstructs calligraphy images and provides detailed guidance for the transfer network.

  • Adversarial loss is used with reconstruction loss to make generated calligraphy images look more realistic.The paper motivates adversarial training as a way to learn the ground-truth image distribution and produce sharper outputs.
  • The proposed module combines a Supervise Network and a Transfer Network trained jointly end-to-end.The Transfer Network performs font-to-calligraphy translation, while the Supervise Network supplies training supervision.
  • A. Supervise Network: The Supervise Network is an autoencoder that reconstructs calligraphy images from latent representations.Its encoder produces a compact feature representation, and its decoder generates an image resembling the input.
  • A. Supervise Network: Skip connections preserve stroke details that a simple encoder-decoder otherwise loses.The authors report that the simple architecture retains rough layout but loses stroke information, whereas skip connections provide detailed information to the decoder.

B. Transfer Network

The Transfer Network converts standard-font images into calligraphy-like images while addressing differences in stroke layout and style. Residual connections and supervision from the calligraphy autoencoder support this transformation.

  • B. Transfer Network: The Transfer Network takes a standard font image as input and generates a calligraphy-like image.Its encoder-decoder structure is similar to the Supervise Network but uses different skip connections.
  • B. Transfer Network: Residual blocks connect corresponding layers to learn minute differences between standard-font structure and specified calligraphy style.The residual output is summed with the later-layer feature to increase the network’s capacity for these transformations.
  • B. Transfer Network: Because standard-font and calligraphy images differ in stroke position and style, high-level layout features alone are insufficient for clear, style-specific strokes.The paper therefore uses the Supervise Network to guide the Transfer Network during training.
  • B. Transfer Network: The Transfer Network is trained within a GAN framework using adversarial loss in addition to reconstruction objectives.Adversarial loss is intended to make the transferred calligraphy output look real.

C. End-to-End Joint Training

The two subnetworks are trained jointly from paired standard-font and calligraphy images. Training combines calligraphy reconstruction, target-image reconstruction, low-level feature reconstruction, and adversarial objectives.

  • C. End-to-End Joint Training: Each training pair contains a standard-font image x and a calligraphy image y for the same character.The two subnetworks are optimized together in an end-to-end training procedure.
  • C. End-to-End Joint Training: The Supervise Network reconstructs the calligraphy image y using an L1 reconstruction loss.The paper chooses L1 rather than L2 because it tends to produce sharper and cleaner images.
  • C. End-to-End Joint Training: The Transfer Network reconstructs the target calligraphy image and the Supervise Network’s low-level feature representation.Its reconstruction objective uses the target image y together with low-level features S from the Supervise Network.
  • C. End-to-End Joint Training: The adversarial objective evaluates real pairs (x, y) against generated pairs (x, G(x)).This adversarial term complements the reconstruction losses in the final training objective.

D. Implementation details

The implementation preprocesses images as binary 256 × 256 inputs, augments them by horizontal flipping, and trains the model end-to-end with Adam.

  • D. Implementation details: Input images are resized to 256 × 256, converted to binary images, and horizontally flipped with probability 0.5 for augmentation.The augmentation is used to enlarge the dataset and reduce overfitting.
  • D. Implementation details: Both subnetworks use eight convolutional encoder blocks and eight deconvolutional decoder layers to generate 256 × 256 binary images.The decoder consists of seven deconvolutional blocks followed by a final deconvolution layer.
  • D. Implementation details: Training uses Adam with an initial learning rate of 0.002 and batch size 16 until the output becomes stable.The transfer decoder is supervised at feature-map sizes from 16 × 16 through 128 × 128, with λs and λr set to 100.
  • D. Implementation details: Only the Transfer Network is used during testing to generate calligraphy images.

IV. EXPERIMENTS

The paper introduces CCS-4, a four-style benchmark collected from ancient Chinese calligraphers, and evaluates calligraphy-generation methods on its subsets.

  • IV. EXPERIMENTS: The experiments compare the proposed method with neural image-translation baselines on the four benchmark subsets.
  • A. Dataset: CCS-4 contains four subsets representing Mi Fu, Zhao Zhiqian, Liu Gongquan, and Shen Yinmo, with varied calligraphic styles.The styles include running, official, and regular script categories.
  • A. Dataset: Each CCS-4 subset contains about 7,000 images divided into 6,000 training images and the remainder for validation.

B. Baseline Methods

The baseline methods include Rewrite, a convolutional neural style-transfer model trained with reconstruction and total-variation losses.

  • B. Baseline Methods: Rewrite transfers one typographic font to another stylized font using a top-down convolutional network.
  • B. Baseline Methods: Rewrite stacks convolution, batch-normalization, and ReLU blocks, followed by max pooling, dropout, and sigmoid layers.
  • B. Baseline Methods: The Rewrite network is optimized with L1 loss and total variation loss.

2) Encoder-Decoder Network:

The encoder-decoder baseline uses L1 and adversarial losses, while the proposed model is evaluated against it and other baselines across four calligraphy styles.

  • 2) Encoder-Decoder Network:: The encoder-decoder baseline is trained with L1 and adversarial losses, following its use in image-translation tasks.
  • 2) Encoder-Decoder Network:: The proposed method is evaluated against baselines using Hei font inputs to reduce stylistic similarity between source and target images.This setting increases the difficulty of generation while testing robustness and effectiveness.
  • 2) Encoder-Decoder Network:: Rewrite and the encoder-decoder baseline perform worst, preserving component structure but producing unsatisfactory stroke layouts and styles.On Mi Fu and Zhao Zhiqian, nearly all generated strokes fail to match the ground truth.
  • 2) Encoder-Decoder Network:: UNet improves over the encoder-decoder baseline but struggles on Mi Fu, where standard-font and calligraphy stroke layouts differ substantially.
  • 2) Encoder-Decoder Network:: Adding adversarial loss produces sharper images with more vivid valid details and less blur.
  • 2) Encoder-Decoder Network:: The proposed method achieves the best results on all four calligraphy subsets, including realistic global styles and local stroke details.It also handles characters with complex stroke structures.

2) Effect of The Supervise Network:

The supervise network improves the perceptual quality of generated calligraphy by guiding fine stroke layout and style details.

  • 2) Effect of The Supervise Network:: The supervise network helps the model learn low-level stroke details that ordinary autoencoders handle poorly.
  • 2) Effect of The Supervise Network:: With the supervise network, generated images outperform the version without it in perceptual quality and fine-detail accuracy.The ablated model retains general character structure but contains incorrect fine details.

3) Effect of The Adversarial Loss:

GAN loss improves generated calligraphy quality by reducing blur and adding vivid details. The model also produces photo-realistic results when Kai Font replaces Hei Font as the standard input.

  • GAN loss makes generated calligraphy sharper and more detailed than using L1 loss alone.The added adversarial loss mitigates blur and restores vivid character details.
  • The model still outputs photo-realistic calligraphy when Kai Font is used as the standard font.This result is presented as evidence of robustness to the choice of standard font.
  • The proposed approach achieves the best results compared with recent Chinese calligraphy generation works.
Loading 1706.08789v1…