Source-linked AI summary
Deep Learning for Classical Japanese Literature
Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, David Ha
TL;DR
Classical Japanese literature contains extensive cultural knowledge, but many historical texts remain inaccessible because most Japanese people cannot read Kuzushiji. The paper releases three datasets spanning cursive Japanese characters and applies them to classification and domain transfer, aiming to engage machine learning with Japanese literature. The datasets preserve real class imbalance and support benchmark and creative modeling research, while their train and test sets share source books.
Problem
Millions of historical Japanese books and documents are difficult for the general public to access because Kuzushiji is no longer widely taught or readable.
Method
The paper releases Kuzushiji-MNIST, Kuzushiji-49, and Kuzushiji-Kanji, and applies classification baselines and generative domain transfer from Kuzushiji Kanji to Modern Kanji.
Results
The datasets provide 10, 49, and 3,832 character classes across benchmark and creative tasks, with Kuzushiji-MNIST designed as an MNIST-compatible replacement.
Takeaways & Limitations
The Kuzushiji datasets offer machine-learning benchmarks and creative modeling material connected to classical Japanese literature and cultural heritage.
Takeaways & Limitations
Train and test characters in all three datasets are sampled from the same 35 books, so their class distributions are consistent across splits.
Abstract
from arXiv · showhide
Much of machine learning research focuses on producing models which perform well on benchmark tasks, in turn improving our understanding of the challenges associated with those tasks. From the perspective of ML researchers, the content of the task itself is largely irrelevant, and thus there have increasingly been calls for benchmark tasks to more heavily focus on problems which are of social or cultural relevance. In this work, we introduce Kuzushiji-MNIST, a dataset which focuses on Kuzushiji (cursive Japanese), as well as two larger, more challenging datasets, Kuzushiji-49 and Kuzushiji-Kanji. Through these datasets, we wish to engage the machine learning community into the world of classical Japanese literature. Dataset available at https://github.com/rois-codh/kmnist
1 Introduction
Historical Japanese texts preserve cultural knowledge but are difficult for most people to read because modernization displaced cursive Kuzushiji from formal education. The paper frames its datasets and experiments as a way to connect machine learning with this inaccessible literary heritage.
- Motivation: Most Japanese today cannot read books over 150 years old because Kuzushiji is no longer taught in the official school curriculum.The script was used for over 1,000 years before language and writing systems changed during modernization after 1868.
- Historical context: Figure 2 contrasts an Edo-period book printed in 1772 with a textbook printed in 1900 after Japanese standardization.The paired examples illustrate changes in Japanese text across the modernization period.
- Motivation: More than 1.7 million pre-1867 books and many additional historical documents remain difficult for the general public to access.Digitization preserves copies, but limited Kuzushiji education restricts who can read and work with them.
- Paper approach: Figure 3 depicts generating Modern Kanji from Kuzushiji Kanji for unseen characters through a domain-transfer experiment.This experiment extends the paper beyond dataset release toward machine-learning methods for classical Japanese materials.
- Paper approach: Kuzushiji-MNIST contains 10 classes, while the accompanying datasets address larger character inventories and creative transfer tasks.Figure 1 presents the modern Hiragana counterparts alongside Kuzushiji classes and a historical literature scroll.
2 Kuzushiji Dataset
The paper releases three preprocessed Kuzushiji datasets from classical Japanese books, spanning balanced small-scale classification, imbalanced Hiragana, and highly imbalanced Kanji recognition and creative tasks.
- The datasets are preprocessed from characters scanned from 35 classical books printed in the 18th century.
- Kuzushiji-49 contains 48 Hiragana characters and one iteration mark, while Kuzushiji-Kanji contains 3,832 Kanji classes including rare characters.
- Kuzushiji-MNIST is a 10-class drop-in replacement for MNIST, with 6,000 training and 1,000 test images per class.
- Hentaigana creates many-to-one mappings because a Hiragana class can contain multiple writing forms derived from different Kanji.
- Kuzushiji-Kanji preserves natural class imbalance, with class sample counts ranging from over a thousand to one.
- The datasets target both classification benchmarks and creative experimental tasks beyond character recognition.
3 Experiments
The experiments establish classification baselines and explore domain transfer from Kuzushiji-Kanji pixel images to Modern Kanji in pixel and stroke-based formats.
- Classification: The paper presents baseline classification results on Kuzushiji-MNIST and Kuzushiji-49 using four model families.
- Domain transfer: The domain-transfer model generates Modern Kanji versions of Kuzushiji-Kanji inputs in both pixel and stroke-based formats.
- Domain transfer: Two separate VAEs encode Kuzushiji-Kanji and pixel-rendered Modern Kanji into respective 64-dimensional latent spaces.
- Domain transfer: A Mixture Density Network models P(znew|zold) as a mixture of Gaussians and samples Modern Kanji latent vectors from Kuzushiji-Kanji representations.
- Domain transfer: A Sketch-RNN decoder generates stroke-based Modern Kanji conditioned on the target-domain latent representation.
- Domain transfer: The approach covers around 3,600 Kanji characters overlapping between the two datasets, while non-overlapping characters use KanjiVG encodings directly.
4 Future Directions
The authors propose expanding Kuzushiji datasets and community activity to support broader machine learning research and preserve Japanese cultural knowledge and heritage.
- The datasets are intended to support classification as well as generative modelling, adversarial examples, few-shot learning, transfer learning, and domain adaptation.
- The authors plan machine learning competitions to encourage development across these research areas.
- The full Kuzushiji dataset is planned to expand to over a million character images.
- The authors connect broader cross-field collaboration with preserving Japanese historical cultural knowledge and heritage.