PaperlayerView exact revision on arXiv

Source-linked AI summary

EMNIST: an extension of MNIST to handwritten letters

Gregory Cohen, Saeed Afshar, Jonathan Tapson, André van Schaik

arXiv:1702.05373v2cs.CV

TL;DR

MNIST’s widespread use and near-saturated performance motivate a more challenging but equally accessible benchmark. The paper constructs EMNIST by converting NIST Special Database 19 into MNIST-compatible datasets and evaluates them with benchmark classifiers. The resulting suite expands the number and variety of classification tasks while preserving MNIST’s structure and compatibility.

  • Problem

    MNIST’s widespread use is supported by accessibility, but its high reported accuracies have reduced its challenge as a benchmark.

  • Method

    The paper converts NIST Special Database 19 characters into MNIST-compatible datasets, documents the process, and evaluates them with benchmark classifiers.

  • Results

    EMNIST provides six datasets with more samples, output classes, and varied and challenging classification tasks than MNIST while maintaining its structure and nature.

  • Takeaways & Limitations

    EMNIST offers a modern performance benchmark that remains directly compatible with networks and systems designed for MNIST.

Abstract

from arXiv · show

The MNIST dataset has become a standard benchmark for learning, classification and computer vision systems. Contributing to its widespread adoption are the understandable and intuitive nature of the task, its relatively small size and storage requirements and the accessibility and ease-of-use of the database itself. The MNIST database was derived from a larger dataset known as the NIST Special Database 19 which contains digits, uppercase and lowercase handwritten letters. This paper introduces a variant of the full NIST dataset, which we have called Extended MNIST (EMNIST), which follows the same conversion paradigm used to create the MNIST dataset. The result is a set of datasets that constitute a more challenging classification tasks involving letters and digits, and that shares the same image structure and parameters as the original MNIST task, allowing for direct compatibility with all existing classifiers and systems. Benchmark results are presented along with a validation of the conversion process through the comparison of the classification results on converted NIST digits and the MNIST digits.

I. INTRODUCTION

MNIST became widely used because it is accessible and easy to benchmark, but its near-saturated accuracy reduced its challenge. EMNIST addresses this by providing standardized, more challenging datasets derived from NIST and compatible with MNIST systems.

  • Motivation: Standardized benchmarks provide quick, quantitative, and fair comparisons of learning approaches.They are especially useful when tasks are intuitive and conceptually simple.
  • Motivation: Multiple groups reported accuracies above 99.7% on MNIST, weakening its role as a challenging benchmark.The paper notes that labeling can become questionable at this accuracy level.
  • Motivation: MNIST remains widely used because it is small, free, straightforwardly encoded, and accessible across platforms and programming languages.Its accessibility has contributed substantially to widespread adoption.
  • Motivation: NIST Special Database 19 contains handwritten numerals and letters and supports broader tasks than MNIST.Potential tasks include writer identification, transcription, and case detection.
  • Motivation: NIST remained difficult to access and use, while its structure and images differed from MNIST.These differences limited direct compatibility with existing MNIST systems.
  • Contribution: EMNIST introduces more challenging classification datasets derived from NIST and designed as drop-in replacements for MNIST systems.The datasets match MNIST image specifications, organization, and file formats.
  • Contribution: The paper documents the conversion process and uses benchmark results to characterize and validate the datasets.The benchmark results support evaluation of the resulting dataset suite.

A. The MNIST and NIST Dataset

NIST Special Database 19 is a large, multi-organization handwriting collection containing digits and uppercase and lowercase letters. Its hierarchies define different classification organizations, with By Class and By Merge most directly aligned with MNIST-style tasks.

  • Dataset contents: NIST Special Database 19 contains binary handwriting scans and individually segmented, labeled digits and letters collected from over 500 writers.It was published in 1995 and re-released in a modern file format in 2016.
  • Data hierarchies: The database provides five hierarchies for optical character recognition tasks, including page, author, field, class, and merge organizations.These organizations vary in whether data are grouped by forms, writers, fields, or character classes.
  • Data hierarchies: The By Class hierarchy contains 62 classes covering digits, lowercase letters, and uppercase letters, with suggested training and testing splits.It is described as the most useful organization from a classification perspective.
  • Data hierarchies: The By Merge hierarchy combines selected uppercase and lowercase letter classes into a 47-class task.The conversion process applies to By Class and By Merge, which are the primary focus because they are MNIST-compatible.
  • Dataset contents: The original training and testing splits contain 731,668 training samples and 82,587 testing samples in each By Class and By Merge hierarchy.Together, each hierarchy contains 814,255 handwritten characters, nearly half of which are digits.
  • Scope: Writer identification using the By Author organization was identified as an opportunity but left beyond the scope of this work.This defines a boundary on the paper’s classification focus.

II. METHODOLOGY

The paper converts NIST handwriting images into MNIST-compatible datasets using a documented, reproducible preprocessing pipeline. Deterministic OPIUM-based classifiers provide benchmark results for validating and characterizing the resulting tasks.

  • Method: The paper introduces OPIUM-based classifiers to validate and characterize EMNIST datasets through benchmark classification results.The classifiers also help explain dataset nature and organization through their results.
  • Conversion: The conversion process reproduces the overall paradigm used to create MNIST while adapting down-sampling for NIST character variation.The method is designed to maximize reproducibility and accessibility.
  • Classifiers: OPIUM-based classifiers provide deterministic, single-step analytical solutions based on a pseudo-inverse network formulation.This avoids multiple iterative passes over the dataset.
  • Accessibility: The original NIST release used a proprietary compressed format that remains difficult to compile and run on modern systems.The paper therefore includes extraction and conversion code, while its post-processing also supports the PNG-based second edition.
  • Conversion: The conversion transforms 128 × 128 binary images into 28 × 28 images with 8-bit grayscale resolution matching MNIST characteristics.Only the By Class and By Merge hierarchies are used, with identical conversion apart from class labels.
  • Preprocessing: Each character is individually loaded, Gaussian-blurred, and bounded before being prepared for down-sampling.The bounding box handles variation in character size and shape across classes and writers.
  • Preprocessing: Characters are centered in a square frame, padded with a 2-pixel border, and down-sampled to 28 × 28 using bicubic interpolation.The resulting intensities are scaled to 8-bit grayscale.

B. Training and Testing Splits

The paper questions whether the recommended student-derived testing split is sufficiently comparable with the training participants. EMNIST therefore follows MNIST by recombining the original splits and drawing a new random split.

  • Split design: The NIST collection combines handwriting from Census employees and high-school students, with students suggested as a harder testing corpus.The paper notes that this recommendation raises questions about similarity and consistency between participant groups.
  • Split design: EMNIST combines the original training and testing sets before drawing new random training and testing partitions.This follows the methodology used in the original MNIST paper.

C. The EMNIST Datasets

EMNIST derives multiple organized datasets from NIST characters, converting images to MNIST-compatible 28 × 28 grayscale representations and adding balanced subsets for fairer evaluation.

  • Dataset structure: The By Class and By Merge datasets use the same NIST images but different class labels and contain uneven numbers of samples per class.By Class retains 62 classes, while By Merge uses 47 classes by merging selected uppercase and lowercase characters.
  • Dataset structure: Because the full datasets contain far more digit than letter samples, four additional subsets address class imbalance and task-specific needs.Letter-class sample counts approximately reflect English-language frequencies.
  • Image conversion: The conversion pipeline smooths, crops, centers, pads, downsamples, and rescales 128 × 128 binary NIST images into 28 × 28 grayscale images.A Gaussian filter with σ = 1 is applied, followed by a 2 pixel border, bi-cubic interpolation, and intensity scaling to [0, 255].
  • Balanced subsets: EMNIST Balanced selects an equal-sized subset of all By Merge classes to create a fair, balanced, sufficiently challenging 47-class task.The By Merge hierarchy avoids errors arising purely from confusing uppercase and lowercase letters.
  • Balanced subsets: EMNIST Letters merges uppercase and lowercase classes into a balanced 26-class task, while EMNIST Digits contains 28,000 samples per digit.EMNIST MNIST is designed to match the original MNIST size and specifications as a drop-in replacement.
  • Validation partitions: Balanced datasets include a specially balanced validation subset, while the remaining training data and testing data retain separate roles.The validation subset is equal in size to the testing set and contains equal numbers of samples for each task.

E. Classifiers

The benchmark uses OPIUM-based classifiers and a linear pseudoinverse baseline to characterize EMNIST datasets across tasks with different class structures and sample counts.

  • Classifier design: A simple three-layer ELM network and a linear classifier provide benchmark classification results for the EMNIST datasets.The linear classifier is the analytical pseudoinverse solution for a network without a hidden layer.
  • Classifier design: OPIUM handles large datasets by iteratively computing the exact pseudoinverse solution through Online Pseudo-Inverse Update rather than one batch calculation.The full pseudoinverse cannot be calculated in a single step because of dataset and network size.
  • Benchmark tasks: The benchmark includes a 47-class task with 131,600 samples and a 62-class task with 814,255 samples.The class compositions distinguish digit classes from letter classes in the dataset organization.

26 Classes 145,600 Samples

The EMNIST suite visualizes class composition and train/test splits across varied tasks, while OPIUM supplies deterministic analytical baselines for dataset characterization.

  • Dataset breakdown: The dataset breakdown shows class membership, samples per class, and training/testing splits for six EMNIST datasets.Merged datasets use arrows to identify the class receiving a lowercase letter.
  • Classifier methodology: OPIUM iteratively calculates an exact pseudoinverse solution for output weights and supports datasets of any size.The experiments vary hidden-layer size while keeping training order fixed and report mean accuracy and standard deviation across trials when appropriate.
  • Classifier methodology: The benchmark methods are intended as deterministic, instructive baselines for exploring and validating the datasets rather than cutting-edge techniques.Given the same network structure and training order, the outcomes are deterministic.

III. RESULTS

Results on EMNIST Balanced show increasing OPIUM accuracy with hidden-layer size, while matched MNIST experiments establish that EMNIST is the more challenging task for these networks.

  • A. EMNIST Balanced Dataset: 78.02% ± 0.09% was the maximum mean accuracy over twenty trials using 10,000 hidden-layer neurons on EMNIST Balanced.Larger hidden layers were not explored because of memory and processing constraints.
  • A. EMNIST Balanced Dataset: Accuracy increased with each tested hidden-layer size on the EMNIST Balanced classification task.The experiments trained OPIUM-based classifiers on the entire training set and evaluated final accuracy on unseen testing data.
  • A. EMNIST Balanced Dataset: The matched MNIST comparison used identical network structures, hidden-layer sizes, and random weights for each trial.Twenty trials were conducted and mean accuracy with standard deviation was plotted for comparison.
  • A. EMNIST Balanced Dataset: The comparison indicates that EMNIST provides a more challenging classification task while retaining MNIST's structure, simplicity, and accessibility.The same networks produced results consistent with previously published results on MNIST.

B. EMNIST By Merge and By Class Dataset Results

The EMNIST By Merge dataset outperformed By Class across hidden-layer sizes, while EMNIST Letters reduced case-related variability by combining letter cases and excluding digits. Classification remained limited by ambiguous character pairs and uneven class distributions.

  • Dataset organization: By Class contains 62 classes, whereas By Merge merges selected letter classes into 47 classes while retaining all 814,255 training characters.The datasets differ in class count, samples per class, and character ordering.
  • By Merge and By Class: By Merge outperformed By Class at every hidden-layer size, reaching 80.87% ± 0.05% versus 77.57% ± 0.08% with 10,000 hidden neurons.Linear classifiers reached 50.51% on By Merge and 51.80% on By Class.
  • By Merge and By Class: By Merge improves case-related errors, but classifiers still commonly confuse lowercase L with digit 1 and other visually similar classes.The reported performance difference is primarily attributed to uppercase–lowercase misclassification in By Class.
  • EMNIST Letters: EMNIST Letters combines uppercase and lowercase versions into one class and removes digits, creating a 26-class letter-only task.This requires classifiers to associate two distinct letter representations with one class label.
  • EMNIST Letters: EMNIST Letters reached 85.15% ± 0.12% with 10,000 hidden neurons, compared with 80.87% ± 0.05% on By Merge, though the comparison is affected by uneven By Merge test-class sizes.The Letters linear classifier achieved 55.78%, a moderate increase over By Merge.
  • EMNIST Letters: The Letters confusion matrix shows strongest ambiguity between I and L and between G and Q; class accuracy ranged from 65.5% for G to 95.8% for O.These confusions also occur in the full By Merge dataset and become more complicated when digits are included.

D. EMNIST Digit Dataset Results

The EMNIST Digits and EMNIST MNIST datasets adapt handwritten digit classification for compatibility with MNIST while supporting conversion validation. Identical-network comparisons show both EMNIST datasets outperform original MNIST, with the conversion producing a more separable problem.

  • Dataset design: EMNIST Digits contains all digit classes with equal samples, while EMNIST MNIST preserves MNIST’s input size, labels, and train/test structure.EMNIST MNIST also matches the original dataset’s sample count and balances each class for conversion validation.
  • Evaluation: Ten identical OPIUM-based networks of varying hidden-layer sizes were evaluated on EMNIST Digits, EMNIST MNIST, and original MNIST.The original MNIST results provide a reference for validating and characterizing the conversion process.
  • Results: Both EMNIST Digits and EMNIST MNIST outperformed original MNIST at every tested network size.The comparison used the same networks across all three datasets.
  • Results: The conversion produced a more separable digit-classification problem, likely because characters fill more of the 28 × 28 frame and contain fewer blank pixels.The denser framing allows more information to be captured during classification.
  • Caveat: The EMNIST MNIST digit set differs somewhat from original MNIST because balancing requires a different sample selection.The paper notes that this difference may have affected accuracy somewhat but does not explain the significant, consistent EMNIST Digits improvement.

IV. DISCUSSION

Classification accuracy increased with hidden-layer size, reaching the best reported performance at 10,000 hidden neurons, while larger networks were not tested because of resource constraints. Comparisons with prior NIST-based work support the reported results, although direct comparability is limited.

  • EMNIST results: Classification accuracy increased with hidden-layer size, with best performance at 10,000 hidden-layer neurons.Memory and processing constraints prevented exploration of larger networks.
  • EMNIST results: Larger networks could not be explored because of memory and processing constraints.
  • Interpretation: More sophisticated multilayer methods are expected to achieve better performance than the simple classifiers used here.The classifiers were selected for simplicity rather than cutting-edge performance.
  • Prior work: Prior NIST-based studies used different conversions, training/testing splits, or task definitions, limiting direct comparison with these results.Reported prior results include 88.12% ± 0.09% on By Class, 91.79% ± 0.11% on By Merge, 92.42% ± 0.09% on letters, and 99.19% ± 0.02% on digits, though the latter are not directly comparable.
  • Prior work: Despite those differences, prior NIST Special Database 19 results are in line with this paper’s findings and indicate potential performance on EMNIST.The authors use these comparisons to further validate the datasets.

V. CONCLUSIONS

The paper introduces six EMNIST datasets by converting NIST Special Database 19 characters into MNIST-compatible formats. The resulting benchmark is more extensive and challenging than MNIST while preserving its structure and nature.

  • Contribution: EMNIST is a suite of six datasets designed as a more challenging alternative to MNIST.
  • Contribution: NIST Special Database 19 characters were converted to MNIST’s format, enabling compatibility with networks that work with the original MNIST dataset.
  • Validation: Digit-subset comparisons with original MNIST were used to validate the conversion process.
  • Contribution: EMNIST provides more samples, output classes, task variety, and task difficulty than MNIST while maintaining its structure and nature.
  • Significance: The datasets are positioned as a modern performance benchmark for current classification and learning systems.
Loading 1702.05373v2…