PaperlayerView exact revision on arXiv

Source-linked AI summary

Document Image Binarization with Fully Convolutional Neural Networks

Chris Tensmeyer, Tony Martinez

arXiv:1708.03276v1cs.CV

TL;DR

Degraded historical-document binarization is needed before tasks such as OCR and page segmentation. The paper formulates it as pixel classification with a multi-scale FCN trained using a continuous Pseudo F-measure objective. An ensemble outperforms competition winners in 4 of 7 DIBCO competitions and transfers to palm leaf manuscripts.

  • Problem

    Degraded document images contain background, stains, faded ink, bleed-through, and uneven illumination that complicate downstream document analysis.

  • Method

    The paper uses multi-scale FCNs for pixel classification and trains them with a combined continuous Pseudo F-measure and regular F-measure loss.

  • Results

    The FCN ensemble outperforms competition winners on 4 of 7 DIBCO competitions and achieves competitive performance on Palm Leaf Manuscripts.

  • Takeaways & Limitations

    The same FCN approach supports binarization across historical paper and palm leaf manuscript domains without explicit prior information about the ground-truth definition.

Abstract

from arXiv · show

Binarization of degraded historical manuscript images is an important pre-processing step for many document processing tasks. We formulate binarization as a pixel classification learning task and apply a novel Fully Convolutional Network (FCN) architecture that operates at multiple image scales, including full resolution. The FCN is trained to optimize a continuous version of the Pseudo F-measure metric and an ensemble of FCNs outperform the competition winners on 4 of 7 DIBCO competitions. This same binarization technique can also be applied to different domains such as Palm Leaf Manuscripts with good performance. We analyze the performance of the proposed model w.r.t. the architectural hyperparameters, size and diversity of training data, and the input features chosen.

I. INTRODUCTION

The paper frames degraded-document binarization as pixel classification for reducing noise in downstream analysis, and proposes multi-scale FCNs trained with task-aligned losses. The approach targets precise localization, cross-domain generalization, and stronger performance than prior methods.

  • Motivation: Binarization labels each pixel as foreground ink or background to reduce degradation effects in tasks such as OCR and page segmentation.Relevant degradations include stains, faded ink, bleed-through, uneven illumination, and non-uniform backgrounds.
  • Approach: The proposed multi-scale FCN preserves downsampling-based generalization while improving localization for pixel-level binarization.The architecture combines information across image scales, including full-resolution processing.
  • Generalization: FCNs can achieve state-of-the-art performance across historical paper documents and palm leaf manuscripts without hyperparameter tuning.The paper attributes this cross-domain applicability to training the same architecture on different document-image domains.
  • Comparison with prior methods: Unlike thresholding, edge-based, and shape-biased methods, FCNs learn spatial pixel arrangements from training data without handcrafted local-shape assumptions.Threshold methods ignore shape because they depend on image statistics, while other approaches impose foreground-component biases.
  • Training objective: The model directly optimizes a continuous Pseudo F-measure and combines it with regular F-measure to address foreground-border errors.The combined objective avoids relying on heuristic pixel sampling for class balance while preserving border-pixel classification.
  • Contributions: The study reports that FCNs exceed previous DIBCO state of the art, benefit more from diverse than merely larger data, and improve with additional input features.The additional features include outputs from other binarization algorithms.

II. RELATED WORK

Prior binarization work includes optimization-based, statistical, neural, and FCN approaches. The paper positions its architecture against methods with handcrafted feature or shape assumptions and against heavily downsampled FCNs with poor localization.

  • Optimization-based methods: Howe’s MRF method uses Laplacian-based unary energies, Canny-derived pairwise connections, and Max Flow optimization.Variants achieved first place in HDIBCO 2012, 2014, and 2016 and second place in DIBCO 2013.
  • Classification-based methods: Earlier classification approaches used MLPs, neighborhood features, LSTMs, CNNs, or randomized trees to classify document pixels.These methods incorporated combinations of local neighborhoods, global statistics, sequence context, and handcrafted features.
  • Prior FCNs: Prior FCNs for semantic segmentation heavily downsample inputs, producing poor localization for document-image binarization.The paper therefore treats those FCNs as unsuitable models for this task without architectural modification.
  • Position of this work: The paper describes its method as an FCN, multi-scale extension, and proposed loss function for document-image binarization.These components address the representation, localization, and training objectives of the task.

A. Fully Convolutional Networks

The FCN maps an image to per-pixel foreground probabilities through alternating convolution and nonlinear operations. Its layers use learned kernels and biases, while the final output applies a sigmoid; the proposed implementation uses repeated architectural hyperparameters.

  • FCN mapping: An FCN maps an input x ∈ R^D×H×W to an output y ∈ R^H×W containing per-pixel foreground probabilities.The number of input channels D can represent features such as RGB channels.
  • Layer operation: Each FCN layer applies convolution followed by element-wise ReLU, with learned kernels and a bias term.The layer output retains spatial dimensions while changing the channel representation according to the learned kernels.
  • Output: The final one-channel output uses an element-wise sigmoid without ReLU to produce pixel probabilities.This converts the final activations into values in the interval [0, 1].
  • Kernel structure: Each convolution kernel spans all input channels and produces one output channel, with the collection of kernels forming the layer output.The kernels operate spatially while covering the full depth of the preceding layer.
  • Hyperparameters: The implementation primarily fixes layer width at Dℓ = 64 and kernel size at Kℓ = 9 across layers.Using shared values across layers simplifies hyperparameter selection.

B. Multi-Scale

Multi-scale FCNs combine local and global image features while preserving the precise localization required for document binarization. The model also uses a differentiable Pseudo F-measure objective with explicit treatment of foreground border pixels.

  • Multi-Scale: Multi-scale feature fusion improves dense pixel prediction by combining local and global information.Document binarization requires precise localization, which is difficult after downsampling.
  • Multi-Scale: The branching FCN computes features at full, half, quarter, and eighth image scales before upsampling and concatenating them.Two final convolution layers classify pixels from the fused scale outputs.
  • Pseudo F-measure Loss: Pseudo F-measure weights binarization errors according to their potential to obscure individual characters.Pseudo Recall and Pseudo Precision use per-pixel recall and precision weights, respectively.
  • Pseudo F-measure Loss: The continuous Pseudo F-measure can use DIBCO weights, while uniform weights yield the conventional F-measure.The differentiable formulation is intended for stochastic-gradient training.
  • Pseudo F-measure Loss: The derivative of the harmonic mean combines the derivatives of Pseudo Recall and Pseudo Precision with respect to predicted foreground probabilities.The recall derivative is obtained from its numerator, while the precision derivative uses the quotient rule.

D. Datasets and Metrics

The experiments evaluate the method on DIBCO competition images and Palm Leaf Manuscripts using fixed dataset splits and four DIBCO evaluation metrics. Test data is reserved for final evaluation rather than model selection.

  • Datasets: The evaluation uses 86 DIBCO competition images from 2009–2016 and 50 Palm Leaf Manuscript images.DIBCO splits vary by held-out competition year; Palm Leaf Manuscripts use 40 training, 10 validation, and 50 designated test images.
  • Datasets: For each DIBCO year, that year’s images form the test set, 10 other images form validation, and the remainder form training.The Palm Leaf Manuscript split uses randomly selected training and validation images.
  • Evaluation Protocol: Model development uses the H-DIBCO 2016 validation set, while test data is excluded from selecting models, architectures, and features.This procedure is intended to avoid overfitting the testing data.
  • Metrics: The reported metrics are P-FM, FM, PSNR, and DRD, with higher values better for the first three and lower values better for DRD.These metrics follow the DIBCO 2013 evaluation criteria.

E. Implementation Details

The implementation trains FCNs on overlapping image crops using grayscale and Relative Darkness features, then stitches central output patches into whole-image binarizations. Qualitative comparisons target bleed-through noise, while ensemble results are summarized across datasets and losses.

  • Input and Training Data: Training uses 256x256 crops sampled every 64 pixels, discarding crops containing only background pixels.The FCN input combines the grayscale image with locally computed Relative Darkness features.
  • Whole-Image Inference: Whole-image binarization stitches the center 128x128 output patch from each overlapping crop.Using central patches provides each classified pixel with sufficient surrounding context.
  • Optimization: SGD training starts with learning rate 0.001, mini-batches of 10 patches, and L2 weight decay of 0.0005.Gradient clipping and validation-based learning-rate decay are also used.
  • Qualitative Evaluation: Figure 2 compares the proposed FCN ensemble with Howe Binarization on bleed-through-noisy H-DIBCO 2016 test images.The comparison is qualitative and focuses on degraded historical document regions.
  • Quantitative Evaluation: Table I reports average performance for five FCNs on H-DIBCO 2016 and Palm Leaf Manuscripts across loss functions, with ensemble values in parentheses.The table distinguishes individual-FCN averages from combined ensemble performance.

IV. EXPERIMENTS

This section presents results and discussion for experiments validating the proposed model.

  • Experiments: The experiments validate the proposed model through reported results and discussion.

A. Loss Functions

The study compares loss functions and finds that combining P-FM with FM supports strong binarization performance, while ensembles improve results at increased computational cost.

  • Loss-function comparison: P-FM and FM losses were compared with cross entropy on HDIBCO 2016 and Palm Leaf Manuscripts using five independently initialized FCNs per loss.Ensemble outputs were combined by per-pixel majority vote.
  • Ensembling: Ensemble prediction significantly improves performance, but increases computation.The ensemble combines individual FCN outputs through per-pixel majority voting.
  • Loss-function comparison: Optimizing P-FM or P-FM + FM produced the best P-FM, whereas cross entropy performed worst for P-FM.Training with only P-FM reduced other metrics because border pixels were predicted as background.
  • Loss-function comparison: P-FM + FM was selected for later experiments because FM generally outperformed cross entropy on validation data.Cross entropy nevertheless performed better on the HDIBCO 2016 test data.
  • DIBCO performance: The five-FCN ensemble outperformed the best submitted entries in 4 competitions, including HDIBCO 2016.On HDIBCO 2014, performance was poor on one image with P-FM 56.4 but averaged 99.04 across the other nine images.

C. Architecture Search

The architecture search varies depth, width, scale count, and kernel size, finding broadly stable performance except for small kernels and larger gains from ensembling.

  • Architecture search: The search independently varied network depth, width, number of scales, and kernel size on HDIBCO 2016.The base architecture used depth L = 9, width Dℓ = 64, 4 scales, and kernel size Kℓ = 9.
  • Training-data setup: The learning-curve figure compares removing whole training images with retaining all images while cropping them smaller.The two strategies change training-set size differently: one reduces image diversity, while the other retains it.
  • Architecture search: Performance was relatively insensitive to architectural hyperparameters, except that kernel size 3 performed worse than larger kernels.The comparison evaluated ensembles of 5 FCNs for each architecture.
  • Architecture search: Ensemble prediction usually produced larger improvements than the performance differences among architectures.This suggests that ensemble choice mattered more than fine-grained architectural variation in these experiments.

D. How Much Data is Enough?

Training-data diversity matters more than the number of pixels alone: cropping images while retaining variety outperforms removing whole images, although validation bias can distort comparisons.

  • Data diversity: Removing whole images decreases the diversity of training inputs, whereas cropping all images retains input diversity while reducing available pixels.The learning curve keeps validation data fixed while varying the training data.
  • Data diversity: Cropping training images performed better than removing whole images at similar pixel counts, indicating that training-data diversity improves performance.The authors recommend small, diverse images when creating additional ground-truth binarizations.
  • Dataset bias: Training on 10M pixels produced better test performance than training on 20–40M pixels using whole images.The authors attribute this to overfitting the validation set used for model selection.
  • Dataset bias: Validation P-FM was 92.84 for 10M pixels and 95.26 for 20M pixels, despite the lower-pixel model performing better on the test set.The added training images were more similar to validation than test images, indicating dataset bias with few evaluation images.

E. Input Features

The study evaluates densely computed input features for FCNs, finding that Relative Darkness features perform best and Howe’s binarization output performs nearly as well.

  • Relative Darkness features performed best among the evaluated additional input feature types.The experiments used an ensemble of 5 FCNs evaluated on the HDIBCO 2016 validation set using P-FM.
  • Howe’s binarization output performed almost as well as Relative Darkness features.
  • Relative Darkness features count darker, lighter, and similar pixels within a local window around each central pixel.The experiments used a 5x5 window and a similarity threshold of ±10.
  • Additional input features can improve FCN binarization performance, including outputs from other binarization algorithms.
Loading 1708.03276v1…