PaperlayerView exact revision on arXiv

Source-linked AI summary

dhSegment: A generic deep-learning approach for document segmentation

Sofia Ares Oliveira, Benoit Seguin, Frederic Kaplan

arXiv:1804.10371v2cs.CV

TL;DR

Historical document analysis spans diverse tasks that have often required separate, hand-tuned methods for different problems and document typologies. The paper proposes an open-source CNN-based pixel-wise predictor with task-dependent post-processing and evaluates it across several tasks. Using one architecture and mostly simple reusable operations, dhSegment achieves competitive or better results than state-of-the-art methods, with fast training and limited data requirements in some cases.

  • Problem

    Historical document processing uses heterogeneous task-specific methods, while the variability of historical series motivates generic approaches for multiple segmentation tasks.

  • Method

    dhSegment combines a generic CNN-based pixel-wise predictor with task-dependent post-processing and standardized training across segmentation tasks.

  • Results

    Across five evaluated tasks, the same network and nearly identical training configurations produced results competitive with or better than state-of-the-art methods.

  • Takeaways & Limitations

    The approach supports simple off-the-shelf segmentation building blocks that can be trained by non-specialists from target-mask examples and integrated into document-analysis pipelines.

Abstract

from arXiv · show

In recent years there have been multiple successful attempts tackling document processing problems separately by designing task specific hand-tuned strategies. We argue that the diversity of historical document processing tasks prohibits to solve them one at a time and shows a need for designing generic approaches in order to handle the variability of historical series. In this paper, we address multiple tasks simultaneously such as page extraction, baseline extraction, layout analysis or multiple typologies of illustrations and photograph extraction. We propose an open-source implementation of a CNN-based pixel-wise predictor coupled with task dependent post-processing blocks. We show that a single CNN-architecture can be used across tasks with competitive results. Moreover most of the task-specific post-precessing steps can be decomposed in a small number of simple and standard reusable operations, adding to the flexibility of our approach.

I. INTRODUCTION

Historical document processing involves diverse segmentation tasks, but existing methods are often heterogeneous and tailored to particular problems and document typologies. dhSegment addresses this variability with a generic architecture intended to support multiple tasks using recent deep-learning advances.

  • Historical document processing requires page, illustration, symbol, and text-location extraction across varied documents.
  • Existing document-analysis methods are heterogeneous and tailored to specific problems and document typologies.
  • Recent progress in deep-learning architectures suggests generic approaches may now be mature enough to outperform dedicated systems.
  • dhSegment introduces a general, flexible architecture for pixel-wise segmentation tasks on historical documents.The authors report competitive or better results across tasks and provide an open-source implementation.
  • Convolutional neural networks have become widely used across tasks and domains through architectures such as AlexNet, VGG, and ResNet.
  • Neural-network approaches, including U-shaped and MDLSTM architectures, have achieved success in historical document-processing competitions.

III. APPROACH

The system combines a fully convolutional neural network with task-dependent post-processing. The network predicts per-pixel attribute probabilities from document images, which are transformed into task-specific outputs using simple standard operations.

  • The system uses two successive steps: a fully convolutional neural network followed by task-dependent post-processing.
  • The neural network takes a document image as input and outputs a probability map for predicted attributes at each pixel.Training uses masks generated from the training labels.
  • Figure 1 depicts dhSegment as a generic neural network whose probability maps are post-processed for each task.
  • Task-dependent post-processing transforms prediction maps into desired outputs using only simple standard image-processing techniques.The diversity of required outputs motivates the task dependence.
  • The network implementation uses TensorFlow.

B. Network architecture

dhSegment uses a ResNet-50 contracting path and an expanding path that restores feature maps to full input resolution. Skip-style concatenation connects corresponding encoder and decoder features before pixel classification.

  • dhSegment combines a ResNet-50 contracting path with an expanding path that maps low-resolution features to full input resolution.Both paths contain five steps, with each step halving the previous feature-map size in the contracting direction.
  • Pretrained weights in the contracting path provide high-level ImageNet features that add robustness and help generalization.
  • Each expanding block upscales features, concatenates them with corresponding contracting features, and applies a 3x3 convolution followed by ReLU.Features in steps i = 4 and i = 5 are reduced to 512 channels by a 1x1 convolution before concatenation.
  • Upsampling is performed with bilinear interpolation.
  • The architecture contains 32.8M parameters, of which 9.36M require full training.
  • The approach limits post-processing to simple standard operations applied to network predictions.

Thresholding:

Post-processing begins by thresholding prediction maps into binary images, then applies morphological operations and connected-components analysis to refine detected regions. Vectorization converts the resulting regions into coordinates or reduced paths.

  • Thresholding:: Thresholding converts network predictions into binary maps, either class-wise for multiple classes or using a fixed or Otsu-derived threshold.The fixed threshold t lies in [0, 1].
  • Thresholding:: Morphological post-processing is limited to opening and closing operations on binary images.These operations combine erosion and dilation to analyze and process geometrical structures.
  • Thresholding:: Connected-components analysis filters out small components remaining after thresholding or subsequent processing.
  • Thresholding:: Vectorization transforms detected regions into polygonal coordinate sets, commonly bounding boxes or quadrilaterals.For line detections, vectorization instead consists of path reduction.

D. Training

Training uses a largely standardized setup, with resizing selected according to the task’s receptive-field needs and pretrained weights supporting efficiency and robustness.

  • D. Training: Training applies L2 weight decay, exponential learning-rate decay, Xavier initialization, Adam, and batch renormalization.The weight decay is 10^-6, the decay rate is 0.95, and the initial learning rate is in [10^-5, 10^-4].
  • D. Training: The network uses 300 × 300 patches after resizing images to between 6 · 10^5 and 10^6 total pixels.Cropping allows batch training and helps fit the images in memory.
  • D. Training: Training setup is mostly shared across experiments, with input resizing chosen so the receptive field matches the task.This makes resizing size the main experiment-specific training parameter.
  • D. Training: Pretrained contracting-path weights significantly reduce training time and appear to make the model less sensitive to outliers.All training and inference runs use an Nvidia Titan X Pascal GPU.

IV. EXPERIMENTS

The experiments assess dhSegment across multiple historical-document processing tasks, including page extraction, with results compared against existing methods.

  • IV. EXPERIMENTS: dhSegment is evaluated on five document-processing tasks, including page extraction, baseline detection, layout analysis, ornament detection, and photograph extraction.Three tasks are compared with state-of-the-art methods, while two private datasets demonstrate additional applications.
  • IV. EXPERIMENTS: Page extraction addresses border regions in digitized images that can alter document-processing outputs.The method is evaluated on the dataset proposed by.
  • IV. EXPERIMENTS: The page-extraction network predicts a binary per-pixel mask of the main page using 1635 images for 30 epochs.Full images are resized to 6 · 10^5 pixels without patches, and training takes around 4 hours.
  • IV. EXPERIMENTS: Page predictions are converted into page quadrilaterals using Otsu thresholding, morphological opening and closing, and four extreme corner points.These operations produce the final binary page extraction output.
  • IV. EXPERIMENTS: The page-extraction results are reported using mean Intersection over Union (mIoU) in Table I.The supplied table passage identifies the task and metric but does not include the numerical rows.

B. Baseline detection

Baseline detection treats text lines as pixel regions near annotated baselines and converts the predicted mask into polygonal lines through simple post-processing.

  • B. Baseline detection: A baseline is a virtual line where most characters rest and descenders extend below.Baseline detection is presented as an important step for text recognition applications in historical documents.
  • B. Baseline detection: The network predicts pixels within a 5-pixel radius of the training baselines.Images are resized to 10^6 pixels and training runs for 30 epochs.
  • B. Baseline detection: Gaussian filtering and hysteresis thresholding convert the probability map into a binary baseline mask.The Gaussian filter uses σ = 1.5, with p_high = 0.4 and p_low = 0.2.
  • B. Baseline detection: Connected components of the binary mask are converted into polygonal lines.This is the final post-processing step described for baseline extraction.
  • B. Baseline detection: Table II reports results for cBAD: ICDAR2017 baseline detection on the test set.The supplied table passage identifies the benchmark but does not include numerical result rows.

C. Document layout analysis

Document layout analysis assigns semantically meaningful labels to manuscript pixels, using manuscript-specific training and simple class-mask post-processing.

  • C. Document layout analysis: Document layout analysis segments documents into semantically meaningful regions.The experiment uses the DIVA-HisDB dataset and the task formulation from.
  • C. Document layout analysis: The task labels pixels as text regions, decorations, comments, or background, while allowing multi-class labels.A pixel may simultaneously belong to the main text body and a decoration.
  • C. Document layout analysis: DIVA-HisDB contains three manuscripts with 20 training, 10 evaluation, and 10 testing images per manuscript.Each manuscript receives a model trained solely on its corresponding 20 training images.
  • C. Document layout analysis: Post-processing thresholds each class mask at 0.5 and removes connected components smaller than 50 pixels.The page-detection mask is also used to reduce false-positive text detections at image borders.
  • C. Document layout analysis: Table III reports Intersection over Union results for the ICDAR2017 challenging-medieval-manuscripts layout-analysis task.The supplied table passage identifies Task-1 and the IoU metric but does not include numerical rows.

D. Ornament detection

The ornament-detection task targets manuscript decorations of interest to historians, using pixel-wise prediction followed by simple morphological processing and bounding-box extraction. The private dataset contains 912 annotated pages, and evaluation compares detection quality against a region-proposal CNN method.

  • Ornament detection supports historians by filtering pages containing decorations and locating their positions in large manuscript collections.
  • The private dataset contains 912 annotated pages, including 612 pages with one or several ornaments.
  • The dataset is split into 610 training pages, 92 evaluation pages, and 183 testing pages.
  • Predicted masks are thresholded at t = 0.6, morphologically opened and closed, converted to bounding rectangles, and filtered below 0.5% of image area.
  • Evaluation reports precision, recall, f-measure at three IoU thresholds, and mean IoU, compared with a region-proposal method coupled to a CNN classifier.
  • Examples include partial ornament detection, an illustration with a false-positive banner, and correct extraction of multiple ornaments.

E. Photo-collection extraction

The photo-collection task extracts cardboard and photograph regions from high-resolution scans. Pixel-wise class prediction is cleaned morphologically, converted to enclosing rectangles, and evaluated against ground-truth rectangles using mIoU and recall at stringent IoU thresholds.

  • The task extracts cardboard and photograph regions from high-resolution scans of cardboard pieces bearing old photographs.
  • Annotations assign background, cardboard, and photograph classes directly on scans, with 100 training, 20 validation, and 150 testing scans.
  • Predicted classes are cleaned with morphological opening, converted to smallest enclosing rectangles, and optionally constrained by photograph containment within cardboard.
  • Evaluation compares extracted rectangles with smallest ground-truth rectangles using mIoU and recall at IoU thresholds of 0.85 and 0.95.
  • The photo-collection examples show more precise extraction because the zone to be extracted is better defined than in the ornament case.

V. DISCUSSION

Across five tasks, the same network and nearly identical training configurations achieved competitive or better results than state-of-the-art methods. The discussion presents generic segmentation bricks as a route toward accessible document-analysis pipelines, while identifying multi-task training as future work.

  • Across five evaluated tasks, the same network and almost the same training configurations produced results competitive with or better than state-of-the-art methods.
  • Training took less than an hour in some cases, and the approach required little training data, benefits attributed to the pretrained network component.
  • A generic architecture retrained for specific segmentation tasks can, in certain cases, outperform dedicated systems.
  • The results open an avenue toward off-the-shelf programming bricks trainable by non-specialists for large series of document-analysis problems.
  • Future work should study incremental or parallel training across tasks to improve transfer learning and potentially enable a universal segmentation engine.
Loading 1804.10371v2…