Source-linked AI summary
DE-GAN: A Conditional Generative Adversarial Network for Document Enhancement
Mohamed Ali Souibgui, Yousri Kessentini
TL;DR
Degraded documents are difficult to read and can impair OCR, motivating restoration across several document-enhancement tasks. DE-GAN uses conditional GANs for end-to-end enhancement and reports strong results across cleaning, binarization, and watermark-removal settings, while its evaluation is constrained by limited paired data in some comparisons.
Problem
Document degradation can obscure text and impair OCR, while GAN-based document enhancement had not previously been studied across these restoration problems.
Method
DE-GAN is a conditional GAN that generates a clean document image from a degraded or watermarked input using adversarial and additional log losses.
Results
DE-GAN produces strong results across document clean up, binarization, and watermark removal, outperforming fully convolutional networks, CycleGAN, and pix2pix-HD in the reported experiments.
Takeaways & Limitations
The results support using a simple end-to-end conditional-GAN framework across multiple document-enhancement tasks and degradation types.
Takeaways & Limitations
The pix2pix-HD comparison used few DIBCO samples, and the authors expect more data could improve its performance.
Abstract
from arXiv · showhide
Documents often exhibit various forms of degradation, which make it hard to be read and substantially deteriorate the performance of an OCR system. In this paper, we propose an effective end-to-end framework named Document Enhancement Generative Adversarial Networks (DE-GAN) that uses the conditional GANs (cGANs) to restore severely degraded document images. To the best of our knowledge, this practice has not been studied within the context of generative adversarial deep networks. We demonstrate that, in different tasks (document clean up, binarization, deblurring and watermark removal), DE-GAN can produce an enhanced version of the degraded document with a high quality. In addition, our approach provides consistent improvements compared to state-of-the-art methods over the widely used DIBCO 2013, DIBCO 2017 and H-DIBCO 2018 datasets, proving its ability to restore a degraded document image to its ideal condition. The obtained results on a wide variety of degradation reveal the flexibility of the proposed model to be exploited in other document enhancement problems.
1 INTRODUCTION
Degraded documents are difficult for both human and computer interpretation because damage can obscure text, while existing restoration needs span multiple degradation types. The paper proposes conditional GANs as a flexible document-enhancement framework and reports state-of-the-art gains in degraded-document binarization.
- Motivation: Document degradation from stains, wrinkles, shadows, blur, lighting variation, distortion, and watermarks can make text difficult to recover.The problem is especially difficult when damage overlaps text or has the same or darker color than the document font.
- Motivation: The target is a clean grayscale or binary document that removes noise and watermarks while retaining text quality.The framework addresses degraded-document recovery and watermark removal without prior knowledge of the degradation or watermark.
- Approach: Conditional GANs learn an output image conditioned on an input image, matching document enhancement as a text-preserving image-to-image translation task.The paper motivates cGANs because they can generate a corresponding output image from an input image.
- Contributions: The paper presents cGANs as a framework for document clean up, binarization, and watermark removal, including dense watermark and stamp removal.It describes the approach as a simple but flexible architecture intended for different document degradation problems.
- Contributions: The approach experimentally achieves higher performance than state-of-the-art methods in degraded document binarization.The paper also presents the framework as applicable across multiple document-enhancement problems.
2 RELATED WORKS
Document enhancement includes cleaning, binarization, and watermark removal, with classical methods based largely on thresholding, energy models, morphology, or matting. Deep learning and GAN-based image translation provide alternative ways to learn restoration from degraded inputs, although paired data remains important for the proposed direction.
- Document enhancement: Document enhancement covers degraded-document clean up, binarization, watermark removal, and other restoration tasks such as unshadowing, super-resolution, deblurring, and dewarping.The paper focuses on clean up, binarization, and watermark removal while placing them within a broader enhancement field.
- Degraded document recovery and binarization: Document binarization classifies pixels as text or degradation to produce a clean binary image, whereas cleaning can preserve grayscale or color values.This distinction explains why the paper treats cleaning and binarization as related but not identical problems.
- Degraded document recovery and binarization: Classical binarization methods use global or local thresholding, while later approaches apply edge detection, variational models, wavelet shrinkage, energy functions, or mathematical morphology.These methods estimate or separate document structure through hand-designed image-processing procedures.
- Degraded document recovery and binarization: Deep learning methods directly separate foreground text from background noise using paired degraded and clean images rather than predicting a threshold.The paper notes that these methods require a considerable amount of paired data.
- Watermark removal: The paper introduces watermark removal in document images as a distinct problem related to binarization and image matting.In this formulation, text is treated as background and the watermark as foreground, enabling comparison with natural-image watermark removal.
- GAN-based enhancement: GAN research motivates document enhancement as image-to-image translation, including applications such as semantic segmentation and super-resolution.The related work uses these examples to motivate exploiting GAN mechanisms for document processing.
3 PROPOSED APPROACH
DE-GAN treats document enhancement as conditional image-to-image translation, generating a clean document from a degraded or watermarked input. Its adversarial training combines a cGAN objective with an additional text-preserving loss, using U-net and fully convolutional discriminator components.
- 3 PROPOSED APPROACH: DE-GAN models document enhancement as conditional image-to-image translation from a degraded document to a clean image.The discriminator also receives the conditioned degraded image when judging the generated or ground-truth output.
- 3 PROPOSED APPROACH: The generator follows a U-net architecture, while the discriminator is a six-layer fully convolutional network producing a 2D realness-probability map.The discriminator concatenates the degraded image with the clean ground-truth or generated image before classification.
- 3 PROPOSED APPROACH: The generator produces a clean image from the degraded or watermarked input and aims to match the ground-truth clean image.The paper denotes the degraded or watermarked image by I_W, the generated clean image by I_C, and the ground truth by I_GT.
- 3 PROPOSED APPROACH: An additional log loss encourages the generated image to preserve the same text as the ground truth.The paper also states that this added loss boosts training speed.
- 3 PROPOSED APPROACH: After training, only the generative network is used to enhance degraded images.The discriminator serves to force better generator outputs during training and is then discarded.
4 EXPERIMENTS AND RESULTS
DE-GAN was evaluated on document cleaning, binarization, watermark removal, and deblurring-related enhancement tasks, using benchmark and newly created data. Across these experiments, it produced high-quality restorations and competitive or superior results, while dense degradation remained difficult in some cases.
- Document cleaning: DE-GAN recovered documents close to the ground truth in the document-cleaning experiments.The evaluation used 112 training images and 32 test images from the Noisy Office Database, with overlapped 256 × 256 patches.
- Document binarization: DE-GAN was superior to current state-of-the-art methods on DIBCO 2013 according to PSNR, F-measure, Fps, and DRD.The comparison used 6,824 training pairs extracted from 80 images, and qualitative results were shown for degraded documents.
- Document binarization: DE-GAN outperformed competing approaches on DIBCO 2017 and achieved the best performance on its test set.The DIBCO 2017 comparison included the top five ranked approaches from a competition involving 18 research groups and 26 algorithms.
- Document binarization: On H-DIBCO 2018, DE-GAN achieved the second-best DRD, PSNR, F-measure, and pseudo F-Measure while using a simple end-to-end model without additional processing.The competition winner used extensive preprocessing and post-processing, whereas DE-GAN maintained good performance across several datasets and enhancement tasks.
- Watermark removal: Watermark-removal experiments used a newly created database of 1000 paired images, including dense synthetic watermarks and stamps.Training used 7658 patches from 800 watermarked documents, with 200 documents reserved for testing.
- Watermark removal and OCR: DE-GAN removed dense watermarks while preserving text quality, and reduced character error rate from 0.37 on degraded documents to 0.01 on enhanced documents.The OCR evaluation used four images, including DIBCO documents and documents with dense watermarks.
5 CONCLUSION
DE-GAN restores severely degraded documents across multiple enhancement tasks and outperforms several comparison methods on benchmark datasets. The method also substantially improves baseline OCR performance, motivating OCR-aware discriminator evaluation as future work.
- DE-GAN restores severely degraded document images using a modified pix2pix architecture with a deeper generator and an additional adversarial-plus-log loss.The model generates an enhanced document from its degraded version.
- DE-GAN achieves strong results across document enhancement tasks and outperforms fully convolutional networks, CycleGAN, and pix2pix-HD.The reported tasks include document cleaning, binarization, deblurring, and watermark removal.
- DE-GAN improves results over many recent state-of-the-art methods on DIBCO 2013, DIBCO 2017, and H-DIBCO 2018.
- DE-GAN boosts baseline OCR performance by a large margin, while future work proposes adding OCR evaluation to the discriminator.The authors also plan to test mobile-captured documents with shadows, blur, and low resolution.