Source-linked AI summary
Bringing Old Photos Back to Life
Ziyu Wan, Bo Zhang, Dongdong Chen, Pan Zhang, Dong Chen, Jing Liao, Fang Wen
TL;DR
Old-photo restoration is difficult because real images contain complex mixed degradations and differ from synthetic training data. The paper proposes triplet domain translation with VAE latent spaces and dual restoration branches, achieving strong visual quality and a 64.86% higher likelihood of first-place selection in a user study.
Problem
Real old photos contain complex, mixed degradations, while models trained on synthetic data suffer from a domain gap and generalize poorly to real images.
Method
The method translates between aligned old-photo and clean-image VAE latent spaces using synthetic paired data, then combines global and local branches for structured and unstructured defects.
Results
64.86% more likely to be chosen as the first-rank result in a 22-user study of 25 old photos, while also achieving the most visually pleasant restoration results.
Takeaways & Limitations
Latent-space domain translation and fused global-local restoration provide a strong approach for restoring severely degraded old photos with mixed defects.
Takeaways & Limitations
The approach assumes that real old photos and synthetic images can be encoded into the same VAE latent space.
Abstract
from arXiv · showhide
We propose to restore old photos that suffer from severe degradation through a deep learning approach. Unlike conventional restoration tasks that can be solved through supervised learning, the degradation in real photos is complex and the domain gap between synthetic images and real old photos makes the network fail to generalize. Therefore, we propose a novel triplet domain translation network by leveraging real photos along with massive synthetic image pairs. Specifically, we train two variational autoencoders (VAEs) to respectively transform old photos and clean photos into two latent spaces. And the translation between these two latent spaces is learned with synthetic paired data. This translation generalizes well to real photos because the domain gap is closed in the compact latent space. Besides, to address multiple degradations mixed in one old photo, we design a global branch with a partial nonlocal block targeting to the structured defects, such as scratches and dust spots, and a local branch targeting to the unstructured defects, such as noises and blurriness. Two branches are fused in the latent space, leading to improved capability to restore old photos from multiple defects. The proposed method outperforms state-of-the-art methods in terms of visual quality for old photos restoration.
1. Introduction
Old photos deteriorate through defects that are difficult to restore automatically, while synthetic-data-trained models struggle to generalize to real images. The proposed method uses triplet domain translation and latent-space restoration to address this gap and mixed degradation.
- Manual restoration is laborious and time-consuming, motivating automatic repair for severely degraded old photos.
- Traditional inpainting methods repair localized defects but do not address spatially uniform effects such as film grain, sepia, and color fading.
- Complex, poorly modeled degradation creates a domain gap that causes models trained on synthetic data to generalize poorly to real old photos.
- Old photos combine unstructured defects requiring neighborhood information with structured defects requiring global image context.
- The method learns translation among real old photos, synthetic degraded images, and clean images in latent space, while using a partial nonlocal block for structured defects.
2. Related Work
Earlier restoration methods largely target individual or localized defects using handcrafted or synthetic-data-driven techniques. The paper addresses the mixed, unknown degradations of real old photos with a data-driven restoration approach.
- Single degradation image restoration: Learning-based restoration methods achieve strong results but depend on synthetic-data quality and suffer from domain gaps on real old photos.
- Mixed degradation image restoration: Mixed degradation restoration is less explored because real images may combine scratches, resolution loss, color fading, and film noise.
- Old photo restoration: Earlier old-photo methods detect scratches and blotches from low-level features and inpaint them using nearby textures.
- Old photo restoration: Handcrafted models and low-level features are difficult to use effectively, and prior methods do not jointly restore color fading or low resolution.
- Old photo restoration: The proposed data-driven approach restores multiple defects simultaneously and converts heavily damaged old photos toward a modern style.
3. Method
The method treats restoration as translation across real, synthetic, and clean-image domains, learning the restoration mapping in aligned latent spaces. Its architecture combines domain alignment with local and global processing for mixed defects.
- 3.2. Multiple degradation restoration: The network addresses mixed degradation by combining local processing with a global branch that uses nonlocal context for structured defects.
- Triplet domain translation uses real photos, synthetic degraded images, and paired clean ground truths rather than only two image domains.
- Synthetic degraded images are paired with clean images, enabling supervised learning of the restoration mapping while real photos remain unpaired.
- 3.1. Restoration via latent space translation: Encoders map the real, synthetic, and clean domains into latent spaces, with real and synthetic corrupted images aligned through a shared representation.
- 3.1. Restoration via latent space translation: The latent restoration mapping translates corrupted features into the clean-image latent space, which is then decoded to produce restored images.
II. T
The restoration network aligns real and synthetic photos in a shared VAE latent space, learns latent restoration from synthetic pairs, and combines local and global branches for mixed defects.
- Domain alignment: VAE1 encodes real and synthetic images into aligned latent spaces using reconstruction, adversarial, and latent adversarial objectives.VAE2 separately projects clean images into a corresponding latent space.
- Latent restoration: The latent mapping network is trained on synthetic corrupted-clean pairs while the two VAEs remain fixed.Its loss combines latent ℓ1, adversarial, and feature-matching terms.
- Latent restoration: The latent restoration mapping is intended to generalize from synthetic images to real photos because real and synthetic domains share an aligned compact representation.Learning in the compact latent space is described as easier than learning directly in high-dimensional image space.
- Multiple degradation restoration: A partial nonlocal global branch uses masked long-range context to inpaint structured defects without adopting corrupted pixels as completion context.The affinity is mask-modulated, and the output is a weighted average of correlated features.
- Multiple degradation restoration: The global and local branches are fused through mask-guided feature composition, enabling the latent restoration network to handle multiple degradations.The local branch uses residual-block transformations, while the global branch targets inpainting.
4. Experiment
Experiments evaluate implementation choices, synthetic-data restoration, real-photo generalization, user preference, and ablations. The method performs strongly perceptually but remains limited on complex shading artifacts.
- Experimental setup: The evaluation uses Pascal VOC for training, DIV2K corrupted images and a real old-photo test set, with comparisons against several restoration baselines.Baselines include operation-wise attention, deep image prior, Pix2Pix, CycleGAN, and sequential BM3D plus EdgeConnect.
- Quantitative comparison: PSNR/SSIM rank the method second-best, while LPIPS and FID place it among the best-performing methods on synthetic DIV2K images.The operation-wise attention method has the best PSNR/SSIM, whereas the method and Pix2Pix perform best on LPIPS; the method has a slight FID advantage.
- Qualitative comparison: On real old photos, qualitative comparisons show restoration of both unstructured and structured degradation with more visually pleasant results than competing methods.Reference-based metrics cannot be applied because real-photo restoration ground truth is unavailable.
- User study: 64.86% more likely to be selected as the first-rank result, according to a user study of 25 photos and 22 users.The study reports subjective restoration-quality rankings against other methods.
- Ablation studies: The ablations test image-level Pix2Pix, VAE latent regularization, two-stage VAE training, and latent adversarial loss as components are added.Additional ablations examine the partial nonlocal block, which better inpaints structured defects without affecting non-hole regions.
- Limitation: The method cannot handle complex shading artifacts because the dataset contains few old photos with such defects.The authors suggest explicitly synthesizing shading effects or adding more such training photos.
5. Discussion and Conclusion
The paper concludes that triplet latent-domain translation and partial nonlocal restoration improve severe old-photo restoration, while complex shading remains outside the method’s demonstrated capability.
- Conclusion: The triplet domain translation network reduces the domain gap and learns translation to clean images in latent space, improving generalization relative to prior methods.The framework uses real old photos, synthetic images, and corresponding clean images.
- Conclusion: A partial nonlocal block leverages global context so scratches can be inpainted with better structural consistency.This complements restoration of mixed degradations in severe old photos.
- Limitation: Complex shading remains a limitation because the training dataset contains few old photos with such defects.Potential remedies include modeling shading during synthesis or adding more affected photos to training data.