Source-linked AI summary
CollaGAN : Collaborative GAN for Missing Image Data Imputation
Dongwook Lee, Junyoung Kim, Won-Jin Moon, Jong Chul Ye
TL;DR
Missing image inputs can bias applications, and conventional imputation remains difficult for high-dimensional natural images. CollaGAN reformulates image imputation as multi-domain image translation using a single generator and discriminator with remaining clean inputs, achieving higher visual quality than existing methods across various tasks.
Problem
Missing input images can introduce substantial bias, while standard imputation methods are limited for high-dimensional image data.
Method
CollaGAN uses multiple available images from different domains with a single generator and discriminator to estimate missing image data.
Results
CollaGAN produces higher visual quality than existing methods across various image-imputation tasks.
Takeaways & Limitations
Combining information from available domains enables CollaGAN to support missing image-data imputation with promising performance across real-world applications.
Takeaways & Limitations
For MR contrast imputation, CycleGAN and StarGAN are limited because they use a single input contrast, whereas accurate intensity estimation requires pixels from at least three contrasts.
Abstract
from arXiv · showhide
In many applications requiring multiple inputs to obtain a desired output, if any of the input data is missing, it often introduces large amounts of bias. Although many techniques have been developed for imputing missing data, the image imputation is still difficult due to complicated nature of natural images. To address this problem, here we proposed a novel framework for missing image data imputation, called Collaborative Generative Adversarial Network (CollaGAN). CollaGAN converts an image imputation problem to a multi-domain images-to-image translation task so that a single generator and discriminator network can successfully estimate the missing data using the remaining clean data set. We demonstrate that CollaGAN produces the images with a higher visual quality compared to the existing competing approaches in various image imputation tasks.
1. Introduction
Missing image data can bias analysis, while conventional imputation methods are limited for high-dimensional images. CollaGAN addresses this by using multiple clean inputs to estimate missing domains and reports strong performance across image-imputation tasks.
- Motivation: Missing inputs can introduce substantial bias, errors in analysis, and reduced statistical efficiency.Acquisition cost, time, and systematic dataset errors can make complete input sets difficult to obtain.
- Motivation: Standard mean, regression, and stochastic imputation methods are limited for images because they require knowledge of the high-dimensional image manifold.
- Problem formulation: Image imputation differs from translation because it estimates an infrequently missing domain using the remaining clean domains.The algorithm should estimate missing data in any domain by exploiting the other domains.
- Contribution: CollaGAN learns the shared image manifold synergistically from multiple inputs rather than a single input, supporting more accurate missing-data estimation.
- Contribution: CollaGAN retains a one-generator architecture similar to StarGAN, making it more memory-efficient than CycleGAN.
- Results: The authors report the best performance among state-of-the-art algorithms across various image-imputation tasks.
2. Related Work
GAN-based image translation methods provide foundations for modeling image-domain relationships, but existing approaches face limitations in cycle consistency, scalability, or input usage. CollaGAN extends multi-domain translation toward image imputation with multiple inputs.
- GAN foundations: GANs use a generator and discriminator, with the discriminator distinguishing real from fake samples while the generator synthesizes realistic samples.
- GAN foundations: Conditional GANs control generated outputs by providing labels or other condition information to the generator.
- Multi-domain translation: CycleGAN and DiscoGAN use cycle consistency to preserve attributes but require separate generator pairs for domain pairs, creating N(N-1) generators across N domains.
- Multi-domain translation: StarGAN and Radial GAN handle multiple domains with a single generator, while StarGAN also uses domain classification in its discriminator.
3. Theory
CollaGAN uses one generator for multi-input, single-output image imputation across domains, with collaborative mappings, multiple cycle consistency, adversarial realism, and domain classification. Its losses combine reconstruction and perceptual consistency with discriminator objectives that enforce realistic, correctly classified outputs.
- Image imputation using multiple inputs: CollaGAN processes multiple input domains with one generator to synthesize the missing target domain, unlike single-input translation.The target domain is specified during generation, and training samples multiple input-target combinations.
- Image imputation using multiple inputs: Randomly selecting the N possible input-target combinations trains the generator to learn mappings toward multiple target domains.The target index κ identifies the desired output domain.
- Network losses: Multiple cycle consistency reconstructs the original domains by feeding generated outputs into N−1 backward multi-input mappings.For N = 4, the forward output supports three backward combinations for reconstructing the other domains.
- Network losses: The discriminator jointly evaluates real-versus-fake status and domain class using adversarial and domain-classification losses.Its two paths share neural-network weights except for their final layers; Least Square GAN loss is used for adversarial training.
- Network losses: SSIM is added to multiple cycle consistency to preserve image structure, while the generator is trained to produce samples classified as the target domain.The SSIM loss is differentiable and supplements the cycle-consistency objective.
- Mask vector: A binary one-hot mask vector supplies the target-domain label to the single generator through concatenation with the input images.The mask has N class channels and the same spatial dimensions as the input images.
4. Method
CollaGAN uses a generator–discriminator framework to impute missing images from multiple related inputs, with task-specific generator designs. Its implementations use U-net-based generators and discriminators that assess both realism and domain.
- Network architecture: The proposed method consists of a generator and discriminator, with task-specific redesigns built around a shared general architecture.The networks were adapted to each task while retaining similar overall structure.
- Generators: The generators use U-net structures with encoder–decoder paths, instance normalization, and Leaky-ReLU activations.U-net connects encoder and decoder components through contracting paths.
- Generators: For MR contrast translation, two convolution branches with 1x1 and 3x3 filters capture multi-scale features before concatenation.The design addresses both voxel-wise processing and neighborhood or larger-field information.
- Generators: The illumination generator uses the original U-net with instance normalization, while facial-expression translation handles multiple, non-strictly aligned facial images.The facial-expression inputs contain subject head movements between expressions.
- Discriminator: The discriminator uses convolutional layers and Leaky-ReLU, with separate headers for real/fake discrimination and domain classification.PatchGAN classifies local patches, dropout helps prevent overfitting, and MR translation additionally uses multi-scale branches.
- Network training: Training uses Adam with learning rate 0.00001, β1 = 0.9, and β2 = 0.999, preceded by 10 epochs of classifier training on real images.The generator and discriminator were then trained simultaneously.
5. Experimental Results
CollaGAN was evaluated on MR contrast, illumination, and facial-expression imputation against CycleGAN and StarGAN. Across these tasks, it produced more natural or accurate reconstructions and remained effective with incomplete inputs.
- Experimental setup: Experiments used subject-level train, validation, and test splits, with CycleGAN and StarGAN as comparison models.All experiments evaluated images from subjects unseen during training.
- MR contrast imputation: CollaGAN reconstructed four MR contrasts with minimum errors compared with CycleGAN and StarGAN, while StarGAN produced poor results.The method combined multiple contrasts, whereas CycleGAN and StarGAN used a single input contrast.
- MR contrast imputation: Combining multiple MR contrasts enabled accurate reconstruction of regions that single-input StarGAN could not recover from neighboring pixels alone.The example contrasts bright cerebrospinal fluid in T2-weighted images with dark values in T2-FLAIR* inputs.
- Illumination imputation: CollaGAN generated natural illumination images while maintaining facial color, brightness balance, and textures with minimum NMSE/SSIM errors.CycleGAN and StarGAN showed reddish or unnatural lighting and weaker representation of facial structure.
- Facial-expression imputation: For facial-expression imputation, CollaGAN produced the most natural images with minimum NMSE and best SSIM compared with CycleGAN and StarGAN.The competing models used only the neutral-expression input, whereas CollaGAN used multiple facial expressions.
- Incomplete input sets: With one facial-expression input substituted from another subject, CollaGAN generated results similar to those from complete input sets.The substituted information from another subject was used to impute a missing expression.
6. Conclusion
CollaGAN imputes missing image data by combining available inputs with a single generator and discriminator, producing higher-visual-quality images than existing methods.
- CollaGAN combines available data using a single generator and discriminator for missing image data imputation.
- The method produces higher visual quality than existing methods across missing image data imputation applications.
7. Collaborative training
Collaborative training evaluates how input redundancy and input quantity affect imputation. Randomly nulling inputs encourages the model to use multiple inputs collaboratively, while more available information improves reconstruction quality.
- Random input nulling is used to control missing inputs and encourage collaborative learning rather than reliance on one informative input.
- The study varies missing inputs from seven to one to evaluate how input quantity affects reconstruction.
- Increasing input information qualitatively improves facial-expression reconstruction results.
8. Implementation Details
The implementation uses task-specific generator and discriminator designs for MR contrast, illumination, and facial-expression imputation, with U-net-based components and multi-scale or multi-branch processing. Experiments also vary the number of available facial-expression inputs from seven to one.
- Network implementation: Each of the three imputation tasks uses redesigned generators and discriminators while retaining a similar general network architecture.
- MR contrast translation: The MR generator is a U-net redesigned with CCNR units, which include convolution, concatenation, instance normalization, and Leaky-ReLU.
- MR contrast translation: The MR discriminator uses three branches to process features at original, quarter-resolution, and progressively reduced scales before concatenation.
- Facial expression translation: Facial-expression experiments reconstruct target expressions using other 8-N expressions as inputs while varying Missing N from seven to one.
- Illumination translation: The illumination generator follows a U-net with instance normalization and Leaky-ReLU, while its discriminator uses strided convolutions and real/fake and domain-classification branches.
- Facial expression translation: Facial-expression translation uses a multi-branched U-net with separate input encoders, concatenated decoder features, and eight facial-expression output channels.
9. Additional evaluation results
Additional evaluation uses NMSE and SSIM for facial-expression and illumination imputation, compares CollaGAN with pix2pix, CycleGAN, and StarGAN, and includes a Mechanical Turk assessment. CollaGAN achieves the best reported quantitative comparison and is selected as best in 70.8% of reconstruction results.
- Quantitative evaluation: Facial-expression and illumination imputation are quantitatively evaluated using NMSE and SSIM.
- Quantitative evaluation: The additional comparison includes pix2pix to test whether multiple cycle consistency losses improve performance beyond single-pair supervision.
- Quantitative evaluation: CollaGAN shows the best performance against CycleGAN, StarGAN, and pix2pix in the reported quantitative results, including paired data.
- Qualitative evaluation: 70.8% of reconstruction results were selected as best by 30 Mechanical Turk participants evaluating image quality and target-expression representation.
10. Ablation study
The ablation study evaluates multiple cycle consistency and SSIM losses on the RaFD dataset. Replacing multiple cycle consistency with direct L1 regression performed worse, while LSSIM improved reconstruction performance.
- The ablation study used the RaFD dataset to evaluate the proposed multiple-cycle consistency and SSIM losses.
- Replacing multiple cycle consistency with direct L1 regression from multiple inputs to one target produced inferior performance.
- LSSIM improved reconstruction performance according to NMSE and SSIM.