Network-to-Network Translation with Conditional Invertible Neural Networks

1 Introduction

• Massive domain-specific models demand substantial data and computational resources, yet learning representations that also support cross-domain transfer remains challenging.The introduction frames reuse of these invested resources as a motivation for connecting existing representations.Source: §1 · p. 1
• The approach translates between arbitrary fixed, off-the-shelf representations without altering or retraining the expert models.It is contrasted with approaches that require learning or fine-tuning existing representations.Source: §1 · p. 2
• A conditional invertible neural network models translation ambiguity as a residual, supporting BERT [15] to BigGAN [4] text-to-image transfer and content creation.The stated contributions also include model diagnostics and computationally affordable transfer without gradients through expert models.Source: §1 · p. 2

2 Related Work

• Most approaches to deep-learning-based domain-to-domain translation rely on generative models—VAEs [37] [58], GANs, autoregressive models [70], or normalizing flows [50]—that transform simple base-distribution samples into complex target distributions.Source: §2 · p. 2
• Conditional synthesis ranges from class labels [47] [36] and attributes to text [82] [55] and image-to-image tasks such as grayscale, low-resolution, edge, segmentation, and keypoint inputs.Image-to-image methods may use aligned or unaligned pairs.Source: §2 · pp. 2–3
• The paper presents post-hoc fusion of two deep networks, using a conditional INN to transfer between their representations while modeling ambiguity as a residual.It frames this as a way to combine pretrained models for conditional content creation without making the pretrained models aware of the new capability.Source: Figure 2 · p. 3, §2 · p. 3

3 Approach

• The approach assumes access to already trained models, but not practical access to their training procedures.The paper gives model complexity and unavailable components as examples of why training access may be missing.Source: §3 · p. 3
• Because representations may be non-isomorphic and invariant to task-irrelevant details, translation is modeled probabilistically as p(zΘ|zΦ).A residual v is introduced so that, for a given zΦ, it uniquely determines zΘ.Source: §3 · p. 4
• The conditional invertible network τ estimates residual v from both representations, capturing information in zΘ that is not already represented in zΦ.Invertibility allows τ to be reversed with respect to v for a given zΦ.Source: §3 · p. 4
• Making v independent of zΦ is pursued by minimizing the divergence between p(v|zΦ) and an independent prior q(v), chosen as standard normal in practice.When the objective is minimized, p(v|zΦ)=q(v), and sampling v from q(v) followed by τ produces samples from p(zΘ|zΦ).Source: §3 · p. 4
• The objective also upper-bounds mutual information I(v, zΦ), so minimizing it works toward independence between the residual and conditioning representation.The paper describes this interpretation in connection with the information bottleneck [1].Source: §3 · p. 4
• The cINN guarantees perfect reconstruction of zΘ through invertibility, avoiding the β trade-off described for the deep variational information bottleneck [1].The method operates on hidden representations, trains independently of the domains, and requires no gradients with respect to expert models.Source: §3 · p. 5

4.1 Translation to BigGAN

• BERT-to-BigGAN synthesis captures fine- and coarse-grained captions and is on-par with state-of-the-art Inception and FID scores without training a text encoder or image decoder.The comparison uses COCO-stuff captions.Source: Table 1 · p. 5, §4.1 · p. 6

4.2 Repurposing a single target generator for different source domain models

• Different segmentation representations produce varied images, and using logits instead of final segmentation outputs reduces diversity, exposing additional information such as animal color.Selecting different expert layers provides control over synthesis.Source: §4.2 · p. 6, Figure 3 · p. 6
• Sobel-edge inputs enable edge-to-image translation, while texture-biased ImageNet ResNet features preserve texture patches in images completed from masked inputs.Source: §4.2 · pp. 6–7
• Combining a 32 × 32 autoencoder with a 256 × 256 autoencoder captures translation ambiguities and enables generative superresolution.Source: §4.2 · p. 7, Figure 4 · p. 7

4.3 Evaluating image modification capabilities of our generic approach

• Attribute modifications are more coherent than [8] and yield higher-quality images by FID; glasses FID is high because it is calculated over the complete dataset.Source: §4.3 · p. 8, Figure 6 · p. 8
• Exemplar-guided transfer recombines an exemplar’s residual with another image’s segmentation, while spatial deformation supports unsupervised shape disentangling and models also transfer between unpaired datasets.Source: §4.3 · p. 8, Figure 7 · p. 9

5 Conclusion

• The conditionally invertible translation network transfers between existing fixed domain representations without altering them or requiring costly retraining or finetuning.The conclusion describes the method as plug-and-play and independent of individual domains.Source: §5 · p. 9
• The method combines pretrained domain experts for new tasks, including BERT and BigGAN, with a single GPU rather than the vast multi-GPU resources required to train such models.The paper identifies speech, music, and brain signals as possible future transfer domains.Source: §5 · p. 9
• Figure 8 illustrates transfers from oil portraits and anime to FFHQ, between FFHQ and CelebA-HQ, and between human and animal faces.The figure caption describes these examples as unpaired transfer.Source: Figure 8 · p. 9

Broader Impact

• Reusing powerful existing models can reduce computational demands instead of repeating costly training for every new application.The paper links this concern to the large environmental footprint of single training runs of large-scale models.Source: Broader Impact · p. 10
• Efficiently connecting expert models across diverse data and problems may support new research directions and interdisciplinary work.The discussion specifically identifies vision and language as disciplines whose expert models can be difficult to bring together.Source: Broader Impact · p. 10
• Computationally efficient transfer without costly retraining can increase research opportunities for economically weaker institutions and countries.The paper notes that the cost of training state-of-the-art models can limit who can afford their transfer and application.Source: Broader Impact · p. 10
• The approach can widen the scope in which publicly shared pretrained models are applied.The discussion frames this as increasing the applicability of research output funded by public resources.Source: Broader Impact · p. 10
• Controlled image synthesis and modification have broad creative applications but can also be misused for forgery and manipulation.Source: Broader Impact · p. 10

Overview

• The appendix surveys requirements and costs, derives the training objective, demonstrates unsupervised shape–appearance disentangling, and gives unpaired-translation examples and technical details.It also describes an MLP ablation, the reusable autoencoder architecture and training, and cINN architecture and BERT-to-BigGAN details.Source: Overview · p. 15

A Computational Cost and Energy Consumption

• The cINN's training costs are two orders of magnitude smaller than those of the domain experts, and the comparison covers single-run computational costs.The authors argue that expert training costs can be amortized across new tasks by recombining the experts; Table S1 specifies assumptions used for energy, cost, and emissions estimates.Source: A Computational Cost and Energy… · p. 15, Table S1 · p. 15

B Training Objective

• The derivation of Eq. (4) uses a change of variables and the invertibility of τ to express the conditional density of v in terms of the density of zΘ and a Jacobian determinant.Taking the expectation over zΦ yields Eq. (4).Source: B Training Objective · p. 16

C Unsupervised Disentangling of Shape and Appearance

• In the Animals setting, randomly deforming inputs supplies training pairs; the learned residual v represents shape independently of appearance and can be recombined with another image's appearance.The reported result is unsupervised disentangling of shape and appearance; Figure S1 illustrates the recombination.Source: C Unsupervised Disentangling of Shape… · p. 17, Figure S1 · p. 17

D Unpaired Image Translation

• For unpaired translation, the residual v captures commonalities between source and target sets and supports meaningful correspondences; examples preserve face viewpoint across datasets.The appendix also shows bidirectional human–animal examples and uses the same v projected onto different datasets.Source: D Unpaired Image Translation · p. 18, Figure S2 · p. 18, Figure S3 · p. 18

• At the deepest segmentation layer, the MLP's mean over possible outputs is not meaningful, while the cINN still samples coherent translations; FID scores validate this behavior across the test set.Source: E Ablation Study: Replacing our cINN… · p. 19
• For early segmentation layers, where translation ambiguity is low, both the cINN and MLP handle translation successfully.Source: E Ablation Study: Replacing our cINN… · p. 19
• At a deeper layer with moderate ambiguity from increased segmentation-model invariances, the MLP does not accurately reflect the translation space.Source: E Ablation Study: Replacing our cINN… · p. 19

• The same autoencoder g weights and architecture serve Sec. 4.2, exemplar-guided Sec. 4.3, and supplementary samples, pairing a ResNet-101 encoder with a BigGAN decoder.Source: F Architecture and Training of the… · p. 20
• Without class information, the encoder's sampled Gaussian latent passes through a fully connected softmax layer to replace BigGAN's one-hot class vector.Source: F Architecture and Training of the… · p. 20
• Training combines VGG-16 perceptual reconstruction loss and learnable output variance γ with a PatchGAN discriminator for improved image quality.Source: F Architecture and Training of the… · p. 20
• Landscape samples are diverse and realistic when generated from label maps or segmentation-model outputs translated into g's space.Source: Figure S6 · p. 21
• Unlike [20], the reconstruction weighting uses an adaptive λ computed from gradient ratios at the decoder's last layer.Source: F Architecture and Training of the… · p. 21
• Table S2 describes autoencoder architectures for CelebA and Animals at 128 × 128 resolution.Source: Table S2 · p. 22

G.1 Architecture of the conditional INN

• G.1 Architecture of the conditional INN: The cINN uses 20 blocks, each combining affine coupling, actnorm, and fixed permutation layers.Source: Figure S7 · p. 22, §G.1 · p. 22
• G.1 Architecture of the conditional INN: Alternating coupling layers mix input components, while a non-invertible embedding H processes conditioning y and leaves τ conditionally invertible.Source: §G.1 · pp. 22–23
• G.1 Architecture of the conditional INN: Typical training uses batch sizes of 10–25, 4–12 GB VRAM, and converges in less than a day.Source: §G.1 · p. 23

G.2 Training Details for Bert-to-BigGAN Translation

• G.2 Training Details for Bert-to-BigGAN Translation: Additional BERT-to-BigGAN examples show high diversity in synthesized outputs.Source: Figure S8 · p. 23
• G.2 Training Details for Bert-to-BigGAN Translation: A variational autoencoder stochastically reconstructs BigGAN's discrete class embedding h to dequantize it and prevent τ from collapsing onto one dimension.Source: §G.2 · pp. 23–24, Table S4 · p. 24
• G.2 Training Details for Bert-to-BigGAN Translation: τ is trained to map BERT representations zΦ and sampled ambiguities v to BigGAN representations zΘ using the objective from Eq. (4).Source: §G.2 · p. 24