TSIT: A Simple and Versatile Framework for Image-to-Image Translation

• Unpaired and paired translation tasks have different data and modeling needs, making their specialized components difficult to integrate without quality degradation.Unpaired tasks may require additional constraints, while paired semantic synthesis is data-dependent and uses spatially coherent structures.Source: §1 · p. 2
• TSIT applies a symmetrical two-stream network to capture and fuse multi-scale structure and style representations through coarse-to-fine feature transformations.FADE preserves content spatial structure, while FAdaIN applies style information.Source: §1 · p. 2
• The framework targets diverse tasks in supervised and unsupervised settings without added constraints such as cycle consistency.Source: §1 · p. 2
• TSIT achieves comparable or better results than task-specific baselines in perceptual quality and quantitative evaluations.Source: §1 · pp. 2–3

Semantic image synthesis

• Unsupervised methods use unpaired data and may add constraints, whereas supervised semantic synthesis depends on well-annotated paired samples.Semantic synthesis targets photorealistic images from segmentation masks and must preserve semantic fidelity without losing diversity.Source: §2 · pp. 3–4

Image-to-image translation

• Some one-to-one GAN-based translation methods produce images with low diversity, while multi-domain and multi-modal approaches explore greater generation diversity.MUNIT disentangles domain-invariant content from domain-specific style; DMIT is designed to capture each domain's multi-modal image nature.Source: §2 · p. 3

Arbitrary style transfer

• The framework seeks to balance and adaptively fuse content and style, enabling user-controllable multi-modal style manipulation with one model.Source: §2 · p. 3
• TSIT reports better synthesis quality than customized style-transfer methods across natural images, real-world scenes, and artistic paintings.Source: §2 · p. 4

Semantic image synthesis

• TSIT learns multi-scale feature-level structure and style representations rather than resizing the input mask, aiming to retain fidelity without diminishing diversity.Source: §2 · p. 4

3 Methodology

• TSIT targets a simple, robust framework that adaptively fuses semantic structure and style in feature space without additional constraints.The design requirements also call for easy training across diverse tasks.Source: §3 · p. 4

3.1 Network Structure

• Symmetrical content and style streams extract multi-scale features for a matching generator, which injects them coarse-to-fine alongside Gaussian noise; multi-scale discriminators assess global structure and fine details.FADE modulates normalized generator activations, while FAdaIN applies style information; discriminators also provide features for feature matching loss.Source: §3.1 · pp. 5–6, Figure 3 · p. 6

3.2 Feature Transformation

• FADE uses multi-scale content features to modulate generator activations, while FAdaIN derives adaptive affine parameters from multi-scale style features for end-to-end fusion.The authors state that FAdaIN enables multi-modal synthesis with arbitrary style control.Source: §3.2 · pp. 7–8

3.3 Objective

• The objective combines hinge adversarial loss with perceptual and feature-matching losses, and the two-stream design makes the typical KL loss optional.The generator and discriminator are trained alternately in a min-max game.Source: §3.3 · p. 8

4.1 Settings

• Experiments cover unsupervised style transfer, supervised semantic synthesis, and multi-modal translation, using Yosemite, BDD100K, Photo→art, Cityscapes, and ADE20K datasets.Comparisons include MUNIT and DMIT for unsupervised tasks, and CRN, SIMS, pix2pixHD, SPADE, and CC-FPSE for supervised tasks.Source: §4.1 · p. 9

4.2 Results

• Across Yosemite, BDD100K day→night, and photo→art transfer, TSIT produces clear, photorealistic or well-stylized outputs while preserving content structure.MUNIT and DMIT show task-specific issues including dark objects, unnatural colors, artifacts, homogeneous color, or grid-like distortions; DMIT is slightly better stylized in some photo→art cases.Source: §4.2 · p. 10, Figure 4 · p. 10, Figure 5 · p. 10
• TSIT outperforms baselines quantitatively across arbitrary style-transfer tasks, although its performance gap is relatively small for photo→art.Table 1 reports FID and IS; lower FID and higher IS indicate better performance.Source: §4.2 · pp. 10–11, Table 1 · p. 11
• For semantic synthesis, TSIT shows better perceptual quality than SPADE and CC-FPSE across varied scenes, with fewer distortions and greater robustness.The comparison includes street, road, outdoor, and indoor scenes.Source: §4.2 · p. 11, Figure 7 · p. 12
• TSIT achieves state-of-the-art segmentation performance on Cityscapes and the best FID score on ADE20K, while matching strong specialized methods overall for semantic synthesis.The comparison uses mIoU, pixel accuracy, and FID.Source: §4.2 · pp. 11–12, Table 2 · p. 12
• A single model translates sunny BDD100K images to different times and weather conditions, enabling style control while retaining photorealistic quality.For snowy outputs, the method introduces snow-like effects on trees and ground despite snow being less apparent in the dataset.Source: §4.2 · p. 13, Figure 8 · p. 13
• In inverse settings, TSIT performs much better than MUNIT and SPADE; MUNIT fails at semantic synthesis, while SPADE loses object details and adds strong artifacts.The tests use these specialized methods without architecture changes, tuning loss weights for the comparison.Source: §4.2 · pp. 13–14, Figure 9 · p. 13
• Ablations show that removing the style stream prevents multi-modal synthesis, while replacing FADE or FAdaIN with concatenation causes content-overload failures or overly strong style regionalization.Removing the content stream weakens semantic structure and produces sky artifacts.Source: §4.2 · pp. 14–15, Figure 10 · p. 14

• TSIT conditions image generation on multi-scale semantic structure and style through feature transformations in a symmetrical two-stream network.Source: §5 · p. 15
• A systematic study verifies TSIT's effectiveness across diverse tasks compared with state-of-the-art task-specific methods.Source: §5 · p. 15
• The authors identify unified frameworks for more tasks as an important direction and suggest unconditional synthesis and greater two-stream or latent-space variability for future work.Source: §5 · p. 15

• Section A details the applications explored, while Section B describes the experimental datasets.Source: Appendix · p. 18
• Section C provides additional implementation details, and Section D presents supplementary ablation studies.Source: Appendix · p. 18
• Section E shows additional images generated by the method.Source: Appendix · p. 18

• The framework explores arbitrary style transfer without paired data, supervised semantic image synthesis, and multi-modal image synthesis for generation diversity.Source: A Application Exploration · p. 18
• For arbitrary style transfer, content and style images enter separate streams, and FADE and FAdaIN adaptively fuse their feature maps at different generator scales.Source: A Application Exploration · p. 18
• The authors describe the method as adaptable to style transfer across natural images, real-world scenes, and artistic paintings.Source: A Application Exploration · p. 18
• For supervised semantic synthesis, segmentation masks provide structure to the content stream and corresponding real images provide style to the style stream.Source: A Application Exploration · p. 18
• Despite a large domain gap, the framework yields comparable or better results than state-of-the-art task-specific methods on semantic image synthesis.Source: A Application Exploration · p. 18
• The authors attribute high-fidelity multi-modal synthesis to disentangling semantic structure and style through the feature transformations and symmetrical two-stream network.Source: A Application Exploration · p. 18

• The unpaired Yosemite dataset contains 1,231 summer and 962 winter training images at 256 × 256 for season transfer.Source: B Dataset Details · p. 19
• 12,454 daytime and 22,884 nighttime BDD100K training images support day-to-night translation, with images scaled to 512 × 256.Source: B Dataset Details · p. 19
• Photo-to-art training combines 6,853 photos and 3,492 paintings, with artworks including Monet, Cézanne, van Gogh, and Ukiyo-e.Images are uniformly resized to 256 × 256.Source: B Dataset Details · p. 19
• Cityscapes provides 2,975 training and 500 evaluation images with dense annotations of 30 classes; images are adjusted to 512 × 256.Source: B Dataset Details · p. 19
• A dataset passage describes wild images with fine annotations of 150 semantic classes.Source: B Dataset Details · p. 19
• 10,000 sunny BDD100K images and 10,000 images across night, cloudy, rainy, and snowy conditions form a 512 × 256 training set.Each non-sunny condition contributes 2,500 images.Source: B Dataset Details · p. 19

Network architecture specifics

• The content/style streams use seven residual blocks with nearest-neighbor downsampling, while the generator upsamples through FADE residual blocks with FAdaIN applied before each block.Content/style blocks use instance normalization; the generator uses SyncBN.Source: C Additional Implementation Details · p. 20
• The generator’s FADE submodule performs element-wise denormalization using learned affine parameters γ and β.FAdaIN layers precede each FADE residual block.Source: C Additional Implementation Details · p. 20
• The model uses multi-scale patch-based discriminators matching, with the same feature-matching layers as those methods.Content/style feature channels progress from 64 to 128, 256, 512, then 1024; resolution halves after each residual block.Source: C Additional Implementation Details · p. 20

Additional training details

• Training runs for 200 epochs on Cityscapes and ADE20K, while unsupervised tasks use task-specific schedules ranging from 10 to 200 epochs.Semantic synthesis uses batch size 16; arbitrary style transfer and multi-modal synthesis use batch size 1. Perceptual feature reconstruction loss requires a content target.Source: C Additional Implementation Details · pp. 20–21

Quantitative evaluation of key component ablation studies

• The full model achieves the lowest FID and highest IS in multi-modal image synthesis; removing any key module reduces quantitative performance.Lower FID and higher IS indicate better performance.Source: D Supplementary Ablation Studies · p. 21

Feature channel ablation studies

• Across unsupervised and supervised tasks, the full TSIT model performs best, while reducing stream channels degrades quality and increasing them raises memory consumption exponentially.The channel counts in the content and style streams are changed together to preserve symmetry.Source: D Supplementary Ablation Studies · p. 22, Table 4 · p. 22, Table 5 · p. 22

Feature-level/Image-level injection ablation studies

• Image-level injection with AdaIN or SPADE performs worse than TSIT’s feature-level FAdaIN or FADE injection in the reported tasks.The comparisons cover arbitrary style transfer on day-to-night translation and semantic synthesis on Cityscapes.Source: D Supplementary Ablation Studies · p. 22, Table 7 · p. 23, Table 8 · p. 23

More examples of generated images

• The authors report that all synthesized images are photorealistic, including examples of arbitrary style transfer and semantic image synthesis.The examples include Yosemite season transfer, BDD100K day-to-night translation, and photo-to-art transfer, as well as Cityscapes and ADE20K synthesis.Source: E More Examples of Generated Images · p. 23, Figure 11 · p. 24, Figure 12 · p. 25
• Fig. 11 presents unsupervised Yosemite summer-to-winter, BDD100K day-to-night, and photo-to-art results.Rows 1–3 show Yosemite, rows 4–6 BDD100K, and rows 7–9 photo-to-art.Source: Figure 11 · p. 24
• Fig. 12 presents supervised semantic synthesis on Cityscapes and ADE20K, including outdoor and indoor scenes.Rows 1–2 show Cityscapes; rows 3–6 show ADE20K outdoor and indoor results.Source: Figure 12 · p. 25