Source-linked AI summary
Multimodal Transfer: A Hierarchical Deep Convolutional Neural Network for Fast Artistic Style Transfer
Xin Wang, Geoffrey Oxholm, Da Zhang, Yuan-Fang Wang
TL;DR
High-resolution feed-forward style transfer can lose correct texture scales and intricate brushwork. The paper proposes multimodal hierarchical stylization using multiple scales and color-luminance representations, and reports better capture of coarse and intricate texture patterns while retaining near-real-time performance. The authors conclude that the scheme produces more visually appealing high-resolution results, while noting future work on lower-memory losses for larger images.
Problem
High-resolution feed-forward style transfer can produce texture-scale mismatch and fail to capture small, intricate textures such as brushwork.
Method
A hierarchical convolutional network stylizes images at increasing scales with multiple losses and uses both color and luminance representations.
Results
The network better captures both coarse and intricate texture patterns and generates high-resolution stylizations with speed and memory usage close to a shallower comparison network.
Takeaways & Limitations
Multimodal hierarchical transfer generates more visually appealing high-resolution results while handling style and texture cues at multiple scales.
Takeaways & Limitations
The authors plan to investigate lower-memory loss networks to extend the scheme to much larger images.
Abstract
from arXiv · showhide
Transferring artistic styles onto everyday photographs has become an extremely popular task in both academia and industry. Recently, offline training has replaced on-line iterative optimization, enabling nearly real-time stylization. When those stylization networks are applied directly to high-resolution images, however, the style of localized regions often appears less similar to the desired artistic style. This is because the transfer process fails to capture small, intricate textures and maintain correct texture scales of the artworks. Here we propose a multimodal convolutional neural network that takes into consideration faithful representations of both color and luminance channels, and performs stylization hierarchically with multiple losses of increasing scales. Compared to state-of-the-art networks, our network can also perform style transfer in nearly real-time by conducting much more sophisticated training offline. By properly handling style and texture cues at multiple scales using several modalities, we can transfer not just large-scale, obvious style cues but also subtle, exquisite ones. That is, our scheme can generate results that are visually pleasing and more similar to multiple desired artistic styles with color and texture cues at multiple scales.
1. Introduction
Fast style transfer replaces slow iterative optimization with offline-trained feed-forward inference, but high-resolution stylization can lose texture scale and intricate brushwork. The proposed multimodal hierarchical network addresses these issues using multiple scales and color-luminance representations.
- Motivation: Offline-trained feed-forward networks reduce style-transfer computation by requiring only one inference pass at application time.They are reported to be hundreds of times faster than optimization-based methods.
- Problem: High-resolution stylization exposes texture-scale mismatch and loss of small, intricate textures such as brushwork.Models trained at one style-image resolution may produce textures that are too small when applied to larger images.
- Approach: The proposed hierarchical network learns coarse texture distortion and fine brushwork by using multiple scales of a style image.Its training scheme also combines multiple models into one network for increasingly larger image sizes.
- Approach: The network represents both color and luminance channels rather than relying only on RGB color channels.This multimodal design is part of the proposed approach to capturing style cues at multiple scales.
- Result: Multimodal transfer captures multiple levels of texture, including style, color, large texture distortion, and fine brushwork.The authors report that it simulates the artwork’s brushwork more closely than the compared singular-transfer networks.
2. Related Work
Related work established optimization-based neural style transfer and later accelerated it with feed-forward networks. Subsequent methods improved feed-forward stylization, while this paper extends that line toward progressively larger image sizes.
- Deep-network representations: Deep-network analysis methods have visualized responsible image patches, internal-neuron computations, and information preserved in layer representations.Some of these methods use optimization to generate visualization images for perceptual understanding.
- Optimization-based style transfer: Gatys et al. synthesized textures by computing an objective loss from representations of a pretrained convolutional neural network.This work differed from earlier non-parametric texture-synthesis algorithms.
- Feed-forward generation: Optimization-based image-generation methods are computationally expensive because they rely on iterative optimization.Feed-forward methods instead use perceptual objectives computed from a neural network to synthesize images.
- Fast style transfer: Johnson et al. introduced a feed-forward network for fast style transfer and super-resolution using perceptual losses from Gatys et al.This established a faster alternative to optimization-based style transfer.
- Fast style transfer: Instance normalization was reported to improve the quality of generated images for fast style transfer.This work followed earlier feed-forward style-transfer architectures while targeting progressively larger image sizes.
3. Multimodal Transfer Network
The multimodal transfer network combines hierarchical subnetworks, multi-scale stylization losses, and color–luminance representations to transfer coarse and fine artistic cues while maintaining texture scale.
- Overall architecture: The MT Network is a hierarchical residual CNN with style, enhance, and refine subnetworks that generate increasingly larger stylized outputs.Each subnet is parameterized separately, and the network produces multiple outputs at increasing sizes.
- Hierarchical losses: Each output is evaluated separately by a loss network, and the total objective combines the resulting stylization losses across scales.The loss network computes a stylization loss for every output, while the hierarchical loss is a weighted sum of these losses.
- Hierarchical processing: At test time, the image is stylized at 256, enhanced at 512, and refined at 1024 to preserve large-scale traits and correct texture scale.The style subnet captures large color and texture traits, the enhance subnet strengthens stylization, and the refine subnet removes local pixelization artifacts.
- Stylization losses: The stylization loss combines content and texture losses, whose feature representations measure spatial content similarity and texture statistics, respectively.Content loss compares corresponding feature maps, while Gram-matrix texture representations discard spatial information but retain color and intensity statistics.
- Hierarchical training: Parallel back-propagation assigns losses to different layer ranges, while later subnetworks also refine earlier outputs and can use shallower CNN structures.This coupling allows former subnetworks to contribute to later losses while reducing memory and runtime requirements for later subnetworks.
- Multi-scale and multimodal cues: Different style-image scales train different subnetworks to learn distinct artistic texture levels, addressing singular-transfer limits that trade coarse texture against fine brushwork.The network also uses separate RGB and luminance branches because visual perception is more sensitive to luminance changes than color changes.
4. Experiments
Experiments show that multimodal transfer captures artistic styles across coarse and fine scales, including when stylizing high-resolution images and combining multiple styles. It maintains near-comparable speed and memory use while improving visual similarity to the target artworks.
- Singular transfer comparison: The style subnet produced qualitatively comparable results and better captured texture details than competing singular transfer networks in some examples.It improved color and texture-scale matching in one example and simulated small, detailed texture better in another.
- High-resolution stylization: On 1024×1024 outputs, multimodal transfer matched both coarse texture structure and fine brushwork more closely than singular transfer.Singular transfer trained at 256 produced smaller-than-target textures, while the 1024 model failed to learn distortion and brushwork.
- Multiple styles: Multimodal transfer fused coarse texture from one style with fine brushwork from another while preserving the content of a separate test image.The two-style demonstration combined Picasso’s large texture distortion with Tomas King’s small, detailed brushwork.
- Processing speed and memory use: MT Net generated 1024×1024 images in 0.54s using 3100 MB, compared with Johnson Net’s 0.42s and 2400 MB.Despite being more than twice as deep as Johnson Net, MT Net had close speed and memory usage because most computation occurred at low resolutions.
5. Conclusion
The paper concludes that multimodal transfer learns color, coarse texture, and fine brushwork at multiple scales while addressing texture-scale mismatch in high-resolution stylization. Future work will examine alternative losses and lower-memory loss networks for larger images.
- 5. Conclusion: The hierarchical training scheme learns artistic style cues at multiple scales, including color, coarse texture structure, and fine brushwork.The conclusion identifies these multimodal, multiscale cues as the basis of the proposed fast style-transfer scheme.
- 5. Conclusion: The scheme addresses texture-scale mismatch and produces more visually appealing stylized results on high-resolution images.The stated conclusion concerns the method’s high-resolution stylization behavior.
- 5. Conclusion: Future work will investigate losses that better capture artistic style at different scales and lower-memory loss networks for much larger images.These are the authors’ stated directions for extending the scheme.