Source-linked AI summary
Deep Image Spatial Transformation for Person Image Generation
Yurui Ren, Xiaoming Yu, Junming Chen, Thomas H. Li, Ge Li
TL;DR
Pose-guided person image generation must spatially transform source images while preserving appearance, a challenge for CNNs that cannot readily rearrange inputs. The paper introduces a global-flow local-attention framework that reassembles source features, and reports strong results on person generation with extensions to view synthesis and video animation.
Problem
Pose-guided person image generation requires spatially transforming source images while retaining appearance details, but CNNs have limited ability to spatially rearrange input data.
Method
The model predicts global flow fields from source-target correlations, extracts flowed local feature patches, and warps source features using content-aware local attention.
Results
The model generates target images with correct poses and vivid details, while experiments report superior subjective and objective performance and applicability to view synthesis and video animation.
Takeaways & Limitations
Feature-level global-flow local-attention reassembly supports pose-guided person generation and can be extended to spatial deformation tasks including view synthesis and video animation.
Abstract
from arXiv · showhide
Pose-guided person image generation is to transform a source person image to a target pose. This task requires spatial manipulations of source data. However, Convolutional Neural Networks are limited by the lack of ability to spatially transform the inputs. In this paper, we propose a differentiable global-flow local-attention framework to reassemble the inputs at the feature level. Specifically, our model first calculates the global correlations between sources and targets to predict flow fields. Then, the flowed local patch pairs are extracted from the feature maps to calculate the local attention coefficients. Finally, we warp the source features using a content-aware sampling method with the obtained local attention coefficients. The results of both subjective and objective experiments demonstrate the superiority of our model. Besides, additional results in video animation and view synthesis show that our model is applicable to other tasks requiring spatial transformation. Our source code is available at https://github.com/RenYurui/Global-Flow-Local-Attention.
1 . Introduction
Pose-guided person image generation requires spatially transforming source information while preserving appearance details, but CNNs lack mechanisms for rearranging inputs. The proposed global-flow local-attention framework addresses this limitation at the feature level and extends to other spatial deformation tasks.
- Pose-guided person image generation transforms a source person image from a source pose to a target pose while retaining appearance details.
- CNN equivariance supports aligned tasks but limits networks’ ability to spatially rearrange input data.
- Pixel-level flow methods restrict new-content generation and make large motions difficult because they require full-resolution flow fields.
- The proposed framework combines global flow estimation with local attention to relate each output location to a local source feature patch.
- Feature-level warping can generate new contents, reduce flow-estimation difficulty, and support view synthesis and video animation.
3 . Our Approach
The approach estimates global source-target motion and occlusions, then uses local feature patches and content-aware sampling to reassemble source features at target locations. Flow supervision and local affine regularization constrain the learned transformation.
- Local Neural Texture Renderer: The Local Neural Texture Renderer samples and reassembles source features according to the estimated flow fields.
- Global Flow Field Estimator: The Global Flow Field Estimator takes source image and structure guidance for source and target, producing flow fields and occlusion masks without flow labels.
- Global Flow Field Estimator: Sampling correctness loss constrains flow fields to sample semantically similar regions using warped and target VGG features.
- Global Flow Field Estimator: The flow regularizer penalizes local regions whose transformation departs from an affine transformation.
3 .2. Local Neural Texture Renderer
The Local Neural Texture Renderer transforms source features into target images by combining flowed local attention with content-aware sampling and occlusion-aware feature selection.
- The renderer takes source images, target poses, flow fields, and occlusion masks to generate the result image.
- It extracts target and flow-shifted source feature patches, then predicts local kernels from their concatenation for content-aware attention.The source patch is bilinearly sampled when flow-shifted coordinates are non-integer.
- The predicted kernels are normalized with softmax and applied over each extracted source patch to construct the warped feature map.The warped feature map is produced by repeating this operation at every spatial location.
- An occlusion mask with values from 0 to 1 selects between attended source features and target features, enabling generation of contents absent from the source.
- Training uses a joint reconstruction ℓ1, adversarial, perceptual, and style loss.The adversarial loss uses a discriminator, while perceptual and style losses use activation maps from a pretrained network.
4 .1. Implementation Details
Experiments use Market-1501 and DeepFashion, with quantitative and subjective evaluation alongside qualitative comparisons across several state-of-the-art methods.
- The experiments use Market-1501 and DeepFashion, whose identities are separated between training and testing sets.Market-1501 contains 32668 low-resolution images, while DeepFashion contains 52712 high-quality model images.
- Qualitative comparisons include Def-GAN, VU-Net, Pose-Attn, and Intr-Flow on Fashion and Market-1501 datasets.
- The implementation uses auto-encoder structures with residual blocks and trains on 256 × 256 Fashion images with two local attention blocks.
4 .2. Comparisons
The model achieves competitive objective and subjective results across Market-1501 and Fashion, while producing detailed transformations with fewer parameters than competing methods.
- The model achieves competitive results on both Market-1501 and DeepFashion, generating realistic images with fewer perceptual reconstruction errors.
- The model achieves the best subjective result on Fashion and competitive results on Market-1501 in the JND evaluation.The test compares real and generated images over 800 images per model and dataset, with five comparisons per image.
- Competitors struggle with complex textures, predefined affine transformations, or side-pose synthesis, whereas the model preserves vivid clothing details.The model regresses flow fields without supervised ground-truth flow and handles global patterns and details such as lace and shoelaces.
- Table 2 reports evaluation results for the ablation study.
- The model achieves high performance with less than half the parameters of competing methods.The authors attribute this efficiency to attention blocks that avoid requiring many convolution layers.
4 .3. Ablation Study
The ablation study compares baseline, global-attention, bilinear-sampling, and full models to evaluate sparse flow-based attention and content-aware sampling.
- The ablation models include an auto-encoder baseline, a global-attention variant, a bilinear-sampling variant, and the proposed full model.The variants isolate global versus local attention and bilinear versus content-aware sampling.
- Figure 5 presents qualitative ablation results.
- Figure 6 visualizes attention coefficients, with red rectangles marking target locations and blue indicating low weights.
- Global attention degrades performance relative to the baseline, while flow-based Bi-Sample and Full models improve performance by enforcing sparse attention.
- The full model gains further performance from content-aware sampling with an adjustable receptive field, unlike Bi-Sample’s predefined limited receptive field.
- Baseline and Global-Attn preserve structure but lose source textures, whereas flow-based methods reconstruct more vivid details.
- Bi-Sample introduces artifacts because bilinear sampling cannot find exact sampling locations, while the full model flexibly adjusts sampling weights.
5 . Application on Other Tasks
The global-flow local-attention module extends beyond person image generation to view synthesis and video animation using images and structure guidance. These applications demonstrate its versatility for spatial transformation tasks.
- The module can be applied flexibly to tasks requiring spatial transformation without additional information beyond images and structure guidance.
- View Synthesis: For view synthesis, the model generates novel views from input views whose appearances are highly correlated.
- Image Animation: For image animation, edge maps provide structure guidance while the source frame and previous generated frames serve as references.
- Image Animation: On 1000 news-briefing videos, the model generates realistic animation results with vivid movements.
- The paper reports that the model can generate target-pose images while maintaining vivid details and extend to view synthesis and video animation.
A . Warping at the Feature Level
Feature-level warping addresses limitations of pixel-level flow operations but can suffer unstable gradients and bad local minima. The paper analyzes these issues through bilinear sampling and flow-field behavior.
- Feature-level warping can generate new contents and better capture large motions, but flow-based networks may become stuck in bad local minima.
- Bilinear sampling assigns each output location a source sampling position through flow-field offsets and interpolates the corresponding input feature.
- The traditional operation computes input-feature gradients according to the sampling offsets, with analogous expressions for other gradient terms.
- When input features are meaningless, gradients based on adjacent-feature differences cannot provide correct flow fields.
- Strong correlations between adjacent features can make flow-field gradients small, limiting stable propagation and large-motion capture.
B . Analysis of the Regularization loss
The regularization loss models local deformations as affine transformations to improve flow-field sampling. Ablation results show smoother fields and fewer incorrect sampling regions, alongside qualitative comparisons on two datasets.
- The regularization loss assumes that local regions such as arms and clothes undergo deformations that can be modeled by affine transformations.
- Using the regularization loss produces smoother flow fields and avoids unnecessary jumps compared with training without the term.
- Without regularization, directly warped source images reveal incorrect sampling regions; the regularized fields improve sampling correctness.
- Qualitative comparisons include Def-GAN, VU-Net, Pose-Attn, and Intr-Flow on DeepFashion and Market-1501.
C .2. Additional Results of View Synthesis
Additional qualitative results examine view synthesis and image animation, while the appendix documents the network components, architecture, and training configuration.
- Additional Results of View Synthesis: Additional view-synthesis results compare Appearance Flow, the proposed model, and ground-truth images for each group.
- Additional Results of Image Animation: Additional image-animation results present a source image followed by generated images for each row.
- The network uses an auto-encoder structure with residual blocks, instance normalization, Leaky-ReLU activations, and spectral normalization in the discriminator.
- Training first fits the Flow Field Estimator and then trains the whole model end-to-end with ADAM.