Source-linked AI summary
ReImagine: Rethinking Controllable High-Quality Human Video Generation via Image-First Synthesis
Zhengwentai Sun, Keru Zheng, Chenghong Li, Hongjie Liao, Xihe Yang, Heyuan Li, Yihao Zhi, Shuliang Ning, Shuguang Cui, Xiaoguang Han
TL;DR
Human video generation must jointly control appearance, motion, and viewpoint despite limited high-quality multi-view data. ReImagine decouples appearance learning from temporal consistency through pose- and viewpoint-guided image synthesis followed by training-free video refinement. The method produces high-quality, temporally coherent videos and releases canonical human-generation resources.
Problem
Joint control of human appearance, pose, and camera viewpoint remains limited because high-quality multi-view video data are scarce and existing methods often model these factors separately.
Method
ReImagine combines a pretrained image backbone with SMPL-X motion guidance for frame synthesis and a pretrained video diffusion model for training-free temporal refinement.
Results
The method produces higher visual fidelity while maintaining temporal coherence compared with video-first training, with improvements across spatial and temporal metrics.
Takeaways & Limitations
An image-first formulation can synthesize temporally coherent, pose- and viewpoint-controllable human videos without high-quality video data or video-specific training.
Takeaways & Limitations
The image generation module can still produce minor high-frequency artifacts, such as jitter around details.
Abstract
from arXiv · showhide
Human video generation remains challenging due to the difficulty of jointly modeling human appearance, motion, and camera viewpoint under limited multi-view data. Existing methods often address these factors separately, resulting in limited controllability or reduced visual quality. We revisit this problem from an image-first perspective, where high-quality human appearance is learned via image generation and used as a prior for video synthesis, decoupling appearance modeling from temporal consistency. We propose a pose- and viewpoint-controllable pipeline that combines a pretrained image backbone with SMPL-X-based motion guidance, together with a training-free temporal refinement stage based on a pretrained video diffusion model. Our method produces high-quality, temporally consistent videos under diverse poses and viewpoints. We also release a canonical human dataset and an auxiliary model for compositional human image synthesis. Code and data are publicly available at https://github.com/Taited/ReImagine.
1 Introduction
ReImagine addresses the difficulty of jointly controlling human appearance, pose, and camera viewpoint under scarce high-quality multi-view video data through an image-first design. It combines pose- and viewpoint-guided image synthesis with training-free temporal refinement and adds canonical-data resources for compositional human generation.
- Motivation: Jointly controlling human appearance, body pose, and camera viewpoint remains difficult because high-quality multi-view video data are scarce.Existing approaches often handle these factors separately, limiting joint modeling and creating a trade-off between controllability and visual fidelity.
- Method: ReImagine first learns high-quality human appearance from image generation, then uses a pretrained video diffusion model for training-free temporal consistency.This decouples appearance modeling from temporal dynamics and avoids task-specific video training under limited multi-view data.
- Additional resources: The authors construct a paired canonical dataset and an end-to-end model for generating full-body human images from disentangled face, clothing, footwear, and SMPL-X inputs.These resources support compositional and attribute-consistent human synthesis and are publicly released with the code and dataset.
- Method: The pipeline generates videos from canonical front–back images and SMPL-X motion sequences with explicit pose and viewpoint control.Frame generation uses rendered SMPL-X normal maps for pose and viewpoint guidance before temporal refinement.
2 Related Works
Prior work has progressed from GAN-based human image synthesis to diffusion models with controllable pose, identity, garments, and video motion. The paper positions its pose- and view-guided DiT module within this evolution by incorporating 3D human geometry and canonical appearance cues.
- Image-based human generation: Early human image-generation work used GANs for photorealistic unconditional synthesis and broader appearance, pose, and garment-transfer tasks.StyleGAN-Human focused on unconditional human images, while HumanGAN supported global appearance sampling, pose transfer, and part or garment transfer.
- Image-based human generation: Diffusion models shifted controlled human generation toward preserving attributes such as pose and identity through injected structural conditions.ControlNet, HumanSD, and HyperHuman exemplify skeletal or structure-guided conditioning for more controllable synthesis.
- Video-based human generation: Human-centric video generation extends latent diffusion with temporal attention or 3D human guidance to produce motion-controlled human videos.AnimateAnyone and related methods add temporal modeling, while Champ incorporates a 3D human model into latent diffusion.
- ReImagine: ReImagine’s generation module unifies SMPL-X pose parameters and canonical front–back appearance cues in a shared token sequence processed by a DiT backbone.The generated latent is decoded into the final image using a VAE decoder, with condition-aware RoPE used in the DiT processing.
3 Method
ReImagine decomposes controllable human video synthesis into pose- and view-guided image generation followed by training-free temporal refinement. It combines SMPL-X geometry and canonical front–back appearance cues with token-based diffusion conditioning, then suppresses frame-level artifacts during inference.
- Overview: ReImagine first synthesizes pose- and view-consistent human image frames, then refines them into temporally coherent videos.The pipeline separates image generation from temporal consistency processing.
- Pose- and View-Guided Image Synthesis: World-coordinate SMPL-X normal maps encode articulated pose and view-dependent orientation for joint pose–view control.The normal map is injected through a ControlNet-style encoder as multi-scale pose features.
- Pose- and View-Guided Image Synthesis: Canonical front-view and back-view images provide appearance tokens that reduce view ambiguity and support large viewpoint changes.The two images are encoded by a shared VAE into front- and back-view token sequences.
- Pose- and View-Guided Image Synthesis: Condition-aware positional encoding separates pose, appearance, and noise tokens while preserving spatial structure in the DiT backbone.Image and noise tokens retain grid coordinates, whereas pose tokens receive fixed pseudo-spatial coordinates.
- Training-Free Temporal Consistency: The image generation stage can still produce minor high-frequency artifacts, including jitter around clothing wrinkles or motion boundaries.The temporal refinement stage is introduced to address these artifacts without additional training.
- Training-Free Temporal Consistency: The training-free temporal module combines low-noise re-denoising with dynamic spatiotemporal regularization to correct high-frequency inconsistencies while preserving global structure.Filtering is applied during the first 35% of inference steps, and the first-frame latent remains unchanged as a temporal anchor.
4 Experiments
Experiments evaluate ReImagine on MVHumanNet++ and DNA-Rendering against image-editing and video-generation baselines, using quantitative, qualitative, and temporal-consistency analyses. ReImagine remains more stable under larger pose changes and challenging viewpoints, improves temporal consistency, and outperforms a matched video-first baseline.
- Datasets and Baselines: ReImagine is trained on 5,000 MVHumanNet++ subjects with four views and evaluated zero-shot on 15 DNA-Rendering subjects.The training views are near front, back, left, and right; DNA-Rendering is not used for training.
- Datasets and Baselines: The evaluation compares ReImagine with Qwen-Image-Edit, Wan-Animate, Wan-Fun-Control, and Human4DiT under shared appearance and pose-conditioning protocols.Metrics cover spatial fidelity, perceptual realism, and temporal coherence through PSNR, SSIM, LPIPS, FID, and FVD.
- Qualitative Results: Under larger pose changes and challenging viewpoints, ReImagine remains more stable and produces more consistent appearances than the compared baselines.The qualitative comparisons cover MVHumanNet++ and zero-shot DNA-Rendering evaluation.
- Quantitative Results: 0.275 FVD on MVHumanNet versus 0.403 for Wan-Animate demonstrates ReImagine's stronger temporal consistency.Qwen-Image-Edit reaches SSIM 0.831 but has FVD 1.442, illustrating the difference between individual-frame quality and temporal coherence.
- Ablation Study: Re-denoising with 3DFFT-based spatiotemporal regularization produces the smoothest and most coherent tracking trajectories.Without refinement, trajectories fragment and jitter; median filtering oversmooths and drifts, while removing the Wan backbone harms temporal continuity and motion.
- Ablation Study: Front-view input alone yields plausible unseen appearance, but back-facing controllability degrades when front-view elements propagate to the back.The result highlights the importance of explicit front–back appearance conditioning for accurate view-dependent generation.
- Video-First Comparison: Against a matched video-first baseline, ReImagine improves all reported metrics and produces sharper textures, more faithful colors, and clearer facial details.Both methods use the same inputs and MVHumanNet++ training data, differing in generation strategy.
5 User Study
A user study compares ReImagine with four baselines using matched appearance references and comparable pose sequences, with preferences aggregated from pairwise judgments.
- User Study: Participants perform pairwise comparisons of randomly ordered human videos generated with the same appearance reference and comparable pose sequences.The compared methods are Qwen, Wan-Animate, Wan-Fun, and Human4DiT.
6 End-to-End Extension with Canonical Asset Data
The extension constructs a canonical asset dataset and uses disentangled face, clothing, and shoe components for end-to-end human image synthesis. It preserves the original image-generation module and training procedure while replacing full-body inputs with component assets.
- Extension: The canonical asset dataset enables end-to-end human image synthesis from disentangled face, clothing, and shoe components.It relaxes the framework's input requirements through compositional assets.
- Data Construction: The dataset construction extracts canonical front- and back-view frames, standardizes illumination, and segments faces, garments, and shoes into approximately 1,600 identities.The pipeline applies super-resolution, relighting, detection, segmentation, canonicalization, augmentation, and manual filtering.
- End-to-End Training: End-to-end training injects face, clothing, and shoe assets with condition-aware positional encoding while keeping the remaining architecture and training procedure unchanged.The extension reuses the image-generation module described earlier.
7 Conclusion
The conclusion presents ReImagine as an image-first human video generation approach that combines pretrained image synthesis with SMPL-X pose guidance and temporal refinement. It emphasizes temporally coherent videos without high-quality video data or video-specific training, alongside released canonical-data resources.
- Conclusion: ReImagine treats high-quality pose- and viewpoint-conditioned human image generation as the primary challenge in human video generation.The approach uses pretrained image synthesis backbones and structured SMPL-X pose guidance.
- Conclusion: The method synthesizes temporally coherent human videos without requiring high-quality video data or video-specific training.The conclusion attributes this to pretrained image synthesis backbones and structured pose guidance.
- Conclusion: The paper releases a canonical dataset and an end-to-end variant enabled by synthetic canonical data for future human-generation research.
Supplementary Material
The supplementary material provides additional discussion, experimental details, ablation studies, and qualitative results that complement the main paper.
- The supplement adds further discussion beyond the main paper.
- It includes experimental details and ablation studies.
- It provides qualitative results complementing the main paper.
A Extended Temporal Consistency Evaluation
The extended evaluation compares temporal strategies using spatial quality and temporal stability metrics. The proposed 3D FFT refinement preserves competitive image quality while improving temporal coherence and reducing flow-based error.
- Several temporal strategies are evaluated using spatial quality and temporal stability metrics.
- The image-first pipeline without temporal processing achieves the best per-frame fidelity, while refinement slightly lowers spatial metrics.The decrease is attributed to temporal smoothing suppressing high-frequency details.
- The proposed 3D FFT approach maintains competitive image quality while improving temporal coherence.
- The proposed 3D FFT module significantly reduces optical-flow reconstruction error, indicating improved frame-to-frame consistency.
- Temporal error is measured by averaging the L1 difference between a warped frame estimate and the subsequent frame.Optical flow between consecutive frames guides differentiable warping.
B Additional Qualitative Results
Additional qualitative results show that the framework maintains identity and visual consistency across diverse poses and substantial viewpoint changes. The results also demonstrate pose and viewpoint control from canonical or in-the-wild appearance inputs.
- The model preserves identity and clothing details from a single in-the-wild reference across diverse poses and viewpoints.
- Under large viewpoint changes, the model synthesizes consistent human images as the camera rotates around the subject.
- The qualitative results demonstrate control over both pose and viewpoint.
- Supplementary videos provide additional qualitative results and full sequences for observing temporal behavior and viewpoint transitions.
C Additional Applications with Canonical Asset Training
Canonical asset-based training supports compositional human synthesis beyond standard controllable video generation. Disentangled assets enable appearance recomposition and pose transfer under arbitrary viewpoints, though the more flexible interface slightly lowers quantitative performance.
- Canonical asset-based training enables coherent human synthesis from disentangled face, clothing, and shoe inputs with pose conditioning.Condition-aware positional encoding associates each component with its spatial and semantic role during training.
- The disentangled setting provides a more flexible, user-friendly interface but introduces a more challenging generation task.
- The more challenging disentangled setting leads to slightly lower quantitative performance.
- The disentangled interface allows different identities and clothing assets to be combined into novel character appearances.
- The model supports pose transfer under arbitrary viewpoints while maintaining appearance consistency.