Source-linked AI summary
From Frames to Sequences: Temporally Consistent Human-Centric Dense Prediction
Xingyu Miao, Junting Dong, Qin Zhao, Yuhang Yang, Junhao Chen, Yang Long
TL;DR
Temporally consistent human-centric dense prediction is limited by flickering video outputs and scarce paired supervision for multiple dense tasks. The paper combines synthetic frame and sequence data with a unified ViT-based model using human geometry priors and channel reweighting, achieving strong benchmark performance and in-the-wild generalization. Its main scope boundary is that complex textures and rapid large motions can still produce pseudo-geometric artifacts and weaken temporal correspondences.
Problem
Existing dense predictors often flicker under motion, occlusion, and lighting changes, while paired human video supervision for multiple dense tasks is scarce.
Method
The method combines synthetic static and motion-aligned sequence data with a unified ViT predictor using CSE human geometry embeddings and channel reweighting.
Results
The method achieves strong results on THuman2.1 and Hi4D and generalizes to in-the-wild videos.
Takeaways & Limitations
Large-scale synthetic data, temporal supervision, and human priors can effectively support human-centric video perception with stable multi-task predictions.
Takeaways & Limitations
Highly complex textures can still introduce pseudo-geometric artifacts, while rapid large movements weaken temporal correspondences and cause local instability.
Abstract
from arXiv · showhide
In this work, we focus on the challenge of temporally consistent human-centric dense prediction across video sequences. Existing models achieve strong per-frame accuracy but often flicker under motion, occlusion, and lighting changes, and they rarely have paired human video supervision for multiple dense tasks. We address this gap with a scalable synthetic data pipeline that generates photorealistic human frames and motion-aligned sequences with pixel-accurate depth, normals, and masks. Unlike prior static data synthetic pipelines, our pipeline provides both frame-level labels for spatial learning and sequence-level supervision for temporal learning. Building on this, we train a unified ViT-based dense predictor that (i) injects an explicit human geometric prior via CSE embeddings and (ii) improves geometry-feature reliability with a lightweight channel reweighting module after feature fusion. Our two-stage training strategy, combining static pretraining with dynamic sequence supervision, enables the model first to acquire robust spatial representations and then refine temporal consistency across motion-aligned sequences. Extensive experiments show that we achieve state-of-the-art performance on THuman2.1 and Hi4D and generalize effectively to in-the-wild videos.
1 Introduction
Human-centric dense prediction in unconstrained videos remains difficult because existing methods often lack paired video annotations and explicit temporal constraints. The work addresses this gap through scalable human-centric data synthesis and a unified architecture for temporally consistent multi-task prediction.
- Paired human video data for dense depth, surface-normal, and segmentation supervision remains scarce.
- Per-frame models can flicker or produce abrupt discontinuities under motion, occlusion, and lighting changes.
- The synthesis pipeline produces photorealistic frames and motion-aligned sequences with masks, depth, and surface normals.
- A unified model uses human-centric CSE priors to predict segmentation, depth, and surface normals without task-specific fine-tuning.
- The method achieves state-of-the-art depth and surface-normal results on THuman2.1 and Hi4D and transfers well to in-the-wild videos.
2 Related Work
Human-centric dense prediction builds on advances in annotation, representation learning, and transformer-based architectures. Existing work spans manually assisted or reconstructed datasets, pose and part understanding, and scalable pretrained backbones for dense tasks.
- Dense human depth and surface-normal datasets remain difficult to annotate manually, motivating multi-view capture and reconstruction.
- Human-centric vision expanded from keypoint and body-part segmentation methods to dense depth and surface-normal prediction.
- ViT encoders with lightweight convolutional decoders provide fine-grained and globally consistent dense predictions.
- Figure 1 presents synthetic RGB data with ground-truth depth, surface-normal, and segmentation-mask annotations.
- Large-scale self-supervised pretraining and pseudo-labeled real-image collections improve dense prediction transfer and accuracy.
3 Methodology
The method combines synthetic human-centric data generation with a ViT-based architecture and two-stage training to predict temporally consistent depth, normals, and segmentation.
- Human-centric Synthetic Data Pipeline: The pipeline composes diverse clothed human models and renders RGB images, depth, normals, and masks for static and dynamic data.Dynamic sequences use AMASS motion capture and randomized camera settings.
- Model Architecture: The model combines DINO and DPT features with lightweight task heads for depth, surface normals, and foreground segmentation.The encoder extracts global representations, while the decoder produces multi-scale features for the three predictions.
- Model Architecture: A CNN branch adds edge and texture cues, while channel reweighting suppresses appearance-dominated signals and emphasizes geometry-related channels.The fused representation concatenates decoder and CNN features before channel-wise adaptation.
- Model Architecture: CSE embeddings inject human geometric priors by projection and element-wise addition with decoder features.The embeddings provide shape-aware constraints for human-centric dense prediction.
- Training Strategy: Two-stage training first learns spatial representations from synthetic images, then uses synthetic videos and flow-guided stabilization to learn temporal consistency.The spatial stage retains depth, normal, and segmentation objectives before temporal refinement.
- Static Image Model Training: The static objectives include depth, normal, and segmentation losses, with gradient regularization encouraging sharp boundaries and local continuity.The normal formulation combines L1 and cosine terms, while the gradient term supports sharper depth structure.
4 Experiments
Experiments evaluate the method on real-world human datasets for static and video dense prediction, with ablations examining human geometry priors and channel adaptation. Results show strong performance across depth, normals, segmentation, and temporal consistency settings.
- Evaluation setup: The evaluation uses THuman2.1 and Hi4D for depth and surface normal estimation, alongside P3M benchmarks for soft foreground segmentation.
- Video consistency: Temporal evaluation includes qualitative comparisons of video depth and surface normals, with a flow-based TC-Abs metric measuring discrepancies in temporal surface-orientation changes.
- Static prediction: Static depth results show both model variants outperforming general-purpose and human-centric baselines on THuman2.1 and Hi4D.
- Static prediction: The Large model matches or exceeds larger Sapiens models on Hi4D static depth, indicating parameter efficiency and cross-dataset generalization.
- Segmentation: SAD decreases from 78.17 to 70.71 and Conn from 74.72 to 68.32 on PPM-100, while results remain competitive on P3M validation sets.
- Ablation studies: The ablation study finds that CSE human priors strengthen body-aligned geometry and local orientation details, whereas CWA improves prediction stability.
5 Conclusion
The work presents a human-centric dense prediction framework that combines synthetic static and dynamic supervision for temporally consistent video predictions. It reports strong results on THuman2.1 and Hi4D, generalization to in-the-wild videos, and future extensions to complex scenes and 3D reconstruction.
- The framework jointly learns segmentation, depth, and surface normals from pixel-accurate static and dynamic synthetic annotations.
- Experiments show strong results on THuman2.1 and Hi4D and effective generalization to in-the-wild videos.
- Future work will extend the framework to more complex scenes and downstream tasks such as human 3D reconstruction.
A.1 Data synthesis pipeline
The data synthesis pipeline creates diverse clothed human identities, assigns realistic motion, and renders synchronized supervision from randomized Blender cameras. It produces RGB frames with depth, normals, and segmentation masks for static and dynamic training.
- Clothed human models are built with DAZ 3D, MakeHuman, and Character Creator, then diversified through texture augmentation.
- AMASS skeletal trajectories are retargeted to the identities, assigning realistic motion while preserving consistent rigging across characters.
- Animated models are placed in Blender with randomized camera poses, focal lengths, and tracking behavior to increase viewpoint diversity.
- Rendering produces synchronized RGB images, depth maps, surface normals, and segmentation masks for both static and dynamic data.
A.2 Discussion
Prior human-centric synthetic pipelines provide useful supervision but often emphasize static images or simplified human appearance. The discussion highlights the need for more realistic data supporting dense video learning.
- Existing pipelines primarily focus on static image generation or structural parameter supervision rather than comprehensive realistic human video annotations.
- SURREAL provides synthetic videos with dense annotations but lacks realistic clothing and hair geometry because it relies on simplified texture mappings.
A.3 Additional qualitative results
The paper presents additional qualitative results in Figures 8 and 9, with video results available in the supplementary materials.
- Figures 8 and 9 contain additional qualitative results.
- The supplementary materials include video results.
- The section supplements the paper with visual and video-based examples.
A.4 Additional ablation study
The ablations show that local texture modeling, human-structure priors, balanced multi-task supervision, data scaling, geometric regularization, and temporal modeling each affect prediction quality and stability. Channel reweighting is used to reduce texture-induced geometric artifacts, while flow-based temporal supervision improves consistency across motion and occlusion.
- Channel weight adaptation: CNN features improve local continuity and edge detail but can introduce texture-related artifacts from shadows, clothing patterns, and tattoos.CWA suppresses channels weakly related to geometry and emphasizes shape-correlated features.
- Backbone and human priors: DINOv3 improves consistently over DINOv2 across benchmarks, while the paper attributes broader gains to the model design and curated training data together.The strongest stability differences are reported on Thuman-FullBody and Hi4D.
- Backbone and human priors: Human-structure priors improve depth, and additive fusion outperforms concatenation for depth and surface-normal estimation.Addition is described as directly aligning prior and learned features rather than requiring fusion of heterogeneous representations.
- Multi-task loss and data scaling: The best multi-task weighting uses λd=1 and λn=0.1, allowing depth to remain primary while lightly weighted normal supervision provides complementary regularization.Equal weighting performs weakest, while reducing the normal weight to 0.5 harms normal accuracy.
- Multi-task loss and data scaling: Scaling training data from 300K to 2M improves depth and normal prediction, but gains become marginal between 600K and 2M.The results suggest that further progress may require stronger architectures or greater data diversity.
- Geometric and temporal regularization: Normal regularization improves surface-normal prediction without materially changing depth metrics, providing explicit local-orientation guidance.The ablation reports a significant normal improvement while depth metrics remain nearly unchanged.
- Geometric and temporal regularization: Flow-based temporal supervision improves stability because it establishes cross-frame correspondences and extends supervision from depth to directional normal consistency.Gradient-based temporal loss improves depth consistency but remains limited for fast motion, occlusion, and normal estimation.
- Geometric and temporal regularization: Removing the temporal layer increases depth TC-RMSE from 0.0189 to 0.0276 and normal TC-Mean from 3.27 to 4.55.The degradation is reported especially for frames with fast motion or occlusion.
A.5 Model parameters comparison
The paper compares representative human-centric models by parameter count and computational cost, and reports the foreground segmentation branch as an auxiliary component evaluated with DINOv3-B.
- A.5 Model parameters comparison: Table 17 compares representative human-centric methods using parameter counts and GFLOPs to characterize relative model complexity.The comparison covers the proposed models alongside existing state-of-the-art approaches.
- A.5 Model parameters comparison: The foreground segmentation ablation is reported with DINOv3-B on Hi4D and THuman2.1.The branch improves depth and normal estimation on both datasets.
A.6 Implementation details
Implementation uses separate image and video training stages, standard depth and normal metrics, and temporal metrics based on adjacent-frame flow and warping. The approach remains challenged by complex textures, large rapid movements, occlusions, and non-rigid deformations.
- Training: Both training stages use AdamW with weight decay 0.05, β1=0.9, and β2=0.95, followed by warmup and polynomial learning-rate decay.The schedule uses a 2000-iteration linear warmup and polynomial decay with power 1.5.
- Training: Stage 1 uses image training and Stage 2 uses video training, with separate hyperparameter tables documenting each stage.The supplied implementation passages identify the two-stage setup and its corresponding configuration tables.
- Evaluation: Evaluation reports RMSE and AbsRel for depth, angular errors and threshold accuracies for normals, plus temporal consistency metrics for adjacent video frames.Video metrics use optical flow and a warping operator to compare neighboring frames.
- Limitations: Complex textures can still create pseudo-geometric artifacts, while rapid motion, occlusion, and non-rigid deformation can cause local prediction instability.The remaining difficulties include clothing patterns, shadows, fluttering skirts, moving sleeves, and hair.