PaperlayerView exact revision on arXiv

Source-linked AI summary

HumanSD: A Native Skeleton-Guided Diffusion Model for Human Image Generation

Xuan Ju, Ailing Zeng, Chenchen Zhao, Jianan Wang, Lei Zhang, Qiang Xu

arXiv:2304.04269v1cs.CV

TL;DR

Precise pose control in human image generation is difficult because dual-branch diffusion must reconcile frozen SD outputs with added skeleton conditions. HumanSD fine-tunes SD natively with heatmap-guided denoising and large-scale text-image-pose data, outperforming ControlNet in pose control and image quality, particularly for sophisticated skeleton guidance.

  • Problem

    Dual-branch controllable diffusion methods can conflict with frozen SD generation when enforcing precise skeleton conditions for human images.

  • Method

    HumanSD fine-tunes the original SD model with skeleton conditions and a heatmap-guided denoising loss, using three large-scale human-centric datasets.

  • Results

    HumanSD outperforms ControlNet in pose control and human image quality, especially when skeleton guidance is sophisticated.

  • Takeaways & Limitations

    Native fine-tuning strengthens pose conditioning while preserving SD generation ability without extra inference computation.

  • Takeaways & Limitations

    HumanSD can fail on extremely crowded scenes and complex or rare actions, and its evaluation system is not comprehensive.

Abstract

from arXiv · show

Controllable human image generation (HIG) has numerous real-life applications. State-of-the-art solutions, such as ControlNet and T2I-Adapter, introduce an additional learnable branch on top of the frozen pre-trained stable diffusion (SD) model, which can enforce various conditions, including skeleton guidance of HIG. While such a plug-and-play approach is appealing, the inevitable and uncertain conflicts between the original images produced from the frozen SD branch and the given condition incur significant challenges for the learnable branch, which essentially conducts image feature editing for condition enforcement. In this work, we propose a native skeleton-guided diffusion model for controllable HIG called HumanSD. Instead of performing image editing with dual-branch diffusion, we fine-tune the original SD model using a novel heatmap-guided denoising loss. This strategy effectively and efficiently strengthens the given skeleton condition during model training while mitigating the catastrophic forgetting effects. HumanSD is fine-tuned on the assembly of three large-scale human-centric datasets with text-image-pose information, two of which are established in this work. As shown in Figure 1, HumanSD outperforms ControlNet in terms of accurate pose control and image quality, particularly when the given skeleton guidance is sophisticated.

1. Introduction

Controllable HIG needs precise pose guidance, but dual-branch diffusion methods must reconcile conflicts between frozen SD generation and added conditions. HumanSD instead fine-tunes SD natively with heatmap-guided denoising and large-scale human-centric data, outperforming ControlNet in pose control and image quality.

  • Precise position control remains difficult for text-conditioned diffusion models, particularly for deformable human subjects.
  • ControlNet and T2I-Adapter improve controllability by adding learnable condition branches above frozen pre-trained SD models.
  • These dual-branch methods face conflicts because the frozen SD branch may generate humans inconsistent with the requested skeleton, forcing the added branch to perform difficult feature editing.
  • HumanSD directly fine-tunes SD with skeleton conditions and heatmap-guided denoising to focus on foreground humans while reducing catastrophic forgetting.
  • The method uses three large-scale human-centric datasets containing text, image, and pose information, including two datasets established in this work.
  • HumanSD outperforms state-of-the-art methods such as ControlNet in pose control and human image generation quality, especially with sophisticated skeleton guidance.

2. Related Work

Prior HIG methods face limitations in cross-modality alignment, source-pair diversity, and scenario coverage. HumanSD addresses the data gap with a standard process for large-scale, multi-scenario text-image-pose datasets.

  • Earlier GAN- and VAE-based methods struggle with training, multimodality fusion, feature alignment, and diverse arbitrary-pose generation.
  • ControlNet, T2I-Adapter, and GLIGEN extend text-to-image diffusion models with arbitrary condition mechanisms, including additional trainable modules.
  • Existing HIG datasets often provide noisy paired images, limited scenarios, or insufficient scale for broad human-centric model training.
  • Human-Art supplies precise pose and text annotations across natural and artificial scenes but its 50K-image scale is inadequate for large-model training.
  • This work provides a standard development process for large-scale multi-scenario text-image-pose datasets targeted at skeleton-guided HIG.

3. Preliminaries and Motivation

Latent diffusion generates images in a VAE latent space, while dual-branch condition injection can create conflicts between frozen generation features and trainable pose features. HumanSD motivates native fine-tuning to avoid these conflicts.

  • Stable Diffusion is a latent diffusion model that projects images into VAE latent embeddings and guides generation with text conditions in latent space.
  • The authors argue that conflict between frozen image-generation and trainable condition-injection branches degrades pose control.
  • ControlNet clones an SD branch for condition features, connects trainable and frozen blocks through convolution, and adds the trainable output to frozen features.
  • The additional branch must identify, suppress, and augment noise components, while the frozen branch permanently preserves conflicting behavior during inference.

4. Method

HumanSD fine-tunes the stable diffusion model natively with skeleton conditions and a heatmap-guided denoising loss, rather than adding a separate condition branch. Its method also relies on diverse human-centric datasets constructed for pose, text, and image training.

  • Native Skeleton Conditioning: HumanSD adds pose conditions by encoding a skeleton image and concatenating its latent embedding directly with noisy latent embeddings.This aligns pose information with the input latent representation and processes it at the same density level.
  • Heatmap-guided Denoising Loss: The heatmap-guided denoising loss emphasizes newly added condition information while leaving condition-invariant image regions to the pretrained backbone.The loss explicitly adds an aggregated heatmap weight Wa to the original diffusion loss.
  • Heatmap-guided Denoising Loss: Wa must vary across diffusion steps because constant weighting can disrupt training when condition injection is not equally important throughout denoising.The method derives step-wise heatmaps from differences between estimated and ground-truth noise using a pretrained human-pose heatmap estimator.
  • Dataset Construction: GHI contributes 1M pose-image-text pairs generated from SD, filtered for image quality, human count, body integrity, and pose annotation cleanliness.Its prompts span 14 scenes and 6,826 human actions.
  • Dataset Construction: LAION-Human contributes about 1M filtered internet image-text pairs, while Human-Art supplies 50k precisely annotated images across natural and artificial scenarios.LAION-Human emphasizes diverse actions and photorealistic images; Human-Art supports training and quantitative evaluation.
  • Dataset Construction: The training ensemble combines GHI, LAION-Human, and Human-Art to provide large-scale human-centric images with pose and text information.HumanSD is trained on the union and evaluated on Human-Art’s validation set.

5. Experiments

HumanSD is evaluated against SD- and GAN-based baselines across image quality, pose accuracy, text-image consistency, and inference time. It achieves stronger pose control, faster inference, and competitive image quality, while showing limitations under text-pose inconsistency.

  • 5.2. Comparison with SD-based Methods: HumanSD improves pose accuracy by 34.8%–109.1% over the best ControlNet or T2I-Adapter results, especially for challenging poses.The comparison uses AP, AP(m), and PCE under a fair 0.2M-image training setting.
  • 5.2. Comparison with SD-based Methods: HumanSD infers faster than ControlNet and T2I-Adapter because its single-branch design avoids an additional condition-learning branch.The text also reports that T2I-Adapter’s compressed condition branch sacrifices quality relative to ControlNet.
  • 5.2. Comparison with SD-based Methods: HumanSD, ControlNet, and T2I-Adapter show similar FID and KID, indicating comparable preservation of Stable Diffusion’s image-generation abilities.HumanSD preserves quality by concentrating fine-tuning on human regions, whereas the baselines retain frozen Stable Diffusion branches.
  • 5.2. Comparison with SD-based Methods: HumanSD and ControlNet show a small decline in text-image consistency when text and pose conditions are potentially inconsistent.The results suggest both methods prioritize pose over text in such cases.
  • 5.3. Comparison with GAN-based Methods: NTED and TIPS fail on unconventional poses, attaining AP scores of 2.79 and 17.65, respectively.These results indicate limited precise pose control for the compared GAN-based methods in open-scenario poses.
  • 5.4. Ablation Study: Adding the heatmap-guided denoising loss raises AP from 30.63 to 32.66 and CLIPSIM from 32.55 to 32.98.The ablation attributes these gains to more focused human optimization, improved pose guidance, and better background preservation.
  • 5.4. Ablation Study: Combining GHI, LAION-Human, and Human-Art improves AP and provides better trade-offs among PCE and CLIPSIM than individual dataset settings.GHI alone provides stronger human-number accuracy and text-image consistency but suffers from low image quality without real images.

6. Conclusion

The paper presents HumanSD for precise pose- and text-conditioned human image generation using a heatmap-guided denoising loss and large-scale human-centric datasets. It reports effective and efficient generation, while identifying failures in crowded or rare-action scenes, dataset-content risks, and incomplete evaluation.

  • 6. Conclusion: HumanSD combines native Stable Diffusion fine-tuning, heatmap-guided denoising, and over 2M text-image-pose pairs for multi-scenario human generation.The framework is designed to preserve pretrained generation ability while concentrating learning on foreground humans.
  • 6. Conclusion: HumanSD improves controllability, but still easily fails on extremely crowded scenes and complex or rare actions.This is identified as a remaining limitation of the proposed system.
  • 6. Conclusion: The training pairs may still contain social biases and violent content despite filtering, and the evaluation system remains incomplete.Both dataset-content risk and evaluation coverage are explicitly listed as limitations.

Supplementary Materials

The supplementary material provides additional experimental and implementation information omitted from the main paper. It covers experimental details, quantitative and qualitative results, and future work.

  • Supplementary Materials: The supplement contains experimental details, additional quantitative and qualitative results, and future-work discussion.The listed sections include experimental details, more quantitative results, more qualitative results, and future work.

A. Experimental Details

The supplementary experimental details explain the evaluation metrics, HumanSD training procedure, and heatmap-guided denoising loss. These materials expand the methodological description beyond the main paper.

  • A. Experimental Details: Supplementary Section A details the evaluation metrics, HumanSD training procedure, and heatmap-guided denoising loss.The listed subsections are A.1 for metrics, A.3 for training, and A.4 for the loss.

A.1. Evaluation Metrics

The evaluation uses text-image consistency, FID, KID, and pose accuracy measures to assess generated-image quality and pose control across Human-Art scenarios.

  • Image quality metrics: FID compares Inception-v3 feature distributions between Human-Art images and generated images.The distributions are modeled as multivariate Gaussians, using a default 64-dimensional feature layer.
  • Image quality metrics: KID measures maximum mean discrepancy between features from Human-Art images and generated images.KID and FID are averaged across scenarios in the Human-Art validation set.
  • Pose accuracy metrics: AP and CAP jointly evaluate pose controllability, with CAP removing positional effects while AP measures pose and position agreement.An example gives AP=0 and CAP=1 when two people share a pose but occupy non-overlapping positions.
  • Text-image consistency: Text-image consistency uses CLIP-ViT-base-patch16 to extract text and image features.The image encoder is a ViT-B/16 Transformer and the text encoder is a masked self-attention Transformer.
  • Evaluation setup: The evaluation also analyzes prompt composition, whose ordering affects generation results, and reports implementation training details.The prompt design uses 18 parts describing image, human, and scene components; training uses four A100 GPUs for about 95,000 iterations.

A.4. Detailed Implementation of the Heatmap-Guided Denoising Loss

The heatmap-guided denoising loss derives a pose heatmap from denoising errors, converts it into an embedding, and uses it to weight training toward human regions.

  • Heatmap-guided weighting: The denoising-error difference map is decoded through the VAE and processed by a pretrained pose estimator to produce joint heatmaps.The difference map is M = VAEdecoder(ϵ − ϵθ), followed by H = F(M), where H contains heatmaps over human joints.
  • Heatmap estimator: The implementation uses HigherHRNet pretrained on MSCOCO and Human-Art to estimate heatmaps from blurred noise-difference maps.The combined training sources are used to improve inclusiveness for the heatmap estimator.
  • Heatmap-guided weighting: A thresholded summed heatmap identifies human regions, which are re-encoded as a heatmap embedding for loss weighting.The method thresholds the heatmap at 0.1, then passes the resulting mask through the VAE encoder.
  • Heatmap-guided weighting: The weighting function combines the heatmap embedding with a residual baseline, using w=0.05 by default.The weighting is defined as Wa = w · HE + 1.

B. Quantitative results

Quantitative evaluations show that HumanSD generally improves pose accuracy across scenarios and retains stronger performance as the number of generated humans increases.

  • Comparisons across scenarios: Pose AP varies substantially by scenario, with garage kits highest and shadow play lowest, while complex acrobatics and dance scenes remain difficult.The passage links these differences partly to uneven dataset distributions and pose complexity.
  • Comparisons across scenarios: HumanSD has higher AP than ControlNet and T2I-Adapter in every scenario except shadow play, with gains of 18.2% and 24.1% in cosplay.The reported exception is attributed to limited shadow-play training data for the competing methods.
  • Comparisons by human count: HumanSD retains relatively strong pose AP as the number of humans increases and performs better than both baselines for images containing 1–13 humans.Results for 14–19 humans fluctuate because that range contains few images, but HumanSD remains relatively stronger.

C. Qualitative Results

Qualitative comparisons cover natural, sketch, rare-scene, and artistic settings, while the paper also identifies crowded scenes, rare actions, and whole-body detail retention as limitations.

  • Qualitative comparisons: Figures 12–16 compare SD, ControlNet, T2I-Adapter, and HumanSD across varied scenes, poses, orientations, rare scenarios, and artistic details.Each condition is generated with three seeds to display diversity while holding the text and pose inputs fixed.
  • Failure cases: HumanSD still fails in extremely crowded scenes and complex or rare actions.The paper identifies infrequent poses, such as standing upside down, as more failure-prone.
  • Failure cases: Whole-body generation creates a trade-off with local facial detail retention, especially for ControlNet.The paper contrasts high-fidelity faces in local generation with substantial facial-detail decline when forcing whole-body images.
  • Evaluation limitations: The text-and-pose evaluation system remains incomplete and introduces substantial randomness into assessment.This is listed as an open issue alongside the need for augmentations covering complex poses and human orientations.
Loading 2304.04269v1…