PaperlayerView exact revision on arXiv

Source-linked AI summary

Implicit Preference Alignment for Human Image Animation

Yuanzhi Wang, Xuhua Ren, Jiaxiang Cheng, Bing Ma, Kai Yu, Tianxiang Zheng, Qinglin Lu, Zhen Cui

arXiv:2605.07545v1cs.CVcs.AI

TL;DR

High-fidelity hand motion remains difficult in human image animation, while DPO requires costly and often impractical good-bad pairs. IPA instead aligns the model using self-generated good samples under a pretrained prior constraint and steers optimization toward hands, with experiments reporting improved hand fidelity and video quality. The authors position IPA as a resource-efficient alternative when paired preference data are scarce.

  • Problem

    Complex hand motion remains difficult to generate, while constructing strict good-bad preference pairs for dynamic hands is costly and often impractical.

  • Method

    IPA maximizes the likelihood of self-generated high-quality samples while constraining deviation from the pretrained model, with Hand-Aware Local Optimization focusing alignment on hands.

  • Results

    IPA and Hand-Aware Local Optimization improve hand-generation fidelity and overall video quality, outperforming existing state-of-the-art methods.

  • Takeaways & Limitations

    IPA offers a specialized, resource-efficient preference-alignment alternative for scenarios where high-quality paired preference data are scarce.

  • Takeaways & Limitations

    IPA is not claimed to be inherently superior to DPO in all general scenarios and is positioned for settings with scarce high-quality preference pairs.

Abstract

from arXiv · show

Human image animation has witnessed significant advancements, yet generating high-fidelity hand motions remains a persistent challenge due to their high degrees of freedom and motion complexity. While reinforcement learning from human feedback, particularly direct preference optimization, offers a potential solution, it necessitates the construction of strict preference pairs. However, curating such pairs for dynamic hand regions is prohibitively expensive and often impractical due to frame-wise inconsistencies. In this paper, we propose Implicit Preference Alignment (IPA), a data-efficient post-training framework that eliminates the need for paired preference data. Theoretically grounded in implicit reward maximization, IPA aligns the model by maximizing the likelihood of self-generated high-quality samples while penalizing deviations from the pretrained prior. Furthermore, we introduce a Hand-Aware Local Optimization mechanism to explicitly steer the alignment process toward hand regions. Experiments demonstrate that our method achieves effective preference optimization to enhance hand generation quality, while significantly lowering the barrier for constructing preference data. Codes are released at https://github.com/mdswyz/IPA

1. Introduction

Human image animation has advanced, but complex, high-degree-of-freedom hand motions still produce blur and malformations. IPA addresses costly preference-pair construction by aligning from good samples and focusing optimization on hands.

  • Challenge: High degrees of freedom and ten flexible fingers make hand motions especially complex, often causing blur and malformations.Hands can contain substantial motion even when other body regions remain still.
  • Preference Alignment: DPO typically requires manually annotated good-bad video pairs, but strict pairs for dynamic hands are prohibitively expensive and impractical.Different seeds generate candidate videos that must be manually sorted by hand quality.
  • IPA: IPA eliminates bad samples by maximizing likelihood of self-generated high-quality samples while constraining deviation from the pretrained model.The framework is theoretically grounded in implicit reward maximization and aims to avoid mode collapse.
  • IPA: Hand-Aware Local Optimization explicitly steers alignment toward hand regions to mitigate geometric distortions and blurring artifacts.The mechanism prioritizes fine-grained hand structure during optimization.
  • Results: Experiments report improved hand-generation fidelity and overall video quality, outperforming existing state-of-the-art methods.The reported evaluation includes quantitative and qualitative experiments.

2. Related Work

Human image animation research progressed from GAN-based approaches to diffusion architectures, which target higher-fidelity generation and temporal consistency. Recent systems align reference appearance with motion and support increasingly capable video-generation backbones.

  • GAN-based Approaches: Early human image animation approaches predominantly relied on GANs and motion networks that warped source appearance onto target poses.These methods estimated dense appearance flows for feature warping.
  • Diffusion-based Approaches: Recent research has largely pivoted toward diffusion models because of their training stability and high-fidelity generation capabilities.The shift reflects broader adoption of continuous-time modeling.
  • Diffusion-based Approaches: Animate Anyone introduced ReferenceNet to extract human appearance features from an input image and align them with motion generation.UniAnimate later aligned reference-image and video features within a shared space.
  • Large-model Architectures: DiT-based large-model architectures have further advanced video generation, improving character realism and temporal consistency in human image animation.UniAnimate-DiT extended UniAnimate to the Wan2.1 video foundation model, while VACE supports human image animation.

3. Preliminaries

Flow Matching models continuous-time transport between source and target distributions, while RLHF aligns outputs with human preferences under a pretrained-model constraint. DPO simplifies RLHF by directly optimizing annotated preference pairs.

  • Generative Modeling via Flow Matching: Flow Matching transforms a source distribution p0 into a target distribution p1 through a continuous-time vector field.Rectified Flow defines the probability path through linear interpolation between source and target samples.
  • Generative Modeling via Flow Matching: The generative model is trained to approximate the constant velocity field using mean squared error, with conditioning information such as prompts or reference images.Flow Matching supports stable training and efficient straight-line inference paths.
  • Reinforcement Learning from Human Feedback: RLHF maximizes a human-preference reward while restraining deviation from the initial pretrained model.A KL-divergence penalty controls the strength of this restraint.
  • Reinforcement Learning from Human Feedback: DPO directly optimizes the policy using winner-loser preference pairs, bypassing explicit reward modeling.Variants have been developed for diffusion and flow-based generative paradigms.

4. Method

IPA replaces strict good-bad preference pairs with alignment from self-generated good samples while constraining deviation from the pretrained model. A hand-aware weighting mechanism then prioritizes hand-region quality during optimization.

  • Motivation: Human image animation is difficult for hands because their high motion complexity often causes distortion and collapse.
  • Motivation: DPO is impractical when valid bad counterparts cannot be consistently sampled, because only the consistently good-versus-bad scenario satisfies its preference-pair requirement.
  • Implicit Preference Alignment: IPA aligns the model using only self-generated high-quality samples, eliminating the need for strict preference pairs and bad samples.The approach is intended to reduce preference-data production costs.
  • Implicit Preference Alignment: The alignment objective uses a divergence constraint coefficient β to limit deviation from the pretrained model and reduce overfitting during fine-tuning.Larger β keeps the aligned model closer to the reference model, whereas smaller β permits larger deviation.
  • Implicit Preference Alignment: IPA's objective maximizes consistency with preferred data while minimizing divergence from the pretrained reference model through an implicit reward formulation.The paper states that minimizing the log-sigmoid loss is equivalent to maximizing the implicit reward.
  • Hand-Aware Local Optimization: Hand-aware local optimization injects a spatial weight matrix into the objective so alignment prioritizes hand regions.The matrix uses a binary hand mask and a hand enhancement coefficient λ.

5. Experiments

Experiments evaluate IPA and Hand-Aware Local Optimization across standard, challenging, hand-region, qualitative, and ablation settings. The method improves overall and hand-specific generation quality, with β = 600 and λ = 10 identified as effective settings.

  • Evaluation Setup: The evaluation uses TikTok sequences 335–340 plus a disjoint 100-case benchmark covering complex hand dynamics.The study compares four image-generation and five video-generation baselines.
  • Baseline Comparisons: IPA achieves the best performance across all TikTok evaluation metrics, reducing FID-VID from 8.6 to 5.9 and FVD from 316 to 255 versus Wan-Animate.It also achieves SSIM 0.841 and PSNR 23.8.
  • Hand-Region Evaluation: On hand regions, IPA consistently outperforms all baselines on the SSIM-Hand and PSNR-Hand metrics.These metrics quantify hand-region structural integrity and texture details using hand masks.
  • Qualitative Results: Qualitative comparisons show clearer, anatomically correct hands with distinct fingers under challenging motion, unlike baselines’ blurred, distorted, or collapsed appearances.The visual comparisons include sharp edges and distinct finger separation relative to the Ground Truth.
  • Ablation Studies: Ablations show that IPA improves performance, HALO provides further gains, and their combination produces distinct hand structures and texture details.Removing both components causes severe hand malformations, while removing HALO leaves blurry artifacts.
  • Hyperparameter Studies: Performance peaks at β = 600; increasing λ from 0.1 to 10 improves all metrics, whereas λ = 100 slightly saturates or degrades performance.Excessive β imposes an overly strict constraint, while excessive λ can disrupt global video quality.

6. Broader Discussion for IPA and DPO

The discussion positions IPA as a resource-efficient alternative to DPO for complex hand-animation settings where strict preference pairs are scarce. It distinguishes IPA’s novel derivation and practical justification from the algebraic structure it shares with Flow-DPO.

  • Structural Comparison and Novelty Positioning: IPA and Flow-DPO have mathematically equivalent structural forms, but IPA derives the form by minimizing the KL-divergence gap under a strict prior constraint.The paper attributes Flow-DPO’s structure to the Bradley-Terry model and IPA’s form to a first-principles derivation.
  • Structural Comparison and Novelty Positioning: The paper claims novelty in theoretically and practically justifying the reduction rather than inventing a new algebraic operator.Its positioning emphasizes why the reduction is viable and essential for complex generation tasks.
  • Data Constraints and Comparison Fairness: Only 7 of 93 high-quality samples, approximately 7.5%, could be paired with valid bad samples for direct DPO training.This scarcity makes equal-cost, equal-volume comparison between IPA and DPO infeasible.
  • Positioning IPA and DPO: DPO is preferable when good and bad pairs are easy to distinguish or extensive generation and annotation resources are available.In those settings, DPO can use explicit negative feedback as a robust optimization signal.
  • Positioning IPA and DPO: IPA is preferable for highly complex tasks such as dynamic hand articulation or resource-constrained settings because it uses only self-generated good samples.The paper presents this as a trade-off based on task complexity and resource availability, not universal superiority over DPO.

7. Conclusion

The conclusion presents IPA as a data-efficient post-training framework for high-fidelity hand motion generation. It combines implicit reward maximization with Hand-Aware Local Optimization and reports experimental validation.

  • Conclusion: IPA addresses the persistent challenge of generating high-fidelity hand motions in human image animation.The framework is described as novel and data-efficient.
  • Conclusion: IPA eliminates strict preference-pair construction by aligning the model solely with good samples through an implicitly derived reward-maximization objective.This reduces the need for expensive paired preference data.
  • Conclusion: Hand-Aware Local Optimization explicitly steers the optimization trajectory toward hand regions.The conclusion identifies this mechanism as a component of the proposed framework.
  • Conclusion: Extensive experiments validate the effectiveness of IPA.

Impact Statement

The paper identifies broad application potential for human image animation while warning that high-fidelity generation can be misused to create misleading content. It advocates responsible deployment safeguards.

  • Impact Statement: Human image animation has potential applications in film production, virtual reality, and digital content creation.
  • Impact Statement: High-fidelity generative technologies carry a risk of misuse for creating misleading content.
  • Impact Statement: The paper advocates watermarking and detection mechanisms to safeguard against malicious use.It also states that a classifier can distinguish real and generated videos using texture features.

A.1. Ablation Study of Our Method and Supervised Fine-Tuning

The ablation study compares the proposed method with supervised fine-tuning using the same curated high-quality samples. It evaluates both approaches on TikTok and the proposed benchmark.

  • Ablation Study: The ablation study compares IPA with standard Supervised Fine-Tuning.
  • Ablation Study: The SFT model uses exactly the same curated high-quality samples as IPA for a fair comparison.
  • Ablation Study: SFT is trained with the standard generative flow matching objective and the proposed Hand-Aware Local Optimization.The study reports quantitative results for this comparison.
  • Ablation Study: The quantitative comparison is conducted on both the TikTok dataset and the proposed benchmark.

A.2. Ablation Study of Regularized SFT

Ablations show that IPA outperforms regularized SFT and KTO, while its training dynamics exhibit initialization, active-learning, and gradient-saturation phases. Hyperparameter and human-preference studies further characterize effective hand generation and consistent preference for IPA.

  • Regularized SFT: IPA substantially outperforms regularized SFT, whose L2 anchor regularizer mitigates catastrophic forgetting but remains far behind IPA.The comparison uses identical settings aside from the training objective and regularization.
  • Comparison with KTO: IPA significantly outperforms KTO when both methods use the same 93 high-quality videos, base model, and training steps.KTO additionally uses 93 randomly sampled unpaired videos as bad samples.
  • Training dynamics: During steps 0–100, vθ ≈ vref makes Δ ≈ 0 and produces an initial loss of approximately 0.69, yielding a strong gradient toward high-quality samples.The loss is −log σ(0) ≈ 0.69.
  • Training dynamics: During steps 100–600, the hand-structure term shrinks while Δ becomes increasingly positive because the prior-distance term remains constant.This is described as the active-learning phase of IPA training.
  • Training dynamics: During steps 600–1000, the sigmoid output approaches 1.0, −log(1.0) approaches 0, and the loss curve plateaus, empirically supporting saturation.The tracked dynamics are reported across 1,000 training steps.
  • Hyperparameter β: β = 200 produces an anatomically impossible third hand, β = 2000 causes blurry distortions, while β = 600 yields clear hand structures without impossible artifacts.The visual study examines β values spanning 200 to 2000 on two benchmarks.
Loading 2605.07545v1…