PaperlayerView exact revision on arXiv

Source-linked AI summary

FashionChameleon: Towards Real-Time and Interactive Human-Garment Video Customization

Quanjian Song, Yefeng Shen, Mengting Chen, Hao Sun, Jinsong Lan, Xiaoyong Zhu, Bo Zheng, Liujuan Cao

arXiv:2605.15824v2cs.CV

TL;DR

Existing human-centric video customization methods provide limited fine-grained garment control despite demand for low-latency, interactive applications. FashionChameleon addresses this with an autoregressive framework for interactive garment switching, achieving coherent motion, long-video extrapolation, and 23.8 fps generation on a single GPU.

  • Problem

    Existing customization methods emphasize subject consistency over fine-grained garment control, despite practical needs for low-latency, streaming, and interactive garment editing.

  • Method

    FashionChameleon combines in-context teacher modeling, streaming distillation, and training-free KV cache rescheduling for interactive autoregressive human-garment video customization.

  • Results

    23.8 fps enables real-time 720p generation on a single GPU, while FashionChameleon outperforms baselines in temporal consistency, video quality, and three garment consistency metrics.

  • Takeaways & Limitations

    The framework supports interactive garment switching during generation while preserving coherent human motion and consistent long-video extrapolation for applications such as e-commerce and content creation.

  • Takeaways & Limitations

    Limited garment categories and variations may restrict generalization, while complex human motions and camera movements remain challenging.

Abstract

from arXiv · show

Human-centric video customization, particularly at the garment level, has shown significant commercial value. However, existing approaches cannot support low-latency and interactive garment control, which is crucial for applications such as e-commerce and content creation. This paper studies how to achieve interactive multi-garment video customization while preserving motion coherence using only single-garment video data. We present FashionChameleon, a real-time and interactive framework for human-garment customization in autoregressive video generation, where users can interactively switch garment during generation. FashionChameleon consists of three key techniques: (i) Instead of training on multi-garment video data, we train a Teacher Model with In-Context Learning on a single reference-garment pair. By retaining the image-to-video training paradigm while enforcing a mismatch between the reference and garment image, the model is encouraged to implicitly preserve coherence during single-garment switching. (ii) To achieve consistency and efficiency during generation, we introduce Streaming Distillation with In-Context Learning, which fine-tunes the model with in-context teacher forcing and improves extrapolation consistency via gradient-reweighted distribution matching distillation. (iii) To extend the model for interactive multi-garment video customization, we propose Training-Free KV Cache Rescheduling, which includes garment KV refresh, historical KV withdraw, and reference KV disentangle to achieve garment switching while preserving motion coherence. Our FashionChameleon uniquely supports interactive customization and consistent long-video extrapolation, while achieving real-time generation at 23.8 FPS on a single GPU, 30-180$\times$ faster than existing baselines.

1 Introduction

FashionChameleon addresses the need for low-latency, interactive garment-level video customization by enabling garment switching during autoregressive generation while preserving coherent human motion. Its teacher learning, streaming distillation, KV-cache rescheduling, and data curation pipeline support consistent long-video extrapolation and real-time 720p generation.

  • Garment-level customization is important for filmmaking, e-commerce, and entertainment, yet existing methods emphasize subject consistency over fine-grained attributes and lack low-latency interactive garment control.
  • FashionChameleon enables users to interactively switch garments during autoregressive video generation while maintaining coherent human motion.
  • The Teacher Model uses in-context learning with mismatched reference and target garments to implicitly preserve coherence during single-garment switching without multi-garment video training.
  • Streaming Distillation uses in-context teacher forcing to remove data-intensive ODE initialization and gradient-reweighted distribution matching to improve long-video extrapolation consistency.
  • Training-Free KV Cache Rescheduling combines garment KV refresh, historical KV withdraw, and reference KV disentangle to switch garments, suppress outdated garments, and preserve motion coherence.
  • 23.8 FPS on a single H200 GPU enables real-time 720p customization, while experiments demonstrate interactive multi-garment customization and consistent long-video extrapolation.The data curation pipeline includes coarse-to-fine video filtering, static-dynamic captioning, garment image extraction, and adaptive reference image extraction.

2 Related Works

Related work spans subject-to-video customization, hybrid autoregressive video generation, and streaming-video applications. These directions advance identity preservation, generation efficiency, and interactive downstream use cases.

  • Subject-to-Video Customization: Subject-to-video methods evolved from few-shot tuning to U-Net fine-tuning and DiT-based human-centric customization.The progression improves generalization, identity preservation, editing flexibility, and text-image alignment.
  • Hybrid Autoregressive Video Generation: Hybrid autoregressive methods combine diffusion-based frame modeling with autoregressive prediction across frames to balance fidelity and efficiency.CausVid distills a slow bidirectional teacher into a few-step autoregressive student using distribution matching distillation, avoiding training from scratch.
  • Applications of Streaming Video Generation: Streaming video generation has been applied to audio-driven avatars, video super-resolution, interactive motion control, and interactive prompt control.LiveAvatar, FlashVSR, MotionStream, and LongLive extend hybrid autoregressive generation to these downstream tasks.

3 Preliminary

The preliminary section reviews video diffusion models and hybrid autoregressive video generation. It introduces latent video prediction with conditional flow matching, autoregressive frame generation, self-rolling, and distribution-matching distillation.

  • Video Diffusion Models: Video diffusion models use a variational encoder–decoder pair and a transformer-based prediction network to model videos in latent space.The encoder transforms an F-frame video into a latent sequence.
  • Video Diffusion Models: Conditional flow matching trains the prediction network to regress a conditional vector field for noisy latent sequences.The forward process linearly interpolates between the data distribution and a standard normal distribution.
  • Hybrid Autoregressive Video Generation: Hybrid autoregressive generation factorizes the joint video distribution into conditional frame distributions, with each frame or chunk generated autoregressively.CausVid models each conditional distribution with diffusion models.
  • Hybrid Autoregressive Video Generation: Self-Forcing improves autoregressive training by conditioning on self-generated history, while DMD distills multi-step bidirectional teachers into few-step autoregressive students.DMD minimizes an approximate KL divergence between the student and teacher-related distributions, updating the student and fake-score models while freezing the real-score model.

4 Methodology

FashionChameleon combines in-context teacher modeling, streaming distillation, and KV-cache rescheduling to enable interactive garment switching with coherent autoregressive video generation. Its methodology uses shared latent representations, specialized teacher forcing, and cache management for stable long-video extrapolation.

  • Teacher Model: The teacher model uses in-context learning with a reference image and single garment image in a unified backbone, retaining the image-to-video training paradigm without auxiliary encoders.Reference and garment images are processed as discrete inputs through in-context token concatenation.
  • Teacher Model: Reference, garment, and noisy video latents share one VAE-derived semantic space, while multimodal attention enables global interaction and outputs only the video latent.Reference and garment latents remain noise-free as conditional inputs, whereas the video latent is noised during training.
  • Streaming Distillation: Streaming distillation initializes an autoregressive student with in-context teacher forcing and then applies gradient-reweighted distribution matching to improve extrapolation consistency.The teacher-forcing mask lets generated frames attend to conditioning signals and clean historical frames without accessing future generated frames.
  • KV Cache Rescheduling: KV-cache rescheduling enables interactive switching through garment KV refresh, historical KV withdrawal, and reference KV disentanglement while preserving motion coherence.The cache persistently stores reference and garment conditioning entries, retains the initial frame as an attention sink, and removes older entries using first-in-first-out management when necessary.
  • Data Curation: A four-stage data-curation pipeline constructs reference images, garment images, video sequences, and prompts for teacher pre-training and streaming distillation.The stages are coarse-to-fine video filtering, static-dynamic captioning, fine-grained garment extraction, and adaptive reference construction.

5 Experiments

Experiments compare FashionChameleon with representative multi-reference customization baselines and evaluate identity, garment consistency, motion, temporal smoothness, alignment, and visual quality. Qualitative results and ablations show advantages in long-video extrapolation, interactive garment switching, teacher training, streaming distillation, and KV cache rescheduling.

  • Evaluation Settings: The evaluation compares FashionChameleon with VACE, Kaleido, MAGREF, SkyReels-A2, Phantom, and a Qwen-Image-Edit plus WAN-5B-TI2V pipeline.Baselines generate videos at their respective native resolutions and durations.
  • Quantitative Comparisons: The experiments measure identity consistency, text alignment, motion magnitude, temporal smoothness, visual quality, and garment consistency.Garment consistency is evaluated with Gemini-3.0 using high- and low-level garment-consistency aspects.
  • Long-Video Extrapolation: FashionChameleon maintains character and garment consistency across long temporal ranges through autoregressive long-video extrapolation, unlike bidirectional methods that synthesize all frames jointly.Figure 6 demonstrates long-video extrapolation results.
  • Interactive Customization: FashionChameleon supports interactive garment switching during generation while preserving coherent human motion, beyond the capability of existing methods.This capability is enabled by KV Cache Rescheduling and shown in Figure 6.
  • Ablation with Teacher Model: In-context learning outperforms channel-wise concatenation across several metrics, while Full FT performs best overall among the evaluated teacher-model fine-tuning strategies.The compared strategies are Full FT, Attn FT, and LoRA FT; Full FT is adopted for streaming distillation.
  • Ablation with Streaming Distillation: Gradient-Reweighted DMD produces coherent and anatomically consistent human structures during 165-frame extrapolation, avoiding distortions or duplicated limbs associated with naive DMD.The comparison uses qualitative and quantitative evaluations in Table 3 and Figure 7.

6 Conclusion

FashionChameleon is a real-time, interactive framework for autoregressive human-garment video customization that enables garment switching during generation while preserving motion coherence. It combines in-context teacher modeling, streaming distillation, and training-free KV cache rescheduling, achieving 720p generation at 23.8 fps on a single GPU.

  • Conclusion: FashionChameleon enables interactive garment switching during autoregressive human-garment video generation while preserving coherent human motion.The framework targets interactive multi-garment customization without requiring multi-garment video training data.
  • Conclusion: The framework combines a Teacher Model with In-Context Learning, Streaming Distillation with In-Context Learning, and Training-Free KV Cache Rescheduling.These techniques respectively encourage coherence during single-garment switching, improve efficient inference and consistent long-video extrapolation, and support interactive multi-garment customization.
  • Conclusion: 23.8 fps is achieved for real-time 720p video generation on a single GPU.Extensive experiments report superiority over existing approaches alongside this real-time generation performance.

A Data Curation Pipeline Details

The data curation pipeline has four stages: coarse-to-fine video filtering, static-dynamic captioning, fine-grained garment extraction, and adaptive reference-image construction. It ultimately produces manually verified reference–garment–video triplets for training.

  • Pipeline Overview: The pipeline comprises four stages: video filtering, static-dynamic captioning, garment image extraction, and adaptive reference image construction.These stages are illustrated in Figure 8.
  • General Coarse-to-Fine Video Filtering: Videos are filtered through shot segmentation, single-person detection, moderate-to-large motion estimation, and aesthetic and visual-quality assessment.PySceneDetect creates 3–5 second subclips; YOLOv8-Seg, UniMatch, Q-Align, and FAST-VQA-M provide the filtering steps.
  • Static-Dynamic Video Captioning: Gemini-3.1 generates separate static captions for scene, appearance, and garment details and dynamic captions for expressions, actions, camera motion, and transitions.Static elements remain unchanged over time, whereas dynamic elements typically change temporally.
  • Fine-Grained Garment Images Extraction: Qwen-Image-Edit extracts garments from initial frames, while VLM checks verify semantic and textural consistency and exclude non-garment context.Extraction is repeated until all checks pass; repeatedly failing samples are discarded.
  • Adaptive Reference Images Construction: Adaptive reference construction classifies extracted garment type, retrieves a compatible garment, performs image try-on, and verifies unchanged non-edited regions.The reference garment should differ from the extracted garment, and repeated reconstruction failures cause sample discard.
  • Dataset Yield: 82K triplets were curated, and 62K triplets remained after manual verification.Each triplet contains a reference image, garment image, and corresponding video.

B Training Details · C HGC-Bench Details

Training uses a curated 62K paired video dataset with standardized 81-frame, 1280 × 704 inputs, while pre-training and distillation employ FSDP and specified optimization settings. HGC-Bench evaluates customization using anonymized Internet reference images, paired garment images, and Gemini-3.0-generated prompts.

  • B Training Details: Pre-training keeps the VAE in float32 and fully fine-tunes the transformer in bfloat16 using FSDP with a global batch size of 64.Optimization uses AdamW with β1 = 0.9, β2 = 0.999, and weight decay 0.01, followed by warm-up and two-stage learning-rate decay.
  • B Training Details: Post-training maintains the VAE and transformer in bfloat16 and uses FSDP with a global batch size of 64.The generator is initialized from the pre-trained teacher model for teacher forcing.
  • B Training Details: 4000 steps fine-tune the teacher-forced generator with AdamW at a learning rate of 1 × 10^-6 and timestep schedule [1000, 750, 500, 250].Teacher forcing uses β1 = 0.0, β2 = 0.999, and weight decay 0.01; distribution matching initializes score networks from the pre-trained teacher.
  • C HGC-Bench Details: HGC-Bench samples illustrate diverse scenarios and semantic content, varied garment-category proportions, and triplets of reference image, garment image, and input prompt.These dataset analyses and representative samples are summarized in Figure 9.
  • B Training Details: 62K paired samples support both pre-training and post-training, each containing a reference image, garment image, and video sequence.Sequences contain 81 frames; video and reference images are resized to 1280 × 704, while garment images are center-padded to that resolution.
  • C HGC-Bench Details: HGC-Bench curates high-aesthetic reference images from the Internet, anonymizes faces through face swapping, and pairs them with garment images from a collected database.The benchmark is designed for comprehensive evaluation.
  • C HGC-Bench Details: Gemini-3.0 generates a corresponding prompt from each reference-image and garment-image pair, including concise static descriptions such as accessories and scene information.The supplied passage introduces the prompt-generation process but truncates the full prompt specification.

D Additional Ablation Studies on Distillation Prompts · E Additional User Study

Additional ablations compare distillation caption strategies and find that long static-dynamic captions perform best. A user study with 672 valid responses reports superior human preference for FashionChameleon across four video-quality dimensions.

  • D Additional Ablation Studies on Distillation Prompts: The teacher model uses a hybrid caption strategy with 70% dynamic content and 30% static-dynamic content to extract static information from reference images.
  • D Additional Ablation Studies on Distillation Prompts: Table 4 quantitatively compares different distillation captions at τ = 0.2 using Cur., GME, Amp., Smoo., VQ, HGC, LGC, and NTP.
  • E Additional User Study: FashionChameleon achieves superior human preference rates across ID Consistency, Garment Consistency, Temporal Coherence, and Visual Quality.
  • D Additional Ablation Studies on Distillation Prompts: Using different caption types during streaming distillation produces different distilled results.
  • D Additional Ablation Studies on Distillation Prompts: Long captions containing static-dynamic contents yield superior performance in the distillation ablation.
  • E Additional User Study: The user study collected 672 valid responses comparing FashionChameleon with other baselines.
  • E Additional User Study: Participants selected videos with the best ID Consistency, Garment Consistency, Temporal Coherence, and Visual Quality.

F Evaluation Details

This section defines the metrics used to evaluate identity, text alignment, motion, temporal smoothness, visual quality, inference efficiency, and garment consistency in generated videos.

  • Evaluation Metrics: Cur Score measures identity consistency by comparing ArcFace facial-embedding cosine similarity between the reference image and generated video frames.Facial embeddings are extracted from the reference image and each video frame.
  • Evaluation Metrics: GME Score assesses semantic alignment between the generated video and input prompt using a Qwen2-VL-fine-tuned vision-language model.The model is selected for handling long and complex text descriptions.
  • Evaluation Metrics: Amplitude, Smoothness, and VQ Score evaluate motion magnitude, temporal coherence, and visual quality using optical flow, Q-Align, and MUSIQ, respectively.Amplitude averages optical-flow magnitude across pixels and frames; Smoothness uses Q-Align; VQ Score averages MUSIQ frame scores.
  • Evaluation Metrics: FPS measures inference efficiency as the number of frames generated by the backbone network divided by inference time.The metric reports generation speed in frames per second.
  • Evaluation Metrics: Garment consistency is evaluated with Gemini-3.0 across high-level consistency, low-level consistency, and non-target garment preservation.The paper notes that no established metric is available for this purpose.

G Limitations and Future Work

FashionChameleon’s generalization remains limited by restricted garment diversity and challenging human or camera motion, motivating better data curation, larger datasets, and stronger video backbones.

  • Limitations and Future Work: Limited garment categories and variations in the curated training data may restrict FashionChameleon’s generalization to complex scenarios.Future work could develop a more efficient data curation pipeline and scale up training datasets.
  • Limitations and Future Work: Imperfect open-source video generation backbones, such as Wan, challenge the model on complex human motions and camera movements.Future work could explore stronger video generation backbones to address these limitations.

H Potential Negative Societal Impact

FashionChameleon enables human-garment video customization but may pose societal risks if deployed irresponsibly or used maliciously. Key risks include inappropriate content, amplified stereotypes or bias, and realistic misleading videos.

  • Responsible Deployment: Because irresponsible or malicious deployment can introduce nontrivial societal risks, the method should be accompanied by appropriate protections against misuse.The authors frame these risks as relevant to human-centric content creation scenarios.
  • Sexually Explicit or Violent Content: FashionChameleon may generate sexually explicit, violent, or otherwise inappropriate material without safeguards, potentially harming audiences psychologically or emotionally.The paper identifies these risks as consequences of insufficient safeguards.
  • Stereotypes and Bias: Training-data biases in character and garment information may be reflected or amplified, reinforcing harmful stereotypes or discriminatory visual representations.These concerns specifically involve unintended biases carried into generated content.
  • Misleading Content: The model may be misused to create realistic but false video advertisements, accelerating the spread of misleading information at scale.The risk concerns deceptive advertising enabled by realistic human-garment video customization.

I Additional Qualitative Comparison … N System Prompts of VLM

Additional qualitative results show that FashionChameleon preserves character, garment, and motion consistency across standard, short, interactive, and long-video customization settings. The appendix also documents figure-generation prompts and VLM system prompts for dataset construction and garment-consistency evaluation.

  • I Additional Qualitative Comparison: FashionChameleon produces more consistent characters and garments, with more coherent and higher-quality results than competing baselines.These findings are illustrated by additional qualitative comparisons in Figures 11 and 12.
  • J Additional Examples of Short Video Customization: FashionChameleon generates coherent, high-fidelity customized videos at the 81-frame training length.The model is trained on 81-frame clips and supports short videos of the same length, with examples in Figures 13 and 14.
  • K Additional Examples of Interactive Customization: KV cache rescheduling enables interactive multi-garment generation while preserving motion continuity during reference-image switching.Users can freely switch reference images at different generation stages, unlike conventional methods requiring a reference image in advance.
  • L Additional Examples of Long Video Customization.: FashionChameleon extrapolates beyond the training sequence length while maintaining long-range character and garment consistency.Additional long-video extrapolation results are provided in Figures 17 and 18.
  • M Prompt List of Figures: The documented figure prompts include detailed human actions, garment descriptions, settings, camera movements, gaze changes, and physically natural motion.Examples cover qualitative comparisons, short-video customization, interactive customization, and long-video extrapolation.
  • N System Prompts of VLM: The Gemini-3.1 VLM prompt requires structured multilingual descriptions focused on visible dynamic video content and semantically aligned Chinese and English outputs.It requires objective, accurate descriptions without speculation, with long descriptions comprehensive and short descriptions focused.
Loading 2605.15824v2…