PaperlayerView exact revision on arXiv

Source-linked AI summary

Kandinsky S2V 1.0: Audio-Driven Avatar Video Generation

Nikolay Semenov, Artem Riabinin, Sergey Bassenko, Anna Averchenkova, Julia Agafonova, Anastasia Aliaskina, Kirill Kozlov, Lev Novitskiy, Nikita Samsonov, Nikolai Gerasimenko, Denis Dimitrov

arXiv:2610.19290v1cs.CV

TL;DR

Audio-driven human animation must align speech with facial and body motion while preserving the identity of a person in a reference image. The open Lite and Pro models combine Qwen2.5-Omni text and audio features with reference-aware visual memory and are trained on a bilingual, single-speaker video corpus. Pro performs comparably to or better than the strongest open-source baseline and two commercial models on average in human evaluation, while distilled Pro leads both aggregate benchmark measures across EMTD, CelebV-HQ, and HDTF.

  • Problem

    Audio-driven human animation must align speech with facial and body motion while preserving the identity of a person in a reference image.

    Source: §1 · p. 3

  • Method

    The open Lite and Pro models combine Qwen2.5-Omni text and audio features with reference-aware visual memory and are trained on a bilingual, single-speaker video corpus.

    Source: §1 · p. 3, §7 · p. 23

  • Results

    Pro performs comparably to or better than the strongest open-source baseline and two commercial models on average in human evaluation, while distilled Pro leads both aggregate benchmark measures across EMTD, CelebV-HQ, and HDTF.

    Source: §6.3 · p. 22, §7 · p. 23

  • Takeaways & Limitations

    Kandinsky S2V 1.0 Lite (2B) and Pro (21B) generate long-form avatar video at 25 FPS from a reference image, speech, and a text prompt.

    Source: §7 · p. 23

    The corpus consists of publicly posted videos of real people, but the source videos, extracted clips, captions, and source URL lists are not distributed.

    Source: §2.4 · p. 7

Abstract

from arXiv · show

We present Kandinsky S2V 1.0, an open family of speech-to-video models for controllable audio-driven human animation, comprising Kandinsky S2V 1.0 Lite (2B parameters) and Kandinsky S2V 1.0 Pro (21B parameters). From a single reference image, a speech waveform, and a text prompt, the models generate long-form 25 FPS video in which lip motion, expression, pose, and gesture follow the speech. The architecture draws text and audio conditioning from one Qwen2.5-Omni model and adds a reference-aware three-scale visual memory. Training starts from the Kandinsky 5.0 Video foundation models [1] and combines mixed reference-memory layouts with an area-adaptive face loss. The training corpus contains 1.17M single-speaker clips selected by active-speaker detection, with captions generated directly in English and Russian. A side-by-side human evaluation against the strongest open-source baseline and two proprietary commercial models shows that Kandinsky S2V 1.0 Pro performs comparably to or better than these models on average and is also preferred for lip synchronization. On common speech to video benchmarks, Kandinsky S2V 1.0 Lite and Kandinsky S2V 1.0 Pro rank among the top performing models according to mean rank over five reported metrics. Beyond the base models, we provide {\pi}-Flow distilled variants of both Kandinsky S2V 1.0 Lite and Kandinsky S2V 1.0 Pro. The distilled Kandinsky S2V 1.0 Pro model generates video in 16 sampling steps without classifier-free guidance, achieving a 5.4 times end-to-end speedup over the base model, while matching or exceeding it on nearly all reported benchmark metrics. To accelerate open research and deployment in speech-to-video generation, we release the code, model checkpoints, and diffusers integration under the MIT license.

1 Introduction

Kandinsky S2V 1.0 introduces open 2B and 21B speech-to-video models for long-form avatar animation, combining Qwen2.5-Omni text and audio conditioning with reference-aware visual memory. Its evaluation reports strong benchmark aggregates and human comparisons, including a lip-synchronization preference for Pro.

1.2 Kandinsky S2V 1.0

  • A single Qwen2.5-Omni model supplies text and audio features, while a reference-aware three-scale visual memory carries context across chunks.Source: §1 · p. 3, §1.2 · p. 4

1.3 Contributions

  • The 2B Lite and 21B Pro models generate SD and HD video at 25 FPS with multilingual prompting and autoregressive long-form continuation.Both are publicly released.Source: §1.3 · p. 5

1.2 Kandinsky S2V 1.0

  • Training combines mixed reference-memory layouts, area-normalized face loss, and a curated multilingual corpus of 1.17M single-speaker clips captioned in English and Russian.Source: §1.2 · p. 4, §1.3 · p. 5

1.3 Contributions

  • Pro performs comparably to or better than the strongest open-source baseline and two commercial models on average, and is preferred for lip synchronization.Source: §1.3 · p. 5
  • Distilled Pro achieves the best mean rank and min-max score on EMTD, CelebV-HQ, and HDTF among evaluated models.Source: §1.3 · p. 5
  • The paper provides π-Flow distilled Lite and Pro variants; distilled Pro uses 16 sampling steps without classifier-free guidance and is 5.4× faster end-to-end than base Pro while matching or exceeding it on nearly all reported metrics.Source: §1.3 · p. 5

2 Data Processing Pipeline

The data pipeline builds a 1.17M-clip, bilingual corpus from two video sources, using active-speaker filtering, source-specific quality filtering, and cross-source deduplication. Captions describe only the target chunk when a clip supplies memory context; the source videos, clips, captions, and source URL lists are not distributed.

2.1 Data Sources

  • The corpus combines the Kandinsky 5.0 web collection, filtered for technical quality, and TalkVerse Panda-70M videos longer than five seconds, which receive no additional quality filtering.Source: §2.1 · pp. 5–6

2.2 Active-Speaker Detection and Clip Extraction

  • Both sources undergo LR-ASD filtering; web clips use the highest-scoring qualifying 10-second window, while TalkVerse clips come from the beginning of each video.Deduplication then runs jointly across the sources.Source: §2.2 · p. 6

2.3 Captioning

  • For 10-second training clips, only the second five seconds are captioned because the first half supplies memory context rather than the denoising target.Source: §2.3 · p. 6

2.4 Provenance and Exclusion of Evaluation Data

  • The corpus consists of publicly posted videos of real people, but its source videos, extracted clips, captions, and source URL lists are not distributed.Source: §2.4 · p. 7
  • Evaluation clips and matching benchmark identities are excluded before captioning, so reported scores assess generalization to unseen people rather than unseen clips of known people.Source: §2.4 · p. 7

2.5 Resulting Corpus

  • The final 1.17M-clip corpus is nearly evenly split between the web and TalkVerse sources; English accounts for slightly over half and Russian roughly a quarter.Training with bilingual captions supports Russian-language text conditioning without a dedicated language-specific fine-tuning stage.Source: §2.5 · p. 7, Figure 1 · p. 8

3 Kandinsky S2V 1.0 Architecture

Kandinsky S2V 1.0 uses text, speech, a reference image, and visual memory to generate 5-second, 25 FPS avatar-video chunks. Its Lite and Pro variants share a 3D Diffusion Transformer design with distinct model sizes and audio-injection schedules.

3 Kandinsky S2V 1.0 Architecture

  • The Lite (2B) and Pro (21B) models generate 125-frame, 25 FPS chunks conditioned on a reference image, speech, and text; later chunks also use prior generated frames.The variants differ in visual depth, model and feed-forward dimensions, text and audio stack depth, head dimension, and audio-injection schedule.Source: §3 · p. 8

3.1 Conditioning streams

  • Text prompts and speech are encoded by trainable DiT stacks from frozen Qwen2.5-Omni features, while a frozen VAE encodes the reference image and optional past-frame memory.The audio stream drives lip and head motion through audio blocks and V2A cross-attention; the text prompt conditions the visual stream and supplies a global style embedding.Source: §3.1 · p. 9

3.2 Latent video tokenization

  • A three-scale FramePack memory uses 24 oldest, 4 intermediate, and 2 newest latent frames, with coarser patches for older frames to reduce their token count.Memory, reference, and target tokens are joined into one visual sequence; the memory spans up to 30 latent frames, about 4.8 seconds of history.Source: §3.2 · pp. 9–10, Figure 3 · p. 10

3.3 DiT backbone

  • The 3D Diffusion Transformer uses text and audio encoders to prepare prompt and speech features, while a visual decoder generates video and attends to both through cross-attention.The visual decoder has 32 blocks in the 2B model and 60 in the 21B model; text and audio encoders use 2 and 4 blocks, respectively, in those models.Source: §3.3 · pp. 10–11

3.4 Reference image and memory placement

  • Training mixes memory-free steps with steps using past-frame memory, while placing reference and memory tokens at separated temporal RoPE positions to discourage reference-pose reuse at chunk boundaries.The memory-free regime is sampled with probability 0.75 and uses a nearby source-video reference; the memory-conditioned regime is sampled with probability 0.25 and places the reference farther in the past.Source: §3.4 · p. 11

3.5 Audio injection

  • Audio cross-attention is inserted in selected decoder blocks and aligns each target latent frame only with audio tokens from the corresponding time span; reference and memory tokens skip this layer.Audio is injected at 13 of 32 blocks in Lite and 18 of 60 blocks in Pro, with spacing narrowing toward the output.Source: §3.5 · p. 12, Figure 4 · p. 12

4.1 Continuous Pre-training

Continuous pre-training uses rectified-flow matching with an auxiliary face-region loss, initializing the DiT from Kandinsky 5.0 text-to-video checkpoints. The two model runs use multi-resolution training, shared optimization settings, and distributed computation.

4.1.1 Flow-matching objective

  • The model predicts the target velocity for a noisy video latent conditioned on text, audio, reference, and memory, and is trained with mean-squared error against that velocity.The noisy latent interpolates between the clean video latent and Gaussian noise; timestep sampling shifts probability toward high-noise states using scale s = 5.Source: §4.1.1 · pp. 12–13

4.1.2 Face loss

  • An adaptive face-area-based weight scales the auxiliary face loss, with masked-mean loss intended to stabilize face-region gradients across face sizes.With fixed weighting, large-face clips could dominate the face gradient and the authors observed tighter face framing during training; the adaptive weight is intended to balance the face term and preserve signal for small faces.Source: §4.1.2 · p. 13

4.1.3 Pretraining initialization

  • The DiT starts from corresponding Kandinsky 5.0 text-to-video checkpoints, while new conditioning layers and encoders are initialized from scratch and the Qwen, CLIP, and VAE modules remain frozen.All DiT blocks and newly introduced conditioning layers are trainable.Source: §4.1.3 · p. 13

4.1.4 Multi-resolution training

  • Training samples each step from 512 × 512, 768 × 512, or 512 × 768 resolution while keeping the clip length fixed at 125 frames.A resolution-aware batch accumulator keeps microbatch shapes consistent within each gradient-accumulation step.Source: §4.1.4 · p. 13

4.1.5 Optimization

  • Both runs use AdamW with a constant learning rate after linear warmup, bfloat16 parameters and outputs, float32 gradient reduction, and an EMA of weights for inference.The learning rate is 3 × 10^-5, and the EMA decay is 0.999.Source: §4.1.5 · pp. 13–14

4.1.6 Distributed training

  • The 21B model is trained across 256 devices using FSDP and tensor parallelism, while the 2B model uses 64 GPUs with FSDP.For the 21B run, tensor parallelism shards the visual sequence along height; activation checkpointing is applied to every visual decoder block.Source: §4.1.6 · p. 14

4.1.7 Training setup

  • The 2B run uses 14,000 steps and the 21B Pro run 9,500 steps, both at global batch size 128 and with the adaptive masked-mean face loss.Both runs use dense FlashAttention, independent text, audio, and reference dropout at probability 0.1, and memory-conditioned steps at probability 0.25.Source: §4.1.7 · p. 14, Table 3 · p. 14

4.2 Distillation

π-Flow distillation converts the Lite and Pro models into 16-step samplers without classifier-free guidance. The method imitates guided teacher trajectories, and the training setup uses a separate curated set of clips.

4.2 Distillation

  • Both Lite and Pro are distilled with π-Flow to generate in 16 sampling steps without classifier-free guidance.The distillation procedure is shared by both models.Source: §4.2 · p. 14

4.2.1 Method

  • The student predicts a five-point grid of clean-latent estimates per segment, interpolates between them, and is trained to match the guided teacher’s average velocity.The output projection is widened five-fold, with teacher weights replicated across the five grid points at initialization.Source: §4.2.1 · pp. 14–15
  • Imitation stays on teacher-trajectory states throughout training, rather than transitioning to states from the student’s own rollout.Each teacher step uses guided velocity to advance the state; the student learns the policy’s average velocity over that step.Source: §4.2.1 · p. 15

4.2.2 Training Setup

  • Distillation uses 9.5K curated single-speaker clips, while training retains memory dropout at 0.5 and disables text, audio, and reference dropout.The curated set is separate from the main corpus and excludes evaluation data by source video and ArcFace identity.Source: §4.2.1 · p. 15, §4.2.2 · p. 16

5 Inference

Inference generates long videos as successive 125-frame chunks, reusing a fixed visual-memory window and the same reference image. Distilled models reduce sampling to 16 network evaluations per chunk, and distilled Pro achieves a 5.4× end-to-end speedup in the reported SD setup.

5 Inference

  • Long-form generation composes 125-frame chunks, with later chunks reusing the reference image and memory latents from a fixed window of generated history.Memory is read before decoding, avoiding VAE re-encoding, and the fixed history window keeps the cache bounded as output length grows.Source: §5 · pp. 16–17
  • Base sampling uses 50 steps with conditional and unconditional evaluations per step, while distilled sampling uses 16 policy segments with no unconditional pass.Both use timestep shift s = 10; distilled models retain the same chunking, reference, and memory handling.Source: §5 · p. 17
  • At SD, eight-GPU Pro sampling time falls from 162 to 57 seconds per chunk, and at HD from 568 to 145 seconds.Lite sampling time on one GPU falls from 220 to 43 seconds per SD chunk; both models can generate HD at 25 FPS without further training.Source: §5 · p. 17, Table 5 · p. 17
  • 5.4× end-to-end speedup: distilled Pro generates a three-chunk SD video in 102.2 seconds versus 554.9 seconds for base Pro.With weights kept on eight H100 GPUs and torch.compile enabled; a two-chunk HD video takes 280 seconds versus 1563 seconds, a 5.6× reduction.Source: §5 · p. 17

6 Evaluation

Evaluation finds distilled Pro ranks first on aggregate benchmark measures across three datasets, while human comparisons show strengths in synchronization alongside trade-offs in visual quality and, for distilled Lite, synchronization. Distillation largely preserves Pro’s benchmark performance, with human comparisons showing model-specific differences.

6.1 Comparison with Competing Models

  • Distilled Pro ranks first by mean rank and min-max score across EMTD, CelebV-HQ, and HDTF among the evaluated models.It leads LongCat 1.5, and also achieves the highest CSIM on EMTD, highest IQA on CelebV-HQ and HDTF, and highest ASE on HDTF.Source: Table 6 · p. 18, §6.1 · p. 19

6.2 Human Evaluation

  • On the Russian set, Pro is favored for synchronization in all three comparisons, with 54%–62% of judgments versus 4%–15% for the other system.It is also preferred for motion naturalness and hand generation; visual quality is its only clear loss, against LongCat 1.5 at SD.Source: Figure 5 · p. 19, §6.2 · p. 20
  • On the multilingual set, Pro leads in synchronization and hand generation but trails both systems in visual quality.HeyGen Avatar IV is also preferred for motion naturalness, teeth and mouth generation, and identity preservation; motion naturalness against LongCat 1.5 is even.Source: §6.2 · p. 20, Figure 6 · p. 21
  • On the Russian set, Lite beats LongCat 1.5 in synchronization and teeth and mouth generation but loses on visual quality; against Pro, Lite loses on synchronization and visual quality.Most other criteria in the Lite-versus-Pro comparison are ties.Source: §6.2 · p. 20, Figure 7 · p. 21

6.3 Distillation Results

  • Distilled Pro leads both aggregate benchmark measures and improves CSIM, IQA, and ASE over base Pro on all three benchmarks, though Sync-C is slightly lower on EMTD and CelebV-HQ.Distilled Lite improves min-max scores on all three benchmarks, but has lower Sync-C and higher Sync-D than base Lite.Source: Table 6 · p. 18, §6.3 · p. 22
  • Against base Pro, distilled Pro is preferred for teeth and mouth generation, 38% to 3%, and is not worse on any of the seven human-evaluation criteria.For Lite, most judgments are ties, but base Lite retains an edge in synchronization, 10% to 0%; explanations proposed for visual gains remain untested hypotheses.Source: §6.3 · pp. 22–23, Figure 8 · p. 22, Figure 9 · p. 22

7 Conclusion

The conclusion reports strong benchmark and human-evaluation results for Kandinsky S2V 1.0, alongside long-form generation and a faster distilled Pro model. The ethical considerations identify misuse risks, deployment guidance, and limitations involving training-data bias and output accuracy.

7 Conclusion

  • The family ranks among leading models by mean rank and min-max score across EMTD, CelebV-HQ, and HDTF; distilled Pro leads both aggregates on all three.Source: §7 · p. 23
  • In side-by-side human evaluations, Pro performs comparably to or better than the strongest open-source baseline and two commercial models and is preferred for lip synchronization.Source: §7 · p. 23
  • Lite (2B) and Pro (21B) generate long-form 25 FPS video from a reference image, speech, and text prompt, using three-scale reference-aware memory.Source: §7 · p. 23
  • Both models were trained on 1.17M single-speaker clips captioned in English and Russian.Source: §7 · p. 23
  • Distilled Pro retains strong benchmark performance with 16 sampling steps and a 5.4× end-to-end speedup over base Pro.Source: §7 · p. 23

Ethical Considerations

  • Realistic video generation can be misused for deception, impersonation, disinformation, fraud, or harassment; the authors discourage such uses and specify prohibited depictions.Source: Ethical Considerations · p. 23
  • The authors encourage synthetic-content labeling, legal compliance, and safeguards such as moderation; intended uses include research, education, and creative work.Source: Ethical Considerations · p. 23
  • Training-data biases may include uneven representation across demographic groups, cultures, and languages, while outputs may contain factual, physical, or cultural inaccuracies.Source: Ethical Considerations · p. 23

Related papers

Loading 2610.19290v1…