PaperlayerView exact revision on arXiv

Source-linked AI summary

HeyGen Voice: Scene-Aware Voice Cloning for Talking Videos

Pavel Andreev, Andrei Gusev, Adithya Narayan, Yi Ren, Ivan Somov, Zijian Li, Sergey Demyanov, Viktor Goriachko, Alan Tan, Nik Nolte, Sam Prokopchuk, Andy Pan, David Chou, Kirill Leonov, Penghan Wang, Jiewei Yuan, Zhibin Hong, Rong Yan

arXiv:2610.30470v1cs.SD

TL;DR

Audio-only voice cloning carries a reference recording’s delivery and acoustics into new pictured scenes, while prior image-conditioned approaches do not jointly retain a chosen speaker and adapt to the scene. HeyGen Voice conditions acoustic generation on the image, captions, reference voice, and continuation state, and listeners preferred it over leading commercial systems across several evaluations.

  • Problem

    Audio-only cloning carries a reference recording’s delivery and acoustics into new scenes, while prior image-conditioned approaches do not jointly preserve a chosen speaker and adapt speech to the pictured setting.

    Source: §1 · p. 3

  • Method

    HeyGen Voice conditions a multimodal autoregressive transformer on the script, image, speaker and background captions, and reference recording, with clone policies and an audio anchor supporting scene adaptation and continuity.

    Source: §1 · p. 3, §4.3 · p. 11, §8 · p. 26

  • Results

    HeyGen Voice ranked first on the Artificial Analysis controlled-voice leaderboard (Elo 1201, October 2026); own-voice users chose it over ElevenLabs v4 and Cartesia in 62.2% and 54.1% of blind pairwise votes.

    Source: §8 · p. 26

  • Takeaways & Limitations

    Across a third-party arena, production votes, and talking-video studies, listeners preferred HeyGen Voice over leading commercial systems.

    Source: §8 · p. 26

    Image-designed voices are authored choices for pictured scenes, not recovery of the pictured person’s actual voice.

    Source: §7 · p. 26

Abstract

from arXiv · show

Most voice-cloning systems are built for audio alone: they copy the reference recording's delivery and acoustics, so a voice cloned on a phone sounds like a phone recording whether the speaker is pictured in a studio, on a street, or in a cathedral. Speech for a talking video should sound like the pictured person speaking in the pictured place. We present HeyGen Voice, a scene-aware voice-cloning model that separates who is speaking from how and where: it keeps the speaker's voice from an everyday recording while the target image sets the delivery and acoustics. Prior image-conditioned TTS does half of this job: it either adds scene acoustics without keeping a specific speaker or infers a voice from a face. HeyGen Voice is a multimodal autoregressive transformer conditioned on the image, speaker and scene captions, and the voice recording, trained in five stages that move from multilingual TTS pretraining through simulated and real cross-scene adaptation to reinforcement and preference optimization. Two clone policies trade scene adaptation against fidelity to the original recording, and an audio anchor holds long scripts in one voice and one acoustic environment. Listeners prefer HeyGen Voice over leading commercial systems across a third-party arena, production votes, and talking-video studies: it ranks first (Elo 1201) on the public Artificial Analysis controlled-voice TTS leaderboard (October 2026); in a blind test, users cloning their own voice chose it over ElevenLabs v4 and Cartesia Sonic 3.6 in 62% and 54% of votes; and volunteers rated talking videos of their own digital twin about 10 to 30 points higher (out of 100) when voiced by HeyGen Voice than by leading commercial alternatives. Try HeyGen Voice and hear samples at https://heygen.com/heygen-voice.

1 Introduction

Talking-video speech should match the pictured person and place, but audio-only cloning carries the reference recording’s delivery and acoustics into every scene. HeyGen Voice separates speaker identity from scene performance, and evaluations report listener preference over leading commercial systems.

1 Introduction

  • Prior approaches each leave part of the task unresolved: they control delivery without the picture, infer a voice without preserving a chosen speaker, add scene acoustics without that speaker, or generate with the picture without a chosen voice.The paper identifies scarce genuine same-speaker recordings across different scenes and acoustic ambiguity in long, chunked scripts as further challenges.Source: §1 · p. 3
  • HeyGen Voice keeps the reference speaker’s voice while adapting delivery and acoustics to the pictured scene.It conditions on the image, captions, and reference tokens, with a flow-matching decoder rendering the audio.Source: §1 · p. 3
  • The system offers two clone policies, a five-stage cross-scene training curriculum, and an anchor-based method for keeping long-form output consistent.The evaluation includes a 331-case cross-scene benchmark, a corrected ten-language content-fidelity benchmark, multilingual listening tests, production votes, a third-party arena, and talking-video studies.Source: §1 · pp. 3–4
  • On the Artificial Analysis controlled-voice leaderboard, HeyGen Voice ranked first with Elo 1201 in October 2026; in blind tests, own-voice users chose it over ElevenLabs v4 and Cartesia Sonic 3.6 in 62% and 54% of votes.Volunteers rating talking videos of their own digital twins scored both HeyGen Voice settings about 10 to 30 points higher out of 100 than Cartesia Sonic 3.6, ElevenLabs v4, and Seedance 2.5.Source: §1 · p. 4

2 Model Design

HeyGen Voice conditions acoustic generation on the script, target image, captions, reference voice, and continuation state. Its model separates inputs for speaker identity, scene performance, and acoustic continuity, then renders predicted acoustic codes as waveform audio.

2.1 Problem formulation

  • The problem formulation assigns the script to what is said, the image and captions to the intended performance and setting, and the reference and continuation context to the voice and any acoustics to retain.A clone policy determines whether output is designed for the pictured scene or continues the reference recording.Source: §2.1 · p. 5

2.2 Multimodal autoregressive acoustic model

  • A causal transformer processes a conditioning prefix, tokenized script, and acoustic codes as one sequence, with each conditioning source compressed into a fixed number of pooled tokens.The image tokens summarize the person and scene; separate caption tokens represent speaker performance and background environment.Source: §2.2 · pp. 5–6, Figure 2 · p. 6

2.3 Reference voice and continuity

  • Reference tokens carry timbre, accent, and cadence, while a consistency embedding of the selected anchor and preceding acoustic context maintain voice, delivery, and scene acoustics across chunks.The anchor is either the voice-design output or the reference recording, corresponding to the two clone policies.Source: §2.3 · pp. 6–7
  • Pitch variation controls vocal pitch movement, signal-to-noise ratio controls background level relative to speech, and ambient uniformity controls background steadiness across the clip.Table 2 describes the conditioning prefix as fixed-count pooled tokens per source, while script and continuation lengths vary.Source: §2.3 · p. 7, Table 2 · p. 7

2.4 Flow matching and waveform synthesis

  • A flow-matching model uses transformer hidden states and reference audio conditioning to synthesize continuous audio latents, which a waveform decoder renders as audio.Visual and caption information reaches the waveform through the transformer hidden states; the decoder itself has no image input.Source: §2.4 · p. 7

3 Training Strategy

HeyGen Voice trains in stages to separate speaker identity from scene and delivery, progressing from multilingual TTS through cross-scene adaptation to intelligibility and expressiveness optimization.

3 Training Strategy

  • Training progresses through five stages: multilingual TTS, cross-scene adaptation, measured intelligibility and speaker fidelity, then expressive delivery.The cross-scene stages teach the model not to treat the reference microphone, room, and performance as fixed speaker attributes.Source: §3 · p. 7

3.1 Data engine and infrastructure

  • HELIOS schedules more than 5,000 GPUs, while its data engine sustains over 200,000 concurrent tasks and an eight-stage audio pipeline builds adaptation data.The pipeline includes speaker clustering, cross-scene pair mining, and audits for false speaker matches and residual overlap.Source: §3.1 · pp. 7–8

3.2 Supervised TTS pretraining

  • Supervised TTS pretraining teaches the acoustic model to predict the next acoustic code from the script, preceding codes, and reference-voice conditioning.Image, caption, and continuity routes are introduced during adaptation.Source: §3.2 · p. 8

3.3 Simulated cross-scene adaptation

  • Simulated cross-scene adaptation perturbs the reference’s channel and timing while keeping the target and its image and captions fixed, separating speaker from scene conditioning.These perturbations cannot reproduce changes in intent, audience, or delivery between scenes.Source: §3.3 · pp. 8–9

3.4 Supervised fine-tuning on real cross-scene pairs

  • Real-pair fine-tuning uses same-speaker recordings from different videos, with the reference establishing identity and the target audio, image, and captions specifying performance and setting.Pair quality matters: mismatched speakers teach the wrong voice, while similar recording conditions offer little cross-scene adaptation signal.Source: §3.4 · p. 9

3.5 GRPO for intelligibility and speaker similarity

  • GRPO optimizes intelligibility and speaker similarity using ASR- and speaker-verification-scored candidate groups, with clean references used for this stage.Its clipped PPO objective includes a KL penalty to the reference policy.Source: §3.5 · p. 9

3.6 DPO adapter for expressiveness

  • DPO trains a low-rank expressiveness adapter first on judge preferences and then human preferences, while retaining the GRPO model as the adjustable base.The preference signal evaluates expressive delivery while checking that the reference speaker’s identity is preserved.Source: Figure 3 · p. 8, §3.6 · p. 9

4 Inference

Inference uses an image-guided workflow to design a scene-specific realization from a reusable voice reference, with clone policies and an anchor supporting control and continuity.

4.1 Cross-scene authoring workflow

  • For each scene, the user supplies an image, script, and delivery style; a language model writes speaker and background captions before speech synthesis.A separate video pipeline renders the talking video.Source: §4.1 · p. 10

4.2 Image-based voice design

  • Voice design combines the image, script, captions, and optional reference audio; its generated utterance becomes an anchor for subsequent scripts in the chosen voice and scene.Without reference audio, the image, captions, and text define a new voice; with reference audio, the speaker is preserved while scene and delivery follow the image and captions.Source: §4.2 · p. 10

4.3 Clone policies

  • voice-clone adapts delivery and acoustics to the pictured scene, while full-clone carries over the reference’s acoustic conditions and has the highest similarity to the original recording.Reference audio is capped at 60 seconds.Source: §4.3 · p. 11, Table 3 · p. 11

4.4 Expressiveness control

  • The expressiveness adapter adds a scaled low-rank update to transformer projections, and setting its strength to zero recovers the base model.The adapter strength is an inference control.Source: §4.4 · p. 11

4.5 Long-form synthesis and voice consistency

  • Long-form synthesis reuses the anchor-derived consistency embedding across chunks and updates a moving window of preceding acoustic context after each generated chunk.Together, these inputs target sequence-level consistency and local continuity.Source: §4.5 · p. 11

5 Evaluation

The evaluation spans audio benchmarks, listening tests, a public arena, and video studies, assessing speaker identity, scene adaptation, content fidelity, and fit between speech and the pictured person and scene. In a 50-scene study, reviewers favored HeyGen Voice for ambient fit against all three systems and across all measures against Cartesia, while other comparisons were mixed; participant comments reflected the trade-off between scene-matched delivery and fidelity to the original recording.

5.1 Cross-scene benchmark

  • voice-clone leads scene alignment (audio–audio 0.637; audio–image 0.282) and target similarity (0.590), while full-clone leads reference similarity (0.757).The benchmark compares 331 matched cases; the two policies trade adaptation to the target scene against preservation of the source recording.Source: §5.1 · p. 13, Table 5 · p. 13

5.2 Multilingual content fidelity

  • HeyGen Voice’s macro CER is 0.015%, near ElevenLabs v4 (0.013%), Qwen3-TTS (0.028%), and Inworld (0.035%), and below Cartesia (0.079%).The agreement-only CER counts script deviations confirmed by both ASR transcription and independent audio review; the authors treat CER as a content-fidelity diagnostic, not a quality measure.Source: §5.2 · pp. 13–14, Table 6 · p. 14

5.3 Human listening evaluation

  • voice-clone is preferred over ElevenLabs v4 (61.5 ±3.4%), Cartesia Sonic 3.6 (77.9 ±2.8%), and Inworld TTS-2 (68.1 ±3.3%) for narrator choice.In the fixed podcast-studio setting, it also leads on clarity and native accent, while full-clone preserves more reference similarity.Source: §5.3 · pp. 14, 16, Table 9 · p. 16
  • full-clone has the highest reference-similarity score (1144 ±17 Elo), while voice-clone leads narrator choice, clarity, and native accent; both HeyGen policies lie on the trade-off front.The studio-scene benchmark pits professional-narrator delivery against user-like references that are often noisy, reverberant, and conversational.Source: §5.3 · pp. 14, 17, Table 10 · p. 16, Figure 6 · p. 17

5.4 Production voice-clone preference

  • Users chose HeyGen Voice over ElevenLabs v4 in 62.2 ±1.1% of 8,332 votes and over Cartesia Sonic 3.6 in 54.1 ±1.2% of 7,286 votes.In the blind voice-cloning flow, users selected among unlabeled clones of their own voice; HeyGen Voice used full-clone.Source: §5.4 · p. 20, Table 11 · p. 20

5.5 Public arena ranking

  • HeyGen Voice in voice-clone ranks first on the controlled-voice leaderboard (1201 ±17), ahead of Qwen-Audio-3.1-TTS-Plus (1182 ±15).The October 9, 2026 snapshot compares delivery and quality with all systems cloning the same eight reference voices.Source: §5.5 · p. 20, Figure 9 · p. 21

5.6 Video-based evaluation

  • Participants described voice-clone as clear but sometimes less like themselves, while full-clone preserved recording traits; voice-clone scored slightly higher on average.This reflects the intended trade-off: voice-clone adapts delivery and acoustics to the pictured studio, while full-clone continues the recording as it is.Source: §5.6 · p. 23
  • The study tested scene placement across 50 varied environments using image-designed reference voices not recorded in target scenes and blind, counterbalanced video comparisons.Five reviewers compared each system pair on overall preference, acoustic fit, and ambient-sound fit; each comparison and aspect received 250 votes.Source: §5.6 · p. 23
  • Against Cartesia Sonic 3.6, HeyGen Voice won overall, acoustic-fit, and ambient-fit votes at 64.5%, 65.0%, and 73.3% (28, 27, and 36 scenes; all significant).The reported significance is after correction.Source: Table 14 · p. 23, §5.6 · p. 23
  • Against ElevenLabs v4, ambient fit favored HeyGen Voice at 69.7% (31 scenes won against 15, p = 0.002), while overall preference (50.7%) and acoustic fit (55.0%) were near parity.Source: Table 14 · p. 23, §5.6 · p. 23
  • Against Seedance 2.5, HeyGen Voice won ambient fit at 65.3% (31 scenes won against 14, p = 0.017), while overall preference (55.9%) and acoustic fit (52.5%) were indistinguishable from parity.For acoustic fit, 111 of 250 votes were judged the same.Source: Table 14 · p. 23, §5.6 · p. 23

6 Related Work

Related work spans zero-shot TTS architectures, language and visual controls, joint audio-video generation, and evaluation; HeyGen Voice combines specific-speaker preservation with image- and caption-guided scene control.

Zero-shot speech synthesis

  • HeyGen Voice uses an autoregressive planner over mel-derived codes and a diffusion-transformer flow decoder, adding image, caption, and continuity conditioning for talking videos.The related-work discussion contrasts this design with autoregressive and non-autoregressive TTS families.Source: §6 · p. 25

Reward and preference optimization for speech

  • Speech-model training uses automatic-reward reinforcement learning and preference optimization, including GRPO and DPO approaches for intelligibility, speaker similarity, expressiveness, and naturalness.HeyGen Voice applies GRPO rewards on the base model and DPO to a low-rank adapter, using judge-scored and human-preference pairs.Source: §6 · p. 25

Description, image, and scene control

  • Prior description-guided systems control delivery from text, while visual-acoustics methods use scene imagery and face-conditioned systems infer a plausible voice from a face.HeyGen Voice instead preserves a specific reference speaker while image and captions govern delivery and acoustics, with policies balancing those goals and a fixed realization across long scripts.Source: §6 · p. 25

Speech inside generated video

  • Speech generation research includes joint audio-video systems that render dialogue and ambience, while audio-driven animation systems generate a person from imagery and require a speech track from elsewhere.HeyGen Voice supplies such a track; the cited video studies evaluate whether scene-fitting speech improves coherence.Source: §6 · pp. 25–26

Evaluation

  • The evaluation follows standard ASR-based content-fidelity and speaker-similarity measures, adding cross-scene similarity measures, narrator-choice listening tests, own-voice user votes, and video studies.The cross-scene benchmark reports similarity to the target recording separately from similarity to the reference.Source: §6 · p. 26

7 Ethics and Safety

The paper describes consent, moderation, watermarking, and data controls for voice cloning, while clarifying that an image-designed voice is not a recovery of the pictured person's actual voice.

7 Ethics and Safety

  • The platform requires verified consent for voice cloning, allows the person to request voice removal, and moderates uploaded and generated content through automated and human review.Reference recordings, cached voices, and evaluation assets are governed by access, retention, and deletion controls.Source: §7 · p. 26
  • AudioSeal watermarking supports provenance detection, and the paper states that a voice designed from an image is an authored choice rather than recovery of the pictured person's actual voice.The authors internally test whether irrelevant visual attributes affect accent or perceived identity and whether explicit captions remain effective across appearances.Source: §7 · p. 26

8 Conclusion

HeyGen Voice uses a reference voice and target scene information to preserve speaker identity while adapting delivery and acoustics. Its evaluation reports listener preference over leading commercial systems across arena, production, and talking-video settings.

8 Conclusion

  • HeyGen Voice keeps the speaker's voice while adapting delivery and acoustics to a target image and captions, using a five-stage training curriculum.The curriculum progresses from supervised TTS pretraining through simulated and real cross-scene adaptation, GRPO, and a DPO expressiveness adapter.Source: §8 · p. 26
  • Two clone policies trade scene adaptation against preserving the reference recording, while an anchor-derived consistency embedding and local acoustic context maintain one voice and room across long-form output.Source: §8 · p. 26
  • HeyGen Voice ranks first on the Artificial Analysis controlled-voice leaderboard (Elo 1201, October 2026), and users chose it over ElevenLabs v4 and Cartesia in 62.2% and 54.1% of blind pairwise votes.Volunteers rated its talking videos about 10 to 30 points above Cartesia Sonic 3.6, ElevenLabs v4, and Seedance 2.5.Source: §8 · p. 26
  • The model separates speaker identity from the delivery and room that follow the pictured scene.Source: §8 · p. 26

Contributions and Acknowledgments

The authors frame the work as spanning model research, training data, evaluation, and production engineering, and acknowledge participants and colleagues who supported evaluation and deployment.

Contributions and Acknowledgments

  • The work brings together contributions in model research, training data, evaluation, and production engineering.Source: Contributions and Acknowledgments · p. 27
  • The authors thank listening- and video-study participants and colleagues who supported evaluation and deployment.Source: Contributions and Acknowledgments · p. 27
  • The listed authors are Pavel Andreev, Andrei Gusev, Adithya Narayan, Yi Ren, Ivan Somov, Zijian Li, Sergey Demyanov, Viktor Goriachko, Alan Tan, Nik Nolte, Sam Prokopchuk, Andy Pan, David Chou, Kirill Leonov, Penghan Wang, Jiewei Yuan, Zhibin Hong, and Rong Yan.Source: Contributions and Acknowledgments · p. 27

Related papers

Loading 2610.30470v1…