Source-linked AI summary
Text-based Editing of Talking-head Video
Ohad Fried, Ayush Tewari, Michael Zollhöfer, Adam Finkelstein, Eli Shechtman, Dan B Goldman, Kyle Genova, Zeyu Jin, Christian Theobalt, Maneesh Agrawala
TL;DR
Changing speech in talking-head video while preserving seamless audio-visual flow is difficult. The paper edits transcripts, selects and stitches suitable facial-motion snippets, and renders photorealistic speech video; it demonstrates diverse edits, translation, and sentence synthesis, with 59.6% of edits rated real in a user study.
Problem
Existing tools and text-based editing methods cannot synthesize new audio-visual speech content, making some transcript changes require re-shooting or overdubbing.
Method
The method uses transcript edits, phoneme and viseme alignment, dynamic-programming snippet selection, parameter blending, and recurrent neural rendering to synthesize edited talking-head video.
Results
59.6% of pipeline-generated edits were rated real in a crowd-sourced user study, alongside demonstrations of word edits, translation, and full-sentence synthesis.
Takeaways & Limitations
The approach provides a first step toward fully text-based editing and synthesis of general audio-visual content.
Takeaways & Limitations
The system requires a re-timed background video, which can misalign eye blinks and gestures with speech, and phoneme retrieval is agnostic to mood.
Abstract
from arXiv · showhide
Editing talking-head video to change the speech content or to remove filler words is challenging. We propose a novel method to edit talking-head video based on its transcript to produce a realistic output video in which the dialogue of the speaker has been modified, while maintaining a seamless audio-visual flow (i.e. no jump cuts). Our method automatically annotates an input talking-head video with phonemes, visemes, 3D face pose and geometry, reflectance, expression and scene illumination per frame. To edit a video, the user has to only edit the transcript, and an optimization strategy then chooses segments of the input corpus as base material. The annotated parameters corresponding to the selected segments are seamlessly stitched together and used to produce an intermediate video representation in which the lower half of the face is rendered with a parametric face model. Finally, a recurrent video generation network transforms this representation to a photorealistic video that matches the edited transcript. We demonstrate a large variety of edits, such as the addition, removal, and alteration of words, as well as convincing language translation and full sentence synthesis.
1 INTRODUCTION
The paper introduces transcript-based editing for talking-head video, addressing the difficulty of changing speech while preserving seamless audio-visual flow. Its system synthesizes new dialogue from text edits and supports diverse edits, while raising ethical concerns about misuse.
- Motivation: Existing editing tools require skilled manual selection and careful cut placement to minimize disruptions in talking-head video.Conventional tools can cut and assemble footage, but editors must scrub raw video and manage audio-visual continuity.
- Prior limitations: Prior text-based editing can move or delete segments but produces artifact-free transitions only at constrained, well-behaved boundaries.The method is limited, for example, when the subject remains still between phrases or sentences.
- Contribution: Unlike conventional and earlier text-based tools, the proposed method synthesizes new audio-visual speech content from transcript edits, including difficult mid-utterance transitions.The system completes transcript-based operations by generating convincing new video rather than only rearranging existing segments.
- Contribution: The pipeline selects and stitches motion snippets using dynamic programming, blends head parameters, and uses recurrent neural rendering to produce photorealistic video.The contributions include inserted text, viseme-sequence assembly, seamless parameter blending, and conversion of composite inputs into photorealistic frames.
- Results: The authors demonstrate word addition, removal, and alteration, as well as translation and full-sentence synthesis, with edits rated real in 59.6% of user-study cases.The reported user study provides a realism result for the demonstrated edits.
- Ethical considerations: The authors warn that highly realistic manipulation can enable falsified statements and slander, motivating disclosure, watermarking, and performer permission.They frame these safeguards as measures intended to reduce misuse while preserving creative applications.
2 RELATED WORK
The related work spans facial reenactment, visual dubbing, speech animation, transcript-based editing, audio synthesis, generative models, and monocular face reconstruction. The paper distinguishes its approach by synthesizing photorealistic video of a real speaker from text edits alone.
- Facial reenactment: Facial reenactment methods control a target face’s pose, expression, gaze, or lip motion, providing the visual basis for the paper’s editing system.The paper uses facial reenactment and neural face rendering to visualize text-based edits.
- Visual dubbing: Visual dubbing transfers facial motion across languages or performers, but prior speech-driven methods may require new audio resembling the original speaker.The paper instead synthesizes new video using text-based edits and supports phenomena such as lip rolling in more general settings.
- Speech animation for rigged models: Speech-animation methods produce speech curves for rigged 3D characters, whereas this approach animates a real person from text and a monocular recording.The distinction is between artist-supplied character rigs and photorealistic video of a recorded subject.
- Text-based video and audio editing: Transcript-based tools support audio editing, review, segmentation, summaries, or rearrangement, but the proposed system also synthesizes new video by typing text.This capability differs from prior tools that primarily cut, copy, paste, or assemble existing material.
- Audio synthesis: Audio synthesis can accompany transcript-based video editing, and the paper’s video method is independent of audio so it can use multiple TTS approaches.The related work contrasts parametric TTS with unit selection and mentions coupling the method with VoCo for new words.
- Generative models and face reconstruction: The system combines a parametric 3D face model with neural rendering to generate realistic edited imagery from tracked facial geometry and appearance.Monocular face reconstruction supplies model-based geometry and appearance representations used in reenactment and visual dubbing.
3 METHOD
The method converts transcript edits into realistic talking-head video by matching phoneme and viseme sequences, blending tracked 3D face parameters, and neural-rendering the edited motion. This pipeline supports text-only additions, rearrangements, and deletions while producing seamless localized edits.
- 3 METHOD: The system accepts a talking-head video, transcript, and transcript edit operations for adding, rearranging, or deleting consecutive words.It represents each edit as the edited word sequence plus its correspondence to the original transcript.
- 3 METHOD: It preprocesses each input video by aligning transcript phonemes and registering a 3D parametric head model with every frame.The model exposes head pose, geometry, reflectance, expression, and scene illumination for later selective blending.
- 3 METHOD: Viseme search partitions the edited phoneme sequence and uses modified Levenshtein matching to select input-video subsequences with similar phoneme and viseme patterns.The matching cost distinguishes identical phonemes, same-viseme substitutions, and different-viseme substitutions, while accounting for phoneme lengths; dynamic programming encourages longer subsequences.
- 3 METHOD: Selected snippets are retimed in parameter space and blended because direct concatenation can mismatch timing and create visible jump cuts from pose or posture changes.The blending strategy combines tracked 3D-head parameters rather than directly joining the original frames.
- 3 METHOD: The renderer masks the lower face, composites a parametric synthetic lower face onto background video, and uses a recurrent neural network to produce photorealistic frames matching the edited motion.Training combines photometric reconstruction with spatial and temporal adversarial losses; the generator is shared across time steps.
- 3 METHOD: The resulting localized edits seamlessly blend into the original video, making the subject appear to speak the new phrase from text-derived processing.The pipeline renders photo-realistic frames that match the modified parameter sequence.
4 RESULTS
The approach produces photorealistic, temporally coherent talking-head edits across deletion, rearrangement, insertion, translation, and full-sentence synthesis, while seamlessly recomposing modified segments into original footage.
- Runtime Performance: 110ms per frame is reported for 3D face reconstruction and 132ms per frame for neural face rendering.For a one-hour recording, phoneme alignment takes 20 minutes and viseme search takes 10 minutes to 2 hours for insertion operations.
- 4.1 Video Editing: The system supports moving and deleting phrases, adding unspoken words, and replacing words with photorealistic results and good audio-to-video alignment.It also synthesizes a photorealistic mouth interior with highly detailed teeth.
- 4.1 Video Editing: The neural rendering backend conditions video generation on the background, enabling dynamic-background handling that the DVP baseline lacks.This conditioning also helps preserve independently moving foreground elements such as hands.
- 4.1 Video Editing: The approach seamlessly recomposes modified segments into original full-frame footage and blends new segments into longer videos across arbitrary framing, resolution, and aspect ratio.Localized edits can preserve most of the original video and fit into a standard editing pipeline.
- 4.2 Translation: The method supports video translation when source material contains similar visemes to the target language, demonstrated by making an English speaker appear to speak German.Its viseme search is language agnostic and requires phoneme conversion for a new language.
- 4.3 Full Sentence Synthesis Using Synthetic Voice: The system synthesizes full sentences from a synthetic voice and supports adaptive instruction or storytelling videos for different audiences.The paper connects this capability to producing video for arbitrary voice-assistant utterances.
5 EVALUATION, ANALYSIS & COMPARISONS
The evaluation examines data requirements, synthesized-phrase behavior, parameter blending, comparisons with existing methods, ablations, and perceptual realism. Results show graceful degradation with less data, temporally coherent synthesis, advantages over comparison methods, and edits rated real in many cases.
- 5.1 Size of Input Video: More training data improves phoneme-retrieval quality, but results degrade gracefully as the dataset shrinks from 100% to 5%.Best results were obtained with the full dataset.
- 5.1 Size of Input Video: 0.018 mean RMSE at 100% training data, 0.019 at 50%, and 0.021 at 5% indicate the neural renderer needs less data than viseme retrieval.The result suggests certain edits, such as deletion, can be performed on shorter videos.
- 5.2 Synthesized Phrases: Phrase length does not directly predict synthesis quality; viseme composition and phoneme alignment quality have greater influence.This finding comes from synthesizing varied phrases in “I love the smell of X in the morning.”
- 5.3 Parameter Blending: Parameter-space blending prevents temporal instability, replacing abrupt mouth closure with smooth transitions between frames.Without blending, the editing results are temporally unstable.
- 5.4 Comparison to MorphCut: Our method removes jump cuts under head motion where MorphCut fails, and produces seamless word additions where MorphCut creates large head jumps.MorphCut requires relatively still performers and cannot synthesize new words.
- 5.5 Comparison to Facial Reenactment Techniques: Compared with Deep Video Portraits and Face2Face, the method produces more detailed, temporally coherent mouth imagery and better handles dynamic foregrounds and backgrounds.The ablation shows that mouth masking and background conditioning positively contribute, especially to teeth detail and mouth quality.
- 5.7 User Study: 59.6% of edited clips were rated “real” by 138 participants, compared with 80.6% for real base videos and 82.2% for real target videos.The study collected 2993 responses using a 5-point Likert scale.
6 LIMITATIONS & FUTURE WORK
The method has several practical scope boundaries and open directions, including dependence on retimed background video, mood-insensitive retrieval, slow search, and substantial data requirements.
- 6 LIMITATIONS & FUTURE WORK: The synthesis approach requires a re-timed background video, which can misalign eye blinks and gestures with the speech.The authors mitigate this by using a longer retiming region with a smaller retiming factor.
7 CONCLUSION
The paper presents transcript modification as a way to edit talking-head video across additions, removals, alterations, translation, and sentence synthesis. It frames this as an initial step toward fully text-based editing and synthesis of general audiovisual content.
- 7 CONCLUSION: The approach supports adding, removing, and altering words, convincing language translation, and full sentence synthesis through transcript edits.The authors describe it as the first approach enabling text-based editing of talking-head video by modifying its transcript.
A PHONEME & VISEME CONTENT
The analysis finds that long exact phoneme or viseme matches are uncommon even in phonetically rich input, motivating retrieval from shorter subsequences and parameter blending.
- A PHONEME & VISEME CONTENT: Exact matches longer than 4–6 visemes or 3–5 phonemes are rare in TIMIT.Figure 17 uses leave-one-out matching over subsequences of lengths 1–10.
- A PHONEME & VISEME CONTENT: Variation in viseme duration across instances motivates incorporating viseme distance and length into search and blending.The analysis observes variation both between different visemes and within a viseme class.