Source-linked AI summary
Towards Automatic Face-to-Face Translation
Prajwal K R, Rudrabha Mukhopadhyay, Jerin Philip, Abhishek Jha, Vinay Namboodiri, C. V. Jawahar
TL;DR
The paper addresses translating talking-face videos across languages while retaining realistic lip synchronization, beyond existing speech-to-speech translation. It builds a modular speech-to-speech pipeline and adds LipGAN for audio-conditioned face generation, reporting superior LipGAN evaluations and improved user experience in multimodal applications.
Problem
Existing systems translate audio-visual content only at the speech-to-speech level, producing voice mismatch and unsynchronized lips when translated speech is overlaid on original video.
Method
The authors combine speech recognition, neural translation, speech synthesis, voice transfer, and LipGAN to generate target-language talking-face videos.
Results
LipGAN achieves state-of-the-art scores across all quantitative metrics on the LRW test set and outperforms existing works in quantitative and human-based evaluation.
Takeaways & Limitations
The pipeline demonstrates face-to-face translation and supports proof-of-concept applications including multimodal content consumption and automatic dubbing.
Abstract
from arXiv · showhide
In light of the recent breakthroughs in automatic machine translation systems, we propose a novel approach that we term as "Face-to-Face Translation". As today's digital communication becomes increasingly visual, we argue that there is a need for systems that can automatically translate a video of a person speaking in language A into a target language B with realistic lip synchronization. In this work, we create an automatic pipeline for this problem and demonstrate its impact on multiple real-world applications. First, we build a working speech-to-speech translation system by bringing together multiple existing modules from speech and language. We then move towards "Face-to-Face Translation" by incorporating a novel visual module, LipGAN for generating realistic talking faces from the translated audio. Quantitative evaluation of LipGAN on the standard LRW test set shows that it significantly outperforms existing approaches across all standard metrics. We also subject our Face-to-Face Translation pipeline, to multiple human evaluations and show that it can significantly improve the overall user experience for consuming and interacting with multimodal content across languages. Code, models and demo video are made publicly available. Demo video: https://www.youtube.com/watch?v=aHG6Oei8jF0 Code and models: https://github.com/Rudrabha/LipGAN
1 INTRODUCTION
The paper extends speech-to-speech translation to face-to-face translation, motivated by increasingly visual communication and the poor lip synchronization of translated audio over original video. It combines speech, language, voice, and visual modules, introducing LipGAN for realistic target-language talking faces.
- Motivation: Existing systems can translate speech but overlaying translated audio on the original video produces unsynchronized lip movements.Translated speech also sounds different from the original speaker’s voice.
- Pipeline: The proposed pipeline recognizes source speech, translates the text, synthesizes target-language speech, personalizes the voice, and generates a talking face.The system is developed for English and Hindi in this paper.
- LipGAN: LipGAN generates realistic talking faces conditioned on audio in any language and uses an adversary to measure lip synchronization.The model also handles random face poses without template-pose realignment.
- Contributions: The complete system automatically translates a person’s talking face into a target language with realistic lip synchronization.The authors describe this as the first automatic pipeline for face-to-face translation.
- Results: LipGAN outperforms existing works in quantitative and human-based evaluation, while the pipeline achieves state-of-the-art Hindi-English neural machine translation results.The introduction reports these outcomes without providing the underlying LipGAN metric values.
2 BACKGROUND
Face-to-face translation combines established speech and language components with talking-face generation to convert a source-language video into a synchronized target-language video. LipGAN addresses generalization and blending challenges by conditioning generation on audio and target-face pose.
- Face-to-Face Translation: Face-to-face translation aims to generate a lip-synchronized video of a speaker speaking in target language LB from a source-language video in LA.The system combines modules from speech, vision, and language.
- Pipeline: The pipeline recognizes source speech, translates the text, synthesizes target speech, optionally transfers the speaker’s voice, and generates the target-language face.Figure 2 specifies English as LA and Hindi as LB for the paper’s setting.
- Prior Work: Prior talking-face methods struggle with unseen speakers, unseen audio, or canonical-pose requirements, limiting generalization and blending into original videos.The related work describes these as limitations of existing approaches.
- LipGAN: LipGAN generates realistic talking faces from audio and accepts target-face pose information to preserve pose during generation.Its design uses additional pose information rather than normalizing every input face to a canonical pose.
3 SPEECH-TO-SPEECH TRANSLATION
The speech-to-speech system adapts existing recognition, translation, synthesis, and voice-transfer models for English-Hindi translation. The reported evaluations cover translation quality, Hindi speech quality, and personalization of the synthesized speaker voice.
- System Components: The system uses a pretrained Deep Speech 2 recognizer for English and adapts state-of-the-art neural translation and speech-synthesis architectures for Indian languages.The paper frames this adaptation as necessary because few speech and language systems are available for Indian languages.
- Translation: 3 BLEU points is the reported increase over Google Translate on the IIT-Bombay Hindi-English Parallel Corpus test set.The comparison uses BLEU scores and identifies Google Translate as Online-G.
- Speech Synthesis: The Hindi text-to-speech model adapts DeepVoice 3 and is trained on about 10,000 audio-text pairs, with quality evaluated through a 25-participant MOS study.The Hindi dataset was curated from recorded sentences based on crawled news articles.
- Evaluation: The Hindi TTS MOS is reported as comparable to the same architecture trained on the LJSpeech English TTS dataset.The comparison is stated in the Table 2 caption without numerical MOS values in the supplied passages.
- Voice Personalization: CycleGAN voice transfer personalizes the synthesized Hindi speech using non-parallel target-speaker audio while preserving linguistic features.The method uses about 10 minutes of speaker audio and synthetic TTS speech for each speaker.
- Evaluation: Voice-transfer quality is evaluated with 30 participants across 20 generated samples transferred to five famous personalities.The Table 3 caption reports consistently reasonable cross-language voice transfer.
4 TALKING FACE GENERATION
The section formulates talking-face generation as producing synchronized speech frames for unseen languages and faces, using LipGAN’s adversarial synchronization objective and pose-aware generator design.
- Problem: LipGAN generates a sequence of face frames from a face image and segmented speech while targeting proper lip synchronization for unseen languages and faces.The model is also intended to learn self-supervisedly because annotated multilingual data is difficult to collect.
- Method: The model uses a generator conditioned on audio and a discriminator that tests whether generated faces and audio are synchronized.Adversarial training encourages photo-realistic faces that remain accurately synchronized with the input audio.
- Face encoder and decoder: Pose information is supplied during training so the decoder changes the mouth shape while better preserving the input face’s pose and expression.This design enables seamless pasting of the generated face crop into the video without artefacts caused by uncontrollable pose variations.
- Generator network: The generator combines face and audio encoders, then decodes their joint embedding into a talking face through masked-region inpainting.The audio encoder produces an embedding from an MFCC heatmap, while the decoder reconstructs the mouth region and preserves facial features through six skip connections.
- Joint training of the GAN framework: The discriminator receives synthetic, synchronized real, and unsynchronized real samples so it learns to evaluate synchronization rather than image quality alone.The generator is trained to reconstruct the target face with an L1 loss within the joint GAN objective.
D La(G, D) + LRe (6)
The paper evaluates LipGAN through quantitative metrics, visual comparisons, activation heatmaps, and human studies, then assesses the complete face-to-face translation pipeline.
- Quantitative evaluation: LipGAN is evaluated on the LRW test set using PSNR, SSIM, and Landmark distance, with random speaker frames preventing lip-information leakage.Inference uses 350ms audio segments and masked current frames to provide pose information without revealing the target lip movements.
- Visual comparison: LipGAN produces consistent, accurate, natural talking faces across languages, unlike baselines with poor lip synchronization or unnatural lip movements.
- Quantitative evaluation: LipGAN achieves significant improvements over existing competitive approaches across all standard quantitative metrics.
- Discriminator analysis: The lip-sync discriminator makes LipGAN highly attentive to lip and mouth regions and provides direct lip-synchronization supervision absent in prior works.
- Human evaluation: LipGAN receives significantly higher human-evaluation scores than existing works for both realistic rate and extent of lip synchronization.The study uses 150 generated samples, three models, and 20 participants rating videos from 1 to 5.
- Complete pipeline evaluation: Adding each automatic module increases user-experience scores, and the complete system significantly outperforms traditional text-based and speech-based translation systems.The complete pipeline is compared with subtitles, automatic dubbing, voice transfer, lip synchronization, and human dubbing variants.
5 APPLICATIONS
The face-to-face translation framework is demonstrated for movie dubbing, educational videos, and cross-language access to interviews, while complex movie speech remains challenging.
- 5.1 Movie dubbing: The pipeline can synthesize and synchronize lips in manually dubbed movie videos, automatically correcting dubbing errors.
- 5.1 Movie dubbing: Automatic dubbing of complex movie speech still requires significant improvements to the speech-to-speech pipeline for realistic results.
- 5.2 Educational videos: Face-to-face translation can dub educational videos while preserving lip synchronization, avoiding the visual discrepancy created by speech-to-speech dubbing.
- 5.3 Interviews: The system could let viewers consume interviews and other important information in a chosen language irrespective of the source language, with lip synchronization.
6 CONCLUSION
The paper extends machine translation to talking-face videos, introduces LipGAN, and demonstrates applications while identifying several open research directions.
- 6 CONCLUSION: The work extends automatic machine translation to audio-visual face-to-face translation, where both input and output are talking-face videos.
- 6 CONCLUSION: It introduces a talking-face generation approach, contributes to language-processing tasks for resource-constrained languages, and demonstrates practical applications.
- 6 CONCLUSION: Translation changes speech duration, creating open challenges for transforming corresponding gestures, expressions, and background content.