PaperlayerView exact revision on arXiv

Source-linked AI summary

No Distillation Needed: Single-Pass Real-Time Talking Heads via Acausal Noise Shaping

Yu Han, Dejan Markovic, Alexander Richard, Wojciech Zielonka, Akshay Venkatesh, Cheng-hsin Wuu, Michael Zollhoefer

arXiv:2610.11070v1cs.CV

TL;DR

Streaming facial animation needs frame-level generation from audio observed up to the current time, while diffusion's iterative sampling is costly online. FaceGAN uses a single-pass generator with acausal noise shaping; it matches compared baselines at low latency, with future synthetic noise sampled without delaying the audio path.

  • Problem

    Diffusion models produce high-fidelity, diverse motion but require tens to hundreds of evaluations per frame, while causal time-invariant generators driven by i.i.d. noise face limits on suppressing output spectrum.

    Source: §1 · p. 2, §3 · p. 4

  • Method

    FaceGAN uses a single-pass generator with acausal shaping of synthetic noise, keeping the audio-to-expression pathway causal without audio latency.

    Source: §1 · p. 2

  • Results

    FaceGAN matches compared baselines at 43 ms latency and 12× real time on a single GPU.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Because the driving noise is synthetic, its future can be sampled now, so acausal noise shaping does not add latency to the causal audio pathway.

    Source: §1 · p. 2

    The spectral limitation is stated for causal, time-invariant generators driven by i.i.d. noise.

    Source: §3 · p. 4

Abstract

from arXiv · show

Audio-driven facial animation underpins real-time avatars, telepresence, and embodied virtual agents. And it must run online: each frame emitted from audio observed up to the current time, at interactive rates. Recent progress is dominated by diffusion models, which need many network evaluations per sample and are therefore a poor fit for streaming. We argue the cost is unnecessary in this domain. Audio-conditioned facial motion occupies a comparatively low-dimensional manifold, a regime where a single-pass GAN suffices. The obstacle is not capacity but stochastic structure. We show that a causal, time-invariant generator driven by i.i.d. noise cannot suppress its output spectrum over a band without collapsing its per-step innovation. We proposed FaceGAN, which dissolved the limitation by shaping the noise pathway acausally. Because the driving noise is synthetic, its future can be sampled now, so the audio-to-expression path stays causal, and the model supports fully causal operation. FaceGAN emits expression and head pose in a single forward pass per frame and matches or outperforms state-of-art approaches in generation quality. Being feed-forward with bounded attention windows, it generates indefinitely without drift.

1 INTRODUCTION

The introduction argues that iterative diffusion sampling is unnecessary for low-dimensional audio-driven facial motion, but causal single-pass generation faces a stochastic-structure constraint. FaceGAN addresses it with latency-free acausal noise shaping and matches compared baselines in single-pass generation.

1 INTRODUCTION

  • Diffusion models generate high-fidelity, diverse motion but require tens to hundreds of network evaluations per frame, poorly fitting streaming inference.The introduction frames online frame emission from current audio and small lookahead as the deployment constraint.Source: §1 · p. 2
  • Audio-conditioned facial expression lies on a comparatively low-dimensional manifold, where a single-pass GAN can match or exceed diffusion quality at low latency.The paper presents this regime as one where the advantage motivating iterative refinement largely disappears.Source: §1 · p. 2
  • A causal, time-invariant generator driven by i.i.d. noise cannot suppress output spectrum over a band without collapsing per-step innovation.The paper attributes the resulting jitter-versus-muted-motion trade-off to the generated process, not network capacity.Source: §1 · p. 2
  • Acausal shaping of synthetic noise lets the audio-to-expression path remain causal because required future noise can be sampled now.The approach drives the generator with non-i.i.d. noise and adds no audio latency.Source: §1 · p. 2
  • FaceGAN combines a single-pass GAN with noise shaping to match or surpass diffusion baselines at a fraction of their inference latency.The introduction presents the system as the combination of single-pass efficiency and the noise-shaping technique.Source: §1 · p. 2
  • FaceGAN matches compared baselines at 43 ms latency and 12× real time on a single GPU.The paper presents it as a single-pass system for audio-driven facial expression generation.Source: §1 · p. 2

2 RELATED WORK

Related work spans multiple facial-motion representations and generative approaches, while highlighting the challenge of streaming diverse motion. The paper positions a single-pass GAN with acausal noise shaping as a frame-level streaming alternative to iterative or sequential methods.

2.1 AUDIO-DRIVEN FACIAL MOTION GENERATION

  • Facial-motion generation spans 2D portrait video, 3D mesh templates, and neural rendering with NeRF or Gaussian Splatting.These approaches target different output representations for synthesizing facial movement from speech.Source: §2.1 · p. 2
  • Only FaceGAN is both single-pass and frame-level streaming among the compared methods.The table defines a live system as frame-level streaming with low latency, not throughput alone.Source: Table 1 · p. 3
  • Deterministic regression can smooth away stochastic detail; diffusion and autoregressive methods restore diversity but incur iterative, chunk-based, or token-sequential inference costs.The cited streaming-oriented diffusion methods operate at chunk granularity, while autoregressive methods decode tokens sequentially.Source: §2.1 · p. 3

2.2 GENERATIVE MODELS FOR MOTION: DIFFUSION VS. SINGLE-PASS GENERATOR

  • Diffusion's many sequential evaluations motivate efforts to compress it toward one step, while GANs can be strong on lower-dimensional or structured domains under limited data.The paper describes audio-conditioned expression and head pose as a comparatively low-dimensional manifold with scarce paired data.Source: §2.2 · p. 3
  • The paper locates the obstacle to cheap single-pass motion generation in the spectral limits of causal i.i.d.-noise inputs, not an inherent GAN weakness.Acausal noise shaping removes that limitation without audio-side latency.Source: §2.2 · pp. 3–4

3 SPECTRAL LIMITATION OF CAUSAL GENERATION

The section establishes that any causal, time-invariant generator using fresh i.i.d. noise cannot produce an exact spectral stopband while retaining per-step innovation. Deeper suppression over a wider band further limits innovation, while acausal shaping of synthetic noise escapes this constraint without audio lookahead.

3 SPECTRAL LIMITATION OF CAUSAL GENERATION

  • The impossibility argument concerns a causal, time-invariant mapping from i.i.d. noise to output motion, with the generator allowed to be any measurable function.The section frames the target as motion with energy concentrated at low frequencies and none above a cutoff.Source: §3 · p. 4

3.1 NO PERFECT SPECTRAL STOPBAND

  • Any causal generator that uses current i.i.d. noise with nonzero conditional innovation has output spectral density greater than zero almost everywhere, ruling out a perfect stopband.The result applies to arbitrary measurable generators, not only a particular architecture.Source: §3.1 · p. 4, §3 · p. 4

3.2 QUANTITATIVE TRADE-OFF: STOPBAND DEPTH VS. INNOVATION

  • For a 40 dB stopband across 90% of the spectrum, at most approximately 0.025% of input noise variance survives as new per-step information.The bound follows when the output spectrum is scaled to peak at 1.Source: §3.2 · p. 4
  • Acausal preprocessing makes future-shaped noise predictable from its past, removing the i.i.d.-noise condition that binds the downstream causal generator, without audio lookahead.Future noise can be sampled in advance because it is synthetic.Source: §3.2 · p. 4

4 METHOD

FaceGAN combines speech embeddings with a temporally shaped noise prior to generate expression and head-pose motion in a single pass. Its bounded-attention generator is paired with multiscale discriminators and an objective that encourages realistic, varied expression while supervising pose.

4 METHOD

  • The system maps frozen Mimi audio embeddings and a noise latent to 137-D motion codes at 25 Hz, with adversarial training addressing the one-to-many mapping from speech to facial performance.The motion representation shares Agrawal et al.’s latent space and matches Chu et al.’s predicted signal and decoder for comparison.Source: §4 · p. 5

4.1 GENERATOR

  • A two-layer bidirectional noise stack builds temporal structure before six causal audio-conditioning layers emit the full sequence in one pass; one audio frame of lookahead costs 40 ms.Windowed attention bounds the receptive field independently of sequence length and supports KV-cached incremental decoding.Source: §4.1 · p. 5

4.2 DISCRIMINATORS

  • The conditional discriminator judges audio–expression correspondence, while the unconditional discriminator also receives pose; both assess motion at four temporal scales.The scales are S={1, 2, 4, 8}, spanning per-frame articulation through multi-second head motion.Source: §4.2 · p. 6

4.3 TRAINING OBJECTIVE

  • Training combines multiscale hinge adversarial loss and feature matching with a mode-seeking term; pose receives ℓ1 supervision and first- and third-order smoothness regularization.The first 5,000 steps use reconstruction and smoothness terms before generator and discriminator updates alternate.Source: §4.3 · p. 6

5 EXPERIMENTS

Across the experiments, FaceGAN matches or exceeds the compared methods while streaming in real time, and user preferences and ablations support its perceptual quality and design choices. Its bounded-window streaming implementation sustains real-time generation across many concurrent streams.

5.1 EXPERIMENTAL SETTINGS

  • The dataset contains 34,906 clips from 832 subjects (469.6 hours), partitioned into 784 training and 48 test identities; results are reported on a fixed set of 64 held-out clips.Source: §5.1 · p. 6

5.2 RESULTS

  • FaceGAN achieves the best Sync Score and within-clip pose variance, ranks second on every other metric, and runs in real time with single-step inference.The comparison covers Fallingwater, DiffPoseTalk, MemoryTalker, and ARTalk.Source: Table 2 · p. 7

5.3 USER STUDY

  • Pooled user preferences are at parity for lip sync and favor FaceGAN for naturalness, with no baseline ahead on either criterion.The study included 19 participants and 684 comparisons; per-opponent results are only sensitive to differences of roughly ten points.Source: §5.3 · p. 8, Table 3 · p. 8

5.4 ABLATION STUDY

  • Causal noise stacks under-articulate at 0.92–0.93 and lose diversity, while acausal settings stay within 3% of 1.0; removing key discriminators or the mode-seeking term also degrades quality.At matched width, causal stacks lose 14% of expression diversity and 22% of pose diversity; the ablations use three-run comparisons.Source: Table 4 · p. 8, §5.4 · p. 9, Table 5 · p. 9

5.5 COMPUTATIONAL PERFORMANCE

  • A single stream runs at 43.3 ms and 12.1× real time; latency remains unchanged at 256 streams and real time holds to 4096 streams at 62 ms.Measurements are streaming-mode generator performance on one H200 in fp32, including the 40 ms lookahead.Source: §5.5 · p. 9, Table 6 · p. 9

6 CONCLUSION

The conclusion argues that audio-driven facial animation can use single-pass generation, with FaceGAN matching compared baselines and running in streaming mode. The paper also reports AI assistance in writing and identifies both risk-reducing properties and residual misuse risk.

6 CONCLUSION

  • FaceGAN matches the compared baselines with one forward pass and runs offline or streaming, at 43 ms for one stream and 62 ms for 4096 streams on one GPU.The conclusion attributes its quality to both discriminators and an acausal noise stack, which avoids under-articulation and diversity loss seen with a causal stack.Source: §6 · p. 9

AI USE STATEMENT

  • The authors used AI tools to polish and draft some text and format tables, reviewed AI-assisted work, and attribute conceptual contributions and experimental work to the authors.The authors state that they take responsibility for the final content.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

  • The system outputs expression and head-pose coefficients rather than pixels and lacks speaker identity or style conditioning, but its real-time operation lowers misuse costs without eliminating risk.Photorealistic video requires a renderer not contributed by the authors.Source: ETHICS STATEMENT · p. 10

A BACKGROUND FOR SPECTRAL LIMITATION

The background defines the process and information conditions used to reason about spectral limitation. It relates fresh-noise usage and past-output predictability to the process’s per-step innovation.

  • The process’s spectrum distributes its variance across frequencies, while its one-step linear prediction error measures new randomness; zero prediction error makes the output recoverable from its past.A perfect stopband is defined as zero spectral density on a set of positive measure.Source: A BACKGROUND FOR SPECTRAL LIMITATION · p. 14
  • The fresh-noise assumption requires that varying the current noise while fixing past noise can vary the output; the text gives examples that violate or satisfy this condition.A non-trivial generator sampling fresh noise per frame satisfies the stated assumption.Source: A BACKGROUND FOR SPECTRAL LIMITATION · p. 14
  • Past outputs are deterministic functions of past noise, so past outputs contain no more information than past noise; the law of total variance gives a lower bound on conditional output variance.The stated bound drops a non-negative variance term when moving from conditioning on past noise to conditioning on past outputs.Source: A BACKGROUND FOR SPECTRAL LIMITATION · p. 14

B STYLE-CONDITIONED RESULTS

The appendix reports style-conditioned baseline results alongside the unconditioned model as a reference. It describes the metrics and the table’s ranking conventions.

  • The appendix compares style-conditioned variants of four baselines and includes the unconditioned model as a reference.The main-paper comparison uses baselines without style conditioning for direct comparability.Source: B STYLE-CONDITIONED RESULTS · p. 14, Table 7 · p. 14
  • Expr/Pose Var measure within-clip variance ratios against ground truth, with values closest to 1.0 ranked best.Source: Table 7 · p. 14
  • Table 7 marks style conditioning and uses gold, silver, and bronze markers for per-column ranking, following each metric’s direction.Source: Table 7 · p. 14

C ADDITIONAL QUALITATIVE COMPARISONS

The supplemental material includes baseline-comparison videos and a qualitative single-frame example; Figure 4 shows head-pose variation across three identities.

D INFERENCE CHARACTERISTICS IN DETAIL

Table 1 summarizes inference characteristics of the compared methods, while Table 8 provides per-method verification of step counts.

E REAL-TIME BENCHMARK OF THE COMPARED METHODS

The benchmark contrasts first-frame latency and compute for FaceGAN and the baselines, with architectural differences explaining latency and sequential-step costs.

Loading 2610.11070v1…