Source-linked AI summary
BEACON: Behavior and Appearance Control for Subject-Specific Video Generation
Pokrzywa Baptiste, Nabyl Quignon, Yara Bahram, Muhammad Osama Zeeshan, Antitza Dantcheva, Eric Granger
TL;DR
Existing image-conditioned methods often preserve appearance but provide limited control over person-specific expressive dynamics. BEACON separates identity-image and expressive-video conditioning, and experiments on MEAD and RAVDESS report improved expressive consistency while maintaining strong identity preservation.
Problem
Many subject-to-video methods preserve identity but remain limited in faithfully preserving person-specific expression dynamics.
Source: §2 · p. 5Method
Results
Experiments on MEAD and RAVDESS show improved expressive behaviour consistency over existing subject-to-video approaches while maintaining strong identity preservation.
Source: §6 · p. 14Takeaways & Limitations
The factorization supports identity-expression transfer as well as joint preservation of identity and expressive behaviour.
Source: §6 · p. 14Minor appearance leakage, including eye-colour influence from the expressive reference, can still occur in some cases.
Source: §5.2 · p. 14
Abstract
from arXiv · showhide
Generating human-centric videos that preserve both visual identity and person-specific expressive behavior remains a fundamental challenge. In addition to reproducing appearance, a model must replicate the facial behaviors that characterize how a subject expresses emotion over time. However, most state-of-the-art methods condition generation on a single reference image, which contains no information about these temporal dynamics. As a result, they tend to preserve the subject's visual identity but often produce expressions with limited variation and weak subject specificity. To mitigate this issue, we introduce BEACON, a lightweight framework for person-specific video generation that produces more expressive videos by disentangling visual identity from expressive behavior. BEACON conditions generation on two complementary signals: a reference image encoding the identity and a reference video capturing subject-specific facial dynamics. By conditioning on these complementary signals, BEACON generates videos that better preserve both the subject's appearance and characteristic facial dynamics, while also supporting identity-expression transfer. Our experiments on the MEAD and RAVDESS datasets show that by fine-tuning on approximately 2,000 pairs and updating about 1% of the pretrained Wan video diffusion model, BEACON improves facial expressivity over state-of-the-art video generation methods while maintaining competitive identity preservation.
1 Introduction
Human-centric video generation has made progress in preserving who appears, but often offers limited control over how a person expresses emotion over time. BEACON conditions generation on a reference image and a reference video to separate identity from subject-specific expressive behaviour.
1 Introduction
- Image-conditioned methods preserve visual identity but provide little temporal information about a person’s characteristic facial behaviour, leaving expressions limited and weakly subject-specific.Facial behaviour can vary in timing, intensity, asymmetry, and motion patterns.Source: §1 · p. 2
- BEACON complements the text prompt and identity image with a reference video that captures subject-specific expressive behaviour.The video provides cues such as timing, intensity, gaze, and head-motion dynamics.Source: §1 · pp. 2–3
- BEACON models identity and expressive behaviour separately, enabling independent control over appearance and behaviour during generation.The framework also supports identity-expression transfer across subjects.Source: Figure 1 · p. 2, §1 · p. 3
- BEACON adds image and video conditioning branches to pretrained Wan, using IEDA and CPM to keep identity and behavioural signals distinct.Its training strategy uses diverse identity-behaviour combinations to encourage disentanglement.Source: §1 · p. 3
2 Related Work
Prior human-centric video generation methods chiefly condition on reference images to preserve identity, while providing limited control over person-specific expressive dynamics. BEACON’s motivating distinction is to model identity and expressive behaviour as separate factors.
2 Related Work
- Text-to-video foundation models generate realistic, temporally coherent videos, but preserving identity and fine-grained expression control remains challenging.Reference-conditioned subject-to-video methods add identity images to improve control over the person depicted.Source: §2 · p. 4
- End-to-end identity-conditioning methods preserve identity but require jointly learning identity conditioning and video generation, which is computationally expensive.VACE, Phantom, and ConsisID are examples of this approach.Source: §2 · p. 4
- Lightweight identity adapters improve efficiency and modularity, but their static-image conditioning provides little information about natural movement or expression.Stand-In serves as the basis of BEACON’s work.Source: §2 · p. 4
- Slot-ID uses a reference video to enrich identity representation, but leaves appearance and behavioural cues entangled rather than explicitly controllable.BEACON advocates explicitly disentangling visual identity from expressive behaviour.Source: §2 · pp. 4–5
3 Proposed BEACON Method
BEACON factorizes subject conditioning into identity from a reference image and expressive behaviour from a reference video. Its conditioning branches and training pairs are designed to preserve those factors as distinct sources of control.
3 Proposed BEACON Method
- BEACON formulates subject conditioning as separate identity-image and expressive-video signals alongside the text prompt.This formulation is intended to enable independent control of appearance and expressive dynamics.Source: §3 · p. 5
3.1 Background
- Stand-In supplies the lightweight identity-conditioning foundation, encoding reference-image latents as tokens that video tokens can attend to.Restricted Self-Attention keeps reference tokens stable, while CPM gives them a separate positional coordinate system.Source: §3.1 · pp. 5–6
3.2 Expressive Video Conditioning Branch
- The expressive branch encodes the reference video alongside identity and generated-video tokens, while the pretrained generation backbone remains frozen.It provides cues including facial motion, expression intensity, head movement, and emotion over time.Source: §3.2 · p. 6, Figure 2 · p. 6
- InsightFace face localization filters expressive tokens to facial regions, reducing the set from 16.3k tokens to 2.2k.The selection targets facial expressive behaviour rather than scene dynamics.Source: §3.2 · p. 7
- Generated video tokens attend to both conditioning branches, while identity and expressive branches remain isolated and use independent self-attention.Extended CPM places the conditioning branches in a shared offset region while preserving expressive temporal coordinates and original generated-video coordinates.Source: §3.2 · pp. 7–8
3.3 Training Scheme
- Real MEAD pairs use a neutral identity image and a different same-subject performance of the target emotion and intensity as expressive input.Using different performances avoids trivially copying the target sequence while supervising subject-specific dynamics.Source: §3.3 · p. 8
- Synthetic cross-identity pairs use expressive videos from a different identity, encouraging the expressive branch to encode behaviour rather than appearance.Synthetic samples make up 25% of each training mini-batch; the remainder are real MEAD videos.Source: Figure 3 · p. 8, §3.3 · p. 9
4 Experimental Methodology
BEACON is trained as a lightweight adaptation of Wan using MEAD identity-expression pairs, and evaluated on MEAD and RAVDESS. The evaluation combines identity, expressive-behaviour, and video-quality measures on held-out MEAD subjects.
4.1 Implementation Details
- BEACON fine-tunes Wan with rank-128 LoRA on conditioning-token QKV projections, using approximately 2,000 MEAD identity-expression pairs.Training runs for 3k steps and includes 500 synthetic mismatched identity-expression pairs.Source: §4.1 · p. 9
4.2 Dataset
- MEAD provides repeated scripted performances across eight emotions and three intensity levels; RAVDESS tests generalization on 24 actors performing eight emotions.RAVDESS includes two intensity levels for most emotions.Source: §4.2 · p. 9
4.3 Evaluation Metrics
- Evaluation measures identity with FaceSim, expressive behaviour with E-FID and AUE, and video quality with VBench.The VBench measures are Subject Consistency, Aesthetic Quality, and Motion Smoothness; lower E-FID indicates better expressive-behaviour agreement.Source: §4.3 · p. 10
4.4 Evaluation Protocol
- The MEAD protocol evaluates eight held-out subjects by generating ten videos per emotion for each subject, totalling 640 videos.The subset is selected to estimate distribution-based metrics while limiting computational cost.Source: §4.4 · pp. 10–11
5 Results and Interpretation
BEACON shows stronger expressive-behaviour consistency than existing methods on MEAD and outperforms Stand-In on expressivity on RAVDESS. Its identity-expression transfer results support disentanglement, though some appearance leakage remains.
5.1 Comparison with State-of-the-art Methods
- On MEAD, BEACON substantially outperforms other methods in expressive-behaviour consistency.The comparison is reported in Table 1.Source: §5.1 · p. 11, Table 1 · p. 11
- 79.0% FaceSim for BEACON versus 82.6% for Stand-In with neutral references; the trend reverses when references match the generated emotion.The authors suggest Stand-In’s lower expressive variation keeps its output closer to a neutral appearance.Source: §5.1 · p. 11, Table 2 · p. 11
- On RAVDESS, BEACON outperforms Stand-In on expressivity, while the FaceSim trend reverses.The authors suggest face-recognition sensitivity to pose and illumination may explain the FaceSim difference.Source: §5.1 · pp. 11–12, Table 3 · p. 12
- BEACON produces richer, more subject-specific facial deformations than Stand-In and ConsisID in the qualitative comparisons.For fear, BEACON better captures facial dynamics such as forehead wrinkles visible in the expressive reference.Source: §5.1 · pp. 12–13, Figure 4 · p. 12
- Stand-In’s generated videos have lower E-FID with neutral references across target emotions, whereas BEACON has the lowest E-FID for the correct emotion.The matrices compare generated videos with same-subject references across emotions on MEAD.Source: Figure 5 · p. 12, §5.1 · p. 13
5.2 Analysis of Disentanglement
- With cross-identity training pairs, Face Similarity to the identity reference rises from 0.10 to 0.69, while similarity to the expressive-reference identity falls from 0.67 to 0.25.The transfer evaluation uses identity images and expressive videos from different subjects; identity-expression transfer may still show minor appearance leakage, such as eye color influenced by the expressive reference.Source: §5.2 · pp. 13–14, Table 4 · p. 13, Figure 6 · p. 14
6 Conclusion
BEACON jointly conditions person-specific video generation on an identity image and expressive reference video, improving expressive-behaviour consistency while preserving identity and supporting identity-expression transfer. Its current focus is controlled talking-head videos, with more diverse, unconstrained scenarios left for future work.
6 Conclusion
- BEACON improves expressive-behaviour consistency on MEAD and RAVDESS compared with existing S2V approaches while maintaining strong identity preservation.The framework also supports identity-expression transfer.Source: §6 · p. 14
- BEACON jointly conditions generation on an identity image and expressive reference video, separating identity from expressive behaviour.This dual conditioning supports person-specific facial dynamics.Source: §6 · p. 14
- The framework focuses on controlled talking-head videos with static cameras and relatively simple backgrounds; extending it to diverse, unconstrained scenarios remains future work.Recent S2V models can generate subjects across more diverse scenes and poses.Source: §6 · p. 14