PaperlayerView exact revision on arXiv

Source-linked AI summary

Unveil the Mask: Jointly-Optimized Diffusion Masked Pretraining for Appearance-Agnostic Operator Verification

Katsuya Ogata, Ryoya Yoshiyama, Kawachi Hodaka, Yuta Nakashima

arXiv:2610.35414v1cs.CV

TL;DR

Photorealistic avatars separate rendered appearance from the person operating them, making operator verification from appearance alone unsuitable. The paper combines diffusion-pretrained landmark motion representations with a video-motion branch and reports 0.769 overall AUC on WBA2026. Fusion also helps cross-mode verification, while generalization to unseen generators remains a challenge.

  • Problem

    Photorealistic avatars decouple rendered appearance from the operator, so verification must identify the driver regardless of the animated source.

    Source: §1 · pp. 1–2

  • Method

    The framework combines appearance-normalized landmarks with diffusion masked pretraining, behavioral auxiliary tasks, generator-simulating transformations, and a video branch based on consecutive-frame differences.

    Source: §1 · p. 2, §4.1 · p. 5, §4.2 · p. 8

  • Results

    0.769 overall AUC on WBA2026; the authors also report that fusion helps cross-mode verification.

    Source: §6 · p. 14

  • Takeaways & Limitations

    Video embeddings retain minimal avatar information, and combining modalities helps cross-mode verification, while unseen-generator generalization remains challenging.

    Source: §6 · p. 14

    Test samples have no mode labels at inference, and Diff Gen uses distinct, unseen generation pipelines.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Photorealistic avatar videos decouple the appearance of a rendered face from the person who operates it, which makes conventional face recognition unsuitable for identifying the operator. Verifying the operator therefore requires representations of how a face moves that are insensitive to what the avatar looks like. We propose a verification framework built without any external pretrained model. It uses two complementary modalities provided for each sample: facial landmark sequences and video frames. From the landmark modality, it learns motion representations by diffusion masked pretraining over grouped landmarks, guided by lightweight auxiliary tasks that predict behavioral cues such as motion intensity and eye openness, and turns them into operator embeddings with a contrastive linear probe. To generalize to unseen avatar generators, the pretraining input is randomly transformed to simulate how a generator may alter the operator's motion. From the video modality, it captures fine facial motion through differences between consecutive frames. The embeddings of the two modalities are combined into one. Our approach reaches an overall AUC of 0.769 on the WBA2026 benchmark. Linear probing shows that the video embedding retains little information about the avatar's identity, whereas the landmark embedding retains some, and fusing the two modalities improves verification in particular across speaking and listening modes.

1 Introduction

Photorealistic avatars separate rendered appearance from the person operating them, making conventional face recognition unsuitable for operator verification. The paper introduces a multimodal, appearance-invariant framework and reports an overall WBA2026 AUC of 0.769.

1 Introduction

  • Appearance decoupling makes rendered face texture and geometry unreliable for identifying the physical operator.The observed appearance belongs to the synthesized avatar rather than the person behind it.Source: §1 · p. 1
  • The WBA task verifies the driver regardless of the animated source, confronting appearance confounding, speaking–listening behavior differences, and unseen-generator variation.Speaking is dominated by larger lip and jaw motion, while listening involves subtler expressions, gaze shifts, and head nodding.Source: §1 · p. 2
  • 0.769 overall AUC on WBA2026 is achieved by a framework combining landmark and video branches without external pretrained weights or distillation.The landmark branch uses diffusion masked pretraining, auxiliary behavioral tasks, and a driver-aware contrastive linear probe; the video branch uses consecutive-frame differences.Source: §1 · p. 2

2 Related Work

Related work motivates identity representations based on facial behavior rather than static appearance, and places landmark modeling within the progression of self-supervised masked prediction.

2.1 Behavioral Biometrics and Facial Dynamics

  • Behavioral biometrics identify people through characteristic motion, and facial dynamics offer person-specific cues beyond static appearance.Prior work cited in the paper examines head and mouth motion, expression dynamics, action-unit statistics, and micro-expressions.Source: §2.1 · p. 2

2.2 Self-Supervised Predictive and Masked Modeling

  • Masked prediction progressed from language modeling to visual and temporal signals; DiMP combines masked modeling with diffusion, which this work adapts to facial landmarks.The passage cites BERT, masked autoencoding, VideoMAE, MotionBERT, masked motion prediction, and DiMP.Source: §2.2 · p. 3

3 Task Formulation

WBA asks whether avatar videos share an operator despite potentially differing rendered sources. Evaluation uses synchronized landmark and video observations, tests generator and speaking–listening conditions, and reports AUC over probe–gallery pairs.

3 Task Formulation

  • WBA asks whether two avatar videos are driven by the same person even when their rendered avatars may differ.The task is appearance-agnostic operator identity verification.Source: §3 · p. 3

3.1 Problem Setting

  • Each sample pairs a facial-landmark sequence with its synchronized rendered video.The landmark sequence contains 106 two-dimensional landmarks; the video is 512 × 512 RGB.Source: §3.1 · p. 3
  • Test samples have no speaking/listening labels, and Diff Gen uses avatars from distinct, unseen generation pipelines.The dataset also includes Same Gen samples from rendering pipelines represented in training.Source: §3.1 · p. 3
  • Model selection uses validation samples from 20 held-out drivers and 12 held-out sources absent from the training split.The passage identifies these as validation-set identities.Source: §3.1 · p. 3
  • The encoder maps each sample to a unit-norm embedding, and cosine similarity is an accepted verification score.The challenge protocol accepts one embedding per test sample and permits cosine similarity or Euclidean distance.Source: §3.1 · p. 3
  • Table 1 summarizes dataset statistics, with validation drivers and sources excluded from the training split.The supplied table caption identifies the validation set as held out from official training data.Source: Table 1 · p. 4

3.2 Evaluation Metrics

  • AUC is computed over probe–gallery pairs across four speaking/listening combinations and two generator conditions.The mode combinations are S→S, S→L, L→S, and L→L; generator conditions are Same Gen and Diff Gen.Source: §3.2 · p. 4
  • Overall AUC is the mean of the Same Gen and Diff Gen scores, each averaged across its four mode combinations.The validation score instead averages S→S, L→L, and a combined cross-mode AUC.Source: §3.2 · p. 4

4 Proposed Method

The framework trains landmark and video branches independently, then fuses their embeddings for operator verification. The landmark branch removes static appearance cues and combines diffusion-based pretraining with behavioral supervision, generator simulation, and contrastive probing.

4.1 Appearance-Agnostic Landmark Tokenization

  • Four per-sample normalizations remove face position, scale, avatar shape, and overall motion amplitude while retaining relative motion.The 106 landmarks are grouped into spatial, functional-anatomical, or synchronized-motion tokens.Source: §4.1 · pp. 5–6

4.2 Diffusion Masked Pretraining (Stage 1)

  • Diffusion masked pretraining recovers masked landmark coordinates and centroids while independently modeling their motion from visible-token context.The decoder reconstructs masked coordinates, and a separate diffusion process predicts noise added to masked-token displacements.Source: §4.2 · pp. 7–8

4.3 Auxiliary Behavioral Tasks

  • Three label-free regression tasks guide visible-token features toward motion energy, facial-region motion, and eye openness cues.Eye openness is measured by the height-to-width ratio of eye-landmark bounding boxes.Source: §4.3 · p. 9

4.4 Pseudo-Generator Randomization

  • Randomized transformations simulate six possible generator effects, while GroupDRO emphasizes motion families with the highest prediction loss.The transformations include time warping, facial-part shifts or filtering, tracking-like noise, and compositions of these changes.Source: §4.4 · p. 9

4.5 Driver-Aware Contrastive Probing (Stage 2)

  • A frozen pretrained encoder feeds a trained linear embedding layer, with batches arranged so same-driver positives cross avatars and negatives share an avatar.This batch construction makes avatar matching insufficient to solve the contrastive objective.Source: §4.5 · p. 10

4.6 Video Branch on Landmark-Guided Face Crops

  • Consecutive-frame differencing emphasizes dense facial motion beyond the 106 tracked landmarks while suppressing static frame features.Landmark-guided crops preserve head motion, and the video models train on feature or pixel differences.Source: §4.6 · pp. 10–11

4.7 Ensembling and Score-Level Fusion

  • The final embedding concatenates landmark and video ensemble members with modality weights, making cosine similarity a weighted average of member similarities.The fusion does not require the test sample's speaking or listening mode.Source: §4.7 · p. 11

5 Experiments and Results

The experiments evaluate landmark and video models on WBA2026, including unseen-generator and cross-mode verification, and test auxiliary tasks, fusion, and token groupings. The final submission reaches 0.769 overall AUC, while analysis finds different strengths for the two modalities.

5.1 Experimental Setup

  • All models are randomly initialized; validation drivers and sources are disjoint from training and guide early stopping, configuration comparisons, and fusion-weight selection.Stage 1 uses a 6-layer Transformer encoder and 4-layer decoder; Stage 2 and video models are early-stopped on validation mean AUC.Source: §5.1 · pp. 11–12

5.2 Benchmark Performance on WBA2026

  • 0.769 overall AUC: the final submission uses a fused ensemble for Same Gen and a pseudo-generator-randomized landmark ensemble for Diff Gen.The benchmark comparison reports that landmark models generalize better to unseen generators than the video branch, while a single landmark-video fusion already beats every single-branch result.Source: §5.2 · pp. 12–13, Table 2 · p. 12

5.3 Ablation Studies

  • All three auxiliary tasks yield the best reported task-combination result (0.834, +2.3 points); fusion beats either branch alone across a wide range of α, and combining all token groupings reaches 0.862.Anat and Motion tokens score 1.6 and 2.0 points above Geo tokens, respectively; every combination beats any single grouping.Source: §5.3 · p. 13, Table 3 · p. 13

5.4 Analysis

  • The video branch predicts avatar source at 2.1% versus 1.3% chance, while the landmark branch reaches about 10%; the analysis uses training generators only.Frame differencing removes most avatar appearance, but landmark motion retains some rendered-avatar properties; the analysis cannot measure cues specific to unseen generators.Source: §5.4 · p. 14, Table 4 · p. 14
  • Fusion raises cross-mode AUCs to 0.925 and 0.938, while Diff Gen video models score 0.52–0.56 versus 0.58–0.63 for landmark models.Landmark models reach 0.937 for S→S but 0.817 and 0.850 for S→L and L→S; the authors therefore use only landmark models for Diff Gen.Source: §5.2 · p. 12, §5.4 · p. 14

6 Conclusion

The paper concludes with a two-branch operator-verification framework that achieves 0.769 AUC on WBA2026. Its analysis finds little avatar-identity information in video embeddings and cross-mode gains from fusion, while unseen-generator generalization remains the primary challenge.

6 Conclusion

  • The framework combines diffusion masked pretraining, auxiliary behavioral tasks, and inter-frame video differences, achieving 0.769 AUC on WBA2026.The conclusion identifies generalization to unseen generators as the primary challenge.Source: §6 · p. 14

Related papers

Loading 2610.35414v1…