Source-linked AI summary
FATE: Frame-Level Audio-Visual Temporal Embedding
Kaisi Guan, Bingzi Zhang, Xihua Wang, Ying Ba, Xin Cheng, Yijing Chen, Ruihua Song
TL;DR
Audio-visual models often capture semantic matching or temporal synchronization, but not both. FATE preserves and aligns frame-level audio-video features in a shared embedding space using joint semantic-temporal contrastive learning, outperforming baselines across retrieval, localization, and generation evaluation.
Problem
Existing audio-visual models typically emphasize semantic matching or temporal synchronization, leaving representations limited in the other dimension.
Method
FATE preserves frame-level audio-video features, aligns them on the physical timeline, and trains a shared embedding space with joint semantic and temporal contrastive objectives.
Results
FATE outperforms the strongest synchronization baseline by over 13 points in temporal retrieval R@3 and over 20 points in semantic retrieval accuracy, while achieving 48.3% average accuracy in zero-shot event localization.
Takeaways & Limitations
Across three tasks, FATE provides a reusable representation that captures semantic correspondence and temporal synchronization and agrees most closely with human synchronization judgments.
Takeaways & Limitations
Fine-grained synchronization evaluation remains an open problem because all metrics, including FATE, show modest absolute sample-level correlations, and model-ranking correlation uses only five generation models.
Abstract
from arXiv · showhide
When a dog opens its mouth and barks, humans naturally recognize what the sound is and when it occurs. Building audio-visual models with this same ability requires representations that capture both semantic and temporal alignment. Current approaches fall short on one side or the other: embedding models match semantic but lose temporal information; synchronization models capture temporal offsets but lack semantic understanding. To bridge this gap, we propose FATE, Frame-level Audio-visual Temporal Embedding. Unlike prior embedding models that pool each modality into a single embedding and discard temporal information, FATE retains frame-level sequences, aligns them on the physical timeline, and computes similarity over strictly aligned frame pairs. Unlike synchronization models that output only an offset prediction, FATE encodes synchronization in a reusable embedding space, trained with a joint objective combining cross-video semantic and within-video temporal contrastive learning to capture both what sounds and when it occurs. Across three tasks, FATE surpasses the strongest baseline on temporal and semantic retrieval by a large margin, matches fully supervised methods on event localization in a zero-shot setting, and achieves the best correlation with human judgments as a generation evaluation metric. The source code can be found at \texttt{https://github.com/guankaisi/FATE}.
Introduction
FATE addresses the gap between semantic audio-visual embeddings and temporally precise synchronization models by preserving frame-level features, aligning modalities on the physical timeline, and learning a reusable embedding space. Across three downstream tasks, it improves retrieval, zero-shot event localization, and agreement with human synchronization judgments.
- Motivation: Existing audio-visual embedding models emphasize semantic alignment, whereas synchronization models capture temporal offsets but yield weak semantic representations.Omni LLMs also lack fine-grained synchronization at low sampling rates and produce no reusable embeddings for downstream tasks.
- Method: FATE preserves frame-level audio and video features, aligns them on the physical timeline, and computes similarity over strictly aligned frame pairs.This design captures synchronization directly in a reusable embedding space without task-specific prediction heads.
- Training objective: FATE combines cross-video semantic contrastive learning with within-video temporal soft-contrastive learning to capture both what sounds and when it occurs.The two objectives are complementary according to extensive ablations.
- Evaluation: Over 13 points in R@3 and over 20 points in semantic retrieval accuracy separate FATE from the strongest synchronization baseline and prior methods, respectively.FATE also reaches 48.3% average accuracy for zero-shot audio-visual event localization and agrees most closely with human synchronization judgments as a generation metric.
Related Works
Related work spans multimodal contrastive representation learning, audio-visual synchronization, and evaluation of generative models. Existing approaches motivate FATE’s frame-level design by lacking either sufficiently fine temporal granularity or reliable synchronization evaluation.
- Multimodal Representation Learning: Contrastive multimodal learning, pioneered for image-text alignment by CLIP, extends naturally to audio-visual representation learning.Recent omni-modal LLMs jointly process audio and video but use sparse visual sampling, such as 2 fps in Qwen3-Omni.
- Multimodal Representation Learning: Recent omni-modal models provide no reusable embeddings for downstream tasks and sample visuals too coarsely for synchronization.The supplied passage identifies sparse visual sampling and the absence of reusable embeddings as limitations motivating a frame-level design.
- Audio-Visual Synchronization: Audio-visual synchronization research progressed from binary lip-speech detection to open-domain modeling and quantitative temporal-offset estimation.SyncNet introduced two-tower CNNs for lip-speech synchronization; AVST generalized the setting, while SparseSelector and Synchformer estimated offsets.
- Generative-Model Evaluation: Audio-visual generative models are advancing rapidly, but reliable synchronization evaluation remains limited.Existing metrics are organized into three paradigms, including rule-based metrics such as AV-Align.
Method
FATE uses a dual-encoder architecture that preserves frame-level audio-visual sequences, aligns them on a shared timeline, and computes similarity over corresponding frame pairs. It jointly trains semantic and temporal contrastive objectives so one embedding space captures both what sounds and when it occurs.
- Learning objectives: FATE satisfies both semantic and temporal discrimination by defining similarity over temporally aligned frame pairs in a single embedding space.Conventional embedding models capture semantic but not temporal discrimination, whereas synchronization models show the opposite limitation.
- Architecture: FATE discards global pooling, retains frame-level tokens, aligns visual and audio sequences, and computes similarity as the mean diagonal of their pairwise frame-level inner products.The pipeline comprises frame-level feature extraction, temporal alignment, and frame-level similarity computation.
- Temporal alignment: Audio anchors the timeline while video is downsampled and nearest-neighbor aligned, preserving transient synchronization cues and matching corresponding time intervals.The design reflects video’s temporal redundancy and audio’s transient synchronization information.
- Frame-level similarity: Mean diagonal similarity enforces temporal order: temporal shifts move true correspondences off-diagonal, reducing similarity as correctly aligned content shrinks.This produces a similarity peak at zero offset and enables synchronization-aware retrieval without a task-specific prediction head.
- Learning objectives: The total objective is Ltotal = λ1Lsem + λ2Ltemp, combining cross-video semantic contrast with within-video temporal soft-contrastive supervision.The temporal loss creates a smooth alignment landscape that peaks at zero offset and decreases with displacement; T0 = 1 s sets its support boundary.
Experimental Settings
FATE is fine-tuned parameter-efficiently on temporally segmented VGGSound clips and evaluated across retrieval, event localization, and generation-quality assessment. The experiments test temporal discrimination, cross-video generalization, zero-shot localization, and correlation with human synchronization judgments.
- Training: FATE initializes from PE-AV-small, discards its text encoder and global pooling head, and fine-tunes with LoRA while updating only 2.3% of parameters.Encoder weights remain frozen during fine-tuning.
- Training: Training uses approximately 183k VGGSound 10-second clips across 309 categories, segmented into 2-second clips with a 0.5-second stride.This produces approximately 17 clips per video.
- Temporal Cross-Modal Retrieval: Temporal cross-modal retrieval evaluates aligned-clip selection on AVSync15 and VGG-Sync, using approximately 17 candidates per video and leakage checks by video ID.The benchmarks contain 150 and 595 videos, respectively, each approximately 10 seconds long.
- Temporal Cross-Modal Retrieval: Intra-Video retrieval isolates fine-grained temporal discrimination, whereas Inter-Video retrieval uses M=50 distractor videos and approximately 850 candidates per query.The inter-video setting requires identifying both the correct video and aligned segment.
- Baselines: FATE is compared with six audio-visual foundation models and Synchformer, whose retrieval scores are cosine similarity and predicted probability at zero offset, respectively.Foundation-model baselines include ImageBind, LanguageBind, CAV-MAE Sync, PE-AV, CAVP, and PEAVS.
- Downstream Evaluation: Additional evaluation covers zero-shot Audio-Visual Event localization against supervised baselines and human-rated synchronization of videos generated by five joint audio-video models.AVE reports segment-level accuracy for A2V, V2A, and their average; generation evaluation uses 10 annotators’ 1-5 ratings aggregated as MOS.
Experimental Results
FATE improves temporal and semantic audio-visual evaluation by preserving frame-level structure, outperforming pooling-based and specialized synchronization baselines across retrieval, localization, and human-alignment metrics. Its frame-level similarity is temporally sensitive, while joint semantic and temporal learning provides complementary gains, although fine-grained human agreement remains limited.
- Temporal and Semantic Retrieval: FATE lifts AVSync-15 V2A R@1 from 5.33 to 22.37, a 4× improvement over PE-AV, by preserving temporal structure and adding temporal supervision.FATE and PE-AV use the same encoder; pooling-based models remain near the 5.9% intra-video chance level.
- Human-Perception Agreement: FATE achieves the highest average sample-level correlation of 17.24 and model-ranking ρ of 44.41, ahead of CAVP’s 14.15 and AV-Align’s 24.99.It is the only compared metric that performs best on both dimensions and requires no metric-specific training.
- Limitations: Fine-grained synchronization evaluation remains difficult: all metrics show modest absolute sample-level correlations, and model-ranking ρ uses only five generation models.FATE narrows the evaluation gap but does not close it, so its model-ranking correlation is indicative rather than conclusive.
- Temporal Sensitivity Analysis: FATE’s similarity peaks at temporal offset ∆t=0 and decays as offsets increase, unlike PE-AV’s nearly flat response to temporal shifts.Frame shifts disrupt physical-time alignment, while soft labels produce a gradual rather than binary transition.
- Effectiveness of Loss Components: Combining semantic and temporal losses reaches 34.13 V2A R@3, exceeding Lsem alone at 19.78 and Ltemp alone at 15.56.Lsem learns cross-video semantics without within-video temporal precision, whereas Ltemp lacks semantic separation across videos.
Conclusion
FATE unifies semantic correspondence and temporal synchronization in one audio-visual representation by retaining frame-level features, aligning them on physical time, and jointly training semantic-temporal objectives. Experiments across three complementary tasks validate this design.
- FATE unifies semantic correspondence and temporal synchronization within a single audio-visual representation space.
- Dense frame-level features, physical-time alignment, and joint semantic-temporal training avoid the information bottleneck imposed by global pooling.
- Experiments across three complementary tasks validate FATE’s design.
Supplementary Material for FATE
The supplementary material details FATE’s efficiency, preprocessing configuration, human synchronization evaluation protocol, qualitative validation, and comparison setup against omni-modal LLMs. These experiments assess reusable embeddings, annotation reliability, perceptual alignment, and controlled zero-shot evaluation.
- Efficiency: FATE requires 10.8 s per video for inference, nearly matching PE-AV at 9.6 s, while Synchformer requires 168.0 s, roughly 15× slower.FATE and PE-AV share a dual-encoder architecture; Synchformer requires a full forward pass for every query because it cannot pre-extract modality-independent embeddings.
- Implementation: Preprocessing resizes video frames to 336 × 336 and resamples audio to 48 kHz before feature extraction.These settings are listed in the complete preprocessing and optimization configuration.
- Human evaluation: Human synchronization judgments cover 750 generated videos in 150 groups, with five shuffled, anonymized model outputs per text prompt.The setup prevents annotators from inferring generator identities from file ordering.
- Human evaluation: Annotators rank each five-video group from 1 to 5 without ties, and ten independent rankings are averaged into each video’s mean human score.Annotators compare videos side by side and may repeatedly scrub around sound-triggering keyframes.
- Qualitative validation: FATE’s qualitative scores track human synchronization judgments, rewarding aligned sound events and penalizing unsynchronized content across generated-video comparisons.In one example, BridgeDiT scores 12.48 for premature mouth closure, whereas LTX-2 scores 24.75 for aligned mouth movements and audio onsets.
- Omni-modal LLM comparison: Qwen3-Omni-30B-A3B and Gemini-3.6-Flash are evaluated with the same zero-shot prompt, identical preprocessed clips, greedy decoding, and one query per video.The protocol excludes model-specific tuning, few-shot exemplars, and chain-of-thought elicitation while isolating temporal synchronization from semantic relevance.