Audio-Visual Intelligence in Large Foundation Models

1 Introduction

• AVI unifies audio and vision for understanding, prediction, control, generation, and interactive reasoning across temporally and spatially constrained real-world environments.Industry systems such as Meta MovieGen [439] and Google Veo-3 exemplify end-to-end audio-vision modeling and coordinated synthesis.Source: §1 · p. 5
• The field spans perception, generation, and interaction tasks requiring synchronization, grounding, temporal reasoning, latency awareness, and user-intent alignment.Examples include speech recognition, localization, retrieval, question answering, talking heads, video-conditioned audio, multimodal storytelling, streaming inference, tool use, and embodiment.Source: §1 · p. 5
• AVI methods combine modality tokenization, cross-modal fusion, autoregressive and diffusion generation, self-supervised learning, instruction tuning, preference optimization, and large-scale data curation.Scaling laws, data mixtures, and curation jointly shape capability and robustness while raising concerns about coverage, bias, and licensing.Source: §1 · p. 5
• The survey addresses fragmented terminology and taxonomies, inconsistent evaluation protocols, and uneven treatment of privacy, consent, provenance, watermarking, and training-energy concerns.These gaps particularly affect reproducibility, open-ended generation, alignment quality, temporal coherence, and human-centric judgments [55].Source: §1 · p. 5

1.1 Contribution Summary

• The work presents the first systematic survey of AVI in large foundation models, unifying perception, generation, and interaction through a coherent framework.It also establishes a principled taxonomy clarifying task scope, assumptions, and relationships across speech, music, sound events, video, and open-world settings.Source: §1.1 · p. 5
• The survey synthesizes core methods, curates benchmarks and metrics, identifies assessment gaps, shares resources, and highlights future challenges in synchronization, grounding, controllability, safety, and governance.Its organization proceeds from modalities and taxonomy through foundation techniques, perception, generation, interaction, applications, and open challenges including streaming efficiency and evaluation.Source: §1.1 · p. 6, §1.2 · p. 6

2.1 The Audio Modality: Data and Representation

• Digital audio begins as sampled microphone signals represented as mono, stereo, or multichannel waveforms, with common inputs consisting of raw waveforms and log-Mel spectrograms.Waveforms encode temporal samples, while STFT magnitudes are projected onto the Mel scale and logarithmically compressed.Source: §2.1 · p. 6, (b) Visual data representation · p. 8

(b) Visual data representation

• Audio encoders use global semantic embeddings, dense feature sequences preserving temporal or frequency structure, and discrete tokens that enable language-model-style generation.Dense representations are downsampled latent maps or sequences, while vector quantization maps inputs to codebook token IDs.Source: (b) Visual data representation · p. 8

2.2 The Visual Modality: Data and Representation

• Visual data comprises RGB images and frame sequences, while broader multimodal applications also use human-centric content and 3D formats such as depth maps, point clouds, meshes, NeRFs, and Gaussian splats.Images are represented as H × W × 3 tensors, and videos as ordered T-frame 4D tensors, often processed at fixed frame rates.Source: §2.2 · p. 8
• Visual representations similarly include global embeddings, spatially preserving continuous feature maps or sequences, and vector-quantized image-token sequences.CNNs and vision transformers can produce global features, while learned codebooks discretize image features into token IDs.Source: §2.2 · pp. 8–9

3 Task Taxonomy

• Audio-visual intelligence is organized into three dimensions—understanding, generation, and interaction—that together provide a roadmap toward AGI.The detailed task families are developed in Sections 5, 6, and 7.Source: §3 · p. 9

3.1 Understanding the World: Audio-Visual Perception

• Audio-visual perception progresses from pixel-level grounding, through semantic content understanding, to logical reasoning over causes, outcomes, and physical commonsense.Pixel-level tasks align sounds with visual origins; content understanding interprets multimodal events and relationships; logical reasoning infers beyond direct observation.Source: §3.1 · p. 9

3.2 Creating the World: Audio-Visual Generation

• Audio-visual generation comprises conditional generation, cross-modal generation, and joint generation, with increasing integration of audio and vision.These paradigms emphasize controllability and semantic alignment, cross-modal temporal and causal grounding, and synchronized, causally consistent scene evolution.Source: §3.2 · pp. 9–10

Audio-Visual Interaction

• Conditional generation follows text, labels, or structured attributes, whereas cross-modal generation synthesizes one modality from another, including audio-to-visual and video-to-audio tasks.Representative settings include speech or effects from text or audio context, images or videos from text or visual cues, talking-heads and avatars from audio, and soundtracks or spatial audio from video.Source: §3.2 · p. 10

3.3 Interacting with the World: Audio-Visual Unified Perception and Generation

• Unified audio-visual systems combine ongoing perception, contextual reasoning, and timely output or action through conversational interfaces and embodied interaction.Conversation systems may be audio-centric, visual-centric, or omni-modal, while embodied systems support navigation, question answering, and manipulation in physical environments.Source: §3.3 · pp. 10–11

4.1 Representation-centric Methods

• Representation-centric methods convert audio, images, and video into continuous embeddings or discrete tokens that support downstream audio-visual intelligence.They include feature extraction, variational reconstruction, and discrete tokenization.Source: §4 · p. 11, §4.1 · p. 11

4.1.1 Audio-Visual Feature Extraction and Representation

• Audio-visual representations learn shared semantics from synchronized signals using self-supervision, contrastive alignment, correlation modeling, and task-oriented encoders.Representative mechanisms include correspondence and temporal-synchronization objectives, contrastive learning, cross-modal attention, and masked prediction [486].Source: §4.1.1 · p. 11

4.1.2 Audio-Visual Variational Auto-Encoding and Reconstruction

• VAEs map audio or visual inputs into structured probabilistic latent spaces for reconstruction, while discrete tokenizers use vector quantization to create compact sequences for autoregressive multimodal modeling.Multimodal VAEs separate shared and modality-specific information, while audio, visual, and video tokenizers balance reconstruction quality, token rate, and semantic content [141] [690].Source: §4.1.2 · p. 12, §4.1.3 · pp. 12–13

4.2 Generation-centric Methods

• Generation-centric methods support unimodal and cross-modal synthesis or editing, requiring semantic coherence and temporal synchronization across audio-visual outputs.The main paradigms are GANs, diffusion and continuous-time models, autoregressive generation, and masked autoregressive generation.Source: §4.2 · p. 13, Figure 6 · p. 13

4.2.2 Diffusion-based Generation

• Diffusion and related continuous-time models have become prevalent for audio-visual generation, scaling from UNets to Diffusion Transformers and supporting higher resolution, longer duration, and improved temporal consistency.They underpin text-to-image, text-to-video, speech, sound-effect, audio-conditioned video, visually grounded sound, and joint audio-video generation [53] [54] [607] [166] [278] [548] [15] [165] [369].Source: §4.2.2 · pp. 13–14

4.2.3 Autoregressive Generation

• Autoregressive generators formulate multimodal synthesis as next-token prediction, whereas masked autoregressive models parallelize token refinement but depend on tokenization quality and decoding schedules.AR systems support interleaved multimodal prompting but face sampling latency and error accumulation for long sequences; MAR methods trade strict sequencing for faster parallel refinement.Source: §4.2.3 · p. 14, §4.2.4 · p. 14

4.3 LLM-Centric Methods

• LLM-centric methods connect modality encoders, external generators, unified backbones, agents, or action spaces to provide multimodal perception, generation, interaction, and embodied control.Encoder+LLM systems are modular but can limit fine-grained alignment and incur token overhead, while LLM+Generator pipelines enable reusable tools but risk interface mismatches, error accumulation, and synchronization problems [239, 638, 613].Source: §4.3 · p. 14, §4.3.1 · pp. 14–15, §4.3.2 · pp. 15–16

4.3.3 Unified Model for Joint Perception and Generation

• Unified models reduce brittle interfaces and latency through native tokenizers and decoders, while agentic systems dynamically plan tool use and VLA models close the perception-reasoning-action loop.Unified audio-visual modeling remains early, with challenges in heterogeneous tokenization, long-context video, streaming memory, interruption handling, and open-ended evaluation; agents additionally face grounding, state-management, and error-propagation issues [275] [354] [156].Source: §4.3.3 · pp. 16–17, §4.3.4 · p. 17, §4.3.5 · p. 17

4.3.5 Visual-Language-Action Models for Embodied Interaction

• Hierarchical policies such as RT-H separate high-level planning from low-level control, while RT-Trajectory uses hindsight trajectory sketches to improve task generalization.Source: §4.3.5 · p. 17
• OpenVLA combines large-scale multimodal pretraining with robot demonstrations to generalize strongly across tasks and embodiments.Source: §4.3.5 · p. 17
• VLA research also targets efficient deployment and complex control through lightweight models such as TinyVLA and SmolVLA [493], whole-body systems such as GR00T-N1 [41] and Gemini Robotics [519], and diffusion- or planning-based policies including π0 [42] and Octo [520].Other directions include unified perception-action models, trajectory reasoning, modular action heads, failure recovery, and world-model integration.Source: §4.3.5 · p. 18
• VLA models integrate perception, reasoning, and action through multimodal pretraining and language-conditioned planning, but deployment requires robust execution across dynamic environments and robot platforms.Key challenges include long-horizon planning, cross-embodiment transfer, data efficiency, safety, and generalization.Source: §4.3.5 · p. 18

• Audio-visual perception progresses from pixel- and sample-level structure through semantic content understanding to reasoning about event relationships over time.Figure 8 schematizes this three-stage progression.Source: §5 · p. 18
• The section covers structure within each modality as well as joint audio-visual settings before addressing higher-level interpretation.Source: §5 · p. 18
• Unimodal material is treated as brief previews, while audio-visual problems receive more detailed coverage of cross-modal alignment, data, and benchmarks.This distinction follows the survey scope established in Section 3.Source: §5 · p. 18

5.1.1 Audio Perception

• Audio perception extracts waveform- or time-frequency-level primitives, including event presence, temporal boundaries, onsets, beats, voice activity, and separated sources.Self-supervised encoders support classification, tagging, and detection, while task-specific methods address localization and separation; general-purpose audio-text LLMs still lag on fine-grained temporal perception.Source: §5.1.1 · p. 19

5.1.2 Visual Perception

• Visual pixel perception grounds objects, actions, and events through spatially resolved detection, segmentation, tracking, and temporal action localization.Representative approaches span Faster R-CNN, YOLO, DETR, Mask2Former, SAM, ByteTrack, and transformer-based temporal detectors, supporting audio-visual grounding and synchronization.Source: §5.1.2 · pp. 19–20

5.1.3 Audio-Visual Event Localization/Grounding

• Audio-visual event localization identifies when and where events are simultaneously audible and visible, using cross-modal attention, adaptive fusion, semantic modeling, and pretrained encoders.Open-vocabulary and dense variants extend localization to unseen categories and untrimmed videos with overlapping events, supported by AVE, LLP, and UnAV-100.Source: §5.1.3 · p. 20

5.1.4 Audio-Visual Segmentation

• Audio-visual segmentation delineates sounding objects at pixel level while suppressing silent or off-screen sources, evolving from fused representations toward foundation-model prompting, reasoning, and generation.SAM-like priors, diffusion, and MLLMs improve generalization and reduce reliance on dense annotations.Source: §5.1.4 · pp. 21–22
• Ref-AVS and OmniAVS extend segmentation to textual and multimodal referring expressions, shifting toward reasoning-centric handling of open-vocabulary sounds, abstract references, and complex scenes.Methods include reinforcement learning, SAM-based prompting, and agentic reasoning, while benchmarks increasingly test semantic labels, distractors, long videos, and omnimodal queries.Source: §5.1.4 · p. 22

5.1.5 Audio-Visual Synchronization

• Audio-visual synchronization detects alignment, estimates temporal offsets, or classifies predefined offset intervals, progressing from lip-speech alignment to general cross-modal correspondence.Dual encoders and contrastive objectives dominate, with synchronized pairs as positives and shifted or cross-video pairs as negatives.Source: §5.1.5 · p. 23
• Offset classification increasingly serves as the preferred synchronization formulation because it offers more stable training and better reflects perceptual tolerance to small misalignment.Datasets range from AudioSet and VGGSound for representation learning to AVE, VGGSync, LRS, and RealSync for correspondence-focused evaluation.Source: §5.1.5 · pp. 23–24

5.2 Audio-Visual Content Understanding

• Audio-visual content understanding maps audio and visual signals to semantic interpretations, structured outputs, questions, and retrieval targets, emphasizing shared semantics and spatio-temporal evidence.The section distinguishes per-modality recognition from joint audio-visual reasoning and retrieval.Source: §5.2 · p. 24

5.2.1 Audio Understanding

• Audio understanding covers speaker recognition, emotion analysis, captioning, and audio question answering, progressing from engineered or task-specific systems toward neural, self-supervised, retrieval-augmented, and audio-language models.Representative approaches include d-vector [542], x-vector [496], ECAPA-TDNN, WavLM, Wav2Vec 2.0, Emotion2Vec, SALMONN [514], Audio Flamingo [279], and Qwen-Audio [106].Source: §5.2.1 · pp. 24–25

5.2.2 Visual Understanding

• Visual understanding includes recognition, grounding, captioning, and question answering, with multimodal large language models extending these capabilities while remaining limited in temporal reasoning, long-context memory, spatial intelligence, and reliability.Grounding additionally requires temporal localization, while captioning faces hallucination and evaluation limitations.Source: §5.2.2 · p. 25

5.2.3 Audio-Visual Question Answering

• Audio-visual question answering requires associating sounds with visual events and temporal dynamics, and the field is shifting toward language-centric AV-LLMs, fine-grained grounding, modality-aware routing, bias mitigation, and open-ended reasoning.Benchmarks increasingly test robustness, openness, audio indispensability, shortcut resistance, continual learning, and hallucination, including AVQA, MUSIC-AVQA, FortisAVQA [381], AVQACL, Valor32k-AVQA v2.0, DAVE [448], AVUT, and AVHBench [269].Source: §5.2.3 · pp. 26–27

5.2.4 Audio-Visual Cross-Modal Retrieval

• Audio-visual cross-modal retrieval learns shared representations across heterogeneous signals, advancing from coarse clip-level matching and paired discrimination toward contrastive and masked modeling, unified embeddings, dense correspondence, and finer-grained localization.Methods use hard-negative mining, multi-view objectives, asynchronous audio-visual relations, and equivariant constraints. Masked and contrastive objectives improve cross-modal semantics, alignment, efficiency, spatial localization, and fine-grained retrieval.Source: §5.2.4 · pp. 27–28

5.3.1 Audio Reasoning

• Audio reasoning adapts prompting, instruction tuning, and multi-task supervision to elicit multi-step inference from acoustic evidence, but complex temporal and causal reasoning remains difficult.Benchmarks including CLEAR, MMAR [388], and CMDAR [302] evaluate compositional, multimodal, temporal, causal, and domain-knowledge reasoning, with current Audio LLMs stronger on perception than complex inference [175].Source: §5.3.1 · p. 29

5.3.2 Visual Reasoning

• Visual reasoning evolved from symbolic task-specific systems toward vision-language foundation models using prompting, chain-of-thought supervision, and reinforcement learning for grounded, consistent, and tool-assisted inference.Video reasoning further incorporates temporal-aware rewards to align conclusions with relevant moments, while persistent hallucinations and spurious correlations motivate stronger grounding rewards and faithfulness benchmarks.Source: §5.3.2 · pp. 29–30

5.3.3 Audio-Visual Reasoning

• Audio-visual reasoning jointly interprets video and audio to infer why, when, and how events unfold through temporal alignment, causal inference, cross-modal grounding, and faithful rationales.Recent work targets grounded reasoning, self-correction, long-horizon credit assignment, hallucination suppression, and reduced visually dominated shortcuts through preference optimization, RL-style post-training, and specialized decoding.Source: §5.3.3 · p. 30
• Industrial and open omni-modal systems increasingly make reasoning explicit, including OpenAI-o3, Gemini 2.5/3.0 Pro [111], and Qwen3-Omni-Thinking [639].Benchmarks now extend beyond short closed-set QA toward temporal alignment, modality complementarity, hallucination diagnosis, cross-modal dependency, and long-form reasoning, yet reliable evaluation remains immature.Source: §5.3.3 · pp. 30–31, Table 9 · p. 31

6 Audio-Visual Generation

• The section organizes audio-visual generation into conditional unimodal generation, cross-modal translation, and joint audio-visual generation, reflecting increasingly coupled regimes.These regimes are summarized in Figure 9.Source: §6 · p. 31
• The field has progressed from composing strong single-modality generators toward explicitly coupled and unified architectures.Diffusion, flow matching, and multimodal transformers are the dominant design choices.Source: §6 · p. 31

6.1 Conditional Audio/Visual Generation

• Conditional generation covers independently controlled audio and visual synthesis or transformation, preceding cross-modal and joint audio-video generation.It uses text, reference media, structured controls, or instructions as external conditioning signals.Source: §6.1 · p. 31, Figure 9 · p. 32

6.1.1 Conditional Audio Generation

• Conditional audio generation spans text-to-audio synthesis, transformation and separation, and instruction-guided editing, with tokenizers and latent representations supporting fidelity, controllability, and efficiency.Representative systems include AudioLDM and Stable Audio Open, AudioLM, AudioX [529], TF-GridNet, Dual-Path Mamba, and InstructAudioEdit [588].Source: §6.1.1 · p. 32

6.1.2 Conditional Visual Generation

• Conditional visual generation includes text-to-image/video synthesis, controllable video generation, and image/video editing that preserve appearance, structure, motion, identity, or temporal coherence.The literature progresses from DALL-E [454], Stable Diffusion, Make-A-Video [495], and Imagen Video [216] toward newer foundation and product systems, while ControlNet, ControlVideo [725], InstructPix2Pix [48], and Video-P2P establish structured control and instruction-based editing.Source: §6.1.2 · pp. 32–33

6.2.1 Video-to-Audio Generation

• V2A generation is shifting from direct video-conditioned audio synthesis toward structured audiovisual planning followed by rendering, with stronger control, open-domain coverage, and longer temporal windows.Methods include contrastive audiovisual pretraining with latent diffusion, onset conditioning, flow matching, multimodal joint training, text-guided control, and multimodal reasoning modules [729] [295] [587] [726] [554] [479] [348] [347] [155].Source: §6.2.1 · pp. 33–34

6.2.2 Audio-to-Video Generation

• A2V generation is underdetermined because the same audio can support many visually plausible videos, making controllability increasingly important alongside realism despite reference images, motion priors, and explicit control signals.Source: §6.2.2 · pp. 34–35

6.2.3 Audio-Synchronized Image Animation

• Audio-synchronized image animation extends beyond classical talking heads to general audio-conditioned animation across multiple object and scene classes, with text providing auxiliary control over motion, emotion, or scene behavior.Keyframe-aware generation supports highly dynamic motions, while recent systems achieve more controllable general video generation and open-domain audio-synchronized visual animation.Source: §6.2.3 · pp. 35–36

6.2.4 Audio-Driven 3D Visual Generation

• Audio-driven 3D generation maps sound into dynamic facial, head, or body representations, offering viewpoint consistency, explicit geometry, and rendering compatibility while making subtle motion realism harder to learn.Approaches include explicit mesh deformation, neural rendering and Gaussian splatting, emotion disentanglement, gesture modeling, and coordinated body and background dynamics.Source: §6.2.4 · pp. 36–37

6.3 Joint Audio-Visual Generation

• Joint audio-video generation encompasses coupled synthesis and editing rather than attaching a unimodal output to another model afterward.The section covers text- and image-conditioned joint synthesis as well as workflows that edit video and sound together while preserving semantic consistency and alignment.Source: §6.3 · p. 37, §6.3.3 · p. 39

6.3.1 Text-to-Audio-Video Generation

• Text-to-audio-video generation requires within-modality quality, prompt faithfulness, and cross-modal synchronization, with architectures evolving from partially separate branches toward shared or tightly coupled attention.Synchronization uses temporal priors, frame-level cross-attention, aligned RoPE positions, and data scaling; multi-task systems extend beyond T2AV to A2V, V2A, and real-time generation.Source: §6.3.1 · p. 37
• Commercial T2AV systems remain ahead through larger audio-video corpora, stronger base generators, and heavier post-training, although LTX-2 [200] and MOVA [684] show open-source models are catching up.The remaining gap reflects combined data scale, base-model maturity, and post-training depth rather than a single architectural trick.Source: §6.3.1 · p. 38

6.3.2 Image-to-Audio-Video Generation

• Image-to-audio-video generation is moving from cascaded image-to-video and image-to-audio pipelines toward unified models that jointly infer motion and sound under tight spatial constraints.Integrated systems include Animate and Sound an Image, UniVerse-1 [552], Ovi [369], Klear [555], ALIVE [193], and SkyReels-V4 [69], but standardized benchmarks remain scarce.Source: §6.3.2 · p. 39

6.3.3 Joint Audio-Video Editing

• Joint audio-video editing is shifting from coarse modifications toward local, synchronous, minimal-side-effect intervention supported by finer supervision and unified audio-visual foundation models.Benchmarks have progressed from prompt-based and zero-shot settings to mask-aware and paired source-target evaluation of edit faithfulness, preservation, and synchronization.Source: §6.3.3 · pp. 39–40

7 Audio-Visual Interaction

• Audio-visual interaction comprises conversational responses in language or speech and embodied physical actions, making it a stricter test of AV intelligence than static understanding.Both settings require tightly coupled perception, memory, reasoning, and generation under latency constraints.Source: §7 · p. 40

7.1.1 Audio-Driven Conversation

• Audio-driven conversation is evolving from cascaded ASR–LLM–TTS pipelines toward end-to-end and token-based hybrids that better preserve latency, prosody, and acoustic detail.Cascades remain modular and controllable, while end-to-end systems improve latency and prosody preservation; token-based hybrids offer a practical compromise.Source: §7.1.1 · pp. 40–41

7.1.2 Unified Visual Understanding and Generation

• Unified visual models couple understanding and generation through MLLM-to-diffusion bridges, block-level fusion, shared-token autoregression, masked autoregression, or discrete diffusion.Hybrid systems generally deliver better visual quality, especially for video, while continuous or latent-token autoregression offers a potential middle ground.Source: §7.1.2 · pp. 42–43
• Unified video modeling remains less diverse and largely hybrid, pairing MLLMs with dedicated generators while shared-video-token and generation-centric designs remain early-stage.Existing benchmarks separately assess understanding and generation, leaving long-context failures across both capabilities insufficiently tested.Source: §7.1.2 · p. 42

7.1.3 Omni-Modal Audio-Visual Conversation

• Omni-modal conversation unifies audio-visual grounding with text or streaming speech through integrated encoders, LLM backbones, speech decoders, and increasingly any-to-any generation.Naturalistic multi-turn AV dialogue remains under-benchmarked, and scarce authentic conversational data—not architecture alone—is the primary bottleneck.Source: §7.1.3 · pp. 43–44

7.2.1 Audio-Visual Navigation

• Audio-visual navigation is shifting from reactive control toward deliberative, language- and memory-assisted policies, but improvements depend on tight grounding in low-latency sensory feedback.The task includes AudioGoal and AudioPointGoal settings, with methods addressing semantic events, noise, moving targets, sim-to-real transfer, and acoustic-field modeling.Source: §7.2.1 · pp. 44–45

7.2.2 Audio-Visual Scene Understanding and Reconstruction

• Embodied AV scene understanding prioritizes spatially consistent, queryable representations of geometry, acoustics, and language-grounded semantics that directly guide navigation, manipulation, and world modeling.The field is moving from echo-based reconstruction and joint visual-acoustic fields toward reusable spatial memory and updateable audiovisual world models.Source: §7.2.2 · pp. 45–46

7.2.3 Audio-Visual Embodiment Interaction and Manipulation

• Audio-visual embodiment interaction and manipulation use passive or active acoustic sensing to infer contact events such as slip, collision, pouring progress, and material change.Audio supplies contact-rich feedback that can resolve ambiguities left by RGB alone.Source: §7.2.3 · p. 47
• Audio-visual embodiment spans contact-aware visual enhancement, acoustic object-state sensing, audio-integrated world models, and auditory-token VLA control for closed-loop interaction.Representative works include SonicSense [349], world-model approaches [236] [678] [710], and VLA integration [591] [602] [738].Source: §7.2.3 · p. 47
• Embodied AVI benchmarks span in-the-wild demonstrations, multi-robot trajectories, real-world assembly, perception-action coupling, collision-aware manipulation, and sim-to-real evaluation.ManiWAV [366], ARIO [591], Kaiwu [248], AV grounding-and-act, and Audio-VLA [602] provide complementary benchmark settings; Audio-VLA introduces Task Completion Rate (TCR).Source: §7.2.3 · p. 47
• Sim-to-real transfer remains the defining obstacle because acoustic simulation is especially unreliable for contact-rich manipulation, where subtle sound differences reveal major task-state changes.Progress requires better acoustic rendering, domain-invariant representations, and training recipes that exploit simulation scale without overfitting to unrealistic sound statistics.Source: §7.2.3 · p. 47
• Audio-visual foundation models serve as a core intelligence layer across AIGC, human-centric services, immersive environments, and ubiquitous perception.Figure 12 summarizes this broader AVI application landscape.Source: Figure 12 · p. 48

8.1 AIGC and Creative Content Production

• Foundation models streamline film, video, music, and game workflows through synchronized Foley, dubbing, scene generation, text-to-audio creation, source separation, editing, and procedural sound synthesis.Video-to-audio models automate temporally aligned Foley and ambient audio, while newer systems generate short-form audio-visual scenes from text; music and sound tools support rapid prototyping, remixing, restoration, and context-aware game audio.Source: §8.1 · pp. 48–49

8.2 Digital Humans and Social Interaction

• Digital-human systems have advanced from pixel-level lip synchronization to expressive 3D avatars and identity-consistent personas for influencers, telepresence, companions, accessibility, and multilingual interaction.Mesh-based and neural-rendering methods support speech, facial expression, and co-speech gestures, while one-shot or few-shot identity priors preserve appearance across poses, expressions, sessions, and media.Source: §8.2 · p. 49

8.3 Human-Centric Intelligent Services

• Audio-native and omni-modal systems enable natural dialogue and meeting analytics, while education and accessibility applications adapt instruction, generate synchronized lessons, and translate sensory content across modalities.These systems preserve paralinguistic cues, extract meeting actions and summaries, adapt tutoring to engagement, create realistic training scenarios, and provide captioning, audio description, audio question answering, and sign-language avatars.Source: §8.3 · pp. 49–50

8.4 Immersive Experience and Metaverse

• Immersive systems jointly model visual scenes and acoustic propagation to deliver position-dependent spatial audio, semantic 3D interaction, and virtual experiences, but require inference latency often below 20ms.Neural acoustic fields and AV-NeRF support 6DoF rendering with attenuation, occlusion, and room acoustics; audio-visual language maps and echo-based reconstruction enable spatial queries and geometry inference when line-of-sight is limited.Source: §8.4 · p. 50

8.5 Embodied AI and Robotics

• Embodied agents use complementary visual and acoustic evidence for navigation, manipulation, and human-robot collaboration, including sound-source localization, contact sensing, verbal control, and feedback-driven action.Audio supports perception under occlusion, low light, and beyond the visual field, while contact sounds reveal material, slip, and grasp stability; deployment emphasizes real-time reliability, failure detection, and sim-to-real transfer.Source: §8.5 · pp. 50–51

8.6 Ubiquitous Perception and Security

• Ubiquitous audio-visual perception improves public safety, forensics, industrial monitoring, and IoT by correlating complementary evidence for anomaly detection, authentication, maintenance, inspection, and environmental sensing.Practical deployments require efficient edge architectures and increasingly privacy-preserving local processing or semantic transmission, while deepfake detection examines lip-speech consistency to protect trust in digital evidence.Source: §8.6 · p. 51

9 Open Challenges, Limitations, and Future Directions

• The field’s central challenge is modeling audio and vision as partially observed views of dynamic worlds shaped by events, space, materials, occlusion, and social context.AVI must explain why modalities co-occur and how their evidence relates, rather than treating them as generic multimodal signals.Source: §9 · p. 51
• Future AVI should advance along six axes: causal event-source grounding, audio-visual world models, context memory, causal intervention, verifier ecosystems, and interactive responsible systems.Figure 13 frames the roadmap from correspondence, perception, and generation through interactive embodied systems toward causal-contextual and verifiable agentic AVI.Source: §9 · p. 51, Table 24 · p. 52, Figure 13 · p. 52

9.1 From Temporal Synchronization To Causal Event-Source Grounding

• Causal event-source grounding must move beyond synchronization to explain which source produced each sound through which causal path, using event graphs and counterfactual supervision.Training should distinguish visible-but-silent sources, off-screen events, shifted onsets, material conflicts, and mixtures with only one correct grounding.Source: §9.1 · p. 53, Figure 14 · p. 53

9.2 From Paired Clips to Action-Conditioned Audio-Visual World Models

• Action-conditioned audio-visual world models should predict compact latent states covering sources, materials, acoustics, affordances, intent, and uncertainty for embodied action and counterfactual queries.Evaluation should prioritize navigation gains, manipulation success, spatial reasoning, and counterfactual prediction over waveform or pixel reconstruction alone.Source: §9.2 · pp. 53–54

9.3 Audio-Visual Context Memory

• Audio-visual context memory should replace raw token accumulation with layered buffers, event memory, evidence-tethered summaries, and continuously updated user/task state.OmniVideoBench duration splits show that high short-video accuracy does not automatically yield robust ultra-long-video accuracy; benchmarks should test delayed recall, audio-critical questions, conflicts, and belief revision.Source: §9.3 · p. 54, Figure 15 · p. 54

9.4 Causal AV Intervention

• Causal AV intervention should represent scenes with graphs and disentangled latents, changing requested consequences while preserving unrelated sources, identities, ambience, and narrative state.Verifier feedback must inspect both modalities because visual improvement can otherwise conceal synchronization drift.Source: §9.4 · p. 55

9.5 Verifier and Reward Ecosystems

• AVI evaluation and training need layered verifiers for signal fidelity, synchronization, source grounding, causal coherence, utility, safety, identity, and provenance.Scalar metrics can miss wrong sources, temporal hallucinations, or physically inconsistent prompts, so optimization should use adversarial and counterfactual probes with explicit failure taxonomies.Source: §9.5 · pp. 55–56, Figure 16 · p. 55

9.6 Interactive and Responsible AVI

• Interactive and responsible AVI must coordinate fast reactions, deliberation, uncertainty-based deferral, privacy safeguards, provenance, consent, and user-controlled memory in streaming embodied systems.The overall frontier is integrating causal grounding, world models, faithful memory, causal control, multimodal verification, and safe human deployment.Source: §9.6 · p. 56

Conclusion

• The survey unifies AVI research into representation-centric signal embedding, generation-centric cross-modal synthesis, and LLM-centric reasoning methods.It traces understanding from event recognition to spatial reasoning and emphasizes diffusion and autoregressive modeling for synchronized, high-fidelity generation.Source: §10 · p. 56
• Cross-modal alignment remains the foundational challenge across audio-visual understanding, generation, and reasoning tasks.Source: §10 · p. 56
• Unified omni architectures that combine perception and generation introduce new challenges in large-scale temporal synchronization and training stability.Source: §10 · p. 56
• As AVI expands into embodied robotics and immersive XR, responsible progress requires efficiency, robustness, safety, and mitigation of deepfake and privacy risks.Source: §10 · p. 56