PaperlayerView exact revision on arXiv

Source-linked AI summary

EgoAVU: Egocentric Audio-Visual Understanding

Ashish Seth, Xinhao Mei, Changsheng Zhao, Varun Nagaraja, Ernie Chang, Gregory P. Meyer, Gael Le Lan, Yunyang Xiong, Vikas Chandra, Yangyang Shi, Dinesh Manocha, Zhipeng Cai

arXiv:2602.06139v1cs.CV

TL;DR

Egocentric audio–visual understanding lacks coherent joint-modality supervision, limiting evaluation of whether MLLMs connect sounds with visual events. EgoAVU generates enriched multimodal narrations and QA data through cross-modal modeling and curation, yielding up to 113% relative improvement on EgoAVU-Bench after finetuning, with gains transferring to other benchmarks.

  • Problem

    Egocentric videos challenge visual-only understanding, while existing data and benchmarks provide limited supervision and evaluation for integrated audio–visual reasoning.

  • Method

    EgoAVU enriches narrations with multimodal context, filters videos, models cross-modal relations with context graphs, and generates diverse QA pairs.

  • Results

    Up to 113% relative performance improvement is achieved on EgoAVU-Bench after finetuning, with up to 28% relative gains transferring to EgoTempo and EgoIllusion.

  • Takeaways & Limitations

    EgoAVU-Bench reveals vision bias and audio neglect in existing MLLMs, while EgoAVU-Instruct improves joint audio–visual understanding across egocentric benchmarks.

Abstract

from arXiv · show

Understanding egocentric videos plays a vital role for embodied intelligence. Recent multi-modal large language models (MLLMs) can accept both visual and audio inputs. However, due to the challenge of obtaining text labels with coherent joint-modality information, whether MLLMs can jointly understand both modalities in egocentric videos remains under-explored. To address this problem, we introduce EgoAVU, a scalable data engine to automatically generate egocentric audio-visual narrations, questions, and answers. EgoAVU enriches human narrations with multimodal context and generates audio-visual narrations through cross-modal correlation modeling. Token-based video filtering and modular, graph-based curation ensure both data diversity and quality. Leveraging EgoAVU, we construct EgoAVU-Instruct, a large-scale training dataset of 3M samples, and EgoAVU-Bench, a manually verified evaluation split covering diverse tasks. EgoAVU-Bench clearly reveals the limitations of existing MLLMs: they bias heavily toward visual signals, often neglecting audio cues or failing to correspond audio with the visual source. Finetuning MLLMs on EgoAVU-Instruct effectively addresses this issue, enabling up to 113% performance improvement on EgoAVU-Bench. Such benefits also transfer to other benchmarks such as EgoTempo and EgoIllusion, achieving up to 28% relative performance gain. Code will be released to the community.

1 Introduction

EgoAVU addresses limited coherent audio–visual supervision for egocentric videos by generating enriched multimodal data and benchmarks. Its datasets expose vision bias in existing MLLMs and support substantial gains after finetuning.

  • Dynamic camera motion and limited field of view make visual-only egocentric understanding challenging, motivating persistent audio context.
  • Existing egocentric datasets emphasize human–object interactions, while existing benchmarks mainly assess visual cues rather than integrated audio–visual reasoning.
  • EgoAVU enriches narrations with environmental context, object details, audio captions, cross-modal event relations, and diverse understanding tasks.
  • 3M audio–visual-language samples form EgoAVU-Instruct, while EgoAVU-Bench contains 3K manually verified samples from 900 videos.
  • Up to 113% relative performance improvement on EgoAVU-Bench follows finetuning, with gains transferring by up to 28% to EgoTempo and EgoIllusion.

2 Related Work

Recent MLLMs extend language models to unified visual and auditory understanding, while egocentric video research has advanced through large-scale datasets. However, existing egocentric MLLMs still face limitations in incorporating audio.

  • Recent MLLMs extend large language models beyond text to unified understanding of visual and auditory inputs.
  • Leading egocentric video models such as MM-Ego and EgoVLPv2 lack the ability to incorporate audio.
  • Egocentric video understanding is increasingly studied because of its relevance to augmented reality and embodied AI.
  • Large-scale datasets such as Ego4D, EPIC-KITCHENS, and Ego-Exo4D have driven progress in egocentric video–language research.

3 Method

EgoAVU is a scalable, modular pipeline that converts egocentric videos and narrations into diverse, cross-modally grounded audio-visual data. It enriches and filters clips, explicitly models cross-modal relations, and generates unified narrations and QA datasets.

  • 3.1 Data Collection: EgoAVU collects Ego4D videos with audio tracks and groups consecutive narration segments into clips lasting 10–360 seconds.Narration timestamps define temporal boundaries, while grouping addresses the limited information in segments averaging 3 seconds.
  • 3.2 Multimodal Narration Enhancement: The pipeline separately generates detailed visual, video-level, and auditory descriptions to reduce errors caused by joint audio-visual modality bias.Qwen2.5-VL captions spatial object details, while Qwen2.5-Omni independently captions visual temporal dynamics and foreground or background sounds.
  • 3.2 Multimodal Narration Enhancement: 54.3% audio and 25.4% visual consistency error rates for Qwen2.5-Omni motivate modular processing after models omit sounds or associate audio with incorrect visual events.MiniCPM-o shows corresponding error rates of 68.2% and 31.2%, respectively, in the reported consistency experiment.
  • 3.3 Video Filtering for Diversity Enhancement: EgoAVU filters for multimodal diversity by computing MATTR over combined narrations and retaining videos with MATTR above τ = 0.3.The threshold removes the bottom 25% of the distribution and produces a final video count of 9,900.
  • 3.4 Audio-Visual Narration Generation: A Multimodal Context Graph represents interacted objects, background objects, foreground sounds, and background sounds before an LLM fuses them into coherent audio-visual narrations.The graph makes cross-modal relationships explicit, including links between action sounds and their corresponding actions or visible scene elements.

4 Results

EgoAVU-Bench exposes substantial weaknesses in existing MLLMs’ joint audio–visual reasoning, while fine-tuning on EgoAVU-Instruct produces consistent gains across tasks and transfers to other benchmarks.

  • Main Results: Below 1.6 out of 5, all baseline MLLMs score on Source-Sound Localization, indicating difficulty associating sounds with their visual sources.The metric is the LLM-as-a-judge score (S).
  • Main Results: Below 2.4, even Qwen2.5-Omni (7B) scores on AVDN and AVSN, alongside low ROUGE-L and METEOR caption metrics.These results indicate weak coherence and temporal alignment in audio-visual narrations.
  • Main Results: 53.2% is the highest Temporal Reasoning accuracy, achieved by Qwen2.5-Omni (7B), while VideoLLaMA2 and MiniCPM-o perform considerably worse.The result reflects limited temporal reasoning over joint audio-visual inputs.
  • Main Results: Below 43% accuracy, baseline MLLMs perform on Audio-Visual Hallucination, frequently hallucinating during audio-visual reasoning.Figure 6 illustrates fabrication when models are asked about sound sources absent from the video.
  • Main Results: Up to 113.3% and 44.5% relative performance improvements are achieved on open- and close-ended EgoAVU-Bench tasks after fine-tuning on EgoAVU-Instruct.Both LoRA and full fine-tuning provide considerable gains.
  • Generalization: Up to 28.1% accuracy gains transfer to EgoTempo and EgoIllusion, while EgoSchema decreases marginally by 0.1%.The fine-tuned model also maintains competitive performance on EgoSchema and strong performance on VideoMME and AVQA.
  • Error Analysis: Over 72% of MiniCPM-o and Phi4-mm errors arise from incorrect or missed sound descriptions rather than misidentified human–object interactions.This error pattern links poor performance primarily to sound perception and interpretation.

5 Conclusion

The paper presents EgoAVU as a data engine and shows that its datasets improve joint egocentric audio–visual understanding. It also identifies residual noise from open-source MLLM outputs as a limitation.

  • Conclusion: EgoAVU enhances egocentric narrations with modularized MLLMs and uses multi-modal context graphs to generate diverse, high-quality audio-visual QA pairs.The approach addresses data limitations in egocentric audio-visual understanding.
  • Conclusion: EgoAVU-Bench reveals consistent vision bias in existing MLLMs, including neglected or hallucinated audio information.EgoAVU-Instruct mitigates this gap and improves performance on EgoAVU-Bench and existing egocentric benchmarks.
  • Limitation: The training data still contains noise from open-source MLLM outputs despite carefully designed filtering techniques.The paper identifies more sophisticated approaches as future work and expects improvement as unimodal MLLM capabilities advance.

A Additional Details on EgoAVU

The paper documents prompts used to generate multimodal context graphs, audio-visual narrations, and task-specific question–answer pairs.

  • Prompt Structure: EgoAVU prompts define objectives, input formats, task descriptions, and general instructions for multiple generation modules.The modules include the Multi-modal Context Graph, Audio-Visual Narration, and task-specific QA generation.

A.2 MATTR

MATTR filters videos for multimodal diversity by retaining clips whose moving-average type-token ratio exceeds a manually validated threshold.

  • Filtering Criterion: MATTR with window size w = 200 and threshold τ = 0.3 removes the bottom 25% of videos to filter static or repetitive descriptions.The threshold was selected through manual inspection of 100 randomly sampled videos across MATTR ranges.
  • Filtering Criterion: Videos below τ = 0.3 predominantly contain repetitive actions, limited object diversity, and minimal auditory variation.Videos above the threshold exhibit richer multimodal dynamics.

A.3 Ablation on Multi-Modal Context Graphs

The ablation compares EgoAVU’s MCG-based narration pipeline with a direct baseline on 200 video clips. MCG substantially reduces errors in sound-source associations and action-sequence coverage while improving overall audio-visual coherence.

  • Experimental Setup: The ablation evaluated MCG against a direct LLaMA-3-70B baseline on 200 randomly sampled video clips.The baseline generated narrations directly from enhanced video, image, audio, and action narrations without an intermediate MCG.
  • Evaluation Criteria: The manual evaluation measured sound-source associations, action sequences, and overall audio-visual coherence.
  • Results: 41.0% of direct-baseline captions contained errors, including missed or misassociated sounds and omitted action or interaction details.Errors affected 82 of 200 captions: 48 involved sound sources, 31 involved action sequences or interactions, and 17 involved both issues.
  • Results: 10.5% of MCG-based captions contained errors, representing a 76.1% relative error reduction versus the direct baseline.The MCG pipeline reduced errors from 82 to 21 captions out of 200.

B Manual Effort for EgoAVU-Bench Construction

EgoAVU-Bench underwent manual verification across 3,000 question-answer pairs from 900 egocentric videos. Annotation checked temporal and audio-visual grounding, corrected open-ended responses, and improved close-ended distractors.

  • Verification Scope: Annotators manually verified all 3,000 question-answer pairs covering 900 egocentric videos.Each video review took approximately 2–3 minutes and checked temporal alignment and audio-visual correspondence.
  • Verification Scope: 50.8% of question-answer pairs were modified during manual verification.
  • Correction Types: Open-ended-task corrections addressed missing sounds, incorrect human-object interactions, and sound-source misalignments.The open-ended tasks included SSA, AVDN, and AVSN.
  • Correction Types: Close-ended verification checked answer correctness and made distractors challenging and plausible without being similar to the correct answer or obviously wrong.This process applied to temporal reasoning and audio-visual hallucination tasks.
  • Annotation Effort: The complete manual verification required approximately 225 hours of human annotation effort.

D Additional Experiment Details

Additional experiments describe EgoAVU’s training configuration, LLM-as-judge validation, and prompt-based procedures for extracting multimodal information and generating narrations and question-answer pairs. The prompts cover spatial, temporal, audio, action, object, hallucination, and reasoning tasks.

  • Training Details: Training uses 30,000-token contexts, 1 FPS sampling, 256 × 256 frames, DeepSpeed ZeRO-3, five epochs, and balanced task sampling.The learning rate is 1 × 10^-4 with a cosine schedule and 10% warmup, for both LoRA and full fine-tuning.
  • Evaluation: The LLM-as-judge achieves an 87.6% human-alignment rate on 300 randomly sampled open-ended EgoAVU-Bench question-answer pairs.The evaluation follows prior work and measures agreement between LLM-based scoring and human judgments.
  • Narration Pipeline: The pipeline organizes enriched narrations into a Multi-Modal Context Graph before parsing it into a dense audio-visual narration.The resulting narration is then used to generate question-answer pairs for joint audio-visual understanding.
  • Question-Answer Generation: Prompted QA generation covers sound-source association, audio-visual hallucinations involving sounds, actions, and objects, and temporal reasoning over event order.Temporal prompts generate before, after, and event-ordering questions from chronological narrations.
  • Narration and Evaluation Prompts: Dense narration prompts combine chronological 10-second captions into a single fluent paragraph, while grading prompts compare predicted answers with grounding answers for a question.
Loading 2602.06139v1…