PaperlayerView exact revision on arXiv

Source-linked AI summary

EVA-Cap: Optimizing Audiovisual Video Captioning via Event-Centric Alignment

Jiapeng Shi, Qiuxia Hou, Jiaqi Leng, Junke Wang, Yanhao Zhang, Haonan Lu, Ziyi Ye, Zuxuan Wu

arXiv:2610.22186v1cs.CVcs.CL

TL;DR

Holistic text supervision can miss video event structure and fine-grained audiovisual alignment. EVA-Cap uses event-centric data and SFT plus GRPO to optimize event alignment, and outperforms existing open-source models across six audiovisual benchmarks.

  • Problem

    Holistic caption-level supervision can overlook video structure and event-level correspondences, while narrow supervision struggles to achieve fine-grained audiovisual alignment.

    Source: §1 · pp. 1–2

  • Method

    EVA-Cap uses event-centric quality-controlled data and two-stage SFT and GRPO, with rewards for event completeness, character attribution, and audiovisual synchronization.

    Source: §1 · p. 2

  • Results

    EVA-Cap outperforms existing open-source models across 6 audiovisual benchmarks and surpasses existing methods by 3.3% on WorldSense.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results suggest event-centric alignment is an effective path toward more faithful and temporally coherent audiovisual video captioning.

    Source: §5 · p. 11

    In rapid scene transitions, visual omissions and audio-visual misalignment still occur; in multi-character scenes with off-screen speech, speaker identity can be bound incorrectly.

    Source: §A.5 · p. 18

Abstract

from arXiv · show

Recent advances in Omni Language Models (OLMs) provide a unified solution for audiovisual video captioning. Nevertheless, existing approaches predominantly treat captioning as an unstructured text generation process, neglecting the inherent semantic structure of videos and leading to suboptimal fine-grained audiovisual alignment. To bridge this gap, we propose EVA-Cap, a novel framework that decomposes captions into atomic audiovisual events and transforms unstructured text supervision into event-centric alignment. We train EVA-Cap via Supervised Fine-Tuning (SFT) and Group Relative Policy Optimization (GRPO) on a self-curated dataset, EVA-Data, which is built with event-centric quality control. During the GRPO phase, the model is explicitly optimized against three event-centric alignment targets: global event completeness, intra-event attribution, and inter-event synchronization. Extensive experiments demonstrate that EVA-Cap significantly outperforms existing methods in event-centric evaluation (e.g., +9.0% average gain on ChronusAV), while achieving competitive performance on general audiovisual captioning benchmarks (e.g., +3.3% on WorldSense).

1 Introduction

Existing audiovisual captioning often relies on holistic text supervision, which can miss videos’ event structure and fine-grained audio-visual correspondences. EVA-Cap uses event-centric data and post-training to target these alignments, and is reported to outperform existing methods across six audiovisual benchmarks.

1 Introduction

  • Holistic caption-level supervision can miss video structure and event correspondences, while supervision limited to dialogue or object attributes struggles to deliver fine-grained audiovisual alignment.The introduction identifies a need for a caption representation that unifies video-grounded structural semantics with language modeling.Source: §1 · pp. 1–2
  • EVA-Cap represents videos as sequences of atomic audiovisual events, aligning event content and correspondences to compose coherent captions.Each event is a tuple of type, character, and content; the framework models attribution within events and temporal order and synchronization between events.Source: §1 · p. 2
  • EVA-Cap builds on Qwen2.5-Omni and trains on 63K EVA-Data examples using SFT followed by GRPO with rewards for event completeness, character attribution, and synchronization.The dataset uses an initialization–refinement pipeline and per-event verification of timestamped audiovisual events.Source: §1 · p. 2
  • EVA-Cap outperforms existing open-source models across 6 audiovisual benchmarks, including surpassing existing methods by 3.3% on WorldSense.On event-centric evaluation, it reports 55.4% Visual-to-Audio inference accuracy, 56.2% Audio-to-Visual grounding accuracy, and 40.1% speaker attribution accuracy.Source: §1 · p. 2

2 Related Work

Prior audiovisual captioning work addresses data quality and alignment at different levels, from holistic supervision to narrowly scoped caption aspects. Prior captioning reinforcement-learning methods likewise use limited or coarse objectives, motivating joint event-centric rewards.

2.1 Audiovisual Video Captioning.

  • AVoCaDO builds large-scale audiovisual video-caption pairs but has insufficient factual verification and coarse temporal alignment; UGC-Captioner [43] uses holistic supervision, while DiaDem and D-ORCA [30] target specific caption aspects.These approaches are contrasted with event-enhanced data quality control and event-level alignment training.Source: §2.1 · p. 3

2.2 Reinforcement Learning for Visual Content Captioning.

  • CapRL and RubiCap [16] use verifiable rubrics for dense image captioning, while OwlCap, VideoCap-R1 [22], D-ORCA [30], and TimeChat-Captioner target key information, dialogue, or timestamp alignment.The paper characterizes these approaches as limited in scope or reliant on coarse audiovisual synchronization, in contrast to its three jointly applied event-centric rewards.Source: §2.2 · p. 3

3 Method

EVA-Cap pairs an event-checked audiovisual caption dataset with SFT and GRPO that explicitly rewards event coverage, character attribution, and audiovisual synchronization.

3 Method

  • EVA-Cap combines a 63K-scale audiovisual caption dataset with a two-stage SFT–GRPO strategy and event-centric rewards for fine-grained alignment.The method is built on Qwen-2.5-Omni and integrates event-level supervision into post-training.Source: §3 · p. 3, §3.2 · p. 5

3.1.1 Annotation Pipeline

  • The annotation pipeline drafts descriptions with AVoCaDO, refines temporal alignment with Gemini-2.5-Pro, then validates event hits and discards samples below 0.8.An event hit requires audio and visual events to be entailed with consistent temporal synchronization.Source: §3.1.1 · pp. 3–4

3.1.2 Data Statistics

  • From 100K videos under 200 seconds, EVA-Data retains 63K pairs; sampled captions contain audio and visual events without modality bias.Videos average 73.7 seconds, and captions average 504.3 words.Source: §3.1.2 · pp. 4–5

3.2.1 Training Pipeline

  • SFT on 63K EVA-Data precedes GRPO on 3.2K rollout videos, with two reward rounds progressing from event coverage and synchronization to character attribution.The first round sets attribution reward to zero; the second uses all three reward terms.Source: §3.2.1 · p. 5, §3.2.2 · p. 7

3.2.2 Event-Centric Reward Modeling

  • Captions become modality-grounded event sequences encoding modality, type, character, and content, scored for event coverage, character assignment, and audiovisual synchronization.The three targets are global event completeness, intra-event attribution, and inter-event synchronization.Source: §3.2.2 · p. 6
  • Synchronization is scored as retrieval: an LLM selects visual candidates synchronized to each audio query, then set-level F1 compares predicted and reference indices.The reference query includes an audio event, candidate visual events, and synchronized visual indices.Source: §3.2.2 · pp. 6–7

4 Experiments

EVA-Cap is evaluated on general captioning, QA-cascaded, and event-centric benchmarks, with ablations of its post-training stages and rewards. Results show gains on QA-cascaded and event-centric tasks, while the ablations examine how SFT, reward choices, and training order contribute.

4.1 Experimental Setup

  • The evaluation covers direct caption-quality assessment, QA-cascaded benchmarks, ChronusAV synchronization subsets, and AV-SpeakerBench speaker-recognition samples.ChronusAV uses 1,066 videos under 200 seconds; AV-SpeakerBench evaluation uses 422 QA samples.Source: §4.1 · pp. 7–8

4.2 Main Results

  • +3.3% on Daily-Omni and +3.3% on WorldSense are EVA-Cap’s reported gains over existing open-source models in QA-cascaded evaluation.Gemini-2.5-Pro answers questions based on generated captions, and QA accuracy reflects caption quality.Source: Table 2 · p. 8, §4.2 · p. 9
  • EVA-Cap significantly outperforms the compared models across all event-centric evaluation metrics.The evaluation includes V2A and A2V grounding and speaker attribution.Source: §4.2 · p. 9, Table 3 · p. 9

4.3 Ablation Studies

  • After SFT on EVA-Data, V2A accuracy rises from 8.4% to 44.8% and A2V accuracy from 11.9% to 46.1%.The ablation also reports improvements in general audiovisual captioning.Source: §4.3 · p. 9, Table 4 · p. 9
  • The completeness reward yields accuracy gains of +8.5% on V2A, +2.1% on AV-Speaker, and +3.7% on Daily-Omni; synchronization and attribution rewards add task-specific gains.The latter rewards yield +2.1% on A2V and +3.1% on AV-Speaker accuracy, respectively.Source: §4.3 · pp. 9–10, Table 4 · p. 9
  • Introducing attribution reward in the first round degrades general and event-centric performance, while progressive weighting moves from coarse structure to character-level detail.The authors hypothesize that early synchronization and attribution rewards conflict; a comparison indicates later gains reflect reward design rather than data scaling alone.Source: Table 4 · p. 9, §4.3 · p. 10

4.4 Visualizations

  • In a multi-view sidewalk scene, EVA-Cap aligns a puppy’s speech with the conversing characters, whereas AVoCaDO links it to the adjacent crosswalk scene and misattributes the speaker.Figure 5 marks synchronization features in bold, accurate attribution underlined, and correct or incorrect alignments in green or red.Source: §4.4 · p. 10, Figure 5 · p. 10

5 Conclusion

EVA-Cap combines event-centric data curation with two-stage SFT and GRPO training, using rewards for event completeness, attribution, and synchronization. The reported findings support event-centric alignment as an effective path toward faithful, temporally coherent audiovisual captioning.

5 Conclusion

  • EVA-Cap transforms holistic caption optimization into event-centric alignment and supports event-level supervision with EVA-Data and two-stage SFT and GRPO.The training framework decomposes captions into modality-grounded events and uses rewards for event completeness, character attribution, and audiovisual synchronization.Source: §5 · pp. 10–11
  • SFT on EVA-Data improves OLMs’ event-centric knowledge, while subsequent GRPO improves event-centric alignment and maintains strong general captioning performance.Source: §5 · p. 11
  • The results suggest event-centric alignment offers an effective path toward more faithful and temporally coherent audiovisual video captioning.Source: §5 · p. 11

A.1.1 Formulation of Supervised Fine-Tuning

SFT trains on EVA-Data to generate long-form audiovisual captions from video-instruction pairs, using next-token prediction and providing initialization for GRPO.

A.1.1 Formulation of Supervised Fine-Tuning

  • SFT uses standard next-token prediction on ground-truth captions to train long-form outputs integrating audio and visual information.The training example is a video-instruction pair and its ground-truth caption.Source: §A.1.1 · p. 14
  • SFT provides the initialization for the subsequent GRPO optimization.Source: §A.1.1 · p. 14

A.1.2 Formulation of Group Relative Policy Optimization

GRPO samples groups of captions, scores them against reference captions and event structures, and optimizes the policy using normalized group-relative advantages and a clipped objective with KL regularization.

A.1.2 Formulation of Group Relative Policy Optimization

  • For each audiovisual instruction, the old policy samples a group of outputs and scores each against the reference caption and event structures.Each output receives an event-centric reward.Source: §A.1.2 · p. 14
  • Each response’s advantage is its reward normalized by the mean and standard deviation of rewards within the sampled group.Source: §A.1.2 · p. 14
  • The policy uses a clipped surrogate objective with a KL penalty relative to the reference policy.The clipping range and KL-penalty weight control the objective.Source: §A.1.2 · p. 14

A.2.1 Details about the Construction of Reference Structures

These supplementary sections detail how EVA-Cap constructs event-level GRPO supervision, tests the data and reward components, and discusses evaluation conditions, limitations, and broader impacts.

A.2.1 Details about the Construction of Reference Structures

  • For 6.4K videos with 100% Stage 3 hit ratios, reference structures pair filtered audio events with five neighboring visual candidates to create synchronization queries and hard negatives.Audio events spanning more than five visual-event indices are discarded; these structures support event-level reward calculation for GRPO.Source: §A.2.1 · pp. 14–15

A.2.2 Details about the Rollout Process

  • Rollouts score attribution and synchronization; query- and video-level filtering selects 3.2K samples, and two GRPO rounds progress from completeness plus synchronization to all three rewards.The rollout process generates eight captions per video; filtering requires at least five valid synchronization queries and retains the top half of qualified videos by reward-score variability.Source: §A.2.2 · p. 15

A.3.1 Ablation on the Independent Effect of Data and Reward

  • EVA-Data’s event-centric refinement improves A2V grounding by +4.9% and Daily-Omni by +3.6%; the reward framework also improves event-centric and general benchmarks on ASID-Captioner.The data and reward suites are assessed separately; additional experiments use a reduced GRPO run with 1.6K samples and Qwen3.5-27B as judge.Source: §A.3.1 · pp. 15–16, §A.3 · p. 15, Table 5 · p. 16

A.3.2 Robustness of the Reward Judge

  • Reward-judge agreement between Qwen3.5-27B and Gemini-2.5-Pro is around 85%, while Qwen3.5-27B and Gemma-4-31B-it yield similar improvements over SFT.The reported consistency spans event-centric and general benchmarks and supports robustness to judge choice.Source: §A.3.2 · pp. 16–17, Table 7 · p. 16

A.3.3 Length-Controlled Hallucination Evaluation

  • Under length-matched evaluation, EVA-Cap has lower hallucination than DiaDem in short and mid-length splits, while DiaDem is lower in the longest split.The authors characterize EVA-Cap as more reliable for compact captions but still behind strong baselines on long-form outputs.Source: Table 8 · p. 17, §A.3.3 · p. 17

A.3.4 Experiments with Event Precision Reward

  • Completeness reward increases average caption length from 633.0 to 748.2 tokens while per-token hallucination changes from 8.2 to 8.1 Hall./1k tokens; adding precision shortens captions but raises Hall./1k from 8.1 to 8.9.Precision also increases miss rate, slightly hurts general performance, and adds judge calls; the authors exclude it, noting attribution reward more directly penalizes hallucinated or incorrectly grounded events.Source: §A.3.4 · p. 17, Table 9 · p. 17

A.4 Discussion with Recent Work

  • EVA-Cap’s reward process decouples audio and visual events, unlike AVSCap’s unified synchronized-event treatment; experiments comparing these choices are left for future work.The discussion also reports omissions and cross-modal misalignment in rapid transitions, and incorrect speaker binding with off-screen speech; EVA-Data still relies on proprietary MLLMs, with cross-verification proposed to reduce dependence on single models.Source: §A.4 · p. 18, §A.5 · pp. 18–19

A.6.1 Prompt for Annotation Pipeline

  • Annotation prompts refine captions, extract timestamped event tuples, verify event hits, and standardize reward judgments; the paper notes risks including misleading errors, privacy concerns, and misuse for surveillance or profiling.The stated broader-impact benefit is that better audiovisual alignment may reduce misleading descriptions when both modalities are needed.Source: §A.6.1 · p. 19, §A.6.2 · p. 19, §A.7 · p. 19
Loading 2610.22186v1…