PaperlayerView exact revision on arXiv

Source-linked AI summary

MLLM Frame Subset Ensembling for Audio-Visual Video QA and MLLM-based Reranking for Ad-hoc Video Search in TRECVID 2025

Andreas Goulas, Damianos Galanopoulos, Ioannis Patras, Vasileios Mezaris

arXiv:2610.23214v1cs.CV

TL;DR

The paper addresses video retrieval from free-text queries and question answering about videos in TRECVID 2025. It combines foundation-model AVS retrieval and reranking with fine-tuning and frame-subset prediction aggregation for VQA. The reported AVS results favor reranking, while the VQA system ranks first across three Answer Generation metrics and second in Multiple Choice.

  • Problem

    The paper addresses the TRECVID tasks of retrieving relevant shots from free-text queries and answering questions about videos.

    Source: §1 · p. 1

  • Method

    For AVS, it combines query augmentation and multiple VLMs for retrieval with MLLM reranking; for VQA, it fine-tunes an audio-visual MLLM and aggregates predictions from different frame subsets.

    Source: §2.1 · p. 3, §4 · p. 10

  • Results

    AVS reranking improves the final rankings, while the VQA system achieves the highest scores across three Answer Generation metrics and the second-highest score in Multiple Choice.

    Source: §4 · p. 10

  • Takeaways & Limitations

    The reported results show the AVS framework producing more contextually relevant rankings and the VQA approach performing strongly in both tasks.

    Source: §4 · p. 10

    Evaluation parsing errors affected 0.7% of multiple-choice questions, and those answers were marked incorrect in the ablation study.

    Source: §3.4.3 · pp. 9–10

Abstract

from arXiv · show

This paper presents the overview of the runs related to the Ad-hoc Video Search (AVS) and Video Question Answering (VQA) tracks of TRECVID 2025 on behalf of the CERTH-ITI team. For the AVS track, we introduce a two-stage framework built on foundation models. In the first stage, multiple vision-language models (VLMs) encode both the input query, augmented through LLM-generated rephrasings, and the candidate video shots, producing weighted similarity scores for initial retrieval. In the second stage, we utilize a Multimodal-LLM(MLLM)-based reranking module that evaluates the semantic alignment between each shot among the top-N highest-ranked ones and the original query, generating updated relevance scores for reordering these shots. This MLLM-driven reranking significantly improves contextual matching and produces more accurate final rankings without requiring any model training. Regarding the VQA track, we fine-tune an audio-visual MLLM model on the provided TRECVID training dataset and we implement an inference-time scaling technique to enhance the multimodal understanding capabilities of the MLLM. For the open-ended Answer Generation (AG) task, we aggregate multiple model responses per question via a majority vote. The responses are generated with greedy sampling from different random frame subsets of the video and they are ranked based on the number of votes. For the Multiple-Choice (MC) task, instead of voting, we use mean pooling on the logits assigned by the fine-tuned model to each candidate response. Through the combination of fine-tuning and frame subset ensembling we achieve the highest score across 3 metrics in the VQA AG task and the second highest in the VQA MC task.

1 Introduction

The paper reports CERTH-ITI’s approaches to the TRECVID 2025 Ad-hoc Video Search and Video Question Answering tracks. It uses foundation models for retrieval and answering in these two distinct tasks.

1 Introduction

  • AVS retrieves relevant video shots for free-text queries, while VQA answers questions about a specific video in open-ended or multiple-choice form.These are the two TRECVID tracks addressed by the paper.Source: §1 · p. 1
  • For AVS, the team builds a training-free pipeline combining pretrained vision-language models, LLM query augmentation, and contextual reranking.The approach builds on the team’s previous submissions.Source: §1 · pp. 1–2
  • For VQA, the team fine-tunes an audio-visual MLLM and aggregates predictions generated from different random video-frame subsets.The paper introduces these approaches as its use of foundation models across both tracks.Source: §1 · pp. 1–2

2 Ad-hoc Video Search

The AVS system uses foundation models in a two-stage retrieval and reranking pipeline, with query augmentation and contextual shot scoring. In the reported evaluations, its 4,000-candidate reranking run leads the team’s runs across four years, and the no-reranking baseline scores lowest.

2.1 Approach

  • The retrieval stage augments the query, combines similarity scores from five VLM families, and normalizes them against background-query similarities before retrieving top-N shots.The fixed family weights are CLIP:BLIP:BLIP-2:SLIP:BEiT3 = 10:5:5:1:20; background queries come from AVS 2022, 2023, and 2024.Source: §2.1 · pp. 2–3
  • Qwen2.5-VL-7B reranks retrieved shots by assessing each shot’s semantic alignment with the original query.The MLLM revises the initial query–shot similarities to produce a final ranking.Source: §2.1 · pp. 2–3

2.2 Submission

  • The four submitted runs vary reranking depth: 4,000, 2,000, or 1,000 candidates, with the base run omitting reranking.Run 1 also uses 20 automatically generated query rephrasings.Source: §2.2 · pp. 3–4

2.3 Experimental Results

  • Run 1 achieves the highest MXinfAP scores across all four evaluation years, while the base run without reranking consistently scores lowest.The results cover the 2022–2025 tasks; reducing reranking depth to 2,000 candidates causes only a slight drop, while 1,000 candidates yields a more noticeable decrease.Source: Table 3 · p. 4, §2.3 · p. 4
  • The AVS 2025 figures rank submitted runs against other teams’ runs using combined 2024–2025 ground truth or 2024 ground truth alone.The evaluation uses MXinfAP on V3C2; Table 3 also reports internal evaluations for the 2022 and 2023 queries.Source: §2.2 · p. 3, §2.3 · p. 4, Figure 2 · p. 5

3 Video Question Answering

The VQA track includes answer generation and multiple-choice tasks on short videos with audio and visual streams. The team fine-tunes an audio-visual MLLM on the provided training data and aggregates predictions from different frame subsets during inference.

3 Video Question Answering

  • VQA comprises answer generation, which ranks up to 10 natural-language responses, and multiple choice, which ranks four candidate responses.The evaluation dataset contains 2,000 short videos with audio and visual streams.Source: §3 · p. 5
  • The team fine-tunes an audio-visual MLLM on the TRECVID training dataset and aggregates predictions from different frame subsets at inference.The paper presents the architecture and reports official evaluations alongside additional experiments.Source: §3 · p. 5, Figure 4 · p. 6

3.1 Approach

The approach fine-tunes an audio-visual MLLM, then scales inference by aggregating predictions generated from different video-frame subsets. Answer Generation uses majority voting, while Multiple Choice pools candidate-answer logits.

3.1.1 Base Model Selection and Fine-tuning

  • Qwen2.5-Omni 7B’s Thinker is fine-tuned on 500 audio-visual videos using correct answers, with 10 sampled frames per video during training.Fine-tuning uses supervised fine-tuning with LoRA for five epochs.Source: §3.1.1 · p. 6

3.1.2 Frame Subset Ensembling

  • For open-ended VQA, greedy generation from different frame subsets supplies the response diversity for majority voting, without external-model supervision.The authors avoid bin-wise sampling because the videos are short.Source: §3.1.2 · p. 6
  • Inference samples 10 random frame subsets from the entire video and combines each subset with audio embeddings from the full video.Each generation can use up to 16,384 visual tokens.Source: §3.1.2 · p. 6
  • For Multiple Choice, mean pooling aggregates candidate-letter log-probabilities across responses; ranking by mean response-token log probability significantly underperformed direct letter generation.The pooled probabilities determine the candidate ranking.Source: §3.1.2 · p. 7

3.2 Submission - Answer Generation

The Answer Generation submissions compare prediction aggregation and generation settings using the fine-tuned model. The main run aggregates predictions from different frame subsets.

  • Run 1, the main run, aggregates 10 greedy-sampled predictions per question from different frame subsets using the fine-tuned model.Source: §3.2 · p. 7
  • Run 2 generates one response per question with the fine-tuned model.Source: §3.2 · p. 7
  • Run 3 matches Run 1 but uses multinomial sampling instead of greedy sampling.Source: §3.2 · p. 7
  • Run 4 matches Run 1 but removes quote characters from final responses to check CSV parsing by the evaluation platform.Source: §3.2 · p. 7

3.3 Submission - Multiple Choice

The Multiple Choice submissions compare base and fine-tuned models, with aggregation by either mean-pooled candidate logits or confidence estimation. Each run aggregates 10 predictions.

  • Runs 1 and 2 aggregate 10 predictions by mean pooling the candidate logits.Source: §3.3 · p. 7
  • Runs 2 and 4 use the fine-tuned model; Runs 1 and 3 use the base model without fine-tuning.Source: §3.3 · p. 7
  • Runs 3 and 4 aggregate 10 predictions and choose the answer by confidence estimation.Source: §3.3 · p. 7

3.4 Experimental Results

The submitted VQA Answer Generation runs rank highest across all three evaluation metrics, while the Multiple Choice approach ranks second among participants. Additional evaluations show that the main AG run outperforms the single-generation baseline under top-1 selection, and that inference-time scaling improves both MC setups.

3.4.1 Results for the Answer Generation Task

  • The official AG comparison finds Run 1 significantly outperforms multinomial-sampling Run 3, but Run 1 and single-generation Run 2 are not directly comparable because they submit different response counts.Run 1 and Run 4 perform identically; Run 4 differs only by removing quote characters.Source: §3.4.1 · p. 7, Table 4 · p. 8
  • The AG runs achieve the highest scores among participants across METEOR, BERTScore, and STS; the ranking is sorted by STS.Run IDs in red identify the team’s submissions.Source: §3.4.1 · pp. 7–8, Figure 5 · p. 8

3.4.2 Discussion on Answer Generation Task and Additional Evaluation

  • Aggregating metrics across ranked AG responses penalizes runs that submit more answers, motivating a top-1 evaluation with one response per run.The additional evaluation keeps the highest-ranked answer and discards the rest.Source: §3.4.2 · p. 8, Table 5 · p. 9
  • Under top-1 selection, Run 1 significantly outperforms the Run 2 baseline across all metrics, while multinomial sampling in Run 3 continues to hurt performance.Top-1 selection retains the highest-ranked answer and discards additional responses; Table 5 reports this evaluation.Source: §3.4.2 · p. 9, Table 5 · p. 9

3.4.3 Results for the Multiple Choice Task

  • The MC approach achieves the second-highest score among participants, and the fine-tuned model outperforms the base model for both evaluated methods.Direct generation also significantly outperforms confidence estimation.Source: §3.4.3 · p. 9, Figure 6 · p. 9
  • Additional MC ablations show a clear performance gain from inference-time scaling in both evaluated setups.The experiments use organizer-provided ground-truth responses.Source: §3.4.3 · p. 9, Table 7 · p. 10
  • The MC evaluation dataset contained incorrectly escaped quotes in 0.7% of questions; affected answers were marked incorrect in the ablation study to match official scoring.The entries were manually escaped before official submission, but diagnostic files indicated that automatic evaluation still marked the responses incorrect due to parsing.Source: §3.4.3 · pp. 9–10

4 Conclusions

The paper concludes that its foundation-model AVS pipeline improves retrieval through MLLM reranking. For VQA, fine-tuning combined with inference-time scaling achieves the highest scores across three AG metrics and the second-highest score in MC.

4 Conclusions

  • MLLM-based reranking improves query-to-shot semantic alignment and produces more accurate, contextually relevant AVS rankings.The AVS framework uses foundation models for both retrieval and reranking.Source: §4 · p. 10
  • Fine-tuning and aggregating predictions from different frame subsets achieve the highest scores across three AG metrics and the second-highest score in MC.The paper also reports additional experiments on framework components and evaluations for more direct comparison.Source: §4 · p. 10

Related papers

Loading 2610.23214v1…