PaperlayerView exact revision on arXiv

Source-linked AI summary

VIOLA: Towards Video In-Context Learning with Minimal Annotations

Ryo Fujii, Hideo Saito, Ryo Hachiuma

arXiv:2601.15549v1cs.CVcs.AI

TL;DR

Video ICL must adapt MLLMs to novel domains despite scarce expert-labeled demonstrations and impractical large annotated pools. VIOLA combines density-uncertainty-weighted selection with pseudo-annotation, confidence-aware retrieval, and prompting, and experiments across nine benchmarks and four MLLMs show consistent gains over baselines with minimal annotation costs.

  • Problem

    Novel video domains require adaptation beyond pre-trained knowledge, while specialized-domain expert annotation makes the large labeled pools used by standard ICL impractical.

  • Method

    VIOLA selects a small labeled subset using density-uncertainty weighting, combines it with pseudo-labeled data, and uses confidence-aware retrieval and prompting.

  • Results

    Experiments across nine diverse benchmarks with four MLLMs show VIOLA consistently outperforms state-of-the-art baselines, including specialized-domain gains of +53.6% on industrial ENIGMA and +38.2% on EgoPet with Qwen2-VL-7B.

  • Takeaways & Limitations

    VIOLA supports robust video ICL adaptation by synergizing strategic data selection with confidence-aware use of labeled and unlabeled demonstrations.

  • Takeaways & Limitations

    The framework assumes a strict annotation budget B much smaller than the unlabeled pool size N and contrasts with settings where the demonstration pool is fully annotated and sufficiently large.

Abstract

from arXiv · show

Generalizing Multimodal Large Language Models (MLLMs) to novel video domains is essential for real-world deployment but remains challenging due to the scarcity of labeled data. While In-Context Learning (ICL) offers a training-free adaptation path, standard methods rely on large annotated pools, which are often impractical in specialized environments like industrial or surgical settings since they require the experts' annotations. To bridge this gap, we introduce VIOLA (Video In-cOntext Learning with minimal Annotation), a label-efficient framework that synergizes minimal expert supervision with abundant unlabeled data. First, to maximize the efficiency of a strict annotation budget, we propose density-uncertainty-weighted sampling. Unlike standard diversity or uncertainty strategies that risk selecting visual outliers, our method leverages density estimation to identify samples that are simultaneously diverse, representative, and informative. Second, to utilize the remaining unlabeled data without noise propagation, we construct a hybrid pool and introduce confidence-aware retrieval and confidence-aware prompting. These mechanisms explicitly model label reliability, retrieving demonstrations based on a composite score of similarity and confidence while enabling the MLLM to adaptively distinguish between verified ground truths and noisy pseudo-labels. Extensive experiments across nine diverse benchmarks using four MLLMs demonstrate that our framework significantly outperforms various baselines in low-resource settings, achieving robust adaptation with minimal annotation costs.

1 Introduction

Video ICL needs adaptation to unseen domains, but expert-labeled demonstrations are costly and standard large annotated pools are impractical. VIOLA combines minimal selective annotation with pseudo-annotation in a confidence-aware hybrid pool for low-cost adaptation.

  • MLLMs must adapt beyond pre-trained knowledge as novel video domains, scenarios, and tasks emerge in real-world applications.
  • ICL adapts models without parameter updates by conditioning inference on example inputs and demonstrations.
  • Expert-labeled video demonstrations are difficult and expensive to obtain in specialized domains, making large annotated retrieval pools impractical.
  • VIOLA integrates selective annotation and pseudo-annotation into a hybrid pool, bridging minimal human supervision with abundant unlabeled videos.
  • Density-uncertainty-weighted selection targets samples that are diverse, representative, and informative while reducing semantic-outlier selection.
  • Confidence-aware retrieval and prompting model label reliability so demonstrations can distinguish verified ground truths from potentially noisy pseudo-labels.

2 Related Work

Related work establishes video ICL as an efficient adaptation paradigm but identifies annotation scarcity as a central limitation. VIOLA adapts semi-supervised active learning by combining selective annotation and pseudo-labeling for low-resource video ICL.

  • Video ICL extends in-context adaptation to video understanding through example selection and confidence-based inference.
  • Standard video ICL frameworks rely on large annotated retrieval pools, which are impractical in specialized domains because expert annotation is costly.
  • Selective annotation optimizes labeling budgets by choosing valuable unlabeled examples, commonly emphasizing diversity or uncertainty.
  • Pseudo-annotation scales demonstrations by converting model predictions on unlabeled data into examples, but video ICL remains sensitive to label noise.
  • VIOLA unifies selective annotation and pseudo-labeling under semi-supervised active learning to synergize minimal expert supervision with abundant unlabeled data.

3 Methodology

VIOLA builds a hybrid video-ICL demonstration pool from a small expert-labeled subset and confidence-filtered pseudo-labeled data. It selects annotations using density and uncertainty, then retrieves and formats demonstrations according to both similarity and label reliability.

  • Preliminaries: Standard video ICL retrieves semantically relevant demonstrations from a fully annotated pool, an assumption that fails when expert annotations are scarce.The context contains K demonstrations selected by visual similarity, but the standard pool is assumed to be large and fully annotated.
  • Hybrid Demonstration Pool: VIOLA allocates a strict budget B by selecting an optimal subset of unlabeled videos for expert annotation.The selected samples form the labeled set D_L after oracle-provided annotations.
  • Selective Annotation: Density-uncertainty-weighted sampling combines semantic density with model uncertainty to select diverse, representative, and informative examples while filtering outliers.The method fits a GMM, estimates zero-shot uncertainty from minimum generated-token probability, and scores samples using a density–uncertainty trade-off controlled by λ.
  • Selective Annotation: One sample maximizing the selection score is chosen from each cluster, covering all K semantic modes while balancing prototypical and hard examples.Higher λ prioritizes uncertain samples; lower λ favors high-density samples, and the resulting labeled set is diverse, representative, and informative.
  • Pseudo Annotation: For remaining unlabeled videos, VIOLA generates in-context pseudo-labels with confidence scores and retains only predictions above the 95th confidence percentile.The expert-labeled set supplies domain-specific context, distinguishing this stage from noisier zero-shot estimation.
  • Confidence-Aware Inference: The hybrid pool combines expert labels and retained pseudo-labels, while confidence-aware retrieval ranks demonstrations using visual similarity and confidence.Ground-truth samples receive confidence 1.0, and the composite score balances semantic relevance with label trustworthiness through τ.
  • Confidence-Aware Inference: Confidence-aware prompting formats retrieved answers according to their demonstration-source reliability.The formatting function Φ explicitly communicates confidence for retrieved demonstrations.

4 Experiments

Across nine video-understanding benchmarks and multiple MLLMs, VIOLA is evaluated under a 20-sample annotation budget, varying budgets, and targeted ablations. It generally improves over zero-shot, selective-annotation, and pseudo-annotation baselines, while its components support stability and cross-model effectiveness.

  • Experimental Setup: Experiments cover classification and captioning across nine diverse benchmarks spanning medical, industrial, ego-centric, and surveillance domains.Classification uses accuracy across seven benchmarks, while captioning is also evaluated.
  • Main Results: With only 20 labeled samples, VIOLA generally outperforms zero-shot inference across datasets and architectures, including +53.6% on ENIGMA and +38.2% on EgoPet with Qwen2-VL-7B.The comparison uses a fixed labeled pool of 20 samples.
  • Main Results: VIOLA surpasses selective-annotation and pseudo-annotation baselines, including a +34.8% margin over VoteK on EgoPet and a 0.365 ROUGE-L score on Bora versus 0.338 for Random and 0.359 for VoteK.On CapERA, the framework also consistently outperforms baselines.
  • Main Results: Performance gains extend across VideoLLaMA3-7B, Qwen3-VL-8B, and LLaVA-Video-7B, with gains up to +40.0% on EgoPet for VideoLLaMA3-7B.ENIGMA is the sole reported outlier for LLaVA-Video-7B.
  • Budget Scaling: Across annotation budgets from 20 to 100 samples, VIOLA generally outperforms Random and VoteK and remains more stable than VideoICL in limited-budget settings.VideoICL underperforms Random on most datasets, except EgoSurgery and Xsports.
  • Ablation Studies: Ablations show that combining density and uncertainty at λ = 0.5 outperforms either extreme, while joint confidence-aware retrieval and prompting yields improvements such as +10.3% on Drive&Act.In-context pseudo-annotation also improves downstream performance, including +3.7% on EgoSurgery.

5 Conclusion

VIOLA combines density-uncertainty-weighted sampling with confidence-aware retrieval and prompting to support label-efficient video ICL. Across diverse benchmarks and four MLLMs, it consistently outperforms state-of-the-art baselines in low-resource adaptation, while relying on pre-trained visual embeddings creates a domain-shift limitation.

  • VIOLA filters semantic outliers while selecting informative annotation samples through density-uncertainty-weighted sampling.The method addresses the trade-off between diversity and representativeness in video selection.
  • Confidence-aware retrieval and prompting help distinguish verified ground truths from potentially noisy pseudo-labels in hybrid pools.These mechanisms follow in-context pseudo-annotation to model label reliability during inference.
  • VIOLA consistently outperforms state-of-the-art baselines across diverse benchmarks, including specialized medical and industrial domains, using four MLLMs.
  • Limitations and Future Work: Pre-trained visual embeddings may distort semantic spaces under significant distribution shifts in highly specialized domains, compromising clustering and retrieval accuracy.The authors identify domain-adapted encoders as a direction for improving robustness.

A Datasets

The evaluation spans video-language classification and captioning across datasets covering medical, industrial, ego-centric, surveillance, wildlife, and aerial-video scenarios. The listed datasets define varied recognition and description settings for assessing VIOLA.

  • EgoSurgery benchmarks surgical phase recognition in open surgery using head-mounted-camera footage.It addresses data scarcity in open surgery relative to minimally invasive procedures.
  • Xsports evaluates extreme-sport action classification across 18 categories in rapidly moving egocentric video.Its split contains 1,191 training samples and 351 test samples.
  • The UAV dataset evaluates textual description generation for overhead footage of traffic, concerts, and harvesting.Its experiment uses 1,473 training samples and 1,391 test samples.

B Prompt Details

The prompts constrain classification outputs with domain-specific questions and complete candidate lists, while captioning prompts request textual descriptions.

  • Classification prompts provide domain-specific questions, list all candidate categories, and require the model to answer only with a category or behavior name.
  • Captioning prompts instruct the model to generate captions for the input video.

C Qualitative Results

Qualitative comparisons show VIOLA retrieving more semantically appropriate context than the baseline on UCF-Crimes and EgoSurgery. These retrieved examples support correct classifications despite minor context noise and fine-grained visual ambiguity.

  • UCF-Crimes: On UCF-Crimes, VIOLA retrieves high-confidence “Normal” pseudo-labels, avoiding the baseline’s irrelevant crime clips and hallucinated “Assault” prediction.
  • EgoSurgery: On EgoSurgery, VIOLA retrieves four correctly pseudo-labeled “Dissection” videos instead of the baseline’s visually similar but incorrect “Hemostasis” examples.
  • EgoSurgery: The accurate retrieved context enables the MLLM to distinguish fine-grained tool-tissue interactions and correct the surgical-phase prediction.
Loading 2601.15549v1…