PaperlayerView exact revision on arXiv

Source-linked AI summary

VIOLA: Towards Video In-Context Learning with Minimal Annotations

Ryo Fujii, Hideo Saito, Ryo Hachiuma

arXiv:2601.15549v1cs.CVcs.AI

TL;DR

Adapting MLLMs to unseen video domains is challenging when labeled examples are scarce, while standard ICL assumes a large annotated demonstration pool. VIOLA combines density-uncertainty-weighted sample selection with confidence-aware use of labeled and pseudo-labeled examples. Across diverse benchmarks, it consistently outperforms state-of-the-art baselines.

  • Problem

    Novel video domains call for model adaptation, yet standard ICL assumes a fully annotated, sufficiently large pool covering the test distribution.

    Source: §1 · p. 1, §3.1 · p. 7

  • Method

    VIOLA combines density-uncertainty-weighted sample selection with a hybrid pool of expert-labeled and pseudo-labeled videos, using confidence-aware retrieval and prompting.

    Source: §1 · p. 4, §5 · p. 15

  • Results

    VIOLA consistently outperforms state-of-the-art baselines across diverse benchmarks; on ENIGMA with Qwen2-VL-7B, it achieves a +53.6% increase over zero-shot inference.

    Source: §4.2 · p. 11, §5 · p. 15

  • Takeaways & Limitations

    The findings support combining strategic data selection with confidence-aware mechanisms for label-efficient video in-context learning.

    Source: §5 · p. 15

    The paper frames its standard ICL setting as assuming a fully annotated, sufficiently large pool that covers the test distribution.

    Source: §3.1 · p. 7

Abstract

from arXiv · show

Generalizing Multimodal Large Language Models (MLLMs) to novel video domains is essential for real-world deployment but remains challenging due to the scarcity of labeled data. While In-Context Learning (ICL) offers a training-free adaptation path, standard methods rely on large annotated pools, which are often impractical in specialized environments like industrial or surgical settings since they require the experts' annotations. To bridge this gap, we introduce VIOLA (Video In-cOntext Learning with minimal Annotation), a label-efficient framework that synergizes minimal expert supervision with abundant unlabeled data. First, to maximize the efficiency of a strict annotation budget, we propose density-uncertainty-weighted sampling. Unlike standard diversity or uncertainty strategies that risk selecting visual outliers, our method leverages density estimation to identify samples that are simultaneously diverse, representative, and informative. Second, to utilize the remaining unlabeled data without noise propagation, we construct a hybrid pool and introduce confidence-aware retrieval and confidence-aware prompting. These mechanisms explicitly model label reliability, retrieving demonstrations based on a composite score of similarity and confidence while enabling the MLLM to adaptively distinguish between verified ground truths and noisy pseudo-labels. Extensive experiments across nine diverse benchmarks using four MLLMs demonstrate that our framework significantly outperforms various baselines in low-resource settings, achieving robust adaptation with minimal annotation costs.

1 Introduction

VIOLA addresses the scarcity of expert-labeled video demonstrations by combining minimal human annotations with unlabeled videos in a confidence-aware pool. It selects representative, informative examples and uses label reliability during retrieval and prompting.

1 Introduction

  • Novel video domains demand adaptation beyond pretrained knowledge; fine-tuning is costly, while ICL adapts at inference but depends on high-quality labeled demonstrations.Expert annotation is especially costly in specialized industrial and surgical settings.Source: §1 · pp. 1–2
  • VIOLA achieves robust adaptation with significantly reduced annotation costs compared with Naive ICL, averaged across seven classification datasets.The figure presents the problem setup and performance comparison.Source: Figure 1 · p. 2
  • VIOLA bridges minimal expert supervision and abundant unlabeled videos by combining annotated examples with pseudo-labeled data in a confidence-aware hybrid pool.The paper identifies the combination of selective annotation and pseudo-annotation as underexplored.Source: §1 · pp. 3–4
  • Density-uncertainty-weighted selection uses GMM clusters and density-weighted uncertainty to choose diverse, representative, informative samples while avoiding semantic outliers.This addresses video selection methods that can favor diverse but unrepresentative outliers.Source: §1 · pp. 3–4
  • Confidence-aware retrieval combines visual similarity with label confidence so limited context slots favor relevant, reliable demonstrations over merely similar examples.Reliability varies between ground truths and pseudo-labels, and also among pseudo-labeled examples.Source: §1 · pp. 3–4
  • Confidence-aware prompting encodes demonstration source and confidence, helping the MLLM calibrate reliance on ground truths versus noisy pseudo-labels in few-shot video ICL.Video token costs constrain ICL to few shots, making it sensitive to erroneous demonstrations.Source: §1 · pp. 3–4

2 Related Work

VIOLA addresses video ICL’s dependence on large annotated retrieval pools by combining selective expert annotation with pseudo-labeling in a unified semi-supervised active-learning framework. It targets robust adaptation when expert labels are scarce and demonstrations may be noisy.

In-Context Learning

  • ICL adapts models through input-output demonstrations without parameter updates, while VideoICL extends example retrieval and iterative inference to out-of-distribution video understanding.Standard video ICL nevertheless depends on large annotated pools, which are impractical in specialized domains where expert labels are costly.Source: §2 · pp. 4–5

Selective Annotation

  • Diversity-based selection can choose noisy outliers, while uncertainty-based selection can confuse useful epistemic uncertainty with aleatoric ambiguity; VIOLA combines density and uncertainty to select representative, informative samples.This density-uncertainty-weighted sampling addresses video-domain noise that can undermine either selection strategy.Source: §2 · p. 5

Pseudo-Labeling

  • Video ICL’s few-shot regime makes it sensitive to erroneous pseudo-labels, and similarity-based retrieval does not distinguish them from ground truths; VIOLA models label reliability in retrieval and prompting.Unlike text many-shot learning, video ICL faces prohibitive token costs, so one incorrect demonstration can cause negative transfer.Source: §2 · p. 5

Semi-Supervised Active Learning

  • Unlike prior work treating selective annotation and pseudo-labeling separately, VIOLA adapts semi-supervised active learning to video ICL, combining minimal expert labels with abundant unlabeled data via pseudo-annotation.The framework aims to improve performance and robustly adapt to low-resource target domains.Source: §2 · pp. 5–6

3 Methodology

VIOLA builds a hybrid demonstration pool from a small expert-labeled set and confidence-filtered pseudo-labels, then uses label reliability in retrieval and prompting. Its annotation strategy combines semantic density with prediction uncertainty to select demonstrations under a strict budget.

3.1 Preliminaries

  • Standard ICL retrieves K demonstrations by visual similarity and conditions its prediction on them, assuming a large, fully annotated pool covers the test-query distribution.Similarity is computed as cosine similarity between visual embeddings.Source: §3.1 · pp. 6–7

3.2 Overview

  • Under a strict annotation budget B ≪ N, VIOLA combines expert-labeled samples with pseudo-labeled videos from the remaining unlabeled data to form a hybrid pool for inference.Experts label B selected samples; the model generates pseudo-labels for the remainder using the expert-labeled set as context.Source: §3.2 · pp. 7–8

3.3 Selective Annotation via Density-Uncertainty-weighted Selection

  • With budget B, VIOLA clusters video embeddings using a GMM with K=B and selects one density- and uncertainty-weighted sample per cluster to balance representativeness and informativeness.Zero-shot uncertainty is estimated from the minimum token probability in the generated sequence; the weighting parameter balances uncertainty against density.Source: §3.3 · p. 8

3.4 In-Context Pseudo-Annotation

  • For each remaining unlabeled video, VIOLA uses visually similar expert-labeled examples as in-context demonstrations, then keeps its pseudo-label only if confidence exceeds the 95th percentile.The in-context examples align predictions with domain semantics; retained pseudo-labels and confidence scores join the expert-labeled set in the hybrid pool.Source: §3.4 · p. 9

3.5 Confidence-Aware Inference

  • VIOLA retrieves demonstrations using a composite of visual similarity and label confidence, then marks expert labels as ground truth and pseudo-labels with their confidence to guide the MLLM.Expert-labeled samples receive confidence 1.0, while source-specific prompt formatting acts as a soft gate between verified patterns and pseudo-labels.Source: §3.5 · pp. 9–10

4 Experiments

Across nine video benchmarks and four MLLMs, VIOLA is evaluated against zero-shot, annotation-selection, and pseudo-annotation baselines, including under varying annotation budgets. The experiments report gains over these baselines and ablations supporting its selection and confidence-aware inference components.

4.1 Experimental Settings

  • Experiments cover seven classification and two captioning datasets across medical, industrial, ego-centric, and surveillance domains, using accuracy and ROUGE-L respectively.The evaluated MLLMs are Qwen2-VL-7B, VideoLLaMA3-7B, Qwen3-VL-8B, and LLaVA-Video7B; the setup uses m = 8 demonstrations, 16 frames each, and filters pseudo-demonstrations at the 95th-percentile confidence threshold. Videos are sampled at 1 FPS and sequences over 32 seconds are downsampled to 32 frames; pseudo-labeled demonstrations precede labeled ones, with each group sorted by similarity.Source: §4.1 · pp. 10–11

4.2 Main Results

  • With 20 labeled samples, VIOLA improves Qwen2-VL-7B results over zero-shot by +53.6% on ENIGMA and +38.2% on EgoPet, and exceeds VoteK by +34.8% on EgoPet.VideoICL scores 11.2% versus Random’s 20.9% on Drive&Act with Qwen2-VL-7B, while pseudo-annotation drops to 8.0% on Xsports.Source: §4.2 · pp. 11–12, Table 1 · p. 12
  • VIOLA achieves ROUGE-L 0.365 on Bora, above Random’s 0.338 and VoteK’s 0.359, and gains persist across other models, with ENIGMA the sole LLaVA-Video-7B outlier.On VideoLLaMA3-7B, gains reach +40.0% on EgoPet; Qwen3-VL-8B also improves, and CapERA results consistently outperform baselines.Source: §4.2 · pp. 12–13

4.3 Performance under Varying Annotation Budgets

  • As the labeled pool grows from 20 to 100 samples, VIOLA generally outperforms Random and VoteK, with continuing gains and no premature plateauing.VideoICL underperforms Random on most datasets except EgoSurgery and Xsports; VIOLA also maintains an advantage in captioning tasks.Source: Figure 3 · p. 13, §4.3 · p. 13

4.4 Ablation Studies

  • Density-uncertainty-weighted selection with λ = 0.5 outperforms density-only and uncertainty-only selection, which respectively miss informative signals and select noisy outliers.Uncertainty-only selection degrades EgoSurgery performance by -21.2%; ablations use an annotation budget of |DL| = 20.Source: §4.4 · p. 13, Table 2 · p. 14
  • Combining confidence-aware retrieval and prompting yields robust improvements across datasets, including +10.3% on Drive&Act, whereas either component alone gives limited or unstable gains.The combined mechanisms filter the hybrid pool for reliability and guide the MLLM to distinguish ground truths from pseudo-labels.Source: §4.4 · p. 14, Table 3 · p. 14
  • In-context pseudo-label generation conditioned on expert examples improves downstream performance over zero-shot pseudo-labeling, including +3.7% on EgoSurgery.Retrieved expert data provides domain guidance for constructing a more reliable hybrid pool.Source: §4.4 · p. 14, Table 5 · p. 14
  • VIOLA also outperforms baselines with smaller Qwen2-VL-2B and VideoLLaMA3-2B models, including gains over random selection.This ablation evaluates whether the framework remains effective at reduced model capacity.Source: §4.4 · p. 14, Table 4 · p. 14

5 Conclusion

VIOLA combines minimal expert supervision with abundant unlabeled data for label-efficient video in-context learning. It pairs density-uncertainty-weighted sampling with confidence-aware use of pseudo-labels, and its pre-trained-embedding reliance may limit performance under severe domain shifts.

5 Conclusion

  • VIOLA combines density-uncertainty-weighted sampling to filter semantic outliers and select informative annotations with confidence-aware retrieval and prompting to distinguish verified labels from noisy pseudo-labels.In-context pseudo-annotation precedes confidence-aware retrieval and prompting in the hybrid pool.Source: §5 · pp. 14–15
  • Across diverse benchmarks, including medical and industrial domains, VIOLA consistently outperforms state-of-the-art baselines using four MLLMs.The results support combining strategic data selection with confidence-aware inference for low-resource adaptation.Source: §5 · p. 15
  • VIOLA relies on pre-trained visual embeddings, which distribution shifts in highly specialized domains may distort and thereby compromise clustering and retrieval accuracy.Future work will explore domain-adapted encoders to improve robustness in such scenarios.Source: §5 · p. 15

Supplementary materials · A Datasets

The supplement describes nine video benchmarks across action, interaction, phase, behavior, and captioning tasks, specifying their experimental formulations and dataset splits.

A Datasets

  • Drive&Act evaluates action recognition on Kinect-IR segments, averaging three official splits of approximately 2,000 training and 600 test examples each.Action segmentation labels are used to extract the video segments.Source: A Datasets · p. 20
  • EgoPet uses a curated single-interaction subset to classify the interacted-with object into 18 categories, with 387 training and 448 test samples.The source collection contains approximately 84 hours of pet egocentric video.Source: A Datasets · p. 20
  • EgoSurgery tests surgical-phase recognition across nine phases using 10 consecutive frames to classify the middle frame, with 1,886 training and 501 test samples.The benchmark covers 15 hours of open-surgery footage captured by head-mounted cameras.Source: A Datasets · p. 20
  • ENIGMA evaluates right-hand object-interaction recognition across 18 categories (644 training, 323 test), while UCF-Crime classifies 13 crime types plus normal footage.UCF-Crime uses four official splits of 532 training and 168 test samples each and averages performance across them.Source: A Datasets · p. 20
  • Xsports evaluates classification of 18 extreme-sport actions from dynamic egocentric footage, using 1,191 training and 351 test samples.Examples include climbing, flipping, jumping, and vaulting.Source: A Datasets · p. 20
  • MammAlps evaluates animal behavior across 11 activity categories (1,249 training, 466 test), while Bora tests domain-specific medical video captioning (4,407 training, 490 test).MammAlps provides 8.5 hours of densely annotated camera-trap footage with hierarchical activity and action labels; Bora spans endoscopy, ultrasound, MRI, and cell tracking.Source: A Datasets · p. 21
  • CapERA evaluates aerial-video captioning from UAV footage describing events, objects, and settings, using 1,473 training and 1,391 test samples.The dataset comprises 2,864 videos across scenarios including traffic, concerts, and harvesting.Source: A Datasets · p. 21

B Prompt Details

The section documents the exact dataset prompts used for reproducibility, distinguishing classification prompts from concise-event captioning prompts.

B Prompt Details

  • For classification, prompts ask a domain-specific question, list all candidate categories, and append “Just answer the name of the category” (or behavior) to constrain valid outputs.For captioning, prompts instruct the model to describe events concisely; the exact prompts for each dataset are provided in Table 6.Source: B Prompt Details · p. 21

C Qualitative Results

On UCF-Crimes and EgoSurgery, the qualitative examples show VIOLA retrieving useful pseudo-labeled videos where the baseline’s retrieved context leads to incorrect predictions.

C Qualitative Results

  • For a “Normal” UCF-Crimes query, VIOLA retrieves high-confidence “Normal” pseudo-labels and avoids the baseline’s irrelevant crime clips and hallucinated “Assault” prediction.Despite minor noise in the retrieved context, the semantic signal guides the model to the correct classification, reducing false positives.Source: C Qualitative Results · p. 21
  • For a “Dissection” EgoSurgery query, VIOLA retrieves four correctly pseudo-labeled videos, enabling the model to distinguish fine-grained tool-tissue interactions and correct the baseline’s “Hemostasis” prediction.The baseline retrieves visually similar but semantically incorrect examples from the limited labeled pool; VIOLA’s context demonstrates the effectiveness of expanding the knowledge base.Source: C Qualitative Results · pp. 21–22
  • VIOLA corrects the baseline’s errors in both qualitative examples by retrieving context that supports the query’s true class.On UCF-Crimes, the query is “Normal”; on EgoSurgery, it is “Dissection.”Source: C Qualitative Results · pp. 21–22, Figure 4 · p. 23

Related papers

Loading 2601.15549v1…