PaperlayerView exact revision on arXiv

Source-linked AI summary

Demo-ICL: In-Context Learning for Procedural Video Knowledge Acquisition

Yuhao Dong, Shulin Tian, Shuai Liu, Shuangrui Ding, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Jiaqi Wang, Ziwei Liu

arXiv:2602.08439v1cs.CV

TL;DR

Existing video benchmarks largely test static knowledge or facts visible in a target video, leaving limited evaluation of learning new procedures from demonstrations. This paper introduces Demo-driven Video In-Context Learning, Demo-ICL-Bench, and Demo-ICL with two-stage training; experiments report that current MLLMs struggle while Demo-ICL improves video understanding and in-context knowledge acquisition.

  • Problem

    Existing video benchmarks mainly assess static internal knowledge or visible target-video facts rather than learning and adapting to new procedures from demonstrations.

  • Method

    The paper introduces three demo-driven video ICL settings, constructs Demo-ICL-Bench, and trains Demo-ICL with video supervised finetuning and information-assisted DPO.

  • Results

    Demo-ICL outperforms existing MLLMs on Demo-driven ICL, VideoMMMU, and VideoMME, while Gemini-2.5-Pro reaches 46.6% and 32.0% accuracy on text and video demonstrations.

  • Takeaways & Limitations

    Demo-driven ICL and Demo-ICL-Bench provide an evaluation setting for acquiring and applying new procedural knowledge from in-context text or video demonstrations.

  • Takeaways & Limitations

    Demo-ICL uses a customized training strategy rather than a specialized architecture, and the paper does not explore learning from diverse modalities or resources jointly.

Abstract

from arXiv · show

Despite the growing video understanding capabilities of recent Multimodal Large Language Models (MLLMs), existing video benchmarks primarily assess understanding based on models' static, internal knowledge, rather than their ability to learn and adapt from dynamic, novel contexts from few examples. To bridge this gap, we present Demo-driven Video In-Context Learning, a novel task focused on learning from in-context demonstrations to answer questions about the target videos. Alongside this, we propose Demo-ICL-Bench, a challenging benchmark designed to evaluate demo-driven video in-context learning capabilities. Demo-ICL-Bench is constructed from 1200 instructional YouTube videos with associated questions, from which two types of demonstrations are derived: (i) summarizing video subtitles for text demonstration; and (ii) corresponding instructional videos as video demonstrations. To effectively tackle this new challenge, we develop Demo-ICL, an MLLM with a two-stage training strategy: video-supervised fine-tuning and information-assisted direct preference optimization, jointly enhancing the model's ability to learn from in-context examples. Extensive experiments with state-of-the-art MLLMs confirm the difficulty of Demo-ICL-Bench, demonstrate the effectiveness of Demo-ICL, and thereby unveil future research directions.

1 S-Lab, Nanyang Technological University 2 Shanghai AI Lab 3 CUHK-MMLab https://github.com/dongyh20/Demo-ICL

Figure 1 presents three settings for Demo-driven Video In-Context Learning, while an example asks what action follows heating oil for Mexican Rice.

  • An example question asks what to do after heating the oil when cooking Mexican Rice.
  • Demo-driven Video In-Context Learning includes text demonstrations, video demonstrations, and demonstration selection from a candidate pool.Text instructions, reference videos, and relevant-video selection provide the three task settings.
  • The figure's task overview distinguishes demonstrations provided as text instructions from those provided as videos.

1. Introduction

The paper introduces demo-driven video in-context learning to evaluate whether models can acquire task-specific procedural knowledge from demonstrations rather than relying only on static knowledge or visible target-video facts. It also presents Demo-ICL-Bench and a two-stage-trained Demo-ICL model, with experiments showing current MLLMs struggle while Demo-ICL performs better across evaluated video understanding tasks.

  • Existing video benchmarks mainly test internal pre-trained knowledge or visible facts, not learning a new process from demonstrations and applying it to a related target video.The paper motivates demonstration-based procedural learning with instructional examples such as cooking Mexican Rice.
  • Demo-driven ICL presents target videos and questions with text or video demonstrations, including a setting that selects relevant examples from a candidate pool.The three settings are text-demo learning, video-demo learning, and demonstration selection.
  • Demo-ICL-Bench contains instructional-video demonstrations, target videos, questions, and answers derived from collected YouTube videos and summarized subtitles.Video search and an LLM were used to identify similar instructional videos and construct candidate pools.
  • 46.6% and 32.0% accuracy were achieved by Gemini-2.5-Pro on text and video demonstrations, respectively, indicating the benchmark remains difficult.
  • Demo-ICL uses video supervised finetuning followed by information-assisted Direct Preference Optimization and outperforms existing MLLMs on Demo-driven ICL, VideoMMMU, and VideoMME.
  • The paper contributes three tasks, a benchmark for evaluating them, and a model intended to improve learning and adaptation from in-context demonstrations.

2. Related Work

Related work traces multimodal video understanding from perception toward knowledge acquisition and places Demo-driven Video In-Context Learning as a benchmark for acquiring new concepts from demonstrations.

  • Instructional video datasets support a shift from low-level perception toward extracting, structuring, and applying knowledge from complex procedures.HowTo100M provides 1.2 million narrated videos and 136 million clip-caption pairs for procedure recognition and cross-task transfer.
  • Demo-driven Video In-Context Learning evaluates whether models can acquire new concepts through provided video demonstrations in settings closer to real-world use.

3. Demo-ICL: Procedural Knowledge Learning from In-Context Demonstrations

Demo-driven Video In-Context Learning evaluates whether models can acquire procedural knowledge from text or video demonstrations and transfer it to new target videos. The benchmark construction and two-stage training pipeline target this capability through structured instructional data, demonstration selection, and information-assisted preference optimization.

  • 3.1. Task Formation: The task asks models to interpret a text or video demonstration, then predict subsequent actions in a related target video using its available context.It includes text-demo, video-demo, and demonstration-selection settings.
  • 3.1. Task Formation: The three settings progress from retrieving textual instructions and extracting knowledge from video demonstrations to selecting relevant demonstrations from distractor pools.Demonstration selection uses a candidate pool containing one relevant and several irrelevant videos.
  • 3.2. Dataset Construction: Text demonstrations are generated through coarse-to-fine transcript summarization, filtering, and multimodal refinement against the corresponding video clips.The refinement jointly considers step descriptions and visual clips to improve contextual alignment.
  • 3.2. Dataset Construction: Questions target intermediate steps and require prediction of the next action, while video demonstrations are validated for task comparability and meaningful visual evidence.Demonstration-selection instances add three irrelevant videos and dynamically vary candidate pools.
  • 3.3. Training Strategy: Demo-ICL uses video supervised fine-tuning followed by information-assisted DPO to strengthen fine-grained comprehension and learning from demonstrations.Assistive information includes timestamps for text-demo tasks and textual guidance paired with video demonstrations; iterative preference training produces progressively stronger models.

4. Experiments

Experiments show that current MLLMs struggle with video-demonstration ICL, while Demo-ICL improves demonstration-based learning through targeted data and information-assisted training. Analyses identify temporal alignment, abstraction, and knowledge transfer as central challenges.

  • Text-demo In-context Learning: Models improve with text demonstrations, but gains depend strongly on scale: small models typically gain less than 10 points, while Qwen2.5-VL-72B gains over 20 points.On Demo-ICL, SFT improves by over 10 points, and DPO achieves state-of-the-art results among similarly sized models.
  • Video-demo In-context Learning: Video demonstrations provide limited gains for many models, with InternVL-3, Qwen2-VL, and Ola even experiencing performance degradation.Demo-ICL consistently benefits from video demonstrations, unlike several existing MLLMs.
  • Demonstration Selection: Demonstration selection tests whether models can retrieve relevant reference videos and answer associated questions, exposing weak knowledge extraction, transfer, search, and selection capabilities.The task approximates real-world retrieval from large video pools.
  • Analysis Experiments: Fine-grained visual cues are critical for video-demo ICL, as denser frame sampling improves Demo-ICL performance.The analysis evaluates the role of densely sampled frames in video-demo in-context learning.
  • Analysis Experiments: Direct grounding is easier than transferring knowledge through demonstrations, while clip-based contexts expose failures in temporal evidence alignment and fine-grained abstraction.Using 128 frames from an identical reference video provides the full content as context; ASR transcripts and captions yield additional improvements.
  • Analysis Experiments: Instructional-video training, information-assisted DPO, and iterative training collectively improve Demo-ICL's ability to adapt to novel scenarios.Vanilla DPO yields noisy pairs and marginal improvements, whereas information-assisted DPO provides richer feedback signals.

5. Conclusion

The paper introduces Demo-driven Video In-Context Learning, Demo-ICL-Bench, and Demo-ICL to evaluate and improve learning from instructional demonstrations. Experiments find that existing MLLMs struggle with the task, whereas Demo-ICL improves video understanding and in-context knowledge acquisition.

  • Conclusion: Demo-driven Video In-Context Learning focuses on learning from in-context instructional demonstrations, and Demo-ICL-Bench contains 1,200 challenging questions for evaluation.The benchmark is designed to assess demo-driven video in-context learning capabilities.
  • Conclusion: Demo-ICL uses enhanced in-context learning abilities to achieve superior video understanding and in-context knowledge acquisition compared with existing MLLMs.The conclusion reports improved performance on the proposed task.

A.1. Experiment Details

The experiments evaluate Demo-ICL across general video understanding and demonstration-based settings. Video-MME tests performance across varied video durations, while figures illustrate text-demo and video-demo task inputs.

  • Task Visualization: The Video-demo task provides a demonstration video together with the target video input.Figure 4 visualizes two examples of this task design.
  • Video-MME Evaluation: Demo-ICL is evaluated on Video-MME across short, medium, and long video tracks.Video-MME includes 900 videos spanning 11 seconds to 1 hour, with six visual domains and 30 subfields.
  • Video-MME Results: Demo-ICL achieves strong results across all Video-MME duration tracks.It surpasses similarly sized 7B open-source video MLLMs, matches larger 34B models comparably, and competes closely with some commercial MLLMs.
  • Task Visualization: The Text-demo task provides textual instructions alongside a target video as model inputs.Figure 3 visualizes two examples of this task design.

C.2. Results on Video-MMLU Benchmark

Video-MMLU evaluates whether demo-driven video in-context learning supports lecture comprehension and knowledge acquisition. Demo-ICL maintains competitive caption performance while substantially improving reasoning performance on the Quiz track.

  • Interpretation: The evaluation examines the effectiveness of both demo-driven video in-context learning and information-assisted DPO for general video knowledge acquisition.The experiment is presented as a validation of the paradigm and training strategy beyond Demo-ICL-Bench.
  • Benchmark Setup: Video-MMLU assesses lecture comprehension and video-based knowledge acquisition using caption and reasoning questions.Its 1,065 lecture-focused videos use open-ended answers evaluated by an LLM-as-a-Judge framework.
  • Results: Demo-driven video in-context learning substantially improves Demo-ICL’s Quiz-track performance over previous models.The model uses videos as demonstrations for subsequent reasoning questions and achieves markedly better Quiz results with a Qwen2.5-7B backbone.
  • Results: Demo-ICL maintains competitive performance on the Video-MMLU caption task.The reported gains are especially pronounced on the Quiz track rather than the caption task.

C.4. Why Demonstration Selection Difficult?

Demonstration Selection remains difficult because current models struggle to identify and use the correct examples among multiple candidates. Even correct selection does not ensure successful transfer to the target question.

  • Selection Difficulty: Existing models perform poorly when selecting correct demonstrations from multiple candidates.The analysis attributes this to difficulty effectively utilizing the appropriate in-context demonstrations in complex scenarios.
  • Transfer After Selection: 22.2% final accuracy follows correct demonstration selection, compared with 25.4% in the Video-demo ICL setting.These figures are reported for Qwen2.5-VL on the same questions.
  • Transfer After Selection: The results indicate that recognizing correct videos does not guarantee effective knowledge transfer to new tasks.The comparison shows that selection success alone is insufficient for high final-answer accuracy.
  • Dependence on Demonstrations: Gemini-2.5-Pro answers only 5% of original questions without demonstrations or answer options.This validation uses open-ended evaluation with Qwen2.5-72B as the judge model.
  • Dependence on Demonstrations: Demo-ICL-Bench requires in-context demonstrations because its questions contain detailed, task-specific action sequences.The reported validation concludes that internal knowledge alone is insufficient.

C.6. More Evaluation Results

Additional evaluations compare retrieval- and agent-based methods on Demo-ICL-Bench and situate the benchmark within prior video-understanding work. These results show that perceptual alignment does not by itself solve demonstration-to-target knowledge transfer.

  • RAG-Based Methods: VideoRAG performs well on Text-demo ICL but underperforms on Video-demo ICL and Demonstration Selection.Its indexing and multimodal retrieval align videos with provided text demonstrations, but transfer to target-video tasks remains difficult.
  • Agent-Based Methods: VideoAgent improves Text-demo ICL performance through stronger video-text grounding but underperforms on the other tracks.The results suggest that agentic strategies improve perceptual alignment without resolving knowledge transfer to new tasks.
  • Open Challenge: Knowledge transfer from demonstrations to new tasks remains an open challenge for current video MLLMs.This challenge persists beyond video knowledge acquisition and perceptual accuracy.
  • Related Work Context: Prior work includes instructional-video grounding, multimodal in-context learning, temporal reasoning, and video retrieval-augmented generation.These lines of research provide context for Demo-ICL-Bench’s focus on demonstration-based knowledge acquisition.

E. Limitations and Future Directions

The authors identify two limitations: Demo-ICL relies on training rather than a specialized architecture, and it does not address learning from diverse contextual resources or modalities.

  • Demo-ICL uses a customized training strategy instead of a specialized architecture for demo-driven video in-context learning.This design aims to add the capability to current MLLMs without architectural modifications.
  • The work does not explore learning from diverse contexts, including combinations of modalities or resources such as text instructions and instructional videos.Combining varied contextual information to improve learning on new tasks remains an open challenge.
Loading 2602.08439v1…