CoS: Chain-of-Shot Prompting for Long Video Understanding

• Shot selection must minimise redundant distractions while preserving sparsely distributed task-relevant information.Source: Figure 1 · p. 1, §1 · pp. 1–2
• Long videos remain difficult for MLLMs because massive visual-token inputs can exceed model context and computational capacity.Source: §1 · p. 1
• Different shot selections from the same video can produce sharply different semantic interpretations, exposing visual selection as a reasoning issue.Source: Figure 2 · p. 2, §1 · p. 2
• CoS introduces a training-free test-time optimisation strategy that treats video inputs as visual prompts for long-video understanding.Source: §1 · p. 2
• Binary Video Summary uses mosaicing-based binary coding for pseudo temporal grounding, while Video Co-Reasoning constructs task-relevant positive and irrelevant negative videos to filter noise.Source: §1 · p. 2
• Experiments across five datasets and 17 models validate CoS’s effectiveness for dynamically optimising task-specific video inputs.Source: §1 · p. 2

MLLMs for visual understanding

• MLLM research improves cross-modal alignment and visual-token handling through projectors, pooling, token dropping, and related connector enhancements.Source: §2 · pp. 2–3

Prompt Engineering

• CoS’s framework uses mosaicing binary coding for pseudo temporal grounding and then combines task-relevant positive with irrelevant negative shots for co-reasoning.Source: Figure 3 · p. 3

MLLMs for Long Video Understanding

• Long-video methods extend context, compress inputs, or preserve key information through fine-grained data, parallel processing, and compact representations.Source: §2 · p. 3

Prompt Engineering

• Prompt-engineering methods such as VideoCoT and VoT mainly optimise textual reasoning inputs, whereas CoS models temporal and spatial structure in visual inputs.Source: §2 · p. 3

Overview

• CoS dynamically optimises visual inputs at test time for each video-understanding task by constructing task-relevant positive and irrelevant negative shots.The method is training-free and uses an MLLM’s spatial reasoning and summarisation abilities for per-instance shot selection.Source: §3 · pp. 3–4

Motivation

• Long-video shot selection must balance capturing sparse task-relevant information against reducing redundant and distracting shots.Low sampling rates risk omissions, whereas denser sampling increases irrelevant content and computational burden.Source: §3.1 · p. 4, §3.2 · p. 4

Binary Video Summary

• Binary video summarisation asks whether each shot or aggregated shot group contains task-relevant content, producing a binary coding for pseudo temporal grounding.Grouping k consecutive shots into mosaicing images reduces evaluation cost while preserving temporal information; the implementation uses k=4 and LLaVA.Source: §3.2 · p. 4

Constructing Balanced Sub-Shots

• Positive shots are selected from shots labelled relevant, while negative shots use irrelevant shots and replacement rules to maintain balanced sequence lengths.If suitable neighbouring shots are unavailable, the positive sequence falls back to the original video, while negative construction can use a black shot.Source: §3.3.1 · pp. 4–5

Co-Reasoning with Sub-Shots

• The model jointly reasons over the original sequence and constructed positive and negative sub-shots, with a weight α controlling their influence.A smaller positive-shot ratio indicates stronger selection and increases α; when α=0, reasoning uses only the original sequence X.Source: §3.3.2 · pp. 5–6

Results

• CoS significantly improves three baselines across short, medium, and long videos, with larger gains on LLaVA-Video and LongVA than Video-XL.The smaller Video-XL gains are attributed to its built-in context attention mechanism overlapping with CoS’s design.Source: §4.2 · p. 7
• CoS outperforms closed-source methods and other open-source 7B-scale models on the MLVU and LongVideoBench development sets.These results cover two long-video benchmarks.Source: §4.2 · p. 7
• CoS delivers significant improvements on NEXT-QA and MVBench across two baselines, achieving leading performance on both short-video benchmarks.This indicates that the visual prompting modification generalizes beyond long-video tasks.Source: Table 3 · p. 6, §4.2 · p. 7

Module Analysis

• Ablations show that binary video summary, original videos, positive shots, negative shots, and dynamic weighting each contribute to the full CoS model.Removing binary video summary regresses toward the baseline, while removing original videos eliminates a margin of error for shot-selection mistakes.Source: §4.2 · p. 7, Table 4 · p. 7

Shot Selection Model Analysis

• Using LongVA for shot selection performs worst, whereas other MLLMs improve results, suggesting diverse model strengths can mitigate biases in unlabelled shot selection.Qwen2 and LLaVA1.5 perform comparably in this analysis.Source: §4.2 · pp. 7–8

Impact of Shot-sampling Rate

• With 64 sampled shots performance is mediocre, but it remains stable from 96 to 192 shots, indicating robustness to sampling rate.The analysis attributes the 64-shot result to inadequate sampling and suggests dynamic selection helps when relevant shots are sparse.Source: Table 5 · p. 7, §4.2 · p. 8

Image Aggregation Shot Count Analysis

• Aggregation count 4 balances grounding granularity and inference speed, while counts above 4 reduce grounding accuracy and spatial positioning.Count 2 offers finer grounding but is more time-consuming; count 4 was selected for effective temporal-spatial grounding.Source: Table 5 · p. 7, §4.2 · p. 8

• CoS dynamically selects shots for each video instance and query task, constructing task-relevant positive and task-irrelevant negative videos to improve reasoning performance.The method is presented as a training-free test-time optimisation plug-in mechanism.Source: §5 · p. 8