CG-Bench: Clue-grounded Question Answering Benchmark for Long Video Understanding

• Long-video MCQs can reward elimination based on textual information or sparse video knowledge rather than engagement with relevant content.Source: §1 · p. 1
• A comprehensive benchmark for credible evaluation of generalist MLLMs on diverse long videos remains missing; NExT-GQA is limited to one dataset and primarily short videos.Source: §1 · pp. 1–2
• CG-Bench tests clue-grounded QA through white-box clue intervals, black-box alignment of video- and clue-level MCQs, and a human-clue-aided heuristic for open-ended QA.Source: §1 · p. 2
• Its 1,219 videos and 12,129 QAC triplets span 14 primary, 171 secondary, and 638 tertiary categories, plus perception, reasoning, and hallucination questions.Source: §1 · p. 2
• 53.9 to 21.7: existing MLLMs’ scores decrease under credibility and open-ended evaluations, despite relatively high long-video MCQ accuracy.Source: §1 · p. 2

• Video models have explored frame-token compression and long-context system optimization, but real-world long-video performance remains underexplored.Source: §2 · p. 3
• Short-video benchmarks include TVQA, NextQA, VideoBench, MVBench, and EgoSchema, whose clips range from seconds to tens of seconds.Source: §2 · p. 3
• Long-video benchmarks such as VideoMME and MLVU broaden evaluation, but commonly rely on MCQs whose negative-option design affects difficulty.Source: §2 · p. 3
• Sparse frames and common-sense reasoning can let MLLMs eliminate wrong options and inflate MCQ performance; CG-Bench uses clue intervals for credibility evaluation.Source: §2 · p. 3

• Dataset construction proceeds through video collection, QAC annotation, and quality-review iteration.Source: §3.1 · p. 3
• Videos are manually collected from the internet, defined as long above 10 minutes, tagged for diversity, reviewed over seven rounds, and retained with subtitles and audio.Source: §3.1 · pp. 3–4
• Annotators create 6–15 QAC triplets per video across perception, reasoning, and hallucination, selecting one or more intervals that cover relevant short moments.Source: §3.1 · p. 4
• Reviewers reject or revise annotations and check question rationality, video dependence, negative-option difficulty, and clue-interval positional diversity.Source: §3.1 · p. 4

3.2.1 DATASET STATISTICS

• 1,219 videos span 10–80 minutes, with 20–30-minute videos most prevalent and content tagged across 171 secondary and 638 tertiary categories.The collection uses a three-tier tagging system to characterize video content.Source: §3.2.1 · p. 4, Figure 4 · p. 5
• 14,362 clue intervals average 19.24 seconds, and their distribution across normalized video timestamps is described as unbiased.The benchmark contains 12,129 question-answer-clue annotations.Source: §3.2.1 · pp. 4–5, Figure 5 · p. 5

3.2.2 COMPARISON WITH PREVIOUS BENCHMARKS

• CG-Bench-QG has the most videos and longest average duration among the compared question-clue-grounding benchmarks, whose videos are primarily action- or egocentric-domain academic samples.The comparison covers NextGQA, Ego4D-NLQ, MultiHop-EgoQA, E.T. Bench, and RexTime.Source: §3.2.2 · p. 5, Table 2 · p. 5
• CG-Bench-Clue is presented as the largest, held-out, open-domain, multimodal short-video QA benchmark compared with TempCompass, MVBench, and MMBench-Video.It is derived by transforming QAC triplets.Source: §3.2.2 · p. 5, Table 2 · p. 5
• For long-video QA, CG-Bench is reported to exceed prior benchmarks in video count, length, question quantity, and annotation quality.Its clue intervals also support reliable long-video evaluation and clue-assisted open-ended evaluation, unlike Video-MME and MLVU as described here.Source: Table 2 · p. 5, §3.2.2 · p. 6

3.3.1 MULTIPLE-CHOICE QUESTION EVALUATION

• Clue-based MCQ restricts input to the annotated clue interval, while Long-Video MCQ presents the full video; multiple clues for one question are merged into one segment.Both settings ask the model to choose an option using the question and candidate answers.Source: §3.3.1 · p. 6

3.3.2 CREDIBILITY EVALUATION

• White-box evaluation scores predicted clue intervals against ground-truth intervals using tIoU, mIoU, recall at IoU thresholds, and accuracy conditioned on overlap.Multiple intervals may be predicted; the default acc.@IoU threshold τ is 0, requiring at least some overlap as well as a correct answer.Source: §3.3.2 · p. 6
• Black-box evaluation defines Clue Recovery Rate (CRR) as the minimum of Long-Video MCQ and Clue-based MCQ accuracy, probing clue retrieval under diluted video context.A CRR below 100% indicates that short-clue retrieval from long-video representations is not optimal.Source: §3.3.2 · pp. 6–7

3.3.3 CLUE-AIDED OPEN-ENDED QA EVALUATION

• Open-ended answers may express the same correct response differently, creating ambiguity for LLM evaluators of long-video QA.The proposed heuristic first judges whether text alone suffices, then requests visual clues sampled from annotated question-clue intervals when needed.Source: §3.3.3 · p. 7, Figure 6 · p. 7

4.1 SETTINGS

• The experiments evaluate commercial models including GPT4o, Gemini-1.5, and Claude-3.5, alongside open-source models including LLaVA-OneVision, Qwen2-VL, and InternVL2.Long-Video MCQ uses 128 uniformly sampled frames; Clue-based MCQ uses 32.Source: §4.1 · p. 7

4.2 MAIN RESULTS

• 45.2%: GPT4o’s long-acc. approaches this level, while its Acc@IoU falls to 4.38 and OE-acc. to 39.5, illustrating weaker performance on credibility and open-ended evaluation.The authors report GPT4o ahead of other MLLMs across all metrics; Qwen2-VL is comparable to GPT4o on long-acc. and clue-acc.Source: §4.2 · p. 8, Table 3 · p. 8

4.3 ANALYSIS

• When FT and ST are both added, mIoU rises from 3.39 to 9.68 and Acc@IoU from 10.7 to 26.7; combining S, FT, and ST gives the best performance across metrics.Subtitles help long-acc. more than clue-acc.; audio yields no significant gain and can slightly degrade results, while visual input reduces subtitles’ impact.Source: §4.3 · p. 9, Table 4 · p. 9
• All three evaluated MLLMs improve gradually as frame counts rise, with GPT-4o consistently outperforming the others across metrics.Qwen2VL is comparable to GPT-4o on long-acc. and OE acc.; Gemini performs better than Qwen2VL on mIoU and Acc@IoU, while GPT-4o has more consistent CRR across frame counts.Source: Figure 7 · p. 9, §4.3 · p. 10
• GPT-4o has the highest evaluator stability and smallest deviation from human scores; the proposed heuristic has the lowest evaluation bias.On 200 manually annotated samples, visual requests trigger for 14% and have 88% recall; fully using visual information improves accuracy but increases time and cost.Source: §4.3 · p. 10, Table 5 · p. 10
• With 128 frames, GPT-4o-0806 struggles with undersampling, especially on longer videos.The human evaluation of 30 videos and 296 questions using 128 sampled frames achieved 59.85% accuracy, which the authors say highlights the benchmark’s challenge and room for model improvement.Source: §4.2 · p. 9, §4.3 · p. 10, Figure 9 · p. 10
• Higher resolution offers some long-acc. improvement for GPT4o-0806, whereas FFmpeg keyframe extraction has no significant impact.The sampling-strategy experiment used 50 uniformly sampled frames.Source: §4.3 · p. 10, Table 6 · p. 10

5 CONCLUSION AND FUTURE WORK

• CG-Bench combines 1,219 curated videos and 12,129 QA pairs across perception, reasoning, and hallucination question types with clue-grounded white-box and black-box evaluation.The benchmark uses a three-tier video categorization system.Source: §5 · pp. 10–11
• The authors report that current MLLMs significantly underperform in long-video understanding compared with short-video understanding.They present CG-Bench as a resource for developing more trustworthy and capable long-video MLLMs.Source: §5 · p. 11