Source-linked AI summary
ANCHOR: Audio-Visually Grounded Chain-of-Thought Reasoning Benchmark
Joel Julin, Souraja Kundu, Liza Dahiya, George Z. Wei, Liuyue Xie, Jingru Yi, Xu Zhang, Hao Yang, Laszlo A. Jeni
TL;DR
Answer-only benchmarks provide limited evidence about whether multimodal rationales identify the perceptual evidence supporting a prediction. ANCHOR evaluates evidence-linked reasoning across audio-visual video QA, and its results show that answer performance alone does not ensure grounded explanations. Cue-removal tests also show a larger accuracy drop from removing video than from removing audio or bounding boxes.
Problem
Existing evaluations mainly score final answers and rarely test whether MLLM reasoning identifies relevant visual and audio evidence.
Source: §1 · p. 1, §2.2 · p. 3Method
Results
Answer accuracy does not guarantee grounded reasoning; removing video reduces accuracy by 38.4–41.8 percentage points in the tested models.
Source: §4.3 · p. 8, §5 · p. 9Takeaways & Limitations
The findings support evaluating perceptual evidence use alongside answer correctness in audio-visual reasoning.
Source: §5 · p. 9ANCHOR is limited by the scale of human-refined annotations, reliance on pseudo ground-truth traces, and coverage of the source datasets.
Source: §5 · p. 9
Abstract
from arXiv · showhide
Multimodal large language models (MLLMs) perform well on audio-visual question answering, but correct answers do not necessarily imply that their stated reasoning is grounded in relevant perceptual evidence. Existing benchmarks mainly evaluate final-answer accuracy, offering limited insight into whether model-generated rationales correctly identify visual regions, temporal intervals, and audio events supporting predictions. We introduce ANCHOR, a benchmark and evaluation framework for measuring grounded-rationale alignment in multimodal reasoning, comprising 2,319 videos and 8,272 question-answer pairs with structured reasoning traces, temporal evidence, spatial bounding boxes, and salient audio cues, including a human-refined test set of 224 videos and 975 question-answer pairs. We separately measure answer correctness, hallucination, logical divergence, and conciseness, with direct spatial and temporal grounding measurements on a focused subset. Across the full benchmark, twelve contemporary MLLMs show a substantial alignment gap: models with comparable answer performance can differ markedly in whether their explanations are supported by annotated evidence. On a matched subset of 100 videos and 446 question-answer pairs, rankings remain highly stable across human-refined, Gemini-generated, and GPT-generated references (mean Pearson r=0.99), providing evidence against model-reference circularity within this subset. Grounding-cue ablations of three MLLMs on the full human-refined test set reveal a consistent modality hierarchy: removing video reduces accuracy by 38.4-41.8 points, while removing bounding boxes (3.2-7.7 points) and audio (4.2-5.6 points) causes smaller, complementary, model-dependent drops. These findings show answer-only evaluation obscures important grounding failures and position ANCHOR as a testbed for evaluating accurate and evidence-aligned audio-visual reasoning.
1 Introduction
MLLMs can produce plausible multimodal answers without grounding their reasoning in perceptual evidence, while answer-only evaluation rarely tests whether they used visual and audio signals. ANCHOR evaluates that alignment by requiring evidence-linked reasoning traces.
1 Introduction
- Answer-only evaluations rarely test whether MLLMs reason over visual and audio signals rather than exploit linguistic correlations.Correct conclusions can still rest on ungrounded reasoning, which the paper links to shortcut learning and vulnerability in high-stakes multimodal scenarios.Source: §1 · pp. 1–2
- As Figure 1 illustrates, Gemini-3-Flash answers correctly but produces verbose, uninformative “filler” reasoning, while GPT-5.2 prematurely stops audio-visual processing and answers incorrectly based on incomplete evidence.These examples show that neither final accuracy nor explanation length guarantees perceptual faithfulness.Source: §1 · p. 2
- ANCHOR evaluates multimodal reasoning with structured Grounded Chain-of-Thought traces that cite supporting timeframes and evidence.Its 2,319 videos and 8,272 question-answer pairs include spatial bounding boxes and fine-grained visual and audio event cues.Source: §1 · p. 2
2 Related Works
Prior work studies text-only Chain-of-Thought and multimodal understanding, but plausible rationales and correct answers alone do not establish which perceptual evidence supported a prediction. ANCHOR evaluates generated audio-visual video-QA rationales against aligned evidence.
2.1 Chain-of-Thought Reasoning.
- In perceptual tasks, plausible text rationales do not establish that a model identified the correct evidence; ANCHOR links rationales to video intervals, audio events, and localized entities.The resulting observable rationales can be compared with annotated perceptual evidence.Source: §2.1 · pp. 2–3
2.2 Multimodal Understanding Benchmarks.
- Most multimodal benchmarks focus on final-answer correctness, while ANCHOR evaluates generated rationales against aligned temporal, spatial, visual, and audio evidence.Its comparison with prior benchmarks highlights dense spatial grounding and human-verified reasoning traces for evidence-linked evaluation.Source: §2.2 · p. 3, Table 1 · p. 3
3 Benchmark Design
ANCHOR assembles audio-visually grounded video questions with aligned reasoning traces, temporal evidence, audio cues, and spatial boxes. Its design includes varied videos, a predominantly hard question set, and human refinement of perceptual and reasoning annotations.
3.1 Dataset Construction Pipeline
- ANCHOR combines videos requiring visual or audio evidence with multiple-choice and open-ended questions spanning diverse domains and temporal and spatial reasoning.Source datasets contribute examples selected for cross-modal grounding, fine-grained temporal boundaries, instructional structure, and long-horizon retrieval.Source: §3.1 · pp. 3–4
- The construction pipeline augments existing benchmark answers with aligned reasoning traces, temporal evidence, audio cues, and spatial bounding boxes.The benchmark reuses source-dataset videos and answers rather than authoring them from scratch.Source: §3.1 · pp. 3–4
3.2 Dataset Statistics
- 2,319 videos and 8,272 question-answer pairs include a human-refined subset of 224 videos and 975 pairs.Videos span 4.9 seconds to 114.3 minutes; 37.86% are under one minute, 54.12% one to ten minutes, and 8.02% over ten minutes. Figure 2 summarizes category, duration, and resolution distributions.Source: §3.2 · p. 4, Figure 2 · p. 4
3.3 Difficulty Rating
- Difficulty labels use rollout disagreement and intra-rollout similarity to identify questions requiring multimodal reasoning while filtering apparent difficulty from unstable outputs or ambiguous annotations.Four experts reviewed questions labeled hard; about 80% were confirmed as hard, and 53.8% of AI-labeled questions were hard.Source: §3.3 · pp. 4–5
3.4 Human-in-the-Loop Video Annotation
- Human annotators refine bounding boxes, audio-visual event spans, and reasoning steps against raw perceptual evidence in a synchronized annotation interface.An IRB-approved study with 26 participants refined 224 videos and 975 question-answer pairs; a fixed 100-video, 446-pair subset supports reference-source and direct grounding analyses.Source: §3.4 · pp. 5–6, Figure 3 · p. 5
4 Experiments
The experiments evaluate both the quality of models’ reasoning traces and the effect of grounding cues on answer correctness. They also test reference-source robustness and judge agreement.
4.1 Experimental Setup
- For clips over 20 seconds, models process sequential chunks with prior reasoning and evidence history before a final text-only synthesis pass.Frame-based models use 120-second chunks; native-video models use 30-minute chunks.Source: §4.1 · p. 7
- On the focused validation subset, scores correlate strongly across human-refined, GEMINI-, and GPT-generated references (Pearson r = 0.96–1.00).GEMINI does not score highest under its own generated reference on any facet.Source: §4.1 · p. 7
4.2 CoT Evaluation
- GEMINI-3-FLASH and GPT-5.2 have similar Answer scores (3.83 and 3.77), but differ in hallucination and divergence on fully correct answers.Among 2,012 items where judges rated both models’ answers fully correct, GPT-5.2’s Hallucination score was at least one point lower on 250 items and at least 1.5 points lower on 81; three recurring failures involved wrong events, wrong entities or intervals, and evidence not connected to the answer.Source: Table 2 · p. 6, §4.2 · p. 8
- Models given frames and ASR rather than native audio report no usable audio more often, but input format and architecture are confounded.This occurs in 25.3% of GPT-5.2, 25.0% of Claude, and 20.9% of Grok responses, versus 3.7% for Nova and 0.1% for Gemini-3-Flash; 85% of GPT-5.2’s cases depend on informative non-speech audio.Source: §4.2 · p. 8
4.3 Ablation on Grounding Cues
- Removing video reduces accuracy by 38.4–41.8 percentage points, versus 4.2–5.6 points for audio and 3.2–7.7 points for bounding boxes.Removing audio and bounding boxes together reduces accuracy by 7.5–9.6 points; the relative importance of those cues varies by model.Source: §4.3 · p. 8, Table 4 · p. 8
4.4 Ablation on Judge Models
- Judge agreement is moderate to high for hallucination and divergence, but panel ratings correlate only moderately with human ratings (Pearson r=0.50).On 100 sampled responses, humans rated hallucination higher than the panel (4.35 vs. 2.35).Source: §4.4 · pp. 8–9, Figure 5 · p. 9
5 Conclusion
ANCHOR evaluates whether MLLMs ground their reasoning in audio-visual evidence, combining structured perceptual anchors with a construction pipeline and multidimensional evaluation. The conclusion reports a persistent gap between answer accuracy and grounded reasoning, while noting limits in human-refined annotation scale, pseudo-ground-truth traces, and source-dataset coverage.
5 Conclusion
- Strong answer accuracy does not guarantee grounded reasoning; ablations indicate spatial and visual evidence matter.The benchmark evaluates grounded CoT with explicit bounding boxes, representative frames, and salient audio events.Source: §5 · p. 9
- The conclusion identifies limited human-refined annotation scale, reliance on pseudo-ground-truth traces, and source-dataset coverage as constraints.It calls for benchmarks that directly stress-test perceptual evidence use rather than evaluating answer correctness alone.Source: §5 · p. 9
A Appendix
- The appendix supplies curation and evaluation details, including difficulty assignment, annotation schema, judge rubrics, and bounding-box target selection.Full prompts and extended quantitative results are in the supplementary material.Source: A Appendix · p. 13
A.1 Difficulty Assignment
- Difficulty is the maximum severity indicated by rollout disagreement or intra-rollout answer diversity.The procedure is intended to distinguish robust multimodal reasoning cases from questions answerable through simple recognition or linguistic priors.Source: §A.1 · p. 13
A.2 Grounded Chain-of-Thought Annotation Schema
- Grounded traces separate visual cues, audio cues, and the reference answer, and human annotators refine the reasoning and its grounding links.Visual evidence includes relevant temporal spans and spatial evidence; audio evidence includes speech, sound events, music, or other acoustic information.Source: §A.2 · pp. 13–14
A.3 Judge Rubrics
- ANCHOR scores answer correctness, hallucination, and divergence with multiple LLM judges, while measuring conciseness separately from token counts.The distinct rubrics distinguish unsupported reasoning from changes to the reasoning trajectory and from verbosity.Source: §A.3 · p. 14
A.4 Bounding-Box Target Selection
- Bounding-box targets are concrete visible entities; annotators verify and refine SAM3 proposals to align boxes with the trace's perceptual evidence.Abstract events or actions are mapped to the visible entity performing or undergoing them.Source: §A.4 · p. 14
A.5 Reference-Source Circularity
On the focused validation subset, model scores were strongly correlated across human-refined and model-generated reference sources. GEMINI did not score highest under its own generated reference on any of the three judged facets.
A.5 Reference-Source Circularity
- Pearson r = 0.96–1.00 (mean 0.99) and Spearman ρ = 0.88–1.00 (mean 0.95) across reference-source pairs; mean absolute score difference was 0.05 points.The comparison used n=9 models, with a maximum absolute score difference of 0.08 on the 1–5 scale.Source: §A.5 · pp. 14–15
- GEMINI scored highest under the human-refined reference, not its own generated reference, on all three judged facets.Source: §A.5 · p. 15
A.6 Direct Spatial and Temporal Grounding
The focused subset evaluates box overlap and temporal agreement against human-refined tracks. GPT-5.6-LUNA showed stronger spatial and temporal alignment, while CLAUDE-HAIKU-4.5 had higher temporal recall but lower precision.
A.6 Direct Spatial and Temporal Grounding
- GPT-5.6-LUNA achieves stronger spatial and temporal alignment; CLAUDE-HAIKU-4.5 attains higher temporal recall but lower precision.IoU measures spatial overlap, while temporal precision, recall, and F1 measure agreement over visible frames.Source: §A.6 · p. 15, Table 5 · p. 15
- Direct grounding metrics complement semantic evaluation because exact overlap can penalize alternative valid evidence.Source: §A.6 · p. 15
A.7 Human-Judge Validation
Human evaluation found moderate correlation between LLM-judge scores and human ratings, but limited absolute calibration. The supplementary material contains full prompts and extended results.
A.7 Human-Judge Validation
- Judge-panel scores correlate moderately with human ratings (Pearson r = 0.50), while exact-category agreement is modest (weighted Cohen’s κ = 0.11 to 0.39).Human ratings exceeded judge-panel ratings across all three facets, especially for hallucination (4.35 vs. 2.35 on the 1–5 scale).Source: §A.7 · p. 15
A.8 Relationship to the Supplementary Material; B.1 Overview
- The supplementary material provides complete prompts and extended results, including judge-agreement plots and performance by difficulty and annotation source.Source: §A.8 · p. 15, §B.1 · p. 16
B.2 Additional Benchmark Details
This part details taxonomy labeling, pseudo-ground-truth reasoning and bounding-box label generation, difficulty assignment, and evaluation-judge criteria, including a prompt example illustrating a rating of 3 for partially aligned reasoning with different intermediate steps.
B.2 Additional Benchmark Details
- GPT-5.2 labels question-answer pairs with one MAVEN3 top-level and one subcategory, using the question and answer as primary signals and analysis only as support.When uncertain, the prompt directs the labeler to choose the closest category and lower confidence; Table 8 reports category-wise, subcategory-wise, and joint taxonomy distributions.Source: §B.2 · p. 16
B.3 Prompts for CoT Generation
- GEMINI-2.5-PRO generates pseudo-ground-truth chain-of-thought for each question-answer pair.Source: §B.3 · p. 16
B.4 SAM3 Bounding Box
- GPT-4O-MINI extracts a concrete visible object label from the question, answer, and Gemini analysis for frame-by-frame SAM3 bounding-box generation.The prompt favors specific, central, consistently visible entities and substitutes a visible entity when the question concerns an abstract action or concept.Source: §B.4 · p. 17
B.5 Difficulty-Rating Procedure
- Difficulty is the more severe tier from answer disagreement and judge similarity: at least 10 or 5 mismatches indicates difficult or medium, while similarity scores 5 or 3–4 do likewise.The procedure assigns easy below those thresholds and takes the maximum severity of the two tiers; the judge semantically checks 16 candidate answers and rates their overall similarity from 1 to 5.Source: §B.5 · pp. 17–18
B.6 Judge Prompts for Evaluation
- The three Likert-scale judges separately assess lenient final-answer correctness, unsupported content in reasoning, and divergence from reference reasoning, with divergence ignoring conciseness and hallucination.Accuracy focuses on semantic consistency with the ground truth; hallucination measures unsupported entities, facts, assumptions, or steps; divergence measures differences in reasoning logic, structure, and interpretation.Source: §B.6 · pp. 18–20
- Rating 3 indicates partial alignment with the ground truth alongside different intermediate reasoning steps.The example also begins to describe an omission, but the supplied passage is incomplete.Source: §B.6 · p. 21
B.7 Additional Experiments
The appendix reports extended experiments and a per-model view of agreement among judges across three evaluation dimensions.
B.7 Additional Experiments
- The appendix reports additional experimental results beyond those included in the main paper.The supplied passage does not specify those results.Source: §B.7 · p. 21
B.8 Judge Agreement
- Per-model Cohen’s κ plots show how consistently judge models rate evaluated models on Divergence, Hallucination, and Final Answer.The plots provide a finer-grained view than the aggregate agreement results in the main paper.Source: §B.8 · p. 21
B.9 Ablation #1: Performance by Difficulty Level
This section examines how model performance varies across question-difficulty splits.
B.9 Ablation #1: Performance by Difficulty Level
- Model performance is reported by facet across Easy, Medium, and Difficult question splits, using the dataset’s difficulty labels.The passage describes the analysis and splits but provides no performance outcomes.Source: §B.9 · p. 21
B.10 Ablation #2: Annotation Separation
This section compares model performance across human-only, human-plus-AI, and AI-only annotation subsets.
B.10 Ablation #2: Annotation Separation
- Performance remains broadly consistent across annotation-source splits, with modest differences in answer quality and reasoning behavior.Table 7 reports per-facet scores for each model on the human-only (H), human+AI (H+AI), and AI-only (AI) subsets.Source: §B.10 · p. 21, Table 7 · p. 26