Source-linked AI summary
CLBench-V: Evaluating Multimodal Context Learning from Grounding to Knowledge Acquisition
Lai Wei, Chengqi Li, Jiapeng Li, Ruina Hu, Yue Wang, Weiran Huang
TL;DR
Existing context-learning evaluations mainly use text, despite many real-world tasks requiring models to learn from multimodal evidence. CLBench-V organizes multimodal context learning into three capability levels and finds that current systems remain unreliable, with the best overall score reaching only 0.2847.
Problem
Existing context-learning benchmarks are largely unimodal, leaving multimodal evidence use insufficiently evaluated despite its prevalence in science, finance, spatial reasoning, and web information seeking.
Method
CLBench-V benchmarks context grounding, new information application, and new knowledge learning using converted public benchmarks and automatically constructed, filtered domain-specific tasks.
Results
Across six multimodal models, the best overall score is only 0.2847, with InternVL3.5-30B-A3B strongest on grounding and knowledge learning and Qwen3.5-Plus strongest on information application.
Takeaways & Limitations
Reliable multimodal context use requires more than accepting multimodal input, because models differ in grounding evidence, applying supplied information, and acquiring context-defined knowledge.
Takeaways & Limitations
The benchmark integrates heterogeneous data sources with differing annotation styles and evaluation protocols, while some subsets are small diagnostic probes rather than comprehensive task distributions.
Abstract
from arXiv · showhide
Real-world tasks often require models to learn from task-specific context rather than relying only on pre-trained knowledge. While recent work has highlighted this capability as context learning, existing evaluations mainly focus on textual contexts. In many practical settings, however, the context to be learned from is multimodal: scientific findings are conveyed through figures and tables, financial indicators are scattered across converted reports, and spatial decisions depend on maps, scenes, or web pages. We introduce CLBench-V, a benchmark for multimodal context learning that addresses the difficulty of localizing where context use breaks down by organizing tasks around three dimensions: context grounding, new information application, and new knowledge learning. CLBench-V combines converted public benchmarks with newly constructed datasets spanning domains such as science, finance, long-document understanding, spatial reasoning, and web-based visual question answering. To reduce the cost of constructing domain-specific context-learning tasks, we further use automated construction and filtering procedures for our newly built datasets. Across 3,443 instances and six recent multimodal models, the best overall score is only 0.2847, indicating that multimodal context learning remains far from saturated. Moreover, InternVL3.5-30B-A3B performs best on context grounding and new knowledge learning, while Qwen3.5-Plus performs best on new information application. We further analyze judge reliability, context length, image count, and representative failure cases. Code is available at https://github.com/IamLihua/CLBench-V.
1 Introduction
CLBench-V evaluates multimodal context learning through a three-level hierarchy that separates context grounding, new information application, and new knowledge learning. Across 3,443 instances and six multimodal models, the best overall score is 0.2847, showing that current systems still struggle with reliable multimodal context use.
- Benchmark construction: Automated construction and filtering procedures synthesize candidate instances at scale while retaining task-specific quality control.These procedures address the high cost of manually annotating high-quality context-learning tasks.
- Benchmark framework: CLBench-V organizes multimodal context learning into L0 Context Grounding, L1 New Information Application, and L2 New Knowledge Learning.The hierarchy distinguishes accessing relevant evidence, applying newly provided information, and acquiring knowledge from context rather than relying on prior knowledge.
- Benchmark construction: The benchmark combines converted public benchmarks with newly constructed domain-specific tasks, including medical-paper conclusion inference and financial-report ROE analysis.All tasks are evaluated under a unified inference and scoring framework.
- Results: 0.2847 was the best overall score across 3,443 instances and six multimodal models.The experiments indicate that current systems continue to struggle with multimodal context learning.
- Results: InternVL3.5-30B-A3B performs best on L0 Context Grounding and L2 New Knowledge Learning, whereas Qwen3.5-Plus performs best on L1 New Information Application.The level-wise variation indicates that accepting multimodal input alone does not ensure reliable context use.
2 Related Work
Prior work evaluates long-context retrieval, reasoning, and multimodal document understanding, while CLBench-V extends these foundations by distinguishing evidence finding from information application and knowledge acquisition.
- Context learning and long-context evaluation: Long-context benchmarks evaluate retrieval, long-document question answering, summarization, few-shot learning, and reasoning across documents, code, and structured data.Position-sensitive analyses and synthetic needle-in-a-haystack tests probe whether models use evidence buried in distractor contexts.
- Context learning and long-context evaluation: Many-shot in-context learning studies whether expanded context windows enable learning from hundreds or thousands of demonstrations at inference time.CL-Bench further evaluates acquisition of new information, rules, procedures, and empirical patterns beyond retrieval or simplification.
- Multimodal long-context modeling: Multimodal long-context research examines training recipes, sequence-length exposure, and data-mixture design, with evaluations spanning multi-image and page-rich document contexts.Controlled visual needle tests isolate retrieval, counting, and evidence-po… capabilities.
- Multi-image and visually rich document understanding: Multi-image and document benchmarks cover comparison, co-reference, temporal ordering, cross-image composition, visual reading, numerical reasoning, page selection, and layout preservation.CLBench-V treats these perceptual and retrieval demands as prerequisites, then diagnoses whether evidence is found, applied as new information, or generalized as newly acquired knowledge.
3 CLBench-V: Task and Benchmark Design
CLBench-V organizes multimodal context-learning tasks by three capability levels—context grounding, new information application, and new knowledge learning—to diagnose distinct failure modes. It combines integrated public benchmarks with newly constructed domain-specific tasks under a unified schema and evaluator interface.
- Capability hierarchy: The three-level hierarchy separates context grounding, new information application, and new knowledge learning, with L2 requiring synthesis and transfer of newly learned multimodal knowledge.The levels distinguish missing evidence, misuse of context-specific information, and failure to acquire context-defined knowledge.
- Capability hierarchy: The levels are cumulative but analytically distinct, and orthogonal dimensions expose differences in operations such as chart reading, route validation, page search, conclusion induction, and numeric evaluation.This diagnostic design allows tasks at the same level to stress different multimodal operations and failure modes.
- Benchmark construction: CLBench-V combines integrated public benchmarks with newly constructed domain-specific tasks, mapping both sources into one normalized schema organized by primary capability.The integration source provides broad capability diversity, while the construction source targets under-covered settings for context-grounded information use and knowledge acquisition.
- Benchmark construction: Public-benchmark admission requires visual dependency and low irrelevant context, excludes video and quasi-video tasks, and selects sources spanning documents, science, maps, sports, search, and multi-image reasoning.The curation objective is to evaluate context learning rather than shortcut retrieval or capabilities primarily supported by training exposure to video.
- New domain-specific tasks: The financial-report task converts PDFs into Markdown-style multimodal documents and asks models to compute ROE through DuPont-style reasoning over report quantities.The task requires identifying and combining net income, revenue, total assets, and shareholders’ equity; evaluation focuses on final ROE exact match.
4 Experiments
Experiments evaluate six multimodal models under standardized task inputs and report low, heterogeneous performance across context-learning levels and datasets. InternVL3.5-30B-A3B leads overall and grounding, while Qwen3.5-Plus leads new-information application, with no model dominating every dataset.
- Experimental setup: Six multimodal models receive the same task prompt, input context, and answer-format instruction, with minor formatting changes only for model-specific API constraints.Qwen3.6-27B also serves as the main judge for judge-based tasks.
- Experimental setup: Final answer extraction and task-specific answer-format instructions are used to reduce ambiguity and limit judge sensitivity to irrelevant reasoning text.Multiple-choice tasks require an option letter, while open-ended tasks require a concise final answer after reasoning.
- Results by level: 0.2847 is the best overall score, showing that multimodal context learning remains challenging for recent systems and is not saturated by one model family.Scores are sample-weighted over the currently completed subsets.
- Results by level: InternVL3.5-30B-A3B achieves the highest overall and L0 grounding scores, whereas Qwen3.5-Plus achieves the strongest L1 information score.These contrasting profiles indicate stronger grounding for the former and better use of newly supplied information for the latter than acquisition of context-defined knowledge.
- Results by dataset: Dataset leadership is fragmented: Qwen3.6-27B leads Pix2Fact, Insight-O3, and CL-Bench-Table; Qwen3.5-Plus leads MMLongBench-Pic and PRISMM-Bench; Kimi-K2.6 leads all ReasonMap subsets; InternVL3.5-30B-A3B leads Paper Conclusion and MIRBench; and GPT-5.4 leads BrowseComp-V3.Persistent difficulties include Pix2Fact, BrowseComp-V3, CL-Bench-Table, and CourtSI, revealing weaknesses in visual fact grounding, web-style evidence synthesis, context-defined rule following, and spatial intelligence.
5 Do LLM Judges Matter?
LLM judge choice materially affects scores on open-ended multimodal context-learning tasks. Qwen3-VL-4B-Instruct achieves the highest average score in the comparison, while Qwen3-VL-32B-Instruct achieves the lowest.
- Judge reliability: Judge models produce non-negligible score variation on semantically evaluated multimodal context-learning tasks.Scores are reported as macro averages over datasets requiring LLM judging for each prediction set.
- Judge reliability: Qwen3-VL-4B-Instruct gives the highest average score, whereas Qwen3-VL-32B-Instruct gives the lowest average score.The comparison reports macro scores across two prediction sets.
- Judge reliability: The findings motivate transparent reporting of judge models and further calibration in future benchmark versions.Evaluation quality is important because CLBench-V includes both structured and open-ended tasks.
6 Diagnostic Analysis
Diagnostic analyses show that input length alone does not explain failures: performance depends more on usable context limits, while image-count effects vary by model. Manual inspection identifies six recurring failure types involving visual evidence, context application, prior knowledge, and induction.
- Input complexity: For Qwen3.6-27B, token length is essentially uncorrelated with score, with Pearson r = −0.0042 and Spearman ρ = 0.0500.The passage also reports that image count is only weakly negatively correlated, but the value is truncated.
- Input complexity: For financial reports, token-score Pearson correlation becomes positive for InternVL3.5-30B-A3B at r = 0.0510 and Qwen3.6-27B at r = 0.1429.These reports are the longest inputs, and InternVL3.5-30B-A3B exceeds its context limit on them.
- Input complexity: The analysis suggests failures depend on crossing a model’s usable context limit rather than on input length within that limit.This conclusion follows from the contrast between near-zero correlations generally and positive correlations on financial reports that exceed a model’s context limit.
- Input complexity: Image-count effects are model-dependent: InternVL3.5-30B-A3B improves with more images, while Qwen3.6-27B does not.InternVL3.5-30B-A3B has ρ = 0.1217 overall and ρ = 0.2317 excluding financial reports; Qwen3.6-27B has ρ = −0.0739 and ρ = −0.0402, respectively.
- Failure cases: Manual inspection categorizes representative failures into six types: evidence missing, evidence misbinding, context misuse, prior override, incomplete induction, and one additional type not specified in the supplied passage.The passage defines evidence missing as failing to attend to the relevant visual region, evidence misbinding as attaching evidence to the wrong entity or source, context misuse as applying read information incorrectly, prior override as relying on parametric knowledge despite contradictory context, and incomplete induction as failing to infer the broader context-defined rule from local facts.
7 Limitations
CLBench-V is an initial benchmark with heterogeneous data sources, small diagnostic subsets, and reliance on LLM judges that may introduce bias. Its financial-report task currently evaluates only final ROE values rather than intermediate DuPont reasoning.
- Benchmark scope: CLBench-V combines heterogeneous data sources with differing annotation styles and evaluation protocols.This limits uniformity across benchmark components.
- Benchmark scope: Some subsets are small and should be interpreted as diagnostic probes rather than comprehensive task distributions.Their limited size constrains broad distributional conclusions.
- Evaluation: LLM judges are necessary for several open-ended tasks but may introduce model-specific biases despite answer extraction and rubric design.Judge dependence is therefore a limitation of the evaluation process.
- Task coverage: The financial-report task evaluates only final ROE values and does not fully score intermediate DuPont reasoning steps.This leaves intermediate reasoning insufficiently assessed.
8 Conclusion
CLBench-V is a multimodal context-learning benchmark that organizes tasks around context grounding, new information application, and new knowledge learning. It combines converted public datasets with newly constructed scientific-paper and financial-report tasks and evaluates models with task-specific protocols.
- Benchmark design: CLBench-V benchmarks multimodal context learning across context grounding, new information application, and new knowledge learning.This organization distinguishes failures of visual access from failures of contextual reasoning and knowledge acquisition.
- Benchmark design: The benchmark combines converted public datasets with newly constructed scientific-paper and financial-report tasks.These datasets are paired with task-specific evaluation protocols.
- Analysis: Preliminary experiments analyze model performance and judge reliability.
Ethics Statement
CLBench-V uses publicly available benchmark, scientific-paper, and financial-report sources while preserving source attribution for public components. The authors report no human-subject experiments or private personal data, document safeguards and limitations, and treat AI assistance as subject to author review and validation.
- Source Data: CLBench-V is built from converted public benchmarks and newly constructed tasks based on publicly available scientific papers and financial reports.Public benchmark components follow the original datasets’ intended research use, with source attribution retained in data records.
- Ethics Review: The benchmark involves no human-subject experiments, private user data, or newly collected personal information.Converted examples are screened for obvious personally identifiable information, and unnecessary content is excluded.
- Ethics Review: The tasks evaluate models’ use of provided context rather than inferring sensitive attributes or supporting decisions about real people.Released data will include documentation on source provenance, task construction, evaluation scripts, and known limitations.
- AI Assistant Usage: AI assistants supported data processing and benchmark construction, while authors reviewed benchmark definitions, filtering criteria, evaluation protocols, and paper claims.AI-generated code or text was treated as an aid rather than an authority, with benchmark outputs subject to manual inspection or task-specific validation.
A Dataset-Level Details · B Prompt Templates · C Additional Bad Cases
The appendix details CLBench-V’s broad input-length coverage and unified prompt pipeline, including task-specific templates, answer extraction, judging, and representative failures across grounding, value binding, and procedural reasoning.
- A Dataset-Level Details: CLBench-V spans short visually grounded tasks and very long document-centered tasks, with several subdatasets contributing many examples above 16k tokens.Token counts use the Qwen3-VL-4B tokenizer over normalized model inputs.
- B Prompt Templates: A unified message-construction pipeline normalizes each query and image list while optionally adding system, answer-format, repair, extraction, or judge prompts.Image placeholders use <image> and are replaced with image payloads at inference time.
- B Prompt Templates: Public benchmark conversions preserve original queries when possible and add minimal answer-format constraints, including JSON outputs for webpage reasoning and structured route segments for subway maps.Scientific-paper inconsistency tasks require selecting one option from A–D.
- C Additional Bad Cases: Representative bad cases cover visual grounding and entity-to-region binding, context-specific row–column value binding, and failure to induce procedural semantics for scientific-figure inconsistency repair.These cases align with the L0/L1/L2 taxonomy.
- B Prompt Templates: Newly constructed datasets use prompts for DuPont-based ROE computation from financial reports and concise experimental-conclusion inference from research papers.The financial prompt requests extracted quantities, intermediate calculations, and ROE to three significant digits.
- B Prompt Templates: Evaluation uses final-answer extraction followed by task-specific judges, including strict semantic equivalence, all-or-nothing rubric, and scientific-claim list-count scoring.List-count matching is one-to-one and based on scientific claims rather than wording.
C.1 Failure Taxonomy Case Studies
Three representative cases show failures escalating from incorrect visual evidence binding (L0), through incorrect context-specific value application (L1), to incomplete learning and execution of context-defined procedures (L2). Together, they illustrate how early context-learning mistakes propagate into downstream answer errors and support the diagnostic value of the taxonomy.
- Case A — L0: Evidence Misbinding in Visual Grounding: L0 failure: the model grounds to a nearby white sign, extracting x = 102 instead of 14 and returning no answer for 2113 rather than 5.4%.The task required using the “Zone Leidseplein” board to compute 2011 + x and retrieve China’s Q1 year-on-year GDP growth for 2025.
- Case B — L1: Context Misuse in Chart Value Binding: L1 failure: retrieval and localization succeed, but the model binds the query to the wrong respondent and response category, returning 33% instead of 18%.The error concerns Democratic respondents and the “neither party” response in the Pew chart.
- Case C — L2: Incomplete Induction of Context-Defined Procedure: L2 failure: the model treats an inconsistency-repair task as local layout checking, repositioning denoising without updating graph topology after OT matching.The correct procedure requires inferring that changes in edge-noise composition must be reflected in the post-denoising graph topology.
- Cross-case diagnosis: The cases progress from incorrect evidence binding (L0), through incorrect value application (L1), to failure to learn and execute context-defined rules (L2).Each case corresponds to one context-learning level and illustrates how an early mistake propagates into downstream answer errors.