Source-linked AI summary
BEAR-Bench: A Bilingual Enterprise and Academic Reasoning Benchmark for Multimodal Models
Liubov Chubarova, Alexandra Kuleshova, Daniil Volkov, Kirill Sultanov, Alexey Zaytsev
TL;DR
Reasoning over text-dense professional documents is insufficiently evaluated, especially beyond English and Chinese. BEAR-Bench addresses this with a bilingual, self-contained benchmark and finds substantial performance headroom, while hallucination detection remains imperfect.
Problem
Existing benchmarks incompletely cover multi-step reasoning over text-dense professional documents and substantially underrepresent Russian and other Slavic languages.
Method
BEAR-Bench provides 1,000 human-annotated, document-grounded questions in English and Russian and evaluates 16 MLLMs alongside hallucination-detection methods.
Results
75.4% was the highest overall accuracy, every model scored lower on Russian items, and best hallucination-detection balanced accuracy reached 0.74 for proprietary and 0.81 for open-weight outputs.
Takeaways & Limitations
BEAR-Bench offers a common setting for tracking progress in professional-document reasoning and hallucination detection.
Takeaways & Limitations
The benchmark evaluates only single-page English and Russian documents from public sources, limiting transfer to multi-page reasoning, other languages, domains, and private corpora.
Abstract
from arXiv · showhide
While Multimodal Large Language Models (MLLMs) have made significant strides in visual comprehension, their ability to reason about text-dense, professional documents remains incompletely evaluated. Existing benchmarks emphasize information extraction, require external domain knowledge, or cover professional documents only as one of many settings. They are also largely English- or Chinese-centric, leaving other languages and Russian, in particular, substantially underrepresented. To address these limitations, we introduce BEAR-Bench (Bilingual Enterprise and Academic Reasoning), a self-contained, complex English-and-Russian benchmark comprising 1000 human-annotated questions based on text-rich business and scientific documents. We evaluate 16 proprietary and open-weight MLLMs, including Gemini 3.1 Pro and Qwen3.5-397B, on BEAR-Bench and observe clear headroom even for the strongest systems. Finally, we use the resulting model outputs to compare existing hallucination detection methods, evaluating not only how often models fail on BEAR-Bench but also how reliably those failures can be identified.
1 Yandex 2 Applied AI Institute · 1 Introduction
BEAR-Bench evaluates document-grounded multimodal reasoning on complex, text-dense enterprise and scientific documents in English and Russian. It also assesses 16 MLLMs and compares hallucination-detection methods across proprietary and open-weight deployment settings.
- 1 Introduction: Existing benchmarks incompletely cover professional-document reasoning by emphasizing OCR or extraction, requiring specialized factual knowledge, or treating professional documents only superficially.DocVQA emphasizes OCR and information extraction; MMMU often requires specialized factual knowledge; OCR-Reasoning includes professional documents without examining the setting in depth.
- 1 Introduction: BEAR-Bench contains 1000 human-annotated questions on text-dense enterprise and scientific documents in English and Russian.Its science tasks interpret academic figures, transcribe mathematical formulas, and analyze plots, while enterprise tasks cross-reference financial reports and reason over business charts.
- 1 Introduction: The benchmark prioritizes document-grounded reasoning over factual recall or shallow extraction and extends evaluation beyond predominantly English- and Chinese-language resources through Russian tasks.This design targets reasoning grounded in the supplied documents rather than reliance on external factual knowledge.
- 1 Introduction: The study compares token-level uncertainty scores, representation-based detectors, supervised hidden-state probes, and MLLM-as-a-judge methods for hallucination detection.The evaluation addresses whether failures in text-dense professional-document reasoning can be detected reliably, not only how often models fail.
- 1 Introduction: BEAR-Bench evaluates 16 proprietary and open-weight MLLMs, including Gemini-3.1-Pro, Claude Opus 4.6, Qwen3.5, and gemma-4.Even the strongest evaluated systems leave clear headroom on the benchmark.
- 1 Introduction: Hallucination detectors are compared under direct internal-signal access for open-weight models and proxy-based detection for proprietary models.The methods span uncertainty-, representation-, and judge-based approaches for OCR-intensive professional-document reasoning.
2 Related Work
Prior multimodal benchmarks underrepresent self-contained, multi-step reasoning over text-dense professional documents, especially in Russian, while hallucination detectors have not been systematically evaluated in this setting. BEAR-Bench addresses these gaps with scientific and business documents and enables detector comparison on such inputs.
- Multimodal benchmarks for professional documents: Professional-document benchmarks are largely OCR-based, emphasizing extraction over cross-referencing text and visuals, or long-page settings that conflate multimodal reasoning with long-context handling.
- Multimodal benchmarks for professional documents: Multimodal reasoning benchmarks provide little professional-document signal, and text-dense reasoning coverage is concentrated in English or Chinese, leaving Russian underrepresented.
- Error detection for text-dense multimodal inputs: Hallucination detectors span tool-augmented methods, multi-query methods, and lightweight white-box methods based on token uncertainty or internal model states.Tool-augmented methods rely on auxiliary models, multi-query methods require repeated generation or verification calls, and white-box methods operate during a single forward pass.
- Summary: Existing work lacks a Russian-inclusive benchmark for self-contained, multi-step reasoning over text-dense professional documents, motivating BEAR-Bench’s scientific and business documents.
- Summary: BEAR-Bench enables systematic comparison of hallucination detectors on text-dense professional document images requiring multi-step reasoning, a setting absent from prior evaluations.
3 BEAR-Bench
BEAR-Bench is organized into disjoint Business and Science domains covering six document categories that require multimodal, multi-step reasoning. After automated filtering, expert annotation, judge validation, and quality control, it contains 1,000 document images paired with 1,000 QA instances across four domain–language groups.
- Benchmark domains: BEAR-Bench spans disjoint Business and Science domains, with no source document appearing in both subsets.Business includes financial reports, investor presentations, and flowcharts or organisational diagrams; Science includes formulae, scientific figures and plots, and academic layouts.
- Business Domain: Business documents test tabular reasoning, year-over-year calculations, chart and KPI interpretation, infographic maps, and tracing corporate or process structures.The subset uses financial reports, investor presentations, and flowcharts or organisational diagrams.
- Science Domain: Science documents test symbol-level recognition, axis and legend interpretation, reading-order resolution, and cross-referential reasoning.The subset covers mathematical and physical formulae, scientific figures and plots, and multi-column academic layouts.
- Data processing: Up to 1,500 images per language survived visual-content and textual-density filtering for stratified random sampling into the human-annotation pool.The pipeline used content-positive pages and retained pages at or above the 67th percentile of OCR character count within each language group.
- Annotation and evaluation: 1,000 document images and 1,000 human-authored QA instances remained after quality-control filtering across four domain–language groups.Thirteen technical-domain experts authored one question–answer pair per image, and GPT-4o judged responses with 99.0% overall human agreement in a 200-verdict audit.
4 Dataset Statistics and Analysis
BEAR-Bench is balanced across domain and language cells, with Russian business the largest subset and English business the smallest. Its annotations indicate predominantly 4–5-step reasoning, while visual content is led by figures and diagrams, with equations concentrated in science subsets.
- Domain and language balance: Russian business is the largest cell with 352 items (35.2%), while English business is the smallest with 180 items (18.0%); science cells contain 266 and 202 items.The science subsets are more evenly distributed than the business subsets.
- Reasoning depth: Reasoning-depth annotations cover 940 of 1,000 items, span 2–10+ steps, and peak at 4–5 steps with moderate positive skew.The step counts are treated as coarse descriptive metadata rather than an objective difficulty score because annotators may decompose tasks differently.
- Visual content types: Figures and diagrams are most prevalent across subsets; equations appear almost exclusively in science subsets, while code fragments are comparatively rare.This distribution reflects infographics in corporate reports and scientific papers, plus the mathematical character of the arXiv and CyberLeninka sources.
5 Experiments
BEAR-Bench experiments evaluate diverse MLLMs under a fixed zero-shot protocol, revealing substantial performance gaps across models, languages, resolutions, and document types. Error analysis and hallucination-detection experiments further show that failures are predominantly perceptual and that detector performance depends on access regime and response length.
- Evaluation setup: 16 MLLMs spanning open-weight and proprietary systems were evaluated zero-shot using only each image and raw question, with decoding temperature 0.6.Open-weight models ran locally on NVIDIA H100/L40 hardware, while proprietary systems were queried through OpenRouter.
- Main results: 75.4% and 75.1% were the highest overall accuracies, achieved by Qwen3.5-397B-A17B and Gemini 3.1 Pro, respectively, with strengths varying across subsets.Qwen3.5-397B-A17B led Science and Equations, whereas Gemini 3.1 Pro edged ahead on Business and English items; all models dropped from English to Russian.
- Effect of image resolution: 49.3%→45.0%→33.0%→12.6%→4.7% was Qwen3.5-9B’s accuracy as downsampling increased from the original resolution to c = 1.5, 2, 3, and 4.The pronounced degradation indicates that preserving fine-grained visual detail is critical for reasoning over text-dense professional documents.
- Failure analysis: Spatial misgrounding and OCR/visual-attribute errors together account for the majority of Gemini 3.1 Pro failures in the annotated subsample.The taxonomy was developed from 150 incorrect responses, then applied to a diverse subsample; the annotated distribution contained n = 62 responses and allowed multiple labels.
- Hallucination detection: SUQ achieved BalAcc 0.67–0.74 on all eight proprietary models but only BalAcc 0.60–0.67 on two open-weight models, while no detector won everywhere.Performance depended on access regime and response length; median responses were under 150 words for proprietary models and over 1,000 words for open-weight models, limiting last-token signals for earlier errors.
6 Conclusion
BEAR-Bench is a bilingual benchmark of 1,000 human-authored questions for multi-step reasoning over text-dense business and scientific documents. Evaluation shows substantial performance gaps, especially on Russian items, and imperfect reliability of hallucination detection methods.
- Benchmark contribution: BEAR-Bench contains 1,000 human-authored questions requiring context-grounded, multi-step reasoning over text-dense business and scientific documents.Items include figures, tables, charts, equations, and diagrams.
- Benchmark results: 75.4% is the highest overall accuracy across 16 proprietary and open-weight MLLMs, with every evaluated model performing worse on Russian items.The benchmark evaluates both proprietary and open-weight MLLMs.
- Error analysis: Spatial grounding is among the common failure types identified through error analysis.The supplied conclusion passage truncates the remainder of the listed failure types.
- Hallucination detection: Hallucination-detection performance varies by target model and output type: supervised probes lead on proprietary outputs, while an MLLM judge leads on verbose open-weight outputs.This comparison uses the model responses generated on BEAR-Bench.
- Hallucination detection: 0.74 is the best balanced accuracy for proprietary outputs versus 0.81 for open-weight models, indicating that incorrect responses are not always identified reliably.The values are reported for existing hallucination detection methods evaluated on the resulting model responses.
Limitations
BEAR-Bench is intentionally narrow: it evaluates single-page reasoning on English and Russian enterprise and academic documents, limiting assessment of multi-page, cross-document, and broader-domain generalization.
- Scope limitations: The benchmark excludes multi-page and cross-document reasoning because every question is scoped to a single rendered page image.It covers only English and Russian enterprise and academic documents from public disclosure and preprint sources, so findings may not transfer to other languages, domains, or private enterprise corpora.
- Statistical limitations: With 1,000 items and uneven language–domain cell sizes, subset estimates carry more variation.
A Error taxonomy · B Illustrative items from MWS Vision Bench · C Accuracy vs Reasoning Depth
The section defines five visual-document error categories, contrasts MWS Vision Bench with BEAR-Bench’s text-dense multi-step focus, and shows that accuracy-depth trends differ across model families. Frontier proprietary models appear perception-limited, while reasoning-tuned Qwen3.5 models remain stable and instruction-tuned Qwen3-VL models degrade on deeper items.
- A Error taxonomy: C1 covers spatial localization and object matching, including object relations, boundaries, labels, arrows, and links between blocks.Examples include selecting neighboring elements, misreading containment, and confusing label-to-object correspondence.
- A Error taxonomy: C2 covers counting and aggregation mistakes involving visual elements, including missed, extra, or incorrectly summed objects and values.The category includes points, circles, arrows, rows, columns, people, links, paths, labels, and table values.
- A Error taxonomy: C3–C5 cover misreading text or visual attributes, extracting chart values incorrectly, and making semantic, instruction-following, logical, or arithmetic errors.These include OCR issues, color or legend mistakes, incorrect scales or trends, misunderstood conditions, wrong formulas, assumptions, and incomplete answers.
- B Illustrative items from MWS Vision Bench: MWS Vision Bench mixes business scans with handwriting, receipts, and forms, whereas BEAR-Bench targets multi-step questions on text-dense scientific and business pages.A large share of MWS tasks involve OCR, grounding, and key-information extraction.
- C Accuracy vs Reasoning Depth: Accuracy across annotated reasoning-step counts is non-monotonic for several proprietary models, suggesting visual perception constrains knowledge-free tasks more than long-chain execution.The analysis is presented in Figure 6.
- C Accuracy vs Reasoning Depth: Qwen3.5 reasoning-tuned models maintain stable accuracy as annotated reasoning depth grows, unlike instruction-tuned Qwen3-VL models.This comparison covers four open-weight models and is shown in Figure 7.
- C Accuracy vs Reasoning Depth: Instruction-tuned Qwen3-VL models degrade on items requiring seven or more reasoning steps, while reasoning-tuned models incur little extra accuracy cost on longer chains.The passage attributes this contrast to reasoning-tuned models’ training to produce long reasoning chains.
D Additional data statistics: response length … G.4 Demographics
The appendix details model response lengths, hallucination-detector crossvalidation, the chain-of-thought prompt, and human annotation procedures, recruitment, ethics, and demographics. It emphasizes self-contained multi-hop question design and substantial differences in model answer length.
- D Additional data statistics: response length: Response lengths differ sharply: several API systems have median answers of 2–3 words, whereas reasoning-oriented open-weight models often exceed 1,000 words.Lengths are computed by whitespace-splitting each model’s stored answer field.
- E Hallucination Detection Metrics: The hallucination-detector appendix reports full 5-fold crossvalidation for uncertainty scores, ContextualLens, SUQ Probe, and an MLLM-as-a-judge baseline.It evaluates proxy-based detection for proprietary models and native-signal detection for open-weight models using AUROC, AUC-PR, and balanced accuracy.
- F Chain-of-Thought Prompt: The CoT evaluation prepends a system prompt to each image–question pair.The prompt is shown in Figure 8 and is the condition reported in Table 3.
- G.1 Instructions for annotators: Annotators were instructed to create complex multi-hop questions answerable solely from image content, without external expert knowledge.Examples of good and bad questions illustrated the task requirements.
- G.1 Instructions for annotators: Annotators were instructed to formulate answers as briefly as possible, such as a single number.This instruction concerned the answer format rather than the complexity of the questions.
- G.2 Recruitment & payment: Candidates qualified by generating 10 probe questions intended to elicit incorrect Gemini 3.1 Pro answers, with selection requiring more than 4 failures out of 10.Selected annotators were compensated at 4.4 times the Russian minimum wage.
- G.3 Ethics & Consent: No formal ethics review was required for the noninvasive annotation task, and all participants provided informed consent.The passage presents this as the study’s ethics and consent procedure.
- G.4 Demographics: All annotators were aged 22–25, held at least a technical bachelor’s degree, and reported English proficiency of CEFR B2 or higher; 60% were male and 40% female.These figures describe the composition of the annotation sample.
H Broader Impact, Data Use, and Compute Details
BEAR-Bench is intended as a research benchmark for multimodal reasoning and hallucination detection, not autonomous or high-stakes decision-making. Its public-source data may contain information from original records, while evaluation used specified hardware for open-weight models and an API for proprietary models.
- Potential risks: Errors from misreading text, tables, figures, or equations may produce incorrect calculations or unsupported conclusions, especially without human verification.Such failures could lead to incorrect analyses or decisions in financial, scientific, or other high-stakes workflows, and performance varies across languages.
- Intended use: BEAR-Bench is intended for research evaluation, not autonomous decision-making or deployment certification in high-stakes financial, legal, or scientific settings.It does not provide step-by-step rationale annotations and therefore does not directly support explicit chain-of-thought training or evaluation.
- Privacy and content: All source pages came from publicly accessible official disclosure, government, intergovernmental, or academic sources, without direct personal-data collection or additional content screening.Public documents may still contain names, affiliations, or other information from original records; the authors retain source attribution and recommend treating the benchmark as a research resource rather than information about individuals.
- Compute infrastructure: Open-weight models were evaluated on an internal server with two NVIDIA H100 GPUs and five NVIDIA L40 GPUs, while proprietary models were accessed through the OpenRouter API.The underlying hardware configuration and parameter counts are not publicly available for all proprietary models.