PaperlayerView exact revision on arXiv

Source-linked AI summary

SIN-Bench: Tracing Native Evidence Chains in Long-Context Multimodal Scientific Interleaved Literature

Yiming Ren, Junjie Wang, Yuxin Meng, Yihang Shi, Zhiqiang Lin, Ruihang Chu, Yiran Xu, Ziming Li, Yunfei Zhao, Zihan Wang, Yu Qiao, Ruiming Tang, Minghao Liu, Yujiu Yang

arXiv:2601.10108v1cs.CLcs.AIcs.MM

TL;DR

Long-form multimodal scientific evaluation lacks measures requiring native, cross-modal evidence-linked reasoning rather than answer matching alone. SIN-Bench operationalizes this need through interleaved evidence-chain tasks, finding grounding remains the primary bottleneck across evaluated MLLMs.

  • Problem

    Existing long-context evaluations often use synthetic layouts or answer accuracy, limiting assessment of native document dependencies and evidence-linked reasoning.

  • Method

    The paper introduces FITO, SIN-Data, SIN-Bench’s four progressive tasks, and No Evidence, No Score for evaluating and diagnosing interleaved evidence chains.

  • Results

    Grounding is the primary bottleneck: Gemini-3-pro achieves the best average overall score (0.566), while GPT-5 achieves the highest SIN-QA answer accuracy (0.767) but lags in evidence-aligned scores.

  • Takeaways & Limitations

    Correct answers do not necessarily provide traceable support, so explicit evidence chains expose a grounding gap in current multimodal language models.

  • Takeaways & Limitations

    Evaluation breadth is constrained because few multimodal models support long-context interleaved inputs, excluding domain-specialized experts.

Abstract

from arXiv · show

Evaluating whether multimodal large language models truly understand long-form scientific papers remains challenging: answer-only metrics and synthetic "Needle-In-A-Haystack" tests often reward answer matching without requiring a causal, evidence-linked reasoning trace in the document. We propose the "Fish-in-the-Ocean" (FITO) paradigm, which requires models to construct explicit cross-modal evidence chains within native scientific documents. To operationalize FITO, we build SIN-Data, a scientific interleaved corpus that preserves the native interleaving of text and figures. On top of it, we construct SIN-Bench with four progressive tasks covering evidence discovery (SIN-Find), hypothesis verification (SIN-Verify), grounded QA (SIN-QA), and evidence-anchored synthesis (SIN-Summary). We further introduce "No Evidence, No Score", scoring predictions when grounded to verifiable anchors and diagnosing evidence quality via matching, relevance, and logic. Experiments on eight MLLMs show that grounding is the primary bottleneck: Gemini-3-pro achieves the best average overall score (0.573), while GPT-5 attains the highest SIN-QA answer accuracy (0.767) but underperforms on evidence-aligned overall scores, exposing a gap between correctness and traceable support.

1 Introduction

The paper argues that existing long-context multimodal evaluations fail to distinguish genuine scientific comprehension from answer inference, especially when they rely on synthetic retrieval tasks and answer accuracy alone. It introduces FITO, SIN-Data, and SIN-Bench to evaluate cross-modal evidence chains through progressive tasks and evidence-grounded scoring.

  • Motivation: Existing evaluations can mistake parametric inference for comprehension of complex document logic.This problem is pronounced in lengthy, symbol-dense multimodal scientific documents.
  • Limitations of Existing Evaluation: NIAH tests artificial, semantically isolated needles rather than native literature dependencies, such as linking figure interpretation to methodology.Answer-only metrics also obscure the reasoning process behind correct responses.
  • FITO Paradigm: FITO requires models to aggregate interconnected knowledge units across a native document ecosystem into explicit cross-modal evidence chains.The paradigm shifts evaluation from answer correctness toward explicated evidence retrieval.
  • Benchmark Contributions: SIN-Bench operationalizes FITO through SIN-Data and four tasks spanning evidence discovery, hypothesis verification, grounded QA, and evidence-anchored synthesis.SIN-Data preserves the interleaved structure of long-form scientific documents.
  • Empirical Findings: 0.566 was Gemini-3-pro’s best average overall score, while 0.767 was GPT-5’s highest SIN-QA answer accuracy despite weaker evidence-aligned overall scores.These results identify evidence grounding as the primary bottleneck and motivate No Evidence, No Score with Logic, Matching, and Relevance dimensions.

2 Related Work

Prior long-context MLLM benchmarks test retrieval with synthetic text or image needles but largely miss document-spanning cross-modal reasoning. SIN-Bench addresses this gap through Fish-In-The-Ocean, requiring coherent evidence chains in interconnected scientific literature.

  • MLLM Long-Context Understanding Tasks: MM-NIAH and MMLongCite insert synthetic text or image needles into long contexts to probe MLLM retrieval boundaries.These paradigms primarily assess recall limits through Needle-In-A-Haystack setups.
  • MLLM Long-Context Understanding Tasks: Existing benchmarks emphasize single-point extraction or page-layout perception while overlooking long-range cross-modal reasoning across documents.The limitation motivates a paradigm focused on linked evidence rather than isolated retrieval.
  • Evaluating Scientific Reasoning in MLLMs: Fish-In-The-Ocean requires models to construct coherent cross-modal evidence chains within highly interconnected scientific literature.SIN-Bench operationalizes this paradigm by simulating a complete scientific research workflow and evaluating deep reasoning.

3 The Fish-in-the-Ocean Paradigm

FITO evaluates comprehension of inherently complex, long-form multimodal documents by tracing evidence across sections and modalities rather than relying on synthetic noise. Its objective shifts from answer-only prediction to process-oriented reasoning that explicitly models and validates an evidence chain before deriving the answer.

  • FITO paradigm: FITO targets inherently complex, long-form multimodal documents instead of synthetic-noise Needle-in-a-Haystack settings.The paradigm preserves native document complexity rather than artificially injecting noise.
  • FITO paradigm: Evidence chains span sections and modalities, with long-range dependencies linking methods to results and conclusions.The paradigm requires connecting evidence across text, figures, and tables, including disjoint document sections.
  • Evaluation objective: FITO models the joint probability of an answer and its supporting evidence chain rather than maximizing QA likelihood alone.The evidence chain is represented as a latent variable E alongside answer A, document D, and query Q.
  • Evaluation objective: A valid system must explicitly instantiate and validate the sufficiency of E before deriving A.This changes evaluation from result-oriented prediction to process-oriented reasoning.

4 SIN-Bench: Evidence-based Evaluation

SIN-Bench operationalizes the FITO paradigm as a long-context multimodal benchmark built on semantics-aware scientific interleaving and four evidence-chain tasks. Its “No Evidence, No Score” evaluation jointly tests evidence quality, answer correctness, and logical support.

  • SIN-Data Infrastructure: SIN-Data reconstructs scientific documents into a linear interleaved stream that preserves logical coupling between text and visual evidence.The format is designed to move beyond PDF layout or raw LaTeX representations.
  • SIN-Data Infrastructure: 4,000 high-quality documents span 12 top-level disciplines and over 80 subfields after visual-density and citation-link quality filtering.Filtering removes samples with sparse visual context or broken citation links, while taxonomy-based annotation preserves domain diversity.
  • Benchmark Tasks: SIN-Bench organizes professional reading into four hierarchical tasks: Discovery, Verification, Question Answering, and Synthesis.Each instance uses a single document with a query, answer, and interleaved evidence chain containing visual and textual anchors.
  • Benchmark Tasks: SIN-Verify tests whether evidence sufficiently supports a conclusion, whereas SIN-QA and SIN-Summary require generated answers or syntheses anchored to verifiable evidence.Negative perturbations in SIN-Verify omit premises or mismatch conditions; SIN-Summary additionally evaluates holistic, long-range document integration.
  • Evaluation Protocol: “No Evidence, No Score” evaluates generated evidence chains for SIN-Find, SIN-QA, and SIN-Summary, adds answer assessment for SIN-QA, and uses accuracy for SIN-Verify.The protocol addresses cases where models obtain correct answers through incorrect reasoning paths.
  • Evaluation Protocol: MRL metrics assess evidence-chain Matching, Relevance, and Logic at the adjacent visual-text anchor-pair level.Matching uses LLM-based semantic adjudication, while Relevance uses F1 and Logic uses Kendall–Tau similarity.

5 Experiments

Experiments across eight MLLMs show that evidence grounding, native interleaving, and explicit evidence chains substantially shape performance beyond answer accuracy. Models remain vulnerable to near-miss evidence, long-context instability, and missing or spurious reasoning links.

  • Overall Performance: 8 MLLMs are evaluated, including 5 proprietary and 3 open-weight models spanning dense and MoE architectures.The proprietary models are Gemini-3-pro-preview, Gemini-2.5-pro, GPT-5, Grok 4, and Claude-sonnet-4.5; the open-weight family is Qwen3-VL.
  • Overall Performance: 0.566: Gemini-3-Pro-Preview attains the highest average overall score, while GPT-5 leads SIN-QA AnsAcc at 0.767 but trails Gemini-3-pro overall.Gemini-3-pro scores 0.399 and 0.567 on SIN-Find and SIN-QA overall, versus GPT-5’s 0.378 and 0.522, exposing a gap between answer correctness and grounding.
  • Task-Level Results: Qwen3-VL-8B surpasses Qwen3-VL-30B-A3B across nearly all metrics, while Claude-sonnet-4.5 leads SIN-Find overall at 0.460 and Gemini-2.5-pro leads logic at 0.536.The results favor reasoning-oriented fine-tuning over raw parameter count or MoE architecture; multiple open-source models also receive invalid scores for evidence-formatting failures.
  • Task-Level Results: 1.000 → 0.208: GPT-5’s SIN-Verify accuracy drops sharply with hard negatives, while Qwen3-VL-8B falls from 1.000 → 0.044.Standard-setting accuracy is tightly clustered at Acc=0.667–0.697, indicating limited logical discrimination for near-miss evidence.
  • Importance of Interleaved Input: +0.102 and +0.129: preserving native interleaving improves Gemini-3-pro’s SIN-QA and SIN-Summary over images→text, with interleaved input outperforming text-only captions and image-only pages.Captions retain coarse semantics but lose fine-grained empirical evidence, whereas raw visuals become most useful when locally grounded by adjacent text.
  • Evidence Chains and Error Analysis: 0.694 to 0.726: enforcing evidence-chain generation improves Gemini-3-pro’s SIN-QA performance, while long contexts expose tail-end degradation and reasoning failures.GPT-5 degrades at the SIN-QA tail, Qwen3-VL-2B collapses near zero in SIN-Summary, and errors include omitted prerequisites and irrelevant shotgun citations.

6 Conclusion

The work introduces the Fish-in-the-Ocean paradigm and SIN-Bench for long-context, interleaved scientific reasoning. By requiring explicit evidence chains, it shifts multimodal evaluation beyond response accuracy toward traceable, logically grounded reasoning.

  • Contributions: The study introduces the Fish-in-the-Ocean paradigm and establishes SIN-Bench for long-context, interleaved scientific reasoning.SIN-Bench is described as a comprehensive benchmark tailored to this setting.
  • Grounding: The No Evidence, No Score mechanism requires explicit evidence chains and exposes a critical grounding gap in state-of-the-art MLLMs.The conclusion specifically notes that models like GPT-5 often rely on parametric…
  • Evaluation: The study advocates evaluating multimodal systems beyond response accuracy by rigorously assessing reasoning-process traceability and logic.This proposed shift centers evaluation on whether reasoning is traceable and logically supported.

Limitations

The evaluation is limited by the scarcity of models supporting long-context interleaved inputs and by strict filtering that can exclude imperfect documents. Open-source code and configurable filtering rules provide paths to integrate emerging models and balance data purity against scale.

  • Model coverage: Limited support for long-context interleaved inputs restricts evaluation beyond generalist models such as GPT-5 and Gemini-3-Pro.The study open-sources data-construction and evaluation code to help integrate emerging multimodal models.
  • Data filtering: Strict filtering minimizes parsing noise but can exclude documents with minor imperfections, creating a trade-off in data utilization.Configurable thresholds and voting mechanisms let users balance document purity and scale.

Ethical considerations … B On the Principle of “No Evidence, No Score”

The paper builds SIN-Data and SIN-Bench around native text–figure evidence chains, then enforces “No Evidence, No Score” so answer correctness cannot substitute for verifiable grounding. Its construction combines heterogeneous-source parsing, quality filtering, collaborative validation, and ethical safeguards against misuse.

  • Ethical considerations; A.1 SIN-Data Construction Pipeline: Open-access arXiv and PMC documents are standardized into interleaved scientific records while preserving text–figure dependencies and citation-linked evidence chains.The pipeline bridges LaTeX-based Physics and Computer Science sources with XML-based Biomedicine archives, using unified formatting and controllable quality signals.
  • Ethical considerations; B On the Principle of “No Evidence, No Score”: The paper acknowledges that evidence chains could be reverse-engineered for realistic fabrication and urges shifting research toward fraud detection and transparent technical disclosure.The metric suite captures irrelevant-anchor over-generation through Relevance, missing evidence through recall, and cherry-picked or mis-ordered support through Logic.
  • A.1.1 Stage 1: Element Parsing: SIN-Data parsing separates content from presentation, extracts text and visuals from arXiv and PMC streams, and re-anchors images to their cited textual contexts.ArXiv uses Engrafo and NOUGAT with DOM-based visual matching, while PMC uses a customized s2orc-doc2json parser for JATS XML.
  • A.1.2 Stage 2: Semantic-First Formatting & Signal Extraction: Citation-driven injection places each image placeholder immediately before its first-citing paragraph, while token, visual-density, and interconnectivity signals characterize document structure.Recorded signals include total_tokens, avg_segment_length, image_count, image_ratio, and interleave_segments.
  • A.1.3 Stage 3: Quality Filtering & Taxonomy Alignment: Filtering for dense multimodal interconnectivity reduced 50,000 source packages to 4,000 documents spanning 10+ disciplines and 80+ subfields.The curation removes sparse visual contexts, broken reference chains, and extreme lengths while retaining 32k–1M-token documents.
  • A.2 SIN-Bench Construction Pipeline: SIN-Bench uses human-curated seeds, multi-MLLM synthesis, cross-model auditing, human review, and bootstrapping to produce scalable, structurally coupled samples across four evidence-centered tasks.Validation scores Question Validity, Answer Correctness, and Evidence Consistency on 1–5 scales, retaining samples approved by the majority with at least 4 on every dimension.
  • A.2 SIN-Bench Construction Pipeline: The released golden-sample volume is limited by human-auditing costs, although the semi-automated collaborative pipeline is designed to scale with sufficient computational and human resources.The paper states that sample quantity itself presents no inherent pipeline bottleneck under those conditions.
  • B On the Principle of “No Evidence, No Score”: “No Evidence, No Score” makes evidence quality a first-class scoring component: answer-only correctness cannot produce a high overall score without recoverable document anchors.Minor citation-format variations are accepted when anchors remain matchable; missing or invalid anchors receive zero evidence credit, with Matching and F1 collapsing to zero when no anchor matches.

B.1 General Statistics · B.2 Samples from SIN-Bench · B.4 Discipline Taxonomy List

SIN-Bench combines long, predominantly text-driven interleaved instances with substantial visual content, curated evidence-chain examples, and stable source reconstruction. Its discipline coverage uses a three-level taxonomy spanning 12 major disciplines, 35 domains, and 84 subfields.

  • B.1 General Statistics: SIN-Verify instances are roughly 2 times longer on average than SIN-Find and SIN-QA instances, indicating greater contextual and reasoning load.SIN-Find and SIN-QA have comparable length ranges.
  • B.1 General Statistics: 15% of total tokens are image-related on average, with 6.6 images per instance, requiring visual grounding while remaining predominantly text-driven.The benchmark also reports bold, italic, and title-token statistics as structural formatting cues.
  • B.2 Samples from SIN-Bench: SIN-Bench examples show cropped views of the document, question, annotated evidence regions, and gold outputs for SIN-Find, SIN-QA, SIN-Verify, and SIN-Summary.Figures 8 and 9 provide one curated “golden” instance per task.
  • B.2 Samples from SIN-Bench: The benchmark operates on full interleaved Markdown source files with image assets released using stable identifiers for exact reconstruction.Cropped screenshots are used for visualization because rendering full scientific documents would be space-inefficient and dilute task-relevant signal.
  • B.4 Discipline Taxonomy List: SIN-Bench uses a three-level hierarchical discipline taxonomy referencing arXiv subject classification to support multimodal reasoning evaluation across scientific disciplines.The taxonomy was developed through model-assisted preliminary classification followed by expert correction and subfield refinement.
  • B.4 Discipline Taxonomy List: The structured distribution ensures that SIN-Bench covers a wide spectrum of knowledge across its scientific taxonomy.The taxonomy includes detailed subfields spanning machine learning, computer vision, planetary science, quantum physics, computational social science, and Bayesian statistics.

C Comparisons with Existing Benchmarks … D.2 Models Used in the Benchmark and Experiments

SIN-Bench differs from prior long-document and scientific multimodal benchmarks by requiring evidence-based comprehension over native interleaved documents and explicitly evaluating evidence quality. Its experiments use reproducible model settings, Gemini-3-pro as the primary diagnostic testbed, a text-only modality ablation, and rubric-guided LLM evaluation.

  • C Comparisons with Existing Benchmarks: Table 6 compares SIN-Bench with long-document and scientific multimodal benchmarks using explicit-evidence and evidence-score indicators.“Ev.” denotes whether outputs require an explicit evidence chain, while “Ev.-score” denotes explicit evidence-quality evaluation.
  • C Comparisons with Existing Benchmarks: SIN-Bench targets scientific comprehension in native interleaved image-text documents through four progressive Find, Verify, QA, and Summary tasks.These tasks follow a discovery→verification→synthesis workflow.
  • C Comparisons with Existing Benchmarks: Evidence-quality scores for Matching, Relevance, and Logic diagnose failures beyond answer-only correctness.The framework distinguishes missing prerequisites from spurious citations.
  • C Comparisons with Existing Benchmarks: SIN-Bench is distilled from 50k collected papers through multi-stage cleaning and filtering designed to preserve high-quality scientific documents.The benchmark contains fewer evaluated documents than some large-scale resources.
  • D.1 Baselines: All evaluations, including open-weight models, use temperature 0, with proprietary model version identifiers reported for reproducibility.The listed proprietary baselines include Gemini-3-pro, Gemini-2.5-pro, and Claude-sonnet-4.5.
  • D.2 Models Used in the Benchmark and Experiments: Gemini-3-pro is the primary diagnostic model because it achieves the best average SIN-Bench performance across tasks under constrained ablation budgets.The study evaluates both proprietary and open-source models.
  • D.2 Models Used in the Benchmark and Experiments: The modality study replaces each image slot with a Qwen3-VL-8B-Instruct figure description to preserve document structure while removing direct visual signals.This constructs a “pure text” setting for isolating raw visual-input effects.
  • D.2 Models Used in the Benchmark and Experiments: Semantic evaluation uses rubric-guided LLM judges rather than n-gram overlap metrics to accommodate paraphrases and diverse valid answers.This reduces penalties for surface-form variation and false negatives from under-specified reference strings.

D.3 Judges in Metrics · E Additional Experiments and Result Analysis

The study uses Qwen3-8B as an automated judge and validates it against human expert evaluations across all task instances. The automated scores strongly align with the human consensus, supporting the reliability of the matching and question-accuracy metrics.

  • D.3 Judges in Metrics: Qwen3-8B serves as the automated judge for matching and question-accuracy metrics.The choice was evaluated across all instances for each task.
  • D.3 Judges in Metrics: A dual-scoring protocol compares automated judgments with evaluations from human experts.The protocol was applied to all samples in the consistency study.
  • D.3 Judges in Metrics: Qwen3-8B independently evaluates model predictions, while 2–3 domain-expert Master’s and Ph.D. students perform manual evaluation.The human evaluators have relevant domain expertise.
  • D.3 Judges in Metrics: Majority voting aggregates human ratings into the evaluation gold standard.Automated scores are compared against this human consensus.
  • D.3 Judges in Metrics: The study computes correlations between automated scores and the human consensus.This directly tests alignment between the automated judge and expert evaluation.
  • D.3 Judges in Metrics: Avg Pearson r = 0.825 indicates strong alignment between the automated judge and the human gold standard.Table 7 reports that all results are statistically significant with p < 0.001.

E.1 Discipline-level Performance Analysis … F Case Study

SIN-Bench performance varies by discipline, verification difficulty, and document length: mathematics is hardest, hard-negative verification exposes grounding failures, and longer multimodal contexts increase instance-level variance rather than uniformly degrading performance. Case studies further show that unsupported concept injection and method-inconsistent statistical interpretation produce distinct SIN-QA grounding failures.

  • E.1 Discipline-level Performance Analysis: Economics & Finance and Medicine & Health Sciences are strongest at avg. ≈0.66 and 0.65, while Mathematics is hardest at avg. ≈0.31.The discipline differences indicate that SIN-Bench captures meaningful variation in evidence-grounded scientific comprehension.
  • E.1 Discipline-level Performance Analysis: Gemini-3-pro leads Astronomy & Astrophysics and Medicine & Health Sciences by +0.01 and +0.05 over GPT-5, whereas GPT-5 leads Mathematics by +0.10 over Gemini-3-pro.Gemini-3-pro’s advantages align with visually intensive or layout-heavy evidence, while GPT-5 performs better in more formal or quantitative disciplines.
  • E.2 An Analysis of Near-Miss Evidence in SIN-Verify: SIN-Verify uses easy irrelevant-evidence negatives and hard near-miss negatives whose answers are correct but whose evidence chains fail cross-validation agreement.The evaluation samples 24 instances for each setting.
  • E.2 An Analysis of Near-Miss Evidence in SIN-Verify: MLLMs nearly saturate on easy negatives but accuracy drops sharply on hard negatives, making fine-grained cross-modal evidence sufficiency substantially harder than coarse mismatch detection.Verification requires checking numbers, conditions, and figure–text alignment while avoiding answer-driven rationalization.
  • E.3 Impact of Document Length on Performance Stability.: Across roughly 1k–19k text tokens, Gemini-3-pro and GPT-5 retain substantial mid-to-high score mass without systematic collapse, although GPT-5 drops more as prompts lengthen.For strong models, text length alone is unlikely to be the primary bottleneck; selective evidence identification and fine-grained grounding remain more plausible residual error sources.
  • E.3 Impact of Document Length on Performance Stability.: Qwen3-VL-2B remains concentrated in low-score regions across tasks, indicating a broader gap in robust evidence retrieval and cross-modal grounding rather than merely long-context handling.Longer total prompts can add moderate-score mass for SIN-Find and SIN-QA when salient visual cues or near-surface matching suffice, but not for document-level SIN-Summary.
  • E.3 Impact of Document Length on Performance Stability.: Up to ∼87k text+image tokens, total-length distributions become clustered because visual tokenization reflects figure count, resolution, and model-specific encoding, so this axis requires cautious interpretation.Gemini-3-pro remains comparatively smooth, while GPT-5 bifurcates between high- and low-score modes as longer inputs mix helpful evidence with distractors and resolution-sensitive cues.
  • F Case Study: SIN-QA case studies identify Spurious Reasoning through unsupported concept injection and Information Deficiency through distorted statistical interpretation, including oversimplified standard-deviation reasoning.The latter produces quantitative-looking but method-inconsistent conclusions, while the former induces semantic drift by shifting the explanatory frame.
Loading 2601.10108v1…