PaperlayerView exact revision on arXiv

Source-linked AI summary

SciTrue: Reliable Scientific Claim Validation with Frontier and Open Language Models at the NTCIR SciClaimEval Task

Qiming Bao, Neşet Özkan Tan, Siyuan Wang, Mark Gahegan

arXiv:2609.00654v1cs.AIcs.CL

TL;DR

SciClaimEval asks how reliably systems can verify scientific claims against tables and figures. SciTrue benchmarks eleven multimodal models and combines them with transparent fusion, routing, and a leak-free pair prior. It wins or ties across all four official leaderboard combinations, while its audit finds that many residual errors reflect benchmark artifacts rather than model weakness.

  • Problem

    SciClaimEval evaluates whether systems can verify scientific claims against their underlying table or figure evidence.

  • Method

    SciTrue benchmarks eleven frontier and open multimodal models under an honest per-sample protocol, combining score fusion, evidence-type routing, and a leak-free pair prior.

  • Results

    SciTrue placed first by a clear margin in three of four official blind-test categories and tied for first on the primary metric in the fourth.

  • Takeaways & Limitations

    The pairing structure provides the largest reported performance lever, while most residual errors are visually undetectable label swaps or dataset noise.

  • Takeaways & Limitations

    Position-based ordering can inflate development scores, and many remaining errors cannot be resolved from image evidence alone or reflect dataset label noise.

Abstract

from arXiv · show

We describe the SciTrue team's participation in both subtasks of the NTCIR-19 SciClaimEval task~\cite{sciclaimeval}, which asks systems to verify scientific claims against the tables and figures of a paper. Rather than tuning a single model, we benchmark eleven frontier and open multimodal models under one honest, per-sample protocol and combine them with light, transparent post-processing. On the official, blind test leaderboard (Section~\ref{sec:results}), SciTrue placed first by a clear margin in three of the four evidence-category/subtask combinations, and tied for first on the primary metric in the fourth. Three findings explain the result. First, strong instruction-tuned models are already competitive: Claude Opus~4.8 and Gemma-4-31B each exceed the strongest public baseline (o4-mini), and GPT-5.5 and Claude Fable~5 lead both subtasks (97.7 on Subtask~2). Second, the task's pairing structure is the largest lever: a \emph{leak-free pair prior} that recovers the Supported/Refuted pairing from the claim text alone (a visible field) and assigns Supported to the higher-confidence evidence raises Subtask-1 pair-accuracy from 72.2 to 93.5, far more than any model swap or ensemble weighting. Third, a case-by-case audit finds that most residual errors are visually-undetectable label-mapping swaps or dataset label noise, so measured accuracy understates the true ability and the fixable-by-modeling headroom is small. Controlled fine-tuning, distillation, and agentic consistency-checking support the same conclusions, and we document throughout a measurement leak---label information reaching a system through the packaging of the data rather than its content---in which the released file ordering encodes the label, including one instance that briefly misled our own pipeline.

Subtasks

The paper addresses two complementary subtasks: labeling claims and identifying supporting evidence. The listed authors include an equal-contribution designation and Mark Gahegan's University of Auckland affiliation.

  • Subtask 1 predicts whether a claim is Supported or Refuted, while Subtask 2 predicts which evidence image supports it.
  • The paper marks some authors as equal contributors and co-first authors.
  • Mark Gahegan is affiliated with the University of Auckland.

1 Introduction

SciClaimEval asks systems to verify scientific claims against tables and figures, while SciTrue studies model selection, fusion, and pairing structure under an honest protocol. Its contributions include broad benchmarking, leak correction, failure analysis, and consistency checking.

  • SciTrue frames claim verification against tables and figures as a practical multimodal fact-checking problem.
  • The team benchmarks eleven frontier and open multimodal models under one uniform, honest per-sample protocol.
  • A leak-free pair prior recovers claim pairs from text and lifts pair-accuracy to 93.5.
  • The paper reports measurement-leak correction, failure analysis, and an agentic consistency checker that partially transfers to small open VLMs through QLoRA.

2 Related Work

Prior work spans textual and scientific fact-checking, chart and table understanding, multimodal model families, and reliability-oriented fusion and adaptation methods. SciTrue situates its approach at their intersection.

  • Claim verification and fact-checking: Automated fact-checking includes open-domain verification, scientific claims against abstracts, and claims checked against structured tables.
  • Chart/table understanding and vision-language models: Chart and table understanding commonly involves reading axes, legends, panels, and trends, often after converting image evidence into structured form.
  • Chart/table understanding and vision-language models: The benchmarked model landscape includes instruction-tuned Gemma, Qwen-VL, GLM-V, GPT-4, Claude, and o4-mini families.
  • Reliability, fusion, and benchmark quality: Related reliability work examines grounding failures and weak abstract reasoning, while related methods include aggregation, reasoning-plus-acting, and low-rank adaptation.

3 Task and Data

SciClaimEval evaluates claim verification against original and minimally tampered tables or figures across three scientific domains. It measures both Supported/Refuted labeling and selection of the supporting image from a pair.

  • The dataset covers machine learning, natural language processing, and biomedicine, with tables or figures supplied as PNG evidence.
  • For tables, organizers additionally release structured LaTeX, HTML, and JSON data.
  • Subtask 1: Subtask 1 labels each claim as Supported or Refuted given one evidence item, with 747 development and 917 test claims.
  • Subtask 1: Each claim is paired with original Supported evidence and a minimally tampered Refuted copy, forming 352 development pairs plus supported-only singletons.
  • Subtask 2: Subtask 2 selects the supporting image from two evidence images, using 352 development and 436 test samples and accuracy scoring.

4 Methods

SciTrue combines eleven multimodal models with score fusion, evidence-type routing, and a leak-free pair prior under a uniform per-sample protocol. It excludes position-based label leakage from reported results and applies the same visible-field operations to both development and test data.

  • Pair prior: The pair prior matches identical claim text and breaks ties without using position, converting two absolute judgments into a relative ranking.All three main operations use fields released for both splits and avoid claim_id_pair.
  • Models and inference: Eleven frontier and open multimodal models receive the same prompt and produce support scores for scientific claim verification.The protocol avoids model-specific prompt tuning and uses native visual reasoning alongside released structured table data.
  • Honest per-sample evaluation: Position-based pair resolution is treated as a measurement leak because development row order encodes the Supported label but test ordering is not guaranteed.The ordering-dependent method is reported separately rather than as the headline result.
  • Ensembling and evidence-type routing: Score-level fusion sums sign-oriented support scores, while selecting the strongest models avoids noise from weaker systems.The ensemble is combined with evidence-type routing because model rankings differ between tables and figures.
  • Ensembling and evidence-type routing: Evidence-type routing uses Opus 4.8, Gemma-4-31B, GPT-5.5, and Fable 5 for tables, adding GLM-4.6V-Flash for figures.The routing field evi_type is released for both development and test splits, so the rule transfers unchanged to blind evaluation.

4.4 Legal pair prior

SciTrue exploits the dataset’s repeated claim text to recover hidden Supported/Refuted pairings, then uses evidence confidence to rank the pair without relying on file order. An order-sensitive comparison was retained only as a separate diagnostic because its apparent dev gain was biased.

  • Pair recovery: 352 of 352 development two-member groups are recovered exactly by grouping claims with identical text, matching the hidden pairing field.Each recovered group contains one Supported and one Refuted member.
  • Pair prior: The pair prior assigns Supported to the evidence member with higher confidence, turning two absolute judgments into one relative ranking.This uses the visible claim text to identify the pair rather than the hidden claim_id_pair field.
  • Position-bias caveat: A gated direct comparison appeared to reach 96.9 pair-acc on development, but swapping image order changed performance from 97% to 82%.The approximately 15-point gap indicates strong bias toward image 1, so the result was not used as the primary submission.
  • Agentic checking: The agentic checker retrieves four lexically and numerically overlapping paper paragraphs, then cross-checks image content against the paper excerpt for contradictions.The retrieved paragraphs are truncated to 700 characters each.
  • Distillation: QLoRA distillation trains Qwen3.5-9B and Gemma-4-31B on gold-matching checker traces using the claim, caption, retrieved excerpt, and evidence image.The traces encode what the image shows, what the paper says, consistency, and the verdict.
  • Fine-tuning evaluation: An honest QLoRA study partitions development data 80/20 while keeping both pair members in one fold to prevent leakage.The split contains 596 training and 151 held-out claims.

5 Experiments and Results

SciTrue’s experiments show that transparent pairing-aware post-processing matters more than broad model pooling, while newer frontier models strengthen both subtasks. Official blind-test results validate the approach, but audits expose measurement leaks and visually undetectable residual errors.

  • Official leaderboard: SciTrue placed first in three of four official leaderboard combinations and tied for first on Subtask-1 PNG pair-accuracy.It led Subtask-1 JSON at 98.4 versus 93.2, Subtask-1 TeX/HTML at 98.4 versus 97.7, and Subtask-2 PNG at 98.4 versus 98.2; PNG tied at 98.2.
  • Single models: GPT-5.5 and Claude Fable 5 achieved 97.4 and 97.7 accuracy on Subtask 2, respectively, while Fable 5 reached 74.4 pair-accuracy on Subtask 1.Fable 5 had the smallest table-versus-figure gap among tested models, at +1.8 points.
  • Ensembling and pair prior: 93.5 pair-accuracy came from the leak-free pair prior, versus 72.2 for score-weighted fusion and 73.0 for evidence-type routing.Pooling all eight available models reduced performance to 66.2, while the leaky position tie-break variant reached 94.9.
  • Ensembling and pair prior: 71.9% of pairs were already correct under honest prediction, and correcting only same-label cases recovered 94.6% pair-accuracy.A global threshold sweep improved performance by at most 0.5 points, from 71.9 to 72.4.
  • Subtask 2: Subtask 2’s relative-selection formulation makes Opus 4.8 strong alone at 92.9 accuracy, with three-model fusion adding only to 93.5.This supports using direct Opus predictions for Subtask 2 while reserving fusion for the more complementary Subtask 1 models.
  • Fine-tuning: QLoRA raised Qwen2.5-VL-7B from 77.8 to 84.7 pair-accuracy, but the tuned model remained below the 93.1 training-free ensemble-plus-prior.The authors characterize fine-tuning as a cheap route to competence, with remaining headroom in scale, error-diverse fusion, and pairing structure.
  • Measurement leak: Removing position-based tie-breaking lowered pair-accuracy from the leaky 94.9 to the reported headline value of 93.5.The authors treat row-order information as a measurement leak because it encodes the development labels without reading evidence.
  • Negative results and auditing: Retrieved context reduced accuracy from 92.2 to 84.3 because external text described the untampered evidence and masked manipulations inside the image.The same text became useful when used for consistency checking rather than direct claim answering.

6 Failure Analysis

The audit finds that most residual errors arise from visually undetectable label-mapping swaps or questionable dataset labels, while genuine misses on visible edits are fewer. Performance is strong on value and graph edits but weaker when manipulations rewire label-to-value mappings.

  • Most residual errors are visually-undetectable label-mapping swaps or dataset label noise rather than genuine model misses.Some gold labels are correct but impossible to infer from the image, while other Refuted labels do not actually falsify the claim.
  • A legend swap can leave the chart geometrically identical, so ensemble models assign both pair members the same label despite a genuine ceiling.The relabeling changes legend and axis semantics without moving plotted points.
  • 92–100% pair-accuracy is achieved on swapped or flipped graphs and altered cell values, but legend and category swaps fall to 80–89%.Label-mapping edits preserve internal visual coherence, leaving no image-only signal of tampering.
  • Figures trail tables because legends and category axes enable traceless tamper modes, whereas altered table values can be checked against structured data.The authors identify external consistency checking against structured data as a possible future remedy.

7 Discussion and Limitations

The discussion argues that exploiting pair structure matters more than scaling models, provided position-based leakage is removed. It also identifies benchmark artifacts as the main ceiling and states the pair prior’s test-set assumption.

  • +21 points from the leak-free pair prior dwarfs model swaps and fine-tuning gains once position leakage is removed.
  • The pair prior assumes that each test pair contains one Supported and one Refuted member, an assumption unverifiable without test labels.
  • Most apparent errors are visually-undetectable or mislabeled, leaving only a small fraction fixable through better modeling.

8 Conclusions

SciTrue combines multimodal models with transparent post-processing and achieves top official blind-test performance across the task’s categories. Its audit and controlled studies attribute remaining errors largely to benchmark ceilings and measurement artifacts.

  • SciTrue placed first by a clear margin in three of four official test categories and tied for first in the fourth, with 98.2 Subtask-1 PNG performance.
  • The system combines a frontier-plus-open ensemble, evidence-type routing, and a leak-free pair prior recovered from claim text alone.
  • 93.5 pair-accuracy was reached on development, rising to 96.2 after adding GPT-5.5 and Claude Fable 5 and fixing a pair-forcing bug.
  • Fine-tuning, failure auditing, and agentic checking support the conclusion that most residual errors reflect benchmark ceilings rather than model weakness.
Loading 2609.00654v1…