SciReC: Diagnostic Evaluation of Multimodal, Multi-Turn Relational Reasoning with Adaptive Interaction

• Relational reasoning supports logical thinking and problem-solving in novel scenarios, while scientific expertise depends on interconnected relational concepts.Source: §1 · p. 1
• Prior conversation benchmarks use static inputs and one-sided interactions, motivating double-sided, model-adaptive evaluation of multimodal inputs and long-context dialogue.Predefined scenarios may not reflect real-world dialogue, where context shifts in response to both parties.Source: §1 · pp. 1–2
• Because visual perception and complete relational information are prerequisites for reliable multimodal inference, evaluations must diagnose atomic abilities and conditions that affect performance.Source: §1 · p. 2
• SciReC evaluates eight relational categories in university-level academic contexts through adaptive multimodal dialogue and diagnoses visual understanding, knowledge gaps, and memory retention.Conversation flow responds to model outputs, with flexibility in the number of images and turns.Source: §1 · p. 2
• Up to 73% accuracy for proprietary models and 56% for open-source models were achieved on SciReC; DMRA identifies relational reasoning as the primary failure factor, followed by memory.Source: §1 · p. 2

2.1 Multimodal Multi-turn Datasets

• Prior multimodal-dialog benchmarks vary in task and domain, leaving relational reasoning across scientific and social-science conversations untested; SciReC fills this gap.Existing inter-image benchmarks focus on daily-life contexts and lack diverse relational types and complex scientific concepts, while science-domain datasets do not evaluate relational reasoning.Source: §2.1 · pp. 2–3

2.2 Evaluation Protocols and Diagnostic Error Analysis

• Unlike fixed-sequence evaluations, SciReC selects Knowledge Gap, Visual Perception, or Memory Retrieval questions from a fixed pool based on model-performance thresholds.This creates a conditionally adaptive interaction flow in which subsequent context depends on the model’s performance.Source: §2.2 · p. 3
• SciReC’s deficit-based diagnostics quantitatively compare component deficits to explore potential failure sources, focusing on the weakest-performing component.The paper contrasts this systematic decomposition with prior analyses that identify failure patterns but do not attribute failures to underlying causes.Source: §2.2 · p. 3

3.1 Benchmark Overview

• 189 multimodal conversations contain 656 relational questions, alongside knowledge, visual-understanding, memory-retrieval, and memory-validation questions spanning eight relation types.Relational questions require information from multiple images; the other question types focus on a single image.Source: §3.1 · pp. 3–4

3.2 Benchmark Construction

• The benchmark draws on 12 college-level textbooks across eight domains; GPT-5.4 extracts relations from textbook text without image access, and humans review relational questions for accuracy, relevance, and visual grounding.Unclear or inconsistent relational items are removed after manual review against the figures, ground truths, and source text.Source: §3.2 · p. 4
• Relational and visual questions use templates to limit answer leakage; memory-validation questions paraphrase memory questions and add images to test whether visual context improves performance.Knowledge-based questions use structures and contexts extracted directly from textbook text, while generated questions rely exclusively on that text.Source: §3.2 · p. 4
• A knowledge question precedes each relation question, and scores below 7/10 trigger memory retrieval and, if needed, image-based memory validation to distinguish retrieval failure from knowledge or perception errors.Later knowledge and visual questions are skipped if already covered, but remain in the diagnostic analysis; question selection depends on model performance.Source: §3.2 · p. 5

3.3 Evaluation

• Responses are scored from 1–10, with 7 marking correctness; Claude 4.6 was selected to score answers after evaluation of Claude 4.6 and GPT 5.4 showed stronger alignment with human reviews for Claude.The rubrics cover relational and other image-specific tasks, with scores of 1–3 incorrect, 4–6 partial, 7–8 mostly correct, and 9–10 complete and accurate.Source: §3.3 · pp. 5–6

4 Deficit-Based Multimodal Relational Analysis (DMRA)

• DMRA separates figure-level deficits in memory, knowledge, and visual encoding from cross-figure integration and abstraction errors, using a success threshold of T = 7.For each figure, memory, knowledge, visual, and memory-validation scores are used to calculate upstream deficits.Source: §4 · p. 6
• Memory-validation gaps dynamically adjust task weights: DMRA starts with equal base weights of 0.33, uses λ = 0.2, and applies τ = 0.1 when memory and validation scores match.The memory weight is updated based on its base weight, validation gap, and bias; matching scores slightly reduce it because images did not improve performance.Source: §4 · p. 6
• DMRA weights each figure by its smoothed share of total deficits, then combines figure weight, task weight, and task deficit to aggregate upstream causes across figures.A smoothing term λfig = 0.5 ensures balanced, non-zero figure contributions.Source: §4 · p. 6
• The relational error is the portion of total failure unexplained by upstream deficits; upstream contributions are capped at total failure, and normalized scores quantify each task’s share.When upstream deficits equal or exceed total failure, the outcome is attributed to upstream limitations rather than relational error.Source: §4 · pp. 6–7

5.1 Main Results

• 73.78% overall accuracy makes Claude 4.6 the top model, ahead of GPT-5.4 at 68% and Qwen-3.5 at 56.25%.Gemma-3 and Mistral3 score 50.77% and 48.78%, while other open-source models score below 24%, indicating distinct performance tiers.Source: Table 1 · p. 7, §5.1 · p. 7
• Spatial reasoning is a bottleneck for many open-source models, while proprietary models show weaknesses in sequential and hierarchical relations.Claude 4.6 scores 64% on sequential relations, GPT-5.4 scores 55% on hierarchical relations, and Gemma-3 scores 41% on spatial relations; Qwen3-5 reaches 77% on analogical relations.Source: §5.1 · p. 7

5.2 Performance Across Domains

• Most models struggle most in Astronomy and perform best in Psychology, while Claude 4.6 leads across domains except Economics and Behavioral Neuroscience.GPT-5.4 outperforms Claude 4.6 in those exceptions at approximately 62% and 78%, respectively. SciReC’s relation mix also varies by domain: comparative relations dominate, analogies occur mostly in Physics and Astronomy, and spatial relations predominantly derive from Astronomy.Source: §5.2 · p. 8, Figure 4 · p. 8

5.3 Causal Analysis of Model Errors

• Memory is a leading secondary cause for most models at 37–46%, but knowledge gaps rank second for InternVL3 and MiniCPM-V-4.5, reaching up to 40%.For Claude 4.6, the second-factor knowledge gap is 13%; for Qwen 3.5 it is 25%, while GPT-5.4 has a balanced 19% impact from visual and knowledge causes.Source: §5.3 · pp. 8–9
• Relational reasoning is the primary first cause of failure for all models except MiniCPM-V-4.5; high-performing models show 66–71% relational errors.Relational cause is 61% for Mistral3 and 51% for InternVL3 and Gemma-3; MiniCPM-V-4.5 instead has 39% memory, 30% relational, and 23% knowledge-gap first causes.Source: §5.3 · p. 8, Figure 5 · p. 9
• Difficulty retrieving and integrating chat-history information creates memory bottlenecks that can accumulate across agent interactions and cascade into incorrect reasoning.The limitation persists even in strong models that address knowledge-gap and visual-perception errors.Source: §5.3 · p. 9

6 Conclusion

• Relational reasoning accounts for 50%–71% of incorrect responses, followed by memory at 37–46%; knowledge and visual gaps also contribute, especially for most open-source models.Knowledge gaps and limited visual understanding are less prominent in proprietary models.Source: §6 · p. 9
• Proprietary models score highest on structural relations (78–82%) and lowest on hierarchical (55–70%) and sequential (64%) relations, while most open-source models score 33–41% on spatial relations.Astronomy is the most challenging domain, while Psychology yields the highest performance.Source: §6 · p. 9
• SciReC benchmarks MLLMs’ relational reasoning in domain-specific academic dialogue, and DMRA uses a two-stage deficit-based approach to quantify failure causes across memory, knowledge, and visual reasoning.The benchmark includes upstream tasks covering memory, knowledge, and visual reasoning.Source: §6 · p. 9
• The benchmark excludes daily-life scenarios, generation and evaluation require paid API access keys, and extending DMRA beyond three error factors requires reparameterization because softmax weighting is dimension-sensitive.SciReC data comes from OpenStax textbooks under CC BY-NC-SA 4.0 and is released under the same license.Source: §6 · p. 9

A Dataset Details

• SciReC contains 656 relations across eight subject categories in 189 conversations, with flexible dialogue lengths and a minimum of three turns.The minimum-length dialogue contains caption questions and one relational question.Source: A Dataset Details · p. 14
• Biology has the most dialogues at 43 (22.6%), while Psychology has the fewest at 5 (3.2%), followed by Calculus at 13.Organic Chemistry and Chemistry together have 36 dialogues; Physics and Astronomy have 29 and 26, while Behavioral Neuroscience and Economics have 19 and 17.Source: Figure 6 · p. 14, A Dataset Details · p. 14
• Open-source model evaluations used an A100 GPU, approximately 15–18 hours per model, and a 4096-token model context length.The paper states that SciReC data and code are available online but provides no URLs in this passage.Source: A Dataset Details · p. 14
• The benchmark pipeline extracts chapters, images, relations, and ground truths from textbooks, then generates and filters question-answer pairs through human-rated quality checks.Human raters assess relational category match, accuracy, and visual grounding across multiple rounds.Source: Figure 7 · p. 14, A Dataset Details · p. 14

Societal Impacts

• SciReC aims to support more reliable educational AI and transparent diagnosis of reasoning failures, but tutoring pipelines require human oversight to avoid misleading information.The authors also describe diagnostic results as a roadmap for models.Source: A Dataset Details · p. 14

LLM Usage

• LLMs were used for grammar editing, data processing and filtering, question generation, baseline-response evaluation, coding and debugging, technical understanding, and implementation.Generated questions included relational, knowledge-based, visual, and memory-retention types.Source: A Dataset Details · p. 14

B Evaluation Model Selection

• 1.68 points was the average scoring difference between GPT-5.4 and Claude 4.6 on relational questions, compared with around 1 point on knowledge, memory validation, and memory questions.GPT-5.4 tended to score relational answers higher, often above the threshold, and avoided the 4–6 range on memory and validation questions.Source: B Evaluation Model Selection · p. 15
• The models were compared on 199 Qwen-3.5 responses to Physics questions, scored online with prompt constraints for conversational flow and answer quality.The responses included 82 knowledge-based, 63 visual, 22 relational and memory-related, and 10 memory-validation cases; question counts varied because the flow depended on model performance.Source: B Evaluation Model Selection · p. 15
• Human review selected Claude 4.6 because it assessed relational correctness and the concept integration needed to explain relationships, unlike GPT-5.4’s higher relational scores despite missing context.Annotators reviewed 29 conflicting cases, including the question, ground truth, images, scores, and justifications.Source: B Evaluation Model Selection · p. 15

• Each figure’s dialog uses one relational, two memory, one validation, three knowledge, and three visual questions; scores below 7 are deficits, and Figure 1 skips validation when memory exceeds threshold.The relational question score is 2.0.Source: C Example of DMRA Calculations · pp. 15–16
• Deficits use max(0, 7 - score): Figure 1 deficits were memory 0, knowledge 0.666, visual 1.333; Figure 2 memory 5, knowledge 4.666, visual 1.333.Source: C Example of DMRA Calculations · p. 16
• Default task weights are 0.33, then softmax adjustment gives Figure 2 memory 0.3792, knowledge 0.3104, and visual 0.3104 after a 1.0 validation gap.Figure 1 has no validation gap and retains weights of 0.33 for memory, knowledge, and visual.Source: C Example of DMRA Calculations · pp. 16–17
• With λ = 0.5, Figure 2 supplied 95% of upstream deficits; upstream total was 3.206 versus a relational residual of 1.793.Figure 2’s final failure shares were memory 31.14%, knowledge 23.8%, and visual 6.8%; Figure 1’s were 0%, 0.8%, and 1.6%, respectively.Source: C Example of DMRA Calculations · p. 17
• Most failure was attributed to relational reasoning, followed by Figure 2 memory and knowledge; Figure 1 tasks had minimal impact.Source: C Example of DMRA Calculations · p. 18

D Examples of Relational Questions

• The examples span hierarchical, comparative, cause-effect, spatial-contextual, structural-functional, analogical, and “Other” relations across diverse academic domains.The section presents examples from Psychology, Economics, Calculus, Astronomy, Physics, Behavioral Neuroscience, Biology, Chemistry, and Organic Chemistry.Source: D Examples of Relational Questions · p. 18, Figure 10 · p. 18, Figure 12 · p. 19
• A Psychology example asks a hierarchical relational question, while a Physics example asks how one figure’s structure supports what another describes.The Physics example concerns the electric field outside an infinite conducting plate with surface charge density on each face.Source: Figure 10 · p. 18, D Examples of Relational Questions · pp. 18–19, Figure 11 · p. 18
• Economics and Calculus examples use comparative questions about exchange-rate drivers and what two figures emphasize about a function at the origin.The Economics question contrasts economic drivers of exchange-rate shifts; the Calculus question concerns f(x, y)=xy/√(x^2+y^2) at the origin.Source: D Examples of Relational Questions · pp. 19–20, Figure 12 · p. 19, Figure 14 · p. 20
• Cause-effect examples include an Astronomy question and a Physics question about how a system explains water-pressure differences during heavy use.The Astronomy example is identified by its figure caption; the Physics example asks how Figure 2 uses the system shown in Figure 1.Source: Figure 13 · p. 19, D Examples of Relational Questions · p. 22, Figure 18 · p. 22
• Other examples include an Organic Chemistry “Other” question, a Biology structural-functional question, and a Chemistry analogy question about a shared principle.The captions identify the Organic Chemistry and Biology relation types; the Chemistry question asks what principle connects two figures.Source: Figure 15 · p. 20, Figure 17 · p. 21, D Examples of Relational Questions · p. 22
• A Behavioral Neuroscience spatial-contextual example asks how one figure adds detail to a brain region identified in another.The accompanying text locates a gunshot wound within the ventromedial prefrontal cortex, identified as the orbitofrontal region above the eye sockets.Source: D Examples of Relational Questions · p. 21, Figure 16 · p. 21

E Prompt and Templates

• Figures 20–22 show prompts for extracting textbook relations, validating relationships, creating ground truth, and selecting relation categories.Source: Figure 20 · p. 23, E Prompt and Templates · p. 23, Figure 21 · p. 24
• Figures 23–24 provide prompts for creating template-based visual questions and knowledge-based questions.Source: E Prompt and Templates · p. 23, Figure 23 · p. 26, Figure 24 · p. 27
• Figures 25–26 show prompts for creating template-based relational reasoning questions.Source: E Prompt and Templates · p. 23, Figure 25 · p. 28, Figure 26 · p. 29
• Figure 27 presents the model-evaluation prompt, including chat history and question-specific images.Source: E Prompt and Templates · p. 23, Figure 27 · p. 29
• Figure 28 describes rubric-based scoring of relational reasoning answers against ground truth on a 1–10 scale.Source: E Prompt and Templates · p. 23, Figure 28 · p. 30
• Figure 29 describes rubric-based scoring of other question types against ground truth on a 1–10 scale.Source: E Prompt and Templates · p. 23, Figure 29 · p. 31

E.1 Question Templates

• Figures 30, 31, and 33 provide templates for visual questions.Source: Figure 30 · p. 32, Figure 31 · p. 33, Figure 33 · p. 35
• Figure 32 provides the template for relational questions.Source: Figure 32 · p. 34