Source-linked AI summary
SciReC: Diagnostic Evaluation of Multimodal, Multi-Turn Relational Reasoning with Adaptive Interaction
Nilay Yilmaz, Naga Sai Abhiram Kusumba, Stella Wenxing Liu, Yezhou Yang
TL;DR
The paper addresses limited evaluation of relational reasoning in multimodal, multi-turn interactions, especially when visual, knowledge, and memory factors jointly affect performance. It introduces SciReC and DMRA to evaluate adaptive academic dialogue and diagnose failure causes. Claude 4.6 achieved the highest overall relational score at 73%, while relational reasoning and memory were the leading error factors.
Problem
Existing multimodal conversation benchmarks are noninteractive and often omit visual, knowledge, and memory conditions that influence relational reasoning evaluation.
Method
SciReC is a model-adaptive multimodal academic dialogue benchmark, and DMRA uses deficit-based analysis to quantify visual, knowledge, memory, and relational reasoning contributions to failures.
Results
Claude 4.6 achieved 73.78% overall accuracy, followed by GPT-5.4 at 68%, while open-source models commonly struggled with spatial relations and proprietary models with sequential or hierarchical relations.
Takeaways & Limitations
Across models, relational reasoning was the primary cause of performance breakdowns, followed by memory-related limitations.
Takeaways & Limitations
The benchmark focuses on academic domains rather than daily-life scenarios, and extending DMRA beyond its three-factor decomposition requires reparameterization.
Abstract
from arXiv · showhide
Relational reasoning requires the process of perceptual understanding, comparing, and integrating the underlying relationships between concepts. This ability consists of multiple categories, such as analogical, structural, and cause-effect, each capturing a different aspect of higher-order understanding. To examine the performance of multimodal large language models (MLLM) on these relational inference tasks, we developed SciReC, a model-adaptive multimodal academic dialog benchmark. As the relational reasoning process involves multiple representations and various factors (visual understanding, exhibiting knowledge, and memory recall), we propose DMRA, a deficit-based diagnostic framework that quantifies the contribution of these components to identify the primary cause of unsuccessful cases. Claude 4.6 achieved the best performance on the overall relational score with 73\%, followed by GPT 5.4 with 68\%. Performance trends indicate that open-source models achieve their lowest scores on spatial relations, while proprietary models struggle more with hierarchical and sequential relations. Across domains, model performance is lowest on Astronomy and highest on Psychology. The results of DMRA reveal that relational reasoning is the primary source of error across all models, followed by memory limitations.
1 Introduction
The paper argues that existing multimodal conversation benchmarks are static and often omit upstream abilities that shape relational reasoning. SciReC addresses these gaps with adaptive academic dialogue and diagnostic analysis across relational categories and contributing factors.
- Existing conversation datasets use static inputs rather than model-dependent turn selection, producing noninteractive evaluation flows.
- Visual understanding, memory, and knowledge integration can each affect relational inference, so evaluating only final reasoning performance may mislead results.
- SciReC evaluates multimodal academic dialogue across eight relational reasoning categories while incorporating visual, knowledge-based, and memory factors.
- The benchmark adapts conversation flow according to model outputs, allowing flexible numbers of images and conversational turns.
- Up to 73% accuracy was achieved by proprietary models and 56% by open-source models, while DMRA identifies relational reasoning as the primary failure factor, followed by memory.
2 Related Work
Prior multimodal multi-turn benchmarks cover diverse reasoning and memory capabilities but generally evaluate fixed conversation sequences and provide limited explicit error analysis. SciReC instead combines adaptive conversational evaluation with relational reasoning across scientific and social-science domains.
- Existing benchmarks address multimodal relations, memory, contextual consistency, co-reference, comparison, reasoning, and temporal understanding.
- SciReC evaluates relational reasoning in multimodal conversations across scientific and social-science domains.
- Most benchmarks evaluate fixed conversational sequences, so model performance does not influence subsequent turns.
- SciReC provides adaptive conversational evaluation over a fixed question pool, allowing subsequent context to depend on model responses.
- Prior error analyses include checklist-based, manual qualitative, pattern-based, and hierarchical approaches, but their coverage is limited or differs from SciReC’s diagnostic framing.
3 SciReC Benchmark
SciReC constructs adaptive multimodal academic dialogues from textbook figures and evaluates relational, knowledge, visual, memory, and memory-validation questions. Its interaction flow uses scored responses to select later questions and support diagnosis.
- Benchmark design: SciReC challenges models to integrate complex concepts across two images in multimodal multi-turn academic conversations.
- Question types: The benchmark includes relational reasoning, knowledge-based, visual understanding, memory retrieval, and memory validation questions.
- Dataset scale: 189 multimodal multi-turn conversations and 656 relational questions were generated from textbook chapters, with additional knowledge, visual, memory, and validation questions for each image.
- Academic coverage: The dataset draws on 12 college-level open-source textbooks spanning biology, chemistry, physics, astronomy, economics, psychology, behavioral neuroscience, and calculus.
- Quality control: Human annotators reviewed relational question relevance, ground-truth accuracy, and visual grounding after model-assisted verification.
- Adaptive dialogue: A model’s response is scored immediately, and a 7/10 threshold determines whether subsequent turns ask memory-retrieval questions.
- Evaluation: Responses use a 1–10 rubric, with scores below 7 classified as partially correct or incorrect.
4 Deficit-Based Multimodal Relational Analysis (DMRA)
DMRA decomposes unsuccessful relational answers into upstream deficits and residual relational errors, then weights and aggregates these contributions to estimate primary failure sources.
- Two-stage error decomposition: DMRA separates upstream deficits from relational-stage errors to diagnose whether failures arise from figure-level weaknesses or cross-figure reasoning.The upstream stage covers visual, knowledge, and memory-related deficits, while the relational stage captures integration and abstraction errors.
- Adaptive task weighting: Memory validation questions isolate memory errors from perception and knowledge errors by measuring the gap between memory and validation performance.The validation gap dynamically adjusts factor weights through softmax, with base weights of 0.33, λ = 0.2, and τ = 0.1 for equal scores.
- Adaptive task weighting: Figure weights normalize each figure’s smoothed total deficit using λfig = 0.5, ensuring balanced and non-zero contributions across figures.Each figure’s contribution combines its weight, task weight, and task deficit before aggregation.
- Relational error attribution: Relational error is the unexplained portion of total failure after explainable upstream contributions are subtracted.When upstream deficits meet or exceed total failure, the outcome is attributed to upstream limitations rather than relational error.
- Failure quantification: The total attributed failure sums upstream and relational contributions, then normalizes each scaled task contribution into a percentage of total failure.This produces a quantitative diagnosis of the weaknesses underlying an incorrect relational answer.
5 Evaluation Results
SciReC evaluation shows substantial variation across models, relational categories, and academic domains. DMRA attributes most failures to relational reasoning, with memory and upstream visual or knowledge deficits contributing differently by model.
- Evaluation setup: The evaluation covers open-source and proprietary models across relational categories and academic domains using SciReC.The reported model set includes GPT-5.4, Claude 4.6, Qwen3.5, Gemma-3-27B, Mistral3, MiniCPM-V 4.5, and InternVL3.
- Overall performance: 73.78% overall accuracy makes Claude 4.6 the best-performing model, followed by GPT-5.4 at 68% and Qwen-3.5 at 56.25%.
- Relational categories: Open-source models generally struggle most with spatial relations, whereas proprietary models show weaknesses in sequential or hierarchical reasoning.Claude 4.6 scores 64% on sequential relations, GPT-5.4 scores 55% on hierarchical relations, and Gemma-3 scores 41% on spatial relations.
- Performance across domains: Astronomy is the most difficult domain for most models, while Psychology achieves the highest accuracy for most models.Claude 4.6 is an exception for Astronomy and performs weakest in Economics at 52%; Mistral3 is an exception for Psychology, reaching 38%.
- Causal analysis: Relational reasoning is the primary source of performance breakdowns for all models except MiniCPM-V-4.5, whose main challenge is memory at 39%.For GPT-5.4, Claude 4.6, and Qwen-3.5, relational-reasoning errors range from 66% to 71%, while MiniCPM-V-4.5 has 30% relational-reasoning errors.
- Causal analysis: Memory is the second main cause for most models at 37–46%, while knowledge gaps become the second leading cause for InternVL3 and MiniCPM-V-4.5, reaching up to 40%.Knowledge and visual deficits vary across models, with stronger models showing lower upstream error rates and other models exhibiting larger knowledge or visual limitations.
6 Conclusion
SciReC evaluates multimodal relational reasoning with upstream memory, knowledge, and visual tasks, while DMRA decomposes failures across these factors. Results identify relational reasoning as the main error source, with memory also limiting performance, but the benchmark and framework have defined scope boundaries.
- SciReC evaluates MLLM relational reasoning in multimodal academic dialogue while covering memory, knowledge, and visual upstream tasks.
- Relational reasoning accounts for 50% to 71% of incorrect responses, followed by memory at 37–46%.
- Proprietary models show less prominent knowledge and visual-understanding errors, whereas these remain significant contributors for most open-source models.
- Limitations: The benchmark covers academic scenarios rather than daily-life settings, and question generation and answer evaluation require API access keys that may add cost.
- Limitations: DMRA currently decomposes errors across three factors, and adding further upstream factors would require reparameterization.
- SciReC data derives from OpenStax textbooks under CC BY-NC-SA 4.0 and is released under the same license.
A Dataset Details
SciReC contains flexible-length academic dialogues built from textbook chapters, figures, and extracted relations across eight scientific and social science domains. Its creation pipeline filters generated question-answer pairs through repeated human evaluation, while model use and deployment involve computational and oversight considerations.
- Dataset composition: SciReC contains 656 relations across eight subject categories and 189 conversations, with dialogue length determined by model performance and at least three turns.
- Dataset composition: Biology has the most dialogues with 43 (22.6%), while Psychology has the fewest with 5 (3.2%).
- Implementation: Open-source model evaluation uses an A100 GPU, takes approximately 15–18 hours per model, and uses a 4096-token model context length.
- Societal impacts: Automated pipelines such as tutoring may improve reliable educational AI, but the paper says human oversight remains necessary to avoid misleading information.
B Evaluation Model Selection
The authors compare Claude 4.6 and GPT 5.4 as online scoring models and use human review of disagreements to select the model most aligned with human assessment. This procedure accounts for conditionally varying dialogue flows and score conflicts.
- Claude 4.6 and GPT 5.4 were evaluated on the same Physics-question subset to select a scoring model aligned with human assessment.
- Question counts vary by type because dialogue flow depends conditionally on model performance.
- Human annotators reviewed 29 conflicting cases using the questions, ground truths, images, scores, and score justifications.
C Example of DMRA Calculations
DMRA computes deficit-based contributions from relational, memory, knowledge, and visual tasks across figures, then converts them into factor weights and failure percentages. In the worked example, relational reasoning is the largest failure component, with Figure 2 contributing most upstream deficit.
- Deficit computation: DMRA begins by computing each task’s deficit as max(0, 7 − score), using 7 as the threshold for memory, knowledge, and visual tasks.
- Task weighting: Figure 1 has no validation gap because its memory score exceeds the threshold, so its three task weights remain approximately 0.33.
- Task weighting: Default task weights are 0.33, then softmax adjusts them according to each task’s deficit.
- Task weighting: For Figure 2, a validation gap of 1.0 raises the memory logit from 0.33 to 0.53 and produces weights of 0.3792 for memory and 0.3104 each for knowledge and visual.
- Figure weighting: Figure weights use λ = 0.5 to preserve contributions from other figures, and Figure 2 accounts for 95% of total upstream deficits.
- Total deficit: The framework sums figure-level task deficits, caps upstream total deficit at 5, and treats the remainder as unexplained relational error.
- Final contributions: The worked example assigns Figure 2 deficits of 1.557 for memory, 1.19 for knowledge, and 0.34 for visual, while Figure 1 has minimal contributions.
- Final contributions: Most failures are attributed to relational reasoning, followed by Figure 2 memory and Figure 2 knowledge.
D Examples of Relational Questions
The section illustrates how SciReC instantiates relational questions across academic domains and relation types, using paired figures to elicit comparisons, structural links, spatial context, and causal or analogical reasoning.
- SciReC examples span Psychology, Physics, Economics, Astronomy, Calculus, Organic Chemistry, Behavioral Neuroscience, Biology, and Chemistry.
- Comparative relations: Comparative questions ask models to identify contrasts between paired figures, including economic drivers of exchange-rate shifts and mathematical functions.
- Spatial-contextual relations: Spatial-contextual questions connect a broad anatomical region with a more precise location, such as a gunshot wound within the ventromedial prefrontal cortex.
- Other relational types: Other examples probe shared principles, structural support, cause and effect, and analogical relations between figures.
E Prompt and Templates
The appendix documents prompts and templates for extracting, validating, generating, evaluating, and scoring SciReC questions. These materials enforce distinct image pairing, structured question creation, adaptive evaluation, and rubric-based scoring.
- E Prompt and Templates: The appendix lists prompts for relation extraction, validation, ground-truth creation, category selection, question generation, model evaluation, and answer scoring.
- E Prompt and Templates: Textbook inputs contain numbered images with descriptive text, from which meaningful relations between distinct main-image pairs are identified.
- E Prompt and Templates: Relational question templates use named image placeholders and continued prompt instructions to structure questions consistently.
- E Prompt and Templates: The evaluation prompt incorporates chat history and question-specific images for model assessment.
- E Prompt and Templates: Relational answers are scored against ground truth with a rubric ranging from 1 to 10.
- E Prompt and Templates: Answers to other question types are likewise scored against ground truth using a 1-to-10 rubric.
- E.1 Question Templates: Question-template materials separately cover visual questions and relational questions, including continued visual-question templates.