PaperlayerView exact revision on arXiv

Source-linked AI summary

MUSE: Benchmarking Large Vision-Language Models on Multi-Modal Understanding in Situated Education

Luyao Zhu, Xun Wei Yee, Wei Li, Mun Thye Mak, Wee Siong Ng

arXiv:2609.19088v1cs.AIcs.CLcs.CV

TL;DR

Artistic educational imagery requires semantic, affective, cultural, and contextual understanding, but existing benchmarks offer limited coverage of these capabilities. MUSE reuses structured annotations to generate 12 controllable tasks, and evaluations find uneven performance across capabilities, with particular weaknesses in grounding, affective interpretation, and compositional reasoning.

  • Problem

    Existing benchmarks provide limited joint assessment of the artistic, affective, and culturally situated understanding needed for image-based educational interaction.

    Source: §1 · p. 2

  • Method

    MUSE reuses structured visual-semantic image annotations to generate 12 tasks with controlled question formats and difficulty across multiple capability dimensions.

    Source: §1 · p. 3

  • Results

    Models show task-dependent performance: scene and activity recognition are comparatively reliable, while visual grounding, affective interpretation, and compositional reasoning remain limited.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    MUSE analysis indicates that stronger coarse recognition alone is insufficient; reliable educational VLMs require region-aware grounding and integrated reasoning from perception to evidence and causes.

    Source: §5 · p. 11

Abstract

from arXiv · show

Large vision-language models have achieved remarkable progress in multi-modal understanding, yet their capabilities in educational settings remain insufficiently evaluated. In AI-assisted language learning, models must interpret artistic imagery, understand its semantic, affective, and cultural content, and reason about visual context to support meaningful interaction. However, existing benchmarks primarily focus on real-world images or domain-specific educational reasoning, providing limited coverage of artistic educational content. To address this gap, we introduce MUSE, a benchmark for evaluating large vision-language models on artistic image understanding in situated educational applications. MUSE decouples image annotation from question generation, enabling diverse tasks with controllable difficulty while reducing annotation effort. It comprises twelve tasks spanning visual perception, semantic and affective interpretation, culture understanding, and compositional reasoning, together with diverse artistic images deliberately curated to center Singaporean and Southeast Asian multicultural contexts alongside Western art traditions, covering multiple themes and difficulty levels. Evaluation of open-source and proprietary models reveals substantial disparities across capability dimensions, particularly in affective interpretation and compositional reasoning. Our analysis further identifies common failure modes and key challenges for developing trustworthy multi-modal models for education. We hope MUSE will serve as a standardized benchmark for advancing multi-modal understanding in situated educational applications.

1 Introduction

MUSE addresses the gap between general VLM evaluation and the artistic, educational understanding needed for image-based learning. It reuses structured image annotations to generate controllable tasks, and evaluations reveal uneven model capabilities, especially in grounding, affective interpretation, and compositional reasoning.

1 Introduction

  • Artistic educational imagery requires abilities that existing benchmarks do not jointly assess, including semantic, affective, spatial, compositional, and cultural understanding.Artworks often use stylized forms, non-photorealistic colors, implicit narratives, and culturally dependent cues, so natural-image benchmark performance may not reliably reflect this ability.Source: §1 · pp. 2–3
  • MUSE separates reusable visual-semantic annotations from task-specific question generation, enabling consistent semantics and explicit control over question format and difficulty.The construction is intended to improve scalability, consistency, and controllability while reducing repeated annotation effort.Source: §1 · p. 3
  • The benchmark evaluates five dimensions of multimodal understanding through 12 tasks and combines multiple-choice, numerical, and open-ended responses.Tasks such as visual-clue identification and emotion-cause inference test whether models can ground and articulate understanding rather than only select a correct option.Source: §1 · p. 3
  • Evaluations reveal task-dependent capability gaps: models’ success on general benchmarks or coarse recognition does not reliably transfer to artistic imagery and fine-grained evidence-based reasoning.The reported gaps are particularly pronounced in visual grounding, affective interpretation, and compositional reasoning.Source: §1 · p. 3

2 Related Work

Existing multimodal benchmarks cover broad perception and reasoning, while artistic, affective, and cultural resources generally target individual domains. MUSE jointly evaluates these abilities in artistic imagery and reuses annotations across tasks.

2 Related Work

  • General VLM benchmarks focus on natural images, diagrams, charts, documents, or exams, with limited coverage of artistic stylization, implicit narratives, affective evidence, and culturally situated meanings for language learning.Examples include general multimodal assessment, expert reasoning, and scientific or mathematical benchmarks.Source: §2 · p. 3
  • Artistic, affective, and cultural benchmarks advance individual domains, whereas MUSE jointly evaluates perception, activity and scene understanding, affective evidence and causes, spatial and compositional reasoning, and cultural understanding.The cited resources include ArtEmis, ArtELingo, VQArt-Bench, AICA-Bench, and CVQA.Source: §2 · p. 3
  • MUSE’s decoupled construction reuses annotations across tasks, reducing annotation effort while controlling question formulation and difficulty.Source: §2 · p. 3

3 MUSE Benchmark

MUSE contains 2,400 questions over 1,174 artistic images, organized across 12 tasks, three cognitive-complexity levels, and three spatial granularities. Its annotation and question-generation process supports diverse tasks with quality review and controlled question design.

3 MUSE Benchmark

  • MUSE contains 2,400 questions over 1,174 images and 12 tasks, spanning three cognitive-complexity levels and three spatial granularities.Most tasks use textual or visual multiple-choice questions; Object Count uses numerical prediction, while Visual Clue Identification and Emotion Cause Inference use open-ended responses evaluated by semantic similarity.Source: §3 · p. 4

3.1 Dataset Annotation and Quality Control

  • Each sample is annotated independently by one of 127 annotators, reviewed by two others, and finalized after consensus resolves disagreements.The annotations cover boxes, labels, descriptions, scene and cultural information, counts, visual clues, and emotion causes.Source: §3.1 · p. 4

3.2 Question Generation

  • Question generation enforces diversity in artistic styles, scene themes, and difficulty; four affective tasks form a dialogue sequence that retains earlier questions and answers.The sequence comprises Object Classification, Emotion Detection, Visual Clue Identification, and Emotion Cause Inference.Source: §3.2 · p. 4
  • The affective sequence progresses from character and emotion classification to open-ended descriptions of visual evidence and explanations of emotion causes.Emotion Detection uses categories following Plutchik’s emotion wheel, excluding emotions described as rare or difficult to depict visually.Source: §3.2 · pp. 5–6
  • Activity Localization tests selection of an activity’s box, Activity Description tests understanding within boxes, and Cultural Identification tests recognition of cultural elements within a box.Source: §3.2 · p. 6
  • Jigsaw Puzzle tests patch alignment by shape, color, and texture; Object Count tests counting under occlusion and appearance variation; Relative Position targets viewpoint-based spatial relations.Remote Interaction asks models to identify non-contact interactions and supporting evidence, while Scene Classification integrates global visual and semantic information.Source: §3.2 · p. 6

4 Experiments

Across 30 VLMs, performance varies substantially by task: models handle recognition better than grounding, affective interpretation, and compositional reasoning. Correlation and error analyses show related but non-redundant capabilities, uneven transfer from existing benchmarks, and recurring grounding and spatial-reasoning failures.

4.1 Main Results

  • GPT-5.6-Sol achieves the strongest overall results and surpasses GPT-4o on 10 of 12 tasks, but no model dominates across all capabilities.Qwen3-VL-32b leads Activity Localization and Scene Classification; InternVL3-14b and GLM-4V-9b perform best on Relative Position and Jigsaw Puzzle, respectively.Source: §4.1 · pp. 7–8, Table 2 · p. 7
  • Scene Classification has a median score of 81.0, while Emotion Detection, Relative Position, Remote Interaction, and Jigsaw Puzzle have substantially lower medians.Twenty-three of 30 models score above 75.0 on Scene Classification.Source: §4.1 · p. 8
  • Visual grounding, affective interpretation, and compositional reasoning remain pronounced gaps across the evaluated models.Model families share strengths in scene and activity recognition but diverge on Activity Description, Remote Interaction, and Jigsaw Puzzle; scaling is non-monotonic on some tasks.Source: Figure 5 · p. 7, §4.1 · p. 8

4.2 Inter-Task Correlation Analysis

  • Object Count, Emotion Detection, and Scene Classification are strongly correlated, while Visual Clue Identification tracks Emotion Cause Inference and Jigsaw Puzzle correlates weakly with most tasks.The task relationships indicate related but non-redundant dimensions of multimodal understanding.Source: §4.2 · p. 8, Figure 6 · p. 8

4.3 Complementarity to Existing Benchmarks

  • Existing benchmarks explain only part of model variation on MUSE: MMBench and AI2D correlate strongly with several tasks, while BLINK correlations are inconsistent.AICA-Bench Emotion Reasoning aligns moderately with MUSE affective tasks, suggesting only partial transfer to artworks.Source: §4.4 · p. 9, Figure 7 · p. 9

4.4 Cross-Task Error Analysis

  • In Activity Localization, models often select relevant people or activities but miss the complete target extent; Remote Interaction errors commonly mismatch regions and text.Activity Description errors favor co-occurring or concatenated activities, reflecting difficulty separating the queried event from nearby visual semantics.Source: §4.4 · p. 9, Figure 8 · p. 10

4.5 Affective Computing Analysis

  • Rankings shift across affective tasks, and performance in character recognition or emotion classification does not reliably transfer to visual-evidence grounding or emotion-cause inference.Emotion Detection is the clearest bottleneck; current models lack integrated affective reasoning from recognition through evidence to causal explanation.Source: §4.5 · p. 9

4.6 Visual Grounding is the Prerequisite of Accurate Affective Interpretation

  • Artistic images can make affective reasoning depend on jointly resolving target identity, spatial structure, posture, interactions, and scene context.In one case, a model identifies the target but misreads his stylized expression; other models shift attention to a salient child and explain the wrong character's affect.Source: §4.6 · pp. 9–10, Figure 8 · p. 10

5 Conclusion

MUSE evaluates multimodal understanding of artistic imagery for image-based language learning and educational interaction. Its analyses identify task-dependent model performance and recurring grounding and spatial-reasoning failures.

5 Conclusion

  • MUSE decouples reusable visual-semantic annotations from question generation, supporting 12 tasks across five capability dimensions with control over question format and difficulty.Models recognize scenes and activities more reliably than they ground evidence, interpret affect, or reason compositionally.Source: §5 · p. 10
  • Correlation analyses find that MUSE measures related but non-redundant capabilities and complements general-purpose and emotion-reasoning benchmarks.Error analyses identify failures in region-activity binding, entity-evidence alignment, target grounding, and calibration under ambiguous spatial relations.Source: §5 · pp. 10–11
  • Scaling or stronger coarse recognition alone is insufficient for reliable educational VLMs; the paper identifies region-aware grounding, integrated reasoning, and viewpoint-aware uncertainty modeling as needed capabilities.MUSE provides a foundation for measuring progress on artistic and culturally situated imagery.Source: §5 · p. 11

A.1 Human-Model Comparison

Human answers form the outer performance envelope on nearly all 12 MUSE tasks, showing a substantial gap from current VLMs. The largest deficits occur in Relative Position, Emotion Detection, and Jigsaw Puzzle.

A.1 Human-Model Comparison

  • Humans outperform current VLMs on nearly all 12 MUSE tasks, with the largest gaps in Relative Position, Emotion Detection, and Jigsaw Puzzle.The comparison used 20 randomly sampled questions per task answered by two annotators and models; gaps are narrower for Activity Localization, Object Count, Remote Interaction, and Scene Classification.Source: §A.1 · p. 14, Figure 11 · p. 14

A.2 Performance across Taxonomy Dimensions

Performance varies across MUSE capability, difficulty, and granularity dimensions. Semantic and cultural understanding and image-level understanding are stronger than affective, compositional, and pixel-level performance.

A.2 Performance across Taxonomy Dimensions

  • Semantic and cultural understanding are generally stronger than affective interpretation and compositional visual reasoning, and performance tends to decrease from low- and mid-level tasks to high-level reasoning.Image-level understanding is strongest across spatial granularities, pixel-level understanding weakest, and crop-level performance intermediate.Source: §A.2 · p. 14, Figure 12 · p. 15

A.3 Invalid Response Analysis

Invalid-response rates vary sharply across models, and response-format reliability is a distinct concern from answer correctness.

A.3 Invalid Response Analysis

  • Yi-VL-6b and DeepSeek-VL2-Tiny exceed 12% invalid responses, while most models remain below 1%.Several models produce no invalid responses.Source: §A.3 · p. 16
  • GPT-5.6-Sol retains a 2.62% invalid-response rate despite strong task performance.Source: §A.3 · p. 16
  • Invalid-response rates vary substantially within model families and are not determined solely by model scale.Source: §A.3 · p. 16
  • Response-format reliability is a distinct evaluation concern alongside answer correctness.Source: §A.3 · p. 16

B Examples for Selected Tasks

The example presents a Cultural Identification question alongside its corresponding image, with a red box to aid interpretation.

B. Examples for Selected Tasks

  • The example pairs a Cultural Identification question with its corresponding image.Source: §B.1 · p. 16
  • A red bounding box is included solely to facilitate interpretation.Source: §B.1 · p. 16
  • The red bounding box is not shown to VLMs during inference.Source: §B.1 · p. 16

Question

These examples show how Cultural Identification, Jigsaw Puzzle, and affective-computing questions are posed, including their response constraints.

Question

  • Cultural Identification asks models to select the culture most relevant to content inside a bounding box from four options.The example options are Western, China, Europe, and Muslim; the response must be one letter without explanation.Source: Question · pp. 16–17, Figure 14 · p. 17

B.2 Jigsaw Puzzle

  • Jigsaw Puzzle asks models to choose the candidate piece that best completes an image with a missing region.Exactly one option is correct, and the answer must be a single uppercase letter with no explanation.Source: §B.2 · p. 17, Question · p. 18, Figure 15 · p. 18

B.3 Affective Computing

  • The affective-computing example presents a four-turn sequence covering object classification, emotion detection, visual-clue identification, and emotion-cause inference.Source: §B.3 · p. 18

Question - Object Classification

The examples define prompts for object and emotion recognition, visual evidence, and emotion causes, alongside inference settings and annotation-process details.

Question - Object Classification

Question - Emotion Detection

  • Emotion Detection asks models to identify the person’s emotion within the same bounding box and return a single option letter.The example options range from guilt and confusion to joy, anger, and fear.Source: Question - Emotion Detection · p. 19

Question - Visual Clue Identification

Question - Emotion Cause

  • Emotion Cause asks models to infer the most likely cause from the image, bounding box, and previously supplied visual clues.Source: Question - Emotion Cause · p. 20

C Model Hyperparameters

  • Inference uses BF16 for most evaluated model families and FP16 for LLaVA-NeXT, with CogVLM2 falling back to FP16 when hardware lacks BF16 support.The default generation configuration disables sampling and sets max_new_tokens to 1024; other unspecified settings use model or library defaults.Source: C Model Hyperparameters · p. 20, Table 3 · p. 21, Table 4 · p. 21

D Annotation Process

  • Annotators received a 0.5-hour guidelines-based training, and pilot feedback was incorporated to clarify the instructions.The study recruited 127 undergraduate and postgraduate student annotators.Source: D Annotation Process · p. 20
  • Annotators were informed about the dataset’s purpose, personal data collection was limited to compensation and quality control, and sensitive annotations were carefully reviewed.Source: D Annotation Process · p. 20

Related papers

Loading 2609.19088v1…